Skip to main content
The cluster page shows every command on this page with your cluster’s own addresses filled in, and GET /clusters/{id} returns them under connection. Addresses and SSH commands appear once every machine is up and billed.

​
SSH

Node 0 is the head. Each node is also an instance of its own, so the web terminal and VS Code work on every node from the cluster page, and the instance tools (ssh_execute, file_upload, file_download, /instances/{id}/exec) take a node’s instance_id.
The CLI signs a short-lived SSH certificate for the node each time, so no key is needed. If you added your own SSH key when creating the cluster, each node’s ssh_command works with it too. On a machine with no public address of its own, ssh_command goes through a Runcrate gateway, with its own host and port; everything else works the same. Between the nodes, root can ssh node-1 without a password. That key is only accepted from the cluster’s private network.

​
The hostfile and launcher settings

/shared/hostfile has one line per node with its GPUs:
Container machines share no folder, so there the hostfile is at /etc/runcrate/hostfile on every node. Every login shell has MASTER_ADDR (the head on the private network), MASTER_PORT (29500), NODE_RANK, WORLD_NODES, GPUS_PER_NODE, and NCCL_SOCKET_IFNAME and GLOO_SOCKET_IFNAME set to the private network, from /etc/runcrate/cluster.env.

​
torchrun

Run the same command on every node. From the head, this starts it on each node over the cluster’s own SSH. Commands over SSH don’t start a login shell, so each loads its settings first; the rendezvous gives every node its rank:

​
mpirun

From the head, one process per GPU across the hostfile:

​
DeepSpeed

DeepSpeed reads the same hostfile and starts every node over SSH from the head:
With Slurm, Kubernetes or Ray, see Cluster software for sbatch, kubectl and ray job submit.

​
Shared files

  • On virtual machines and bare metal, /shared is on the head and mounted on every node over the private network. Put code, configs and checkpoints your nodes share there; keep large datasets on each node’s local disk when you can.
  • Every node can read and write everything in /shared, whichever node made it. Files written from the other nodes show as owned by nobody, which changes nothing for root.
  • With a volume attached, its files are in /shared/volume and are synced back to the volume, so they outlive the cluster.
  • Container machines share no folder: each node keeps its own disk, a volume shows on the head only, and you copy files between nodes over SSH.
  • Everything on the machines, /shared included, is deleted when the cluster is terminated.

​
Test the network

The NCCL test runs an all-reduce across every GPU on every node, from 1 MB to 1 GB messages, and reports the bus bandwidth for each size and the peak. It takes a few minutes and stops after 10. One test runs at a time per cluster, and it uses every GPU, so run it before your job. On a slow link it stops after 5 sizes and says so, since the larger sizes would take many minutes of billed time.
  • Dashboard: Run NCCL test under Network test on the cluster page.
  • Agent: run_cluster_nccl_test, then get_cluster for last_test.
  • CLI: runcrate clusters test train.
  • API: POST /clusters/{id}/tests, then GET /clusters/{id}/tests/{testId}.
Right after a cluster becomes ready the test tools may still be preparing in the background. If the test says so, try again a few minutes later.

​
Tunnels

Kubernetes and Ray are never open to the internet. runcrate clusters tunnel train forwards them to your computer over SSH to the head: The cluster’s connection.snippets has both tunnels as plain ssh -N -L commands (kubectl_tunnel, ray_tunnel).