Skip to main content
One request can start a group of jobs: the same command on many machines, each with its own index or its own settings. Use it for batch inference over shards, evaluations, and hyperparameter sweeps.

​
count: shards

Each job sees RUNCRATE_JOB_INDEX (0 to count - 1), RUNCRATE_JOB_COUNT and RUNCRATE_GROUP_ID.

​
params: a sweep

One job per entry. Each entry is merged over env, so the third job gets LR=0.001 and BATCH=128.

​
max_parallel

At most max_parallel jobs of the group are on machines at once (default: all of them). The others stay queued with queue_reason: "max_parallel" and start as earlier ones end. Every job also waits for capacity and retries on its own; its capacity wait starts when max_parallel lets it go.

​
What comes back

The answer is the group with its jobs:
Jobs are named after the group with their index (jobs-8d0b1c2e-3), or name-3 when you give a name.

​
Budget

A group is checked against the environment’s monthly budget before any job starts, by what it could cost: count x the lowest rate of its GPUs x max_runtime_minutes. If that is more than what is left of the budget, the request is refused with BUDGET_REACHED and nothing starts. Each launch is checked again as usual. A volume can be attached to one machine at a time, so storage_id cannot be used with count or params.

​
Ask your agent

The agent calls run_job with params: [{"CKPT": "1"}, ..., {"CKPT": "8"}] and max_parallel: 4, follows each job’s logs, and uses get_job_group to know when all are done.