> ## Documentation Index
> Fetch the complete documentation index at: https://runcrate.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Jobs

> Run one command on its own GPU machine, follow its logs, and stop paying the moment it exits.

A job is one command on its own fresh GPU machine. Runcrate starts the machine, runs your command, streams its output, and stops the machine as soon as the command exits. Billing ends with it.

Every job has a maximum runtime (up to 7 days), so a job never runs unbounded. You do not have to remember to shut anything down.

> **[Capacity and GPU lists](https://runcrate.ai/docs/jobs/capacity.md)**
>
> Try several GPUs in order, and wait for one instead of failing.

> **[Retries](https://runcrate.ai/docs/jobs/retries.md)**
>
> Replace a machine that fails and run the command again.

> **[Arrays and sweeps](https://runcrate.ai/docs/jobs/arrays.md)**
>
> Many copies from one request, a few at a time.

> **[Idempotency and callbacks](https://runcrate.ai/docs/jobs/idempotency-and-callbacks.md)**
>
> Retry a submit safely, and get a signed callback when it ends.

## How a job runs

**1. Submit**

You send a command, a GPU type (or a list of them to try in order) and a maximum runtime. The job is `queued` while a machine is requested, and keeps waiting for capacity if none of the GPUs has any yet.

**2. Start**

A fresh machine with that GPU launches and runs its setup (`starting`). If you attached a volume, its files are in `/workspace`. Some machines install the GPU driver after they boot, and the Finetune and Inference templates install PyTorch and vLLM after setup: the job waits for both, so your command starts with the GPU and its packages ready.

**3. Run**

Your command runs under bash in `/workspace` (`running`). Its stdout and stderr are streamed as logs you can follow live.

**4. Stop**

When the command exits, hits its maximum runtime or is cancelled, the machine stops and billing ends. Files under `/workspace` are saved back to the volume.

While the job runs, its machine also shows in **Instances**. Stopping it there cancels the job.

## Run a job

**Ask your agent**

Connect the [Runcrate MCP server](https://runcrate.ai/docs/mcp/setup.md) to Claude Code, Codex, Cursor or any MCP client, then ask in plain words:

```text theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
Run python train.py on an A4000 with the Finetune template for at most 2 hours with my volume training-data, then tell me the final loss.
```

```text theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
Fine-tune with ./scripts/finetune.sh on 2 H100s, cap it at 6 hours, pass my HF_TOKEN, and tell me if it fails.
```

```text theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
What did my last job print before it stopped? Cancel it if it is still running.
```

What the agent does with the first prompt:

1. `list_volumes` to find the id of `training-data`.
2. `run_job` with `gpu: "A4000"`, `template: "ubuntu-train"`, `command: "python train.py"`, `max_runtime_minutes: 120` and the volume's `storage_id`. It gets the job id back.
3. `get_job_logs` with `follow: true`, passing the returned `cursor` back as `after` each time, until `done` is true. It reads the loss from the output.
4. `get_job` for the final status, exit code and cost, then answers you.

For the second prompt the agent passes the token in `run_job`'s `env` object (`{ "HF_TOKEN": "..." }`). For the third it finds the job with `list_jobs`, reads it with `get_job_logs`, and stops it with `cancel_job`. `run_job` also takes `region`, and `workspace` and `environment` to choose where the job runs. See [MCP tools](https://runcrate.ai/docs/mcp/tools.md#jobs) for every parameter.

**See the code**

The CLI submits the job and follows its logs until it ends:

```bash theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
runcrate run --gpu A4000 --template ubuntu-train --volume <volume-id> --max-runtime 2h -- python train.py
```

The same job over the API:

```bash Submit theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
curl -X POST https://www.runcrate.ai/api/v1/jobs \
  -H "Authorization: Bearer $RUNCRATE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "command": "python train.py",
    "gpu_type": "A4000",
    "template": "ubuntu-train",
    "max_runtime_minutes": 120,
    "storage_id": "<volume-id>",
    "env": { "HF_TOKEN": "hf_..." }
  }'
```

```bash Follow logs theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
cursor=0
while :; do
  page=$(curl -s "https://www.runcrate.ai/api/v1/jobs/$JOB_ID/logs?after=$cursor&follow=true" \
    -H "Authorization: Bearer $RUNCRATE_API_KEY")
  echo "$page" | jq -j '.data.chunks[].text'
  cursor=$(echo "$page" | jq '.data.cursor')
  [ "$(echo "$page" | jq '.data.done')" = "true" ] && break
done
```

```bash Status theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
curl https://www.runcrate.ai/api/v1/jobs/$JOB_ID \
  -H "Authorization: Bearer $RUNCRATE_API_KEY"
```

```bash Cancel theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
curl -X POST https://www.runcrate.ai/api/v1/jobs/$JOB_ID/cancel \
  -H "Authorization: Bearer $RUNCRATE_API_KEY"
```

With the [Python](https://runcrate.ai/docs/sdks/python.md#jobs) or [TypeScript](https://runcrate.ai/docs/sdks/typescript.md#jobs) SDK, submit, print the output as it arrives, and fetch the outputs:

```python Python theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import os

from runcrate import Runcrate

client = Runcrate()  # reads RUNCRATE_API_KEY
job = client.jobs.run(
    command="python train.py",
    gpu_type="A4000",
    template="ubuntu-train",
    max_runtime_minutes=120,
    storage_id="<volume-id>",
)
for chunk in client.jobs.follow_logs(job.id):
    print(chunk.text, end="")

job = client.jobs.wait(job.id)
print(job.status, job.exit_code, job.cost_usd)
for artifact in client.jobs.artifacts.list(job.id):
    client.jobs.artifacts.download(job.id, artifact.path, to=os.path.basename(artifact.path))
```

```typescript TypeScript theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import { basename } from 'node:path';
import { Runcrate } from '@runcrate/sdk';

const rc = new Runcrate(); // reads RUNCRATE_API_KEY
let job = await rc.jobs.run({
  command: 'python train.py',
  gpuType: 'A4000',
  template: 'ubuntu-train',
  maxRuntimeMinutes: 120,
  storageId: '<volume-id>',
});
for await (const chunk of rc.jobs.followLogs(job.id)) process.stdout.write(chunk.text);

job = await rc.jobs.wait(job.id);
console.log(job.status, job.exitCode, job.costUsd);
for (const artifact of await rc.jobs.artifacts.list(job.id)) {
  await rc.jobs.artifacts.downloadTo(job.id, artifact.path, basename(artifact.path));
}
```

The SDKs send an `Idempotency-Key` with every submit, so a retried request never starts a second job.

The full request and response shapes are in the [API reference](https://runcrate.ai/docs/api-reference).

## Templates and Python

`template` picks what the machine has installed. In every template `python` is Python 3, and `pip` and `uv` install into that same Python, with PyTorch held to the CUDA 12.8 build the machines' driver runs.

| Template | Installed for your command |
| - | - |
| `ubuntu-devbox` (default) | Python 3, `pip`, `uv`, git and build tools. No PyTorch: `pip install torch` gets the CUDA 12.8 build. |
| `ubuntu-train` | PyTorch 2.8 (CUDA 12.8), torchvision, Transformers, Datasets, TRL, PEFT, Accelerate, bitsandbytes, Hugging Face Hub, W\&B, NumPy, pandas, Matplotlib. |
| `ubuntu-inference` | vLLM 0.11 with PyTorch 2.8 (CUDA 12.8) and Hugging Face Hub. `vllm serve` and `from vllm import LLM` work as is. |

The Finetune and Inference packages install while the job is `starting`, which adds about a minute to setup and nothing to the maximum runtime. On an instance they finish in the background a minute or so after it is up: `systemctl start runcrate-python` returns once they are in.

Installing more is fine: `pip install trl` or `uv pip install --system <package>` keep the installed PyTorch, because `torch`, `torchvision` and `torchaudio` are pinned for every `pip` and `uv` on the machine (a constraint file, `/etc/runcrate/python-constraints.txt`), and on Inference `vllm` and Transformers (below 5) too. A package that needs a newer PyTorch or vLLM is refused rather than replacing them with builds the driver cannot run. To use another PyTorch anyway, run your command in a container [image](#containers) that has it, or, on an instance, install it into a virtual environment without the pins: `PIP_CONFIG_FILE=/dev/null pip install ...` for pip (`--isolated` does not skip the machine-wide config), and `uv --no-config ...` for uv. On an instance, install your own packages once `systemctl start runcrate-python` has returned, so they do not race the template's install.

## Statuses

| Status | Meaning |
| - | - |
| `queued` | Accepted and waiting for a machine. `queue_reason` says why when it waits: `no_capacity`, `budget`, `max_parallel` or `volume_busy` (a retry waiting for its volume to be free), and `next_attempt_at` when it is tried again. A retried job is `queued` between attempts. |
| `starting` | The machine is launching and running its setup, waiting for the GPU driver, and pulling the `image` if you set one. None of this counts against the maximum runtime. |
| `running` | Your command has started. `started_at` is set. |
| `succeeded` | The command exited with code 0. |
| `failed` | The command exited non-zero (`exit_code` is set), or the machine failed or was stopped, or no GPU had capacity within `capacity_wait_minutes` (`exit_code` is null and `error` says why, for example the budget was reached or the balance ran out). With `max_retries`, only once no retries are left. |
| `cancelled` | Cancelled through the API, CLI or MCP, or the machine was stopped from Instances. |
| `timed_out` | An attempt's command hit its maximum runtime. |

`succeeded`, `failed`, `cancelled` and `timed_out` are final: they never change, and `ended_at` is set on all of them.

## Logs

Output is stdout and stderr combined, in order, delivered as chunks. A chunk is a raw fragment of output, not a line: print its `text` as is.

* **Follow with a cursor.** Each response has a `cursor`. Pass it back as `after` to get only what is new. With `follow=true`, a request waits up to 25 seconds for new output or for the job to end.
* **Stop on `done`.** `done: true` means the job has ended and you have read every chunk.
* **Capped at 10 MB per job.** The first 5 MB and the newest output are kept. When the middle is dropped, `truncated` is true and a marker chunk with `seq: null` sits where the gap is.
* **Very fast output.** When a command writes faster than its output can be sent, the newest output is sent first and the part in between is dropped the same way, so a job's final lines are never held back.
* **Kept 30 days** after the job ends.

## Code

Send your code with the job, as an upload or from a git repository, and it is put in `/workspace/code` before your command starts (while the job is `starting`, so it does not use up the maximum runtime). The command then runs in `/workspace/code`, and `RUNCRATE_CODE_DIR` says where it is. The code is never copied to an attached volume.

* **An upload.** `POST /api/v1/jobs/uploads` with `{ "bytes": <size> }` answers `{ upload_id, url, url_expires_at, expires_at }`. PUT a `.tar.gz` of your directory (at most 1 GB) to `url`, then submit with `"code": { "upload_id": "..." }`. An upload belongs to its environment. One no job used is deleted after 24 hours; a job using it keeps it 7 days, and never while a live job uses it. A workspace holds at most 50 uploads and 10 GB of them at a time across its environments, and creates at most 60 an hour (429 past either). Send `sha256` with `bytes` to get the same upload back when you retry. The job reports the upload's SHA-256 in `code.sha256` once it was fetched.
* **A git repository.** `"code": { "git": { "url": "https://...", "ref": "main", "token_env": "GIT_TOKEN" } }`. `ref` is a branch, tag or commit (default: the default branch), fetched shallow. For a private repository, `token_env` names an env variable or secret holding a token; it is sent to the git server as a header, never written into a URL or the logs. The job reports the commit in `code.commit`.
* **From an agent.** The `run_job` tool takes `files` (path to text, up to 1 MB in all) and packs them into an upload, or `code.git`.

If the code cannot be fetched, the command does not run and the job fails, with the reason in its log.

## Outputs

Write results to `$RUNCRATE_OUTPUT_DIR` (`/workspace/outputs`). When your command ends, whatever its status, the files written there while it ran are uploaded and kept 30 days after the job ends: up to 5 GB and 1,000 files a job (the rest is left out, and the log and the timeline say how many). Files already in the directory before the command started (from a volume) are not uploaded again.

* `GET /api/v1/jobs/{id}/artifacts` lists them: `[{ path, bytes, sha256 }]`. The job's `artifacts_count` counts them.
* `GET /api/v1/jobs/{id}/artifacts/{path}` redirects to a download link valid for an hour.
* `job.finished` webhooks carry `artifacts_url`. MCP: `list_job_artifacts`, `read_job_artifact`.
* Deleting a job deletes its outputs.

## Results and volumes

Attach a volume with `storage_id` (CLI: `--volume`) and its files are in `/workspace` when your command starts. Write checkpoints, metrics and outputs under `/workspace`: they are saved back to the volume when the machine stops, whatever the final status. Without a volume or `$RUNCRATE_OUTPUT_DIR`, files on the machine are gone when it stops.

The volume must be in the same workspace and environment as the job, and not attached to another machine. See [Storage](https://runcrate.ai/docs/storage/overview.md).

## Environment variables

Every job gets its environment's secrets, set in the dashboard, as environment variables, along with `RUNCRATE_ENVIRONMENT`, `RUNCRATE_INSTANCE_ID`, `RUNCRATE_JOB_ID`, `RUNCRATE_JOB_ATTEMPT` (1-based, see [Retries](https://runcrate.ai/docs/jobs/retries.md)), and `RUNCRATE_JOB_INDEX`, `RUNCRATE_JOB_COUNT` and `RUNCRATE_GROUP_ID` (see [Arrays](https://runcrate.ai/docs/jobs/arrays.md)). Pass anything else, such as a one-off `HF_TOKEN`, in `env`; a name set in both takes the value from `env`. The values reach your command, are stored only encrypted and are never shown again: the job only reports their names in `env_keys`. They are kept with the job, encrypted, so a [rerun](#find-rerun-and-delete) can use them again. If the machine cannot get them when it starts, the command does not run and the job fails. Sensitive hides a secret's value in Runcrate. Every machine launched into its environments receives it, so anyone who can launch there can read it on the machine.

> [!WARNING]
>
> A job's command and its logs are visible to everyone in the workspace. Put tokens and passwords in secrets or `env`, never in the command itself.

## Containers

Set `image` to run your command inside a container image instead of on the template directly. `/workspace` is mounted, and the image is pulled while the job is `starting`, so the pull does not use up the maximum runtime. The image needs `bash` or `sh`.

The GPUs are passed into the container. A machine without the NVIDIA container toolkit gets it installed during setup; if the GPUs still cannot be passed in, the job fails before the command runs, unless you set `"allow_cpu_container": true`, which runs the container without them.

For a private image, set `"image_registry_auth": { "username_env": "REGISTRY_USER", "password_env": "REGISTRY_TOKEN" }`, naming two env variables or secrets. The machine signs in to the image's registry (the first part of the image name when it is a host, otherwise Docker Hub) before the pull. If the sign-in or the pull fails, the job fails with the reason in its log.

## Metrics and events

While your command runs, the machine samples each GPU's utilization and memory every 15 seconds.

* `get_job` reports `gpu_util_avg` (percent, over the whole run) and `gpu_mem_peak_mb`.
* `GET /api/v1/jobs/{id}/metrics?step=60` answers `[{ at, gpus: [{ util, mem_used_mb, mem_total_mb }] }]`, one point per `step` seconds (60 to 86400): utilization averaged over the step, memory at its peak. Jobs longer than about 16 hours are kept at a coarser step. Kept 30 days after the job ends.
* `GET /api/v1/jobs/{id}/events` is the job's timeline, oldest first: `[{ at, kind, detail, attempt }]`, with kinds `created`, `queued`, `launch_attempt`, `launch_failed`, `machine_up`, `driver_wait_start`, `driver_ready`, `code_fetch_start`, `code_fetch_end`, `image_pull_start`, `image_pull_end`, `started`, `exited`, `retrying`, `artifacts_uploaded` and `stopped`. Use it to see where a job spent its time, or why it failed before its command ran.
* MCP: `get_job_metrics`, `get_job_events`.

## Workspaces and environments

Jobs belong to a workspace and an environment, like instances. An API key or agent session sees only the jobs in its own environment. Otherwise, pick one with the `X-Environment` header or the `environment` query parameter (MCP: the `environment` parameter).

Every job carries `dashboard_url`, its page in the dashboard: the **Jobs** page, opened in the job's workspace and environment, where you watch its steps, logs and outputs live. A job's machine belongs to the job: it never shows on the **Instances** page or in `GET /api/v1/instances`.

## Billing

A job is billed at its machine's hourly rate (`cost_per_hour`, in dollars) from when the machine is running until it stops. Setup time is included; waiting for capacity costs nothing. `cost_usd` on the job is the cost so far, every attempt added up, and is final once the job ends. See [pricing](https://runcrate.ai/docs/billing/pricing.md) for current rates per GPU-hour.

The maximum runtime is counted from when your command starts, for each attempt, so setup does not use it up.

## Find, rerun and delete

* **Tags.** Give a job `tags`, such as `{ "team": "ml", "run": "sweep-3" }` (at most 20; keys are lower case letters, digits and `_ . : -`, values up to 100 characters).
* **Search.** `GET /jobs` filters by `status`, `tag=key:value` (repeat it: a job must have every one), `q` (part of the name), `group_id`, `gpu_type` and `created_after`. MCP: `list_jobs` with `tag`, `q` and `group_id`.
* **Rerun.** `POST /jobs/{id}/rerun` runs a job again as a new job with any field changed, for example `{ "gpu_count": 2 }`. Its env values are reused without you sending them again, for 30 days after it ends; `env` in the body is merged over them. Only whoever submitted the job reuses them: anyone else sends every `env` value again. The new job has `rerun_of`. MCP: `rerun_job`.
* **Delete.** `DELETE /jobs/{id}` removes a job that has ended, with its logs and outputs (`204`). A job that has not ended is a `409`: cancel it first. MCP: `delete_job`.

## GPU readiness

Your command starts once the GPU driver answers. On machines that install the driver after they boot, the job stays `starting` while it waits, for up to 20 minutes. If the driver is still not ready after that, the command runs anyway, the job log says so, and GPU calls in it will usually fail, so the job usually ends `failed`. With [retries](https://runcrate.ai/docs/jobs/retries.md) on, a machine whose setup never finishes is replaced; otherwise rerun the job, which gets a fresh machine.

## Limits

| Limit | Value |
| - | - |
| Maximum runtime | 1 minute to 7 days per attempt, required on every job |
| GPUs per job | 1 to 8 |
| GPU list | Up to 8 GPUs to try |
| Capacity wait | 0 to 24 hours (default 30 minutes) |
| Retries | Up to 10 |
| Jobs per request | Up to 100 (`count` or `params`) |
| Live jobs per workspace | 500 queued, starting or running at once, across all its environments |
| Tags | 20 per job |
| Idempotency-Key | Unique per environment for 24 hours |
| Command | About 5 KB of ordinary shell (put longer scripts in a file on a volume or in the image and run that) |
| Environment variables | 200 keys, 32 KB per value, 64 KB in total with the environment's secrets |
| Log size | 10 MB per job (first 5 MB and newest output kept) |
| Log retention | 30 days after the job ends |
| Log long-poll | Up to 25 seconds per request |

## Errors

| Code | Status | When |
| - | - | - |
| `validation_error` | 422 | A field is missing or out of range. |
| `BUDGET_REACHED` | 402 | The environment's monthly budget is spent, or a group could cost more than what is left of it. Checked before any machine is requested. |
| `insufficient_credits` | 402 | Your balance cannot cover the job. |
| `PAYMENT_HOLD` | 402 | Launches are paused while a payment on the workspace is disputed. Contact [billing@runcrate.ai](mailto:billing@runcrate.ai). |
| `bad_request` | 400 | The volume cannot be used, for example it is attached to another machine. |
| `not_found` | 404 | The job, group or volume does not exist in this workspace and environment. |
| `conflict` | 409 | You cancelled or deleted a job that already ended or has not ended, another launch claimed the volume first, or an `Idempotency-Key` was used with another request. |
| `rate_limited` | 429 | The workspace already has 500 jobs queued, starting or running across its environments. |
| `launch_refused` | 502 | The machine was refused for this request, not for lack of capacity (the message names the reason: `request_conflict` or `launch_script_rejected`). The job is `failed` at once rather than queued, since the same request fails the same way. |
| `service_unavailable` | 503 | With `capacity_wait_minutes: 0`, no machine of the GPUs asked for is available right now; otherwise jobs are briefly unavailable. Retry after the `Retry-After` seconds (60). |

With `capacity_wait_minutes: 0`, if the machine cannot be launched after the job was created, the error carries `details.job_id` and that job is `failed`. Otherwise the job is answered `queued` and waits.

## When to use an instance instead

A job fits work that starts, runs and ends on its own: training runs, evaluations, batch inference, data processing. Use an [instance](https://runcrate.ai/docs/instances/overview.md) when you need to work on the machine yourself (SSH, Jupyter, VS Code), or keep a server up for others to call.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.