Skip to main content
Long jobs meet the occasional bad machine: the GPU driver never comes up, the machine goes silent, or it stops. With max_retries, Runcrate replaces the machine and runs the command again, so a multi-day training run is not lost to one failure.

​
Turn it on

What counts as a machine failure: the machine stopped or disappeared before the command finished, it never finished setting up, or Runcrate lost contact with the job for 10 minutes. A job that hit its maximum runtime, was cancelled, or was stopped by a budget or a low balance is never retried.

​
What an attempt is

  • Each attempt runs on a new machine, with the whole max_runtime_minutes: the maximum runtime counts per attempt.
  • The volume is saved back when each attempt’s machine stops, so the next attempt starts with the files the last one wrote. Write checkpoints under /workspace and resume from them.
  • The command sees RUNCRATE_JOB_ATTEMPT (1, 2, …).
  • Between attempts the job is queued again, and attempt goes up by one. It waits for capacity like a new job, up to capacity_wait_minutes.
  • The logs go on in one stream. A line marks where each new attempt starts:
Only the last attempt’s outcome is the job’s: job.finished goes out once, when the job really ends.

​
Cost

cost_usd adds up every attempt’s machine. attempts on the job lists each one with its machine, status, exit code, error, times, GPU, region and cost:

​
Ask your agent

The agent calls run_job with max_retries: 3 and the volume’s storage_id, and tells your script to resume from the last checkpoint.