Multilingual video dubbing

A product video in one language reaches a fraction of its audience. Dubbing is a timing problem more than a translation problem: each line has to fit the seconds the original took. This pipeline translates with the seconds in the prompt, voices each line, speeds a long line up by at most 35%, and then listens to its own work: it transcribes every dubbed line and flags the ones that do not say what they should. Two languages for a 7-second ad cost 2 cents.

Build it in your project

Paste this line into your agent (Claude Code, Codex, Cursor, OpenCode, Cline). It sets up your key, then builds the pipeline in your project:

Add Runcrate's Qwen3 TTS to this project: run curl -fsSL "https://runcrate.ai/agents.md?model=Qwen%2FQwen3-TTS&intent=build" and follow it. Then build: a script that dubs a video into the languages given (es,de...): whisper-large-v3-turbo segments, gpt-oss-120b translation that fits each segment's seconds, Qwen3-TTS per line fitted with atempo, a back-transcription check that flags lines under 85% word match, subtitles and an ffmpeg mux.

The code

One file per language. Both read RUNCRATE_API_KEY from the environment; nothing secret is in the code.

"""Dub a video into other languages: transcribe, translate to fit the timing, voice each line, check the dub by ear, mux.

pip install openai      (and ffmpeg on PATH)
RUNCRATE_API_KEY=... python video_dubbing.py video.mp4 es,de
"""
import difflib, json, os, re, subprocess, sys, time
from pathlib import Path

from openai import OpenAI

client = OpenAI(base_url="https://api.runcrate.ai/v1", api_key=os.environ["RUNCRATE_API_KEY"], max_retries=4, timeout=180)
ASR = "openai/whisper-large-v3-turbo"
TRANSLATOR = "openai/gpt-oss-120b"          # on the free tier
TTS, VOICE = "Qwen/Qwen3-TTS", "Serena"      # speaks the language of the text it is given
LANGS = {"es": "Spanish", "de": "German", "fr": "French", "it": "Italian", "pt": "Portuguese", "ja": "Japanese", "ko": "Korean", "zh": "Chinese"}
OUT = Path("out"); OUT.mkdir(exist_ok=True)
ledger = []

def log(step, model, t0, cost):
    ledger.append({"step": step, "model": model, "seconds": round(time.time() - t0, 2), "cost_usd": cost})
    print(f"{step:<14} {model:<34} {time.time() - t0:7.2f}s  ${cost:.6f}")

def ffmpeg(*args):
    subprocess.run(["ffmpeg", "-y", "-loglevel", "error", *args], check=True)

def duration(path: Path) -> float:
    return float(subprocess.run(["ffprobe", "-v", "error", "-show_entries", "format=duration", "-of", "csv=p=0", str(path)], capture_output=True, text=True).stdout)

def srt_time(t: float) -> str:
    return f"{int(t // 3600):02d}:{int(t % 3600 // 60):02d}:{int(t % 60):02d},{int(t * 1000 % 1000):03d}"

def words(s: str) -> list[str]:
    return re.findall(r"\w+", s.lower())

def main(src: str, langs: list[str]) -> None:
    src = Path(src)
    ffmpeg("-i", str(src), "-vn", "-ac", "1", "-ar", "16000", str(OUT / "source.wav"))
    t0 = time.time()
    with open(OUT / "source.wav", "rb") as f:
        raw = client.audio.transcriptions.with_raw_response.create(model=ASR, file=f, response_format="verbose_json")
    segs = [{"start": s.start, "end": s.end, "text": s.text.strip()} for s in raw.parse().segments]
    log("transcribe", ASR, t0, float(raw.headers.get("x-llm-cost", 0)))
    report = {}

    for lang in langs:
        name = LANGS[lang]
        t0 = time.time()
        raw = client.chat.completions.with_raw_response.create(
            model=TRANSLATOR, temperature=0,
            messages=[{"role": "user", "content": f"Translate each line into natural spoken {name} for a voice-over. Each line must be speakable "
                       f"in the seconds given, so shorten rather than overrun. Keep product names.\n\n"
                       + "\n".join(f"{i}. ({s['end'] - s['start']:.1f}s) {s['text']}" for i, s in enumerate(segs))}],
            response_format={"type": "json_schema", "json_schema": {"name": "lines", "strict": True, "schema": {
                "type": "object", "additionalProperties": False, "required": ["lines"],
                "properties": {"lines": {"type": "array", "items": {"type": "string"}}}}}},
        )
        lines = json.loads(raw.parse().choices[0].message.content)["lines"][:len(segs)]
        log(f"translate {lang}", TRANSLATOR, t0, float(raw.headers.get("x-llm-cost", 0)))

        parts, check = [], []
        for i, (seg, line) in enumerate(zip(segs, lines)):
            t0 = time.time()
            raw = client.audio.speech.with_raw_response.create(model=TTS, voice=VOICE, input=line, response_format="wav")
            clip = OUT / f"{lang}-{i:03d}.wav"; clip.write_bytes(raw.content)
            cost = float(raw.headers.get("x-llm-cost", 0))
            # Fit the slot: speed up to 1.35x if the line runs long (atempo keeps the pitch).
            slot = max(0.5, (segs[i + 1]["start"] if i + 1 < len(segs) else seg["end"] + 1.0) - seg["start"])
            speed = min(1.35, max(1.0, duration(clip) / slot))
            fitted = OUT / f"{lang}-{i:03d}-fit.wav"
            ffmpeg("-i", str(clip), "-filter:a", f"atempo={speed:.3f}", str(fitted))
            parts.append((seg["start"], fitted))
            # Check by ear: transcribe the dub and compare it with the line it should say.
            with open(fitted, "rb") as f:
                raw2 = client.audio.transcriptions.with_raw_response.create(model=ASR, file=f, language=lang)
            cost += float(raw2.headers.get("x-llm-cost", 0))
            heard = raw2.parse().text
            match = difflib.SequenceMatcher(None, words(line), words(heard)).ratio()
            check.append({"line": line, "heard": heard.strip(), "match": round(match, 2), "speed": round(speed, 2)})
            log(f"voice {lang} {i}", TTS, t0, cost)

        t0 = time.time()
        inputs, mix = [], []
        for k, (start, path) in enumerate(parts):
            inputs += ["-i", str(path)]
            mix.append(f"[{k + 1}:a]adelay={int(start * 1000)}:all=1[d{k}]")
        track = OUT / f"dub-{lang}.wav"
        ffmpeg("-f", "lavfi", "-t", f"{duration(src)}", "-i", "anullsrc=r=24000:cl=mono", *inputs, "-filter_complex",
               ";".join(mix) + ";[0:a]" + "".join(f"[d{k}]" for k in range(len(parts))) + f"amix=inputs={len(parts) + 1}:normalize=0[a]",
               "-map", "[a]", str(track))
        srt = "".join(f"{i + 1}\n{srt_time(s['start'])} --> {srt_time(s['end'])}\n{l}\n\n" for i, (s, l) in enumerate(zip(segs, lines)))
        (OUT / f"{src.stem}.{lang}.srt").write_text(srt)
        ffmpeg("-i", str(src), "-i", str(track), "-map", "0:v", "-map", "1:a", "-c:v", "copy", "-c:a", "aac", "-shortest", str(OUT / f"{src.stem}.{lang}.mp4"))
        log(f"mux {lang}", "ffmpeg", t0, 0.0)

        report[lang] = {"lines": check, "min_match": min(c["match"] for c in check)}
        flag = [c for c in check if c["match"] < 0.85]
        print(f"  {name}: {len(check)} lines, lowest match {report[lang]['min_match']:.2f}" + (f", {len(flag)} to review" if flag else ""))

    total = sum(s["cost_usd"] for s in ledger)
    Path("run.json").write_text(json.dumps({"segments": segs, "check": report, "steps": ledger, "total_usd": total}, indent=2, ensure_ascii=False))
    print(f"\ntotal ${total:.4f}")

if __name__ == "__main__":
    main(sys.argv[1], sys.argv[2].split(","))

// Dub a video into other languages: transcribe, translate to fit the timing, voice each line, check the dub by ear, mux.
//
// npm install openai      (and ffmpeg on PATH)
// RUNCRATE_API_KEY=... npx tsx video-dubbing.ts video.mp4 es,de
import { execFileSync } from "node:child_process";
import { createReadStream, mkdirSync, writeFileSync } from "node:fs";
import { parse } from "node:path";
import OpenAI from "openai";

const client = new OpenAI({ baseURL: "https://api.runcrate.ai/v1", apiKey: process.env.RUNCRATE_API_KEY, maxRetries: 4, timeout: 180_000 });
const ASR = "openai/whisper-large-v3-turbo";
const TRANSLATOR = "openai/gpt-oss-120b"; // on the free tier
const TTS = "Qwen/Qwen3-TTS"; // speaks the language of the text it is given
const VOICE = "Serena";
const LANGS: Record<string, string> = { es: "Spanish", de: "German", fr: "French", it: "Italian", pt: "Portuguese", ja: "Japanese", ko: "Korean", zh: "Chinese" };
const ledger: { step: string; model: string; seconds: number; cost_usd: number }[] = [];
const ffmpeg = (...args: string[]) => execFileSync("ffmpeg", ["-y", "-loglevel", "error", ...args]);
const duration = (path: string) => Number(execFileSync("ffprobe", ["-v", "error", "-show_entries", "format=duration", "-of", "csv=p=0", path]).toString());
const words = (s: string) => s.toLowerCase().match(/[\p{L}\p{N}]+/gu) ?? [];
mkdirSync("out", { recursive: true });

function log(step: string, model: string, t0: number, cost: number) {
  const seconds = (Date.now() - t0) / 1000;
  ledger.push({ step, model, seconds, cost_usd: cost });
  console.log(`${step.padEnd(14)} ${model.padEnd(34)} ${seconds.toFixed(2).padStart(7)}s  $${cost.toFixed(6)}`);
}

function srtTime(t: number): string {
  const ms = Math.round(t * 1000);
  const p = (n: number, w = 2) => String(n).padStart(w, "0");
  return `${p(Math.floor(ms / 3_600_000))}:${p(Math.floor((ms % 3_600_000) / 60_000))}:${p(Math.floor((ms % 60_000) / 1000))},${p(ms % 1000, 3)}`;
}

/** How alike two word lists are, 0 to 1 (the share of words they have in common, in order). */
function similarity(a: string[], b: string[]): number {
  const dp = Array.from({ length: a.length + 1 }, () => new Array<number>(b.length + 1).fill(0));
  for (let i = 1; i <= a.length; i++) for (let j = 1; j <= b.length; j++) dp[i][j] = a[i - 1] === b[j - 1] ? dp[i - 1][j - 1] + 1 : Math.max(dp[i - 1][j], dp[i][j - 1]);
  return a.length + b.length ? (2 * dp[a.length][b.length]) / (a.length + b.length) : 1;
}

const [src, langArg] = process.argv.slice(2);
const stem = parse(src).name;
ffmpeg("-i", src, "-vn", "-ac", "1", "-ar", "16000", "out/source.wav");
let t0 = Date.now();
const tr = await client.audio.transcriptions.create({ model: ASR, file: createReadStream("out/source.wav"), response_format: "verbose_json" }).withResponse();
const segs = (tr.data.segments ?? []).map((s) => ({ start: s.start, end: s.end, text: s.text.trim() }));
log("transcribe", ASR, t0, Number(tr.response.headers.get("x-llm-cost") ?? 0));
const report: Record<string, { lines: { line: string; heard: string; match: number; speed: number }[]; min_match: number }> = {};

for (const lang of langArg.split(",")) {
  const name = LANGS[lang];
  t0 = Date.now();
  const res = await client.chat.completions
    .create({
      model: TRANSLATOR,
      temperature: 0,
      messages: [
        {
          role: "user",
          content:
            `Translate each line into natural spoken ${name} for a voice-over. Each line must be speakable ` +
            `in the seconds given, so shorten rather than overrun. Keep product names.\n\n` +
            segs.map((s, i) => `${i}. (${(s.end - s.start).toFixed(1)}s) ${s.text}`).join("\n"),
        },
      ],
      response_format: {
        type: "json_schema",
        json_schema: {
          name: "lines",
          strict: true,
          schema: { type: "object", additionalProperties: false, required: ["lines"], properties: { lines: { type: "array", items: { type: "string" } } } },
        },
      },
    })
    .withResponse();
  const lines: string[] = JSON.parse(res.data.choices[0].message.content ?? "{}").lines.slice(0, segs.length);
  log(`translate ${lang}`, TRANSLATOR, t0, Number(res.response.headers.get("x-llm-cost") ?? 0));

  const parts: [number, string][] = [];
  const check: { line: string; heard: string; match: number; speed: number }[] = [];
  for (const [i, line] of lines.entries()) {
    t0 = Date.now();
    const seg = segs[i];
    const speech = await client.audio.speech.create({ model: TTS, voice: VOICE, input: line, response_format: "wav" }).withResponse();
    const clip = `out/${lang}-${String(i).padStart(3, "0")}.wav`;
    writeFileSync(clip, Buffer.from(await speech.data.arrayBuffer()));
    let cost = Number(speech.response.headers.get("x-llm-cost") ?? 0);
    // Fit the slot: speed up to 1.35x if the line runs long (atempo keeps the pitch).
    const slot = Math.max(0.5, (i + 1 < segs.length ? segs[i + 1].start : seg.end + 1) - seg.start);
    const speed = Math.min(1.35, Math.max(1, duration(clip) / slot));
    const fitted = clip.replace(".wav", "-fit.wav");
    ffmpeg("-i", clip, "-filter:a", `atempo=${speed.toFixed(3)}`, fitted);
    parts.push([seg.start, fitted]);
    // Check by ear: transcribe the dub and compare it with the line it should say.
    const back = await client.audio.transcriptions.create({ model: ASR, file: createReadStream(fitted), language: lang }).withResponse();
    cost += Number(back.response.headers.get("x-llm-cost") ?? 0);
    check.push({ line, heard: back.data.text.trim(), match: Math.round(similarity(words(line), words(back.data.text)) * 100) / 100, speed: Math.round(speed * 100) / 100 });
    log(`voice ${lang} ${i}`, TTS, t0, cost);
  }

  t0 = Date.now();
  const track = `out/dub-${lang}.wav`;
  ffmpeg(
    "-f", "lavfi", "-t", `${duration(src)}`, "-i", "anullsrc=r=24000:cl=mono",
    ...parts.flatMap(([, p]) => ["-i", p]),
    "-filter_complex",
    parts.map(([start], k) => `[${k + 1}:a]adelay=${Math.trunc(start * 1000)}:all=1[d${k}]`).join(";") +
      `;[0:a]${parts.map((_, k) => `[d${k}]`).join("")}amix=inputs=${parts.length + 1}:normalize=0[a]`,
    "-map", "[a]", track,
  );
  writeFileSync(`out/${stem}.${lang}.srt`, segs.map((s, i) => `${i + 1}\n${srtTime(s.start)} --> ${srtTime(s.end)}\n${lines[i]}\n\n`).join(""));
  ffmpeg("-i", src, "-i", track, "-map", "0:v", "-map", "1:a", "-c:v", "copy", "-c:a", "aac", "-shortest", `out/${stem}.${lang}.mp4`);
  log(`mux ${lang}`, "ffmpeg", t0, 0);

  const minMatch = Math.min(...check.map((c) => c.match));
  report[lang] = { lines: check, min_match: minMatch };
  const flagged = check.filter((c) => c.match < 0.85).length;
  console.log(`  ${name}: ${check.length} lines, lowest match ${minMatch.toFixed(2)}${flagged ? `, ${flagged} to review` : ""}`);
}

const total = ledger.reduce((s, r) => s + r.cost_usd, 0);
writeFileSync("run.json", JSON.stringify({ segments: segs, check: report, steps: ledger, total_usd: total }, null, 2));
console.log(`\ntotal $${total.toFixed(4)}`);

What the proof run cost

The Python file on the 7-second ad from the video ad recipe, into Spanish and German. Measured on 2026-10-23 against api.runcrate.ai; costs are what the account was billed, read from each response’s X-LLM-Cost header or usage.cost.

Step	Model or tool	Time	Cost
Transcribe	openai/whisper-large-v3-turbo	1.2 s	$0.0025
Translate, Spanish	openai/gpt-oss-120b (free tier)	1.3 s	$0.0000
Voice and check two Spanish lines	Qwen/Qwen3-TTS + whisper	6.5 s	$0.0102
Translate, German	openai/gpt-oss-120b (free tier)	1.2 s	$0.0000
Voice and check two German lines	Qwen/Qwen3-TTS + whisper	7.1 s	$0.0079
Two muxes	ffmpeg	0.2 s	$0
Total		18 s	$0.0205

The TypeScript file: $0.0200 in 18 s. On that run the check flagged one line in each language: both were lines sped up to the 1.35x cap, heard back as “Montente” and “bleibe”.

Scale it

Every language is independent: run them in parallel, or one job per language.
Long videos: segments are short, so a 10-minute video is about 150 lines; the cost is the TTS characters plus two transcriptions of the audio.
Review only what the check flags: the word-match score is in run.json for every line.

Gotchas found in the proof run

ResembleAI/chatterbox-multilingual mangled Spanish and German when given only the text (“sostenable Canoistra”); it takes no language parameter. Qwen3-TTS read the same lines cleanly, so the recipe uses it.
A lip-synced talking head was tried and left out: kuaishou/kling-avatar-v2 answered 502 three times, and Seedance 2.0 Fast refused a photographic portrait with reference audio as sensitive (E005). It goes in when it runs. When you add one: Kling Lip Sync and Kling Avatar v2 make a clip as long as the audio or video you send, so duration is ignored. Runcrate measures your input when you submit and needs that clip’s cost free in your balance, with a second or so of margin; you’re charged only for the clip’s real length, and a balance too low is refused with 402 and "code": "video_hold_unaffordable". See video generation.
Music under the voice is lost: this replaces the audio track. Separate the stems first (an open source-separation model on a GPU job) if the original has a score.
The 1.35x cap is where speech starts to slur; the back-transcription check is how you find those lines.

Start here

Recipes: Media

Recipes: Voice

Recipes: Documents and RAG

Recipes: Agents

Recipes: Training and serving

Text and agents

Vision and documents

Images

Video

Speech

GPUs

Storage

Multilingual video dubbing

Build it in your project

The code

What the proof run cost

Scale it

Gotchas found in the proof run

​Build it in your project

​The code

​What the proof run cost

​Scale it

​Gotchas found in the proof run

Build it in your project

The code

What the proof run cost

Scale it

Gotchas found in the proof run