by default, timestamps are enabled for word and segment level

word_timestamps = output[0].timestamp['word'] # word level timestamps for first sample segment_timestamps = output[0].timestamp['segment'] # segment level timestamps

for stamp in segment_timestamps: print(f"{stamp['start']}s - {stamp['end']}s : {stamp['segment']}")

#### Translating with timestamps

To translate with timestamps:
```python
output = asr_model.transcribe(['2086-149220-0033.wav'], source_lang='en', target_lang='fr', timestamps=True)

segment_timestamps = output[0].timestamp['segment'] # only supports segment level timestamps for translation

for stamp in segment_timestamps:
    print(f"{stamp['start']}s - {stamp['end']}s : {stamp['segment']}")

For translation task, please, refer to segment-level timestamps for getting intuitive and accurate alignment.

Note: If timestamps are not required for your work, you can reduce memory usage by restoring only the .nemo file without the auxiliary CTC model. To do this, extract the .nemo file, remove any timestamps_asr_model files, then repackage it into a new .nemo file.

Software Integration

Runtime Engine(s):

• NeMo main branch (until it is released in NeMo 2.5)

Supported Hardware Microarchitecture Compatibility:

• NVIDIA Ampere
• NVIDIA Blackwell
• NVIDIA Hopper

[Preferred/Supported] Operating System(s):

• Linux

Hardware Specific Requirements: At least 6GB RAM for model to load.

Model Version

Current version: Canary-1b-v2. Previous versions can be accessed here.

Training and Evaluation Datasets

Training

The model was trained using the NeMo toolkit [4], following a 3-stage training procedure:

• Initialized from a 4-language ASR model
• Stage 1: Trained for 150,000 steps on X→En and English ASR tasks using 64 A100 GPUs
• Stage 2: Trained for 115,000 additional steps on the full dataset (ASR, X→En, En→X)
• Stage 3: Fine-tuned for 10,000 steps on a language-balanced high-quality subset of Granary and NeMo ASR Set 3.0

For all the stages of training, both languages and corpora are weighted using temperature sampling (τ = 0.5).

Training script: speech_to_text_aed.py

Tokenizer script: process_asr_text_tokenizer.py

---

Training Dataset

Canary-1b-v2 was trained on a massive multilingual speech recognition and translation dataset combining Nvidia's newly published Granary and in-house dataset NeMo ASR Set 3.0.

Granary Dataset [5] [6] with improved pseudo-labels and efficiently filtered versions of the following corpora:

• YTC [7]
• MOSEL [8]
• YODAS [9]

Granary is now available on Hugging Face.

To read more about the pseudo-labeling technique and pipeline, please refer to the Granary Paper.

NeMo ASR Set 3.0 including human-labeled transcriptions from the following corpora:

• Multilingual LibriSpeech (MLS)
• Mozilla Common Voice (v7.0)
• AMI (70 hrs)
• Fleurs
• LibriSpeech (960 hours)
• Fisher Corpus
• National Speech Corpus Part 1
• VCTK
• Europarl-ASR

Total training hours: 1.7M

• ASR: 660,000 hrs
• X→En: 360,000 hrs
• En→X: 690,000 hrs
• Non-speech: 36,000 hrs

All transcripts include punctuation and capitalization.

Data Collection Method by dataset

• Hybrid: Automated, Human

Labeling Method by dataset

• Hybrid: Synthetic, Human

---

Evaluation Dataset

• Fleurs [10], MLS [11], CoVoST [12]
• Hugging Face Open ASR Leaderboard [13]
• Earnings-22 [14], This American Life [15] (long-form)
• MUSAN [16]

Data Collection Method by dataset

• Human

Labeling Method by dataset

• Human

Benchmark Results

This section reports the evaluation results of the Canary-1b-v2 model across multiple tasks, including Automatic Speech Recognition (ASR), Speech Translation (AST), robustness to noise, and long-form transcription.

---

Automatic Speech Recognition (ASR)

Note: Presented WERs do not include Punctuation and Capitalization errors.

---

Hugging Face Open ASR Leaderboard

More details on evaluation can be found at HuggingFace ASR Leaderboard

---

Speech Translation (AST)

X → English

English → X

---

Noise Robustness

Performance across different Signal-to-Noise Ratios (SNR) using MUSAN music and noise samples [16] on the LibriSpeech Clean test set. Metric: Word Error Rate (WER)

Hallucination Robustness

Number of characters per minute on MUSAN [16] 48 hrs eval set:

---

Long-form Inference

Canary-1b-v2 achieves strong performance on long-form transcription by using dynamic chunking with 1-second overlap between chunks, allowing for efficient parallel processing. This dynamic chunking feature is automatically enabled when calling .transcribe() on a single audio file, or when using batch_size=1 with multiple audio files that are longer than 40 seconds.

Note: Presented WERs do not include Punctuation and Capitalization errors.

---

Inference

Engine:

• NVIDIA NeMo

Test Hardware:

• NVIDIA A10
• NVIDIA A100
• NVIDIA A30
• NVIDIA A5000
• NVIDIA H100
• NVIDIA L4
• NVIDIA L40

---

Ethical Considerations

NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their supporting model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.

For more detailed information on ethical considerations for this model, please see the Model Card++ Explainability, Bias, Safety & Security, and Privacy Subcards here.

Please report security vulnerabilities or NVIDIA AI Concerns here.

Bias:

Explainability:

Privacy:

Safety:

References

[1] Granary: Speech Recognition and Translation Dataset in 25 European Languages

[2] NVIDIA Granary Dataset Card

[3] Fast Conformer with Linearly Scalable Attention for Efficient Speech Recognition

[4] Attention is All You Need

[5] Google Sentencepiece Tokenizer

[6] NVIDIA NeMo Toolkit

[7] Youtube-Commons

[8] MOSEL: 950,000 Hours of Speech Data for Open-Source Speech Foundation Model Training on EU Languages

[9] YODAS: Youtube-Oriented Dataset for Audio and Speech

[10] FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech

[11] MLS: A Large-Scale Multilingual Dataset for Speech Research

[12] CoVoST 2 and Massively Multilingual Speech-to-Text Translation

[13] HuggingFace Open ASR Leaderboard

[14] Earnings-22 Benchmark

[15] Speech Recognition and Multi-Speaker Diarization of Long Conversations

[16] MUSAN: A Music, Speech, and Noise Corpus