Tractus
Picking a model to transcribe Korean meetings on a Mac: Qwen3-ASR, Whisper and Nemotron
TL;DR: On Korean meetings, Qwen3-ASR-1.7B made 36% fewer errors than Whisper-large-v3-turbo (CER 9.49 vs 14.92) and transcribed an hour of audio in 8.2 minutes. Tractus uses it as the default model.
Tractus turns meeting recordings into text on your Mac, so we needed a model that runs on a Mac, handles Korean meetings well, and keeps the English terms people drop into Korean (p95 latency, connection pool). Most published numbers are for read speech or English, so we measured on data closer to real meetings.
What we measured
| Dataset | What it tests | Metric |
|---|---|---|
| FLEURS-ko | Clean read Korean; checks we reproduce published numbers | CER |
| HiKE | Korean–English code-switching | MER |
| AI Hub meeting speech | Real multi-speaker meetings, 4 sessions, 66–90 min, 5.2 h total | CER |
| AMI | English meetings | WER |
Lower is better for every metric. AI Hub data can't be redistributed, so only metrics are published.
Results
| Model | FLEURS-ko CER | HiKE MER | AI Hub meetings CER | AMI WER | 1 h of audio | Peak memory |
|---|---|---|---|---|---|---|
| Qwen3-ASR-1.7B | 2.62 | 24.1 | 9.49 | 9.50 | 8.2 min | 3.6 GB |
| Qwen3-ASR-0.6B | 3.92 | 26.1 | 11.47 | 10.14 | 3.3 min | 2.4 GB |
| Whisper-large-v3-turbo | 3.08 | 26.6 | 14.92 | 14.28 | 6.0 min | 2.2 GB |
| Nemotron 3.5 ASR 0.6B | 7.02 | 45.1 | 10.93 | 14.13 | 7.6 min | 2.0 GB |
- The gap is biggest on real meetings. The 1.7B model makes 36% fewer character errors than Whisper; even the small 0.6B beats Whisper.
- All four models reproduce their published FLEURS numbers within 0.5 points, so the pipeline is trustworthy.
- Long meetings finish. All four 66–90 minute sessions were transcribed to the end.
What we learned
- Forcing the language to Korean makes things worse. English-heavy speech got translated and the small model fell into repetition loops. Tractus detects the language automatically.
- Custom vocabulary works. Term recall rose from 71.6% to 84.2%. On a real meeting, a few domain terms took one misheard word from 10 mistakes to 0 (2026-10-18), so Tractus also feeds in words from the note's title and memos.
- Chunk length barely matters. 15, 30 and 60 second chunks gave CER 9.69, 9.49 and 9.39; cutting at speaker changes cost 0.1 points.
- Prompting Whisper with English terms backfires. Output drifted into English and CER went from 16.7 to 58.3.
Why not Nemotron
Its meeting CER beats the 0.6B model, but on code-switched speech its MER (45.1) is nearly twice the others, and it spells English acronyms in Hangul — term recall was 1.1%.
Limits
- AI Hub covers one domain and four sessions; rankings are reliable, absolute numbers vary by meeting.
- One device: Apple M3 with 16 GB.
Code and full tables: bench/ · original write-up (Korean)