Build Low-Latency Multilingual Voice Agents: Open Weights & Full Deployment Control with NVIDIA Magpie TTS

Back to Articles

Every voice interaction has a latency budget.

By the time a user hears your application respond, you've already spent precious milliseconds capturing audio, transcribing speech, running an LLM, retrieving context, and generating a response. Text-to-speech (TTS) is the final step — and the one users notice most. If speech generation is slow, the whole experience feels slow.

The more of that pipeline you can run and tune yourself, the more of the latency budget you get back.

Voice AI is moving fast. Integrated speech models offer simplicity — one API call, audio in, audio out — but they trade the ability to fine-tune each component for your domain, swap in better models as they ship, enforce data residency, and understand exactly where latency is coming from. For more control, a cascaded architecture — purpose-built ASR, TTS, and LLM components running together — keeps each layer independently tunable and deployable on infrastructure you own.

NVIDIA Magpie Multilingual TTS is built for that. With open weights, production-ready NVIDIA NIM, and support for 12 languages, you can deploy multilingual speech inside your own infrastructure, optimize latency for your workload, and customize the model for your domain — end to end, in your own environment.

The latest release expands multilingual coverage with Modern Standard Arabic, Korean, and Brazilian Portuguese, while improving quality across many existing languages through updated training data and model improvements.

Whether you're building customer support agents, healthcare assistants, enterprise copilots, translation systems, or conversational AI applications, Magpie provides an open foundation for production voice AI.

Voice AI Is Becoming Multilingual by Default

Today's voice applications don't serve a single language.

Global customer support, enterprise assistants, healthcare documentation, retail automation, and translation workflows increasingly require natural conversations across multiple languages — all while maintaining low latency.

Supporting more languages is only part of the challenge. Developers also need the ability to:

• Deploy where their data lives
• Meet enterprise privacy requirements
• Customize pronunciation and voices
• Predict latency under production workloads
• Scale on their own infrastructure

Open models change what's possible on every one of these.

One Open Model, Twelve Languages

Magpie TTS Multilingual is a 364M-parameter open-weights model supporting:

English · Spanish · French · German · Italian · Vietnamese · Mandarin · Hindi · Japanese · Modern Standard Arabic (new) · Korean (new) · Brazilian Portuguese (new)

Each language includes male and female speaker voices through a shared multilingual speaker representation.

This release also improves multilingual flexibility with expanded code-switching support for Hindi and Japanese, enabled through IPA grapheme-to-phoneme processing and custom pronunciation dictionaries — making it easier to accurately pronounce names, technical terminology, and mixed-language content.

Instead of maintaining separate TTS models for different regions, developers can build multilingual applications on a single open foundation.

The Latency Your Users Actually Notice

In conversational AI, text-to-speech is the final stage before users hear a response. That makes Time to First Audio (TTFA) — the delay between speech generation beginning and the first audio reaching the user — one of the most important latency metrics in a voice pipeline.

Because Magpie TTS can be deployed inside your own environment, the latency you measure is the server-side latency you actually control, with no managed-service round-trip in the number.

Source: NVIDIA TTS NIM Performance documentation (v26.07), average of three trials, on-prem.
TTFA = latency to first audio; RTFX = throughput as a multiple of real time.

At 32ms on B200, Magpie's TTFA leaves the rest of the latency budget for ASR and LLM processing — keeping total end-to-end latency within the sub-200ms window natural conversation requires. Across NVIDIA GPUs, Magpie delivers first audio in 32–79ms on a single stream. At 64 concurrent streams, B200 reaches 239ms TTFA while delivering throughput at 320× real time — generating audio more than 300 times faster than it plays back, even under concurrent load.

The table above shows Magpie served as the NVIDIA NIM, measured on-prem — the optimized container running on your own GPU. The open Hugging Face checkpoint is the same model and your path for research and fine-tuning; the NIM is the tuned serving stack that produces these production latencies. Both run on hardware you control.

Because the model runs on your own infrastructure, you can benchmark performance directly, tune it for your deployment, and scale according to your workload. For real-time voice agents, that's the difference between conversations that feel responsive and conversations that feel delayed.

Optimized for Real-Time Speech Generation

Low latency isn't accidental. Magpie introduces two complementary architectural improvements that reduce inference time while maintaining speech quality.

Frame stacking. The decoder predicts two audio frames during each decoding step rather than one. This cuts the number of decoder iterations in half, shortening generation time and improving throughput.

Local transformer. Frame stacking alone would reduce audio quality by introducing dependencies between simultaneously generated codebook tokens. The local transformer models those dependencies and refines the generated audio, recovering the quality that frame stacking would otherwise sacrifice.

Together, these techniques deliver both faster generation and natural speech synthesis. The architecture is described in Frame-Stacked Local Transformers for Efficient Multi-Codebook Speech Generation (ICASSP 2026).

Faster Doesn't Matter If It Doesn't Sound Natural

This release doesn't only add languages — it also improves synthesis quality across many existing ones. Compared to the previous release, Magpie shows reduced character error rates (CER) and higher speaker similarity (SSIM) on several languages, with the clearest gains on French and Spanish:

Source: Magpie TTS Multilingual model card. CER lower is better; SSIM higher is better.

The newly added Arabic (1.62% CER), Korean (2.69%), and Brazilian Portuguese (2.91%) models establish baseline quality for future improvements.

While objective metrics help measure progress, speech quality is ultimately perceptual. You can hear the difference yourself on NVIDIA Build or the Hugging Face demo.

Why Open Weights Matter

Latency you can measure is useful. Latency you can control is even better.

Open weights give developers capabilities that come from owning the deployment. With Magpie you can:

• Deploy on infrastructure you control — run entirely within your own infrastructure, including private or air-gapped environments.
• Own your latency budget — no managed-service round-trip, and you optimize directly for your hardware and workload.
• Customize pronunciation and voices — fine-tune with NeMo for your own brand, domain vocabulary, or speaker data.
• Scale on your own terms — optimize the serving stack for your infrastructure and workload.
• Maintain enterprise control — keep sensitive conversations and customer data inside your environment.

For enterprises building production voice AI, this control over deployment, performance, and customization is often what matters most.

Build Complete Voice Agents — Not Just Better Speech

Voice AI in production is a system of models, not a single one. Magpie TTS is part of the NVIDIA Nemotron Voice Agent Developer Example, a reference implementation showing how purpose-built speech, language, and reasoning models work together as a coordinated system — so you can build always-on voice agents, not just better-sounding speech.

Developers can combine:

• Nemotron Speech for streaming speech recognition
• Magpie TTS for natural multilingual speech synthesis
• Nemotron language and multimodal models for reasoning, tool calling, and multimodal understanding
• NVIDIA NIM for GPU-optimized, production-ready inference microservices
• NeMo for customization and fine-tuning

The Nemotron Voice Agent developer example provides an end-to-end reference implementation that developers can clone, customize, and deploy in hours. It includes production patterns for:

• Real-time interruptible (barge-in) conversations
• Multimodal voice agents with vision understanding
• Multi-agent orchestration and tool calling
• Multilingual voice interactions
• Sub-second end-to-end latency using NVIDIA NIM

Rather than assembling individual components from scratch, developers can start from a complete reference architecture and adapt it to their own applications.

Get Started

Try the model

• NVIDIA Build
• Hugging Face demo

Deploy to production

• NVIDIA Magpie Multilingual TTS NIM — optimized inference containers

Customize for your domain

• NVIDIA NeMo Speech — fine-tuning and training

Build complete voice agents

• NVIDIA voice-agent-examples

Open weights and license

• Model card on Hugging Face — open weights under the NVIDIA Open Model License.

Recommended inference configuration:

cfg_scale = 2.5          # classifier-free guidance — raise for tighter text adherence
temperature = 0.6
top_k = 80
apply_attention_prior = True
prior_epsilon = 0.1