Soul TTS is the voice behind every Soul call. It copies a voice from a recording, speaks in a way that fits the conversation, and responds quickly enough for a natural call.
The Problem
With recent TTS models, cloning a voice with perfect prosody is only part of the problem. The voice must also fit the conversation: excitement at good news, a laugh at a joke, a gentler tone when someone is upset. And it must respond quickly enough for a natural call.
Building for AI friends adds another constraint. Users shouldn’t have to upload hours of studio-quality audio just to clone a voice (which is unfortunately the case for many 'off the shelf' TTS providers).
Soul TTS therefore has three problems to solve. It must copy a voice from a short recording, even a noisy one. It must speak with prosody that naturally fits the conversation. And it must respond quickly enough to sound natural.
Listen
Each model gets the same reference speaker and the same target text. We kept the first result from every model, all at their recommended default settings.
The reference speaker sets the voice. Every model then speaks the target text below in that voice.
“Dobby has to tell you something, sir. Something terrible. You must not trust this place. Dobby was bound once, bound forever to serve. But no longer. Now, Dobby has to use his freedom to keep you safe.”
“Messi picks it up just inside the half, he turns away from one, he glides past another and the whole stadium can feel it coming! He's driving at the heart of the defense now, nobody can get near him. Oh! He's curled it into the top corner and this place has lost its mind! Absolutely magical from the little genius!”
“So I'm standing there trying to act normal, right, and I reach for a drink and somehow knock the entire tray out of the waiter's hands and everyone just turns and stares at me. And I'm thinking, you know, please just let the floor open up and swallow me whole, but no of course not, the night was just getting started.”
Benchmarks
Public English test sets measure word accuracy, reference-speaker similarity, predicted audio quality, and how well speech follows style instructions.
| Model | Word error ↓ | Speaker similaritySimilarity ↑ |
|---|---|---|
| F5-TTS | 2.00% | 67.0 |
| VibeVoice | 3.04% | 68.9 |
| Higgs Audio v2 | 2.44% | 67.7 |
| VoxCPM2 | 1.84% | 75.3 |
| Qwen3-TTS | 1.23% | 71.7 |
| LongCat-Audio-DiT | 1.50% | 78.6 |
| Fish Audio S2 | 0.99% | — |
| Fish Audio Drama 3First take | 2.20% | — |
| Soul TTS | 0.97% | 80.2 |
Similarity is an embedding score shown on a 0–100 scale, not a listener rating. Drama 3 reports 2.20% WER on the first take and 1.54% after regenerating takes that fail signal checks (Table 4); that table does not report similarity. Other rows come from the VoxCPM2 report. These are published results, not a shared rerun.
Published results: VoxCPM2 benchmark tables; Drama 3 report, Table 4.
Word error on standard and challenging English speech.
| Model | English ↓ | Hard English ↓ |
|---|---|---|
| CosyVoice2 | 6.32% | 11.96% |
| CosyVoice3 1.5B | 4.99% | 10.55% |
| VoxCPM2 | 5.00% | 8.48% |
| Fish Audio S2 | 2.43% | 4.40% |
| Soul TTS | 2.38% | 4.31% |
Published results: VoxCPM2 benchmark tables.
VoxCPM2 report
Word accuracy and reference-speaker similarity on the English split of MiniMax’s multilingual test.
| Model | Word error ↓ | Speaker similaritySimilarity ↑ |
|---|---|---|
| MiniMax | 2.164% | 75.6 |
| ElevenLabs | 2.339% | 61.3 |
| Qwen3-TTS | 0.934% | 77.5 |
| Fish Audio S2 | 1.620% | 79.7 |
| VoxCPM2 | 2.289% | 85.4 |
| Soul TTS | 0.915% | 87.1 |
Similarity is shown on a 0–100 scale. This table retains the VoxCPM2 report’s scores and scoring setup; the newer Drama 3 comparison below uses a different evaluation setup.
Published results: VoxCPM2 benchmark tables.
Drama 3 report
The latest English comparison from Fish Audio.
| Model | Word error ↓ |
|---|---|
| OmniVoice | 1.56% |
| FireRedTTS3-Base | 2.12% |
| Fish Audio S2 | 1.62% |
| ElevenLabs v4 | 1.56% |
| Inworld TTS-2 | 1.56% |
| Soniox TTS RT v2 | 1.94% |
| Fish Audio Drama 3 | 1.13% |
| Soul TTS | 1.11% |
Drama 3 uses Whisper-large-v3 and OmniVoice’s evaluation script. ElevenLabs v4, Inworld TTS-2, and Soniox TTS RT v2 were evaluated by Fish Audio through their APIs; OmniVoice, FireRedTTS3-Base, and Fish Audio S2 are quoted from their papers with their own scoring. Similarity is not reported. Compare these scores within this table, rather than pooling them with the VoxCPM2 report above.
Published results: Drama 3 report, Table 6.
Voice cloning on 1,127 test-clean sentences from 39 speakers.
| Model | Word errorWER ↓ | Speaker similaritySIM ↑ | Predicted qualityUTMOS ↑ |
|---|---|---|---|
| CosyVoice 3 | 1.59% | 0.694 | 4.28 |
| Qwen3-TTS | 1.60% | 0.704 | 4.41 |
| OmniVoice | 1.30% | 0.729 | 4.28 |
| Fish Audio Drama 3One reference | 1.85% | 0.605 | 4.31 |
| Fish Audio Drama 3Three references | 1.85% | 0.633 | 4.31 |
| Soul TTSOne reference | 1.27% | 0.744 | 4.50 |
WER uses HuBERT-large. SIM is shown on a 0–1 scale against the first reference; UTMOS predicts perceived quality and is not a listener score. All rows use one reference except the explicitly marked three-reference Drama 3 row. Baseline results are quoted from OmniVoice in the Drama 3 report.
Published results: Drama 3 report, Table 5.
Following instructions for acoustic controls, descriptive styles, and role-play.
| Model | Acoustic controlAPS ↑ | Style descriptionDSD ↑ | Role-playRP ↑ |
|---|---|---|---|
| Hume | 83.0 | 75.3 | 54.3 |
| VoxInstruct | 54.9 | 57.0 | 39.3 |
| Parler-TTS mini | 63.4 | 48.7 | 28.6 |
| Parler-TTS large | 60.0 | 45.9 | 31.2 |
| PromptTTS | 64.3 | 47.2 | 31.4 |
| PromptStyle | 57.4 | 46.4 | 30.9 |
| Mimo-Audio 7B Instruct | 80.6 | 77.6 | 59.5 |
| Qwen3-TTS 1.7B VD | 82.9 | 82.4 | 68.4 |
| VoxCPM2 | 84.2 | 83.2 | 71.4 |
| Soul TTS | 85.9 | 84.9 | 72.8 |
APS measures acoustic control, DSD measures style descriptions, and RP measures role-play.
Published results: VoxCPM2 benchmark tables.
Bold values lead the published rows shown in each table.
Our Approach
Soul TTS is a rectified-flow diffusion transformer over continuous audio latents.
It starts from noise and refines every frame of a line at once, conditioned on three things: whose voice it is, what to say, and the conversation so far.
Generating a block of speech jointly lets the model coordinate prosody across the sentence, including stress, rhythm, and pause placement, rather than committing to each token before the rest of the speech is generated.
A causal speaker encoder processes the reference recording in patches, producing speaker tokens for attention and a pooled vector for adaptive layer normalization (AdaLN). A bidirectional text encoder processes the style instruction, recent conversation, and target text. Diffusion transformer blocks predict how the audio latents should change at each step, and a causal audio VAE decodes the result into speech.
From noise to speech
Soul generates speech in the continuous latent space of an audio VAE, which represents audio in a compressed form. During training, we interpolate between noise and clean speech latents and train the model to predict the direction and rate of change between them. During generation, the model follows these predictions from noise toward speech. A straighter path allows fewer generation steps, reducing latency.
We guide generation separately toward the target text and the reference voice. Text guidance improves word accuracy; speaker guidance preserves voice identity. We drop each condition independently during training so these signals can be computed separately during generation. Projected guidance limits distortion from strong guidance, while CFG-Zero* reduces guidance in the initial steps. Sway sampling allocates more steps early in generation, when voice and pronunciation are established. We then distill the guided model into a student that generates speech in fewer steps, making streaming practical.
Rectified flow jointly refines the speech frames along a straight path from noise toward clean audio latents.
One attention over everything
Within each diffusion transformer block, speech frames attend jointly to other speech frames, speaker tokens, text tokens, and conversation context. The model learns how text aligns with speech without a separate alignment or phoneme-duration model.
A block causal attention mask enables streaming. Each speech frame can attend to all frames within its current block and to earlier blocks, but not to future blocks.
Each row represents a speech frame making an attention query. Filled cells show the inputs it can attend to. The highlighted row moves through the sequence.
Streaming on a call
Soul begins generating speech as soon as the first clause of a reply is available. It generates successive blocks using the block causal attention mask. Each block reuses cached attention keys and values from earlier blocks, reducing computation and preserving continuity between them. Playback begins with the first block while the model generates the next.
A short first block reduces the delay before speech begins. Subsequent blocks are longer, allowing more efficient generation while the earlier audio plays.
The language model produces text faster than it can be spoken. Speech generation begins after the first clause and continues ahead of playback.
Cloning from the recording you have
User recordings rarely have studio conditions. A reference may be a voice note recorded in a kitchen, a video with background music, or audio from a Bluetooth call. The model must recover the voice despite these differences in recording quality.
During training, we deliberately degrade reference recordings with random combinations of noise, room acoustics, telephone bandwidth limits, lossy compression, and packet loss. The target audio remains clean. We select the reference from a different segment of the same recording session and retain it only after verifying that the speaker matches. This trains the model to preserve speaker identity while ignoring recording artifacts.
The model receives a degraded reference and learns to produce clean speech from a different segment. The training objective encourages the encoder to retain speaker identity rather than noise or recording conditions.
Hearing the conversation
The appropriate delivery of a sentence depends on the conversation before it. The same words may express a question, a joke, or reassurance. Soul therefore conditions speech generation on the conversation, rather than on the target sentence alone.
Soul's previous turn is supplied as an audio latent prefix, preserving continuity in tone and energy. Your previous turn contributes pitch contour, energy, and speaking rate, normalized to reduce information about your voice identity. This lets Soul respond to an excited “I got the job!” with appropriate excitement while retaining its own voice. We check that conditioning on your speech does not increase the generated voice's similarity to yours.
The next line is conditioned on three inputs: conversation text with speaker labels, Soul's previous turn as an audio latent prefix, and normalized prosodic features from your previous turn.
Laughs, sighs and "mm-hm"
Natural conversation includes laughter, sighs, and brief acknowledgments such as “mm-hm.” We use event-aware automatic speech recognition to identify and label these sounds in recorded conversations. The labels become tokens in the text sequence, allowing the language model to specify them alongside spoken words.
Inline event tags specify where laughter and sighs occur. The speech model generates these sounds in the same voice, with timing and intensity suited to the surrounding speech.
Learning what sounds right
Pretraining teaches the model to generate speech, but does not directly optimize for listener preference. We address this with Flow-GRPO. For each prompt, the model generates several speech samples using a stochastic sampler. We score them for word accuracy, speaker similarity, audio quality, and listener preference, using a preference model trained on blind A/B comparisons. Samples are evaluated relative to others from the same prompt, and higher-scoring samples guide updates to a small set of LoRA parameters. A KL penalty limits how far the updated model departs from the base model.
Flow-GRPO compares samples generated from the same prompt and uses their relative scores to update the model, without a separate value network.
Improvements
Very short or noisy references remain difficult for TTS models. A few seconds of speech over background music may provide too little information to recover the speaker's identity reliably. Soul can then produce a less distinctive voice or reproduce some of the background noise.
Expressive range also remains limited. Emotions that are uncommon in training recordings, such as crying or shouting, often sound slightly flatter than everyday speech.