Stream speech

Streaming sends the audio of a speech request in pieces while the rest is still being generated, so the listener hears the first words sooner. This page shows how to stream a request, play the audio as it arrives, and measure how long the listener waits.

Information

To get one complete file instead, see Speech generation. For what the listener waits for and where the time goes, see Latency.

How streaming works

A streamed request is a speech generation request with stream set to true. You still send the complete text in one request: only the audio comes back in pieces, as server-sent events.

Event	Fields	What your application does
`speech.audio.delta`	`audio_data`: a base64 chunk of audio	Decode the chunk and pass it to your player, or append it to the file you write.
`speech.audio.done`	`usage`	Stop reading: the audio is complete.

The chunks arrive in order. Put together, they make one audio file in the response_format you asked for, the same as the file of a request without streaming.

Note

Streaming the response doesn't make the request a realtime session: the text must be complete when you send it. To speak text while it's still being written, see Stream text as it's written.

Stream a request

The following script streams a request, records when the first chunk arrives, and saves all the chunks to one MP3 file:

import base64
import os
import time
from pathlib import Path

from mistralai.client import Mistral

client = Mistral(api_key=os.environ["MISTRAL_API_KEY"])

chunks = []
first_audio = None
start = time.perf_counter()

with client.audio.speech.complete(
    model="voxtral-mini-tts-3-0",
    input=(
        "Thanks for calling. Your order shipped this morning and should arrive on Thursday. "
        "You'll get a tracking link by email within the hour."
    ),
    voice_id="gb_jane_neutral",
    response_format="mp3",
    stream=True,
) as stream:
    for event in stream:
        if event.event == "speech.audio.delta":
            if first_audio is None:
                first_audio = time.perf_counter() - start
            chunks.append(base64.b64decode(event.data.audio_data))
        elif event.event == "speech.audio.done":
            print(f"Usage: {event.data.usage}")

output = Path("streamed.mp3")
output.write_bytes(b"".join(chunks))
print(f"First audio after {first_audio:.2f} s, {len(chunks)} chunks, total {time.perf_counter() - start:.2f} s")
print(f"Saved {output.resolve()}")

import base64
import os
import time
from pathlib import Path

from mistralai.client import Mistral

client = Mistral(api_key=os.environ["MISTRAL_API_KEY"])

chunks = []
first_audio = None
start = time.perf_counter()

with client.audio.speech.complete(
    model="voxtral-mini-tts-3-0",
    input=(
        "Thanks for calling. Your order shipped this morning and should arrive on Thursday. "
        "You'll get a tracking link by email within the hour."
    ),
    voice_id="gb_jane_neutral",
    response_format="mp3",
    stream=True,
) as stream:
    for event in stream:
        if event.event == "speech.audio.delta":
            if first_audio is None:
                first_audio = time.perf_counter() - start
            chunks.append(base64.b64decode(event.data.audio_data))
        elif event.event == "speech.audio.done":
            print(f"Usage: {event.data.usage}")

output = Path("streamed.mp3")
output.write_bytes(b"".join(chunks))
print(f"First audio after {first_audio:.2f} s, {len(chunks)} chunks, total {time.perf_counter() - start:.2f} s")
print(f"Saved {output.resolve()}")

Play audio as it arrives

To start playback before the whole file is ready, decode each speech.audio.delta and pass the bytes to a player that accepts a stream, in the order they arrive.

Most players keep a small buffer of audio before they start, and that buffer sets a trade-off:

Player buffer	The listener gets	The risk
Smaller	The first sound sooner	Gaps in playback if the next chunk is late
Larger	Smooth playback	A longer wait before the first sound

Tip

Start with your player's default buffer and reduce it only if the first sound comes too late. Test on the network your users have, not only on a fast connection.

Measure the time to first audio

The time to first audio is what the listener waits for. Measure it in two places:

From sending the request to the first speech.audio.delta, as the example does. This is the wait for the API and the network.
From sending the request to the first sound in your player. This adds the player buffer, and it's the wait the listener notices.

, so measure it over many requests and watch the slowest ones, not only the average. For the full method, see Latency.

Stream text as it's written

Common issues

These issues are specific to streaming. For errors on the request itself, such as an unknown voice or an invalid format, see Troubleshooting.

Symptom	Cause	Fix
Nothing plays until the whole audio is ready	Your client reads the full response before handing it on	Read the events as they arrive, and pass each chunk to the player at once.
Gaps or stutter during playback	The player buffer runs out before the next chunk arrives	Increase the player buffer, or check the network between your application and the API.
The saved file doesn't play	The chunks were saved out of order, or one was dropped	Decode every `speech.audio.delta` and write the chunks in the order they arrived.

Going further