Audio interface

The audio interface connects a session to the sound on your side. It captures the user's audio and hands it to the SDK, and it plays the audio the agent sends back. The SDK only moves audio between your interface and Mistral: transcription, the model, and speech all run at Mistral.

This page applies to Python and Node.js. In the browser, the SDK captures and plays audio for you, see Browser.

You pass an interface with audio_interface (audioInterface in TypeScript) when you create the session. From then on, the SDK drives it:

1. Before the session is created, the SDK runs the interface's preflight check, so a missing device or audio library fails before Mistral allocates a session.
2. When the session connects, the SDK calls start() to open your devices or streams.
3. Once the agent joins, the SDK reads the user's audio from input_audio() and sends each chunk to Mistral. While the microphone is muted with set_input_enabled(False), the SDK keeps reading but drops the chunks.
4. Each time the agent speaks, the SDK passes its audio to output_audio(), chunk by chunk.
5. When the session closes, the SDK calls stop() so you can release your resources.

Audio uses the same format in both directions:

DefaultAudioInterface records from the system's default microphone and plays to the default speakers. In Python, it uses PyAudio, which needs PortAudio. In Node.js, it uses SoX. See Install the SDK.

Before sending the microphone audio, it cleans it up: it removes the agent's own voice picked up by the microphone (echo cancellation), reduces background noise, and evens out the volume. On the way out, it buffers the agent's audio and plays it at a steady pace. If the agent's audio arrives faster than it plays and the buffer fills up, the oldest audio is dropped.

In Python, these options tune it:

Keep echo cancellation on when you use speakers instead of headphones. Otherwise, the agent can hear and answer itself.

from mistralai.extra.voice_agents import DefaultAudioInterface

audio_interface = DefaultAudioInterface(
    noise_suppression=False,  # The input is already clean
    output_buffer_size_ms=500,
)

from mistralai.extra.voice_agents import DefaultAudioInterface

audio_interface = DefaultAudioInterface(
    noise_suppression=False,  # The input is already clean
    output_buffer_size_ms=500,
)

To use another audio source or destination, subclass AudioInterface (implement it in TypeScript) and pass an instance with audio_interface. It has four methods:

from collections.abc import AsyncIterator

from mistralai.extra.voice_agents import AudioInterface

class MyAudioInterface(AudioInterface):
    async def start(self) -> None:
        """Open devices, sockets, or other audio resources."""
        ...

    async def input_audio(self) -> AsyncIterator[bytes]:
        """Yield user audio chunks to send to the agent."""
        ...

    async def output_audio(self, audio: bytes) -> None:
        """Play or forward agent audio."""
        ...

    async def stop(self) -> None:
        """Release resources."""
        ...

session = await client.audio.sessions.init(
    instructions="You are a helpful voice assistant.",
    audio_interface=MyAudioInterface(),
)

from collections.abc import AsyncIterator

from mistralai.extra.voice_agents import AudioInterface

class MyAudioInterface(AudioInterface):
    async def start(self) -> None:
        """Open devices, sockets, or other audio resources."""
        ...

    async def input_audio(self) -> AsyncIterator[bytes]:
        """Yield user audio chunks to send to the agent."""
        ...

    async def output_audio(self, audio: bytes) -> None:
        """Play or forward agent audio."""
        ...

    async def stop(self) -> None:
        """Release resources."""
        ...

session = await client.audio.sessions.init(
    instructions="You are a helpful voice assistant.",
    audio_interface=MyAudioInterface(),
)

In Python, you can also override preflight(). The SDK calls it before creating the session, so a missing device fails early instead of after the session is allocated. To change only part of the default behavior, subclass the default interface instead of starting from scratch.

If start(), input_audio(), or output_audio() raises an error, the session closes with the start_failed, audio_input_error, or audio_output_error disconnect reason. If stop() fails, the session emits a non-fatal error event and still closes.

If you don't pass an audio interface, the session neither records nor plays anything. Send audio to the agent with send_audio() (sendAudio() in TypeScript), in the same format, once the session is connected. Receive the agent's audio from the audio event.

session = await client.audio.sessions.init(
    instructions="You are a helpful voice assistant.",
)

async def handle_audio(audio: bytes) -> None:
    ...  # Play or forward agent audio

session.on("audio", handle_audio)

async with session:
    await session.send_audio(audio_bytes)  # 16-bit PCM, mono, 48 kHz
    await session.wait()

session = await client.audio.sessions.init(
    instructions="You are a helpful voice assistant.",
)

async def handle_audio(audio: bytes) -> None:
    ...  # Play or forward agent audio

session.on("audio", handle_audio)

async with session:
    await session.send_audio(audio_bytes)  # 16-bit PCM, mono, 48 kHz
    await session.wait()