Custom audio interfaces

By default, the SDK records from your microphone and plays to your speakers. When audio comes from somewhere else, or you need to process it, you can take over at two levels: implement your own audio interface, or send and receive the audio yourself. Typical cases are a WebSocket stream, a telephony bridge, a visualization, or a test that feeds recorded audio.

Implement an audio interface when you want the SDK to keep running the session while you own the audio streams, for example to process or visualize them, or to read from another device. Handle audio manually when you want to drive the exchange yourself, for example from a WebSocket, a phone bridge, or a test harness.

An audio interface has four methods. The SDK calls start() when the session opens, then reads your audio from input_audio() and passes the agent's audio to output_audio() for as long as the session runs. It calls stop() when the session ends, so you can release your resources. Pass your interface to init() with audio_interface.

from collections.abc import AsyncIterator

from mistralai.extra.voice_agents import AudioInterface

class MyAudioInterface(AudioInterface):
    async def start(self) -> None:
        """Initialize audio resources."""
        ...

    def input_audio(self) -> AsyncIterator[bytes]:
        """Yield audio chunks from your source."""
        ...

    async def output_audio(self, audio: bytes) -> None:
        """Play or process agent audio output."""
        ...

    async def stop(self) -> None:
        """Clean up resources."""
        ...

# then pass it to init()
session = await client.audio.sessions.init(
    llm={"model": "mistral-small-latest"},
    tts={"voice_id": "fr_marie_neutral"},
    audio_interface=MyAudioInterface(),
)

from collections.abc import AsyncIterator

from mistralai.extra.voice_agents import AudioInterface

class MyAudioInterface(AudioInterface):
    async def start(self) -> None:
        """Initialize audio resources."""
        ...

    def input_audio(self) -> AsyncIterator[bytes]:
        """Yield audio chunks from your source."""
        ...

    async def output_audio(self, audio: bytes) -> None:
        """Play or process agent audio output."""
        ...

    async def stop(self) -> None:
        """Clean up resources."""
        ...

# then pass it to init()
session = await client.audio.sessions.init(
    llm={"model": "mistral-small-latest"},
    tts={"voice_id": "fr_marie_neutral"},
    audio_interface=MyAudioInterface(),
)

To change only part of the default behavior, subclass the default interface instead of starting from scratch.

If you don't pass an audio interface to init(), the session neither records nor plays anything. Send audio to the agent with send_audio() (sendAudio() in TypeScript), and receive the agent's audio from the audio event.

session = await client.audio.sessions.init(
    llm={"model": "mistral-small-latest"},
    tts={"voice_id": "fr_marie_neutral"},
    instructions="You are a helpful voice assistant.",
)

# Send audio to the agent (16-bit PCM, mono, 48 kHz)
await session.send_audio(audio_bytes)

# Receive agent audio output
session.on("audio", lambda audio: handle_audio(audio))

session = await client.audio.sessions.init(
    llm={"model": "mistral-small-latest"},
    tts={"voice_id": "fr_marie_neutral"},
    instructions="You are a helpful voice assistant.",
)

# Send audio to the agent (16-bit PCM, mono, 48 kHz)
await session.send_audio(audio_bytes)

# Receive agent audio output
session.on("audio", lambda audio: handle_audio(audio))