Diarize speakers

Diarization labels each part of a transcript with the speaker who said it, in a recorded file or a live session. This page shows how to turn it on in each mode, read the labels, and name known speakers with reference clips.

Diarization works offline and in realtime, with the same diarize option. Only the way you send it and the way labels come back differ.

To diarize a file, set diarize to true and timestamp_granularities to segment in the transcription request. Diarization labels segments, so the request fails without segment timestamps.

import os

from mistralai.client import Mistral

client = Mistral(api_key=os.environ["MISTRAL_API_KEY"])

with open("customer_call.wav", "rb") as audio:
    transcription = client.audio.transcriptions.complete(
        model="voxtral-mini-transcribe-3",
        file={"file_name": "customer_call.wav", "content": audio},
        diarize=True,
        timestamp_granularities=["segment"],
    )

for segment in transcription.segments:
    print(f"[{segment.start:5.1f}s - {segment.end:5.1f}s] {segment.speaker_id}: {segment.text.strip()}")

import os

from mistralai.client import Mistral

client = Mistral(api_key=os.environ["MISTRAL_API_KEY"])

with open("customer_call.wav", "rb") as audio:
    transcription = client.audio.transcriptions.complete(
        model="voxtral-mini-transcribe-3",
        file={"file_name": "customer_call.wav", "content": audio},
        diarize=True,
        timestamp_granularities=["segment"],
    )

for segment in transcription.segments:
    print(f"[{segment.start:5.1f}s - {segment.end:5.1f}s] {segment.speaker_id}: {segment.text.strip()}")

The response holds the transcript as a list of segments, each with its speaker:

{
  "segments": [
    {
      "type": "transcription_segment",
      "text": "Thanks for calling Northwind Support.",
      "start": 0.0,
      "end": 1.6,
      "speaker_id": "speaker_1"
    },
    {
      "type": "transcription_segment",
      "text": " This is Daniel.",
      "start": 1.7,
      "end": 2.4,
      "speaker_id": "speaker_1"
    },
    {
      "type": "transcription_segment",
      "text": " How can I help you today?",
      "start": 2.6,
      "end": 3.5,
      "speaker_id": "speaker_1"
    },
    {
      "type": "transcription_segment",
      "text": " Hi Daniel.",
      "start": 4.1,
      "end": 4.8,
      "speaker_id": "speaker_2"
    }
  ]
}

{
  "segments": [
    {
      "type": "transcription_segment",
      "text": "Thanks for calling Northwind Support.",
      "start": 0.0,
      "end": 1.6,
      "speaker_id": "speaker_1"
    },
    {
      "type": "transcription_segment",
      "text": " This is Daniel.",
      "start": 1.7,
      "end": 2.4,
      "speaker_id": "speaker_1"
    },
    {
      "type": "transcription_segment",
      "text": " How can I help you today?",
      "start": 2.6,
      "end": 3.5,
      "speaker_id": "speaker_1"
    },
    {
      "type": "transcription_segment",
      "text": " Hi Daniel.",
      "start": 4.1,
      "end": 4.8,
      "speaker_id": "speaker_2"
    }
  ]
}

The Python script prints one line per segment:

[  0.0s -   1.6s] speaker_1: Thanks for calling Northwind Support.
[  1.7s -   2.4s] speaker_1: This is Daniel.
[  2.6s -   3.5s] speaker_1: How can I help you today?
[  4.1s -   4.8s] speaker_2: Hi Daniel.
[  5.1s -  13.2s] speaker_2: I ordered two Quilliera headsets last week, order 4417, and the tracking page still says pending.
[ 13.8s -  14.3s] speaker_1: Let me check.

[  0.0s -   1.6s] speaker_1: Thanks for calling Northwind Support.
[  1.7s -   2.4s] speaker_1: This is Daniel.
[  2.6s -   3.5s] speaker_1: How can I help you today?
[  4.1s -   4.8s] speaker_2: Hi Daniel.
[  5.1s -  13.2s] speaker_2: I ordered two Quilliera headsets last week, order 4417, and the tracking page still says pending.
[ 13.8s -  14.3s] speaker_1: Let me check.

To diarize a live session, set diarize to true in the session.update message, before you send the first audio chunk:

{
  "type": "session.update",
  "session": {
    "audio_format": { "encoding": "pcm_s16le", "sample_rate": 16000 },
    "diarize": true
  }
}

{
  "type": "session.update",
  "session": {
    "audio_format": { "encoding": "pcm_s16le", "sample_rate": 16000 },
    "diarize": true
  }
}

With diarization on, the session sends transcription.segment events instead of transcription.text.delta events. Each event carries a few words and the label of their speaker:

{"type": "transcription.segment", "text": " today?", "start": null, "end": null, "speaker_id": "speaker_1"}
{"type": "transcription.segment", "text": " Hi,", "start": null, "end": null, "speaker_id": "speaker_2"}
{"type": "transcription.segment", "text": " Daniel.", "start": null, "end": null, "speaker_id": "speaker_2"}

{"type": "transcription.segment", "text": " today?", "start": null, "end": null, "speaker_id": "speaker_1"}
{"type": "transcription.segment", "text": " Hi,", "start": null, "end": null, "speaker_id": "speaker_2"}
{"type": "transcription.segment", "text": " Daniel.", "start": null, "end": null, "speaker_id": "speaker_2"}

For a complete realtime client, see the Python script in Name speakers with reference clips: it sends the same session configuration, plus reference clips.

Labels carry the same values in both modes, but arrive in a different place:

To show the transcript as turns, start a new turn each time speaker_id changes and append the text of the following items to it. Both Python examples on this page print the transcript this way.

When two people talk at the same time, the transcript has to place both voices on the same stretch of time. Overlapping speech (two speakers talking at the same time) is not supported in this version.

If your application draws segments on a timeline, or shows them as a chat thread, don't assume that each segment ends before the next one starts:

• Order segments by start, and keep both segments when their times overlap.
• Show two active speakers at once, for example on separate tracks of a timeline, instead of cutting one segment short.
• Merge consecutive segments of the same speaker into turns only when they don't overlap with another speaker.

When you know who speaks, send a short clip of each person's voice under a name. Segments from that voice then carry the name in speaker_id instead of a generic label. Reference clips follow these rules:

The closer the acoustic and speaking conditions of the reference clip are to the audio being transcribed, the better the match. For more guidance on choosing good reference clips, see Naming speakers with reference clips.

{
  "type": "session.update",
  "session": {
    "audio_format": { "encoding": "pcm_s16le", "sample_rate": 24000 },
    "diarize": true,
    "speaker_references": {
      "agent": "<base64 of agent_reference.wav>",
      "customer": "<base64 of customer_reference.wav>"
    }
  }
}

{
  "type": "session.update",
  "session": {
    "audio_format": { "encoding": "pcm_s16le", "sample_rate": 24000 },
    "diarize": true,
    "speaker_references": {
      "agent": "<base64 of agent_reference.wav>",
      "customer": "<base64 of customer_reference.wav>"
    }
  }
}

import asyncio
import base64
import json
import os
import wave

import websockets

MODEL = "voxtral-mini-transcribe-realtime-3"
URL = f"wss://api.mistral.ai/v1/audio/transcriptions/realtime?model={MODEL}"
HEADERS = {"Authorization": f"Bearer {os.environ['MISTRAL_API_KEY']}"}
AUDIO_FILE = "customer_call.wav"  # 16-bit mono PCM WAV

def b64_file(path):
    with open(path, "rb") as f:
        return base64.b64encode(f.read()).decode()

async def send_audio(ws, path):
    with wave.open(path, "rb") as wav:
        while chunk := wav.readframes(wav.getframerate() // 10):
            await ws.send(json.dumps({"type": "input_audio.append", "audio": base64.b64encode(chunk).decode()}))
    await ws.send(json.dumps({"type": "input_audio.end"}))

async def main():
    with wave.open(AUDIO_FILE, "rb") as wav:
        sample_rate = wav.getframerate()

    async with websockets.connect(URL, additional_headers=HEADERS) as ws:
        await ws.recv()  # session.created
        await ws.send(json.dumps({
            "type": "session.update",
            "session": {
                "audio_format": {"encoding": "pcm_s16le", "sample_rate": sample_rate},
                "diarize": True,
                "speaker_references": {
                    "agent": b64_file("agent_reference.wav"),
                    "customer": b64_file("customer_reference.wav"),
                },
            },
        }))
        sender = asyncio.create_task(send_audio(ws, AUDIO_FILE))

        speaker = None
        async for message in ws:
            event = json.loads(message)
            if event["type"] == "transcription.segment":
                text = event["text"]
                if event["speaker_id"] != speaker:
                    speaker = event["speaker_id"]
                    print(f"\n{speaker}: ", end="")
                    text = text.lstrip()
                print(text, end="", flush=True)
            elif event["type"] == "transcription.done":
                print()
                break
            elif event["type"] == "error":
                print("error:", event["error"]["message"])
                break
        sender.cancel()

asyncio.run(main())

import asyncio
import base64
import json
import os
import wave

import websockets

MODEL = "voxtral-mini-transcribe-realtime-3"
URL = f"wss://api.mistral.ai/v1/audio/transcriptions/realtime?model={MODEL}"
HEADERS = {"Authorization": f"Bearer {os.environ['MISTRAL_API_KEY']}"}
AUDIO_FILE = "customer_call.wav"  # 16-bit mono PCM WAV

def b64_file(path):
    with open(path, "rb") as f:
        return base64.b64encode(f.read()).decode()

async def send_audio(ws, path):
    with wave.open(path, "rb") as wav:
        while chunk := wav.readframes(wav.getframerate() // 10):
            await ws.send(json.dumps({"type": "input_audio.append", "audio": base64.b64encode(chunk).decode()}))
    await ws.send(json.dumps({"type": "input_audio.end"}))

async def main():
    with wave.open(AUDIO_FILE, "rb") as wav:
        sample_rate = wav.getframerate()

    async with websockets.connect(URL, additional_headers=HEADERS) as ws:
        await ws.recv()  # session.created
        await ws.send(json.dumps({
            "type": "session.update",
            "session": {
                "audio_format": {"encoding": "pcm_s16le", "sample_rate": sample_rate},
                "diarize": True,
                "speaker_references": {
                    "agent": b64_file("agent_reference.wav"),
                    "customer": b64_file("customer_reference.wav"),
                },
            },
        }))
        sender = asyncio.create_task(send_audio(ws, AUDIO_FILE))

        speaker = None
        async for message in ws:
            event = json.loads(message)
            if event["type"] == "transcription.segment":
                text = event["text"]
                if event["speaker_id"] != speaker:
                    speaker = event["speaker_id"]
                    print(f"\n{speaker}: ", end="")
                    text = text.lstrip()
                print(text, end="", flush=True)
            elif event["type"] == "transcription.done":
                print()
                break
            elif event["type"] == "error":
                print("error:", event["error"]["message"])
                break
        sender.cancel()

asyncio.run(main())

agent: Thanks for calling Northwind Support. This is Daniel. How can I help you today?
customer: Hi, Daniel. I ordered two Quilera headsets last week. Order four, 417. And the tracking page still says pending.
agent: Let me check. I can see order 4417. It left our Rotterdam warehouse yesterday, but the tracking was not updated.
customer: Okay. Can you send me the new tracking link? And I also need an invoice for my company, Alder Creek Design.

agent: Thanks for calling Northwind Support. This is Daniel. How can I help you today?
customer: Hi, Daniel. I ordered two Quilera headsets last week. Order four, 417. And the tracking page still says pending.
agent: Let me check. I can see order 4417. It left our Rotterdam warehouse yesterday, but the tracking was not updated.
customer: Okay. Can you send me the new tracking link? And I also need an invoice for my company, Alder Creek Design.

A realtime session can label speakers or detect turns, not both. If one session.update sets diarize to true and includes turn_detection, the session sends an error event. Turn detection only exists in realtime, so offline requests aren't affected.

For which feature fits your application, see Turn detection and diarization. For the turn detection settings, see Turn detection.

Offline requests return an HTTP error. Realtime sessions send an event of type error, with the message in error.message. These errors are specific to diarization:

For other transcription errors, see Troubleshooting.