Audio in chat completions
Audio in chat completions lets a model listen to a recording and answer about it in text: a summary, an answer to a question, a translation, or structured data. This page shows the request once, then the instruction to write for each task.
To get a word-for-word transcript instead of answers, use File transcription.
Send audio in a chat request
Put the audio and your instruction in the same user message: an input_audio part with the recording, base64-encoded, and a text part with what you want to know. Every example on this page uses the same recording of a support call, so only the instruction changes from one task to the next.
import base64
import os
from mistralai.client import Mistral
client = Mistral(api_key=os.environ["MISTRAL_API_KEY"])
with open("support_call_8k.wav", "rb") as f:
audio_base64 = base64.b64encode(f.read()).decode()
response = client.chat.complete(
model="voxtral-small-latest",
messages=[
{
"role": "user",
"content": [
{"type": "input_audio", "input_audio": audio_base64},
{
"type": "text",
"text": "Summarize this support call in three bullet points: the problem, what was checked, and the outcome.",
},
],
}
],
)
print(response.choices[0].message.content)input_audio takes the base64 string of the file directly. The object form used by OpenAI's API, with data and format fields, is accepted too.
See Limits.
Summarize a recording
To get a summary, describe the one you need in the text part: its length, its format, and who reads it. The example in Send audio in a chat request asks for this:
Summarize this support call in three bullet points: the problem, what was checked, and the outcome.
Ask questions about a recording
To ask about a recording, put your question in the text part, in place of the summary instruction.
Which order number does the customer give, and what does the agent promise to send?
Translate a recording
To translate a recording, name the language of the audio and the language you want in the instruction. The answer is written text, not speech: to hear the translation, send the text to Speech generation.
To get the original words and the translation in one answer, ask for two labeled sections. This instruction, sent with a French recording, returns the French transcript and its English translation:
This recording is in French. Return two sections exactly: ORIGINAL: the verbatim French transcript. TRANSLATION: a faithful English translation. Do not add commentary.
The two sections carry no timestamps, so you can't match a line of the translation to a moment of the audio.
Supported language pairs
For the source and target languages that translation supports, see Translation on Languages.
Extract structured data
With function calling, the model reads the recording and fills in the arguments of a function you define, such as the follow-up actions agreed in a call. It doesn't run the function: your application does. Define the function in tools, and set the tool choice to any so the model must call it.
For how function calling works in chat completions, see Function calling.
import base64
import json
import os
from mistralai.client import Mistral
client = Mistral(api_key=os.environ["MISTRAL_API_KEY"])
with open("support_call_8k.wav", "rb") as f:
audio_base64 = base64.b64encode(f.read()).decode()
tools = [
{
"type": "function",
"function": {
"name": "record_action_items",
"description": "Record the follow-up actions agreed during the call.",
"parameters": {
"type": "object",
"properties": {
"items": {
"type": "array",
"items": {
"type": "object",
"properties": {
"owner": {"type": "string", "description": "Who does it (agent, billing, customer...)"},
"task": {"type": "string"},
"due": {"type": "string", "description": "Deadline if one was stated"},
},
"required": ["owner", "task"],
},
},
"order_id": {"type": "string"},
},
"required": ["items"],
},
},
}
]
response = client.chat.complete(
model="voxtral-small-latest",
messages=[
{
"role": "user",
"content": [
{"type": "input_audio", "input_audio": audio_base64},
{"type": "text", "text": "Extract every follow-up action agreed in this call."},
],
}
],
tools=tools,
tool_choice="any",
)
tool_call = response.choices[0].message.tool_calls[0]
print(tool_call.function.name)
print(json.dumps(json.loads(tool_call.function.arguments), indent=2))The answer carries a tool call to record_action_items, with arguments that follow the schema of the function: the order number, and one item per action with its owner, its task, and its deadline. Your application reads them and acts on them.
Common issues
These problems are specific to audio in chat requests:
| Symptom | Cause | Fix |
|---|---|---|
| The answer rephrases what was said instead of quoting it | A chat model answers your instruction. It doesn't return a word-for-word transcript. | For the exact words, use File transcription. |
429 error, Rate limit exceeded, on a request with audio | Send a shorter clip, or a WAV file at 8 or 16 kHz. See Limits. | |
| Part of the MP3 isn't read. | Send the recording as WAV. See Audio formats. |