Custom vocabulary
Some words are rare in the data a model learns from, such as names, drug names, product terms, or jargon, so the model can get them wrong in both directions: transcription misspells them, and generated speech mispronounces them. This page explains why, and which hint to give the model in each direction.
Which hint applies
Each direction has its own hint, and neither works in the other:
| Transcription (speech to text) | Speech generation (text to speech) | |
|---|---|---|
| What goes wrong | The spelling of the word | The way the word sounds |
| The hint | Context biasing: the terms to expect, or a description of the recording | Custom pronunciation: the sounds of the word, as phonemes |
| What you give | Text | Phonemes |
| Where it works | File and realtime transcription, and the transcription settings of voice agents | Speech generation requests. In voice agents, set phoneme_hints in the tts config of client.audio.sessions.init(). |
| Settings | Bias context and vocabulary | Custom pronunciation |
In transcription: context biasing
The model hears sounds, not spellings. For each stretch of audio, it picks the words that best fit what it learned, and nothing in the audio says that a word is a product name. When a rare name sounds like familiar words, the familiar version wins: a rare surname can come out as a more common one, a drug name as an everyday word, or a product name split into two familiar words. Context biasing tells the model which names and terms to expect, so they come out spelled the way you write them.
Not every name needs help, so transcribe a few recordings without context first to find the terms that come out wrong. Fixing them afterwards with a find-and-replace is fragile: a misheard name doesn't always come out the same way, and context fixes the spelling as the model transcribes.
Two kinds of context
You can tell the model what to expect in two ways: list the terms, or describe the recording in plain language. The model accepts both in the same list, where each entry is either a term or a passage of text.
| Keyphrases | Free-form context | |
|---|---|---|
| What you give | The terms that must come out right, one per entry: names, acronyms, product codes. An entry can hold several words, such as "Customer Success Platform". | A sentence or a paragraph that describes the recording: its subject, the people, and the names and tools that come up. |
| Example, for a medical consultation | Dr. Nakamura, metformin, HbA1c | "Follow-up consultation with Dr. Nakamura about a patient's diabetes treatment: the metformin dose and the latest HbA1c results." |
| Fits when | You know the vocabulary. | You know the situation, but can't list every term in advance. |
Give every term that must come out right its own entry. When the recording has a clear subject, add a sentence that describes it. In realtime, rely on keyphrases: a sentence alone may not fix the terms that keyphrases fix.
What biasing does and doesn't do
Context changes which words the model prefers. It doesn't change what the model hears.
- It favors your terms, but doesn't force them. The model still writes what it hears, so a listed term can still come out wrong. Check the transcripts of a few real recordings after you add context.
- It applies to one request or one session. Send the context with each file you transcribe, or at the start of each realtime session, before any audio. You can't change it once the audio has started.
- It shares room with the audio. Context counts toward the same model context as the audio, so a very long list on a long recording can exceed it. The request then fails with an error: the API doesn't shorten your list. See Bias context and vocabulary for the error.
In speech generation: custom pronunciation
A voice reads text by guessing how each word sounds from its spelling. Common words are easy. A surname from another language, a brand name with an unusual spelling, or an acronym that you read as a word has no obvious reading, so the voice can say it wrong, and the same spelling doesn't tell it which reading you want.
Custom pronunciation removes the guess: for each word that needs it, you give the sounds to say, written as phonemes, the units of a phonetic alphabet. The voice then says that word your way, in every sentence of the request where it appears.
- It works per request. Send the pronunciations with each speech request that contains the words.
- It uses a fixed set of phonemes. The API accepts only the phonemes of its own list, and rejects a request with any other symbol. The list is on Custom pronunciation.
- It changes the words, not the delivery. To change the tone or the pace, use emotion and pace tags or style instructions.
Applications that hear and speak
An application that both listens and talks, such as an voice agent, can meet the same name in both directions: when the user says it, and when the agent says it back. Give each direction its own hint: the written name as context for the transcription, and its phonemes for the voice. In voice agents, context biasing goes in the stt settings and phoneme hints in the tts settings.
Pinning the language is another way to avoid wrong words in transcription, but it works on a different level: it tells the model which language to transcribe, not which words to pick. Set it on File transcription or Realtime transcription, with the codes on Languages.