Online evaluations
Online evaluation runs LLM judges on production traffic. Your application sends OpenTelemetry spans to Mistral, and a judge scores the spans you select. Each score is written back as telemetry, attached to the span it scored, so you can search it, chart it, and open it from the trace.
Nothing happens on the request path. Scoring runs asynchronously after ingestion and cannot slow down or drop your telemetry.
Online evaluations are in Public Preview and are available to Enterprise-tier organizations only, like the rest of Observability. To report an issue or share feedback, contact Mistral support.
How it fits together
You configure two things:
| Object | What it is | Owns |
|---|---|---|
| Scorer | A reusable judge: model, prompt template, output type. | Prompt, model, result spec. Versioned. |
| Live eval | Which spans to score, and with which scorers. | Filter, scorers and their variable mapping, sampling, on/off. |
A live eval attaches one or more scorers. For each attached scorer, the system creates a pipeline config with its own slug (<scorer-slug>-<8 hex>). That slug names the results: every score from that scorer in that live eval has evaluation_name = "judge:<pipeline-config-slug>".
Whenever you need to find results, start from the pipeline config slug. The live eval's own slug and the scorer slug do not appear in results.
Before you start
- Workspace role. Creating and editing scorers works with any role that has AI Studio access, including the default Workspace Contributor role. Creating and managing live evals requires the Workspace Admin or Observability Viewer role. Reading results requires the Workspace Admin, Observability Viewer, or Org Admin role. Create live evals with a user API key: service account keys are rejected. Sending telemetry only requires a valid Mistral API key.
- Span attributes to judge. A judge prompt can use any span attribute, through a mapping or directly with
attr("key"). To judge the conversation itself, the spans needgen_ai.input.messagesandgen_ai.output.messages; the Mistral OTel SDK records them when the operation is created withinclude_content=True. See the Quickstart.
Behaviour to know about
- Changes go live without a deploy. Workers reload configuration when it changes. Creating, pausing, or deleting a live eval takes effect on the next spans that arrive.
- Scores arrive within seconds. Scoring starts right after ingestion, and most judge calls complete in a few seconds.
- Identical inputs are cached. A span whose rendered prompt, model, and parameters match an earlier one gets its score from the cache, without calling the model.
- Failures are visible in the evaluation trace. Every evaluation runs in its own trace span, linked to the span it evaluates. If the judge call or prompt rendering fails, that evaluation span is marked as an error with the reason.
- Overload drops evaluations, not telemetry. If the judge queue is full, the evaluation for that span is dropped: the span gets no score, and your telemetry is unaffected.
- Scorer edits do not reach running live evals by themselves. A live eval uses the scorer version that was current when its scorers were last set. See Scorers.