Observability

AI applications are harder to debug than traditional software. The same prompt can produce different responses, tool calls can fail silently, and quality degrades in ways that unit tests don't catch. You only find out something went wrong when a user tells you.

Observability gives you the data to get ahead of that. The suite has three capabilities:

• Traces: collect and explore every request flowing through your AI application in production.
• Offline evaluations: measure and track pipeline quality systematically, before issues reach users.
• Online evaluations: score production traffic continuously with LLM judges.

Every request in your AI application becomes a trace: a tree of spans representing every step in the execution chain. Each span carries its input, output, latency, token counts, and status.

Traces are collected using OpenTelemetry and are supported across Mistral products: the Mistral SDK (Python and TypeScript), Workflows, and Vibe Work. Any application instrumented with OpenTelemetry can also send traces directly.

After traces are flowing, the Trace Explorer in Studio lets you search and filter across all requests, inspect individual executions end to end, and debug failures at the span level.

To instrument your application, see Send traces. To search, filter, and inspect your traces in Studio, see Explore traces.

Sending traces requires only a valid Mistral API key: no special role or feature flag is needed. If an Org Admin disabled observability for the organization, Mistral rejects the traces.

Reading traces in Studio requires an Enterprise organization and one of the following roles:

Other workspace members don't have access to trace data. A Workspace Admin or Org Admin can assign the Observability Viewer role to grant read-only access without admin privileges.

Production traces tell you what your application did. Offline evaluations tell you how well it performed, and whether it's improving or regressing as you change your pipeline.

You define a set of test cases, run them through your pipeline, and score the outputs against criteria you control. Results upload to Studio, where you can track quality trends over time and compare pipeline configurations side by side.

The evaluation workflow has three building blocks:

• Evaluation SDK (mistralai-evaluations): the Python package that runs evaluations, computes statistics, and uploads results. To run evaluations inside Workflows, use the Workflow evaluation plugin (mistralai-workflows-plugins-evaluations).
• Evaluators: scorers for each metric, either rule-based functions or LLM judges for criteria that code can't capture, such as helpfulness, factual accuracy, or tone.
• Datasets: your test cases as a list of input records, with fields like prompts, expected outputs, and grading guidance.

To get started with the Evaluation SDK, see Offline evaluations. To run evaluations as a workflow, see the Workflow evaluation plugin.

Online evaluations score production traffic as it arrives. You write a scorer, an LLM judge with a prompt and an output type, and attach it to a live eval that selects which spans to score. Each score is stored with the span it scored, so you can search it, chart it, and open it from the trace.

Scoring runs after ingestion, outside the request path of your application.

To set up your first live eval, see Online evaluations.