Home
Vibe
Studio

Offline evaluations

The Evaluation SDK (mistralai-observability) lets you run offline evaluations on your LLM pipelines in a few lines of Python. Define a dataset, a task, and scorers to get results in your terminal and in Studio.

i
Information

Offline evaluations are available to Enterprise-tier organizations only. Contact your Mistral representative to get access to the mistralai-observability package.

The SDK follows a simple mental model:

Dataset  →  Task  →  Evaluators  →  Results
  • Dataset: a list of input records (dicts) with your test cases.
  • Task: an async or sync function that receives a TaskContext and produces an output.
  • Evaluators: functions that receive a ScorerContext and score each output.
  • Results: statistics and per-record scores, displayed in the terminal and uploaded to Studio.
Installation

Installation

Prerequisites: Python 3.12+, uv, and a Mistral API key.

After you receive your entitlement token from Mistral, configure your credentials:

export UV_INDEX_CLOUDSMITH_USERNAME=token
export UV_INDEX_CLOUDSMITH_PASSWORD=<your-entitlement-token>

Add the package and the Mistral private repository to your pyproject.toml:

[project]
name = "my-eval-project"
version = "0.1.0"
dependencies = ["mistralai-observability"]

[[tool.uv.index]]
name = "cloudsmith"
url = "https://dl.cloudsmith.io/basic/mistral-ai/sdk-distribution/python/simple/"

Then install:

uv sync

Set your API key:

export MISTRAL_API_KEY=<your-api-key>
Quickstart

Quickstart

Create an eval.py file:

import asyncio
import os

from mistralai.observability import (
    Evaluation, Evaluator, Goal, Mistral, Project, ScorerContext, System, TaskContext,
)

client = Mistral(api_key=os.environ["MISTRAL_API_KEY"])

dataset = [
    {"sentence": "Hello, how are you?", "groundtruth": "English"},
    {"sentence": "Bonjour, comment ça va?", "groundtruth": "French"},
    {"sentence": "Hola, ¿cómo estás?", "groundtruth": "Spanish"},
]

async def task(ctx: TaskContext) -> str:
    response = await client.chat.complete_async(
        model=str(ctx.system.params["model"]),
        messages=[
            {"role": "system", "content": str(ctx.system.params["system_prompt"])},
            {"role": "user", "content": ctx.input_record["sentence"]},
        ],
    )
    return str(response.choices[0].message.content)

def scorer(ctx: ScorerContext) -> int:
    return 1 if ctx.input_record["groundtruth"].lower() == str(ctx.output).lower() else 0

async def main():
    run = await client.evaluation.run(
        project=Project(name="Language Detection"),
        evaluation=Evaluation(name="Accuracy Eval"),
        system=System(name="mistral-small", params={
            "model": "mistral-small-latest",
            "system_prompt": "What language is this sentence in? Reply with ONLY the language name.",
        }),
        dataset=dataset,
        task=task,
        evaluators=[
            Evaluator(
                name="accuracy",
                description="1 if the detected language matches the groundtruth, 0 otherwise.",
                scorer=scorer,
                goal=Goal.gte(0.8),
            ),
        ],
    )
    run.show(level="records")

asyncio.run(main())

Run it:

uv run eval.py

Results are printed in your terminal and uploaded to Studio under Observability > Evaluate > Evaluations.

Core concepts

Core concepts

A task is an async (or sync) function that receives a TaskContext and returns an output. This is the code you want to evaluate:

from mistralai.observability import TaskContext

async def task(ctx: TaskContext) -> str:
    response = await client.chat.complete_async(
        model=str(ctx.system.params["model"]),
        messages=[
            {"role": "system", "content": str(ctx.system.params["system_prompt"])},
            {"role": "user", "content": ctx.input_record["prompt"]},
        ],
    )
    return str(response.choices[0].message.content)

The task can return any type: strings, Pydantic models, or dicts. The SDK serializes the output for upload.

TaskContext gives you typed access to ctx.input_record (the current dataset record), ctx.system (the System config from evaluation.run()), and ctx.metadata (run metadata).

Organizing runs in Studio

Organizing runs in Studio

Use Projects and Evaluations to organize runs:

  • Project: groups related evaluations for a system (for example, "Language Detection").
  • Evaluation: a specific evaluation within a project (for example, "Accuracy Eval").
  • Run: each call to evaluation.run() creates a new run under the evaluation.
run = await client.evaluation.run(
    project=Project(name="My Project"),       # created if it doesn't exist
    evaluation=Evaluation(name="My Eval"),    # created if it doesn't exist
    dataset=dataset,
    task=task,
    evaluators=[...],
)

You can also reference existing entities by slug:

project=Project(slug="my-project")
evaluation=Evaluation(slug="my-eval")

Tags and metadata help you filter and compare runs in Studio:

run = await client.evaluation.run(
    ...
    tags=["model:mistral-small", "prompt:v2"],
    metadata={"commit": "abc123"},
)
Local mode

Local mode

Use local=True to run evaluations without uploading results to Studio. This is useful for fast iteration during development:

run = await client.evaluation.run(
    dataset=dataset,
    task=task,
    evaluators=[Evaluator(name="accuracy", scorer=scorer)],
    local=True,  # no upload, no project/evaluation required
)
run.show(level="records")

A typical development workflow:

  1. Iterate locally: set local=True and adjust your task and scorers until results look right.
  2. Push to Studio: remove local=True and add project and evaluation to track results over time.
Displaying results

Displaying results

The run.show() method prints results at different levels of detail:

run.show(level="run")          # Summary statistics only
run.show(level="records")      # Per-record results
run.show(level="generations")  # Per-generation details
run.show(level="scores")       # Full score breakdown

Export as JSON for further processing:

run.show(mode="json", level="records")
Results in Studio

Results in Studio

Once a run completes, results are available at Observability > Evaluate > Evaluations in Studio. Each run uploads its scores, statistics, and metadata automatically.

The Studio UI gives you four ways to explore results:

  • Trend charts: score averages over time with min/max bands, so you can spot regressions across runs.
  • Distribution charts: pie and stacked area charts showing score distribution across records and runs.
  • Composable filters: filter by tag, run ID, or metric threshold to isolate specific subsets.
  • Run detail view: open any run to inspect per-record inputs, outputs, and scores — with expandable generation details when num_generations > 1.

Cell values render automatically based on type: plain text, JSON objects, and LLM conversation arrays each get their own display format.

Guides

Guides

FAQ

FAQ