API reference

Reference for the Evaluation SDK's core types and the main evaluation.run() entry point. For the method-level reference of the advanced operations, see Retry failed records, Rescore persisted runs, and Optimize prompts and parameters.

The main entry point for running evaluations.

run = await client.evaluation.run(
    dataset=...,
    task=...,
    evaluators=...,
    # optional parameters below
)

run = await client.evaluation.run(
    dataset=...,
    task=...,
    evaluators=...,
    # optional parameters below
)

It returns an EvaluationRun:

A project groups related evaluations together. Think of it as a folder — for example, "Chatbot QA" or "RAG Pipeline".

Project(name="My Project")       # create or get by name
Project(slug="my-project")       # get by slug

Project(name="My Project")       # create or get by name
Project(slug="my-project")       # get by slug

At least one of name or slug must be provided. If the project doesn't exist yet, it is created automatically.

An evaluation is a named test you run repeatedly over time — for example, "Accuracy on French prompts". Each call to evaluation.run() creates a new run under it, so you can track how scores evolve across runs.

Evaluation(name="My Eval")       # create or get by name
Evaluation(slug="my-eval")       # get by slug

Evaluation(name="My Eval")       # create or get by name
Evaluation(slug="my-eval")       # get by slug

At least one of name or slug must be provided. If the evaluation doesn't exist yet, it is created automatically under the given project.

An evaluator defines how to score each individual record. It pairs a name with a scorer function that receives a ScorerContext and returns a score.

Evaluator(
    name="accuracy",
    scorer=my_scorer,
    description="Optional description",
    tags=["tag1"],
    num_scores=1,
    goal=Goal.gte(0.8),
)

Evaluator(
    name="accuracy",
    scorer=my_scorer,
    description="Optional description",
    tags=["tag1"],
    num_scores=1,
    goal=Goal.gte(0.8),
)

A scorer can return:

• int or float — numeric score (statistics: avg, min, max, std, count).
• str or bool — categorical score (statistics: frequencies, mode).
• Score(value=..., rationale=..., metadata=...) — rich score with an explanation and extra data.

A run evaluator operates on the full set of results after all records have been processed. Use it for aggregate metrics that can't be expressed per-record — like F1 score, global pass/fail gates, or cross-record analysis.

RunEvaluator(
    name="accuracy_gate",
    scorer=my_run_scorer,
)

RunEvaluator(
    name="accuracy_gate",
    scorer=my_run_scorer,
)

The scorer receives a RunEvaluatorContext with access to all records, their scores, aggregate statistics, and system config. See the run evaluators guide for examples.

Factory for creating goal specifications. See Set goals for the full guide.

from mistralai.evaluations import Goal

Goal.gte(0.8)             # gate: score >= 0.8
Goal.lte(0.1)             # gate: score <= 0.1
Goal.between(0.2, 0.8)    # gate: 0.2 <= score <= 0.8

from mistralai.evaluations import Goal

Goal.gte(0.8)             # gate: score >= 0.8
Goal.lte(0.1)             # gate: score <= 0.1
Goal.between(0.2, 0.8)    # gate: 0.2 <= score <= 0.8

A Goal is only a gate. To declare which way is better, set direction="maximize" / "minimize" on the evaluator (it is also inferred from a gte/lte goal).

A system captures the configuration that drives your task — model name, temperature, system prompt, tool definitions, and so on. Storing these as params (instead of hardcoding them) makes them visible in Studio so you can compare runs across different configs. See Configure system params for details.

from mistralai.evaluations import System

system = System(name="small-t0", params={"model": "mistral-small-latest", "temperature": 0})

from mistralai.evaluations import System

system = System(name="small-t0", params={"model": "mistral-small-latest", "temperature": 0})

When system is provided, it is available via ctx.system in tasks and scorers:

async def task(ctx: TaskContext) -> str:
    response = await client.chat.complete_async(
        model=str(ctx.system.params["model"]),
        temperature=float(ctx.system.params["temperature"]),
        messages=[{"role": "user", "content": ctx.input_record["prompt"]}],
    )
    return str(response.choices[0].message.content)

async def task(ctx: TaskContext) -> str:
    response = await client.chat.complete_async(
        model=str(ctx.system.params["model"]),
        temperature=float(ctx.system.params["temperature"]),
        messages=[{"role": "user", "content": ctx.input_record["prompt"]}],
    )
    return str(response.choices[0].message.content)

See Use context objects for the full context reference.