Quickstart: run your first evaluation
This quickstart evaluates a prompt that detects the language of a sentence. You run it on a three-sentence dataset, score each answer, and read the results in your terminal and in Studio.
An evaluation has four parts: a dataset of test cases, a task that produces an output for each case, evaluators that score each output, and the results.
You need Python 3.12 or later, uv, and a Mistral API key. Offline evaluations are available to Enterprise-tier organizations. For other organizations, the Evaluate section doesn't appear in Studio and the evaluations API returns a 403 error.
Install the SDK
Install the SDK
Create a project and add the Evaluation SDK:
uv init first-evaluation && cd first-evaluation
uv add mistralai-evaluations
export MISTRAL_API_KEY="your-api-key"Write the evaluation
Write the evaluation
Create eval.py:
import asyncio
import os
from mistralai.evaluations import (
Evaluation, Evaluator, Goal, Mistral, Project, ScorerContext, System, TaskContext,
)
client = Mistral(api_key=os.environ["MISTRAL_API_KEY"])
# 1. The dataset: each record is a test case with its expected answer.
dataset = [
{"sentence": "Hello, how are you?", "groundtruth": "English"},
{"sentence": "Bonjour, comment ça va?", "groundtruth": "French"},
{"sentence": "Hola, ¿cómo estás?", "groundtruth": "Spanish"},
]
# 2. The task: the code you evaluate. It reads the model and the prompt from the system params.
async def task(ctx: TaskContext) -> str:
response = await client.chat.complete_async(
model=str(ctx.system.params["model"]),
messages=[
{"role": "system", "content": str(ctx.system.params["system_prompt"])},
{"role": "user", "content": ctx.input_record["sentence"]},
],
)
return str(response.choices[0].message.content)
# 3. The scorer: 1 if the answer matches the expected language, 0 otherwise.
def scorer(ctx: ScorerContext) -> int:
return 1 if ctx.input_record["groundtruth"].lower() == str(ctx.output).lower() else 0
# 4. The run: ties the dataset, task, and evaluator together and uploads the results to Studio.
async def main():
run = await client.evaluation.run(
project=Project(name="Language Detection"),
evaluation=Evaluation(name="Accuracy Eval"),
system=System(name="mistral-small", params={
"model": "mistral-small-latest",
"system_prompt": "What language is this sentence in? Reply with ONLY the language name.",
}),
dataset=dataset,
task=task,
evaluators=[
Evaluator(
name="accuracy",
description="1 if the detected language matches the groundtruth, 0 otherwise.",
scorer=scorer,
goal=Goal.gte(0.8),
),
],
)
run.show(level="records")
asyncio.run(main())The System holds the model and the prompt apart from the task code. To compare two prompts or two models, run the same evaluation again with a different System: Studio shows the runs side by side. Goal.gte(0.8) marks the evaluator as passed when the score is at least 0.8.
Run it
Run it
uv run eval.pyYou should see
You should see
The terminal prints the run summary first: the accuracy statistics (average, min, max, standard deviation, and count) and whether the Goal.gte(0.8) goal passes. Then it prints one block per sentence with its input, its accuracy score, and a PASS or FAIL verdict against the goal. Before this output, while the run executes, the SDK logs its progress and, once the run is saved, a link to the run in Studio. To see the model's answer for each sentence, use level="generations".
In Studio, open Evaluate > Evaluations > Offline. The run appears under the Language Detection project and the Accuracy Eval evaluation, with its scores, statistics, and the system params you used. Each new run of the script adds a run to the same evaluation, so you can follow the score over time.
To iterate without uploading anything to Studio, pass local=True to client.evaluation.run(). See Iterate locally.
Going further
Going further