Evaluate Agents SDK agents

If the system you evaluate is an agent built with the Agents SDK, pass it as an AgentSystem instead of describing a System by hand. The task runs the agent, and the run records which agent was evaluated, subagents included, so Studio can show it.

You need the Evaluation SDK (see Installation) and a Mistral API key. Install the Agents SDK, mistralai-agents 1.1.0 or later:

pip install -U mistralai-agents

pip install -U mistralai-agents

The Agents SDK is not a dependency of mistralai-evaluations. The Evaluation SDK uses an agent from code through its own methods, and imports the Agents SDK only to reach a deployed agent.

Create the agent, then write a task that runs it as ctx.agent:

import asyncio
import os

from mistralai import agents
from mistralai.evaluations import (
    AgentSystem, Evaluation, Evaluator, Mistral, Project, ScorerContext, TaskContext,
)

client = Mistral(api_key=os.environ["MISTRAL_API_KEY"])


@agents.tool
async def lookup_order(order_id: str) -> dict[str, str]:
    """Get the status of an order."""
    return {"order_id": order_id, "status": "shipped on Monday"}  # read from your database


agent = agents.Agent(
    name="support-bot",
    model="mistral-small-latest",
    instructions="You are the support agent of Shopi. Look the order up before answering.",
    tools=[lookup_order],
)

dataset = [
    {"question": "Has order 1042 shipped?", "expected": "shipped"},
    {"question": "What is the status of order 2077?", "expected": "shipped"},
]


async def task(ctx: TaskContext) -> str:
    result = await ctx.agent.run(ctx.input_record["question"])
    return result.text or ""


def scorer(ctx: ScorerContext) -> int:
    return 1 if ctx.input_record["expected"] in str(ctx.output).lower() else 0


evaluators = [
    Evaluator(
        name="accuracy",
        description="1 if the answer contains the expected status, 0 otherwise.",
        scorer=scorer,
    ),
]


async def main():
    run = await client.evaluation.run(
        system=AgentSystem(agent=agent),
        task=task,
        dataset=dataset,
        evaluators=evaluators,
        project=Project(name="Support"),
        evaluation=Evaluation(name="Support bot"),
    )
    run.show(level="run")


asyncio.run(main())

import asyncio
import os

from mistralai import agents
from mistralai.evaluations import (
    AgentSystem, Evaluation, Evaluator, Mistral, Project, ScorerContext, TaskContext,
)

client = Mistral(api_key=os.environ["MISTRAL_API_KEY"])


@agents.tool
async def lookup_order(order_id: str) -> dict[str, str]:
    """Get the status of an order."""
    return {"order_id": order_id, "status": "shipped on Monday"}  # read from your database


agent = agents.Agent(
    name="support-bot",
    model="mistral-small-latest",
    instructions="You are the support agent of Shopi. Look the order up before answering.",
    tools=[lookup_order],
)

dataset = [
    {"question": "Has order 1042 shipped?", "expected": "shipped"},
    {"question": "What is the status of order 2077?", "expected": "shipped"},
]


async def task(ctx: TaskContext) -> str:
    result = await ctx.agent.run(ctx.input_record["question"])
    return result.text or ""


def scorer(ctx: ScorerContext) -> int:
    return 1 if ctx.input_record["expected"] in str(ctx.output).lower() else 0


evaluators = [
    Evaluator(
        name="accuracy",
        description="1 if the answer contains the expected status, 0 otherwise.",
        scorer=scorer,
    ),
]


async def main():
    run = await client.evaluation.run(
        system=AgentSystem(agent=agent),
        task=task,
        dataset=dataset,
        evaluators=evaluators,
        project=Project(name="Support"),
        evaluation=Evaluation(name="Support bot"),
    )
    run.show(level="run")


asyncio.run(main())

ctx.agent is the agent to run. It runs in your process, with its real tools and hooks.

ctx.agent is not the Agent itself. ctx.agent.run() is described in Turns that don't complete. Anything else (name, start_session(), the settings) is the Agent's. ctx.agent.agent is the Agent with the run's overrides applied.

ctx.agent.run() takes what Agent.run() takes and returns the Completed turn.

Any other outcome raises AgentTurnError, with the Agents SDK's turn record as turn. This covers a turn that is Failed, Interrupted, or Blocked on an approval, a turn that is Busy because another turn of the session is active, a turn that is Running because its timeout expired, and an outcome that the Agents SDK doesn't know.

When a turn raises, the record fails and the evaluators don't score it. retry_failed_records() runs the record again.

A turn blocked on an approval fails its record too. To let the agent go on, answer the approvals in the task:

async def task(ctx: TaskContext) -> str:
    result = await ctx.agent.run(
        ctx.input_record["question"],
        on_approval=lambda callback, request: agents.AllowApproval(),
    )
    return result.text or ""

async def task(ctx: TaskContext) -> str:
    result = await ctx.agent.run(
        ctx.input_record["question"],
        on_approval=lambda callback, request: agents.AllowApproval(),
    )
    return result.text or ""

With output_type=, a turn whose output doesn't validate is still Completed. Its output is an agents.InvalidOutput, which the task can check, or return for the evaluators to score.

overrides changes some settings of the agent for this run only. Your Agent is not modified, and nothing is created or changed on a deployed agent.

system = AgentSystem(
    agent=agent,
    overrides={"instructions": "Always look the order up before answering.", "temperature": 0.0},
)

system = AgentSystem(
    agent=agent,
    overrides={"instructions": "Always look the order up before answering.", "temperature": 0.0},
)

You can override any setting of the Agent that takes a plain value, except its name: for example model, instructions, temperature, max_iterations, reasoning_effort, or permission_mode. The Agent checks the values before the run starts, so a setting that a newer Agents SDK adds can be overridden as soon as you install it.

Tools, hooks, skills, and subagents are part of what the agent is. To change them, build another Agent. An agent built with its own completion= config sets its model, temperature, and reasoning effort there, so they can't be overridden.

params records what the run varies besides the agent's settings, such as the environment the agent runs in. The run records it with the overrides as the system's params, so you can compare runs on both:

system = AgentSystem(
    agent=agent,
    overrides={"temperature": 0.0},
    params={"sandbox": "mistral"},
)

system = AgentSystem(
    agent=agent,
    overrides={"temperature": 0.0},
    params={"sandbox": "mistral"},
)

params is only recorded, never applied to the agent. A key goes in overrides or in params, not both. A setting of the Agent goes in overrides.

evaluation.optimize() takes an AgentSystem too. Wrap the settings to search in Tunable. Fixed values stay as they are.

from mistralai.evaluations import GEPA, Tunable

system = AgentSystem(
    agent=agent,
    overrides={
        "instructions": Tunable(agent.instructions),
        "temperature": Tunable(0.7, bounds=(0.0, 1.0)),
        "max_iterations": 20,
    },
)
result = await client.evaluation.optimize(
    system=system, task=task, dataset=dataset, evaluators=evaluators, algo=GEPA(),
)

from mistralai.evaluations import GEPA, Tunable

system = AgentSystem(
    agent=agent,
    overrides={
        "instructions": Tunable(agent.instructions),
        "temperature": Tunable(0.7, bounds=(0.0, 1.0)),
        "max_iterations": 20,
    },
)
result = await client.evaluation.optimize(
    system=system, task=task, dataset=dataset, evaluators=evaluators, algo=GEPA(),
)

Each candidate is a run of the agent with the candidate's values. See Optimize prompts and parameters.

The task must take a ctx: TaskContext (annotated), because the agent is reached as ctx.agent. optimize() refuses an AgentSystem with any other task signature before the optimization starts.

result.winner.system (or result.best_attempt.system) holds the winning overrides. Evaluate the optimized agent again with them, then write the values into your Agent, or redeploy it:

if result.winner:
    best = AgentSystem(agent=agent, overrides=result.winner.system)

if result.winner:
    best = AgentSystem(agent=agent, overrides=result.winner.system)

For a deployed agent, the overrides are session settings. Put model and reasoning_effort into the agent's code before you redeploy it. additional_instructions can stay a session override.

Pass the name or id of a deployed agent to evaluate the deployment itself. Each record runs a session on it, so the task needs no code of the agent.

system = AgentSystem(agent="support-bot", overrides={"model": "mistral-medium-latest"})

system = AgentSystem(agent="support-bot", overrides={"model": "mistral-medium-latest"})

This needs the Agents SDK installed and a run saved in Studio (not local=True). The agent is looked up and run with the client's credentials, on the same server as the evaluation.

A deployed agent is changed per session. The overrides are what the Agents SDK lets a session change: model, reasoning_effort, and additional_instructions, which is appended to the deployed instructions. See SessionAgentCustomization.

ctx.agent.run() passes its options to the deployment like Agent.run() does. With output_type=, the Agents SDK validates the answer against the type, but the deployment keeps its own completion config. Unless its instructions ask for that shape, expect an agents.InvalidOutput.

The evaluation client looks the agent up with its HTTP settings (proxy, TLS). The sessions of the deployed agent go through the Agents SDK's own HTTP client, which only picks up proxies from the environment (HTTPS_PROXY). With mistralai 3, whose client runs on httpx2, the lookup uses the Agents SDK's own HTTP settings too.

The Agents API is reached as the evaluation client. A client with an API key sends it.

A client without an API key, whose HTTP client sets the Authorization header on every request, sends every Agents API call through that HTTP client: the agent lookups and the sessions of a deployed agent. See Credentials without an API key. In a Mistral Workflow, these calls run as the worker's service account on a managed deployment. They run as the user who executes the workflow when it is declared with on_behalf_of=True, so a deployed agent is found in that user's workspace and runs as that user.

With mistralai 3, whose client runs on httpx2, the Agents SDK gets an httpx client with the same credentials, headers, timeout, and TLS settings (CA bundle, client certificate). Like the httpx2 client, it reads its proxy from the environment (HTTPS_PROXY) unless that client was built with trust_env=False. A proxy set on the httpx2 client itself is not carried over.

An agent from code runs in your process. Its model calls, connectors, and MCP servers use the credentials of its own environment (MISTRAL_API_KEY), whatever the evaluation client carries.

The task doesn't have to call ctx.agent.run(...). When your agent runs in another process or a sandbox, read the settings of ctx.agent (the agent with the overrides applied) and pass them to where it runs. The run still records the agent, its variants, and its optimization:

from my_agent import agent  # the same Agent your runner builds


async def task(ctx: TaskContext) -> str:
    return await run_in_sandbox(
        prompt=ctx.input_record["prompt"],
        model=ctx.agent.model,
        instructions=ctx.agent.instructions,
        max_iterations=ctx.agent.max_iterations,
    )


system = AgentSystem(
    agent=agent,
    overrides={"model": "mistral-medium-latest"},
    params={"sandbox": "docker"},
)

from my_agent import agent  # the same Agent your runner builds


async def task(ctx: TaskContext) -> str:
    return await run_in_sandbox(
        prompt=ctx.input_record["prompt"],
        model=ctx.agent.model,
        instructions=ctx.agent.instructions,
        max_iterations=ctx.agent.max_iterations,
    )


system = AgentSystem(
    agent=agent,
    overrides={"model": "mistral-medium-latest"},
    params={"sandbox": "docker"},
)

Forward every setting your runner applies: the run records the config of ctx.agent, not what the runner did with it.

The run's system has kind="agent". It holds:

• the overrides and params as its params: what varies between runs, compared and optimized;
• the agent as evaluated, as its spec, recorded as the Agents SDK describes it.

A System built by hand has kind="custom" and no spec.

For an agent from code, the spec is its config as the Agents SDK serializes it (model and completion settings, instructions, builtins, permissions, subagents), with the overrides applied. The Agents SDK adds the agent's tools, hooks, skills, MCP servers, and connectors only when a session starts and doesn't expose them otherwise, so the spec records them as unknown (null). A subagent is compiled with its own tools and hooks, so those are recorded.

For a deployed agent, the spec is the agent's deployed record, as deployed. The session overrides are the run's params.

Both also keep, in spec.base, the values the overrides replace: one entry per override, or null when it replaces nothing, as additional_instructions does. This tells the overrides apart from params.

With an Agents SDK version that can't serialize an agent, the run still starts. The config of an agent from code is recorded as unknown (null) and a warning is logged.

The run stores the config as is. Keep credentials out of it, as the Agents SDK does: it references them by environment variable or workspace secret, never by value.

When a deployed agent of the workspace has the same name (or the run evaluates a deployed agent), the run is linked to it, so Studio can open the agent from its runs. A link by name is marked as such, because the deployed agent may run other code than the one you evaluated.

If the Agents API can't be reached or rejects the client's credentials, the run does not start.

To retry the failed records of an agent run, pass the same AgentSystem to retry_failed_records(). The retry is refused if the agent's config (subagents included) changed since the run, so the records of one run always come from one agent.

• Settings that the environment sets (the completion gateway URL and key variable) are not compared.
• Upgrading the Agents SDK between the run and the retry can change the serialized config and refuse the retry.
• For a deployed agent, the retry compares the agent id and its deployed settings (model, instructions, permissions), not when it was deployed. Redeploying the same agent doesn't refuse the retry. A redeploy that only changes the agent's code, tools, or skills is not detected, because the deployed record doesn't describe them.

• Configure system params: the System you use for pipelines that are not an Agents SDK agent.
• Optimize prompts and parameters: search for a better agent configuration.
• Retry failed records: rerun the records that failed.
• Evaluate agents in a workflow: run the same evaluation in Mistral Workflows.