Home
Studio

Optimize prompts and parameters

Optimization turns an evaluation into a search: instead of measuring a single prompt or set of system params, the SDK automatically explores variations of it and returns the best one it finds — scored against the same evaluators you already use.

You keep your dataset, task, and evaluators. You mark which parameters are free to change, pick a search strategy, and call client.evaluation.optimize().

Dataset  →  Task  →  Evaluators  →  Optimizer  →  Best parameters
i
Information

Optimization requires mistralai-evaluations 0.6.0+. Like the rest of offline evaluations, it is available to Enterprise-tier organizations only.

Mental model

Mental model

  • Search space — the parameters you let the optimizer change (a system prompt, a temperature, a threshold), declared with Tunable slots inside a TunableSystem.
  • Score — what an evaluator returns for a candidate: one number per record (plus its run-level average), exactly like in evaluation.run().
  • Objective — the single, higher-is-better number the optimizer maximizes. It is derived from your evaluators' scores: each evaluator's direction sets which way is better, and the scores are combined into one objective. A Goal threshold additionally acts as a gate — a candidate that misses it can't win.
  • Candidate — one concrete set of system params the optimizer tries (the Tunable slots filled with specific values). Each candidate is a real evaluation run, visible in Studio.
  • Optimizer — the search strategy that proposes new candidates from the results of past ones. Two are built in: SimpleOptimizer and GEPA.
Declaring a search space

Declaring a search space

A TunableSystem has the same shape as the System you pass to evaluation.run(), but any parameter can be wrapped in Tunable(...) to mark it as optimizable. Plain values stay fixed.

from mistralai.evaluations import Tunable, TunableSystem

system = TunableSystem(
    name="candidate",
    params={
        "instruction": Tunable("Summarize the text."),  # optimized
        "model": "mistral-small-latest",                # fixed
        "temperature": Tunable(0.7, bounds=(0.0, 1.0)), # optimized, clamped to [0, 1]
    },
)
  • The seed value (the argument to Tunable) is where the search starts — it becomes the baseline (generation 0).
  • bounds=(low, high) constrains numeric slots; proposals are clamped to the range.
  • Your task reads every slot from ctx.system.params, exactly as with evaluation.run(). During the search the SDK materializes each candidate's values before calling your task, so ctx.system.params["instruction"] always holds the candidate being evaluated.

At least one Tunable slot is required — otherwise there is nothing to search over.

A first optimization

A first optimization

This example optimizes a summarization instruction against two evaluators that pull in opposite directions — coverage (keep the key facts) and conciseness (compress hard). The optimizer has to evolve an instruction that balances both.

import asyncio
import os
from typing import TypedDict

from mistralai.evaluations import (
    GEPA,
    Evaluation,
    Evaluator,
    Mistral,
    Project,
    Score,
    ScorerContext,
    TaskContext,
    Tunable,
    TunableSystem,
)

client = Mistral(api_key=os.environ["MISTRAL_API_KEY"])


class Item(TypedDict):
    text: str
    facts: list[str]  # key facts a faithful summary must preserve


dataset: list[Item] = [
    {"text": "The Eiffel Tower was designed by Gustave Eiffel and completed in 1889 in Paris. "
             "Standing 330 metres tall, it was the world's tallest structure for 41 years.",
     "facts": ["Gustave Eiffel", "1889", "330", "Paris"]},
    # ... more records
]


async def task(ctx: TaskContext) -> str:
    response = await client.chat.complete_async(
        model=str(ctx.system.params["model"]),
        messages=[
            {"role": "system", "content": str(ctx.system.params["instruction"])},
            {"role": "user", "content": ctx.input_record["text"]},
        ],
    )
    return str(response.choices[0].message.content)


def coverage(ctx: ScorerContext) -> Score:
    facts = ctx.input_record["facts"]
    raw = str(ctx.output).lower()
    hits = [f for f in facts if f.lower() in raw]
    value = len(hits) / len(facts) if facts else 1.0
    return Score(value=value, rationale=f"{len(hits)}/{len(facts)} facts kept")


def conciseness(ctx: ScorerContext) -> Score:
    source_words = len(str(ctx.input_record["text"]).split())
    summary_words = len(str(ctx.output).split())
    ratio = summary_words / source_words if source_words else 1.0
    value = max(0.0, min(1.0, (0.6 - ratio) / 0.4))  # full marks at <=20% of source
    return Score(value=value, rationale=f"{summary_words}/{source_words} words")


async def main():
    result = await client.evaluation.optimize(
        project=Project(name="Summarization"),
        evaluation=Evaluation(name="Summary Prompt Optimization"),
        dataset=dataset,
        task=task,
        evaluators=[
            # no goal, no explicit direction → direction defaults to "maximize"
            Evaluator(name="coverage", scorer=coverage),
            Evaluator(name="conciseness", scorer=conciseness),
        ],
        system=TunableSystem(
            name="candidate",
            params={
                "instruction": Tunable("Summarize the text."),  # optimized
                "model": "mistral-small-latest",                # fixed
            },
        ),
        algo=GEPA(iterations=8, pareto_size=3, minibatch_size=5, holdout=0.2),
        steer="Summaries drop the key numbers from the source — keep them.",
        tags=["optimization"],
    )

    result.show()
    if result.winner is not None:
        print("Best instruction:", result.winner.system["instruction"])


asyncio.run(main())

Every candidate is a normal evaluation run: filter them in Studio by the optimization:* tag the SDK adds, and use Compare to inspect the trajectory.

Steering the optimizer

Steering the optimizer

Evaluators tell the optimizer how it's scored; steer tells it what problem you're trying to fix — the reason you launched the run. It's free text, and the default reflective mutator reads it when diagnosing failures, so the rewrites target your intent instead of only whatever the scores happen to surface:

steer="Reduce hallucinations when the retrieved context doesn't contain the answer."

steer never changes the objective or the gates (those stay derived from your evaluators' goals); it only orients the default mutator's diagnosis. It is also recorded on the optimization and shown in Studio. It is capped at 4,000 characters — it's a goal statement, not a place to paste a whole document.

Choosing an optimizer

Choosing an optimizer

Both optimizers work the same way from the outside — you pass one as algo= and read the same OptimizeResult back. They differ in how they search.

SimpleOptimizer (greedy hill-climb)

SimpleOptimizer (greedy hill-climb)

Starts from the seed and, each round, reflects on the current best's lowest-scoring records and proposes a rewrite. It keeps the rewrite only if it beats the current best on the whole dataset. The trajectory reads as a plain climb: gen 0, gen 1, gen 2…

from mistralai.evaluations import SimpleOptimizer

algo = SimpleOptimizer(
    iterations=5,   # rewrites to try
    patience=3,     # stop after this many consecutive non-improvements
    reflection_model="mistral-small-latest",
    mutation_model="mistral-large-latest",  # defaults to reflection_model
)

Use it when you want a simple, legible baseline and have a small dataset. Because it validates on the same set it optimizes, it is greedier and can overfit or stall in a local optimum.

GEPA (Pareto-based reflective search)

GEPA (Pareto-based reflective search)

A more robust, multi-objective optimizer. It splits your dataset into three roles and keeps a Pareto frontier — an archive of candidates that are best on different records — instead of a single champion.

from mistralai.evaluations import GEPA

algo = GEPA(
    iterations=8,        # mutation attempts (search budget)
    pareto_size=3,       # fixed validation set used to accept/reject candidates
    minibatch_size=5,    # fresh examples drawn each iteration for reflection
    holdout=0.2,         # fraction reserved for the final baseline-vs-winner number
    patience=3,          # stop after this many consecutive rejected children
    random_seed=42,      # reproducible split / sampling / selection
    reflection_model="mistral-small-latest",
    mutation_model="mistral-large-latest",
)

The three splits:

  • D_pareto — a fixed validation set, the common yardstick every candidate is scored on.
  • D_feedback — a pool from which a fresh minibatch is drawn each iteration; the optimizer reflects on failures here to propose the next candidate.
  • Holdout — an optional never-seen set used only at the end, to report an honest, selection-bias-free improvement of the winner over the baseline. Set holdout=0 to disable (useful on small datasets where there is no room to spare).

Use GEPA for multi-objective problems, larger datasets, or whenever you want the holdout to guard against overfitting.

:::tip Start small, then scale Begin with SimpleOptimizer and a handful of iterations to confirm your task and evaluators behave, then switch to GEPA for the real run. A cheap reflection_model with a stronger mutation_model is a good default: cheap diagnosis, bold rewrites. :::

Direction, weights, and gates

Direction, weights, and gates

Two things steer the optimizer, and they live in different places:

  • Direction is metric semantics on the Evaluator (direction="maximize" / "minimize"). Left unset it's inferred from the goal (gte → maximize, lte → minimize). weight and min_value/max_value (normalization window) also live here — a weight=0 evaluator is a pure constraint, excluded from the objective.
  • Threshold — an evaluator's Goal (Goal.gte(0.6), Goal.lte(0.1), Goal.between(...)) acts as a gate: a candidate that violates any threshold cannot win on aggregate score alone. Gate-passers are always ranked ahead of gate-violators.
evaluators=[
    Evaluator(name="coverage", scorer=coverage, goal=Goal.gte(0.6)),       # must keep facts
    Evaluator(name="conciseness", scorer=conciseness, goal=Goal.gte(0.4)), # must compress
]

This is how you encode "don't sacrifice factuality to win on brevity." A hard floor is a constraint, not a bigger vote — that's why it's a goal, not a weight. A weighted mean can be gamed (an empty "." summary can score high on conciseness while failing accuracy), but a gate ranks gate-passers ahead of gate-violators before the mean is compared.

:::caution When no candidate clears the gates A candidate only wins (verdict == "success") if it clears every gate. If none does, there is no winner: the result is best_attempt (a candidate outscored the baseline but still fails a gate — surfaced as result.best_attempt) or no_change. A failing threshold never masquerades as a win. :::

Reading the result

Reading the result

optimize() returns an OptimizeResult:

result = await client.evaluation.optimize(...)

result.show()          # prints the winner (or best attempt) + full trajectory to the terminal

result.verdict         # "success" | "best_attempt" | "no_change"
result.summary         # one-line human summary, e.g. "0.50 → 0.72  (+0.22, +44%, success)"

# On "success", the winner (else None); on "best_attempt", the closest try that beat the baseline
best = result.winner or result.best_attempt
if best is not None:
    best.system        # {"instruction": "...", "model": "..."} — the full config
    best.score         # aggregate objective
    best.gain          # absolute delta vs baseline (+ best.gain_pct)
    best.scores        # per-evaluator breakdown: value, distribution, goal
    best.run_url       # link to the run in Studio

result.baseline.score  # the seed's score (baseline/winner scored on the same held-out set)

# Every candidate explored, in generation order
for c in result.trajectory:
    print(c.gen, c.score, c.gate, c.changed)

Because the baseline, winner, and best attempt are scored on the same held-out set, the reported gain is honest — not an artifact of the search picking a lucky configuration.

Rate limits and retries

Rate limits and retries

Optimizations fan out many model calls — one candidate is a full evaluation run, and each generation adds the reflective mutator's diagnose/rewrite calls — so under load you may hit 429 Too Many Requests. Lowering max_concurrency helps but doesn't eliminate them: rate limits are account-wide.

Your task and scorers call the model through your Mistral client, so that's where to make them resilient — the SDK does not silently retry on your behalf. Configure retries once on the client, and every completion your task/scorer makes will back off and retry transient errors (429, 500, 502, 503, 504):

from mistralai.client.utils import BackoffStrategy, RetryConfig
from mistralai.evaluations import Mistral

client = Mistral(
    api_key="…",
    retry_config=RetryConfig(
        strategy="backoff",
        backoff=BackoffStrategy(
            initial_interval=1_000,   # 1s, then 2s, 4s… (jitter added by the SDK)
            max_interval=30_000,      # cap each wait at 30s
            exponent=2.0,
            max_elapsed_time=60_000,  # give up after ~60s so a call fails cleanly instead of hanging
        ),
        retry_connection_errors=False,
    ),
)

The SDK already retries its own internal model calls (the optimizer's reflective mutator), so a transient 429 there won't abort the whole optimization. You only need to handle your own task and scorer calls.

Next steps

Next steps