Use context objects

Tasks and scorers receive their inputs through context objects: typed Pydantic models that bundle all available data into a single parameter. This page is a reference for the three context types.

TaskContext

A task function receives a TaskContext with the input record, system configuration, and run metadata:

from mistralai.observability import TaskContext

async def task(ctx: TaskContext) -> str:
    response = await client.chat.complete_async(
        model=str(ctx.system.params["model"]),
        messages=[{"role": "user", "content": ctx.input_record["prompt"]}],
    )
    return str(response.choices[0].message.content)

from mistralai.observability import TaskContext

async def task(ctx: TaskContext) -> str:
    response = await client.chat.complete_async(
        model=str(ctx.system.params["model"]),
        messages=[{"role": "user", "content": ctx.input_record["prompt"]}],
    )
    return str(response.choices[0].message.content)

Field	Type	Description
`input_record`	`dict[str, Any]`	The current dataset record
`system`	`System \| None`	System config from `evaluation.run(system=...)`
`metadata`	`dict[str, Any] \| None`	Run metadata from `evaluation.run(metadata=...)`

ScorerContext

A scorer function receives a ScorerContext with the input record and the task output:

from mistralai.observability import ScorerContext

def accuracy_scorer(ctx: ScorerContext) -> int:
    return 1 if ctx.input_record["expected"].lower() in str(ctx.output).lower() else 0

from mistralai.observability import ScorerContext

def accuracy_scorer(ctx: ScorerContext) -> int:
    return 1 if ctx.input_record["expected"].lower() in str(ctx.output).lower() else 0

Field	Type	Description
`input_record`	`dict[str, Any]`	The current dataset record
`output`	`Any`	The task output for this generation
`system`	`System \| None`	System config from `evaluation.run(system=...)`
`metadata`	`dict[str, Any] \| None`	Run metadata from `evaluation.run(metadata=...)`

RunEvaluatorContext

Run evaluators receive all records and aggregated statistics after the run completes:

from mistralai.observability import RunEvaluatorContext

def accuracy_gate(ctx: RunEvaluatorContext):
    return ctx.statistics["accuracy"].avg >= 0.8

from mistralai.observability import RunEvaluatorContext

def accuracy_gate(ctx: RunEvaluatorContext):
    return ctx.statistics["accuracy"].avg >= 0.8

Field	Type	Description
`records`	`list[RunEvaluatorRecord]`	All processed records with their scores
`statistics`	`dict[str, EvaluatorStatistics]`	Per-evaluator aggregate statistics
`metadata`	`dict[str, JsonValue]`	Run metadata
`system`	`System \| None`	System config from `evaluation.run(system=...)`

Use get_score(record, "evaluator_name") to access a specific evaluator's score for a given record.