Scorers

A scorer is an abstract judge definition: a model, a prompt template with variables, and an output type. It says how to judge, but not what to judge or where the variables come from. That separation lets one judge be shared between offline evaluations (scoring dataset records) and online evaluation (scoring production spans), with each context supplying its own inputs. In AI Studio, scorers are listed under Evaluate → Judges.

A scorer never runs by itself. Attach it to a live eval to score the spans you select. The live eval pins the scorer to a version and supplies its variable mapping.

Live evals create judge-v1 pipeline configs, which run an LLM judge. The only scorer kind is llm_judge.

Anatomy

{
  "name": "Helpfulness",
  "description": "Rates how well the assistant answered.",
  "kind": "llm_judge",
  "config": {
    "model": "mistral-small-latest",
    "prompt_template": "You grade customer-support answers.\nScore 1 (useless) to 5 (fully resolves the question).\n\nProduct: {{ product }}",
    "output_spec": { "type": "numeric", "min": 1, "max": 5 },
    "params": { "temperature": 0, "random_seed": 42 }
  }
}

{
  "name": "Helpfulness",
  "description": "Rates how well the assistant answered.",
  "kind": "llm_judge",
  "config": {
    "model": "mistral-small-latest",
    "prompt_template": "You grade customer-support answers.\nScore 1 (useless) to 5 (fully resolves the question).\n\nProduct: {{ product }}",
    "output_spec": { "type": "numeric", "min": 1, "max": 5 },
    "params": { "temperature": 0, "random_seed": 42 }
  }
}

Field	Rules
`name`	1–200 characters, starting with a letter. Turned into the slug `<slugified-name>-<8 hex>`.
`description`	Optional, up to 2,000 characters.
`config.model`	Any chat model the workspace can call.
`config.prompt_template`	Jinja2, up to 20,000 characters. Syntax is checked on write.
`config.output_spec`	See Output types.
`config.params`	Optional: `temperature` (0–1.5), `top_p` (0 exclusive to 1), `max_tokens` (≥ 0), `random_seed` (≥ 0).

What the judge model receives

Each evaluation is one chat completion with a single user message: your rendered prompt_template. The judge sees only what the template renders, so to judge the conversation, include it with {{ transcript }} or a mapping to transcript, input_messages, or output_messages.

The response is constrained with a strict JSON schema derived from the result spec (output_spec). The model returns a score or label plus a reason. The reason becomes the result's explanation.

Output types

`output_spec`	Example	Result fields
`{"type": "categorical", "labels": [...]}`	`["good", "bad"]`	`score_label` is one of the labels. Labels must be unique and non-empty.
`{"type": "numeric", "min": a, "max": b}`	`min: 1, max: 5`	`score_value` within `[min, max]`. `min ≤ max`.
`{"type": "boolean"}`	—	`score_label` is `"true"` or `"false"`.

For dashboards, prefer numeric: avg, p50, and the other aggregations work on score_value directly. For a categorical or boolean scorer, count results grouped by score_label instead (see Aggregate).

Template variables

Every undeclared variable in the template (for example {{ product }}) becomes a scorer variable. The scorer response lists them in latest_version.variables. A live eval decides what each variable is filled with, through its mapping.

Some names are always available in the template without a mapping, and take precedence over a mapped value with the same name:

Name	Value
`transcript`	The formatted input and output messages, one per line as `role: text`.
`attr("key")`	The value of span attribute `key`.
`format_input_transcript("key", …)`	Formats a messages attribute as input messages.
`format_output_transcript("key", …)`	Formats a messages attribute as output messages.

A variable with no mapping and no value on the span is left unset, and the template renders it as empty. Use {% if product %}…{% endif %} to keep the prompt clean when an attribute is optional.

Allowed statements are if, set, with, filter, raw, and autoescape. for, include, import, extends, and macro are not available.

Versions

Scorers are versioned. Version 1 is created with the scorer. PATCH /scorers/{slug} with a config creates a new version only if the config actually changed: configs are compared by a SHA-256 hash of their canonical JSON. Editing only name or description never creates a version.

Warning

Live evals pin a version. When you attach a scorer to a live eval, the live eval copies that scorer version's config into its pipeline config. Later scorer edits are not picked up automatically. To roll out a new version, update the live eval with its definitions again (PUT /pipelines/{id}). The response's pipeline_configs[].scorer_version shows which version each config runs.

Creating a scorer is not idempotent. Posting the same body twice creates two scorers with different slugs. To make a script idempotent, look the scorer up first (GET /scorers?q=<name>), then PATCH it.

Writing a good judge prompt

Say what to judge and what to ignore. "Judge only whether the final assistant message answers the user's last question; ignore tone."
Define each label or score band in the prompt. The schema forces the output shape, not its meaning.
Ask for a short reason. It is the only explanation shown next to the score.
Try it before you deploy it. In AI Studio, the judge editor can run the prompt on sample spans before you attach it to a live eval.