Live evals

A live eval chooses which spans to score and which scorers score them. The API calls it a pipeline (/v1/observability/pipelines). In AI Studio it is under Evaluate → Evaluations → Live.

Anatomy

{
  "name": "Support answers",
  "description": "Helpfulness of the support bot on v2 traffic.",
  "selectors": [
    {
      "source_kind": "span",
      "filter": "gen_ai.operation.name = \"chat\" AND service.name = \"support-bot\""
    }
  ],
  "definitions": [
    {
      "slug": "helpfulness-1a2b3c4d",
      "mapping": { "product": "app.product_name" },
      "sampling": { "type": "probabilistic", "percentage": 20 }
    }
  ],
  "enabled": true
}

{
  "name": "Support answers",
  "description": "Helpfulness of the support bot on v2 traffic.",
  "selectors": [
    {
      "source_kind": "span",
      "filter": "gen_ai.operation.name = \"chat\" AND service.name = \"support-bot\""
    }
  ],
  "definitions": [
    {
      "slug": "helpfulness-1a2b3c4d",
      "mapping": { "product": "app.product_name" },
      "sampling": { "type": "probabilistic", "percentage": 20 }
    }
  ],
  "enabled": true
}

Field	Rules
`name`	1–200 characters, unique in the workspace. Turned into the live eval slug.
`description`	Optional, up to 2,000 characters.
`selectors`	At least one. Only `source_kind: "span"` is accepted.
`definitions`	At least one. Each entry attaches one scorer, by scorer slug.
`enabled`	Defaults to `true` on create.

Selecting traffic

A span is selected when it matches any selector. A selector without filter matches every span.

filter uses the same expression language as the trace explorer search bar and is validated on write. A syntax error returns 422.

Syntax	Example
Comparison	`gen_ai.request.model = "mistral-large-latest"`, `gen_ai.usage.output_tokens > 500`
Boolean logic	`A AND B`, `A OR B`, `NOT A`, parentheses. A space between terms means `AND`.
Sets	`service.name IN ("api", "worker")`, `NOT IN (...)`
Presence	`app.customer_id EXISTS`, `NOT EXISTS`
Text	`LIKE`, `ILIKE`, `CONTAINS`, `REGEXP`
Range	`BETWEEN 1 AND 5`

Keys are raw OpenTelemetry attribute names. The worker merges resource, scope, and span attributes into one map, with span attributes winning on conflict, and evaluates the filter against that map. This has two consequences:

Span fields that are not attributes, such as span name, duration, and status, cannot be used. AI Studio refuses them when you start the live eval.
AI Studio rewrites explorer field names to attribute names before saving, so a filter read back from the API can look different from the one you typed.

Function calls such as has(...) and free-text terms pass validation but never match.

To check a filter before using it, build it in the AI Studio Target traffic step, which previews how many spans match.

Mapping variables

mapping fills each scorer variable from the span being scored. Keys must be variables of the scorer, or the request returns 422. Values are one of:

Source	Resolves to
`transcript`	Formatted input and output messages.
`input_messages`	Formatted `gen_ai.input.messages` only.
`output_messages`	Formatted `gen_ai.output.messages` only.
Any other string	The value of the span attribute with that name.

If the source is missing on a span, the variable is left unset and the scorer still runs. Variables with no mapping behave the same way.

Sampling

Sampling is set per attached scorer, so one live eval can run a cheap judge on all traffic and an expensive one on a slice.

`sampling`	Behaviour
omitted	Every selected span is scored.
`{"type": "probabilistic", "percentage": 10}`	Each span has an independent 10% chance.
`{"type": "attribute_hash", "attribute": "user.id", "percentage": 10}`	Hashes the attribute value: the same user is always in or always out. Use it to follow whole sessions or users. Spans without the attribute are never scored.

percentage must be greater than 0 and at most 100. In AI Studio, the Sampling rate section sets one rate for all judges of the live eval, and Advanced → Stable sample cohort switches to attribute_hash. To set a different rate per scorer, use the API.

Lifecycle

Action	Request	Effect
Pause or resume	`PUT /pipelines/{id}` with `{"enabled": false}` or `true`	Applies to all of its scorers.
Change the filter	`PUT` with `selectors`	Applies to all of its scorers.
Change scorers, mappings, or sampling	`PUT` with the full `definitions` list	Re-copies each scorer's latest version.
Delete	`DELETE /pipelines/{id}`	Hard delete, including its pipeline configs. Past results stay in the telemetry platform.

Workers pick up every change shortly after it is committed. No restart or deploy is needed.

In AI Studio

Evaluate → Evaluations → Live → New evaluation builds the same object:

Target traffic. Build the filter, and preview matching span volume over the last 24 hours, 7 days, or 30 days.
Sampling rate. Choose the share of matching spans to score.
Judges. Add saved judges and map their variables. To check them on a sample span, select Test.
Start. Name and describe the live eval.

The detail page shows trace volume, score trend, score distribution, evaluated coverage, and per-span results.