Home
Studio

Quickstart: score your first span with online evaluations

In this quickstart, you create a sentiment judge, attach it to chat spans, send one span, and read back its score.

Before you begin, you need:

  • A user API key for a workspace that meets the prerequisites. Service account keys can't create live evals.
  • curl for steps 1, 2, and 4.
  • Python 3.12 or later for step 3.
export MISTRAL_API_KEY=<your-api-key>
export MISTRAL_BASE_URL=https://api.mistral.ai
1. Create a scorer

1. Create a scorer

curl -s -X POST "$MISTRAL_BASE_URL/v1/observability/scorers" \
  -H "Authorization: Bearer $MISTRAL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Sentiment judge",
    "kind": "llm_judge",
    "config": {
      "model": "mistral-small-latest",
      "prompt_template": "Classify the sentiment of the following conversation as either good or bad.\n\n{{ conversation }}",
      "output_spec": { "type": "categorical", "labels": ["good", "bad"] }
    }
  }'

The response contains the generated scorer slug (for example sentiment-judge-3f9a1c2e) and latest_version.variables, which lists the template variables found in the prompt (here ["conversation"]).

export SCORER_SLUG=<slug from the response>
2. Create a live eval

2. Create a live eval

Select chat spans and map the conversation variable to the formatted transcript of each span:

curl -s -X POST "$MISTRAL_BASE_URL/v1/observability/pipelines" \
  -H "Authorization: Bearer $MISTRAL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Chat sentiment",
    "selectors": [
      { "source_kind": "span", "filter": "gen_ai.operation.name = \"chat\"" }
    ],
    "definitions": [
      { "slug": "'"$SCORER_SLUG"'", "mapping": { "conversation": "transcript" } }
    ]
  }'

Live evals start enabled. The response lists one entry in pipeline_configs per attached scorer. Keep its slug, because that is what identifies the results:

export CONFIG_SLUG=<pipeline_configs[0].slug from the response>
export PIPELINE_ID=<id from the response>
3. Send a span

3. Send a span

This judge reads the conversation, so create the operation with include_content=True to record the messages on the span:

pip install "mistralai-otel-observability[mistral]"
import os

from mistralai.client import Mistral
from mistralai.observability.otel import observability
from mistralai.observability.otel import semconv_constants as genai

client = Mistral(api_key=os.environ["MISTRAL_API_KEY"])
observability.configure_mistral(service_name="my-app")  # reads MISTRAL_API_KEY

messages = [{"role": "user", "content": "I love this!"}]

with observability.operation(
    genai.OPERATION_NAME_CHAT,
    model="mistral-large-latest",
    provider=genai.PROVIDER_NAME_MISTRAL_AI,
    include_content=True,
) as chat:
    response = client.chat.complete(model="mistral-large-latest", messages=messages)
    chat.mistral_completion(response, messages=messages)

observability.shutdown()  # flushes pending telemetry before exit
4. Read the score

4. Read the score

The score is available a few seconds after the span is sent.

curl -s -X POST "$MISTRAL_BASE_URL/v1/observability/spans/evaluations/search" \
  -H "Authorization: Bearer $MISTRAL_API_KEY" \
  -H "Content-Type: application/json" \
  -d "{ \"search_expression\": \"evaluation_name = 'judge:$CONFIG_SLUG'\" }"

The response lists the scores in span_evaluations.results, newest first:

{
  "span_evaluations": {
    "results": [
      {
        "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
        "span_id": "00f067aa0ba902b7",
        "timestamp": "2026-11-03T16:42:11.125403Z",
        "evaluation_name": "judge:sentiment-judge-3f9a1c2e-7b04d1aa",
        "score_label": "good",
        "explanation": "The user expresses clear satisfaction."
      }
    ]
  }
}

The response has more fields than shown here. See The result record.

The same result shows on the live eval's page under Evaluate → Evaluations → Live in AI Studio, and on the span in the trace view.

Clean up

Clean up

curl -s -X DELETE "$MISTRAL_BASE_URL/v1/observability/pipelines/$PIPELINE_ID" \
  -H "Authorization: Bearer $MISTRAL_API_KEY"

Scorers have no delete endpoint. Leaving the scorer in place costs nothing once no live eval uses it.

If no result appears

If no result appears

CheckWhy
The span has gen_ai.input.messages or gen_ai.output.messagesWithout them, transcript is empty and the variable is left unset.
The filter matches the spanFilter keys are raw attribute names. See Live evals.
The live eval and its configs have enabled: truePaused live evals select nothing.
You searched with the pipeline config slugjudge:<scorer-slug> and judge:<live-eval-slug> never match.
Your query window covers the spanSearch defaults to the last hour.
The evaluation trace shows an errorThe judge call or prompt rendering failed. The evaluation judge:<slug> span carries the error.