Home
Resources

[3' read]

View on GitHub ↗Open in Colab ↗

Running Campaigns with LLM Judges using Mistral Observability API

Observability

Objective#

Use campaigns and LLM judges to automatically detect and analyze specific agent behaviors (e.g., excessive apologies) across a timeframe. This workflow demonstrates how to classify agent events at scale using LLM-based criteria, filter datasets, and build evaluation suites.

Prerequisites#

  1. Installation: Install the mistralai package (see below).
  2. API Key: Get yours here.
%pip install mistralai

Workflow#

1. Initialize the Client#

from mistralai.client import Mistral
from getpass import getpass

api_key = getpass("Enter Mistral AI API Key")
mistral = Mistral(api_key=api_key)
print("✅ Client ready")

2. Search for Agent Events#

Note: This example filters by agent ID and timestamp, but many other search parameters are available. To see all available fields, use mistral.beta.observability.chat_completion_events.fields.list().

The search() method supports pagination via page_size and cursor parameters. See the manage_datasets.ipynb cookbook for detailed pagination examples.

agent_id = "ag_00000000000000000000000000000000"  # Your agent ID
time_range = {
    "start": "2026-01-02T03:39:00.000Z",  # ISO 8601 format
    "end":   "2026-04-08T15:39:00.000Z",
}

search_params = {
    "filters": {
        "AND": [
            {"field": "api_agent_id", "op": "eq", "value": agent_id},
            {
                "AND": [
                    {"field": "timestamp", "op": "gte", "value": time_range["start"]},
                    {"field": "timestamp", "op": "lte", "value": time_range["end"]},
                ]
            },
        ]
    }
}

response = mistral.beta.observability.chat_completion_events.search(
    search_params=search_params,
    page_size=25  # Limit to 25 events per page
)

events = response.completion_events.results
print(f"✓ Found {len(events)} event(s)")

3. Create an LLM Judge#

A judge is an evaluator that assesses the quality or characteristics of agent responses. Judges can be one of two types:

  • Classification Judge: Assigns categorical labels to responses (e.g., "apology" / "no-apology", "helpful" / "not-helpful")
  • Regression Judge (also called Scoring Judge): Assigns numerical scores to responses (e.g., a score from 1-5 for quality)

In this example, we'll create a classification judge to detect apologies.

judge = mistral.beta.observability.judges.create(
    name="Apology Detector",
    description="Classifies if the agent apologized (e.g., 'I'm sorry')",
    model_name="mistral-large-latest",
    output={
        "type": "CLASSIFICATION",
        "options": [
            {"value": "no-apology", "description": "No apology detected"},
            {"value": "apology",    "description": "Apology detected"},
        ],
    },
    instructions="""Analyze the assistant's response.
Return 'apology' if it contains explicit apologies (e.g., 'I'm sorry', 'I apologize').
Otherwise, return 'no-apology'.""",
    tools=[],
)
print("✓ Judge created")
print(f"View at: https://console.mistral.ai/observability/judges/{judge.id}")

Note:

  • Use CLASSIFICATION for discrete labels (e.g., "apology" / "no-apology").
  • For numerical ratings (e.g., 1-5), use REGRESSION.

4. Run a Campaign#

A campaign applies a judge to a filtered set of completion events asynchronously. Campaigns are ideal for analyzing agent behavior at scale across thousands of events.

campaign = mistral.beta.observability.campaigns.create(
    name="Apology Detection Campaign",
    description=f"Classify apologies for agent {agent_id} ({time_range['start']} to {time_range['end']})",
    judge_id=judge.id,
    search_params=search_params,
    max_nb_events=10,  # Max events to process, the upper bound is 10000
)
print("✓ Campaign created")
print(f"Monitor at: https://console.mistral.ai/observability/campaigns/{campaign.id}")
print("⚠️ Campaigns run asynchronously. Wait for completion before proceeding.")

Tip:

  • Campaigns may take minutes to hours depending on event volume.
  • Check status via the console link or programmatically (see next step).

5. Check Campaign Status#

Re-run this cell until status == "COMPLETED":

status = mistral.beta.observability.campaigns.fetch_status(
    campaign_id=campaign.id
)
print(f"🔄 Status: {status.status}")

6. Filter Events by Classification#

Once complete, filter events using the judge's output field:
__judge_{judge_id_without_hyphens}.

filtered_params = {
    "filters": {
        "AND": [
            *search_params["filters"]["AND"],  # Original filters
            {
                "field": f"__judge_{judge.id.replace('-', '')}",
                "op": "eq",
                "value": "no-apology",  # Only events with apologies
            },
        ]
    }
}

filtered_response = mistral.beta.observability.chat_completion_events.search(search_params=filtered_params)
filtered_events = filtered_response.completion_events.results

print(f"📊 Found {len(filtered_events)} apology event(s)")

Summary#

This notebook demonstrates how to use campaigns and LLM judges to classify agent events at scale:

  1. Search for agent events within a specific timeframe
  2. Create an LLM judge with classification criteria (e.g., "apology" / "no-apology")
  3. Run a campaign to apply the judge to all matching events asynchronously
  4. Monitor campaign status until completion
  5. Filter events by classification results for further analysis