Home
Resources

[4' read]

View on GitHub ↗Open in Colab ↗

Dataset Management with Mistral Observability API

Observability

Objective#

Create and manage datasets for evaluation or fine-tuning using the Mistral Observability API. This notebook demonstrates multiple methods for populating datasets with conversation records.

Prerequisites#

  1. Installation: Install the mistralai package (see below).
  2. API Key: Get yours here.
%pip install mistralai

What is a Dataset?#

A dataset is a collection of conversation records used for evaluation or fine-tuning. Each dataset contains:

  • Records: Individual conversation examples
  • Metadata: Dataset name and description

Record Structure#

Each record in a dataset contains:

  • Payload: The conversation data following the Chat Completion API format

    • Messages: A list of chat messages with role and content fields
      • Roles: "user", "assistant", "system"
      • Content: Can be a string (simple text) or a list of chunks (multimodal content)

    Example payload:

    {
  "messages": [
    {"role": "user", "content": "What is the capital of France?"},
    {"role": "assistant", "content": "The capital of France is Paris."}
  ]
}

  • Properties (optional): Custom metadata for the record (e.g. grading guidance, expected output...)

Workflow#

1. Initialize the Client#

from mistralai.client import Mistral
from getpass import getpass

api_key = getpass("Enter Mistral AI API Key")
mistral = Mistral(api_key=api_key)
print("✅ Client ready")

2. Create a Dataset#

Create a dataset with a name and description to organize your data.

dataset = mistral.beta.observability.datasets.create(
    name="Demo Dataset",
    description="A sample dataset created to demonstrate the Mistral Observability API"
)

print(f"Dataset created. View at: https://console.mistral.ai/observability/datasets/{dataset.id}")

3. Add Records Manually#

Best for small datasets or creating individual records with custom properties.

# Create first record
record1 = mistral.beta.observability.datasets.create_record(
    dataset_id=dataset.id,
    payload={
        "messages": [
            {"role": "user", "content": "What is the capital of France?"},
            {"role": "assistant", "content": "The capital of France is Paris."},
        ]
    },
    properties={"source": "demo_script", "priority": "high"},
)

print("✓ Record 1 created successfully!")
# Create second record
record2 = mistral.beta.observability.datasets.create_record(
    dataset_id=dataset.id,
    payload={
        "messages": [
            {"role": "user", "content": "What is 2 + 2?"},
            {"role": "assistant", "content": "2 + 2 equals 4."},
        ]
    },
    properties={"expected_output": "4"},
)

print("✓ Record 2 created successfully!")

4. Import Records from JSONL File#

Import multiple records from a JSONL file where each line is a valid JSON object with a messages field.

Example JSONL:

{"messages": [{"role": "user", "content": "How do I reset my password?"}, {"role": "assistant", "content": "You can reset your password by..."}]}
{"messages": [{"role": "user", "content": "What's the weather like in Paris?"}], "properties": {"grading_guindance": "The agent should perform a web search to find the current weather in Paris."}}

4.1. Upload File#

file = mistral.files.upload(
    file={
        "file_name": "dataset-import-example.jsonl",
        "content": open("data/dataset-import-example.jsonl", "rb"),
    },
    purpose="evaluation",
)
print(f"✓ File uploaded (id = {file.id})")

4.2. Import Records from File#

file_import_task = mistral.beta.observability.datasets.import_from_file(
    dataset_id=dataset.id,
    file_id=file.id,
)
print("✓ File import task created")

4.3. Check Import Status#

Status COMPLETED means the import was successful.

file_task = mistral.beta.observability.datasets.fetch_task(
    dataset_id=dataset.id, task_id=file_import_task.id
)

print(f"File import status: {file_task.status}")

5. Import Records from Campaign#

Add records from an existing campaign to the dataset.

5.1. Create Import Task#

campaign_import_task = mistral.beta.observability.datasets.import_from_campaign(
    dataset_id=dataset.id,
    campaign_id="00000000-0000-0000-0000-000000000000",  # Replace with your campaign ID
)
print("✓ Campaign import task created")

5.2. Check Task Status#

campaign_task = mistral.beta.observability.datasets.fetch_task(
    dataset_id=dataset.id, task_id=campaign_import_task.id
)

print(f"Campaign import status: {campaign_task.status}")

6. Import Records from Explorer#

Add records by specifying completion event IDs from the Explorer.

6.1. Create Import Task#

explorer_import_task = (
    mistral.beta.observability.datasets.import_from_explorer(
        dataset_id=dataset.id,
        completion_event_ids=["00000000-0000-0000-0000-000000000000"],  # Replace with your event IDs
    )
)
print("✓ Explorer import task created")

6.2. Check Task Status#

explorer_task = mistral.beta.observability.datasets.fetch_task(
    dataset_id=dataset.id, task_id=explorer_import_task.id
)

print(f"Explorer import status: {explorer_task.status}")

7. List Records with Pagination#

Retrieve records from the dataset with pagination support.

7.1. Fetch First Page#

page = 1
page_size = 3  # Limit to 3 records per page for demonstration

records_response = mistral.beta.observability.datasets.list_records(
    dataset_id=dataset.id,
    page_size=page_size,
    page=page
)

print(f"✓ Found {len(records_response.records.results)} record(s) on this page")
print(f"Total records: {records_response.records.count}")
print(f"Has more pages: {records_response.records.next is not None}")

7.2. Fetch Next Page (if available)#

if records_response.records.next is not None:
    page = 2  # Increment to next page
    
    # Fetch the next page using the page parameter
    next_page_response = mistral.beta.observability.datasets.list_records(
        dataset_id=dataset.id,
        page_size=page_size,
        page=page
    )
    
    print(f"✓ Page {page}: Found {len(next_page_response.records.results)} record(s)")
    print(f"Has more pages: {next_page_response.records.next is not None}")
else:
    print("No more pages available")

7.3. Fetch All Records#

To retrieve all records, iterate through all pages.

all_records = []
page = 1

while True:
    response = mistral.beta.observability.datasets.list_records(
        dataset_id=dataset.id,
        page_size=25,  # Use larger page size for efficiency
        page=page
    )
    
    all_records.extend(response.records.results)
    print(f"✓ Page {page}: Fetched {len(response.records.results)} record(s)")
    
    if response.records.next is None:
        break
    
    page += 1

print(f"\n✓ Retrieved all {len(all_records)} record(s) from the dataset across {page} page(s)")

Summary#

This notebook demonstrates how to create and populate datasets using the Mistral Observability API:

  1. Create a dataset with a name and description
  2. Add records manually for individual records with custom properties
  3. Import from JSONL file for bulk imports from structured files
  4. Import from campaign to reuse existing campaign data
  5. Import from Explorer to select specific completion events
  6. List records to verify and review dataset contents

Datasets created this way can be used for:

  • Running evaluations (see the evaluation workflow cookbooks)
  • Fine-tuning models
  • Building test suites
  • Organizing conversation examples with metadata