Iterate locally

Set local=True to run evaluations without uploading results to Studio. This is useful for fast iteration during development.

Usage

run = await client.evaluation.run(
    dataset=dataset,
    task=task,
    evaluators=[Evaluator(name="accuracy", scorer=scorer)],
    local=True,
)
run.show(level="records")

run = await client.evaluation.run(
    dataset=dataset,
    task=task,
    evaluators=[Evaluator(name="accuracy", scorer=scorer)],
    local=True,
)
run.show(level="records")

In local mode:

No data is sent to Studio.
You don't need to specify a project or evaluation.
Results are only available in your terminal or as JSON.

Recommended workflow

Iterate locally: set local=True and adjust your task and scorers until results look right.
Push to Studio: remove local=True and add project and evaluation to track results over time.

# Step 1: iterate locally
run = await client.evaluation.run(
    dataset=dataset,
    task=task,
    evaluators=[Evaluator(name="accuracy", scorer=scorer)],
    local=True,
)

# Step 2: happy with results, push to Studio
run = await client.evaluation.run(
    project=Project(name="My Project"),
    evaluation=Evaluation(name="My Eval"),
    dataset=dataset,
    task=task,
    evaluators=[Evaluator(name="accuracy", scorer=scorer)],
)

# Step 1: iterate locally
run = await client.evaluation.run(
    dataset=dataset,
    task=task,
    evaluators=[Evaluator(name="accuracy", scorer=scorer)],
    local=True,
)

# Step 2: happy with results, push to Studio
run = await client.evaluation.run(
    project=Project(name="My Project"),
    evaluation=Evaluation(name="My Eval"),
    dataset=dataset,
    task=task,
    evaluators=[Evaluator(name="accuracy", scorer=scorer)],
)

JSON export

Export results as JSON for further processing:

run.show(mode="json", level="records")

run.show(mode="json", level="records")