Testing

Run an agent against a scripted model in your tests: agents.testing.

class ScriptedModel(steps: Sequence[ToolCallStep | ReplyStep])

class ScriptedModel(steps: Sequence[ToolCallStep | ReplyStep])

Answer each model call with the next step of the script, in order.

Pass it to Agent.override, as agent.override(model=model), in a test. A model call past the last step fails with an error that gives the script's length, and a tool call step fails its model call when that call does not offer the tool. It answers only in the test process: agents serve and agents deploy refuse an overridden agent with UnsupportedAgentFeatureError.

For example, in an async test of an agent with a get_weather tool, the model calls the tool from run_typescript, then answers:

model = agents.testing.ScriptedModel([
    agents.testing.run_typescript("return await tools.client.get_weather({city: 'Paris'})"),
    agents.testing.reply("Sunny."),
])
state = await agent.override(model=model).run("Weather in Paris?")
assert state.tool_calls[0].arguments == {"city": "Paris"}
assert state.text == "Sunny."
model.assert_consumed()

model = agents.testing.ScriptedModel([
    agents.testing.run_typescript("return await tools.client.get_weather({city: 'Paris'})"),
    agents.testing.reply("Sunny."),
])
state = await agent.override(model=model).run("Weather in Paris?")
assert state.tool_calls[0].arguments == {"city": "Paris"}
assert state.text == "Sunny."
model.assert_consumed()

def assert_consumed() -> None

def assert_consumed() -> None

Fail if a step of the script was never used.

def testing.reply(text: str) -> ReplyStep

def testing.reply(text: str) -> ReplyStep

Script a model call that answers with text.

def testing.call(name: str, /, **arguments: JsonValue) -> ToolCallStep

def testing.call(name: str, /, **arguments: JsonValue) -> ToolCallStep

Script a model call that asks for the tool name, the name the model sees, with arguments.

The model must be able to call name directly. A tool with the default model_access (an agents.ToolModelAccess) is called from run_typescript: script it with run_typescript.

def testing.run_typescript(code: str) -> ToolCallStep

def testing.run_typescript(code: str) -> ToolCallStep

Script a model call that runs code in run_typescript.

The program calls a tool as tools.<namespace>.<name>(...), as a real model would.