Tool Calling

Tool-calling evaluations are useful when the main risk is model I/O discipline: protocol support, structured outputs, tool-call formatting, retries, and whether the harness can turn task context into tool-aware model messages.

Current Runtime Path

Use harnesses that explicitly support tool-aware model protocols:

Workflow	Benchmark	Harness	Environment
Scientific coding with optional code execution	`scicode`	`scicode_tool_use`	`host_process`
Terminal tasks	`terminal_bench_2`, `terminal_bench_2_1`, `terminal_bench_2_verified`	`terminus2`	remote sandbox
Deep research with web/tool actions	`gaia`, `deepsearchqa`, `frontierscience`	`naive_search_agent`	`host_process`

Confirm available benchmark and harness ids with Supported Components before documenting or launching a run.

uv run agentcompass list benchmark

uv run agentcompass list harness

Tool-Use Run

agentcompass run \
  scicode \
  scicode_tool_use \
  your-model \
  --env <env-provider> \
  --benchmark-params '{"sample_ids":["<task-id>"]}' \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY" \
  --model-api-protocol openai-chat \
  --model-params '{"temperature":0}'

What To Check

Signal	Where to inspect
Final answer format	`details/*.json`
Protocol mismatch	run log under `logs/`
Truncation or empty output	`agentcompass analysis` with output-quality analyzers
Aggregate pass rate	`summary.md`

Get Started

Key Modules

Environments

Cookbooks

Current Runtime Path

Tool-Use Run

What To Check

​Current Runtime Path

​Tool-Use Run

​What To Check

​Related Pages

Current Runtime Path

Tool-Use Run

What To Check

Related Pages