DeepSearchQA

DeepSearchQA evaluates agents on multi-domain search questions with single-answer and set-answer scoring.

Runtime Status

Field	Value
Benchmark id	`deepsearchqa`
Tags	`Deep Research`, `Search QA`, `Judge-Scored`
Execution type	local with judge
Typical harness	`researchharness or naive_search_agent`
Typical environment	`host_process`
Current status	registered in the direct runtime

When to Use

Use DeepSearchQA when you need to measure deep research behavior with the task assumptions described by this benchmark. For large or remote benchmarks, prefer benchmark recipes so images, workspaces, and provider-specific defaults come from task metadata instead of manual CLI flags.

Parameters

Common parameters for this benchmark include:

category
answer_type
judge_model
sample_ids

Shared runtime controls such as k, avgk, sample_ids, resume, and category follow the conventions in Benchmark Parameters.

Run Example

agentcompass run \
  deepsearchqa \
  researchharness \
  your-model \
  --env <env-provider> \
  --benchmark-params '{"sample_ids":["<task-id>"]}' \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY" \
  --model-api-protocol openai-chat

Adjust the harness and environment to the supported combination for your branch and deployment.

Outputs

Per-task details are written to results/deepsearchqa/<model>/<run>/details/. Aggregate metrics are written to summary.md in the same run directory.

Notes

Set answer_type to narrow evaluation when comparing retrieval strategies.

Overview

Runtime

Benchmarks

Developer

Runtime Status

When to Use

Parameters

Run Example

Outputs

Notes

​Runtime Status

​When to Use

​Parameters

​Run Example

​Outputs

​Notes

Runtime Status

When to Use

Parameters

Run Example

Outputs

Notes