Run A Judge-Model Benchmark

Use this pattern for benchmarks whose correctness is determined or assisted by a judge model. Keep the evaluated model and judge model separate when possible.

Command

agentcompass run \
  browsecomp \
  researchharness \
  your-model \
  --env <env-provider> \
  --benchmark-params '{
    "sample_ids": ["<task_id>"],
    "judge_model": {
      "id": "judge-model",
      "base_url": "https://judge-endpoint/v1",
      "api_key": "sk-...",
      "api_protocol": "openai-chat",
      "params": {"temperature": 0}
    }
  }' \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY"

Notes

Topic	Guidance
Credentials	Prefer environment variables or private config for real keys.
Judge stability	Use deterministic judge params unless the benchmark requires otherwise.
Cost control	Start with `sample_ids` before running the full split.
Debugging	Inspect scoring payloads in `details/*.json`.

Deep Research
Model API setup

Run A Service Benchmark Concurrency

⌘I

Get Started

Key Modules

Environments

Cookbooks

Command

Notes

​Command

​Notes

​Related Pages

Command

Notes

Related Pages