Agentic Coding

Agentic coding benchmarks combine repository state, issue text, generated patches or output files, and benchmark-specific scoring. AgentCompass keeps the benchmark, harness, model endpoint, and environment separate so the same task set can be tested with different agent implementations.

Recommended Stack

Need	Benchmark	Harness	Environment
SWE-bench Verified	`swebench_verified`	`mini_swe_agent`, `codex`, `claude_code`, or `openhands`	`modal`, `daytona`, `docker`, or cluster provider
SWE-bench Multilingual / Pro	`swebench_multilingual`, `swebench_pro`	coding harness	recipe-backed provider
Scientific coding	`scicode`	`scicode_tool_use`	local or remote

SWE-bench With Recipe Inference

export MODAL_TOKEN_ID="..."
export MODAL_TOKEN_SECRET="..."

agentcompass run \
  swebench_verified \
  mini_swe_agent \
  your-model \
  --env <env-provider> \
  --benchmark-params '{"sample_ids":["astropy__astropy-12907"]}' \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY"

For supported benchmark/provider combinations, the recipe infers the image and workspace root from task metadata. For SWE-bench Verified, the remote workspace is usually /testbed.

Harness Choice

Harness	Best Use when
`mini_swe_agent`	You want a compact SWE-style repair agent.
`codex`	You want to evaluate the OpenAI Codex CLI as the agent loop.
`claude_code`	You want to evaluate Claude Code in a prepared workspace.
`openhands`	You want to evaluate OpenHands on repository repair tasks.
`scicode_tool_use`	You are running SciCode step-by-step code generation.

Get Started

Key Modules

Environments

Cookbooks

Recommended Stack

SWE-bench With Recipe Inference

Harness Choice

​Recommended Stack

​SWE-bench With Recipe Inference

​Harness Choice

​Related Pages

Recommended Stack

SWE-bench With Recipe Inference

Harness Choice

Related Pages