Skip to main content
Harnesses define how a prepared task is attempted. They adapt a model, command-line agent, or external agent framework to AgentCompass’ PreparedTask -> RunResult contract. Harnesses should not own benchmark scoring. They run the agent, collect trajectory and output, and return normalized data for the benchmark and analyzers.

​
What Harnesses Own

Compatibility

Declare whether the harness supports the selected environment and model protocol.

Session setup

Start any local process, remote entrypoint, tmux session, or framework session needed for tasks.

Task execution

Consume PreparedTask and return RunResult with prediction, trajectory, errors, and metadata.

Cleanup

Close sessions without deleting benchmark-owned result artifacts.

​
Core Interface

​
Harness Families

​
Usage: GUI Grounding Harness

Here screenspot prepares image and instruction data. qwen3vl_gui sends the prepared GUI grounding request to the model endpoint. ScreenSpot still owns scoring.

​
Usage: Coding Agent Harness

The benchmark prepares a repository task under /testbed; the Modal recipe selects the task image; mini_swe_agent runs the repair workflow against that workspace.

​
Usage: Terminal Agent Harness

Terminal harnesses depend heavily on environment behavior. Use remote sandboxes when the task image, command execution, or workspace isolation should not consume local resources.

​
What Harnesses Return

Good harness output includes:
  • final prediction or generated files;
  • structured trajectory steps;
  • model usage, latency, and tool-call metadata when available;
  • error fields that can be parsed by analyzers;
  • enough workspace or command context for benchmark evaluation.

​
Developer Notes

Add a harness when an agent framework, CLI, model interaction protocol, or terminal control loop is new. If only the task image or workspace differs, add a recipe or environment config instead.