Runtime

Runtime is the center of AgentCompass. It takes CLI flags or Python arguments, normalizes them into a RunRequest, loads registered components, executes tasks with bounded concurrency, and writes durable artifacts. Use this page when you need to understand what happens after agentcompass run starts.

What Runtime Owns

Request construction

Merge config/defaults.yaml, component defaults, CLI overrides, and Python kwargs into RunRequest.

Execution planning

Build an ExecutionPlan for each task and let recipes rewrite it before the environment starts.

Bounded concurrency

Enforce per-run task_concurrency and process-global provider limits.

Artifact persistence

Write run info, task details, summary, progress events, logs, and optional analysis output.

Runtime does not implement benchmark scoring, agent behavior, provider SDK calls, or badcase diagnosis. Those belong to benchmarks, harnesses, environments, and analyzers.

Core Objects

Object	Purpose
`RunRequest`	The full request: benchmark, harness, environment, model, execution controls, output controls.
`ExecutionSpec`	Task concurrency, enabled recipes, retry count, and analysis settings.
`ExecutionPlan`	Per-task plan after benchmark and harness planning, before and after recipe rewrites.
`UnifiedEvaluationRuntime`	The in-process executor used by CLI and Python API.
`ProgressEvent`	Structured event emitted during task loading, setup, execution, summary, and release.

Execution Flow

run_evaluation(...) / agentcompass run
  -> build RunRequest
  -> load built-in registries
  -> benchmark.load_tasks()
  -> benchmark.select_tasks()
  -> load reusable details
  -> run pending tasks with task_concurrency
  -> process and save summary
  -> optionally save analysis summary

For each pending task:

planner builds ExecutionPlan
  -> matching recipes apply
  -> environment.open()
  -> benchmark.prepare_task()
  -> harness.start_session()
  -> harness.run_task()
  -> benchmark.evaluate()
  -> analyzers run if enabled
  -> environment.close()

CLI Usage

agentcompass run \
  screenspot \
  qwen3vl_gui \
  qwen3-vl \
  --env <env-provider> \
  --benchmark-params '{"category":"desktop"}' \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY" \
  --model-api-protocol openai-chat \
  --task-concurrency 8 \
  --results-dir results \
  --data-dir data

The three positional arguments select benchmark, harness, and model. The environment, model endpoint, concurrency, and output directories become fields on the same RunRequest.

Python Usage

from agentcompass import run_evaluation

result = run_evaluation(
    benchmark="screenspot",
    harness="qwen3vl_gui",
    model="qwen3-vl",
    environment="host_process",
    benchmark_params={"category": "desktop"},
    model_base_url="https://your-endpoint/v1",
    model_api_key="sk-...",
    model_api_protocol="openai-chat",
    task_concurrency=8,
    results_dir="results",
    data_dir="data",
)

The return value contains aggregate metadata, metrics, summary fields, output paths, and applied recipes. Per-task details remain on disk so large evaluations do not need to keep every artifact in memory.

Configuration Precedence

Source	Role
`config/defaults.yaml`	Project defaults for runtime, execution, components, providers, and analysis.
Component defaults	`benchmarks.<id>`, `harnesses.<id>`, and `environments.<id>` blocks.
CLI flags / Python kwargs	Highest-priority run-specific overrides.
Recipes	Final provider-aware plan rewrites before environment startup.

Common Patterns

Pattern	How to express it
Run one sample	`--benchmark-params '{"sample_ids":["task-id"]}'`
Run with a provider	`--env <env-provider>`; recipes may infer image and workspace for supported benchmark/provider pairs.
Increase throughput	`--task-concurrency 32`, then confirm model endpoint and provider limits.
Reuse existing results	Use output reuse options from the CLI or config; runtime skips already materialized details.
Post-analyze results	Add `--enable-analysis` during a run or use `agentcompass analysis` later.

Developer Notes

Most extensions do not need to modify runtime. Add or change a benchmark, harness, environment, recipe, or analyzer first. Touch runtime only when you need a new cross-cutting execution concern such as a new progress event, new output persistence behavior, or a new request-level control.

Get Started

Key Modules

Environments

Cookbooks

What Runtime Owns

Request construction

Execution planning

Bounded concurrency

Artifact persistence

Core Objects

Execution Flow

CLI Usage

Python Usage

Configuration Precedence

Common Patterns

Developer Notes

​What Runtime Owns

Request construction

Execution planning

Bounded concurrency

Artifact persistence

​Core Objects

​Execution Flow

​CLI Usage

​Python Usage

​Configuration Precedence

​Common Patterns

​Developer Notes

​Related Pages

What Runtime Owns

Core Objects

Execution Flow

CLI Usage

Python Usage

Configuration Precedence

Common Patterns

Developer Notes

Related Pages