Installation

AgentCompass is developed from a source checkout. Install it in editable mode from the repository root, then configure model access on the Setup page.

Requirements

Python 3.10 or newer (3.12+ recommended).
git for cloning and updating the source checkout.
wget and unzip for automatic dataset preparation.
Optional runtime dependencies for selected environments, such as Docker, Modal, Daytona, or HBox credentials.

Clone Repository

git clone https://github.com/open-compass/AgentCompass.git
cd AgentCompass

Install System Tools

sudo apt-get update && sudo apt-get install -y wget unzip

Install Python Dependencies

Choose one setup path. Replace <compatible_python_version> with a supported Python version, such as 3.12 or 3.13.

pip + venv
uv
conda

python -m venv .venv
source .venv/bin/activate

python -m pip install --upgrade pip
python -m pip install -r requirements.txt
python -m pip install -e .

uv venv --python <compatible_python_version>
source .venv/bin/activate

uv pip install -r requirements.txt
uv pip install -e .

conda create -n <env_name> python=<compatible_python_version>
conda activate <env_name>

python -m pip install --upgrade pip
python -m pip install -r requirements.txt
python -m pip install -e .

For the SWE-bench Verified quick start, install the benchmark and harness extras in the same environment:

uv pip install -r requirements/swe.txt
uv pip install -r requirements/mini-swe-agent.txt

The Docker quick start also requires Docker to be installed and running:

docker version

Verify Installation

If the environment is activated:

agentcompass --version

agentcompass --help

agentcompass list benchmark

If you are using uv and prefer not to activate .venv:

uv run agentcompass --version

uv run agentcompass --help

uv run agentcompass list benchmark

Update From Source

Pull the latest source with rebase to keep local commits on top of upstream changes, then reinstall editable dependencies when requirements or package metadata change. Commit or stash local edits before rebasing.

pip + venv
uv
conda

git pull --rebase
source .venv/bin/activate

python -m pip install -r requirements.txt
python -m pip install -e .

git pull --rebase

uv pip install -r requirements.txt
uv pip install -e .

git pull --rebase
conda activate <env_name>

python -m pip install -r requirements.txt
python -m pip install -e .

Data, Cache, And Output Directories

AgentCompass uses predictable local directories by default:

Path	Purpose
`data/`	Downloaded or prepared benchmark datasets. Existing datasets are reused as the local dataset cache.
`results/`	Evaluation outputs under `results/<benchmark>/<model>/<timestamp>/`.
`logs/` inside a run directory	Runtime logs for debugging a specific evaluation.

Override these locations when needed:

agentcompass run <benchmark> <harness> <model> --data-dir <data_dir> --results-dir <results_dir>

Troubleshooting

Symptom	Check
`agentcompass: command not found`	Activate the environment, reinstall with `python -m pip install -e .`, or use `uv run agentcompass ...`.
Python version mismatch	Run `python --version` and recreate the environment with Python `>=3.10`.
Dataset preparation fails	Confirm `wget`, `unzip`, network access, and write permission for `data/` or `--data-dir`.
Remote environment startup fails	Check provider credentials and environment-specific setup for Docker, Modal, Daytona, or HBox.
Model provider errors during a run	Configure model endpoint, key, protocol, and provider naming on the Setup page.

Run Without Editable Install

Editable install is preferred. If you intentionally avoid it, run through the module path:

PYTHONPATH=src python -m agentcompass.cli run \
  swebench_verified \
  mini_swe_agent \
  glm-5.2 \
  --env docker \
  --benchmark-params '{"sample_ids":["astropy__astropy-12907"]}' \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY" \
  --model-api-protocol openai-chat

Get Started

Key Modules

Environments

Cookbooks

Requirements

Clone Repository

Install System Tools

Install Python Dependencies

Verify Installation

Update From Source

Data, Cache, And Output Directories

Troubleshooting

Run Without Editable Install

​Requirements

​Clone Repository

​Install System Tools

​Install Python Dependencies

​Verify Installation

​Update From Source

​Data, Cache, And Output Directories

​Troubleshooting

​Run Without Editable Install

Requirements

Clone Repository

Install System Tools

Install Python Dependencies

Verify Installation

Update From Source

Data, Cache, And Output Directories

Troubleshooting

Run Without Editable Install