Models

Model configuration in AgentCompass is run-local. A model spec tells the selected harness which model id to call, where the endpoint is, which protocol to use, and which generation parameters to pass. AgentCompass does not require a global model registry for normal runs.

Model Spec Fields

Field	Meaning
`id`	Model name sent to the provider and used in result paths.
`base_url`	API endpoint base URL.
`api_key`	API key or token for the endpoint.
`api_protocol`	Protocol adapter, such as `openai-chat`, `openai-responses`, or `anthropic`.
`params`	Sampling and provider-specific request options such as `temperature` or `max_tokens`.

The CLI maps these fields from --model-base-url, --model-api-key, --model-api-protocol, and --model-params.

Basic Usage

export MODEL_BASE_URL="https://your-endpoint/v1"
export MODEL_API_KEY="sk-..."

agentcompass run \
  screenspot \
  qwen3vl_gui \
  qwen3-vl \
  --env <env-provider> \
  --benchmark-params '{"category":"desktop"}' \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY" \
  --model-api-protocol openai-chat \
  --model-params '{"temperature":0}'

Use shell variables for secrets. Shell variable names must use underscores, for example MODEL_BASE_URL, not model-base-url.

Protocol Selection

Protocol	Typical use
`openai-chat`	OpenAI-compatible chat completions style endpoints.
`openai-responses`	OpenAI Responses API style endpoints.
`anthropic`	Anthropic Messages API style endpoints.
JSON list	Ordered fallback list when a harness can choose among supported protocols.

If api_protocol is empty, the harness may choose its own default. For public docs and reproducible commands, prefer passing the protocol explicitly.

Python Usage

from agentcompass import run_evaluation

run_evaluation(
    benchmark="screenspot",
    harness="qwen3vl_gui",
    model="qwen3-vl",
    environment="host_process",
    benchmark_params={"category": "desktop"},
    model_base_url="https://your-endpoint/v1",
    model_api_key="sk-...",
    model_api_protocol="openai-chat",
    model_params={"temperature": 0},
)

Judge Models

Some benchmarks and analyzers need a second model for judging or qualitative analysis. They use the same conceptual fields: model id, base URL, API key, protocol, and params. Examples:

benchmark judge models under benchmark_params.judge_model;
qualitative analyzer models under analysis_params.QualitativeAnalyzer;
comparison analyzers under analysis_params.MultiQualitativeAnalyzer.

Common Failures

Symptom	Likely cause
Provider not detected	The model id needs a provider prefix for the underlying client, or the harness protocol is wrong.
401 or authentication errors	`--model-api-key` is missing or points to the wrong endpoint.
404 or model not found	The model id does not exist on `--model-base-url`.
Unsupported protocol	`--model-api-protocol` is not supported by the selected harness.

Get Started

Key Modules

Environments

Cookbooks

Model Spec Fields

Basic Usage

Protocol Selection

Python Usage

Judge Models

Common Failures

​Model Spec Fields

​Basic Usage

​Protocol Selection

​Python Usage

​Judge Models

​Common Failures

​Related Pages

Model Spec Fields

Basic Usage

Protocol Selection

Python Usage

Judge Models

Common Failures

Related Pages