Modal

Modal is a remote environment provider for task images that can run in cloud sandboxes. It is a good fit for SWE-bench and Terminal-Bench style workloads where the task metadata already identifies a compatible registry image.

Official Setup Links

Need	Link
Create a Modal account	Modal signup
User account setup docs	Modal user account setup
Create automation tokens	Modal service users
Token configuration reference	Modal token config
Token settings page	Workspace token settings

First-Time Setup

Create a Modal account from the official signup page.
Configure Modal credentials for the machine that runs AgentCompass.
Export MODAL_TOKEN_ID and MODAL_TOKEN_SECRET, or let the Modal CLI write credentials to ~/.modal.toml.
Run a single AgentCompass task before increasing concurrency.

For local development, the Modal CLI setup flow is usually enough:

modal setup

For automated runners or shared workspaces, create a service user token in Modal and export it in the shell or CI secret store:

export MODAL_TOKEN_ID="..."
export MODAL_TOKEN_SECRET="..."

AgentCompass authentication precedence is:

Source	Use when
`environments.modal.token_id` and `token_secret`	You intentionally keep credentials in a private `--config` file.
`MODAL_TOKEN_ID` and `MODAL_TOKEN_SECRET`	Recommended for shell sessions, CI, and shared scripts.
`~/.modal.toml`	Convenient for local development after `modal setup` or `modal token set`.

Do not commit Modal tokens to config/defaults.yaml. Use environment variables or a private config file passed with --config.

Verify Credentials

Use the Modal CLI to verify the active credentials:

modal token info

Then run one recipe-backed task with AgentCompass:

SWE-bench Verified

agentcompass run \
  swebench_verified \
  mini_swe_agent \
  your-model \
  --env modal \
  --benchmark-params '{"sample_ids":["astropy__astropy-12907"]}' \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY"

The SWE-bench Modal recipe selects the task image from SWE-bench metadata and sets the workspace root to /testbed. You normally do not need to pass image or memory.

Full Benchmark With Concurrency

agentcompass run \
  swebench_verified \
  mini_swe_agent \
  your-model \
  --env modal \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY" \
  --task-concurrency 32

Use lower concurrency when your model endpoint, Modal account limits, or benchmark image startup time becomes the bottleneck.

Provider Params

Common Modal overrides live under environments.modal or --env-params:

Field	Use
`app_name`	Group AgentCompass sandboxes under a Modal App.
`environment_name`	Route sandboxes to a Modal Environment when your workspace uses environments.
`image`	Use a registry image when no recipe should select the image.
`named_image`	Use a published Modal named image.
`timeout` / `idle_timeout`	Control sandbox lifetime and idle shutdown.
`cpu`, `memory`, `gpu`, `region`	Request resources or placement for heavy tasks.
`env_variables`	Inject non-secret environment variables into the sandbox.

Image Overrides

Use image for a registry image or named_image for a Modal named image only when intentionally overriding recipe behavior.

agentcompass run <benchmark> <harness> <model> --env modal --env-params '{"image":"python:3.13-slim"}'

Get Started

Key Modules

Environments

Cookbooks

Official Setup Links

First-Time Setup

Verify Credentials

SWE-bench Verified

Full Benchmark With Concurrency

Provider Params

Image Overrides

​Official Setup Links

​First-Time Setup

​Verify Credentials

​SWE-bench Verified

​Full Benchmark With Concurrency

​Provider Params

​Image Overrides

​Related Pages

Official Setup Links

First-Time Setup

Verify Credentials

SWE-bench Verified

Full Benchmark With Concurrency

Provider Params

Image Overrides

Related Pages