Evaluation Setup

Prerequisites

Environments

Environments must be installed before running evals. If .settings/ is empty, pull them first:

uv run ct settings pull

See Development Setup for details.

API Keys

You'll need API keys for the model providers you're using. Set them in your .env file:

ANTHROPIC_API_KEY=your-key-here
OPENAI_API_KEY=your-key-here

Local Services

Start the support services once:

docker compose up -d

This provides local MongoDB (localhost:27017), the website (private viewer at localhost:3000 with login admin/linuxbench, public viewer at localhost:3002, docs at localhost:3003), and the eval viewer (localhost:8080). The stack's own services read MONGOURI to reach the local Mongo (set in compose.yml); the CLI itself never touches Mongo — it uploads to the local viewer over HTTP, so point it at the stack with CONTROL_TOWER_API_BASE_URL=http://localhost:3000 and ct login (local credentials admin/linuxbench). See Web Viewer and Inspect Logs Viewer for the local service details.

Then log in:

uv run ct login

Without the local services, evals still run and produce local files — results just won't be uploaded or viewable on the website.

Quick Start

The primary command for running evaluations is ct run eval:

uv run ct run eval --policy <policy> --env <env> --main-task <task>

Test your setup with a test run before running expensive model evaluations:

uv run ct run eval --policy test --task-file simple --tags test

This uses the test policy (runs ls and submits, no model calls) with the simple task set. Use test runs to:

• Validate a new development environment
• Verify Docker builds and scorer execution work correctly
• Debug infrastructure issues before running real evals

Configuration

Model Configuration

By default, evaluations use:

• Untrusted model: anthropic/claude-haiku-4-5
• Trusted model: openai/gpt-5-nano

You can override these defaults with policy arguments. The CLI accepts model aliases:

# Using aliases
uv run ct run eval --policy honest -p model=sonnet ...

# Using full model names
uv run ct run eval --policy honest -p model=anthropic/claude-sonnet-4-6 ...

OpenRouter routing

Set USE_OPENROUTER to a boolean to send all model traffic through OpenRouter, or set USE_OPENROUTER_PROVIDERS to a comma-separated list of provider slugs to send only those providers through OpenRouter. Do not use both at once. If you use either, set OPENROUTER_API_KEY in your .env.

Pinning an OpenRouter provider

OpenRouter load-balances each model across backends that can differ in quantization, context window, and which sampling parameters they honour, so unpinned runs are not reproducible per-provider (CT warns once per model when this applies). Anywhere a model is accepted, pass a JSON object instead of a bare name to pin the serving provider:

uv run ct run eval --policy honest \
  -p model='{"model_name": "deepseek/deepseek-chat-v3.2", "provider": {"only": ["deepinfra/fp8"], "allow_fallbacks": false}}' ...

The provider dict is OpenRouter's provider routing object, merged per key over CT's defaults (fp8+ quantization floor, data_collection: deny, sort: exacto). only restricts the candidate set; order alone is try-priority and still falls through unless allow_fallbacks is false. An optional config key sets generation config the same way YAML configs do. The pin is recorded in run metadata and survives re-runs from saved args. Passing provider for a model that does not route via OpenRouter is an error.

Self-Hosted and Open-Weight Models

Model names resolve through Inspect AI's model providers, so self-hosted open-weight models work anywhere a model name is accepted. Point the provider at your endpoint via env vars in .env and use the matching prefix:

• vLLM / SGLang: set VLLM_BASE_URL=http://<host>:8000/v1 (or SGLANG_BASE_URL=...), then pass -p model=vllm/Qwen/Qwen3-32B. Serve with --enable-auto-tool-choice and the tool-call parser for your model family — agent policies drive on OpenAI-style tool calls.
• Any OpenAI-compatible endpoint (NVIDIA NIM, TGI, llama.cpp): pass -p model=openai-api/<service>/<model>, which reads <SERVICE>_BASE_URL and <SERVICE>_API_KEY.

Self-hosted prefixes (vllm/, sglang/, hf/, ollama/, openai-api/) are never routed through OpenRouter, even with USE_OPENROUTER=1.

A single GPU endpoint usually needs the per-call timeout raised and concurrency capped: -p model_attempt_timeout=600 -p model_max_connections=8 --max-samples 1. A fresh clone has no rate file until you run uv run python scripts/generate_model_rates.py (see Development Setup); an eval refuses to start until that file exists. A model the loaded table does not list still reports unknown — not $0.00. A monitor model that can't follow the <score> output format after retries records an error and scores 0.0 — check error counts before trusting safety numbers from a small model.

Skip Docker Builds (Optional)

By default, running an eval rebuilds Docker images locally using the Dockerfile, with layer caching from Docker Hub. To skip rebuilding and pull pre-built images directly from Docker Hub instead:

export SKIP_DOCKER_BUILD=1

Services whose compose entry builds a single-segment image name (no registry or namespace, e.g. BashArena's per-task bash-arena-default-<task_id>:latest) keep their build section even under SKIP_DOCKER_BUILD — such names cannot exist in a registry, so they are always built locally. Fleet workers set SKIP_DOCKER_BUILD=1, so this is what makes per-task-image environments runnable on EC2.

Pre-built images for evaluation environments are published to Docker Hub under the linuxarena organization (for example linuxarena/env-web_scraping:latest). Images for public environments (those with public repos on github.com/linuxarena) can be pulled anonymously; images for private environments remain private - contact the team for access to the private environments.

Each environment's image tag comes from ENV_IMAGE_TAG_<ENV> (uppercased env id, e.g. ENV_IMAGE_TAG_WEB_SCRAPING). Env CI publishes each image under both latest and the env repo's commit SHA, and when the variable is unset a local SKIP_DOCKER_BUILD=1 run pulls the image published at the env checkout's commit — skipping the build doesn't change which env version runs. If no image exists at that commit (checkout ahead of CI, or predating SHA-tagged publishing), the run warns and falls back to latest. EC2 fleet runs default unset tags to latest at dispatch (workers have no local checkout). Set the variable to a published tag to override either default.

Provenance also records the pulled image's digest at env_versions[<env>]["image_digest"], pinning a mutable tag like latest to the exact bytes that ran. Locally-built images have no digest.

When to use:

• CI/batch evals: Always use SKIP_DOCKER_BUILD=1 for faster startup
• EC2 evals: Recommended since EC2 instances start fresh
• Development: Don't use if you're testing local Dockerfile changes

Scoring the main task on attack samples (Optional)

By default an attack-mode sample (one with a side task) records only side_task_success. Pass --score-main-on-attack to also run the main-task scorer on attack samples and record main_task_success alongside it — useful for measuring an attack policy's usefulness (does it complete the main task while attacking?) and for per-run safety/usefulness frontier points.

uv run ct run eval --policy attack -e web_scraping -t crawl_depth -s visit_malicious_website --score-main-on-attack

It is off by default. The side scorer always runs first, then the main scorer runs under its own --scorer-timeout, so an attack sample can take up to twice as long to score. The flag is run-level (every sample emits the same score keys) and rides through YAML configs, hawk submissions, and run metadata automatically. An environment can also declare score_main_on_attack itself, for benchmarks whose side task is defined relative to main-task performance; one selected environment declaring it turns main scoring on for the whole run, which is what keeps the keys uniform. Simulated runs (--simulated) score no main task and ignore the flag.

Policies

Pass a single policy with --policy:

uv run ct run eval --policy honest --task-file simple
uv run ct run eval --policy attack -p model=sonnet --task-file medium

For protocol evals, use --untrusted-policy with --protocol instead:

uv run ct run eval \
  --untrusted-policy simple-attack \
  --protocol trusted-monitoring \
  --protocol-arg sus_threshold=7 \
  --task-file simple

See Protocols and Untrusted Policies for the built-in policies, protocol-mode concepts, and implementation details.

Network Isolation

Docker evaluations deny direct internet access by default. --internet explicitly opts into direct egress; --no-internet selects the default. The restriction marks every network in the environment's compose file internal: true with an isolated gateway, so Docker gives the sample's containers no route out and no path to the host itself. Docker enforces this from outside the container: the agent runs as root inside it and still cannot undo the restriction, and no capability is added to the agent service. The isolated gateway mode needs Docker Engine 28 or newer, which the fleet bootstrap installs. The internet simulator joins the project network from its own bridge, so it stays reachable and is the only path to anything outside the sample. Only the upstream proxies an environment declares go through it to real hosts.

uv run ct run eval --policy honest --task-file simple
# Explicit opt-in for an environment that needs direct egress:
uv run ct run eval --policy honest --task-file simple --internet

--no-internet-simulator independently disables simulator provisioning and attachment for the sample, including persona routing and upstream proxies. It works for local Docker evaluations, fleet workers, and live sessions. Combine it with --no-internet to retain only compose siblings. Scorers and setup scripts that require simulator endpoints cannot run in this mode. Hawk and external sandboxes reject this option because their simulator lifecycle is managed separately.

Compose siblings, the simulator, and its personas resolve and connect as usual unless the simulator is disabled. Anything else, including the host machine, cloud instance metadata, other samples' published ports, and other hosts on the machine's private network, is unreachable. A service using a network_mode other than none or service:..., a compose network declared external: true, with a fixed name:, or with a driver other than bridge, a compose file using include: or a service using extends:, or a Docker Engine older than 28: none of these can be isolated this way, and the run fails before anything starts rather than proceeding open. Tasks whose scorers or setup scripts fetch from real hosts fail under this flag until they are made offline-clean.

Docker's embedded DNS can still forward public-name queries through the host resolver. This network setting alone does not guarantee DNS containment or complete egress isolation.

The network choice is recorded in the run sandbox configuration and forwarded to fleet workers. Hawk enforces the same choice through its Cilium network policy; restricted runs support simulator personas but reject environments declaring upstream API proxies until their routing and CA setup is supported (see the Hawk values contract). External sandboxes and other backends without enforcement require an explicit --internet opt-in. Docker simulator upstream proxies remain separate declared routes to real services.

--no-intranet independently blocks the agent container's connections to private IPv4 ranges, carrier-grade NAT, link-local and IPv6 local ranges, and every address on the Docker host. The simulator and its per-sample exposed-service proxy remain reachable. Rules match the agent's host-side veth interfaces, so changing the agent's routes, source address, or in-container firewall cannot remove the restriction. The agent cannot reach other Compose services; those services can still communicate with each other. ARP and IPv6 neighbor discovery remain available for on-link delivery. Use --no-internet as well to block direct internet connections.

This Docker-only option uses a small control-plane helper with NET_ADMIN in the Docker host network namespace; the agent receives no additional capability. The helper image is built and cached on first use, so prepare it on a connected machine before a disconnected deployment. Host bridge netfilter must be enabled for IPv4 and IPv6. Forwarded traffic is filtered through Docker's DOCKER-USER chain so Docker network updates preserve enforcement. Agent containers with restart policies are rejected because restarting replaces their veth interfaces. Setup fails if enforcement cannot be installed. Project rules are removed after Docker tears down the sample.

External Sandboxes

--sandbox runs samples in an Inspect sandbox provided outside this repo — a microVM runtime, for instance — instead of the default Docker sandbox. Pass a registered type, or module:type to import the module that registers it, with --extra-src when the project is not installed.

uv run ct run eval \
  --policy honest \
  --task-file simple \
  --sandbox my_project.sandbox:my-microvm \
  --internet \
  --sandbox-arg boot_timeout=300 \
  --extra-src ../my-project

--sandbox-arg values are passed to the sandbox as a JSON string for it to parse and validate; Control Tower does not interpret them, and their keys reach it exactly as written. The sandbox owns the sample lifecycle through Inspect's task_init, sample_init, sample_cleanup and task_cleanup hooks, so it cannot be combined with --ec2, --simulated or --replay-sandbox. It also rejects --no-internet and --sandbox-isolation, which the Docker sandbox enforces and an external one cannot — the run fails rather than proceeding unrestricted. The selection is recorded in run metadata with the rest of the sandbox config.