EC2 Fleet Evaluations

See the generated ct run eval reference for the current EC2 and fleet options.

Isolated settings

Fleet jobs automatically use each isolated setting's dependencies and the shipped Control Tower checkout. No custom worker_setup script is needed to activate the setting. The driver resolves an isolated setting's task combinations inside that setting's venv as well (ct run _combinations). A selection expanded with --all, --just-main-tasks, --just-side-tasks or a task file may span isolated and orchestrator-owned environments; -t/-s select within one setting.

Pricing on workers

The controller uploads the driver's rate table beside the code artifact and each worker prices from it, so a fleet run costs what the same run would cost on the driver; CONTROL_TOWER_PRICING=off on the driver turns pricing off on the workers too.

Worker AMI (Speed Optimization)

Fleet workers boot ~3x faster with a pre-baked AMI that includes Docker and pre-pulled Docker images:

uv run ct cloud build-ami  # ~15-20 min, one-time setup

Without AMI: ~230s bootstrap. With AMI: ~65s bootstrap. See EC2 Fleet docs for details.

Fleet Streaming Logs

When you run --ec2, the controller prints only a compact carriage-returned progress line ([N/M] X done, …). To follow per-trajectory results live — from another shell or an LLM — tail logs/fleet/<fleet_id>/events.jsonl. Every line is one JSON object; types include:

  • fleet_start — fleet_id, expected jobs, worker count
  • worker_online — instance_id, bootstrap_s
  • result — job_id, task, model, per-scorer scores, error, duration_s, instance_id. Emitted the instant each sample finishes mid-batch via an Inspect AI hook (worker.py's FleetSampleStreamer) that posts a sample_done SQS message — not at batch-end. So with --concurrency-per-worker 10, you see 10 separate result events as each of the 10 concurrent samples lands, not one burst when the batch completes.
  • job_done — job_id, status, instance_id, duration_s, failure_category, error_summary. Emitted at batch-end from the SQS done queue (the worker's existing flow). Useful for operational debugging.
  • fleet_summary — final counts

The startup banner prints the exact path.

Recipes:

# Watch failures land:
jq -c 'select(.type=="job_done" and .status!="completed")' \
  logs/fleet/<fleet_id>/events.jsonl

# Live per-task scores as each trajectory finishes:
jq -c 'select(.type=="result") | {task, status, scores, duration_s}' \
  logs/fleet/<fleet_id>/events.jsonl

# Running pass-rate across completed trajectories:
jq -s '[.[] | select(.type=="result")] | {n: length, passed: ([.[] | select(.scores.linux_scorer.value.main_task_success=="C")] | length)}' \
  logs/fleet/<fleet_id>/events.jsonl

# Pull a failed .eval for inspection (workers also upload full .eval at batch-end):
aws s3 cp s3://<bucket>/<fleet_id>/<job_id>.eval /tmp/job.eval
uv run inspect view /tmp/job.eval