Evaluation Output

Eval logs are saved to the logs/ directory by default.

A local eval runs through inspect's eval_set over --log-dir (--run-config delegates to inspect's plain eval and never resumes): rerunning an identical command reuses its completed log, resumes a crashed one, and runs only the samples still missing. Identity is the task name, task args, model and limits; for locally built Docker environments, the task args include a fingerprint of the built image contents, so editing an environment starts a fresh eval instead of combining samples produced by different code. External sandboxes and modes that do not build an environment image cannot fingerprint that source, so after changing one of those, or to collect an independent rollout, point --log-dir at a fresh directory or pass a new --run-name. Expansion selections such as --all get a date-stamped default run name, so resuming one across midnight needs an explicit --run-name. A selection may list several task files; each is its own eval in the directory, and a combination listed in several files keeps its first file's sample. A run that completed under --no-fail-on-error is complete, errored samples included; a rerun reuses them rather than retrying them. With upload enabled, a rerun uploads the reused log as a new run.

By default, ct run eval exits non-zero and does not convert or upload the run if Inspect ends with an error or cancellation. Pass --no-fail-on-error only when sample errors should be retained without failing the command.

# Custom output location
uv run ct run eval --log-dir my-logs/ --run-name experiment1

Eval files use Inspect's .eval format and can be viewed with:

# Open in Inspect viewer
inspect view logs/my-eval.eval

# Register an eval log as a run
uv run ct runs make logs/attack_eval/example.eval