Technical note · Benchmark integrity
One in eight: Terminal-Bench 2.0 tasks whose own reference solution fails
Before running an agent study on Terminal-Bench 2.0 (TB2), I ran each candidate task's reference solution through Harbor's oracle agent and the task's own tests. 8 of 65 tasks failed 3 out of 3 tries. Without that check, those tasks would have been scored as agent failures.
A task's reward is only as good as its environment on the day it runs. A one-command oracle check before each run catches this cheaply.
What breaks
- For two tasks the verifier can't install its own tools before grading: Debian 11 security packages it lists now return 404.
- One reference downloads a 163 MB dataset that can't arrive within the time budget from the original host.
- The other five break on package rot, a breaking release of an unpinned dependency, a reference slower than its timeout, and live external data.
| Task | What breaks on 2.0 | Patch tested | In TB 2.1 |
|---|---|---|---|
qemu-startup | Verifier's curl install hits removed Debian 11 security packages | Fall back to bullseye main (workaround); better: preinstall verifier tools | Still fails (#233) |
qemu-alpine-ssh | Same (curl, sshpass) | Same | Still fails (#233) |
build-cython-ext | Unpinned planarity 1.0.0 breaks 1 of 18 upstream tests | Pin planarity<1 | Still fails (#240) |
make-doom-for-mips | apt install without apt update; 15 packages 404 | Add apt-get update | Fixed |
build-pmars | Exact pin dpkg-dev=1.22.21 no longer exists | Drop the pins | Fixed |
crack-7z-hash | Default john mode finds the password at 15:35, past the 900 s budget | --wordlist mode: 297 s | Passes (timeout raised to 1800 s) |
caffe-cifar-10 | Dataset download at 35–98 KB/s outlasts the 1200 s budget | Ship the data in the image: passes with 8 s to spare | Not in our 2.1 pool |
protein-assembly | Live RCSB search returns sequences the reference can't translate | None (freezing data changes the task) | Still fails: RCSB renamed modified_monomer to non_standard_monomer; accepting both passes |
TB 2.1 (May 2026) changed 5 of the 8, and 3 of those now pass. A full oracle check on 2.1 (13 October, GCP e2-standard-8) kept 57 of 61 candidates. But Harbor's registry entry terminal-bench@2.0 still serves the old version, so studies pinned to it inherit all 8.
Where a task is graded matters
Two more tasks, torch-pipeline-parallelism and torch-tensor-parallelism, passed on a cloud VM but failed 3 of 3 on a laptop because their tests ran past the verifier timeout. The task code was identical; the machine was not. Whether a task passes can depend on where it is graded.
What this means for agents
An oracle failure proves the reference is broken today. It doesn't prove every agent must fail. But each of the 8 changes what the task measures. In an evaluation, that changes what a score means. In RL training, a task the grader rarely rewards spends rollout compute for little learning signal, and a task that only rewards working around package rot rewards something other than what it describes.
Recommendations
- Run the oracle before every study or training run, on the machine that will grade it, with the published timeouts. Drop and report failures; don't patch tasks silently.
- Use the newest benchmark revision, or say why not.
- Maintainers: bake the verifier's tools into the image, ship data instead of fetching it at solve time, pin transitive dependencies, and re-run the oracle on a schedule: rot happens after release.
- Registries: an oracle-pass date per task would show rot before users spend agent budget on it.
Setup and limits
TB2 at commit 69671fb (what terminal-bench@2.0 serves). Candidates: the 65 tasks with agent timeout ≤ 1800 s, memory ≤ 4 GB and ≤ 2 CPUs, excluding 5 pilot tasks, so the 1-in-8 rate applies to this pool, not to all 89 tasks. Harbor's oracle agent, up to 3 tries per task; image-pull failures retried and not counted. Verdicts from a GCP n2-standard-16 VM on 12 October 2026; each patch confirmed once. No agents were run against the 8 tasks; the agent-facing reading above is an interpretation, not a measurement.
harbor run -d terminal-bench@2.0 -a oracle -i <task> # one task's reference against its own tests
This check is part of a pre-registered study of coding-agent harnesses on Terminal-Bench 2.1 (design frozen 15 October 2026). Logs and patched tasks are available on request.