Quality

Verifier Debt: How Tests, Eval Harnesses, LLM Judges and AGENTS.md Files Go Stale, and How to Re-Validate Them

Your checks can keep passing after they stop measuring the right thing. How four kinds of check decay, what four new papers measured, and the one question to add to every release review.

Aamir SiddiquiNovember 2026. 8 min read.
Herbert L. Whittemore with his device for testing the machines that test construction materials, U.S. Bureau of Standards, 1937. Harris & Ewing, Library of Congress. Public domain.
Herbert L. Whittemore with his device for testing the machines that test construction materials, U.S. Bureau of Standards, 1937. Harris & Ewing, Library of Congress. Public domain.

A green check tells you the check ran. It doesn't tell you the check still measures what it measured on the day someone wrote it.

One of the papers below calls this benchmark maintenance debt. I'm going to use a wider name, verifier debt. Every test suite, benchmark grader, LLM judge and agent instruction file you rely on starts drifting the moment it ships, and almost nobody puts time on the calendar to re-check it.

Four papers from the last few weeks measured that drift in four different places. The numbers are not subtle.

1. A red-teamed benchmark still needed 28 of its 89 tasks repaired

Terminal-Bench 2.0 went through extensive human and LM-assisted review and red-teaming before release. Buchanan, Zhou, Linderman and Ré audited it afterward anyway (arXiv:2611.08417). They found issues in roughly 30% of the benchmark and repaired 28 of its 89 tasks, released as Terminal-Bench 2.1.

Re-running 19 agents on the repaired version raised accuracy by over 10 percentage points. Individual scores moved by up to 12 points and leaderboard positions by up to three ranks. The overall order held (Spearman rho 0.958), and that's the line people will quote. The task level is where it gets interesting. Five tasks went from a pass rate of 15% or less to over 50%, after repairs centered on the verifier. Agents had often been producing valid solutions and the tests didn't recognize them.

Some of this was there at launch and got missed. Some only showed up later: models got better and found valid paths the verifier rejects, and the infrastructure under the tasks moved. Even the time budget is part of the measurement. A longer time limit closed a 20-point harness gap on GPT 5.4 and shrank a 30-point gap on GPT 5.4 mini to 5.

Chart 1. Pass rate on each of the 28 repaired tasks, averaged across agents, before and after repair.
Chart 1. Pass rate on each of the 28 repaired tasks, averaged across agents, before and after repair. From Table 7 of arXiv:2611.08417.

2. A harness change flipped 214 of 506 test labels and an accuracy gate passed it

Huo, Wan, Mao and Ginde test a video world model, V-JEPA 2, on 506 paired tests (arXiv:2610.41130). They revised the test harness twice. Headline accuracy moved 0.2 points the first time and 2.6 points the second. A gate that only asks "did accuracy go down?" accepts both.

Underneath, the first revision changed 11 test labels and the second changed 214 of 506. In the second one, 114 tests regressed and 100 improved while accuracy went up. A check that only watches for flipped majority verdicts missed 69 of the 214 changes, including 43 regressions.

Their fix is one I'd steal for any QA org. The harness is versioned as part of the system under test. If the thing doing the measuring changed, the measurement changed.

Chart 2. Two harness revisions on the same model and tests.
Chart 2. Two harness revisions on the same model and tests. Numbers from arXiv:2610.41130.

3. A stale instruction file costs work, and agents never fix it

AGENTS.md and CLAUDE.md files are documentation that agents read on every run. Mehtiyev, Tadesse, Arkoh and Assunção (arXiv:2611.09802) took real stale lines from open-source repos and ran 22 maintenance tasks with each line left stale, corrected, or removed. 732 runs, two models.

Agents followed stale guidance when it was relevant, sometimes even when the right information was elsewhere in the repo. One stale line was a test command pointing at a folder that no longer existed. It collected no tests, and since the agent piped the output through tail or head, the shell reported success. In 19 of Claude Sonnet 5.5's 40 stale runs, the first check of its own fix was a test run that ran no tests. None of its 40 corrected runs did that.

So a command that ran zero tests got reported as a pass and used as proof the fix worked.

The bug still got fixed in nearly every run, so an eval that only checks success sees nothing. Sonnet spent about two extra tool calls per run on that line (7.8 against 5.8, roughly a third more), and the authors call that result exploratory. In the other cases they couldn't detect a difference, and they say so. Across all 732 runs, no agent edited the instruction file.

Chart 3. Claude Sonnet 5.5 on the stale test-command case.
Chart 3. Claude Sonnet 5.5 on the stale test-command case. Numbers from arXiv:2611.09802.

4. An LLM judge is calibrated for one version of the agent

Shahidi and colleagues at Meta, including Nachiappan Nagappan (arXiv:2611.09291), built AutoRefine, which re-tunes a judge prompt from workflow outcomes like landed versus abandoned diffs. On a held-out set with an execution oracle (85 tasks, 13 of them failures), calibration raised chance-corrected agreement, Scott's pi, from 0.36 to 0.60.

The result I care about is on a deployed agent. The recalibrated judge went from 0.16 to 0.51 on 17 held-out diffs from the current workflow. On 16 diffs from older workflows, the previous judge stayed ahead. The judge tuned for today's agent was worse on yesterday's work, and the old one was worse on today's.

Those are small samples. The 95% interval on the 17-diff gain runs from 0.00 to 0.74, so I wouldn't put the exact number on a slide. The direction is the point: a judge is fitted to a moving target, and its calibration has a shelf life.

5. Four checks that rot

CheckHow it decaysWhat was measured
Benchmark verifierRejects valid new solutions as models improve; environments and time budgets drift28 of 89 Terminal-Bench tasks repaired; accuracy up over 10 points after repair
Test harnessA config or scoring change flips labels while the headline number barely moves214 of 506 labels changed for a 2.6-point accuracy move
LLM judgeCalibrated for an earlier version of the agentScott's pi 0.16 to 0.51 after recalibration (17 diffs, small sample)
Agent instruction filePoints at paths and commands that moved; agents work around it instead of fixing it19 of 40 stale runs checked a fix with a test command that ran no tests; no file edited in 732 runs

You don't need a paper to see it either. Mike Taylor at Every wrote about adapting a deck-building skill to a client's template ("AI Could Do Anything. Then It Met PowerPoint."). Claude compared its output to the client's real decks over eight rounds and declared its version better than the originals. Then he opened one. Slides out of place, overlapping text, headshots with the wrong names. Claude admitted it hadn't looked at the presentations. The metrics it wrote checked whether content changes made it into the deck, not how the deck looked. A self-written check, green from the start, measuring the wrong thing.

Diagram. Every layer under a green check is a check too. The stamps are the question to ask, not measured dates.
Diagram. Every layer under a green check is a check too. The stamps are the question to ask, not measured dates.

6. My prediction: by 2028, release reviews ask when a check was last re-validated

In 20 years of QA I've watched teams treat a test suite like a fixed asset. Build it, see it pass, trust it. Nobody puts "re-validate the tests" on a sprint board. At best someone fixes a flaky test once it gets annoying enough.

That habit was survivable when people wrote code slowly and tests aged at the same pace. It breaks when agents write the code, agents write the tests, the harness changes every few weeks and the judge is a prompt someone tuned last quarter.

By 2028 I expect a serious release review to ask a second question next to "did it pass?": when was this check last re-validated, and against what? Teams will version their harnesses, judges and instruction files like code, review changes to them, and regression-test them. A green check nobody has re-validated is a stale measurement, and reviewers will start reading it that way.

What that looks like on a normal week is boring, which is good:

  • Keep the harness, judge prompt and instruction files in version control with the code, and when one changes, diff verdicts per test, not just the total.
  • On a schedule, re-run the reference solution and a few known-bad changes. If the check can't tell them apart anymore, that's debt.
  • Put a "last validated" date next to every pass rate in a release review.
  • Treat a stale AGENTS.md line like a failing test, because for the agent it is one.

The second item is the same check we walk through in How to test code written by Claude Code or Cursor. Step 5 is break the code on purpose, so you can see whether the tests would catch it. Step 7 is re-check the merged result before release.

7. Where OrangePro fits

OrangePro runs after a change is merged. It isn't a PR gate. It challenges the tests you already have with targeted faults (mutation) on code you've already merged, to show which tests would miss a meaningful break today. That's a suite getting re-validated instead of trusted from the day it was written. Where it finds a gap, it produces a reviewable, test-only patch with execution evidence: the run against the original code and the run against the controlled fault.

It doesn't prevent production issues by itself. It gives reviewers evidence they can check.

When did your team last test its tests?

Sources

  1. E. Kelly Buchanan, Crystal Zhou, Scott W. Linderman, Christopher Ré. "Benchmarks as Software: Terminal-Bench and the Case for Continuous Benchmarks." arXiv:2611.08417, 5 Nov 2026. https://arxiv.org/abs/2611.08417
  2. Junyu Huo, Zihao Wan, Ziqi Mao, Gouri Ginde. "One Verdict Is Not Enough: Regression Testing Learned World Models and Their Test Harnesses." arXiv:2610.41130, 30 Oct 2026. https://arxiv.org/abs/2610.41130
  3. Tural Mehtiyev, Biruk Tadesse, Lawrence Arkoh, Wesley Assunção. "When Instruction Files Go Stale: Documentation Debt and Hidden Cost in Coding Agents." arXiv:2611.09802, 6 Nov 2026. https://arxiv.org/abs/2611.09802
  4. Shervin Shahidi, James Poon, et al. (incl. Nachiappan Nagappan). "AutoRefine: Auto-Calibrating Code Judges from Workflow Outcomes." arXiv:2611.09291, 6 Nov 2026. https://arxiv.org/abs/2611.09291
  5. Mike Taylor. "AI Could Do Anything. Then It Met PowerPoint." Every, 29 Jun 2026. https://every.to/also-true-for-humans/ai-could-do-anything-then-it-met-powerpoint

Header photo: "Device for testing testers," Washington, D.C., 1937. Harris & Ewing, Library of Congress (LCCN 2016871741). Public domain, via Wikimedia Commons.

Want to see it on your codebase?

We'll run OrangePro against one of your repos and walk through the report together. No commitment.