The Judge Said It Was Fine: How LLM Judges Passed Forbidden Writes a Simple Rule Check Caught, and Why Agent-Written Tests Lose Their Lead on Bigger Code
An LLM judge can pass an agent that changed data it was not allowed to change. What one tau2-bench study measured on 11 human-confirmed forbidden writes, what a second study found when agent-written tests met larger code, and the one kind of check to keep next to any model verdict.

A customer asks an airline agent to cancel a reservation. The policy does not allow it. The agent cancels it anyway. Native scoring fails the episode because the database is wrong. The conversation ended politely, so the LLM judge passes it.
That is one episode from Sungbin Im's study of tau2-bench (arXiv:2611.14581). A check that grades by reading the story is not a check that grades by trying to break the work.
1. The judge agreed with itself. It did not agree with the facts
Im ran 2,160 episodes: 90 tau2-bench tasks (50 airline, 40 retail), two agent models (gpt-4.1-mini and gpt-4.1), four gate conditions, and three repetitions. Each episode was scored four ways: native scoring against the gold database state, an LLM judge (gpt-4.1-mini, temperature 0.4), the same judge given the policy document, and a boundary evaluator. That last one is a rule check with no model call. It compares two facts: whether a write was expected, and whether one occurred.
When the judge scored the same episode twice, it repeated its verdict 98.2% of the time (95% CI 97.5 to 98.8). When it was compared with native scoring, they agreed on 56.2% of episodes (49.6 to 62.7). Stable, but stably different.
Of the 1,007 episodes native scoring failed, the judge passed 929 (92.3%). The other way around, native passed and the judge failed, was 17 of 2,160 (0.8%). The errors run toward "pass."
Two annotators labeled 120 episodes. Against those human labels, weighted, the judge's false-pass rate on task correctness was 39.9% and its false-fail rate was 1.8%. Adding the policy document changed agreement on policy compliance by -0.001 and on commitment correctness by +0.025, neither significant.
2. On the failure that matters, the judge said it was fine
Human annotators confirmed 11 premature commitments: writes the policy did not allow. That is a small labeled sample, and every number below should be read with that in mind.
In the main run, the judge and the judge given the policy both passed all 11. Native scoring and the rule check caught all 11. A re-run with the full, untruncated transcript barely moved it. The judge passed 10 of the 11. The judge with the policy still passed all 11.
Im then tried stronger models from the same provider, still with untruncated input, on the same 120 labeled episodes (Table 6). A fully correct episode is one the annotators marked yes on all three axes. There are 55 of those.
| Evaluator | Forbidden writes passed (of 11) | Fully correct runs failed (of 55) |
|---|---|---|
| gpt-4.1-mini | 10 | 1 |
| gpt-4.1-mini + policy | 11 | 0 |
| gpt-4.1 | 9 | 4 |
| gpt-4.1 + policy | 10 | 3 |
| gpt-5.6 Sol | 6 | 11 |
| gpt-5.6 Sol + policy | 0 | 40 |
| Rule check (no model) | 0 | 2 |
| Native scoring | 0 | 9 |
A stronger judge trades misses for false alarms. With the policy, gpt-5.6 Sol caught all 11 and failed 40 of 55 correct runs. None of the six judge configurations was significantly above the rule check on task or commitment correctness. All six were significantly below native scoring.
The rule check is not perfect. Its false-pass rate on task correctness was 21.6%, and its overall agreement with the human labels was 0.664, against 0.766 for native scoring. It only checks whether a write was expected and whether one happened, so it will miss an allowed write with the wrong content. What it caught, in this study, was the failure type the judges missed.
Do not take this as "rules beat judges." On this failure type, in this study, a two-line check caught what the judges passed. One benchmark, two domains, simulated customers, models from one provider, a single author, and only 11 forbidden-write cases in the labeled sample.
Forbidden writes were not rare with the gates off: 11.1% of those episodes (60 of 540; 95% CI 6.1% to 16.5%), 19.3% in airline (58 of 300) and 0.8% in retail (2 of 240). Verification, correction, and safe-termination gates did not reduce them significantly. A trace shows the gate's evidence digest dropped the order status and customer authentication.

3. Agent-written tests look strong on small code, then the lead shrinks
A second paper, by Pasternak, Langley, Yotam and Kitanidis at Code Metal (arXiv:2611.21797), measured something adjacent: not whether a judge is right, but whether the tests an agent writes would catch a break.
They took 240 frozen, anonymized repos (Python, C++, and Rust; 30 small under 1k lines, 30 medium at 1k-5k, and 20 large at 5k-10k, per language). Three agents, Claude Code, Cursor and Codex, each with a native frontier model at high reasoning effort, were compared with the projects' human test suites. The yardstick was a fixed sample of automatic planted faults: did any test fail when the code was broken on purpose.
Overall, the agent suites caught a higher share of those planted bugs than the human suites: Claude 0.818, Cursor 0.776, Codex 0.724, human 0.625 (paired gap, agent mean minus human, +0.147 [0.120, 0.174]). Line coverage followed the same pattern: 0.926 / 0.913 / 0.873 against 0.762.
The size split is the part I care about. Averaging the three agents from the paper's Tables 7-9:
- Small: 0.851 against 0.674 (gap 0.177)
- Medium: 0.778 against 0.616 (0.162)
- Large: 0.648 against 0.566 (0.082)
The paper's own paired gaps are 0.176, 0.162 and 0.083. The lead roughly halves from small to large. On large projects, Codex (0.546) sits below the human suites (0.566).

4. The lead is mostly volume, not sharper tests
Each human test killed 8.40 planted bugs, against 0.33 for Claude, 0.48 for Cursor and 1.59 for Codex. The paper also reports that each human test covers 8.5 production lines, against 3.7 for Cursor, 2.2 for Claude Code, and 11.6 for Codex. Per planted bug caught, Claude Code used 7.3 times the input tokens, 6.4 times the output tokens, and 8.5 times the wall-clock time of Codex.
Coverage and test counts flatter the agent suites. The planted-bug numbers show a thinner test, repeated more often, and a lead that shrinks as the code gets bigger.
On C++ to Rust translation (Table 2), human C++ suites sat at 0.559; the transpiled suites at 0.514 (Claude Opus 5), 0.489 (Codex gpt-5.6) and 0.513 (Cursor grok-4.6). In 55% of projects the transpiled suite killed fewer planted bugs, with a worst case of nearly a 70% relative drop.
The planted edits are automatic faults, not production bugs. Projects are capped at 10k lines, and the authors note the corpus is open-source, not enterprise-scale. The gaps on how many tests were written have wide confidence intervals, so I am not quoting those.
5. The files the agents read still go barely reviewed
Yang, He and Zhou looked at 2,870 pull requests that changed AGENTS.md, CLAUDE.md, .cursorrules or SKILL.md files across 2,246 repos (arXiv:2611.18582). Only 792 of 2,870 (27.6%) received a substantive review of the instruction-file change. Against 1,912 human-raised concerns used as the scoring set, three AI reviewers recovered 75 (OpenCode, 3.9%), 62 (Claude, 3.2%) and 25 (Codex, 1.3%).
6. My prediction: by 2028, one non-LLM check per irreversible action
In 20 years of QA I've watched teams treat a passing check as a passing product. When the check is a model reading a transcript, that habit gets more expensive.
By 2028 I expect teams that run agents to keep at least one non-LLM check on every irreversible action: writes, payments, deletes, merges. LLM judges will still be useful as triage. They should not be the verdict. A judge that agrees with itself 98% of the time is consistent, not correct.
What that looks like on a normal week is boring, which is good:
- For any agent that can change data, log whether a write was expected and whether one occurred, and fail the run when those disagree.
- Do not ship on a judge score alone.
- For tests the agents write, measure whether they catch a planted break, not how many tests there are or how much they cover. Re-check that as the code grows.
7. Where OrangePro fits
OrangePro runs after a change is merged. It isn't a PR gate. It challenges the tests you already have with targeted faults (mutation) on code you've already merged, to show which tests would miss a meaningful break today. Where it finds a gap, it produces a reviewable, test-only patch with execution evidence: the run against the original code and the run against the controlled fault.
Pasternak's paper is the case for doing that work. Coverage and test counts make agent suites look strong. The planted-bug numbers show the gap, and the gap is smaller on the small projects than on the large ones.
It doesn't prevent production issues by itself. It gives reviewers evidence they can check.
What's the one check in your agent pipeline that doesn't use a model?
Sources
- Sungbin Im. "When Evaluators of AI Agents Disagree: An Empirical Study of Native Scoring, LLM Judges, and Commit-Boundary Checks on tau2-bench." arXiv:2611.14581, 10 Nov 2026. https://arxiv.org/abs/2611.14581
- Gil Pasternak, AJ Langley, Tom Yotam, Ellie Kitanidis. "Can Coding Agents Write and Transpile Unit Tests?" arXiv:2611.21797, 16 Nov 2026. https://arxiv.org/abs/2611.21797
- Wenhao Yang, Runzhi He, Minghui Zhou. "AI Cannot Review What Humans Catch: A First Look at the Reviews of Agent Instruction Files." arXiv:2611.18582. https://arxiv.org/abs/2611.18582
Header photo: Lewis Hine, "Final Inspector - testing radio frequency alignment and making final test of chassis," RCA Victor, Camden, New Jersey, 1936-1937. U.S. National Archives (518704). Public domain, via Wikimedia Commons.
Want to see it on your codebase?
We'll run OrangePro against one of your repos and walk through the report together. No commitment.