Blog
Notes on proving AI-written code works
Findings from running OrangePro on real repositories: where tests protect changed behavior, where they do not, and how we check the difference.
- Quality
The Judge Said It Was Fine
An LLM judge can pass an agent run that changed data it was not allowed to change. This post walks through a tau2-bench study where judges passed 10 or 11 of 11 human-confirmed forbidden writes, a no-model rule check caught all 11, and a second study in which agent-written tests lose about half their lead on larger code. It gives QA and engineering leads a simple rule: keep at least one check that does not use a model for irreversible actions.
Aamir Siddiqui. November 2026. 8 min read.
- Quality
Verifier Debt
Your checks can keep passing after they stop measuring the right thing. This post shows how four kinds of check decay, with measured numbers from four new papers, and gives QA and engineering leads a short re-validation routine plus the one question to add to every release review.
Aamir Siddiqui. November 2026. 8 min read.
- Assessment
Where Verification Lands
A frontier model and OrangePro's deterministic code graph examined the same repositories. Their findings barely overlapped. The model found mechanisms such as races and silent fallbacks. The graph found exposure. Here is what each caught and where they disagreed.
Aamir Siddiqui. July 2026. 12 min read.