Guide
How to validate AI-written code before you release it
Updated November 17, 2026
AI coding agents made writing code cheap. Knowing that the code is safe to ship did not get cheaper. A green build tells you the tests you have passed. It does not tell you whether those tests would notice if the new code were wrong, and with AI-written code the tests are often written by the same agent, in the same session, from the same understanding of the task.
This guide sets out a layered method that works with any coding agent and any toolchain. Each layer answers a different question. No single tool covers all of them, so the point is to know which question each check answers and where your gaps are.
Why AI-written code needs a different validation habit
The usual signals of a careful change are weaker when an agent writes the code:
- Volume. Agents produce larger and more frequent pull requests. Reviewers skim more and read less.
- Plausibility. Generated code reads well. Mistakes tend to be subtle: an off-by-one boundary, a swapped condition, an error path that returns success, a retry that never stops.
- Self-grading. When the agent writes both the code and the tests, the tests encode the agent's assumptions. If the assumption is wrong, the code and the test agree with each other and the build is green.
- Coverage looks fine. A generated test that calls a function and checks that "something was returned" executes every line. Line coverage goes up; protection does not.
The fix is not to distrust everything. It is to add checks whose answer does not depend on the agent's own judgment.
The six layers of validation
1. Write down the intent before the code
An agent cannot be wrong about a requirement nobody stated. For anything beyond a small fix, keep a short spec in the repository: what the change must do, what it must never do, and the edge cases that matter (empty input, limits, retries, permissions, money, deletion). Anthropic's own Claude Code guidance recommends a written definition of done and giving the agent "a check to run." A one-paragraph spec is also what makes later review and tests meaningful.
2. Run deterministic static checks on every change
Type checking, linting, formatting, dependency and secret scanning, and static analysis catch a large class of mistakes with no judgment involved. Rules-based tools such as SonarQube, Semgrep or CodeQL give the same answer every time, which makes them a good counterweight to model-based tools. Several now offer settings aimed specifically at AI-generated code, for example SonarQube's AI Code Assurance quality gate.
3. Review the change, with a human owner
AI reviewers such as CodeRabbit, Qodo, Greptile, GitHub Copilot code review and Cursor Bugbot are good at catching obvious bugs, risky patterns and missed conventions in a diff. Use one. Then make sure a person who understands the system owns the merge decision for anything touching money, data, auth or deletion. Every reviewer vendor says some version of what GitHub's docs say about Copilot: it "is not guaranteed to spot all problems."
4. Make sure tests run the new behavior
Your CI should run unit, integration and (for user-facing changes) end-to-end tests, and you should know which new code they reach. Coverage reports answer "which lines ran during tests?" That is necessary information, but it is a floor, not a verdict. App-level QA agents can also exercise user flows on preview environments and catch broken journeys before merge.
5. Prove the tests would fail if the behavior broke
This is the layer most teams skip, and the one AI-written code needs most. The question is not "did a test run this code?" but "if this code returned the wrong answer, would any test fail?"
Mutation testing answers it directly. A tool changes the code on purpose (flips a comparison, returns a constant, removes a call) and reruns your tests. If a test fails, the fault was caught. If every test still passes, you have found a function that is executed but not protected. Stryker (JavaScript and TypeScript), PIT (Java) and mutmut (Python) are mature open-source options. Google has run diff-based mutation analysis in code review for thousands of engineers, and Meta has published its mutation-guided, LLM-based test generation system, ACH. The idea is well established; the practical challenge is cost and focus.
A targeted version keeps the cost down: pick the functions that matter most, break each one in an isolated copy, and rerun only the tests that call it. A test that passes on the original code and fails at its own assertion on the broken copy has earned its place. One that still passes is a finding, and a precise one: you know exactly which behavior is unprotected.
6. Check the combined change after merge, before release
Pull requests are reviewed one at a time. Releases ship them together. Between the last review and the release, several agent-written changes land on the main branch and interact. A function that was fine in one PR may now be called from a new path that no test exercises with an assertion.
So before you cut a release, run a check on the merged code that is about to ship: which functions changed since the last release, which of them are risky, and which have a test proven to catch a break. That check should produce evidence your team can keep, not just a pass or fail light.
A worked example
Suppose an agent adds a loyalty discount:
export function applyDiscount(totalCents: number, tier: "gold" | "silver" | "none"): number {
if (tier === "gold") return Math.round(totalCents * 0.9);
if (tier === "silver") return Math.round(totalCents * 0.95);
return totalCents;
}And writes this test:
it("applies a discount", () => {
expect(applyDiscount(10_000, "gold")).toBeGreaterThan(0);
});The test passes. Line coverage for the gold branch is 100%. Now break the function on purpose so it returns totalCents unchanged for gold customers. The test still passes. Replace the body with return 1; and it still passes. The test executes the code but does not protect it.
A test that earns its place looks like this:
it("gives gold customers 10% off", () => {
expect(applyDiscount(10_000, "gold")).toBe(9_000);
expect(applyDiscount(10_000, "silver")).toBe(9_500);
expect(applyDiscount(10_000, "none")).toBe(10_000);
});Run the same faults against it and it fails each time. That pass-then-fail pair is the receipt you want to keep.
Validation checklist
| Layer | Question it answers | Typical tools | Runs |
|---|---|---|---|
| Intent | What must this change do and never do? | Spec file, ticket, CLAUDE.md or AGENTS.md | Before coding |
| Static checks | Does it compile and follow the rules? | Type checker, linter, SonarQube, Semgrep, CodeQL | Every commit |
| Review | Does this diff look right? | Human reviewer plus an AI reviewer | Before merge |
| Tests run | Did tests execute the new code? | Unit, integration, e2e, coverage, app-level QA agents | CI, previews |
| Tests catch faults | Would a test fail if the behavior broke? | Stryker, PIT, mutmut, OrangePro | Changed code, before release |
| Release check | Is the merged code that ships protected? | Mutation proof on changed functions since last release | After merge, before release |
Where OrangePro fits
OrangePro covers layers 5 and 6. It is a free, MIT-licensed local engine and MCP server. It maps every public function in a repository, links each to the tests that actually call it, and ranks unproven functions by likelihood of change, what they can break and how hard a break would be to notice. For a function and test you choose, it breaks the function in an isolated copy and reruns your unchanged test, and records the result as a pass-and-fail receipt. Where no test catches the break, it can draft a test with your own model key and export a test-only patch that lives in your repository.
It runs after merge and before release, and it is not a PR or release gate. Local scans do not upload your source, and mapping, ranking and proof need no model key.
“Even with high code coverage, gaps slipped through. OrangePro checks whether a test actually fails when the behavior is broken on purpose. That is the question coverage never answered for us.”
Frequently asked questions
Layer your checks: a written intent, deterministic static analysis, human-owned review, tests that execute the change, proof that tests fail when the behavior breaks, and a check on the merged code before release. Each layer answers a different question.
No. Coverage shows which lines ran during tests. A test can run every line and assert almost nothing. Mutation testing or targeted fault injection shows whether tests actually detect wrong behavior.
It can, and it often writes useful ones. But tests from the same agent share its assumptions. Check them independently by breaking the code and confirming they fail, and have a person review what the tests assert.
Introduce faults on purpose and see which tests notice. Surviving mutants point to behavior that is executed but not checked. Focus first on functions that handle money, permissions, data deletion and retries.
Both. Review and CI belong before merge. A check of test protection on the combined, merged code belongs after merge and before release, because that is the code you actually ship.
Try it on your code
See which functions in your repository are executed by tests but not protected by them. Run the MIT-licensed engine locally. Local runs do not upload your source.
npx -y @orangepro/mcp-server@latest start .Built for after merge, before release. It is not a pull request or release gate. Local scans do not upload source.