Mutation testing: breaking code on purpose to see whether the tests notice.

01Definition

Small deliberate bugs, and whether the tests catch them.

Mutation testing makes many small changes to a program, each one a plausible bug: a < turned into <=, a true into false, an && into ||, a line deleted. Each changed version is called a mutant. The test suite runs against every mutant. If a test fails, the mutant is caught, or “killed”. If every test still passes, the mutant survived, which means the tests would not have noticed that bug.

The share of mutants caught is the mutation score. Unlike line coverage, which only says a line was executed, a mutation score says whether the tests actually check what the line does.

• Mutants are small plausible bugs
• Caught means a test failed
• Survivors show where tests are thin

02Why it matters

Passing tests prove little if they never check the change.

An AI-written change often arrives with tests, and the tests pass. That is weak evidence on its own: tests can execute code without asserting anything meaningful about it. Mutation testing turns “the tests pass” into “the tests would fail if this code were wrong”, which is the claim a reviewer actually cares about.

Running it on a whole codebase is slow. Running it only on the lines a pull request changed is fast enough for CI, and it answers the question for exactly the code under review.

• Coverage is not checking
• Changed lines keep it fast
• Answers the reviewer's real question

03In Verify

Built: mutation testing of the changed lines.

After a change's own tests pass, Verify mutates the lines the pull request changed, using StrykerJS, and reruns the project's tests. A code change can be accepted only when its tests catch at least 60% of those mutants and every other check holds. The verdict comment names each mutant no test caught, so the reviewer knows exactly where the tests are thin.

This part of Verify is built and runs on pull requests in TraceAxiom's own repository. Verify is not installable yet.