Mutation score: the share of deliberate bugs the tests catch.

01Definition

Caught mutants divided by all mutants.

A mutation score is the number of mutants the tests killed divided by the number of mutants that were tested, usually shown as a percentage. If a tool made ten small bugs in a function and the tests failed on eight of them, the score is 80%.

Some mutants cannot be caught by any test, because the change does not alter behavior; these are called equivalent mutants. Tools handle them differently, which is one reason scores from different tools are not directly comparable.

• Killed ÷ tested
• Equivalent mutants cannot be killed
• Compare scores within one tool

02Reading a score

A threshold is a policy, not a law of nature.

There is no universal good score. A payment calculation deserves near-complete coverage by mutants; glue code may not. What matters is choosing a threshold for a purpose and applying it consistently, and looking at the surviving mutants, because each survivor is a concrete bug the tests would miss.

Scoring only the lines a change touched gives a score for the change rather than for the whole codebase, which is the number a reviewer of that change needs.

• Thresholds fit a purpose
• Survivors are the useful output
• Score the change, not the codebase

03In Verify

Built: an accept needs at least 60% of changed-line mutants caught.

Verify's current policy accepts a code change only when its tests catch at least 60% of the mutants on the changed lines and every other check holds. The comment lists the surviving mutants by line. The threshold is part of a published policy so a reader can see what an accept rests on.

Verify is in development and not installable yet; it posts verdicts on pull requests in TraceAxiom's own repository.