Flaky tests: tests that pass and fail on the same code.

01Definition

Same code, different answers.

A flaky test gives different results on the same code: it passes on one run and fails on the next. Common causes are timing and ordering assumptions, shared state between tests, reliance on the network or the clock, and random data.

Flakiness is common in real codebases, and it is a problem for any system that turns test results into a decision.

• Timing and order
• Shared state
• Network, clock, randomness

02Why it matters

A verdict is only as reliable as the tests under it.

If a test is flaky, a failure on head might not be caused by the change, and a pass might be luck. A verifier that treats every failure as a regression will reject good changes; one that ignores failures will accept bad ones.

The honest approach is to compare base and head, treat unexplained instability as something to report rather than hide, and say plainly when a result could not be trusted.

• False rejections
• False accepts
• Report, do not hide

03In Verify

Compared on base and head, and reported plainly.

Verify runs the same tests on base and head, so a failure that also happens on the base is not counted against the change. When the checks cannot produce a trustworthy result, the verdict is couldn't run or needs review, with the reason, rather than a guess.

Verify is not installable yet.