00Glossary / term
Control tests: planted tests with a known outcome that prove the results are real.
Control tests are planted tests with a known outcome, one that must pass and one that must fail, proving a test run's results are real and were not faked.
01Definition
Two tests whose answers you already know.
A control test is a test planted into a suite with a result known in advance. One must pass. One must fail. If the run reports both correctly, the test runner really executed that file and reported honestly. If the failing control is reported as passing, or neither appears, something between the tests and the report is wrong.
The idea comes from experiments: a control shows the instrument works before you trust its readings.
- One must pass
- One must fail
- Both must be reported correctly
02Why it matters
A change can make tests look green without them running.
A change can skip a test file, mark tests as skipped, replace the test command, or print output that imitates a passing run. Some of these are honest mistakes; some are what an agent does when it is told to make the tests pass. Either way the result looks green.
Controls catch this class of problem mechanically. They do not judge the tests; they prove the run happened as reported.
- Skipped files
- Replaced test commands
- Imitated output
03In Verify
Built: two controls in every test file.
Verify reads each test's result through its own reporter and plants two control tests in every test file before the run, for Node's built-in test runner, Jest and Vitest. If a change switches the tests off, forges their results or imitates a passing run, the controls show it and the verdict says so.
This part is built. Verify is not installable yet.
The waitlist
See this check on your own pull requests.
Verify is not installable yet. Join the waitlist to hear the day it is; teams with coding agents can also become design partners.
Join the waitlist