Your Tests Pass. So What?

Here's an uncomfortable truth about test suites — yours, ours, the ones an AI just generated for you: a passing test proves almost nothing.

A test can pass while asserting the wrong thing. It can pass while exercising none of the logic that matters. It can pass because it never could have failed. "24 tests, all green, grade A" is a vanity metric. The question your senior engineer actually asks in review is sharper:

Every test-generation tool — including the AI ones — stops at "the tests pass." That's where the real question starts. So we built the layer that answers it.

Inject the bug. See who notices.#

The technique is called mutation testing, and it's been the gold standard for measuring test quality for decades (PIT in the Java world, Stryker in JS). The idea is delightfully blunt: if you want to know whether tests catch bugs, put bugs in and check.

QuantumVerifi takes your generated suite, then deliberately introduces small, realistic defects into the source files those tests target — one at a time:

• flip a > to <=
• invert a condition: && becomes ||
• change arithmetic: + becomes -
• negate a boolean: true becomes false

Each of these is a mutant — a version of your code with exactly one planted bug. For every mutant, we re-run the suite and watch:

• A test fails? The bug was killed ✅ — your tests caught it.
• Everything still passes? The bug survived ✗ — a real hole in your coverage.

The kill score — bugs caught ÷ bugs injected — is the honest, industry-grade measure of whether your tests actually work. Not "do they pass." Do they catch.

We ran it on real code, in production#

Not toy examples. Two popular open-source repositories, end to end:

Read the gin result again. The generated suite passed. Grade A. And mutation testing turned around and said: your routergroup tests are airtight — they kill 100% of injected bugs — but your authentication tests miss two-thirds of them.

That's not a number on a dashboard. That's "your auth code has a blind spot, and here's exactly where."

It doesn't just score — it points#

The output isn't a single percentage. It's a per-file breakdown plus the exact surviving mutants: operator, file, line. On gin, flipping == to != and true to false on specific auth.go lines went completely undetected:

Code
Copy
1SURVIVORS — bugs your tests miss:
2   ==  →  !=     auth.go:33
3   false → true  auth.go:34
4   ==  →  !=     auth.go:37
5   true → false  auth.go:38
6   …

Each line is a concrete, fixable instruction: write a test that distinguishes these two behaviors, because right now nothing does. This is the feedback a thorough human reviewer would give — generated automatically, grounded in evidence, in minutes.

Why this is hard (and why most tools don't do it)#

Running mutation testing on a real repository — not a clean textbook example — is genuinely difficult, and getting it right took real engineering:

• Real suites are never perfectly green. Clone a big project into a sandbox and some of its own tests fail (network, environment, flakiness). We score each mutant relative to that baseline — a mutant only counts as killed if it introduces a failure that wasn't already there.
• We measure your tests, not the repo's. We isolate execution to exactly the generated tests, so a project's own suite doesn't get credit for catching our mutants.
• Some bugs make code hang. A mutated loop can spin forever. We cut each mutant off fast — and a test that hangs is catching the bug, so it scores as a kill.
• It runs out of the hot path. Mutation testing is on-demand and asynchronous, so it never slows down your normal analysis. You click "Run mutation testing," and results stream in per file.

The result works across languages — Go and JavaScript today, with the same engine extending to TypeScript, Python, and Java.

The point#

Anyone can generate tests. Anyone can run them and report pass/fail. The thing almost no one does — the thing that actually earns an engineering team's trust — is prove the tests are worth having.

A passing suite told you everything was fine. Mutation testing shows you where it isn't — before it ships. Open any repository analysis, find the Mutation Score tab, and ask your tests the only question that matters: would you catch a real bug?

Now you'll know.