A failure is a claim about your code. An error is a claim about our harness.

Last week our own verification engine told a developer their code was grade F — do not ship.

The code was fine. What actually happened: QuantumVerifi generated a test, the test tried to import the module under test, and — because of a packaging quirk — the import failed. No assertion ever ran. We never checked the code. And yet we slapped a red F on their pull request as if their logic were broken.

That's not a verdict. That's a lie. So we fixed it — and the fix is a 20-year-old idea that the AI-code-tooling world has quietly forgotten.

The two-state trap#

Most verification and CI tooling models the world in two states: pass or fail. Tests passed, ship it; tests didn't pass, it's broken.

That binary is wrong, and every serious test framework already knows it. The third state has a name, and it has had one for decades:

pytest puts assertion failures in a FAILURES section and import/fixture errors in a separate ERRORS section — on purpose. JUnit's spec spells it out: "a failure represents an assertion that didn't meet expectations, while an error indicates an unforeseen exception or problem… proving something wrong versus something breaking unexpectedly."

The distinction is the whole point:

When your tests run to completion and an assertion fails, the tool has evidence your code is wrong. When the tests can't even be collected — an import error, a missing dependency, a compile error in the test harness, a sandbox that didn't boot — the tool has no evidence about your code at all. Reporting "FAIL" in that case isn't strict; it's dishonest. It's asserting something the tool never measured.

Why this matters more for AI-generated code#

Here's the twist that makes this urgent rather than academic.

When a tool generates the tests it runs — as every "AI writes tests for your code" product now does — the harness itself becomes a moving, fallible, AI-authored thing. It will sometimes generate a test that can't import your module. It will sometimes pick the wrong test command. The sandbox will sometimes fail to install a dependency.

In a traditional setup, a human wrote the tests, so "the tests couldn't run" was rare and usually meant your build was broken. In an AI setup, "the tests couldn't run" is a routine event that is frequently the tool's fault, not yours. Collapse that into "fail," and your verification tool spends its day confidently blaming developers for its own stumbles. Trust evaporates in about three false reds.

If the product is trust — and for verifying AI-generated code, trust is the entire product — then conflating "couldn't run" with "failed" isn't a bug. It's an existential one.

The fix: a first-class outcome, not a heuristic#

We had been patching this at the edges — sniffing test output for "ModuleNotFoundError," special-casing it in one code path while three other paths still collapsed it to F. That's the wrong shape. The right shape is to make the distinction first-class and let everything downstream derive from it:

Code
Copy
Outcome:
  Passed   → tests ran, all assertions held
  Failed   → tests ran, an assertion about your code was violated   → 🔴 block
  Errored  → tests could not run (import / collection / setup)       → ⚪ inconclusive, never blocks
  (syntax/build break of your code is handled separately, and stays 🔴)

A single classifier computes the outcome from the test runner's native signals — pytest's E, JUnit's <error> — instead of a tangle of string matches. The grade and the PR verdict both read from it. And the PR check maps Errored → GitHub's neutral conclusion: the API's own sanctioned state for "failing is too harsh and success isn't correct either." Neutral doesn't block your merge. It tells you the truth: we couldn't verify this — here's why — go look.

The same idea is the foundation of mutation testing#

This isn't a one-off. The most rigorous corner of the testing world — mutation testing — is built on exactly this discipline.

PITest refuses to start if your suite isn't green, with a now-famous error: "Mutation testing requires a green suite." It aborts rather than emit a misleading score against a broken baseline. Stryker buckets mutants that error (rather than fail) as Invalid and excludes them from the score. Cosmic Ray has a dedicated INCOMPETENT state for "the mutation just made the code crash."

Every one of them refuses to let "we couldn't run it" masquerade as a result. That's not pedantry — it's the only way a score means anything. When we build mutation scoring into PR checks ("your new tests caught 8 of 10 injected bugs"), it sits directly on top of this taxonomy. You can't honestly grade what you couldn't run.

What we believe#

A verification tool earns trust by being more careful about what it claims, not less. So the rule we hold ourselves to:

• 🟢 Safe to ship — we ran your tests and they passed.
• 🔴 Do not ship — we ran your tests and an assertion failed, or we found a real vulnerability, or your code doesn't build.
• ⚪ Could not verify — our harness couldn't execute the tests. That's on us, not on you. We'll tell you, and we won't pretend otherwise.

A red verdict from us is a promise: we measured this, and there's a problem. If we can't keep that promise, you get a neutral, never a fake red.

A failure is a claim about your code. An error is a claim about our harness. We won't confuse the two — because the entire point of a trust layer is that you can trust it.

QuantumVerifi generates and executes real tests for your code in an isolated sandbox and posts an evidence-backed verdict on every pull request. Learn more →