All posts
Engineeringflywheelfine-tuningQLoRAself-improvingAI

Your Tests Get Smarter Every Run: Inside the QuantumVerifi Flywheel

Most AI test tools ship a frozen model and never change. QuantumVerifi trains a private model for every customer that learns your codebase a little more with each run — a self-improvement flywheel that compounds.

QuantumVerifi Team2 June 20265 min read

Every AI testing tool on the market has the same quiet flaw: the model is frozen.

It was trained months ago, on someone else's code, and it will write tests the exact same way on day 1 and day 500. It never learns your stack. It never picks up that your team always uses data-testid, mocks the payment gateway a particular way, or names integration suites by feature. You adapt to it.

QuantumVerifi works the other way around. The more you use it, the better it gets at your codebase specifically — because behind the product is a loop that turns every successful run into training signal, and quietly fine-tunes a private model that's yours alone.

We call it the flywheel. Here's exactly how it turns.

The loop#

Code
1 ┌──────────────────────────────────────────────────┐ 2 │ │ 3 ▼ │ 4Analyze ──▶ Harvest ──▶ Train ──▶ Activate ──▶ Route ──┘ 5 run save the QLoRA promote generate 6 tests wins adapter the model sharper tests

Five stages. Each turn of the wheel makes the next turn better.

1. Analyze — normal usage

You run a verification — a repo, a live URL, an API, a load test. QuantumVerifi generates tests, executes them in real sandboxes, watches them pass or fail, and self-heals the flakes. This is just the product doing its job. Nothing special yet.

2. Harvest — turn wins into memory

Here's where it gets interesting. Every time an analysis succeeds — graded C or better, with tests that actually ran green — we capture the good test files as training episodes.

An episode is a clean example: given this code and context, here is a test we wrote that passed. We quality-gate them ruthlessly (must contain real assertions, can't have needed three rounds of self-healing, must clear a length floor), dedupe them, and store them per customer. No noise, no failures, no other tenant's data — just your wins.

3. Train — a private model, in minutes

A scheduler watches each customer's episode count. The moment you cross a threshold of fresh examples, it bundles them into a dataset and fires a fine-tuning job on a GPU.

We don't retrain a giant model from scratch — that would be slow and absurdly expensive. We use QLoRA: we freeze a strong open base model (Qwen2.5-Coder-7B) and train a tiny adapter on top of it. The adapter is the only thing that changes, and it's small — around 140 MB — so a full training run finishes in minutes and costs cents, not dollars.

What that adapter learns is you: your frameworks, your file layout, your assertion style, the way your team actually writes tests.

4. Activate — promote the model

A finished adapter doesn't go live automatically — you do that, with one click, and it's versioned and reversible. Activate it and it becomes the live model for an operation like test generation. Train again next week and you get v2, v3 — each one sharper than the last.

5. Route — sharper tests, automatically

From that point on, your test generation routes through your fine-tuned adapter instead of the generic base model. The output reads like your team wrote it, because in a very real sense, it learned from tests your team's runs produced.

And if anything ever goes wrong with the adapter? It falls straight back to the base model. You never get a broken run; you only get a better one.

Then the wheel turns again. New runs → new episodes → a newer adapter → even sharper tests. It compounds.

Why a tiny adapter changes everything#

The reason this is practical — and not a research demo — comes down to one idea: LoRA adapters are small, cheap, and swappable.

  • The expensive 7B base model is frozen and shared. We never copy it per customer.
  • Each customer's learning lives in a ~140 MB adapter that trains in minutes for roughly the cost of a coffee.
  • Adapters are hot-swappable — activate, deactivate, roll back a version, all instantly.

That's what makes "a private model per customer" economically sane. A thousand customers isn't a thousand 7B models; it's one frozen base plus a thousand tiny adapters.

Built for trust#

A self-learning system is only worth having if you can trust it. So the loop is wrapped in guardrails:

  • Per-tenant isolation. Your model trains only on your passing tests. No shared global model, no cross-customer leakage — your code's patterns never train anyone else's adapter.
  • Quality gates. Only verified, green, well-formed tests become training data. Garbage never enters the loop.
  • Human-controlled activation. Nothing routes through a fine-tuned model until you promote it. Versioned, reversible.
  • Fallback by default. Any hiccup in the fine-tuned path silently falls back to the base model. The floor is "as good as before"; the ceiling keeps rising.

Two GPUs, one loop#

Under the hood, the same training contract runs on two very different backends — because different customers have different constraints:

  • Modal — serverless H100s that scale from zero to many on demand. No idle cost; a burst of training jobs just spins up more GPUs. This is the cloud path.
  • On-prem GPUs — for teams that want training to never leave their building, the identical pipeline runs on a local box. Same datasets, same adapters, same flywheel — just on hardware you own.

Same loop, your choice of where it spins.

Watch it turn#

Open the Training Engine page and the whole thing is right there: the five stages lit up as the loop runs, a live counter of episodes harvested, adapters trained, and models live — and a live-model card showing the fine-tuned adapter currently routing your tests, with its training loss and version.

It's not a dashboard of vanity metrics. It's a picture of a system that is, quietly and continuously, getting better at your software.

The compounding advantage#

Static AI tools have a ceiling: they're as good as the day they shipped. A learning system doesn't.

Every verification you run isn't just a test result — it's a lesson. Those lessons accumulate into a model that knows your codebase better than any off-the-shelf model ever could, and that gap widens the more you use it. A competitor with a frozen model can match you on day one. They can't catch a product that compounds.

Your tests get smarter every run. That's the flywheel — and once it's turning, it doesn't stop.