It's Not Just the Model, Dummy: How We Rebuilt Our Code Review Harness

Every other week, a new model drops and our Slack lights up: should we switch? For a while, we always did. Until we noticed something: our biggest gains weren’t coming from the model at all.

Despite an ongoing debate over whether harnesses become irrelevant as models get smarter, we’ve found the opposite: as models get better, the harness around them matters more, not less. The idea of investing in the harness has gained real traction over the last few months, with OpenAI and Birgitta Böckeler giving it a name: Harness Engineering. Measuring a harness, though, is another story. It’s harder, and a lot less sexy than plugging in the latest model and rerunning a benchmark.

Yet, someone has to do it. So we did.

In this post, we’ll describe the recent work we’ve done on our flagship Code Reviewer agent. We’ll describe our KPIs, benchmark and main takeaways from this project.

What are we trying to improve?

We heard you. Here’s what you, our customers, told us you care about.

Namely, 3 main categories, in the following order (in parentheses - the metric used to quantify it):

1. Quality - A code reviewer earns its place only if developers trust it enough to read what it says. It should cover all interesting bugs (recall), but avoid reporting false positive or noisy findings (precision)
2. Latency - coding has accelerated rapidly; we don’t want to be the bottleneck (mean review time)
3. Cost - to keep the entire reviewing process viable (mean review cost)

As the shrewd practitioner already knows, these metrics pull in different directions. Catch more bugs, get more false alarms. Want better quality? Pay for a slower, pricier model. Our goal: push the agent to improve on all fronts.

How do we measure it?

We use an internal benchmark of about a hundred pull requests with hundreds of tagged bugs, drawn from open-source projects such as Django and Airflow plus internal Baz PRs with bugs that require deeper investigation. Each run reviews every PR, and an LLM-as-a-judge matches the findings against the known bugs. From that we compute the metrics above: recall, precision, review time and cost.

What changed in Harness V2?

Last December, we described the architecture of our agentic reviewer in Engineering Intuition at Scale. The core idea was to turn a senior engineer’s review habits into explicit stages: map context, infer intent, ask Socratic questions, then spawn sub-agents that each prove or disprove one risk. A reflection step consolidated their verdicts.

Since then, models have become far more competent, and swapping in a newer one is no longer where most of our gains come from. So our focus has widened from which model runs to how the agent works around it: the graph of stages, what each stage is responsible for, and what context it gets.

The biggest shift in V2 is not which model runs. It is how the agent investigates.

1. A simpler pipeline. Older models benefited from many shallow (and sometimes overlapping) steps and validations. Newer models don’t need that hand-holding, so we cut the pipeline down to a minimal number of unique stages.
2. Discovery and proof are separate jobs. Discovery needs breadth; verification needs focus. In V2, parallel surveys explore different review scopes and propose concrete failure scenarios, not vague “this looks risky” notes. A dedicated verifier then investigates each concern on its own.
3. Every concern comes with a way to prove it wrong. Each concern carries falsification checks: what would have to be true for this bug not to exist. The verifier traces the actual code paths and returns a verdict such as Confirmed, Refuted or Insufficient evidence. LangChain saw the same lever in its own agent: self-verification was one of its biggest harness gains.
4. Evidence drives severity. Before assigning severity, the verifier must establish a reachable trigger, observed behavior and supported impact. A concern can’t be critical just because it sounds alarming.

The model still matters

To be clear: the model still matters. A stronger model reasons more deeply, follows code paths further and writes clearer explanations, and no harness fully makes up for a weak one. Over this period, GPT-5.6 Luna became our default for both discovery and verification, and the broader review stack gained Gemini 3.8 Flash, Sonnet 5.5 and Haiku 5.5.

But a new model rarely drops in cleanly. Each one brings its own quirks, and the harness has to adapt to them:

• Caching. Prompt layout and context ordering that kept one model fast and cheap may not carry over to the next, so caching has to be re-tuned.
• Tool calls. Models format and use tools differently, so we validate tool schemas and arguments rather than assuming every model calls them the same way.
• Prompts and thresholds. A prompt, stage boundary or confidence threshold tuned for one model can quietly become a bottleneck for the next. Prompts alone are a big lever: GEPA (ICLR 2026) evolved a system’s prompts with the model weights fixed, and beat RL fine-tuning by 6 points on average while using up to 35x fewer rollouts.

So every model swap should be considered as a harness change. That’s why we run each one through the full benchmark and compare quality, cost and latency together before it becomes a default.

Results

Over seven weeks of harness iteration, recall rose from 51% to 67%, review time fell by about 60%, and cost per PR roughly halved.

*(note: small denominator is due to a smaller set)
† Precision proxy is findings that match a known bug divided by all findings. It undercounts real precision, because findings outside the golden set may still be valid bugs.

Takeaways (they’re free!)

Seven weeks of harness work taught us a lot. Here’s what it takes to keep a harness working as models evolve.

• Keep the structure simple. Yesterday’s guardrails are today’s speed bumps. Steps that once kept weaker models on the right path (like fine-grained stages and extra validation) now pollute the context, add cost and slow everything down. Every stage has to earn its latency and cost: deduplicate early, and cut whatever doesn’t add signal.
• Make every step visible. Split the work into distinct stages with structured outputs, so you can trace, measure and debug each one, and see which change moved which metric.
• Split wide coverage from deep analysis. Each agent gets one focused task and a fresh context. One agent asked to both find and prove bugs does neither well.
• Evaluate the system, not the model. A new model is a new harness problem. Even Anthropic notes that the same model’s SWE-bench score varies significantly with its scaffolding. Measure quality, cost and latency for the whole pipeline every time something changes.
• Give your agent a skeptic. Lately, the “generator-verifier” pattern has been gaining popularity (e.g., VeriHarness, Huang et al.). In our experience, it lets you generate ideas cheaply (improving recall) while staying critical and investigative (improving precision).

These days, with a new model always around the corner, your harness shouldn’t be the reason you fall behind. You could try to build and maintain all of this yourself. Luckily, you don’t need to: let Baz keep up with the models for you.

Try Baz free, or book a demo to learn more about how Baz fits your team.

Sources

• Baz, Engineering Intuition at Scale: The Architecture of Agentic Code Review, December 2025
• Anthropic, Raising the bar on SWE-bench Verified with Claude 3.5 Sonnet, October 2024
• LangChain, Improving Deep Agents with harness engineering
• OpenAI, Harness engineering: leveraging Codex in an agent-first world, February 2026
• Birgitta Böckeler, Harness Engineering for Coding Agent Users, April 2026
• Huang et al., Large Language Models Cannot Self-Correct Reasoning Yet, ICLR 2024
• Zhang et al., VeriHarness: Scaling Agentic Verification for Long-Horizon Tasks, October 2026
• Agrawal et al., GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning, ICLR 2026