Frontier vs local code review

Background

Why we tested local models

Hosted code review has three practical costs for us: per-review spend, provider rate limits, and sending customer source code outside the network where it is stored.

Finding bugs is only half the job. A reviewer that catches real defects but floods every clean change with false alarms still creates more work than it saves. So we measured detection separately from noise, then checked the precision of the findings each model produced and whether the reviews completed successfully.

We tested three models through the same production review agent, with the same prompts and tools: GPT-5.6 Luna, a hosted frontier model, and two 27B models running locally on a single H200 at bf16, Qwen 3.8 and Northcode Mini.

Reasoning was enabled for all three, but this was not a perfectly controlled variable. Luna exposes an effort setting, which we ran at its highest level; the local models expose only an on/off switch.

The results did not split neatly into hosted and local: Qwen was much closer to Luna than Northcode was to Qwen.

Dataset

How this was measured

Each case replays a real merge request from a production codebase at the commit where the review began. This is a full agentic review, not a single prompt: the model sees the diff, searches the repository, opens files, follows references, and eventually returns findings or a clean verdict.

One merge request can produce more than one case because it may be reviewed at several commits. The noise experiment therefore contains 75 case snapshots from 47 merge requests; the precision experiment contains 50 cases from 50 merge requests. Thirteen merge requests appear in both sets, leaving 84 distinct merge requests overall.

Results

Similar detection, very different noise

Denominators differ between rows because a review that crashed is neither a hit nor a miss, so errored cases are excluded rather than counted as silence.

On case-level detection, the experiment is inconclusive: all three confidence intervals overlap. The clearer difference is what happens when the models speak. Luna stayed silent on 52% of clean cases and had 40.0% precision; Qwen stayed silent on 19% and had 24.4% precision.

Northcode landed on 4 of 22 bug-bearing cases and produced 3 true findings out of 42 settled. Its precision was 7.1%, the lowest of the three.

The models

The three models individually

GPT-5.6 Luna — hosted, frontier

Luna had the highest measured precision, at 40.0%, and lost three cases to model errors. Its detection interval overlaps Qwen's. Every finding named a file, although 24 findings carried no line number.

A median Luna review used 77 tool calls and 10,858 reasoning tokens. We did not run Luna at lower effort, so this experiment cannot tell us how much of its precision advantage comes from spending more compute.

Qwen 3.8 27B — local

Qwen's detection interval overlaps Luna's, but the reviews are substantially noisier. It emitted 97 findings to Luna's 64, stayed silent on only 19% of clean cases versus 52%, and had 24.4% precision versus 40.0%.

It lost one case to its own errors. Its large-diff tier was lost to a dropped network tunnel on three separate attempts, so those results are an infrastructure failure on our side rather than model behaviour, and they are excluded.

Northcode Mini — local

Northcode caught 4 of 22 bug-bearing cases. Of 42 findings with settled judgments, only 3 were true and 2 remained unclear. Grounding was also poor: 49% of its findings named no file and 72% gave no line number, so most of its output cannot be checked against the code without further work. It rated 49% of its findings high severity, against Qwen's 9%.

Failure analysis

How the models fail on known bugs

Aggregate scores showed that Northcode was worse, but not why. To see where the review broke down, we replayed ten merge requests containing eight known defects and recorded what each model actually inspected.

Luna spent about 81 tool calls and 9,872 reasoning tokens per review and found 5 of the 8. Qwen spent 25 calls and found 4. Northcode spent 15 and found none.

We cannot read the local models' reasoning directly, since our serving stack reports zero for that field whichever model runs. Both do reason: a direct probe returned a 1,930-character reasoning block from Northcode, and Qwen's output drops from 2,196 tokens per call to 118 when reasoning is switched off.

On one merge request, Northcode returned a clean verdict after only three tool calls even though the change contained a mis-indented Kubernetes affinity block. Three calls were not enough to inspect all of the changed files.

We then required every finding to include a file and line. Northcode responded by dropping every finding and returning all ten merge requests as clean. That is consistent with a grounding failure rather than a simple formatting failure, but this experiment does not distinguish the two.

Some of its findings contradicted the source rather than merely going unproven. One claimed an array was treated as truthy where the code compares a string. One flagged a newly added constant as a backwards-compatibility break. One called a Postgres-only SQL operator a portability risk, in a file whose path contains postgres.

Misses by Luna and Qwen

Luna and Qwen miss well over half the defects in front of them, and the traces show a different pattern from Northcode's. On one large merge request Luna made 84 tool calls and opened 21 of the 71 changed files. In a separate Qwen run with reasoning disabled, 26 of 31 missed defects sat in files the reviewer never opened; it judged those hunks from the inline diff alone, and only 4 were misses on code it had actually read.

In these traces, many misses coincide with incomplete file coverage before the review budget runs out. We have not yet tested whether a larger budget improves detection.

Hypotheses

What might explain the difference

Three things looked potentially relevant to the gap: reasoning, how much of the code the model investigates, and whether it knows what review procedure to follow. We could test the last two directly on Northcode. Reasoning had to be tested on Qwen, because disabling it on Northcode also disables tool use in our serving stack.

Reasoning volume

Luna emits a median 10,858 reasoning tokens per case. The local models report zero for that field, so we cannot compare them directly on it. We ran the whole 75-case suite twice with the thinking flag changed and everything else held.

Turning reasoning off made individual calls much shorter, 118 output tokens instead of 2,196, but the agent compensated with more tool calls. Thirty-nine of 75 reviews hit the request cap without producing a verdict, and completions fell from 66 cases to 30. The cases that did finish were much faster, with a median of 59 seconds rather than 462.

We therefore treat this as a Qwen-specific result at this request budget. We have not tested whether it holds with a larger budget or with other models.

Forcing more tool calls

If a model stops after three tool calls, the harness can refuse to let it. We ran several variations against Northcode.

Giving it the procedure

We wrote down the procedure visible in Luna's traces (open the changed file, read around the hunk, follow the call sites, check the error path, then judge) and gave it to both local models as instructions.

The procedure helped Qwen: it found defects it had previously missed, including a producer/consumer deadlock, and described the mechanism correctly. Northcode changed its behaviour but not its result. Tool calls doubled and its findings increasingly repeated the instructions back, while detection remained unchanged across six versions of the instruction pack.

Harness changes

Harness and prompt changes

Earlier experiments tried two broad kinds of intervention: changes that imposed deterministic constraints around the model, and changes that depended on the model adapting its own behaviour.

Changes that improved a measured result

Changes that did not improve the measured result

The tool-removal experiment was the most counterintuitive. The relevant code had already been placed in the prompt, so we expected removing search and file-reading tools to reduce cost without affecting recall. Input tokens did fall substantially, but both bugs found by the tool-using runs disappeared. We have not isolated the cause.

The successful interventions all imposed deterministic constraints around the model. None of the attempts that relied on the model adapting its own behaviour improved the measured result. Prompt wording changed how the agent spent its budget more reliably than it changed how much budget it spent, which moved our work toward deterministic controls: which tools the agent gets, what context is assembled before the review, and what evidence the output parser requires.

Summary

Qwen versus Luna, item by item

The Qwen comparison should not be generalized to local models as a class. Northcode ran in the same harness and on the same class of hardware and performed substantially worse, so model selection remains a large source of variation.

What we still don't know

• Most emitted findings are false positives. The highest measured precision was 40.0%, and only 18 of 90 findings with settled judgments were real defects.
• The sample does not separate Luna and Qwen on detection. Their detection intervals overlap substantially. Luna's precision point estimate is higher, but those intervals also overlap, 25–58% against 14–39%. The clearer difference is silence on clean code: Luna returned nothing on 23 of 44 clean cases, compared with Qwen's 8 of 42.
• Northcode is easier to distinguish on output quality than on detection. Its measured precision was substantially lower at 7.1%, and roughly half of its findings do not identify a file.
• The models rarely agree. From the earlier round: of 233 findings on real merge requests, deduplication leaves 163 distinct claims, and only 22% were raised by more than one model. The experiment cannot tell whether that disagreement reflects complementary detection or simply different false positives.
• Corroboration between models is weak as a signal. In that same round three of the five real defects were caught by two models, against a 22% base rate. Five defects is too small a sample to calibrate on.
• The precision labels are themselves model judgments. Each finding was checked against the code at the relevant commit by another language model, blind to which reviewer produced it, rather than by the engineers who own that code.
• Some surviving noise may be preference rather than factual error. A finding can be technically correct but concern behaviour the team has deliberately chosen not to change. We have rejection-history data that could test this, but have not used it yet.

The result is narrower than "local can replace hosted". Qwen's observed detection rate was similar to Luna's in this sample, but its reviews were substantially noisier and its measured precision was lower. For us, that makes deployment location a secondary question. The first question is whether the model can investigate reliably; the next is whether the harness gives it enough coverage and evidence to make its verdict useful.