Ship: Inference Routing for Agentic Code Review
We evaluated Martian's Ship as an execution layer beneath Baz's code review pipeline. The question was narrow: can we change how model calls are executed without changing the review behavior the agent architecture produces?
We evaluated Martian's Ship as an execution layer beneath Baz's code review pipeline.
The question was narrow: can we change how model calls are executed without changing the review behavior produced by the surrounding agent architecture?
For Baz, that behavior includes flow mapping, repository context retrieval, reviewer-specific tool use, deep exploration, security analysis, dependency analysis, and finding generation. These stages use different prompts, tools, models, and timeout budgets.
Ship operates below that layer. Baz decides what work to perform. Ship decides how to execute the model request.
One claim we can confirm at the outset: routing cut cost sharply at matched review quality. On the same 88-pull-request benchmark, the direct GPT-5.4 baseline cost $2.06 per pull request and detected 83 of 148 seeded findings; the Martian-routed configuration cost $1.03 per pull request and detected 82. That is a 50.2% cost reduction at effectively the same recall. This is a matched-quality routing comparison across two named models, not a same-model direct-versus-routed test, and it clears the up-to-30% saving Martian advertises for Ship. The rest of this post is about what that saving is worth once recall and trajectory duration are held to a standard.
Decides what work to perform, which context each stage receives, which tools are available, and what evidence a finding needs.
Decides how the model request runs: which route serves it, which execution strategy applies, and which provider is reached, without touching the review logic above.
Evaluation design
We added direct and Martian-routed model variants to the same internal benchmark harness.
Each run records expected finding coverage, generated findings, tool activity, duration, and token usage. Reports include the exact model pair used for the run so results can be reproduced and compared directly.
The integration remained limited to the model boundary. Baz changed the endpoint, credentials, and model mapping while preserving the rest of the review pipeline.
This matters because an agentic review workload is sensitive to more than final answer quality. A model change can alter tool use, message structure, trajectory length, latency, and stopping behavior. We therefore evaluated the full review trajectory rather than only the final text.
Initial paired comparison
In the first side-by-side runs, Martian-routed GPT-5.4 and GPT-5.4-mini produced results in the same range as the direct routes across initial discovery and deep exploration.
Most expected bugs were found by both routes. When both identified the same issue, the findings were usually similar in structure and wording.
This was the first useful signal. The route change did not materially alter the type of issues the review pipeline surfaced.
Larger benchmark result
In a later paired run, the direct route found 77 of 148 expected findings. The Martian-routed run found 82 of 148.
| Route | Expected findings found | Recall |
|---|---|---|
| Direct | 77 / 148 | 52.0% |
| Martian-routed | 82 / 148 | 55.4% |
We did not treat the five-finding difference as an improvement. The team considered it normal model variance and concluded that the two runs were in the same quality range.
That was the relevant result. The routed path preserved benchmark performance within the variation already present across repeated model runs.
Latency behavior
One trace comparison showed similar median latency between the direct and Martian-routed GPT-5.4 paths, with lower latency in the routed path at P75 and above.
For multi-stage agents, tail latency matters more than isolated median call time. One slow model call can delay the entire review.
We also observed the opposite failure mode in another model test. Per-turn latency improved, but the model used more turns, increasing total review duration.
This is why we measure complete trajectory time rather than call latency alone.
Cost, recall, and duration
A cheaper route is only useful if it stays within the required quality and latency range.
The clearest cost signal comes from the direct baseline. The direct GPT-5.4 benchmark completed all 88 pull requests at $181.38 total, a mean of $2.06 per pull request, and detected 83 of 148 findings. The Martian-routed configuration completed the same benchmark at $90.40 total, or $1.03 per pull request, and detected 82. At matched recall, that is a 50.2% cost reduction. It compares two named models at matched quality rather than one model with and without routing, so we read it as a matched-quality routing comparison.
We tested GLM 5.2 through Martian across 88 pull requests. The run completed all 88 with no execution errors. Total cost was $64.14, or $0.73 per pull request on average. Mean duration was 1,050 seconds. It found 67 of 148 expected findings.
A GPT-5.4-mini route on the same benchmark also completed all 88 pull requests. Total cost was $90.40, or $1.03 per pull request. Mean duration was 328 seconds. It found 82 of 148 expected findings.
| Route | Expected findings found | Recall | Mean cost per pull request | Mean duration |
|---|---|---|---|---|
| GLM 5.2 through Martian | 67 / 148 | 45.3% | $0.73 | 1,050 sec |
| GPT-5.4-mini route | 82 / 148 | 55.4% | $1.03 | 328 sec |
GLM 5.2 was about 29% cheaper per pull request. It also lost 10.1 percentage points of recall and took more than three times longer.
We did not promote that route.
The optimization target is not cost in isolation. It is cost subject to minimum recall, acceptable trajectory duration, and stable tool behavior.
What the results imply
The model is only one component of review performance.
Baz separates flow mapping, deep exploration, code deduplication, security review, and dependency review because each task requires different context and execution behavior. Internal analysis of Advanced Security also suggested that some of its strongest results came from the dedicated review harness, not only the underlying model.
That leads to a two-level architecture.
Baz routes the task. It decides which stages run, which context each stage receives, which tools are available, and what evidence is required before a finding is returned.
Ship routes the model execution inside those stages.
The separation is useful because both layers can change independently. Baz can modify the review architecture without becoming an inference provider. Martian can change execution strategies without requiring Baz to rewrite its review logic.
Conclusion
Our results support a limited claim.
For the Martian-routed configurations we approved, benchmark quality stayed within the same range as the direct model path. In one paired run, the routed path found 82 of 148 expected findings and the direct path found 77.
On cost itself, Martian's claim held, and then some. At matched review quality the routed configuration cost $1.03 per pull request against the direct GPT-5.4 baseline's $2.06, a 50.2% reduction with recall effectively unchanged (82 versus 83 of 148). That is a matched-quality routing comparison across two named models rather than a same-model test, and it clears the up-to-30% saving Martian advertises for Ship. Cost was the one axis that behaved exactly as promised.
Other routes failed the acceptance criteria despite lower cost. A route that was 29% cheaper also reduced recall by 10.1 percentage points and increased mean runtime from 328 seconds to 1,050 seconds.
The practical value of inference routing is therefore conditional. It works when the execution layer can reduce cost or tail latency while preserving the behavior expected by the agent architecture.
Baz controls the review trajectory. Ship controls how the underlying model requests are executed.