Building an AI code review layer for your software factory: four lessons from a CTO

Coding agents helped our team produce more software. They also gave our engineers more work to review. So, we built an excellent code review agent, but it wasn’t enough. We still found ourselves spending more time correcting mistakes, explaining system constraints, and revisiting design decisions after the code was already written. We were getting more output, but too much of the time saved in implementation was being spent downstream.

Improving that meant building a review layer into our development process, from planning through implementation, and building a decisions model to fast-track straightforward PRs. Agents needed shared codebase context, a way to learn from corrections, and clear boundaries for when to proceed independently or bring an engineer in. Connecting those pieces let us guide and verify the work throughout development instead of leaving unresolved decisions for the final pull request review.

Here are the four lessons that shaped that approach.

1. Give coding agents shared codebase context

Our engineers used different coding tools and maintained different local instructions (until, finally, Anthropic announced support for AGENTS.md). One person’s agent knew an internal convention. Another person’s agent didn’t. Reviewers kept correcting mistakes that someone else on the team had already explained. We had useful knowledge, but we weren’t making it consistently available.

We began moving that knowledge into shared instructions and reusable skills that were available in a central and separate repository. I covered which patterns to follow, how to use internal tools, what to verify, and when to ask for help. Engineers owned the guidance, and agents could use it regardless of who initiated the work.

The lesson applies anywhere agents work across a team. Experienced engineers carry context that a model can’t reliably infer from code alone. Making that context shared gives every agent a better starting point and reduces how often reviewers have to supply it afterward.

2. Turn review feedback into continuous learning

Shared instructions only help while they remain accurate.

As our codebase changed, guidance that had been useful could start steering agents toward outdated patterns. Engineers would correct the implementation, but unless someone updated the instructions, the next agent could repeat the mistake. We needed a continuous learning loop between the work being reviewed and the context guiding future work.

We built a Skills Maintainer that reviews code changes weekly and proposes updates to our guidance. Engineers approve those changes, preserving ownership while getting help identifying where instructions have drifted. Review feedback has a role in that loop too. When a correction reveals a recurring constraint, it should become a candidate for shared guidance.

The important change was in what we expected a correction to accomplish. Fixing the current PR was the immediate task. Preserving the lesson made that effort useful to the rest of the team.

For organizations generating more code with agents, this is how review can compound. Engineers shouldn’t have to teach the same lesson at the rate agents produce implementations.

3. Shift review left into the implementation plan

The most expensive review finding can be that the agent built the wrong thing correctly.

An implementation might follow our conventions and pass its checks while resting on an assumption nobody approved. Discovering that in a finished PR means revisiting the requirement and reworking code. Our State of Rework 2026 report, based on twelve weeks of PR activity across engineering organizations on Baz, showed how much review capacity this broader problem consumes: roughly 80% of review activity was associated with PRs that entered rework.

That finding reinforced the importance of delivering useful feedback earlier. We started reviewing consequential decisions during planning, before implementation made them expensive to revisit.

That’s where Baz Planner comes in. Planner reviews each plan using what Baz knows about our system, including our codebase and engineering standards, as well as past developer feedback and responses to findings. This lets us review consequential decisions before the coding agent builds on them.

The approved plan then gives the coding agent a framework to work independently within, making implementation decisions that fit the agreed intent and constraints. If new information takes the work beyond that framework, it knows to bring an engineer back in.

Those same boundaries make verification more concrete. A diff can look correct and still miss what the plan intended, so we built Baz Spec Reviewer. It compares the implementation with the approved specifications, identifies deviations, and surfaces the decisions that need engineering judgment.

The plan connects steering, implementation, and review: engineers establish the direction early, agents work within it, and verification identifies when that direction needs another look.

4. Make human review an escalation, not a default

Human review should be reserved for the PRs that need engineering judgement.

Once review runs through both planning and implementation, engineers have many places to give feedback or approve. By the time a PR opens, the consequential decisions have already been reviewed in the plan. Approving the PR then comes down to two questions: does it follow the approved plan, and is the code correct?

That led us to build another agent, Merger, aimed at closing the loop. It flags dangerous changes for an engineer and lets the majority of PRs merge. In roughly 75% of cases, human insight isn’t needed and the PR simply gets an LGTM, which gives that review time back to engineers.

The lesson is to treat engineering attention as the scarce resource. When people review only the changes that call for their judgment, review stops being a default step on every PR, and engineers save time for the work that needs them.

We learned these lessons so you don’t have to

If I were applying this on another team, I’d choose one workflow the team understands well and trace how knowledge, feedback, and decisions move through it.

First, I’d look for knowledge that currently lives in people’s heads or personal agent setups.

• Can every agent doing this work access the relevant standards and system constraints?

• Which conventions do reviewers repeatedly explain?

• Who owns the shared guidance, and what keeps it current as the codebase changes?

Then I’d follow a correction.

• When an engineer catches a meaningful issue, does that lesson reach future agents?

• Can it become part of the team’s shared guidance?

• Who reviews and approves the update, and how will the team know when that guidance needs to change again?

Third, I’d follow a consequential decision.

• Was it made during planning, or did it first surface in a finished PR?

• Did the plan give the agent clear intent and constraints?

• Was the implementation verified against it?

• Did the agent know when to proceed independently and when to involve an engineer?

The final step is connecting those pieces. A plan should draw on shared context. Verification should use the approved plan. Review feedback should improve the context for the next change. Engineers should be able to define where approval is required throughout that cycle.

That connected process is what we’re building at Baz: shared context, continuous learning, maintained skills, and review that starts with the plan and follows the work throughout the software development lifecycle. Together, those capabilities form a review layer that can sit inside a software factory and help engineering teams ship more without absorbing an ever-growing review queue.

Making that work requires being deliberate about where engineering expertise is needed. As agents take on more implementation, engineers need room to consider the system those implementations are creating: how new features fit together, how the architecture should evolve, and what the organization should build in the first place.

If you’re building this yourself, I hope these lessons save you some time and a few wrong turns. But you don’t have to go at it alone. Let Baz carry some of the burden for you. Try Baz free, or book a demo to learn more about how Baz fits your team.