---
title: "Skills Maintainer: keeping your instruction files true"
description: "Your CLAUDE.md was accurate the day it was written. Skills Maintainer checks each rule against the code and the pull requests that came after, and opens a PR when a rule no longer holds."
canonical: "https://baz.ai/resources/blog/skills-maintainer-keeping-your-instruction-files-true"
---

# Skills Maintainer: keeping your instruction files true

Your CLAUDE.md was accurate the day it was written. Skills Maintainer checks each rule against the code and the pull requests that came after, and opens a PR when a rule no longer holds.

By Omer Militcher | 2026-08-10

Your repository has a `CLAUDE.md`. Probably an `AGENTS.md` in a few packages, and a `.claude/skills/` folder someone filled in during a good week. They were accurate when they were written. Nobody has read them since.

Stale docs and stale instruction files fail differently. A developer reading bad documentation notices and works around it. An agent reading a stale rule just follows it, on every pull request that touches the path.

## What drift looks like

A rule says every API handler must call `getAuthorizedUser` before touching the database. True when written. Six weeks later the team moves authorization into middleware and handlers stop calling it directly. The rule is now wrong, and it pushes new code back toward a pattern the team deliberately left.

A quieter version: "import `Button` from `@/components`." The barrel gets split, the path changes, and agent-written components keep reaching for a path that no longer exists.

Neither is a bug anyone wrote. The file just stopped describing the codebase.

## Why it stays broken

There is no failing test for "this rule is no longer true." No linter, no type error, no CI job.

The only signal is the gap between what the file says and what the last few hundred merged pull requests actually did. Nobody runs that comparison by hand, across nine repositories, for files they did not write.

## What Skills Maintainer does

It runs the comparison. It reads the instruction files Baz already discovers for the [AI Coding Guidelines](/docs/agents/skills-and-instructions) reviewer, checks each rule against the current code and the pull requests that came after it, and opens a pull request when a rule no longer holds.

A pull request, not a dashboard. The output is a diff against `CLAUDE.md` with the reasoning attached, reviewed and merged like anything else.

## How it finds the evidence

The obvious design is to hand a model every rule and every merged pull request and ask what changed. That is the most expensive option and the least accurate one. Everything before the judgment is a narrowing problem, and narrowing is better served by cheap, deterministic signals.

**Drop what carries no signal.** Dependency bumps, reverts, empty titles. Instruction files disabled for review are dropped on the same pass, so the comparison only runs on rules that are actually in force.

**Match pull requests to rules by scope.** Every instruction file governs a path. A skill at `services/api/.claude/skills/foo.md` governs `services/api`, the parent of the agent directory; a file outside an agent directory governs its own parent. Paths named inside the file itself narrow that further. A pull request becomes a candidate for a rule when it changed a file under that scope, and it can be a candidate for several.

That is a boring rule, and it beat the alternatives. We tested it on 170 pull requests against 42 instruction files, alongside a reranker (Cohere rerank-v3.5 on Amazon Bedrock) in three configurations:

| Candidate gate | Cost per run | Time | Result |
| --- | ---: | ---: | --- |
| File-path scope | $0 | instant, offline | best recall, strong precision: 1,133 candidate pairs down to 112 |
| Batch rerank, one query per pull request | ~$0.33 | 40-56s | abstains often, assigned only 39 of 169 pull requests |
| Pairwise rerank, one query per pair | ~$2.27 | slowest | same results as batch |
| Cluster-sampled rerank | ~$0.06 | ~7s | worse than batch |

The gate's job is recall. A rule that never becomes a candidate never gets checked. Reranking optimizes for one confident pick, which is the wrong objective a stage too early: it throws away the pull request that would have proved a rule stale. It is still worth reaching for as a second filter if false positives become the problem, but the free option was better at the job.

**Group pull requests into themes.** One pull request that ignores a rule is an exception. Eight that all do the same new thing are drift, so pull requests get clustered before anything forms an opinion about them.

Each pull request becomes two fingerprints: what it says (title, topics and description, embedded with `text-embedding-3-small`) and where it lands (the directories it touches, as a TF-IDF vector where a rare directory counts for more than a ubiquitous one like `src`). Distance is a blend, 0.8 meaning and 0.2 location. HDBSCAN clusters that, and clusters whose semantic centroids sit within 0.4 of each other are merged back together, since broad themes tend to get over-split.

Scored against 202 pull requests our team grouped by hand into 104 clusters:

| Approach | ARI |
| --- | ---: |
| HDBSCAN on blended distance | 0.638 |
| Semantic distance only | 0.417 |
| KMeans, k chosen by silhouette | 0.357 |
| LLM taxonomy | 0.256 |

The winning configuration also scores 0.855 B-Cubed F1 and 0.646 pairwise F1. Two results are worth keeping from the rest of the sweep. Structure is worth about 0.16 ARI on its own: where a pull request lands says something its description does not. And asking a model to tidy the clusters afterwards made them worse, with essentially every edit disagreeing with the human grouping.

HDBSCAN wins because it is allowed to decline. Loners get labeled noise instead of forced into a group, which is what the hand-labeled set does too: most pull requests are legitimate one-offs and belong nowhere. KMeans and the model have to place every pull request, and that is what costs them.

**Then the agent reads.** By that point it is looking at a handful of grouped, in-scope pull requests per rule rather than a repository's full history. The model is spent on the judgment, which rule no longer holds and what it should say instead, not on the search.

## Keeping it usable

Three properties matter more than the analysis, because they decide whether the output gets read:

**Bounded.** You cap how many pull requests it can have open at once. At the cap it stops until you clear some. The failure mode of a tool like this is not bad suggestions, it is fifty good ones on a Monday morning.

**Scoped.** Run it on one repository or all of them. Repositories without write access are visibly excluded rather than silently skipped.

**Separable.** Every pull request gets a title prefix and a branch prefix you choose, so its output is filterable and routable through your branch rules from the first run.

Runs happen on demand until you trust the output, then on a schedule.

{% app-demo view="skills" caption="One pass: pick repositories, scan, review what landed, then put it on a schedule." %}

The [docs](/docs/agents/baz-agents) cover setup and how to start a run.

## What it will not do

Skills Maintainer keeps instruction files honest. It cannot invent them. Point it at a four-line `CLAUDE.md` and it will have almost nothing to say, correctly.

Maintenance is worth the most where enforcement already happens. The AI Coding Guidelines reviewer applies your rules on every pull request, and enforcement without maintenance is the worst combination available, since it applies yesterday's rules at today's speed.

## Start small

Turn it on for one repository you know well. Set the open pull request cap to three. Scan on demand.

Read that first batch carefully. It says less about the agent than about how far your written standards have drifted from your real ones.

Then widen the scope, put it on a schedule, and stop thinking about it.

---

Skills Maintainer is available now. See the [documentation](/docs/agents/baz-agents) for setup.