Skills Maintainer: keeping your instruction files true

Your repository has a CLAUDE.md. Probably an AGENTS.md in a few packages, and a .claude/skills/ folder someone filled in during a good week. They were accurate when they were written. Nobody has read them since.

Stale docs and stale instruction files fail differently. A developer reading bad documentation notices and works around it. An agent reading a stale rule just follows it, on every pull request that touches the path.

What drift looks like

A rule says every API handler must call getAuthorizedUser before touching the database. True when written. Six weeks later the team moves authorization into middleware and handlers stop calling it directly. The rule is now wrong, and it pushes new code back toward a pattern the team deliberately left.

A quieter version: "import Button from @/components." The barrel gets split, the path changes, and agent-written components keep reaching for a path that no longer exists.

Neither is a bug anyone wrote. The file just stopped describing the codebase.

Why it stays broken

There is no failing test for "this rule is no longer true." No linter, no type error, no CI job.

The only signal is the gap between what the file says and what the last few hundred merged pull requests actually did. Nobody runs that comparison by hand, across nine repositories, for files they did not write.

What Skills Maintainer does

It runs the comparison. It reads the instruction files Baz already discovers for the AI Coding Guidelines reviewer, checks each rule against the current code and the pull requests that came after it, and opens a pull request when a rule no longer holds.

A pull request, not a dashboard. The output is a diff against CLAUDE.md with the reasoning attached, reviewed and merged like anything else.

How it finds the evidence

The obvious design is to hand a model every rule and every merged pull request and ask what changed. That is the most expensive option and the least accurate one. Everything before the judgment is a narrowing problem, and narrowing is better served by cheap, deterministic signals.

Drop what carries no signal. Dependency bumps, reverts, empty titles. Instruction files disabled for review are dropped on the same pass, so the comparison only runs on rules that are actually in force.

Match pull requests to rules by scope. Every instruction file governs a path. A skill at services/api/.claude/skills/foo.md governs services/api, the parent of the agent directory; a file outside an agent directory governs its own parent. Paths named inside the file itself narrow that further. A pull request becomes a candidate for a rule when it changed a file under that scope, and it can be a candidate for several.

That is a boring rule, and it beat the alternatives. We tested it on 170 pull requests against 42 instruction files, alongside a reranker (Cohere rerank-v3.5 on Bedrock) in three configurations:

The gate's job is recall. A rule that never becomes a candidate never gets checked. Reranking optimizes for one confident pick, which is the wrong objective a stage too early: it throws away the pull request that would have proved a rule stale. It is still worth reaching for as a second filter if false positives become the problem, but the free option was better at the job.

Group pull requests into themes. One pull request that ignores a rule is an exception. Eight that all do the same new thing are drift, so pull requests get clustered before anything forms an opinion about them.

Each pull request becomes two fingerprints: what it says (title, topics and description, embedded with text-embedding-3-small) and where it lands (the directories it touches, as a TF-IDF vector where a rare directory counts for more than a ubiquitous one like src). Distance is a blend, 0.8 meaning and 0.2 location. HDBSCAN clusters that, and clusters whose semantic centroids sit within 0.4 of each other are merged back together, since broad themes tend to get over-split.

Scored against 202 pull requests our team grouped by hand into 104 clusters:

The winning configuration also scores 0.855 B-Cubed F1 and 0.646 pairwise F1. Two results are worth keeping from the rest of the sweep. Structure is worth about 0.16 ARI on its own: where a pull request lands says something its description does not. And asking a model to tidy the clusters afterwards made them worse, with essentially every edit disagreeing with the human grouping.

HDBSCAN wins because it is allowed to decline. Loners get labeled noise instead of forced into a group, which is what the hand-labeled set does too: most pull requests are legitimate one-offs and belong nowhere. KMeans and the model have to place every pull request, and that is what costs them.

Then the agent reads. By that point it is looking at a handful of grouped, in-scope pull requests per rule rather than a repository's full history. The model is spent on the judgment, which rule no longer holds and what it should say instead, not on the search.

Keeping it usable

Three properties matter more than the analysis, because they decide whether the output gets read:

Bounded. You cap how many pull requests it can have open at once. At the cap it stops until you clear some. The failure mode of a tool like this is not bad suggestions, it is fifty good ones on a Monday morning.

Scoped. Run it on one repository or all of them. Repositories without write access are visibly excluded rather than silently skipped.

Separable. Every pull request gets a title prefix and a branch prefix you choose, so its output is filterable and routable through your branch rules from the first run.

Runs happen on demand until you trust the output, then on a schedule.

The docs cover setup and how to start a run.

What it will not do

Skills Maintainer keeps instruction files honest. It cannot invent them. Point it at a four-line CLAUDE.md and it will have almost nothing to say, correctly.

Maintenance is worth the most where enforcement already happens. The AI Coding Guidelines reviewer applies your rules on every pull request, and enforcement without maintenance is the worst combination available, since it applies yesterday's rules at today's speed.

Start small

Turn it on for one repository you know well. Set the open pull request cap to three. Scan on demand.

Read that first batch carefully. It says less about the agent than about how far your written standards have drifted from your real ones.

Then widen the scope, put it on a schedule, and stop thinking about it.

---

Skills Maintainer is available now. See the documentation for setup.