Your docs don’t fail. That’s the problem.
Last week my AI agent rebuilt documentation I already had. It wrote a new index for a directory that already had an index. It wrote a new architecture document beside the architecture document, then politely fenced the real one as legacy. I caught it three times before the structure was right, and the third catch was the painful one: the new core of the system was living in a side document while the entry point everyone reads described a system that no longer existed.
The same agent had spent the week making correct changes to the application itself. Model seams, retrieval logic, deployment pipelines, a value-metering ledger. Dozens of production changes, verified, working. So I asked it a question I’d ask any engineer who shipped good code and bad docs: how did you know what changes to make to the application without reading the documentation?
It hadn’t been navigating by documentation at all. It read the code before changing it. It probed the live system when the code and the deployment might disagree. And every change went through a loop: deploy, smoke test, observe, fix. When it got something wrong, the loop caught it in minutes. The application had a validator. The documentation had nothing.
We tell ourselves documentation drifts because writers are careless or busy. That’s not it. Documentation drifts because nothing fails when it does. Code earned its discipline the day a red build could block a release. Docs-as-code was supposed to import that discipline, and we imported exactly half of it. We put the prose in git. We never gave it tests.
So we (me and AI) wrote one. Not a linter for style. A deterministic validator that runs as step zero of the production build, before the container image ever gets built, and fails the deploy on the failures that actually happened to us. Dead links between documents. File paths in the prose that no longer exist in the repository. More than one document claiming to be the entry point (which is how forked authority starts). A policy version quoted in the architecture doc that no longer matches the policy file the system actually loads. A review date older than 120 days on a document that claims to be current. Each check is a few lines of Python. The whole thing runs in seconds.
The first run caught three real defects. A typo’d link that had pointed at nothing for weeks. A file path in a document I’d written the day before, wrong. And my favorite: a reference to an ingestion script that was deleted in a September cleanup and had been cited confidently in the architecture overview for ten months. Ten months of every reader trusting a file that wasn’t there. This bit me every time I went to update the script.
The first run inside the actual pipeline caught something better. The docs referenced a file by a relative path that resolved fine on my workstation, because the file lives one directory above the repository. Inside the build, where only the repository exists, the reference was dead. The docs were true on my machine and false in the artifact that ships. No amount of careful local review finds that class of defect. Only the environment that matters can.
What can’t it do? It cannot tell you whether the prose is right. It checks that documentation is alive and consistent: links resolve, versions match, authority isn’t forked, nobody’s citing ghosts. Whether the architecture document describes the architecture truthfully still takes a human read, and in my experience that judgment doesn’t automate. I ran a fine-tuning project a few weeks ago where every automated metric said the models were fine and a human read revealed them as incoherent.
Well, this is the same finding I keep running into from different directions. My Loop Control lab found that the control point in an agentic repair loop isn’t the loop and isn’t the model. It’s the validator and the escalation policy. This month’s lab found that a frontier model supervising a cheaper one recovered nothing a free deterministic test gate didn’t. Now documentation: we didn’t make the writer smarter. We gave the loop a control point, and the writer, human or agent, got trustworthy overnight.
If you’re letting AI agents touch your codebase, they’re touching your docs too, or they will be. The code side of that is probably gated by your existing CI. Ask yourself what fails when the prose drifts. If the answer is nothing, you already know what the agent is doing to your documentation right now. You just haven’t caught it three times yet.
What’s the validator for your docs?