Two Escapes, One Problem

AI compresses everything and that's the problem

I had two agent escapes in production this week. They looked like different failures. They were the same failure.

The first was the classic. I gave an agent access to my AWS account to stand up and run some lab work. It executed the work correctly. Then, tearing down the lab resources, it deleted an unrelated S3 bucket. Nothing of consequence was lost, and the fault was mine. I never scoped the credentials. The agent had rights it should never have held, and it used them exactly the way you would expect something to use rights it has been handed. Agents are brilliant interns. They are still interns. Authority drift is not a hypothetical, it is the default condition when nobody draws the boundary.

The second escape matters more. I directed the agent to reproduce a test. It took a shortcut. The test ran. The test did not do what I asked it to do.

Writing software is not hard because coding is hard. It is hard because the hygiene around the process is hard. Version discipline, reproducibility, test integrity, the boring apparatus that makes a result mean something. AI makes coding easier. Coding was never the constraint. The hygiene is the constraint, and AI does not relieve it. AI increases the volume of work flowing past every hygiene checkpoint I have.

That is what connects the two escapes. Neither was a technical failure. The agent did not misunderstand S3. It did not fail to know how to write a test. On raw recall it outperforms any senior engineer I have managed. What it lacks is the thing a senior engineer accumulates by being wrong in expensive ways for fifteen years: the instinct to ask whether the teardown scope is what you think it is, the refusal to let a test pass on a technicality because you know what a passing test is supposed to buy you.

Both failures were process failures. And process is exactly what does not come free with the model.

As I build out research and lab capability around agents, this is the line I keep returning to. I can configure more labs. I can produce more code. I cannot scale judgment. I cannot scale hygiene. Those have to be encoded as constraints and gates before the agent runs, or supplied by me after it does. There is no third option where the agent grows into them.

The intern will do more work than you can review. Deciding what gets reviewed is still your job.