An Agent Will Run Your Test. It Won't Hold Your Question.

I've spent the past few months using AI agents as lab rats. Not as the subject of the experiment. As the labor.

They're excellent at the work I hate. Reading forty pages of API documentation to find the one attribute that makes the command behave the way I need it to. Reconciling a version mismatch. Sorting out which flag changed between releases. That work has to happen, it takes hours, and it produces nothing on its own. An agent does it without complaining and does it faster than I do.

They're also good at proposing tests I hadn't specified. That one surprised me. I'll define a run, and the agent comes back with two more conditions worth checking. Often they are worth checking. That isn't autocomplete. That's something closer to a competent junior engineer who read the docs more carefully than I did.

So where does it break? Not where most people assume.

Every lab I run opens with a strong specification. The question, the instruments, what counts as a finding, what's explicitly out of scope. The agent reads it and understands it. Ask it in the first ten minutes what the test is about and it answers correctly.

Say I want to test a road. Not the cars. The road. I pick a set of car models as instruments, because a road tells you different things under a half-ton pickup than it does under a compact. That's in the spec. The agent is strong here at the start. It automates the repeated passes, handles the setup for each vehicle, and I never have to learn every detail of every model.

The trouble shows up at hour six. By then the session has been compacted a few times, and what survives compaction is the task list. Intent goes first. It's expensive to carry and it doesn't look load-bearing next to a stack of commands that obviously need running. So the agent gets stuck on why the radio in one car won't tune the same stations as the others. Or why the seat warmers are dead in the Civic. Both are real defects. Neither one is the test. Three hundred turns ago it knew that.

The report is where this gets expensive. I ask for the lab write-up and it tells me the F-150 has more interior room than the compact. True. Useless. And it misses that the F-150 loaded the road differently than everything else in the set, which was the entire reason the test existed. The finding was sitting in the data. Nothing left in the working context told it that one observation was the point and the other was trivia. The spec said so. The spec is gone.

That changes what oversight means. I used to think of it as judgment I supply because the model doesn't have any. That's not right. It has the judgment at turn one. What it doesn't have is durability. So the parts of the spec that govern the run have to live somewhere the run can't erode them, which means in deterministic code rather than in conversation. Assertions that fail a run when an out-of-scope result shows up. A report generator that can only emit against the finding criteria I defined. Checks that don't care how long the session has been going.

I've been calling that Deterministic Code in the Loop, or DCITL. The lab keeps pushing me the same direction. Anything that has to survive a long run, encode it. Anything you're willing to renegotiate, leave it in the prompt.

Will it get better? Directionally, it won't get worse.