The Model Learned My Voice, Not My Judgment

Two rounds of fine-tuning lost to one paragraph of standing positions. So where does expert judgment actually belong.

One of the projects I’ve been working on over the past few weeks started with a simple question: can I fine-tune a model to reason more like I do?

The obvious approach was to train on my corpus and validate against my 4+1 AI Infrastructure Assessments. I built the validator first, before running a single training job. It included held-out vendor placement probes, framework discipline checks, honesty tests, advisory head-to-heads against a baseline model, and human review of the output. If I was going to claim the model reasoned like me, I wanted an instrument that could catch it when it didn’t. Then I ran multiple fine-tuning rounds.

The results weren’t what I expected.

The model learned bounded behaviors. Forty refusal exemplars measurably taught it to decline questions it couldn’t answer instead of inventing benchmark winners. It got better at reproducing my voice: sentence rhythm, vocabulary, the way I frame an argument. If you read the output cold, you might believe it was mine.

The validator earned its keep before the first run finished. Training loss said every round went fine. The gate said one artifact fabricated performance results and another collapsed into verbatim repetition loops on open-ended questions. That failure mode is invisible to averaged metrics and obvious to any reader. If I had built the model first and the instrument second, I would have shipped something broken.

However, the model didn’t get materially better at applying my judgment. It could sound like me while placing a vendor in a way I never would. In head-to-head advisory questions judged against my real published answers, the untuned base model beat the fine-tuned versions 12-1 and 13-0. And in the strongest control I ran, a single paragraph of my standing positions written into the system prompt outperformed both rounds of fine-tuning on every measured family. All the effort of curating 563 training pairs, running jobs, and evaluating checkpoints lost to a block of instructions that took ten minutes to write.

The full lab writeup, with the gate design, scores, and what each round actually taught, is published at Layer2C Labs: Put the judgment in the constraints, not the weights. This post is the shorter story of how that result changed my thinking.

At the same time, I’ve been using Claude Code on a different problem: combining two very different evidence sources. On one side, quantitative public research from HyperFRAME Research. On the other, fifteen years of podcast interviews and practitioner conversations. That’s the kind of material that never makes it into a survey, because it lives in how a vice president of infrastructure describes a migration going sideways at minute forty-two of a recording.

The goal isn’t to have AI tell me what’s interesting. Models are happy to declare everything interesting. The goal is to have the system surface candidate patterns and then pressure-test them against an explicit judgment instrument — something written down, something I can argue with. That’s where my work on the Decision Authority Placement Model and Deterministic Code in the Loop keeps showing up. Both frameworks exist as artifacts outside any model. They constrain what the AI proposes and give me a fixed standard for evaluating what comes back.

So the more I run these experiments, the less I think expert judgment belongs in model weights at all.

Weights are good at bounded behavior. Tone, format, refusal discipline, domain vocabulary — the things you can demonstrate by example, weights absorb. Judgment is different. Judgment seems to scale through explicit constraints, retrieval, validators, and human review. It wants to live where you can read it, version it, and correct it.

Software engineering settled this argument a long time ago. We don’t train developers to memorize every coding standard and then hope the standards survive in their heads. We build linters and quality gates. The standard lives in a config file that anyone can inspect, and enforcement happens deterministically at the boundary, not probabilistically in someone’s recall. Nobody argues that a linter is less sophisticated than a developer who memorized the style guide. It’s more reliable precisely because it’s dumber and explicit.

I’m starting to think enterprise AI needs the same structure. The instinct right now is to push everything into the model. Fine-tune on your documents, your decisions, your tribal knowledge, and hope the judgment comes along for the ride. My experiment suggests the voice comes along. The judgment stays home.

The interesting part of this work isn’t whether I can build “a Keith model.” It’s whether I can build a system that lets expert judgment accumulate, remain inspectable, and improve over time instead of disappearing into opaque weights, where the only way to audit it is to run it and hope.

That question feels much more important than the one I started with.

The bench work behind this post is Lab 006 at labs.layer2c.com: the 70-probe gate, both training rounds, the constraint-block control, and the production validation. Self-funded, no sponsor.