The Xeon Exit Is a Batch Lane, Not a Serving Lane
Intel and Google market the C4 virtual machine as GPU-comparable for small and mid-size model inference. The chart behind that claim compares the new Xeon to the old Xeon. I bought the comparison the
I have a custom model that no cloud model garden will host. It is a LoRA-tuned Gemma 4 26B mixture-of-experts, merged into open weights. That is the strongest position you can hold at the reasoning layer, because no garden hosts it and therefore no garden can withdraw it. The catch is that holding the position means you have to serve the thing yourself, and to serve it I needed compute I could actually get my hands on.
I could not get NVIDIA GPUs. Requests I filed on June 29 were still unusable two weeks later. A fresh request went in and came back denied in about two seconds. That is the real starting condition of this lab. The CPU lane was not a preference or a thought experiment. It was the one self-serve door that opened, so I walked through it and measured what was on the other side.
The claim I walked in to test is the one Intel and Google make about the C4 virtual machine: GPU-comparable performance for small and mid-size model inference. It is a clean line, and it lands well in a keynote. The chart underneath it compares the new Xeon to the old Xeon, which is a real improvement and also not the comparison the claim implies. I bought the comparison the chart skips. I put my model on the largest Granite Rapids shape the self-serve quota path would give me and measured it across two serving stacks, two prompt shapes, and concurrency from one stream to sixteen.
Twenty-two configurations. None of them served an interactive workload. Not the best case, not a corner case, none of them.
The interactive bar I set was deliberately ordinary: first token in under five seconds, and fifteen tokens per second per stream, both at the same time. The two strongest stacks failed that bar in opposite directions, which is the part worth sitting with. llama.cpp reads back at a genuinely usable pace, north of twenty tokens per second, but it takes thirty-six seconds to produce the first token on a real six-thousand-token retrieval prompt. vLLM gets the first token out in about six seconds and then decodes at roughly half llama.cpp’s rate. One stack makes you wait to start and then reads well. The other starts quickly and then crawls. There was no configuration anywhere in the grid where a user waits an acceptable time and then reads at an acceptable pace. The frontier is empty, and I reported the near-misses honestly because the emptiness is only credible if you can see how close the closest cells came.
The reason matters more than the numbers, because the numbers will move and the reason tells you when. Advanced Matrix Extensions are the load-bearing feature of the entire CPU-inference pitch. AMX is what is supposed to make a Xeon competitive for this work. On this model generation, neither stack ran the mixture-of-experts layers on it. llama.cpp repacked two hundred and six attention and dense tensors onto the matrix units and zero expert tensors. vLLM’s fused expert kernel rejects Gemma 4’s activation function outright and will not run those experts on the AMX path at all. Both stacks then converge on the same throughput ceiling, which is exactly what you would expect if the expert math fell back to the vector units in both cases. What the marketing calls AMX inference is, on the flagship open mixture-of-experts today, AMX prefill and attention stapled to vector-unit decode. That is a software seam, not a property of the silicon, and it is also the present-tense truth of a present-tense marketing claim.
So is the claim real? For an interactive serving tier on a custom model, no, not today. But the lab did not come back with nothing. It came back with a workload classifier, which is more useful than the yes-or-no I went in looking for.
Where the lane earns its keep is latency-tolerant batch on a model no garden will host. Corpus sweeps, evaluation runs, classification passes, overnight generation. Work where nobody is watching a cursor blink. At sustained saturation, counting compute only, the lane produced custom-model tokens at ten to fourteen dollars per million. And it came with an availability property the GPU lane does not offer anywhere: the CPU quota bumps auto-approved in four to six minutes, on the same project and the same day that the GPU request was denied in two seconds. Same automation. Same account. One door negotiates with you in minutes and the other refuses you before you have finished reading the confirmation dialog.
That asymmetry is the durable finding, and it is the sentence I expect to be repeating in briefings for a year. The performance numbers will move with the next software release. The doors are policy. A CPU is a negotiation, a TPU is a capacity lottery, and an NVIDIA GPU is a wall. Substrate choice inside a cloud you have already committed to is an availability decision before it is ever a performance decision, and most architecture conversations still treat it the other way around.
There is a door I have not measured yet. The TPU lane took no quota fight at all and granted a chip in minutes while the GPU requests sat denied. That is the substrate I am building the next lab around right now. Whether a TPU can actually serve this specific custom model is an open question, and open is exactly what I am going to keep calling it until the bench answers. The scaffolding is going up. The measurement comes next.
In the meantime, do not take my summary for it. I published the full lab as machine-readable detail on purpose, so you can point your own model at the real artifact instead of at my account of it. Paste this and interrogate the numbers directly:
Read the Layer2C Lab 007 detail at https://labs.layer2c.com/downloads/gemma4-xeon-inference-lab-detail.md. I am evaluating a Xeon 6 C4 shape to serve a custom model. Using only the measured results, answer three things. First, which of my workloads does this lane actually support and which does it disqualify. Second, what would have to change for the interactive verdict to flip, and is that change silicon or software. Third, what did the lab not measure that I would still need to test before I design around it. Cite the specific layer and finding for every claim.
If your model tells you something my summary did not, that is the point.
Disclosure: Intel is a client of this practice. This lab was not commissioned, funded, or previewed by Intel or any other vendor, and the ruling was formed from the bench alone. It ran on thirty-seven dollars of my own cloud spend.