The Joist, the GPT, and Why Good Plans Fail
When the Flaw Isn’t in the Plan — It’s in What the Plan Forgot to Plan For
Years ago, the staircase to my basement collapsed.
I decided, “I can rebuild this.”
No YouTube tutorials (this was before YouTube), no construction manual — just me, a hammer, and misplaced confidence.
I tore out the remains, cut new stringers, and started attaching them… until I reached the joist.
That’s when I realized I had no idea how to attach the staircase to the floor above.
The result? A shaky set of stairs that eventually got rebuilt by a friend who knew what they were doing.
In that case, getting it wrong mattered — bad stairs can hurt people.
Fast forward to this month.
I was building a custom GPT trained on my public work — my blog, my Substack, and other sources.
The goal: seamless retrieval. Someone asks a question in plain language, and it surfaces the most relevant insight, no matter where I had written it.
I told the GPT in its system prompt:
“Always search all my datasets.”
It didn’t.
Sometimes it pulled only from my blog and ignored my Substack entirely — even when I had relevant posts in both.
The reason: that “always search everywhere” instruction doesn’t change how the Retrieval-Augmented Generation (RAG) pipeline builds the retrieval query.
If the query isn’t explicitly pointed to all sources, the model never even sees the missing data.
The Problem I Didn’t Know to Look For
Here’s the thing — I didn’t even know to read about retrieval routing.
This wasn’t a “known unknown” I could research in advance.
It was an unknown unknown: a failure mode I didn’t realize existed until I tripped over it.
And if this had been a production system, I’d have called in an expert from the start — just like I eventually did with the staircase.
But that wasn’t the goal here.
This was low-stakes, hands-on exploration designed to surface hidden execution constraints I wouldn’t uncover from documentation or a whiteboard session.
By going down this path myself, I uncovered a blind spot that now informs how I approach strategy — even if I’m not the one who will ultimately fix it.
It’s the same as standing in the basement with a cut stringer, suddenly realizing:
“Oh… attaching this to the joist is a separate problem.”
Once you see that constraint, you never design a staircase (or a retrieval pipeline) the same way again.
Governance and Architecture Matter
In an enterprise context, this wasn’t just a technical glitch — it was a governance gap.
The plan was “use all data,” but the architecture didn’t enforce that rule.
This is exactly the kind of oversight an Architecture Review Board (ARB) should catch.
An ARB’s role is to ensure that systems align with governance, manage risk, and establish robust frameworks — including things like prompt testing, retrieval validation, and output tracing.
If a RAG pipeline is silently skipping half your datasets, that’s a data governance failure.
It undermines trust in the system’s outputs, even if the system “looks” like it’s following the plan.
From Joist to Leadership Practice
The staircase had its joist, and my GPT had its own: the underlying RAG pipeline.
My hammer was like my confidence in the system prompt.
Both were insufficient because they didn’t account for a fundamental structural constraint — the joist in one case, the retrieval query logic in the other.
In both cases, the plan failed at a layer beneath where I was working.
The job of a CTO or IT leader isn’t just to draw the blueprint — it’s to make sure someone is testing the structural connections before the whole project is framed.
Low-stakes Proofs of Concept (PoCs) are how you surface the joists before they cause expensive, embarrassing, or even dangerous failures in production.
They create the space for builders to find — and fix — the hidden constraints that high-level planning will always miss.
Strategic takeaway for technology leaders:
Your org’s “always” and “never” rules — in architecture, AI strategy, or go-to-market — can fail silently between the plan and the execution layer.
Knowing that risk exists isn’t enough. You need governance processes, architectural reviews, and deliberate PoCs that permit your teams to look for the joists.
Because in strategy, as in construction, the most dangerous flaw is the one that was invisible on the blueprint — until you try to connect it to the joist.