Bigger Context Windows Won't Save You
The problem was never the space.
The Architecture Series · 9 minute read · By Ed | Founder & CEO, DealiOS
If you've ever re-explained the same thing to the same AI tool for the third time in the same week, congratulations: you've discovered the context problem.
And no, a bigger context window isn't the answer. That's like solving a hoarding problem by buying a bigger house. You still can't find anything. You just have more square footage to lose it in.
The AI industry is in a context window arms race. Gemini supports 2 million tokens. Llama 4 Scout claims 10 million. Claude, GPT, and half a dozen open-source models are all racing past the million-token mark. And the marketing pitch is always the same: more context means better answers.
It doesn't. More context means more noise. And noise at scale is not signal. It's just louder noise.
A bigger context window is like solving a hoarding problem by buying a bigger house. You still can't find anything.
The Re-Explanation Tax
Every time you re-explain something to an AI tool, you're paying a tax. Not in dollars (though you're paying that too, in tokens). In time. In patience. In the slow erosion of trust that happens when a tool that's supposed to make you faster keeps forgetting who you are.
I call this the re-explanation tax, and it's one of the most expensive hidden costs in enterprise AI today.
Here's how it plays out. Monday, you tell the AI about your client. Their industry, their pain points, the three proposals you've already sent. You get a great draft. Tuesday, new conversation. The AI has no idea who the client is. You re-explain. Wednesday, same thing. By Thursday, you've spent more time teaching the AI than doing the actual work. By Friday, you're back in Google Docs because at least a blank page doesn't pretend to know you.
Research from Chroma's 2025 context rot study tested 18 frontier models, including GPT-4.1, Claude, and Gemini 2.5. Every single model showed performance degradation as input length increased. Not some of them. Not most of them. All of them. The study described this as an architectural property of transformer-based attention, not a capability gap that training solves.
The re-explanation tax isn't a user problem. It's an architecture problem. And the solution isn't more tokens. It's more structure.
Why "More Tokens" Is the Wrong Frame
Let's talk about what a context window actually is. It's RAM. Short-term memory. The amount of text a model can hold in its head during a single conversation.
When Gemini advertises 2 million tokens, they're saying: "We can hold approximately 1.5 million words in a single conversation." That's impressive. It's also irrelevant to the problem most people actually have.
Because the problem isn't capacity. The problem is organization.
Giving an AI a bigger context window is like giving a disorganized person a bigger desk. They don't become more organized. They've become expansively disorganized. That's worse. Now they can lose things at scale.
Liu et al.'s landmark 2024 study in the Transactions of the Association for Computational Linguistics established what researchers call the "lost in the middle" phenomenon. Models perform best when relevant information sits at the beginning or end of the context. Performance drops by over 30% when the relevant content lands in the middle. The research team tested this across multiple models and tasks. The finding was consistent: models don't attend to all parts of their context equally.
And this was published in 2024. Despite two years of architectural improvements, Chroma's 2025 follow-up confirmed the pattern persists across every frontier model. The U-shaped attention curve is a feature of transformer architecture, not a bug that gets patched with more training data.
So when someone tells you their AI tool has a million-token context window, ask the follow-up question: what happens to the information at token 500,000? The answer, backed by peer-reviewed research, is: the model probably ignores it.
A Goldfish With a PhD
Here's a metaphor that's more accurate than any benchmarking paper.
Current AI tools are like a goldfish with a PhD. Brilliant within the conversation. Absolutely no memory of what happened yesterday. And if you dump the entire ocean into their bowl, they don't become an oceanographer. They just can't find the castle anymore.
The industry response to this problem has been predictably literal: make the bowl bigger. Gemini's 2-million-token window. Llama's 10-million-token claim. Every provider racing to advertise the largest number.
But size without structure is just storage. And storage is cheap. What's expensive is retrieval. What's really expensive is understanding which pieces of information are related to each other and why.
When a sales rep asks an AI to draft a proposal, the AI doesn't need every email the company has ever sent. It needs to know that this client is in manufacturing, that their last proposal included a 15% discount that was approved by the VP of Sales, that the discount policy changed last quarter, and that the new policy requires a different approval chain for discounts above 10%.
That's not a context window problem. That's a relationship problem. And relationships require architecture, not raw capacity.
Context isn't data. Context is the relationships between data. A million tokens of disconnected information is still disconnected.
Context as a Relationship Graph
Here's where the conversation shifts from what's broken to what works.
Context isn't data. Context is the relationships between data. A million tokens of disconnected information is still disconnected. It's a filing cabinet where every document is labeled "Important" and nothing is cross-referenced.
Think about how context actually works in a real business interaction. A proposal references pricing. Pricing references a discount policy. The discount policy references client tier. Client tier references the contract. The contract references the account. The account references every previous interaction you've ever had.
That's not a list. It's a graph. And the value isn't in any single node. It's in the edges, the connections between them that tell you why a piece of data matters in this specific moment for this specific task.
When I architected DealiOS, the data model was designed to hold these relationships from the first database migration. Proposals know about contracts. Contracts know about pricing. Pricing knows about calculator outputs. Calculator outputs know about discount policies. Everything references the same source of truth, and every reference is bidirectional.
This means when the AI assembles context for a proposal, it doesn't stuff 500,000 tokens of everything into the window and hope the attention mechanism finds the relevant bits. It follows the graph. It starts with the account, walks the relationships to the relevant proposals, contracts, pricing history, and engagement data, and assembles a focused context package that contains exactly what's needed. Nothing more.
The result isn't just faster. It's more accurate. Because the AI isn't scanning for needles in a haystack. It's following a map.
The Five-Layer Context Assembly
Relationship-aware context doesn't assemble itself. It requires a deliberate architecture for deciding what goes in, in what order, and at what priority.
In a well-architected system, prompt assembly follows layers. Think of it like a briefing for a new team member. You don't hand them every document the company has ever produced. You give them the relevant information in a structured order: who they are, what they're supposed to do, who the client is, what's already happened, and what the user is asking for right now.
Layer one is the system identity. Who is the AI in this context? What are its capabilities and constraints? This is the foundation that every other layer builds on.
Layer two is the domain expertise: the feature-specific module and industry knowledge combined. For a proposal, that's the prompt engineering that tells the AI how proposals work, what makes them effective, what to avoid, plus the industry context (SaaS pricing models, cloud infrastructure conventions, etc.). For a contract, it's different. For a presentation, different again. The domain layer changes based on what you're building and what industry you're building it for. (The companion paper "Five Layers Deep" breaks this into separate feature and industry layers for a more granular technical view.)
Layer three is workspace context. Brand tokens, company information, customer-specific calibration. This is the "who is the customer" layer.
Layer four is the specific item context. The account, the deal, the previous proposals, the engagement history. This is where the relationship graph pays off, because the system follows connections rather than keyword-matching across a corpus.
Layer five is the user instruction. What did the human actually ask for?
Five layers. Ordered by priority. Each one scoped to exactly what's relevant. The total token count might be 15,000 or 50,000, depending on complexity. But every token earns its place because the architecture knows why it's there.
Compare that to the alternative: dump everything into a million-token window and let the attention mechanism sort it out. One approach is engineering. The other is hope. I've shipped products on both. Hope doesn't scale.
What Changes When Context Is Structured
When context is a relationship graph instead of a token dump, three things change immediately.
First, updates propagate forward. Change a pricing rule once, and every future document the AI generates reflects the change. Not because someone remembered to update a template. Because the architecture assembles context from the source of truth on every call. The AI doesn't need to be told that pricing changed. It reads the current pricing every time it assembles context.
Second, context stays fresh without re-explanation. The rep doesn't re-explain the client on Tuesday because the system already knows the client. Not because it memorized Monday's conversation, but because the client data lives in the data model and the relationship graph brings it into context automatically when it's relevant.
Third, the AI's output improves because its input is curated. Fewer irrelevant tokens means less noise. Less noise means the attention mechanism works better on the tokens that matter. > The goal isn't to make the window bigger. The goal is to make the window unnecessary.
The "lost in the middle" problem largely disappears when the middle is eliminated. If you only send 20,000 tokens and every one of them is relevant, there's no middle to get lost in.
This is the architectural insight that the context window arms race misses entirely. The goal isn't to make the window bigger. The goal is to make the window unnecessary.
The Cache Layer Nobody Talks About
There's a second dimension to context that doesn't get enough attention: economics.
Every token you send to an AI model costs money. At current pricing, a million-token request to a frontier model can cost $5 to $25 per call. Run that a few hundred times a day across a sales team, and you're looking at infrastructure costs that would make a CFO's eye twitch.
Structured context is also an economic architecture. When you assemble focused context packages of 15,000 to 50,000 tokens instead of dumping 500,000 tokens per request, you're not just getting better results. You're reducing cost by an order of magnitude.
And then there's caching. When context is structured, you can cache the layers that don't change between requests. The system identity doesn't change. The domain expertise doesn't change. The workspace context changes rarely. The item context changes when the data changes. Only the user instruction is truly unique per request.
Cache the stable layers. Reassemble only what changed. The result is faster response times, lower costs, and context that's always current. It's the boring infrastructure that makes the whole system economically viable at scale. Nobody puts "we cache prompt layers" on a pitch deck, but it's the difference between a platform that costs $3 per user per month in AI infrastructure and one that costs $300.
Why This Matters Now
The timing of this conversation matters because the industry is at an inflection point.
For the last two years, the answer to every AI context problem has been "bigger windows." And the windows got bigger. Models went from 8,000 tokens to 128,000 to a million to 10 million. The marketing got louder. The benchmarks got more impressive.
But the problems didn't get solved. Enterprise users are still re-explaining things. Proposals are still missing context. AI tools are still generating confident-sounding output that misses the point because the relevant information was buried at token 400,000 and the model's attention mechanism skipped right over it.
The next phase of enterprise AI won't be won by whoever has the biggest context window. It will be won by whoever builds the best context architecture. The boring, structural, database-schema-level work of making data relate to other data in ways that an AI can follow.
Context isn't a window. It's a relationship graph. And the companies that figure this out will build AI products that feel intelligent, not because the model is smarter, but because the architecture gives the model the right information at the right time.
Which, when you think about it, is how intelligence actually works. It's not about how much you can hold in your head at once. It's about knowing what to pay attention to.
The takeaway: Context is a relationship graph, not a token dump: the next phase of enterprise AI will be won by whoever builds the best context architecture, not whoever has the biggest context window.
References
1. Liu, N.F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2024). "Lost in the Middle: How Language Models Use Long Contexts." Transactions of the Association for Computational Linguistics, 12, 157-173.
2. Chroma Research (2025). "Context Rot: How Increasing Input Tokens Impacts LLM Performance." 18 frontier models tested including GPT-4.1, Claude, Gemini 2.5.
3. Anthropic (2026). Claude Opus 4.6 and Sonnet 4.6: 1M token context window GA at standard pricing, March 2026.
4. Google (2026). Gemini 3 Pro: 2M token context window with native multimodal processing.
5. Meta (2026). Llama 4 Scout: 10M token context window, mixture of experts architecture.
This paper is part of The Architecture Series by Ed, Founder & CEO of DealiOS. For the weekly series and more technical deep-dives, visit dealios.app/library.
See how these principles come together. Explore the platform →