Stop Feeding AI Sentence Fragments
Most retrieval systems chop documents into pieces and hope AI reassembles them.
The Architecture Series · 8 minute read · By Ed | Founder & CEO, DealiOS
The default RAG architecture goes like this: take your documents, chop them into 500-token chunks, embed each chunk, store the embeddings, and when the AI needs context, retrieve the top-k most similar chunks.
It works beautifully for Q&A. "What's our refund policy?" Retrieve the relevant chunk. Answer the question. Done.
Now try generating a 15-page sales proposal that references a past case study. The chunk that scores highest is 500 tokens from the middle of the case study. The client name is in a different chunk. The outcome is in another. The emotional arc that makes the reference compelling is spread across four chunks that the system doesn't know are related.
You optimized for precision. You got fragments. And fragments don't tell stories.
Retrieving chunks from a case study is like reading the punchline without the setup. It's technically the important part. It's also completely useless.
This paper is about a specific architectural bet: that for document generation tasks, retrieving whole documents produces better output than retrieving precise fragments. It's a bet that runs counter to the prevailing RAG orthodoxy, and it's one we made deliberately after watching chunk-based retrieval fail in production for months.
The Fragment Problem
The standard RAG tutorial tells you to chunk your documents into 500-token pieces, generate embeddings for each chunk, and retrieve the most semantically similar chunks at query time. The logic is sound: smaller chunks produce more precise embeddings, which produce more relevant retrieval. For Q&A, this is exactly right.
For document generation, it falls apart.
A case study is not a bag of facts. It's a narrative. The client had a problem. They tried solutions that didn't work. They found the right approach. The results followed. That arc is what makes the reference compelling in a proposal. A 500-token chunk from the results section has the metric but not the client's name. It has the outcome but not the problem it solved. It has the punchline but not the setup.
Retrieving chunks from a case study is like reading the punchline without the setup. It's technically the important part. It's also completely useless without everything that came before it.
This is the fragment problem: chunk-based retrieval optimizes for relevance of individual pieces while destroying the relationships between them. The embedding for "achieved 34% reduction in cycle time" is semantically similar to a query about efficiency improvements. But without the context of who achieved it, in what industry, under what conditions, and through what approach, that fragment is noise dressed as signal.
Chunk-based RAG is like if you asked a friend to recommend a restaurant and they just said "the risotto." Great. Where? What's it called? What's the vibe? "The risotto." Thank you. Very helpful.
The problem compounds when the AI needs to reference multiple source documents. A proposal might draw from a case study, a methodology document, and a pricing framework. Chunk-based retrieval treats these as interchangeable fragments in a ranked list. The AI receives a jumble of decontextualized pieces from different documents and has to reconstruct coherent references from the wreckage. Sometimes it succeeds. Often it hallucinates the connections that the chunking destroyed.
The Architecture Shift
We tried chunk-based retrieval for months. The results were consistent: factual Q&A worked well, document generation produced outputs that felt stitched together from disconnected parts. The AI would reference a case study but get the client name wrong because the name was in a different chunk. It would cite a methodology but describe a different methodology's process because the chunks were semantically similar but from different documents.
The fix was straightforward in concept and significant in implementation: store documents whole with a single embedding per document, and retrieve entire documents instead of fragments.
This inverts the standard tradeoff. Chunk-based retrieval gives you high precision on fragment relevance at the cost of narrative coherence. Document-level retrieval gives you narrative coherence at the cost of precision. For document generation, coherence wins.
The retrieval pipeline returns the top 3 full documents based on semantic similarity, not the top 20 fragments. The AI receives complete case studies, complete methodology descriptions, complete pricing frameworks. It can reference them accurately because it has the full context to reference.
The chunker still exists. It handles token estimation so the system knows whether the retrieved documents fit within the context window. But it's no longer the retrieval unit. The document is.
Behind this sits a three-tier fallback system. First: pgvector cosine similarity against document-level embeddings. This is the primary retrieval path, matching the query against the semantic meaning of complete documents. Second: PostgreSQL full-text search. If vector similarity doesn't surface the right documents (which happens with highly specific terminology that embeddings generalize away), keyword matching catches what vectors miss. Third: most recent documents, pure recency. If neither semantic nor keyword search finds relevant context, the system retrieves whatever the user uploaded or generated most recently.
Something always comes back. The system never generates in the dark.
For document generation, coherence wins over precision. You need the right document, not the right sentence.
When Chunks Win, When Documents Win
This is not an argument that chunk-based RAG is wrong. It's an argument that chunk-based RAG is wrong for a specific category of tasks.
Chunks win for factual Q&A: "What's our refund policy?" The answer lives in one paragraph. You don't need the surrounding document. Chunks win for policy lookup: "What's the SLA for tier-2 support?" The answer is a specific clause. Chunks win for data retrieval: "What was our Q3 revenue?" The answer is a number in a specific location.
Documents win for proposal generation: the AI needs to understand the narrative arc of a case study to reference it compellingly. Documents win for methodology explanations: the AI needs the full process to describe it accurately. Documents win for anything where the source material has an internal structure that chunk boundaries destroy.
Match retrieval architecture to generation task. Chunk-level for lookup. Document-level for creation.
The principle: match retrieval architecture to generation task. Chunk-level for lookup. Document-level for creation.
The industry pretends one size fits all. Every RAG tutorial teaches the same chunking-first approach regardless of the downstream task. The result: systems that are excellent at answering questions and mediocre at generating documents. Nobody notices because most RAG applications are Q&A systems. The moment you try to generate long-form content from retrieved context, the fragment problem hits hard.
The reason the industry defaults to chunks isn't because chunks are better. It's because chunks are cheaper. A 500-token chunk produces one embedding. A 10,000-token document requires either one embedding (which loses detail) or 20 embeddings (which multiplies storage and compute). Document-level retrieval with single embeddings accepts the precision loss as a reasonable trade for narrative coherence. That's a values decision masquerading as a technical one.
Build Routes and Latency Tradeoffs
Not every request to the system uses the same context strategy, and pretending otherwise is how you get unnecessarily slow responses for tasks that don't need retrieval at all.
Build routes handle document generation. Chat routes handle conversational Q&A. They look identical to the user but they behave differently at the retrieval layer.
Build routes skip vector search entirely when the system already has workspace-level context. If the user's workspace contains the case studies, the pricing framework, and the methodology documents, and the build template specifies which documents to reference, the system doesn't need to search for them. It already knows. The 1-2 seconds of retrieval latency that vector search adds is wasted time when the answer to "which documents should inform this proposal?" is already encoded in the workspace configuration.
Chat routes use the full retrieval pipeline. When a user asks "What case studies do we have for healthcare clients?", the system needs to search. Vector similarity finds semantically relevant documents. Full-text search catches the keyword "healthcare" if the embeddings generalized it to something broader. Recency provides a fallback.
The architecture knows which mode it's in and adjusts. A document generation request hits the workspace context first. A search query hits the retrieval pipeline. This distinction matters because retrieval latency compounds. If every request, regardless of type, runs through the full vector search pipeline, a document build that references five internal documents adds 5-10 seconds of unnecessary search time to an already 30-60 second generation process.
The system adapts context strategy to generation mode. Retrieval is not a universal prelude. It's a tool deployed when the system doesn't already know the answer.
The Embedding Quality Question
Single embeddings per document lose detail compared to chunk-level embeddings. A 10,000-token document compressed into a 1,536-dimension vector necessarily sacrifices nuance. The embedding captures the document's general semantic territory, not the specific claims within it.
For document generation, this tradeoff is acceptable because you need the right document, not the right sentence. If the system retrieves the correct case study, the AI can find the relevant details within it. If the system retrieves a chunk from the wrong case study (because that chunk's embedding was more precisely similar to the query), no amount of chunk-level precision fixes the fundamental error.
The analogy: you're looking for a book in a library. Chunk-based retrieval is like searching for a specific paragraph across all books. Document-based retrieval is like finding the right book and reading the relevant chapter. The first approach might find a more precisely relevant paragraph, but it might also find a paragraph from the wrong book that happens to use similar language. The second approach finds the right book every time, then lets you find the paragraph yourself.
Embedding quality at the document level improves significantly with thoughtful preprocessing. Summaries, title-enriched embeddings, and metadata tags all help the single vector capture more of the document's semantic footprint. But the fundamental bet remains: for generation tasks, getting the right document matters more than getting the right fragment.
The Practical Consequences
The shift from chunk-level to document-level retrieval changed three things in production output quality.
First, reference accuracy improved dramatically. When the AI has the full case study, it gets the client name right, the industry right, the outcome right, and the narrative arc right. No more Frankensteined references stitched together from fragments of different documents.
Second, the hallucination pattern shifted. With chunk-based retrieval, the most common hallucination was invented connections between disconnected fragments. The AI would receive chunks from two different case studies and blend them into a single, fictional reference. With document-level retrieval, the AI has clear boundaries. This case study is one story. That case study is a different story. The boundaries are structural, not inferred.
Third, document coherence improved. Proposals generated with full-document context read like a senior sales professional wrote them with the source materials open on their desk. Proposals generated with chunk-based context read like someone described the source materials to the AI over a bad phone connection.
These aren't theoretical improvements. They're the differences we measured after switching retrieval strategies in production across thousands of generated documents. The fragment problem isn't an edge case. It's the default failure mode of chunk-based RAG applied to generation tasks.
The Broader Architectural Lesson
The RAG orthodoxy (chunk everything, embed everything, retrieve the most similar fragments) emerged from the information retrieval tradition, where the task is finding relevant information in a corpus. That tradition optimizes for precision: return the most relevant pieces.
Document generation is a different task. It's not about finding relevant information. It's about providing the AI with the source material it needs to create something new. The quality of the creation depends on the quality of the source material, and quality in this context means completeness, not precision.
This is a pattern that repeats across architecture: the same technique that excels at one task can fail at a related but fundamentally different task. Chunking excels at search. It fails at generation. The failure isn't in the technique. It's in the assumption that one retrieval strategy serves all downstream tasks.
The architectural lesson: every time you choose a retrieval strategy, you're implicitly choosing what downstream tasks it will serve well. If you only ever ask your system to answer questions, chunk-based RAG is the right call. The moment you ask it to create documents, you need a different strategy.
One size doesn't fit all. But the industry pretends it does, and the pretending costs everyone who tries to build a document generation system on top of a Q&A retrieval architecture.
Match retrieval architecture to generation task. That's the whole paper in seven words.
The takeaway: For document generation, retrieve whole documents instead of chunks: you need narrative coherence, not fragment precision, and the industry's default chunking-first RAG architecture is optimized for the wrong task.
References
1. Lewis, P. et al. "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks." NeurIPS, 2020. 2. Gao, Y. et al. "Retrieval-Augmented Generation for Large Language Models: A Survey." arXiv, 2024. 3. Barnett, S. et al. "Seven Failure Points When Engineering a Retrieval Augmented Generation System." arXiv, 2024.
This paper is part of The Architecture Series by Ed, Founder & CEO of DealiOS. For the weekly series and more technical deep-dives, visit dealios.app/library.
See how these principles come together. Explore the platform →