Common failure modes in chunking
No matter how fancy the strategy is — even LLM-based or late chunking — every approach still operates on the same primitive idea: choose cut points and slice the text into chunks.
Fixed-size chunking makes that obvious, but more advanced methods mainly choose better boundaries. The slicing itself is the constant. That means the same classes of failure show up regardless of which strategy you pick.
Understanding these failure modes matters because they directly affect retrieval accuracy, and by extension, the quality of every answer your RAG system produces.
Orphaned facts
A retrieved chunk often depends on context that lives just outside its boundaries. The sentence looks relevant in isolation, but the heading, definition, exception, or procedural step that makes it trustworthy was cut away.
Example. Consider a policy document where a fixed-size chunker produces:
- Chunk 1:
"…end of previous section. 3. Health Insurance" - Chunk 2:
"Employees are eligible for coverage after 90 days…" - Chunk 3:
"…enrollment deadline is December 15. 4. Dental Coverage"
A user asks "When can I enroll in health insurance?" The retriever returns Chunk 2, which contains the eligibility rule — but the section heading that identifies this as health insurance landed in Chunk 1. The model receives a bare eligibility sentence with no label. If the prompt already contains dental or vision chunks, the model may attribute the 90-day rule to the wrong benefit.
Orphaned facts are especially damaging in legal, financial, and compliance documents where a statement's meaning depends entirely on the heading or clause it falls under.
Overlap inflation
Overlap is the standard patch for boundary loss. You slide the window so adjacent chunks share some tokens, reducing the chance of cutting mid-thought. The tradeoff is real, though: overlap duplicates content across your vector store, inflates storage costs, and increases retrieval noise.
Example. With a 200-token chunk size and 50-token overlap, roughly 25% of your stored tokens are duplicates. At scale — say, 100,000 documents — that is a meaningful increase in index size and embedding cost. Worse, when a user query is semantically close to the overlapping region, the retriever may return multiple chunks that are mostly the same text. Your LLM prompt fills up with near-duplicates instead of diverse, relevant passages.
Overlap helps at boundaries, but it does not solve the underlying problem. It is a band-aid that trades storage and retrieval precision for slightly fewer orphaned facts.
Context dilution
When retrieval returns too many partially relevant chunks, you start losing signal inside the LLM's context window. This is sometimes called the "lost in the middle" problem: models pay more attention to the beginning and end of the prompt and tend to underweight information buried in the middle.
Example. Suppose a user asks a question that touches three sections of a technical manual. Your retriever returns the top-8 chunks. Five of them are genuinely relevant, but three are borderline matches dragged in because the chunking boundaries happened to co-locate relevant and irrelevant sentences. The model now has to distinguish signal from noise across thousands of tokens. In practice, answer quality degrades — the model may hallucinate a conflation of two unrelated passages, or simply ignore the most relevant chunk because it landed in position four of eight.
Context dilution gets worse as you increase top_k to compensate for low retrieval precision, which is itself often caused by orphaned facts and overlap noise. The failure modes compound.
Why these failures keep returning
You can patch chunking by attaching extra generated context to each chunk before embedding — for instance, prepending a summary or section heading. But that is still chunking. You are stapling a helper note onto a fragment and hoping the retriever can now find the right fragment.
The core limitation is not which separator you picked or how you tuned your chunk size. It is the decision to retrieve broken-off fragments in the first place. As long as the retrieval unit is a slice of text with arbitrary boundaries, these three failure modes remain in play.
More advanced strategies (semantic chunking, agentic chunking, late chunking) reduce the frequency of these failures. None of them eliminate the failure modes entirely, because the fundamental unit — a chunk — is still a fragment.
TL;DR
Continue reading
- POMA chunksets — how chunksets address these failures
- Strategy comparison — see which strategies are affected
- The full chunking guide — deep dive with all 15 strategies
- PrimeCut product page