Ingestion is key to context engineering because AI agents can't skim
To get the most out of your agent, feed it information that's easy to use.
Context engineering has been the AI industry's hottest buzzword since late 2025 — an eternity in this fast-moving industry — yet how to do it properly remains a challenge.
Even developers who spend all day working on RAG pipelines can find themselves frustrated by the whack-a-mole nature of the problem. In large part, this is due to a structural misunderstanding: if outputs are garbage, the solution is not to beautify the garbage, but rather to fix the inputs.
My colleagues and I have spent the last several years working to solve this exact issue — and won a patent for our efforts. Here's how (and why) we did it.
TL;DR
Fix the information problem upstream
The future of AI is agentic, as every LinkedIn influencer will tell you. The present is, too. And agents need different inputs than humans do, because they intake information in a totally different way than we do. If humans absorb new data "one step at a time," then agents absorb "everything, everywhere, all at once."
That's why our team built a structured ingestion and chunking engine, called PrimeCut. While many have focused on retrieving the most relevant chunks (which is very important), we realized that the key to doing this actually depends even more on how those chunks are created in the first place, at ingestion.
So we built PrimeCut to bolt onto every common OCR (or process more than fifty file formats itself), and map out the full content of each ingested file. By understanding the relationships between pieces of information in the file, our engine is able to produce chunks that deliver full context and traceable justifications.
As you'll see shortly, the key to context engineering is fixing the problem upstream. That might sound counterintuitive — but if you understand vector embeddings, there's a simple and elegant logic to this approach.
Why hierarchy is key to context
Once upon a time, we were trying to build a legal chatbot for the tiny mountain principality of Andorra. And we had one particular nemesis: a document explaining the policy for vanity license plates, such as ALEX1. The chatbot could find the document itself, but since it lacked good captions, the chatbot couldn't connect a specific heading with the tables containing vital information.
The only way to recall this information was by wasting a lot of tokens (by basically dumping the whole thing into the context). This turned out to be a common problem: recently at the Applied AI conference, we heard the same thing from the developers of vespa.ai and hornet.dev. Since agents can't triage information, they're inefficient at distinguishing signal from noise. Imagine a fishing trawler scooping up dolphins, or an entire swimming pool turned red by a single drop of ink.
Or consider Google search: if a human wants to know "what is spam," he or she can distinguish between results that discuss unwanted emails versus canned meat. An agent cannot, which results in poisoned context and the poor thing desperately reasoning over relations between the two.
This is why search is — for the second time in our generation — again becoming so important. We were lucky enough to encounter the problem early, just when AI and vector search were beginning to dominate popular imagination. And it didn't take long before we recognized that hierarchical representation of content and structure were key to improving search in the context of AI.
How we made ingestion better
Ingestion is the first step of any RAG pipeline, and it's also a vital (if often overlooked) aspect of context engineering. Flaws in ingestion percolate down throughout the entire pipeline, usually in ways that are difficult, if not impossible, to fix.
As the saying goes, you can't make chicken salad out of chicken s***.
To avoid this unappetizing scenario, we built PrimeCut to actually understand the input and its implicit structure, as opposed to just raw data. Every element on every page gets parsed by the tool suited to it instead of a one-size-fits-all extractor.
For example: a complex graph on page 4 gets a parser built for graphs, while the surrounding body text gets parsed by a separate tool. A table is handled differently still.
Then, each of those elements is linked back to whatever it depends on — e.g. the heading it sits under, the paragraph that explains it, or the section it belongs to — so the relationships between them survive ingestion rather than being flattened into an undifferentiated wall of text. Same for spreadsheets: instead of extracting only numbers, it extracts the calculation that led to them.
This allows users to understand how A relates to B, and why A + B = C.
How we made chunking better
Better chunking is all about preserving hierarchy — and while we weren't the first people on earth to try this, we were the first to succeed at it. The one weird trick, so to speak, was finding the structure by outmaxing the cross-attention mechanism in LLMs.
Here's how we did it:
Every chunk carries its position in the document's heading tree: how deep it sits, and what it sits under. That is easy to specify and hard to produce.
With the patent now granted, we can describe how we do it. We give the model the full text and ask it to write that text back out — every word, unchanged — but indented like source code. Nothing is summarized and nothing is dropped. The only thing the model contributes is structure.
This is an expensive way to find headings, and we know it. Re-emitting an entire document costs far more tokens than classifying its lines. It is slower, and harder to guardrail. The model has to reproduce its input faithfully rather than paraphrase it, and faithfulness is not the behavior these models are trained toward.
We pay that cost because it lands on the right side of an asymmetry. A document is ingested once and retrieved from many thousands of times. Every token spent at ingestion is amortized across every query that follows, and every structural mistake made at ingestion is paid for on every one of them too.
The reason we reproduce the text rather than label it is attention. Asking the model to emit each line forces it to attend to the whole document while deciding where that line belongs — a heading three sections earlier is still in view when the model commits to an indentation level. Cheaper formulations we tried (like classifying headings in isolation, labeling chunks one at a time, or asking for the outline alone) give the model a narrower window, and the hierarchy degrades accordingly. In our evaluations, full-text reproduction recovered much more of the ground-truth structure than "single-shot outlining."
What comes back, after a long stretch of fine-tuning and harness work, is the same text — shaped like a tree.
Because of all this, we can bundle each chunk with its full lineage — e.g. the heading above it, the section above that, or the document title above that — into a single retrieval unit we call a chunkset. A paragraph three headings deep carries that context instead of losing that context. An optimization pass collapses and merges chunksets that are too similar or too small.
The result is a set of distinct, well-scoped units in place of one chunkset per heading in the tree. None of this works if the input is a mess. That's why the step before chunking — turning PDFs, Excel tables, or other file formats into clean, properly indented structure — matters just as much as the chunking itself.
How we made embedding better
"More input means more dilution" has been accepted wisdom in the world of vector search for years. The more information you add, the more opportunity to get sidetracked. Imagine your grandfather telling a story about the weather one July afternoon in 1957.
It's a simple concept. For a long time, it was true, too.
But what if more input could actually sharpen an embedding? Instead of a train of thought careening off the rails, what if it became more exact as the story progressed? This may sound counterintuitive to vector embedding veterans — but the mechanism makes sense, doesn't it?
The trick is adding the right text. By the time a chunk reaches the embedding model, it's no longer just a stray paragraph. It's already been bundled with its section, its heading, and its place in the document. So instead of a model padding a vector with noise, it's disambiguating one with lineage that was fixed at chunking time (not guessed at afterward).
That's the power of hierarchically organized information. When each piece of data is stacked correctly, adding more of it doesn't lead an agent off track. Instead, it points directly to the desired destination, and with a longer arm.
How we made retrieval better
At its core, retrieval in a RAG pipeline is just a simple vector search.* This is a double-edged sword: if you don't put garbage in, you don't get garbage out. But this also means no tricks or rerankings at this stage can save you from sloppiness earlier in the pipeline.
*Some people say full-text, but we'll get to that in a second.
If you've been diligent about hierarchically chunking your data to provide proper context, however, retrieval is where you start to really reap the rewards. Not only will agents avoid confusion about irrelevant information, they'll use vastly fewer tokens.
If you want to see this end-to-end without tinkering in your current pipeline: our context engine, called Grill, integrates PrimeCut into a full end-to-end RAG pipeline that can be dropped into any agent or application. It delivers 100% recall using only 23% of the tokens required by traditional chunkers, as measured on our OfficeQA benchmark against real filings.
That's compared to normal chunking with the same embedding model and same retrieval logic, i.e. with no tuning in our favor. Agents stop dumping half a document into context to find one number, because the one number already arrived with the context it needed.
Now, back to that vector/full-text question: it is hard to believe that using well-crafted hierarchical chunks as input for the newest state-of-the-art embedding models actually reduces the need for full-text/BM25/hybrid search by a lot, sometimes down to zero. However, it really does: in our own tests we now often reach 90+% of all results from vector search.
As a result, everything else becomes an add-on. We even now replaced RRF with cosine-led fusion, so as not to coarsely "taint" the beautiful vector search results.
Improving context for LLMs
So far, we've talked a lot about the importance of context for AI agent performance. We've also stressed the role of search quality for optimal results. But here's where we should mention that structuring information is also vital for perfecting your context.
That's because agents, much like people, have a U-shaped attention curve. In other words, the information at the beginning and the end of the context get the most attention — and the stuff in the middle often gets lost. Trying to fight this is a fast way to go crazy.
However, it can also be used to your advantage. The solution is simple: put the most important results at the beginning and the end, and stick the worst ones in the middle.
- Score every retrieved chunk before it goes into the prompt.
- Reorder them: strongest result first, second-strongest last, third-strongest second, fourth one second-last, and so on. Weaker results get pushed into the middle, instead of feeding them in the usual raw rank order.
- Let the curve do the work. The agent will notice what's at the edges and skim what's buried, so you place your evidence accordingly instead of hoping the model reads everything with equal care.
In a nutshell: swimming with the current works better than trying to fight it.
TL;DR
In the age of AI agents, all roads lead to search
In the early 2000s, the rise of Google changed the search industry by turbocharging the keyword era with PageRank, citations, and links. This was the culmination of decades' worth of innovation: inverted indices, BM25, and many more advances all built on the principle of matching terms and then ranking them by frequency.
Today, that era is ending as AI ushers in a new type of search that demands different solutions. Context engineering for agents requires fresh tactics and ways of thinking that diverge in significant ways from human-oriented search.
What it doesn't necessarily require, however, is massive and perpetual cost. While proper context engineering comes with higher up-front ingestion costs, this only happens once. But what's ingested can be retrieved thousands of times or more. The amortization is obvious.
This is yet another reason to discard the idea that you need to reinvent the wheel to thrive in the brave new world of search. Many of the time-honored principles still apply — they just need to be applied in modern ways.
Ready to fix context engineering at the ingestion layer?
- Try PrimeCut for free — 1,000 pages free, no credit card required
- PrimeCut product page — how it works
- Pricing — from €0.003/page
Continue reading
- What is Context Engineering? — the full-stack view: why context quality is decided before the prompt
- Document ingestion guide — how to build a production ingestion pipeline that preserves structure
- RAG chunking guide — chunking strategies, tradeoffs, and when to use each one
- Chunksets explained — what a hierarchical retrieval unit actually looks like
- Grill — managed context engine — PrimeCut wired into an end-to-end retrieval pipeline