Better Ingestion Ingestion Chunking Retrieval Ingestion Chunking

Yes, in the same sense that voice calls still matter after the launch of the internet. Technologies can overlap without being in direct competition (and sometimes even benefit from each other). Here’s a short explanation of how this works in the case of Context Engineering and MCP.

Released by Anthropic in late 2024, model context protocol (MCP) provides a standardized way for AI tools to connect with data sources and other tools. From a developer’s standpoint, the benefits are obvious: this one technology allows an AI tool to connect with any content library, staging environment, and so on.

Universal compatibility makes the question of “how do I make X work with Y?” much easier to answer. However, connecting to a database and pulling specific bits of relevant information from it are very different things.

POMA AI specializes in the latter. Using it in conjunction with MCP increases the utility of both technologies, but their core functions are quite different. In fact, as adoption of MCP increases, so does the need for POMA AI—because a bigger pool of information is only useful if you have a way to find the data you actually need.

An LLM’s context window is often compared to its “working memory.” This analogy is both helpful (in understanding its function) and potentially misleading (in understanding its practical applications).

Theoretically, a much larger context window would solve many of the most pressing issues currently facing LLMs. For example: a supersized context window could be expected to greatly reduce hallucinations, since this would enable an LLM to “remember” more information it ingests. As a result, there would be fewer gaps in the LLM’s knowledge base and fewer opportunities for it to invent incorrect information to fill those gaps.

But it’s a bit more complicated in practice.

Larger context windows require more computing power—a lot more, since compute requirements increase quadratically in comparison with the input. In other words: if the context window doubles in size, the LLM requires four times as much power to process the information in it.

In addition to increased compute costs, larger context windows haven’t always led to improved accuracy in real-world use cases. Perhaps the most notorious example is the case of an Australian lawyer who was recently stripped of his ability to practice as a principal lawyer after submitting court documents riddled with nonexistent AI-generated citations. Despite being produced by “reputable” AI-powered legal software, the output was still plagued by hallucinations.

Here’s where the “working memory” analogy is unintentionally apt. The larger the context window, the more information is contained in the middle of the window. And in real world use cases, LLMs have a tendency to skim over those details—much like a human reader when confronted with a solid page of text. Whether you’re a robot or a meatsack, it’s easy to get lost in the middle.

The term “bathtub curve” has become popular among engineers for describing the failure rates of the products they build. This mental image—with the ends of the tub clearly visible, and the middle entirely submerged—can also be useful for understanding how information gets lost even in the biggest context windows.

LLMs might read the first few sentences carefully, but soon their eyes glaze over. Upon reaching the end of the page, their attention may perk up again, but their comprehension of everything in the middle remains hazy. As a result, the LLM remains prone to introducing incorrect information into its outputs.

The dramatic increase in size of context windows has led some people to wonder if Context Engineering is still necessary for LLMs. After all, if an entire database can fit in an LLM’s context window, why would it need to consult an external database?

At the risk of oversimplifying: just because an LLM’s context window can fit a huge amount of information in it doesn’t mean the LLM can make effective use of that information.

The appeal of Context Engineering is its precision. For industries where accuracy is at a premium—like healthcare or the legal profession—simply having a huge amount of information available doesn’t address the actual needs. It would be like having a set of beautiful encyclopedias open to all pages at all times. If that idea breaks your brain a little, that’s the point. It’s physically impossible.

Context Engineering’s ability to quickly (and cost-effectively) provide exactly the information required for a certain task means that it’s highly likely to stay relevant long into the future, regardless of how large context windows may grow.

If Sam Altman and friends do create an omnipotent digital god, then the question is moot and you either have nothing to worry about or much bigger things to worry about.

But in the event that AI development continues on its current trajectory—i.e. lots of incremental improvements with the occasional leap forward—then Context Engineering will certainly be a key driver of this progress, and retain its usefulness for the foreseeable future.

This is because no matter how “smart” AI models become, they’ll still face the fundamental issues of today’s models. To be more specific: it’s impossible to keep every possible piece of information permanently poised at the tip of your tongue (or LLM prompt context), and it’s not cost-effective to read an entire book every time you need to quote a line from Chapter 3.

To use a human analogy: an excellent set of notes is vital whether you’re a 12-year old student or a 32-year old finishing their PhD.

In a nutshell, they’d make less money if they did.

The behemoths of AI development—OpenAI, Anthropic, etc.—generate revenue when their users burn tokens. It’s not in their financial interest for you to use n tokens to get the context you wanted, when you could’ve used n2 tokens instead. Plus, their file search tools utilize rough chunking strategies. That means when you go back to retrieve the files you need, you’ll use even more tokens. Making this process more efficient might save you money, but it cuts into the big AI labs’ profit margins.

In fairness, there’s another reason big AI labs didn’t solve this particular puzzle: it’s really f****** hard. And the OpenAIs of the world have a lot of other initiatives competing for resources and their teams’ attention, from developing chatbots to building short-form video platforms.

POMA AI, on the other hand, is solely focused on making chunking as efficient and effective as possible. This is what our team does all day, so we do it well.

If you’ve already checked out our Pricing pages, you probably know that’s not the answer. And if you continued clicking around our websites, you might’ve seen many of the same terms: structured data, RAG, and so on.

So it’s understandable if you’re scratching your head right now.

Unstructured.io is primarily an element extractor. It pulls elements like images and tables from a document, and converts them into text digestible by LLMs. They do have a built-in chunking function, but that’s more of a side dish than main course.

POMA AI, on the other hand, is all about chunking. We specialize in turning entire documents into chunks, so the information contained in them can be accurately (and efficiently) utilized by RAG-enabled LLMs.

Which tool is best? It depends on your needs. If you’re curious about how POMA AI might work for yours, give our do-it-yourself Demo a try.