POMA AI ⬣ Docs

Appearance

Ingestion patterns ​

Once you know ingestion is more than just "read file, get text," the next question is how to wire it into the rest of your stack. In practice, teams converge on a few common patterns.

File-based batch ingestion ​

This is the default approach for many early RAG deployments: periodically ingest documents from shared folders, buckets, or repositories as batches.

Typical use cases

  • Periodic ingestion of internal knowledge bases and policy documents.
  • Migrating legacy archives of PDFs and Word files into a new RAG system.
  • One-off ingest of open-source document sets such as manuals and standards.

Advantages

  • Simple operational model.
  • Works well for large document types where real-time updates are not required.
  • The normalized corpus can be reused with different embedding models later.

Disadvantages

  • Poor fit for real-time use cases.
  • Coarse-grained error handling.
  • Updates and deletions are harder to reconcile cleanly.

API- and event-based ingestion ​

Here, document ingestion reacts to events. A new ticket is created, a wiki page is updated, or a file is uploaded through an application. The ingestion pipeline is triggered via API, queue, or webhook.

Typical use cases

  • Customer support systems where new tickets should become searchable within seconds.
  • Product docs that must reach RAG-powered chat quickly after an update.
  • Workflow tools that embed RAG inside an existing SaaS product.

Advantages

  • Supports near-real-time updates and deletions.
  • Gives you finer-grained routing and metadata control.
  • Lets you vary parsing strategies by source or format.

Disadvantages

  • Operationally more complex.
  • Harder to rebuild from scratch after systemic changes.
  • Easier to couple tightly to upstream producers.

Connector-based ingestion ​

Many RAG stacks rely on connectors to extract content from SaaS platforms or transactional databases and map it into a neutral internal representation.

Typical use cases

  • Building organization-wide search across many systems.
  • Pulling tickets, CRM data, and knowledge-base content into one retrieval layer.
  • Standardizing authentication, pagination, and rate limits through shared integrations.

Advantages

  • Reduces implementation time.
  • Often aligns initial structure with business semantics such as tickets, issues, or wiki pages.
  • Makes multi-system ingestion easier to centralize.

Disadvantages

  • Limited control over parsing fidelity.
  • Potential vendor lock-in.
  • Not every connector exposes enough structural detail to drive good chunking.

In mature deployments, teams often combine all three patterns: batch for static archives, event-based flows for live content, and connectors for the long tail of SaaS systems.

TL;DR

Batch, event-based, and connector-driven ingestion all have legitimate use cases. The important design requirement is that the pattern you choose still preserves enough structure for chunking and retrieval later.

Continue reading ​