Source: http://www.poma-ai.com/docs/pipelines/azure-document-intelligence-to-mongodb-atlas

# The Missing Link Between Azure Document Intelligence and Optimal Retrieval in MongoDB Atlas

<ByAuthor />

**The short answer:** Azure Document Intelligence's `prebuilt-layout` model gives you excellent reading order and table extraction; MongoDB Atlas folds vector search directly into the aggregation pipeline you already run. Wired together naively — flatten, fixed-split, `insert_one` — the pipeline still produces mediocre RAG, because nothing in between rebuilds the document's hierarchy or projects the metadata Atlas needs at query time. The missing link is POMA: `PrimeCut().ingest()` consumes the raw analyze result (auto-detected, either shape), emits hierarchy-preserving [chunksets](/learn/chunking/chunksets), and `vektoria` keeps the Atlas collection content-free while `assemble()` reconstructs prompt-ready context from a volume.

## What each end of the pipeline actually provides

**Azure Document Intelligence** returns one of two shapes: markdown mode's single reading-order string with inline HTML tables and `<!-- PageBreak -->` delimiters, or JSON mode's `paragraphs[]` (ordered by span offset, role-tagged) plus `tables[]` cell grids. Neither shape links a page-41 heading back to the page-3 title that opened its chapter, and neither decides what a retrieval unit should be. Details: [The Optimal Chunker for Azure Document Intelligence](/optimal-chunker-azure-document-intelligence).

**MongoDB Atlas Vector Search** provides a `$vectorSearch` aggregation stage on top of a document you already control, plus ordinary fields for `$match` filtering and a `$search` compound stage for BM25-style hybrid. What it doesn't provide: any opinion about what a document should contain, or a reminder that its aggregation pipeline drops any field the next stage didn't ask for. Details: [MongoDB Atlas Chunking Strategy for RAG](/optimal-chunks-mongodb-atlas).

## The naive wiring, and where it breaks

The common recipe — flatten Azure's `content` string (or `paragraphs[]`) with a fixed-size splitter, embed, `insert_one` each fragment, `$vectorSearch` at query time — breaks in a way specific to this pair, and it compounds across ingest and retrieval:

- **Azure's page anchors never make it into a `$match`-able field.** Azure only exposes page boundaries through `<!-- PageBreak -->` comments (markdown mode) or span offsets against `paragraphs[]` (JSON mode). A naive flatten-then-split loses that boundary before any page number is attached to a document, so an Atlas `$match` predicate has nothing to filter a citation by.
- **The retrieval side drops what ingest didn't lose.** Even with page numbers correctly attached at ingest, a `$vectorSearch` stage with no follow-up `$project` silently drops every field but the ones MongoDB's aggregation pipeline defaults to returning. A `file_id`/`chunkset_index` written correctly at ingest time is invisible at retrieval time if the next stage doesn't ask for it — the two failures look identical from the application's side (empty context), but one is an ingest bug and the other a query bug.
- **Azure's tables compound the ingest-side mistake.** A table sliced mid-row by a fixed-size splitter becomes two documents that each `$match` the same `file_id`/`page` — duplicate near-identical hits crowding out the passage that actually answers the question, on top of the missing-metadata risk above.

## The pipeline, end to end

```bash
pip install azure-ai-documentintelligence pymongo
```

```python
import json
import os

from azure.ai.documentintelligence import DocumentIntelligenceClient
from azure.core.credentials import AzureKeyCredential
from pymongo import MongoClient
from poma import PrimeCut
from poma.vektoria import assemble, records_from_archive, Volume
from poma.embeddings import get_embedder

# 1. Azure Document Intelligence — your existing call, unchanged.
di = DocumentIntelligenceClient(
    endpoint=os.environ["AZURE_DI_ENDPOINT"],
    credential=AzureKeyCredential(os.environ["AZURE_DI_KEY"]),
)
with open("document.pdf", "rb") as f:
    poller = di.begin_analyze_document(
        "prebuilt-layout",
        body=f,
        output_content_format="markdown",  # omit for classic JSON mode — POMA handles both
    )
analysis = poller.result()
with open("result.azure-di.json", "w") as f:
    json.dump(analysis.as_dict(), f)

# 2. The missing link — raw result JSON in, content-free archive out.
client = PrimeCut()  # reads POMA_API_KEY
client.ingest("result.azure-di.json", download_dir="archives", filename="doc.poma")

embedder = get_embedder("local:BAAI/bge-small-en-v1.5")
vol = Volume("s3://your-bucket/poma")
mongo = MongoClient(os.environ["MONGODB_ATLAS_URI"])
col = mongo["poma"]["chunksets"]
# Atlas vector index (numDimensions, similarity) is created once via the Atlas UI/API,
# named "poma_vec" here — not through the pymongo driver itself.

# 3. Content-free ingest — Atlas holds only routing metadata; content lives on the volume.
records = records_from_archive("archives/doc.poma")  # -> list[Record] (id, text, payload)
for r in records:
    vol.write_doc(r.payload["file_id"], {"file_id": r.payload["file_id"],
                                          "chunks": r.payload["chunks"], "text": r.text})
    col.insert_one({
        "_id": r.id,
        "embedding": embedder.embed([r.text])[0],
        "metadata": {"file_id": r.payload["file_id"], "chunkset_index": r.payload["chunkset_index"]},
    })

# 4. $vectorSearch retrieval — $project the metadata explicitly, then assemble.
qv = embedder.embed(["early termination conditions"])[0]
hits = list(col.aggregate([
    {"$vectorSearch": {"index": "poma_vec", "path": "embedding",
                       "queryVector": qv, "numCandidates": 200, "limit": 10}},
    {"$project": {"_id": 1, "metadata": 1, "score": {"$meta": "vectorSearchScore"}}},
]))
context = assemble(hits, volume=vol)  # -> [{"file_id", "content"}, ...]
```

Because the analysis already ran, POMA charges only for the downstream structure and chunking value — the OCR front-end is skipped. The Atlas collection never stores document content: `vektoria`'s `Volume` holds it, and `assemble()` deduplicates ancestor lineage across retrieved chunksets into one prompt-ready cheatsheet — the discipline behind our reference legal-document benchmark answering the same query with **337 tokens** instead of **1,542** for a recursive-splitter baseline. [Methodology here](/document-ingestion-chunking-rag).

## Metadata mapping: POMA fields → MongoDB Atlas primitives

| POMA chunk field | Atlas primitive | What it enables |
| --- | --- | --- |
| `to_embed` | embedding field, covered by an Atlas Vector Search index | `$vectorSearch` ANN |
| `file_id` | `metadata.file_id`, ordinary field | per-document scope via `$match` |
| `page` (from Azure `PageBreak`/span offsets) | volume document, not a field | content-free retrieval, cheatsheet assembly |
| `depth` | volume document, not a field | content-free retrieval, cheatsheet assembly |
| `chunkset_index` | `metadata.chunkset_index` — must be `$project`-ed | `assemble()` dedupe key |
| chunk content | not stored in the document — lives on `Volume` | content-free index, smaller documents |

## Frequently asked questions

### How do I get Azure Document Intelligence results into MongoDB Atlas for RAG?

Run `begin_analyze_document` with `prebuilt-layout` as you already do, and save the raw analyze result JSON. Hand that unmodified file to `PrimeCut`, which auto-detects the Azure shape and rebuilds the cross-page heading hierarchy into chunks and chunksets. Embed each chunkset's `to_embed` text, `insert_one` it as a document with an `embedding` field and a `metadata` field, and retrieve through `vektoria`'s `assemble()`, which fetches content from a volume and returns prompt-ready cheatsheets.

### Why does POMA assemble return nothing after I ingest Azure Document Intelligence output into Atlas?

This is the one documented Atlas gotcha, and it applies regardless of source: a `$vectorSearch` aggregation stage must be followed by a `$project` that explicitly includes the `metadata` field holding `file_id` and `chunkset_index`. MongoDB's aggregation pipeline drops any field you do not project, so a `$vectorSearch` with no follow-up `$project` on `metadata` is the one documented way `assemble()` comes back empty on a non-empty search.

### What metadata should Azure Document Intelligence chunks carry in MongoDB Atlas documents?

A `metadata` field holding only `file_id` and `chunkset_index` — the fields `$project` must retain for `assemble()` to work. `page` and `depth`, recovered from Azure's `PageBreak` comments or paragraph span offsets, stay on the volume document with the rest of the chunkset content rather than as separate Atlas fields. The embedding itself sits in a field covered by the Atlas Vector Search index.

### Do Azure Document Intelligence's tables survive the trip into MongoDB Atlas?

Yes, if you chunk the raw analyze result instead of a flattened string. Azure Document Intelligence returns tables as inline HTML (markdown mode) or cell grids with merged-cell spans (JSON mode), and POMA converts both into HTML that stays intact inside a single document's content — no row gets split across two MongoDB documents the way a fixed-size splitter would cut it.

### Does this pipeline work with self-hosted MongoDB, not just Atlas?

No — `$vectorSearch` and the Atlas Search vector index type are Atlas-specific features, not part of self-hosted community MongoDB. The chunking and content-free ingest pattern on this page is database-agnostic, but the retrieval stage itself requires an Atlas cluster with a vector index configured.

## Related recipes

Same parser, different store: [Azure Document Intelligence → Qdrant](/pipelines/azure-document-intelligence-to-qdrant) · [Azure Document Intelligence → Pinecone](/pipelines/azure-document-intelligence-to-pinecone) · [Azure Document Intelligence → Redis](/pipelines/azure-document-intelligence-to-redis) · [Azure Document Intelligence → LanceDB](/pipelines/azure-document-intelligence-to-lancedb)

Foundations: [The Optimal Chunker for Azure Document Intelligence](/optimal-chunker-azure-document-intelligence) · [MongoDB Atlas Chunking Strategy for RAG](/optimal-chunks-mongodb-atlas) · [All pipeline recipes](/pipelines/) · [RAG chunking guide](/guides/rag-chunking/)