Source: http://www.poma-ai.com/docs/pipelines/unstructured-to-mongodb-atlas

# The Missing Link Between Unstructured.io and Optimal Retrieval in MongoDB Atlas

<ByAuthor />

**The short answer:** Unstructured.io gives you a typed, ordered element list; MongoDB Atlas gives you `$vectorSearch` as an aggregation stage you can combine with your existing pipeline and filters. Wired together naively — join element text, split, embed, insert — they still produce mediocre RAG, because nothing in between rebuilds the hierarchy Unstructured's typing implies but never states, and Atlas's aggregation semantics silently drop what you don't project. The missing link is POMA: `PrimeCut().ingest()` consumes the raw element list (auto-detected), emits hierarchy-preserving [chunksets](/learn/chunking/chunksets), and `vektoria`'s content-free ingest pattern writes them as documents with the metadata `$vectorSearch` needs. Retrieval comes back through `assemble()` as prompt-ready cheatsheets.

## What each end of the pipeline actually provides

**Unstructured.io** (`partition_pdf` or the hosted API) returns a flat, ordered list of typed elements — `Title`, `NarrativeText`, `ListItem`, `Table`, `Image`, plus page furniture types — each carrying a stable `element_id` and metadata such as `page_number` and `text_as_html`. What it doesn't provide: any record of which `Title` nests under which, or a retrieval-ready unit. Details: [The Optimal Chunker for Unstructured.io](/optimal-chunker-unstructured).

**MongoDB Atlas** provides `$vectorSearch` as a first-class aggregation stage — `numCandidates`, `limit`, and combinable `$match`/compound `$search` filters — inside the same pipeline you already run for everything else in the collection. What it doesn't provide: any opinion about what a document's embedding field should represent, and no free pass on fields you forget to project. Details: [MongoDB Atlas Chunking Strategy for RAG](/optimal-chunks-mongodb-atlas).

## The naive wiring, and where it breaks

The common recipe — concatenate every element's `text`, run a character splitter, embed, `insert_one` — breaks in a pair-specific way. Concatenating Unstructured's flat element stream re-admits `Header`/`Footer`/`PageNumber` noise, and flattening a `Table` element to its plain `text` (instead of using `text_as_html`) mangles row and column structure right at write time — a failure Unstructured itself already prevented by keeping the two representations separate.

Even a correctly built ingest can look broken at query time for an unrelated reason: Atlas's aggregation pipeline only returns fields you explicitly project. A `$vectorSearch` stage with no follow-up `$project` on `metadata` drops `file_id` and `chunkset_index` from every hit — unlike a plain `find()`, there is no implicit "return everything" behavior here — so `assemble()` receives hits with no routing metadata and returns empty from a search that matched correctly. The two failures compound: a team debugging "empty retrieval" is as likely to be chasing a flattened table as a missing `$project`.

## The pipeline, end to end

```bash
pip install 'unstructured[pdf]' pymongo
```

```python
import os
from unstructured.partition.pdf import partition_pdf
from unstructured.staging.base import elements_to_json
from pymongo import MongoClient
from poma import PrimeCut
from poma.vektoria import assemble, records_from_archive, Volume
from poma.embeddings import get_embedder

# 1. Your existing Unstructured call — unchanged.
elements = partition_pdf(
    filename="contract.pdf",
    strategy="hi_res",
    infer_table_structure=True,
    extract_image_block_types=["Image"],
    extract_image_block_to_payload=True,
)
elements_to_json(elements, filename="elements.json")

# 2. The missing link — raw element list in, a .poma archive out.
client = PrimeCut()  # reads POMA_API_KEY
client.ingest("elements.json", download_dir="archives", filename="doc.poma")

embedder = get_embedder("local:BAAI/bge-small-en-v1.5")
vol = Volume("s3://your-bucket/poma")
mongo = MongoClient(os.environ["MONGODB_ATLAS_URI"])
col = mongo["poma"]["chunksets"]

# Atlas vector index (numDimensions, similarity) is created once via the Atlas UI/API,
# named "poma_vec" here — not through the pymongo driver itself.

# 3. Content-free ingest — document holds only routing metadata; content lives on the volume.
records = records_from_archive("archives/doc.poma")  # -> list[Record] (id, text, payload)
for r in records:
    vol.write_doc(r.payload["file_id"], {"file_id": r.payload["file_id"],
                                          "chunks": r.payload["chunks"], "text": r.text})
    col.insert_one({
        "_id": r.id,
        "embedding": embedder.embed([r.text])[0],
        "metadata": {"file_id": r.payload["file_id"], "chunkset_index": r.payload["chunkset_index"]},
    })

# 4. $vectorSearch retrieval — $project the metadata explicitly, then assemble.
qv = embedder.embed(["early termination conditions"])[0]
hits = list(col.aggregate([
    {"$vectorSearch": {"index": "poma_vec", "path": "embedding",
                       "queryVector": qv, "numCandidates": 200, "limit": 10}},
    {"$project": {"_id": 1, "metadata": 1, "score": {"$meta": "vectorSearchScore"}}},
]))
context = assemble(hits, volume=vol)  # -> [{"file_id", "content"}, ...]
```

POMA validates the element list's shape up front (a corrupted or mislabeled upload 422s immediately), drops `Header`/`Footer`/`PageNumber` elements before any document is written, splices `Table` elements' `text_as_html` so rows survive on the volume rather than in a flattened field, and rebuilds the cross-page heading tree before chunking. To force or suppress detection, set `external_ocr_source` to `"unstructured"` or `"none"`.

## Metadata mapping: POMA fields → MongoDB Atlas primitives

| POMA chunk field | Atlas primitive | What it enables |
| --- | --- | --- |
| `to_embed` | embedding field, covered by the Atlas Vector Search index | `$vectorSearch` ANN over chunkset text |
| `file_id` | field inside `metadata` | `$match`/filter scoping; requires `$project` to return |
| `chunkset_index` | field inside `metadata` | required by `assemble()`; requires `$project` to return |
| `page` (from Unstructured `metadata.page_number`) | optional field inside `metadata` | page-cited answers |
| `depth` (rebuilt from flat `Title` elements) | optional field inside `metadata` | hierarchy-aware `$match` filtering |
| `text_as_html` (Unstructured `Table` splice) | stored in the volume document, not an indexed Atlas field | tables reach the prompt intact, never garbled by the aggregation pipeline |

## Frequently asked questions

### How do I get Unstructured.io elements into MongoDB Atlas for RAG?

Save the element list with `elements_to_json`, run `PrimeCut().ingest()` (auto-detected, hierarchy rebuilt), insert each chunkset with an Atlas-indexed embedding field and a `metadata` field for `file_id`/`chunkset_index`. Query with `$vectorSearch` plus `$project`, then hand the result to `assemble()`.

### Why does POMA assemble() return nothing from my MongoDB Atlas plus Unstructured pipeline?

It's Atlas's aggregation contract, not an Unstructured issue: `$vectorSearch` must be followed by a `$project` that explicitly includes `metadata`. Unprojected fields are dropped by the pipeline, so `assemble()` gets nothing back even from a correctly matched search.

### What MongoDB Atlas fields should Unstructured chunk data carry?

At minimum an Atlas-indexed embedding field and a `metadata` field holding `file_id`/`chunkset_index`, always projected. Add `page` (from `page_number`) and `depth` inside `metadata` for citation and hierarchy filtering.

### Do Unstructured's HTML tables survive into MongoDB Atlas retrieval?

Yes, via the volume, not the indexed document — POMA splices `text_as_html` into the chunkset content stored on the volume, so `assemble()` returns the table intact instead of the aggregation pipeline flattening or truncating it.

### Does this pipeline require MongoDB Atlas specifically, or does self-hosted MongoDB work too?

Atlas specifically — `$vectorSearch` and the Atlas Search vector index type don't exist on self-hosted community MongoDB. The PrimeCut and `vektoria` ingest steps are unaffected; only the retrieval query needs Atlas.

## Related recipes

Same parser, different store: [Unstructured.io → Qdrant](/pipelines/unstructured-to-qdrant) · [Unstructured.io → Pinecone](/pipelines/unstructured-to-pinecone) · [Unstructured.io → pgvector](/pipelines/unstructured-to-pgvector)

Foundations: [The Optimal Chunker for Unstructured.io](/optimal-chunker-unstructured) · [MongoDB Atlas Chunking Strategy for RAG](/optimal-chunks-mongodb-atlas) · [All pipeline recipes](/pipelines/) · [RAG chunking guide](/guides/rag-chunking/)