Unstructured.io to MongoDB Atlas: Chunking for RAG
The short answer: Unstructured.io gives you a typed, ordered element list; MongoDB Atlas gives you $vectorSearch as an aggregation stage you can combine with your existing pipeline and filters. Wired together naively — join element text, split, embed, insert — they still produce mediocre RAG, because nothing in between rebuilds the hierarchy Unstructured's typing implies but never states, and Atlas's aggregation semantics silently drop what you don't project. The missing link is POMA: PrimeCut().ingest() consumes the raw element list (auto-detected), emits hierarchy-preserving chunksets, and vektoria's content-free ingest pattern writes them as documents with the metadata $vectorSearch needs. Retrieval comes back through assemble() as prompt-ready cheatsheets.
What each end of the pipeline actually provides
Unstructured.io (partition_pdf or the hosted API) returns a flat, ordered list of typed elements — Title, NarrativeText, ListItem, Table, Image, plus page furniture types — each carrying a stable element_id and metadata such as page_number and text_as_html. What it doesn't provide: any record of which Title nests under which, or a retrieval-ready unit. Details: The Optimal Chunker for Unstructured.io.
MongoDB Atlas provides $vectorSearch as a first-class aggregation stage — numCandidates, limit, and combinable $match/compound $search filters — inside the same pipeline you already run for everything else in the collection. What it doesn't provide: any opinion about what a document's embedding field should represent, and no free pass on fields you forget to project. Details: MongoDB Atlas Chunking Strategy for RAG.
The naive wiring, and where it breaks
The common recipe — concatenate every element's text, run a character splitter, embed, insert_one — breaks in a pair-specific way. Concatenating Unstructured's flat element stream re-admits Header/Footer/PageNumber noise, and flattening a Table element to its plain text (instead of using text_as_html) mangles row and column structure right at write time — a failure Unstructured itself already prevented by keeping the two representations separate.
Even a correctly built ingest can look broken at query time for an unrelated reason: Atlas's aggregation pipeline only returns fields you explicitly project. A $vectorSearch stage with no follow-up $project on metadata drops file_id and chunkset_index from every hit — unlike a plain find(), there is no implicit "return everything" behavior here — so assemble() receives hits with no routing metadata and returns empty from a search that matched correctly. The two failures compound: a team debugging "empty retrieval" is as likely to be chasing a flattened table as a missing $project.
The pipeline, end to end
pip install 'unstructured[pdf]' pymongoimport os
from unstructured.partition.pdf import partition_pdf
from unstructured.staging.base import elements_to_json
from pymongo import MongoClient
from poma import PrimeCut
from poma.vektoria import assemble, records_from_archive, Volume
from poma.embeddings import get_embedder
# 1. Your existing Unstructured call — unchanged.
elements = partition_pdf(
filename="contract.pdf",
strategy="hi_res",
infer_table_structure=True,
extract_image_block_types=["Image"],
extract_image_block_to_payload=True,
)
elements_to_json(elements, filename="elements.json")
# 2. The missing link — raw element list in, a .poma archive out.
client = PrimeCut() # reads POMA_API_KEY
client.ingest("elements.json", download_dir="archives", filename="doc.poma")
embedder = get_embedder("local:BAAI/bge-small-en-v1.5")
vol = Volume("s3://your-bucket/poma")
mongo = MongoClient(os.environ["MONGODB_ATLAS_URI"])
col = mongo["poma"]["chunksets"]
# Atlas vector index (numDimensions, similarity) is created once via the Atlas UI/API,
# named "poma_vec" here — not through the pymongo driver itself.
# 3. Content-free ingest — document holds only routing metadata; content lives on the volume.
records = records_from_archive("archives/doc.poma") # -> list[Record] (id, text, payload)
for r in records:
vol.write_doc(r.payload["file_id"], {"file_id": r.payload["file_id"],
"chunks": r.payload["chunks"], "text": r.text})
col.insert_one({
"_id": r.id,
"embedding": embedder.embed([r.text])[0],
"metadata": {"file_id": r.payload["file_id"], "chunkset_index": r.payload["chunkset_index"]},
})
# 4. $vectorSearch retrieval — $project the metadata explicitly, then assemble.
qv = embedder.embed(["early termination conditions"])[0]
hits = list(col.aggregate([
{"$vectorSearch": {"index": "poma_vec", "path": "embedding",
"queryVector": qv, "numCandidates": 200, "limit": 10}},
{"$project": {"_id": 1, "metadata": 1, "score": {"$meta": "vectorSearchScore"}}},
]))
context = assemble(hits, volume=vol) # -> [{"file_id", "content"}, ...]POMA validates the element list's shape up front (a corrupted or mislabeled upload 422s immediately), drops Header/Footer/PageNumber elements before any document is written, splices Table elements' text_as_html so rows survive on the volume rather than in a flattened field, and rebuilds the cross-page heading tree before chunking. To force or suppress detection, set external_ocr_source to "unstructured" or "none".
Metadata mapping: POMA fields → MongoDB Atlas primitives
| POMA chunk field | Atlas primitive | What it enables |
|---|---|---|
to_embed | embedding field, covered by the Atlas Vector Search index | $vectorSearch ANN over chunkset text |
file_id | field inside metadata | $match/filter scoping; requires $project to return |
chunkset_index | field inside metadata | required by assemble(); requires $project to return |
page (from Unstructured metadata.page_number) | optional field inside metadata | page-cited answers |
depth (rebuilt from flat Title elements) | optional field inside metadata | hierarchy-aware $match filtering |
text_as_html (Unstructured Table splice) | stored in the volume document, not an indexed Atlas field | tables reach the prompt intact, never garbled by the aggregation pipeline |
Frequently asked questions
How do I get Unstructured.io elements into MongoDB Atlas for RAG?
Save the element list with elements_to_json, run PrimeCut().ingest() (auto-detected, hierarchy rebuilt), insert each chunkset with an Atlas-indexed embedding field and a metadata field for file_id/chunkset_index. Query with $vectorSearch plus $project, then hand the result to assemble().
Why does POMA assemble() return nothing from my MongoDB Atlas plus Unstructured pipeline?
It's Atlas's aggregation contract, not an Unstructured issue: $vectorSearch must be followed by a $project that explicitly includes metadata. Unprojected fields are dropped by the pipeline, so assemble() gets nothing back even from a correctly matched search.
What MongoDB Atlas fields should Unstructured chunk data carry?
At minimum an Atlas-indexed embedding field and a metadata field holding file_id/chunkset_index, always projected. Add page (from page_number) and depth inside metadata for citation and hierarchy filtering.
Do Unstructured's HTML tables survive into MongoDB Atlas retrieval?
Yes, via the volume, not the indexed document — POMA splices text_as_html into the chunkset content stored on the volume, so assemble() returns the table intact instead of the aggregation pipeline flattening or truncating it.
Does this pipeline require MongoDB Atlas specifically, or does self-hosted MongoDB work too?
Atlas specifically — $vectorSearch and the Atlas Search vector index type don't exist on self-hosted community MongoDB. The PrimeCut and vektoria ingest steps are unaffected; only the retrieval query needs Atlas.
Related recipes
Same parser, different store: Unstructured.io → Qdrant · Unstructured.io → Pinecone · Unstructured.io → pgvector
Foundations: The Optimal Chunker for Unstructured.io · MongoDB Atlas Chunking Strategy for RAG · All pipeline recipes · RAG chunking guide