Source: http://www.poma-ai.com/docs/pipelines/marker-to-mongodb-atlas

# The Missing Link Between Marker and Optimal Retrieval in MongoDB Atlas

<ByAuthor />

**The short answer:** Marker gives you fast, local PDF parsing with real recovered structure; MongoDB Atlas gives you vector search folded directly into the aggregation pipeline you already run. Wired together naively — flatten the markdown, split, embed, insert, then `$vectorSearch` with no follow-up stage — the pipeline drops every figure before Atlas sees a document, and the aggregation stage silently drops metadata it wasn't told to keep. The missing link is POMA: `PrimeCut().ingest()` consumes the saved Marker JSON (the Document block tree auto-detects), splices the side-channel image bytes back in, and rebuilds the cross-page hierarchy into [chunksets](/learn/chunking/chunksets). `vektoria` keeps Atlas content-free — embeddings and routing metadata only — and `assemble()` turns a correctly-projected `$vectorSearch` result into prompt-ready context.

## What each end of the pipeline actually provides

**Marker** (by Datalab / VikParuchuri) runs on your own hardware and turns PDFs into clean structured output. With the JSON renderer (`--output_format json`) you get a Document block tree — Page children, per-block HTML, full `<table>` elements with row and column spans. What it doesn't provide: cross-page hierarchy, retrieval units, or inline images — figure bytes live in a side-channel `images` dict keyed by name, with only `![](name)` references left in the text. Details: [The Optimal Chunker for Marker](/optimal-chunker-marker).

**MongoDB Atlas** provides a `$vectorSearch` aggregation stage on an Atlas-managed vector index, combinable with `$match` or Atlas Search's compound query type for hybrid, over documents you already model in your own collections. What it doesn't provide: any opinion about what a document should hold, or a `$project` by default — the aggregation pipeline drops any field you didn't explicitly ask for, metadata included. Details: [MongoDB Atlas Chunking Strategy for RAG](/optimal-chunks-mongodb-atlas).

## The naive wiring, and where it breaks

The common recipe — run Marker's default markdown output, `RecursiveCharacterTextSplitter`, embed, `insert_one` with only an `embedding` field, then `$vectorSearch` with no follow-up stage — breaks in a way specific to this pair:

- **Every figure vanishes before Atlas.** Marker's images live in the side-channel `images` dict; the markdown carries only `![](name)` references. A markdown-only pipeline embeds those dangling references as literal text, so the chart that answers a question was never inserted as a document at all.
- **`$vectorSearch` with no `$project` returns bare IDs.** MongoDB's aggregation pipeline only returns fields you explicitly project; a query that ends at `$vectorSearch` and skips a metadata `$project` hands back scores with no way to know which document or chunkset a hit belongs to — `assemble()` gets a non-empty result with nothing extractable in it.
- **Page numbers never make it into a filterable field.** Marker's default markdown output has no page boundaries, so there's no ordinary document field to `$match` on for a page-scoped query — the JSON renderer's Page blocks are the only source for one.
- **Overlap inflates the collection.** Splitter overlap embeds every boundary span twice, and Atlas's vector index cost scales with document count like any other collection index.

## The pipeline, end to end

```bash
# 1. Marker — your existing run, unchanged. The JSON renderer keeps the
#    Document block tree that POMA auto-detects, pages included.
marker_single contract.pdf --output_format json --output_dir out/
```

```python
import os
from pymongo import MongoClient
from poma import PrimeCut
from poma.vektoria import assemble, records_from_archive, Volume
from poma.embeddings import get_embedder

# 2. The missing link — raw Marker JSON in, .poma archive out (chunks +
#    chunksets, image bytes spliced back into their references).
client = PrimeCut()  # reads POMA_API_KEY
client.ingest("out/contract/contract.json", download_dir="archives", filename="contract.poma")

embedder = get_embedder("local:BAAI/bge-small-en-v1.5")
vol = Volume("s3://your-bucket/poma")
mongo = MongoClient(os.environ["MONGODB_ATLAS_URI"])
col = mongo["poma"]["chunksets"]

# Atlas vector index (numDimensions, similarity) is created once via the
# Atlas UI/API, named "poma_vec" here — not through the pymongo driver itself.

# 3. Content-free ingest — Atlas holds only routing metadata; content lives on the volume.
records = records_from_archive("archives/contract.poma")  # -> list[Record] (id, text, payload)
for r in records:
    vol.write_doc(r.payload["file_id"], {"file_id": r.payload["file_id"],
                                          "chunks": r.payload["chunks"], "text": r.text})
    col.insert_one({
        "_id": r.id,
        "embedding": embedder.embed([r.text])[0],
        "metadata": {"file_id": r.payload["file_id"], "chunkset_index": r.payload["chunkset_index"]},
    })

# 4. $vectorSearch retrieval — $project the metadata explicitly, then assemble.
qv = embedder.embed(["early termination conditions"])[0]
hits = list(col.aggregate([
    {"$vectorSearch": {"index": "poma_vec", "path": "embedding",
                       "queryVector": qv, "numCandidates": 200, "limit": 10}},
    {"$project": {"_id": 1, "metadata": 1, "score": {"$meta": "vectorSearchScore"}}},
]))
context = assemble(hits, volume=vol)  # -> [{"file_id", "content"}, ...]
```

POMA validates the payload shape up front — the JSON Document block tree is the strict auto-route fingerprint, and a corrupted or mislabeled upload 422s immediately instead of degrading the index. If what you saved is one of Marker's bare markdown shapes, declare it explicitly with `external_ocr_source="marker"`. On our reference legal document, this chunking answers the same query with **337 tokens** of retrieved context instead of **1,542** for a recursive character splitter — methodology in [Document Ingestion & Chunking for RAG](/document-ingestion-chunking-rag).

## Metadata mapping: POMA fields → MongoDB Atlas primitives

| POMA chunk field | MongoDB Atlas primitive | What it enables |
| --- | --- | --- |
| `to_embed` | `embedding` field, covered by the `$vectorSearch` index | similarity search over the embedded chunkset |
| `file_id` | `metadata.file_id`, ordinary document field | `$match`/filter scoping to one document |
| `chunkset_index` | `metadata.chunkset_index` | `assemble()` lookup key, stable ordering |
| `page` (JSON renderer only) | volume document, not a field | content-free retrieval, cheatsheet assembly |
| chunkset content | not in the document — lives on a `Volume` | small documents, reassembled at retrieval |

## Frequently asked questions

### How do I get Marker output into MongoDB Atlas for RAG?

Run Marker with the JSON renderer and save the result; hand it to `PrimeCut().ingest()`, which auto-detects the Document block tree, splices images back in, and writes a `.poma` archive of chunks and chunksets. Pull records with `records_from_archive()`, embed each record's `to_embed` text, and insert one document per record with an `embedding` field plus a `metadata` field holding `file_id` and `chunkset_index`. Query with `$vectorSearch`, `$project` the metadata explicitly, and hand the aggregation result to `assemble()`.

### Do Marker's extracted images make it into MongoDB Atlas?

Only if something reunites them first. Marker parks figure bytes in a side-channel `images` dict and leaves bare `![](name)` references in the markdown, so a pipeline that embeds only the markdown inserts documents with zero figures represented in the vector field. POMA splices the bytes back into their references as data URIs and describes each figure into searchable text before `embedder.embed()` ever runs, so the figure becomes a real embedded document. References without bytes are neutralized and counted, never silently dropped.

### Why does POMA assemble return nothing from my MongoDB Atlas results after ingesting Marker output?

Almost always a missing `$project`. The `$vectorSearch` aggregation stage must be followed by a `$project` that explicitly includes the `metadata` field holding `file_id` and `chunkset_index` — MongoDB's aggregation pipeline drops any field you didn't ask for, unlike a plain `find()`. If the ingest step wrote metadata correctly but the query stage doesn't project it, `assemble()` gets a non-empty result with nothing to extract, and comes back empty.

### What MongoDB Atlas fields should Marker chunks carry?

An `embedding` field covered by the Atlas vector index, and a `metadata` field holding `file_id` and `chunkset_index` — the same fields `records_from_archive()` hands you after ingesting Marker's JSON block tree. Page numbers as an ordinary indexed field only exist if you ran Marker's JSON renderer; the default markdown output has no page boundaries to store at all.

### Does Marker's JSON renderer matter for a MongoDB Atlas pipeline?

Yes. The JSON renderer's Document block tree is what POMA auto-detects and what preserves per-page structure, which is the only way a page field ends up in your Atlas documents at all; Marker's default markdown output loses page boundaries entirely. The JSON renderer also keeps full HTML tables intact instead of collapsing them into one flat string, which matters once that content is deduplicated into a cheatsheet at retrieval time.

## Related recipes

Same parser, different store: [Marker → Weaviate](/pipelines/marker-to-weaviate) · [Marker → pgvector](/pipelines/marker-to-pgvector) · [Marker → Milvus](/pipelines/marker-to-milvus)

Foundations: [The Optimal Chunker for Marker](/optimal-chunker-marker) · [MongoDB Atlas Chunking Strategy for RAG](/optimal-chunks-mongodb-atlas) · [All pipeline recipes](/pipelines/) · [RAG chunking guide](/guides/rag-chunking/)