Source: http://www.poma-ai.com/docs/pipelines/paddleocr-vl-to-mongodb-atlas

# The Missing Link Between PaddleOCR-VL and Optimal Retrieval in MongoDB Atlas

<ByAuthor />

**The short answer:** PaddleOCR-VL gives you self-hosted, layout-aware OCR — PP-DocLayoutV3 labels every region, PaddleOCR-VL reads it, all behind an HTTP endpoint you operate yourself. MongoDB Atlas gives you vector search folded directly into the aggregation pipeline you already run. Wired together naively — flatten, split, embed, insert — they still produce mediocre RAG, because nothing in between rebuilds the document's hierarchy or respects Atlas's projection rules. The missing link is POMA: `PrimeCut().ingest()` consumes the raw layout-parsing JSON (auto-detected, either accepted shape), emits hierarchy-preserving [chunksets](/learn/chunking/chunksets), and `vektoria`'s content-free ingest pattern writes only routing metadata into Atlas while `assemble()` reconstructs prompt-ready context from a volume at query time.

## What each end of the pipeline actually provides

**PaddleOCR-VL** runs as a self-hosted pipeline — PP-DocLayoutV3 for layout detection, PaddleOCR-VL served via vLLM for reading, behind a `/layout-parsing` HTTP API you operate yourself. It returns either the PaddleX serving envelope (`result.layoutParsingResults[]`, each with a `markdown` object and a `prunedResult` block list) or a raw `save_to_json()` page dict with `parsing_res_list`: flat blocks carrying `block_label`, `block_content`, `block_id`, and `block_order`. What it doesn't provide: an opinion on what a retrieval unit should be, or a link between a `paragraph_title` on page 12 and the `doc_title` on page 1. Details: [The Optimal Chunker for PaddleOCR-VL](/optimal-chunker-paddleocr-vl).

**MongoDB Atlas** provides a `$vectorSearch` aggregation stage on an Atlas-managed vector index, compound hybrid search via Atlas Search, and ordinary document fields for `$match` filtering — all inside the aggregation pipeline you already operate. What it doesn't provide: any opinion about what a document should hold, or which fields survive to the output. It ranks whatever you embedded and returns whatever you `$project`, including nothing at all if you forget. Details: [MongoDB Atlas Chunking Strategy for RAG](/optimal-chunks-mongodb-atlas).

## The naive wiring, and where it breaks

The common recipe — call `/layout-parsing`, concatenate `block_content` sorted by `block_id` (or join `markdown.text` across pages with no page tracking), run a `RecursiveCharacterTextSplitter`, embed, `col.insert_one(...)` per fragment, then a bare `$vectorSearch` query — breaks in a pair-specific way:

- **Sorting by `block_id` instead of `block_order` scrambles reading order, and `page_index` vanishes at the same join.** `block_id` is the model's detection order, not reading order; on a multi-column page the two diverge. A pipeline that flattens the wrong field embeds a logically reordered document with no page left to cite, and any `$match` predicate is left with per-document filtering only.
- **Overlap inflates the collection with near-duplicate documents,** each embedded and indexed separately — more entries for Atlas's vector index to rank through, for no retrieval benefit.
- **The aggregation pipeline drops the exact fields `assemble()` needs.** A `$vectorSearch` stage on its own returns Atlas's default projection, not a custom `metadata` field — without an explicit `{"$project": {"_id": 1, "metadata": 1, "score": {"$meta": "vectorSearchScore"}}}` stage immediately after it, `file_id` and `chunkset_index` never make it into the result, and `assemble()` returns an empty context list from hits that scored just fine.

## The pipeline, end to end

```bash
pip install requests pymongo
```

```python
import json
import os
import requests
from pymongo import MongoClient
from poma import PrimeCut
from poma.vektoria import assemble, records_from_archive, Volume
from poma.embeddings import get_embedder

# 1. Your existing self-hosted call — unchanged. `/layout-parsing` is the
#    PaddleX-compatible endpoint your layout-server + PaddleOCR-VL stack exposes.
resp = requests.post(
    "http://layout-server:40111/layout-parsing",
    json={"file": "<base64-encoded PDF>", "fileType": 0},
)
resp.raise_for_status()
with open("contract.paddleocr-vl.json", "w") as f:
    json.dump(resp.json(), f)

# 2. The missing link — raw result JSON in, a portable .poma archive out.
client = PrimeCut()  # reads POMA_API_KEY
client.ingest("contract.paddleocr-vl.json", download_dir="archives", filename="contract.poma")

embedder = get_embedder("local:BAAI/bge-small-en-v1.5")
vol = Volume("s3://your-bucket/poma")
mongo = MongoClient(os.environ["MONGODB_ATLAS_URI"])
col = mongo["poma"]["chunksets"]

# Atlas vector index (numDimensions, similarity) is created once via the Atlas UI/API,
# named "poma_vec" here — not through the pymongo driver itself.

# 3. Content-free ingest — the collection holds only routing metadata; content lives on the volume.
records = records_from_archive("archives/contract.poma")  # -> list[Record] (id, text, payload)
for r in records:
    vol.write_doc(r.payload["file_id"], {"file_id": r.payload["file_id"],
                                          "chunks": r.payload["chunks"], "text": r.text})
    col.insert_one({
        "_id": r.id,
        "embedding": embedder.embed([r.text])[0],
        "metadata": {"file_id": r.payload["file_id"], "chunkset_index": r.payload["chunkset_index"]},
    })

# 4. $vectorSearch retrieval — $project the metadata explicitly, then assemble.
qv = embedder.embed(["early termination conditions"])[0]
hits = list(col.aggregate([
    {"$vectorSearch": {"index": "poma_vec", "path": "embedding",
                       "queryVector": qv, "numCandidates": 200, "limit": 10}},
    {"$project": {"_id": 1, "metadata": 1, "score": {"$meta": "vectorSearchScore"}}},
]))
context = assemble(hits, volume=vol)  # -> [{"file_id", "content"}, ...]
```

POMA validates the payload shape up front (a corrupted or mislabeled upload 422s immediately), prefers `markdown.text` when present and falls back to `parsing_res_list` sorted by `block_order` when it isn't, drops running furniture (`header`/`footer`/`page_number`/`aside_text_number`), and rebuilds the cross-page heading tree before chunking. Retrieved chunksets share ancestor lineage across a document — `assemble()` deduplicates that lineage into one prompt-ready cheatsheet, the same discipline that answers our reference legal-document benchmark with **337 tokens** instead of **1,542** for a recursive-splitter baseline. [Methodology here](/document-ingestion-chunking-rag).

## Metadata mapping: POMA fields → MongoDB Atlas primitives

| POMA record field | Where it lives | MongoDB Atlas primitive | What it enables |
| --- | --- | --- | --- |
| `id` (`chunkset_uuid`) | Atlas document key | `_id` | deterministic re-upsert — the same chunkset always lands on the same document |
| `text` (`to_embed`) | embedded | `embedding` field, covered by the Atlas Vector Search index | ANN search via `$vectorSearch` |
| `payload["file_id"]` | Atlas document | `metadata.file_id`, must be `$project`-ed | `$match`/filter scoping to one document |
| `payload["chunkset_index"]` | Atlas document | `metadata.chunkset_index`, must be `$project`-ed | `assemble()` content lookup on the volume |
| `payload["chunks"]` (page, depth, content, lineage) | volume only (`Volume.write_doc`) | never an Atlas field | full content and hierarchy reconstructed at `assemble()` time, content-free cluster |

## Frequently asked questions

### How do I get self-hosted PaddleOCR-VL results into MongoDB Atlas for RAG?

Save the raw `/layout-parsing` JSON, run `PrimeCut().ingest()` on it, then pull `records_from_archive(...)`, write each chunkset's content to a volume, embed its `to_embed` text, and `insert_one` the vector plus a `metadata` document (`file_id`, `chunkset_index`) into an Atlas Vector Search-indexed collection. Retrieve with `$vectorSearch` and `assemble()`.

### Why does POMA's assemble() return nothing from my MongoDB Atlas results after a PaddleOCR-VL ingest?

The aggregation pipeline drops fields you don't project — this applies to every source, PaddleOCR-VL included. A `$vectorSearch` stage must be followed by `{"$project": {"_id": 1, "metadata": 1, "score": {"$meta": "vectorSearchScore"}}}`, or `file_id`/`chunkset_index` never reach the result and `assemble()` gets nothing back.

### What MongoDB Atlas fields should PaddleOCR-VL chunksets carry?

An `embedding` field covered by the Atlas Vector Search index, plus a `metadata` document with `file_id` and `chunkset_index`. Index `file_id` as an ordinary field for `$match` scoping — page and depth stay with the chunkset content on the volume, not in the Atlas collection.

### Why does block_order matter more than block_id when writing PaddleOCR-VL chunks into MongoDB Atlas?

`block_id` is detection order, not reading order; `block_order` is. Sorting by the wrong field scrambles multi-column text before it's embedded, so the vector Atlas indexes represents a clause sequence that never occurred in that order in the source document.

### Can I keep documents on my own infrastructure through both OCR and MongoDB Atlas retrieval?

Through OCR, yes — PaddleOCR-VL runs entirely behind your own endpoint, and only the resulting JSON needs to reach POMA's API. MongoDB Atlas is a managed service, so the vector index lives there; the actual chunkset content stays on whichever volume you control.

## Related recipes

Same parser, different store: [PaddleOCR-VL → Qdrant](/pipelines/paddleocr-vl-to-qdrant) · [PaddleOCR-VL → LanceDB](/pipelines/paddleocr-vl-to-lancedb)

Foundations: [The Optimal Chunker for PaddleOCR-VL](/optimal-chunker-paddleocr-vl) · [MongoDB Atlas Chunking Strategy for RAG](/optimal-chunks-mongodb-atlas) · [All pipeline recipes](/pipelines/) · [RAG chunking guide](/guides/rag-chunking/) · [POMA chunksets](/learn/chunking/chunksets)