Skip to content

PaddleOCR-VL to Chroma: Chunking for RAG ​

The short answer: PaddleOCR-VL gives you self-hosted, layout-aware OCR behind your own HTTP endpoint; Chroma gives you the fastest path from zero to working retrieval — an embedded, persistent vector store with metadata filtering and pluggable embedding functions. Wired together naively — flatten, split, embed — the prototype works just well enough to ship its worst decision everywhere, and inconsistent handling of PaddleOCR-VL's two accepted shapes across batch workers compounds it with a metadata schema that's incomplete for a silent subset of documents. The missing link is POMA: PrimeCut().ingest() consumes the raw layout-parsing JSON (auto-detected, either shape, every time), emits hierarchy-preserving chunksets, and each lands in Chroma with file_id, page, depth, and chunkset_index metadata ready for where filters — with the .poma archive as a portable source of truth when the prototype graduates.

What each end of the pipeline actually provides ​

PaddleOCR-VL runs self-hosted — PP-DocLayoutV3 for layout detection, PaddleOCR-VL served via vLLM for reading, behind a /layout-parsing HTTP API you operate. It returns either the PaddleX serving envelope (result.layoutParsingResults[], with a markdown object plus a prunedResult block list) or a raw save_to_json() page dict (parsing_res_list: flat blocks with block_label, block_content, block_id, block_order) — both legitimate, both liable to appear across a multi-worker batch pipeline. Details: The Optimal Chunker for PaddleOCR-VL.

Chroma provides a local-first store that runs embedded or client/server, collections with default or bring-your-own embedding functions, where filters over metadata, and where_document filters over content. What it doesn't provide: any opinion about what a document entry should contain, or metadata it wasn't given consistently. where={"file_id": ...} matches nothing for an entry that never received one. Details: The Optimal Chunks for the Best Retrieval in Chroma.

The naive wiring, and where it breaks ​

The common recipe — call /layout-parsing, hand-roll a block-to-text converter, split, collection.add(documents=windows) — breaks in a pair-specific way:

  • Inconsistent shape handling ships an inconsistent schema. A self-hosted PaddleOCR-VL deployment commonly runs several batch workers; if one script only expects the serving envelope's markdown object and another only expects raw parsing_res_list dicts, the Chroma collection ends up with some entries carrying a page integer and others carrying none — where filters then silently drop the incomplete subset with no error raised.
  • The prototype decision gets baked in. Chroma is where RAG pipelines are born; the chunking chosen in hour one is rarely revisited, so a hand-rolled flattener's gaps travel unnoticed into the production database it eventually ports to.
  • Retrieved fragments arrive without lineage, so the LLM answers out of context — a failure that looks like "Chroma isn't good enough" when the actual defect is upstream, in how the two PaddleOCR-VL shapes were merged.

POMA's chunksets fix this at the source: one chunker, both shapes, the same hierarchy metadata every time. On a notoriously hard reference legal document, chunksets plus cheatsheet assembly delivered the answer in 337 tokens of retrieved context versus 1,542 for a recursive character splitter, with zero information loss (methodology).

The pipeline, end to end ​

bash
pip install requests poma chromadb
python
import json
import requests
import chromadb
from poma import PrimeCut
from poma.utils import unpack_poma_archive

# 1. Your existing self-hosted call — unchanged. `/layout-parsing` is the
#    PaddleX-compatible endpoint your layout-server + PaddleOCR-VL stack exposes.
resp = requests.post(
    "http://layout-server:40111/layout-parsing",
    json={"file": "<base64-encoded PDF>", "fileType": 0},
)
resp.raise_for_status()
with open("contract.paddleocr-vl.json", "w") as f:
    json.dump(resp.json(), f)

# 2. The missing link — raw result JSON in, chunks + chunksets out.
poma = PrimeCut()  # reads POMA_API_KEY
# keep the .poma archive: page numbers live on its chunk records, not on the SDK objects
result = poma.ingest(
    "contract.paddleocr-vl.json",  # PaddleOCR-VL shape auto-detected
    download_dir="store",
    filename="contract.poma",
)

# Chunk lookups: depth from the SDK objects, page from the archive's chunk records.
depth_of = {c.chunk_index: c.depth for c in result.chunks}
page_of = {
    c["chunk_index"]: c.get("page")
    for c in unpack_poma_archive(poma_archive_path="store/contract.poma")["chunks"]
}

# 3. Chroma — persistent local store; the collection's embedding
#    function (default or your own) embeds the documents on add.
chroma = chromadb.PersistentClient(path="./chroma")
collection = chroma.get_or_create_collection(name="contracts")

# 4. Add chunksets — documents, stable IDs, where-filterable metadata.
leaves = [cs.chunks[-1] for cs in result.chunksets]  # document order; last index is the leaf
collection.add(
    ids=[f"{cs.file_id}:{cs.chunkset_index}" for cs in result.chunksets],
    documents=[cs.to_embed for cs in result.chunksets],
    metadatas=[
        {
            "file_id": cs.file_id,
            "page": page_of[leaf],
            "depth": depth_of[leaf],
            "chunkset_index": cs.chunkset_index,
        }
        for cs, leaf in zip(result.chunksets, leaves)
    ],
)

# 5. Query with a metadata filter, then assemble the cheatsheet.
res = collection.query(
    query_texts=["What are the early termination conditions?"],
    n_results=5,
    where={"file_id": "contract.paddleocr-vl.json"},
)
pairs = sorted(
    zip(res["documents"][0], res["metadatas"][0]),
    key=lambda p: p[1]["chunkset_index"],
)
cheatsheet = "\n\n".join(dict.fromkeys(doc for doc, _ in pairs))

POMA validates the payload shape up front (a corrupted or mislabeled upload 422s immediately), prefers markdown.text when present and falls back to parsing_res_list sorted by block_order when it isn't — the same logic for every worker, every document — drops running furniture, and rebuilds the cross-page heading tree before chunking. To force or suppress detection, set external_ocr_source to "paddleocr_vl" or "none".

Prototype today, port tomorrow ​

The .poma archive that PrimeCut produces is the portable source of truth for the document — chunks, chunksets, and hierarchy metadata, independent of any database and independent of which of PaddleOCR-VL's two shapes fed it. Loading it into Chroma is one loop; loading the same archive into Qdrant, Weaviate, or Pinecone is the same loop against a different client, because file_id/page/depth map onto every store's filtering primitive. The prototype and the production system disagree only about infrastructure, never about content — no re-running the self-hosted OCR call, no re-chunk, no drift between what the demo retrieved and what production retrieves.

Metadata mapping: POMA fields → Chroma primitives ​

POMA chunk fieldChroma primitiveWhat it enables
to_embeddocuments entry, embedded by the collection's embedding functionsemantic retrieval; where_document content filters
file_idmetadatas key, where={"file_id": ...}scope queries to one document
page (on the archive's chunk records, from PaddleOCR-VL page_index, 1-based; take the chunkset's leaf chunk)metadatas key, where with range operatorspage-cited answers, page-range filters
depth (of the chunkset's leaf chunk)metadatas keyfilter/re-rank by hierarchy level
chunkset_indexmetadatas key + part of the entry idstable IDs, ordering, independent of block_id
chunkset lineage.poma archive (portable source of truth)identical reload into any other vector DB

Frequently asked questions ​

How do I get self-hosted PaddleOCR-VL results into Chroma for RAG? ​

Save the raw /layout-parsing JSON, run PrimeCut().ingest() on it (auto-detected, hierarchy rebuilt), then collection.add() with each chunkset's to_embed as the document and file_id/page/depth/chunkset_index metadata. page comes from the archive's chunk records (the chunkset's leaf chunk) — the SDK objects expose depth but not page. Query with where filters and assemble cheatsheets client-side.

What goes wrong when different workers process PaddleOCR-VL's two shapes inconsistently? ​

If one worker's script only handles the serving envelope and another only handles raw save_to_json dicts, the resulting collection has a schema gap — some entries lack page — and where filters silently exclude that subset without an error.

What metadata should Chroma store for PaddleOCR-VL chunks so where filters work? ​

file_id, page (from page_index, read off the archive's chunk records), depth, and chunkset_index. POMA writes all four consistently no matter which of the two accepted shapes produced a given chunk.

Will a Chroma prototype built on self-hosted PaddleOCR-VL chunks port to a production vector database? ​

The .poma archive reloads into Qdrant, Weaviate, or Pinecone without re-running OCR or re-chunking. A hand-rolled flattener's schema gaps also port — which is why routing both shapes through one chunker from day one matters.

Does staying self-hosted through Chroma matter for data locality? ​

Yes — PaddleOCR-VL and Chroma can both run entirely on infrastructure you control. Only the layout-parsing result JSON needs to reach POMA's chunking API.

Same parser, different store: PaddleOCR-VL → Qdrant · PaddleOCR-VL → Pinecone · PaddleOCR-VL → Weaviate

Same store, different parser: Mistral OCR → Chroma · LlamaParse → Chroma · Docling → Chroma

Foundations: The Optimal Chunker for PaddleOCR-VL · The Optimal Chunks for Chroma · All pipeline recipes · RAG chunking guide