Skip to content

Marker to Chroma: Chunking for RAG ​

The short answer: Marker and Chroma are the natural local pair — a PDF parser that runs on your own hardware and a vector store that embeds in your process with a PersistentClient. Wired together naively — flatten the markdown, split, collection.add — the prototype works, and that's the trap: the chunking decision made at prototype time is the one that sticks. The missing link is POMA: PrimeCut().ingest() consumes the saved Marker JSON (the Document block tree auto-routes), emits hierarchy-preserving chunksets, and the portable .poma archive means the same retrieval units move unchanged from your Chroma prototype to whatever runs in production.

What each end of the pipeline actually provides ​

Marker (by Datalab / VikParuchuri) is the fast open-source PDF→markdown favourite for local pipelines. With the JSON renderer (--output_format json) it returns a Document block tree: a root block_type: "Document" with Page children, per-block HTML content, tables as full <table> elements with row and column spans, and a side-channel images dict of {name: base64}. What it doesn't provide: cross-page hierarchy or retrieval units. Details: The Optimal Chunker for Marker.

Chroma is local-first: embedded in your process via PersistentClient or run client/server, with collections, where metadata filters and where_document content filters, and a default embedding function so add works with zero configuration — or bring your own. What it doesn't provide: any opinion about what an entry should contain. It retrieves nearest neighbors of whatever you gave it. Details: The Optimal Chunks for the Best Retrieval in Chroma.

The naive wiring, and where it breaks ​

The common recipe — rendered.markdown → RecursiveCharacterTextSplitter → collection.add(documents=fragments) — breaks in a pair-specific way:

  • Non-scalar metadata blows up add. Chroma metadata values must be strings, numbers, or booleans. Marker's run metadata object and its images dict of base64 bytes are neither — pipelines that pass them through fail, or strip them and lose the figures entirely.
  • where filters have nothing to filter on. Marker's markdown output has no page boundaries, so where={"page": {"$gte": 30}} can never work — the metadata was unfillable before the first query.
  • Dangling ![](name) refs become embedded noise. The default embedding function faithfully embeds image-file names that point at bytes left behind in the side channel.
  • The prototype's fragments become the product's fragments. Chroma makes it effortless to ship the first thing that works — and splitter fragments wired directly to Chroma's API have no existence outside that collection. When production wants Qdrant or pgvector, there is nothing coherent to migrate.

The pipeline, end to end ​

bash
pip install marker-pdf poma chromadb
marker_single contract.pdf --output_format json --output_dir out/
python
import chromadb
from poma import PrimeCut

# 1. The missing link — raw Marker JSON in, chunks + chunksets out.
poma = PrimeCut()  # reads POMA_API_KEY
result = poma.ingest("out/contract/contract.json")  # Document block tree auto-detected

# 2. Chroma — embedded, persistent, zero infra. The default embedding
#    function embeds the documents locally on add.
client = chromadb.PersistentClient(path="./chroma")
collection = client.get_or_create_collection(
    name="marker_chunksets",
    metadata={"hnsw:space": "cosine"},
)

# 3. One add call: chunkset text as the document, flat scalar metadata.
leaves = [cs.chunks[-1] for cs in result.chunksets]  # leaf chunk per chunkset
collection.add(
    ids=[f"{leaf.file_id}:{leaf.chunk_index}" for leaf in leaves],
    documents=[cs.to_embed for cs in result.chunksets],
    metadatas=[
        {
            "file_id": leaf.file_id,
            "page": leaf.page,
            "depth": leaf.depth,
            "chunk_index": leaf.chunk_index,
        }
        for leaf in leaves
    ],
)

# 4. Retrieve — metadata-filtered, fully local.
hits = collection.query(
    query_texts=["What are the early termination conditions?"],
    n_results=5,
    where={"$and": [
        {"file_id": {"$eq": "contract"}},
        {"depth": {"$gte": 2}},
    ]},
)

Deduplicate and merge the retrieved chunksets into a cheatsheet — one prompt-ready context block — before handing them to the LLM. POMA validates the upload shape up front (a corrupted or mislabeled payload 422s immediately), splices Marker's side-channel image bytes into described text, strips running headers and footers, and rebuilds the cross-page heading tree before chunking. Bare markdown shapes ({markdown, images, metadata}) are not auto-routed — declare external_ocr_source="marker", or "none" to opt out of detection.

The payoff, measured on our reference legal-document benchmark: 337 tokens of retrieved context instead of 1,542 for a recursive character splitter, zero information loss. Methodology: Document Ingestion & Chunking for RAG.

Metadata mapping: POMA fields → Chroma primitives ​

POMA chunk fieldChroma primitiveWhat it enables
to_embeddocument (embedded by the collection's embedding function)retrieval over self-explanatory chunkset text
file_idmetadata string, where {"$eq": …}scope queries to one document
page (from Marker's Page blocks)metadata int, where {"$gte"/"$lte": …}page-cited answers, page-range filters
depthmetadata int, where filterablefilter or re-rank by hierarchy level
chunk_indexmetadata intstable ordering at assembly time
chunkset lineagethe document text itself + the .poma archiveself-contained hits, portability beyond Chroma

The portability angle: prototype in Chroma, keep your options ​

Chroma's sweet spot is the same as Marker's: everything on your machine, nothing to provision. The risk is that prototype decisions harden. Vectors are disposable — you can always re-embed — but chunks are not: they encode what your retrieval units are. If those units are ad-hoc splitter fragments living only inside a Chroma collection, every future migration re-opens the chunking question from scratch.

The .poma archive breaks that lock-in. It is the durable, database-independent record of the ingestion — chunks, chunksets, hierarchy — so the Chroma prototype and the eventual production store are two views of the same source of truth. PrimeCut().ingest() once; collection.add today; upsert to Qdrant or INSERT into pgvector tomorrow, from the same archive, with identical retrieval units.

Frequently asked questions ​

How do I get Marker output into Chroma for RAG? ​

Run Marker with the JSON renderer, hand the saved result to PrimeCut().ingest() (auto-detected, hierarchy rebuilt), then chromadb.PersistentClient → get_or_create_collection → one collection.add with chunkset to_embed as documents and file_id/page/depth/chunk_index as flat metadata. Fully local, end to end.

Should I use Chroma's default embedding function for Marker output? ​

For prototyping, yes — zero configuration, runs locally. What it embeds matters more: chunkset text, not flattened markdown full of dangling image refs and cut tables. Swap in a stronger model later and re-embed the same chunksets.

How do I filter Chroma queries by page or document for Marker-parsed PDFs? ​

Attach file_id, page, depth, chunk_index as metadata and use where filters ($eq, $gte, $and, …). The prerequisite: run Marker's JSON renderer, because the markdown output has no page boundaries to carry.

Can I move a Chroma prototype built on Marker output to another vector database? ​

Yes — keep the .poma archive as the source of truth. Moving to Qdrant, pgvector, or Milvus is re-embedding and re-upserting the same chunksets; retrieval quality carries over because the retrieval units are identical.

Why does Chroma reject my Marker metadata? ​

Chroma metadata must be scalar (string, number, boolean); Marker's run metadata object and base64 images dict are not. Let POMA consume them first — image bytes become described text, and the emitted chunk fields are flat scalars that map straight onto Chroma metadata.

Same parser, different store: Marker → Qdrant · Marker → pgvector · Marker → Milvus

Same store, different parser: Docling → Chroma · Unstructured → Chroma · Mistral OCR → Chroma

Foundations: The Optimal Chunker for Marker · The Optimal Chunks for Chroma · All pipeline recipes · RAG chunking guide