Skip to content

Docling to Chroma: Chunking for RAG ​

The short answer: Docling and Chroma are the natural local, open-source pairing — a typed document tree from a parser you run yourself, into a vector store that lives on your disk. Wired together naively — flatten, split, collection.add — the demo works and the mistake ships, because the chunking decision made at prototype time is the one that sticks. The missing link is POMA: PrimeCut().ingest() consumes the saved DoclingDocument JSON (auto-detected), emits hierarchy-preserving chunksets with where-filterable metadata, and leaves a portable .poma archive so the prototype's retrieval behavior moves to any production store unchanged.

What each end of the pipeline actually provides ​

Docling (IBM's open-source converter, also served via docling-serve) returns the DoclingDocument: a typed texts/tables/pictures/groups tree in which section_header items carry an explicit numeric level, tables are cell grids, and repeating page furniture is pre-isolated in the furniture group with page_header/page_footer labels. What it doesn't provide: retrieval units — its HybridChunker packs the tree into token windows, which controls length, not meaning. Details: The Optimal Chunker for Docling.

Chroma provides the fastest path from zero to a working retrieval loop: chromadb.PersistentClient(path=...) gives you a durable local collection with no server, the same API scales to client/server mode, where filters act on scalar metadata, where_document adds content conditions, and default embedding functions mean documents= is all you need to pass. What it doesn't provide: any opinion about what a document should contain. Details: The Optimal Chunks for the Best Retrieval in Chroma.

The naive wiring, and where it breaks ​

The common recipe is an afternoon notebook — export_to_markdown() → RecursiveCharacterTextSplitter → collection.add() — and it breaks in a pair-specific way: this is the series' most local, most open-source pipeline, which makes it the one most likely to be built as a quick prototype whose chunking then ships.

  • The demo hides the damage. At notebook scale, Chroma makes fragments look retrievable — the toy queries hit. The splitter becomes the least-examined, most load-bearing line in the stack.
  • Docling's structure is gone before Chroma sees it. Explicit section_header levels, typed tables, and pre-isolated furniture flatten into display text — so there is no file_id/page/depth metadata, and where filters (Chroma's main retrieval lever beyond similarity) have nothing to act on.
  • Overlap fills the collection with near-duplicates that crowd n_results — five results collapse into two distinct passages.
  • Graduation day is a rewrite. When production lands on a server-grade store, fragments chunked for the notebook must be re-chunked, re-embedded, and re-evaluated — the prototype validated nothing that survives.

The pipeline, end to end ​

bash
pip install docling poma chromadb
python
import json

import chromadb
from docling.document_converter import DocumentConverter
from poma import PrimeCut
from poma.utils import unpack_poma_archive

# 1. Docling — local parsing, your existing conversion unchanged.
converter = DocumentConverter()
doc = converter.convert("contract.pdf").document
with open("contract.docling.json", "w") as f:
    json.dump(doc.export_to_dict(), f)

# 2. The missing link — DoclingDocument tree in, chunks + chunksets out.
poma = PrimeCut()  # reads POMA_API_KEY
result = poma.ingest(
    "contract.docling.json",  # Docling shape auto-detected
    download_dir="store",
    filename="contract.poma",  # keep the archive — page numbers live in its chunk records
)

# A chunkset's structural metadata comes from its leaf chunk (cs.chunks is in
# document order, so the last index is the leaf). depth is on the SDK objects;
# page only exists on the archive's chunk records.
depth_of = {c.chunk_index: c.depth for c in result.chunks}
# -1 when the source has no pages (Markdown, text)
page_of = {
    c["chunk_index"]: (c.get("page") if c.get("page") is not None else -1)
    for c in unpack_poma_archive(poma_archive_path="store/contract.poma")["chunks"]
}

# 3. Chroma — a durable local collection; same API in client/server mode.
chroma = chromadb.PersistentClient(path="./chroma")
collection = chroma.get_or_create_collection("contracts")

# 4. Add chunksets: Chroma embeds `documents` with its default embedding
#    function (or bring your own); hierarchy lands as filterable metadata.
collection.add(
    documents=[cs.to_embed for cs in result.chunksets],
    metadatas=[
        {
            "file_id": cs.file_id,
            "chunkset_index": cs.chunkset_index,
            "page": page_of[cs.chunks[-1]],
            "depth": depth_of[cs.chunks[-1]],
        }
        for cs in result.chunksets
    ],
    ids=[f"{cs.file_id}:{cs.chunkset_index}" for cs in result.chunksets],
)

# 5. Query, scoped by the hierarchy metadata.
hits = collection.query(
    query_texts=["What are the early termination conditions?"],
    n_results=5,
    where={"file_id": result.chunksets[0].file_id},
)

POMA validates the payload up front (a corrupted or mislabeled upload 422s immediately, never a silently degraded collection), honors the furniture Docling already isolated, keeps tables as HTML, and accepts docling-serve md_content envelopes via a markdown passthrough. To force or suppress detection, set external_ocr_source to "docling" or "none".

The ingest also produces a portable .poma archive — chunks, chunksets, and hierarchy metadata as the source of truth on your disk, with Chroma as one projection of it. Retrieved chunksets merge into a deduplicated, prompt-ready cheatsheet — on the reference legal-document benchmark, 337 tokens of context versus 1,542 for a recursive-splitter baseline, with zero information loss (methodology).

Metadata mapping: POMA fields → Chroma primitives ​

POMA chunk fieldChroma primitiveWhat it enables
to_embeddocuments (embedded by the default function, or your own)semantic retrieval over self-explanatory units
file_idmetadata scalar → where={"file_id": ...}per-document scoping
page (on the archive's chunk records; take the chunkset's leaf chunk)metadata scalar (int) → $gte/$lte filterspage-cited answers, page-range queries
depth of the leaf chunk (from Docling section_header level)metadata scalar (int)exclude deep appendix material via where
chunkset_indexmetadata scalar / stable ids suffix ({file_id}:{chunkset_index})deterministic ordering at assembly time
chunkset lineage.poma archive (portable source of truth)cheatsheet assembly + store-to-store portability

Frequently asked questions ​

How do I load Docling output into ChromaDB? ​

Save export_to_dict() as JSON and run PrimeCut().ingest() on it — the schema_name fingerprint routes it through the BYOCR seam. Then collection.add the chunksets: to_embed as documents (Chroma embeds them) and file_id/chunkset_index/page/depth as metadata — page read from the .poma archive's chunk records for the chunkset's leaf chunk. Query with query_texts plus a where filter.

Can I run Docling and Chroma together in a local RAG stack? ​

Mostly — Docling parses locally and PersistentClient stores and searches locally with no server. The chunking step runs through POMA's API and returns chunks, chunksets, and a portable .poma archive that lives on your disk. Everything you store, query, and evaluate stays local.

What Chroma metadata should Docling chunks carry? ​

file_id, chunkset_index, page, depth — flat scalars, exactly what Chroma metadata requires. file_id, chunkset_index, and depth come off the SDK objects; page comes from the .poma archive's chunk records, read from the chunkset's leaf chunk. depth comes from Docling's explicit section_header levels, so depth-based where filters act on detected hierarchy; where_document adds content conditions. None of it exists if the tree was flattened first.

Will a Docling plus Chroma prototype survive the move to a production vector database? ​

Yes, if the chunks live outside Chroma as a portable artifact. The .poma archive is the source of truth; moving to Qdrant, pgvector, Weaviate, Milvus, or Pinecone is a re-upsert of the same chunksets with the same metadata, so validated retrieval behavior carries over.

Why not just export Docling markdown, split it, and add it to Chroma? ​

Because the prototype's chunking ships. Flattening discards the heading levels, tables, and furniture labels Docling recovered; the demo still works at small scale, but where filters have no metadata, overlap crowds n_results with near-duplicates, and re-chunking after launch means re-embedding and re-evaluating everything.

Same parser, different store: Docling → Qdrant · Docling → pgvector · Docling → Milvus

Same store, different parser: Marker → Chroma · Unstructured.io → Chroma · Mistral OCR → Chroma

Foundations: The Optimal Chunker for Docling · The Optimal Chunks for Chroma · All pipeline recipes · RAG chunking guide