Docling to Chroma: Chunking for RAG
The short answer: Docling and Chroma are the natural local, open-source pairing — a typed document tree from a parser you run yourself, into a vector store that lives on your disk. Wired together naively — flatten, split, collection.add — the demo works and the mistake ships, because the chunking decision made at prototype time is the one that sticks. The missing link is POMA: PrimeCut().ingest() consumes the saved DoclingDocument JSON (auto-detected), emits hierarchy-preserving chunksets with where-filterable metadata, and leaves a portable .poma archive so the prototype's retrieval behavior moves to any production store unchanged.
What each end of the pipeline actually provides
Docling (IBM's open-source converter, also served via docling-serve) returns the DoclingDocument: a typed texts/tables/pictures/groups tree in which section_header items carry an explicit numeric level, tables are cell grids, and repeating page furniture is pre-isolated in the furniture group with page_header/page_footer labels. What it doesn't provide: retrieval units — its HybridChunker packs the tree into token windows, which controls length, not meaning. Details: The Optimal Chunker for Docling.
Chroma provides the fastest path from zero to a working retrieval loop: chromadb.PersistentClient(path=...) gives you a durable local collection with no server, the same API scales to client/server mode, where filters act on scalar metadata, where_document adds content conditions, and default embedding functions mean documents= is all you need to pass. What it doesn't provide: any opinion about what a document should contain. Details: The Optimal Chunks for the Best Retrieval in Chroma.
The naive wiring, and where it breaks
The common recipe is an afternoon notebook — export_to_markdown() → RecursiveCharacterTextSplitter → collection.add() — and it breaks in a pair-specific way: this is the series' most local, most open-source pipeline, which makes it the one most likely to be built as a quick prototype whose chunking then ships.
- The demo hides the damage. At notebook scale, Chroma makes fragments look retrievable — the toy queries hit. The splitter becomes the least-examined, most load-bearing line in the stack.
- Docling's structure is gone before Chroma sees it. Explicit
section_headerlevels, typed tables, and pre-isolated furniture flatten into display text — so there is nofile_id/page/depthmetadata, andwherefilters (Chroma's main retrieval lever beyond similarity) have nothing to act on. - Overlap fills the collection with near-duplicates that crowd
n_results— five results collapse into two distinct passages. - Graduation day is a rewrite. When production lands on a server-grade store, fragments chunked for the notebook must be re-chunked, re-embedded, and re-evaluated — the prototype validated nothing that survives.
The pipeline, end to end
pip install docling poma chromadbimport json
import chromadb
from docling.document_converter import DocumentConverter
from poma import PrimeCut
from poma.utils import unpack_poma_archive
# 1. Docling — local parsing, your existing conversion unchanged.
converter = DocumentConverter()
doc = converter.convert("contract.pdf").document
with open("contract.docling.json", "w") as f:
json.dump(doc.export_to_dict(), f)
# 2. The missing link — DoclingDocument tree in, chunks + chunksets out.
poma = PrimeCut() # reads POMA_API_KEY
result = poma.ingest(
"contract.docling.json", # Docling shape auto-detected
download_dir="store",
filename="contract.poma", # keep the archive — page numbers live in its chunk records
)
# A chunkset's structural metadata comes from its leaf chunk (cs.chunks is in
# document order, so the last index is the leaf). depth is on the SDK objects;
# page only exists on the archive's chunk records.
depth_of = {c.chunk_index: c.depth for c in result.chunks}
# -1 when the source has no pages (Markdown, text)
page_of = {
c["chunk_index"]: (c.get("page") if c.get("page") is not None else -1)
for c in unpack_poma_archive(poma_archive_path="store/contract.poma")["chunks"]
}
# 3. Chroma — a durable local collection; same API in client/server mode.
chroma = chromadb.PersistentClient(path="./chroma")
collection = chroma.get_or_create_collection("contracts")
# 4. Add chunksets: Chroma embeds `documents` with its default embedding
# function (or bring your own); hierarchy lands as filterable metadata.
collection.add(
documents=[cs.to_embed for cs in result.chunksets],
metadatas=[
{
"file_id": cs.file_id,
"chunkset_index": cs.chunkset_index,
"page": page_of[cs.chunks[-1]],
"depth": depth_of[cs.chunks[-1]],
}
for cs in result.chunksets
],
ids=[f"{cs.file_id}:{cs.chunkset_index}" for cs in result.chunksets],
)
# 5. Query, scoped by the hierarchy metadata.
hits = collection.query(
query_texts=["What are the early termination conditions?"],
n_results=5,
where={"file_id": result.chunksets[0].file_id},
)POMA validates the payload up front (a corrupted or mislabeled upload 422s immediately, never a silently degraded collection), honors the furniture Docling already isolated, keeps tables as HTML, and accepts docling-serve md_content envelopes via a markdown passthrough. To force or suppress detection, set external_ocr_source to "docling" or "none".
The ingest also produces a portable .poma archive — chunks, chunksets, and hierarchy metadata as the source of truth on your disk, with Chroma as one projection of it. Retrieved chunksets merge into a deduplicated, prompt-ready cheatsheet — on the reference legal-document benchmark, 337 tokens of context versus 1,542 for a recursive-splitter baseline, with zero information loss (methodology).
Metadata mapping: POMA fields → Chroma primitives
| POMA chunk field | Chroma primitive | What it enables |
|---|---|---|
to_embed | documents (embedded by the default function, or your own) | semantic retrieval over self-explanatory units |
file_id | metadata scalar → where={"file_id": ...} | per-document scoping |
page (on the archive's chunk records; take the chunkset's leaf chunk) | metadata scalar (int) → $gte/$lte filters | page-cited answers, page-range queries |
depth of the leaf chunk (from Docling section_header level) | metadata scalar (int) | exclude deep appendix material via where |
chunkset_index | metadata scalar / stable ids suffix ({file_id}:{chunkset_index}) | deterministic ordering at assembly time |
| chunkset lineage | .poma archive (portable source of truth) | cheatsheet assembly + store-to-store portability |
Frequently asked questions
How do I load Docling output into ChromaDB?
Save export_to_dict() as JSON and run PrimeCut().ingest() on it — the schema_name fingerprint routes it through the BYOCR seam. Then collection.add the chunksets: to_embed as documents (Chroma embeds them) and file_id/chunkset_index/page/depth as metadata — page read from the .poma archive's chunk records for the chunkset's leaf chunk. Query with query_texts plus a where filter.
Can I run Docling and Chroma together in a local RAG stack?
Mostly — Docling parses locally and PersistentClient stores and searches locally with no server. The chunking step runs through POMA's API and returns chunks, chunksets, and a portable .poma archive that lives on your disk. Everything you store, query, and evaluate stays local.
What Chroma metadata should Docling chunks carry?
file_id, chunkset_index, page, depth — flat scalars, exactly what Chroma metadata requires. file_id, chunkset_index, and depth come off the SDK objects; page comes from the .poma archive's chunk records, read from the chunkset's leaf chunk. depth comes from Docling's explicit section_header levels, so depth-based where filters act on detected hierarchy; where_document adds content conditions. None of it exists if the tree was flattened first.
Will a Docling plus Chroma prototype survive the move to a production vector database?
Yes, if the chunks live outside Chroma as a portable artifact. The .poma archive is the source of truth; moving to Qdrant, pgvector, Weaviate, Milvus, or Pinecone is a re-upsert of the same chunksets with the same metadata, so validated retrieval behavior carries over.
Why not just export Docling markdown, split it, and add it to Chroma?
Because the prototype's chunking ships. Flattening discards the heading levels, tables, and furniture labels Docling recovered; the demo still works at small scale, but where filters have no metadata, overlap crowds n_results with near-duplicates, and re-chunking after launch means re-embedding and re-evaluating everything.
Related recipes
Same parser, different store: Docling → Qdrant · Docling → pgvector · Docling → Milvus
Same store, different parser: Marker → Chroma · Unstructured.io → Chroma · Mistral OCR → Chroma
Foundations: The Optimal Chunker for Docling · The Optimal Chunks for Chroma · All pipeline recipes · RAG chunking guide