Docling to Milvus: Chunking for RAG
The short answer: Docling gives you a typed document tree with explicit heading levels; Milvus gives you vector search that keeps scaling — partitions, filtered ANN, HNSW/IVF/DiskANN, hybrid dense+sparse on recent versions. Wired together naively — flatten, split, insert — the collection scales beautifully while retrieving context-free fragments. The missing link is POMA: PrimeCut().ingest() consumes the saved DoclingDocument JSON (auto-detected), emits hierarchy-preserving chunksets, and those map one-to-one onto a Milvus schema whose scalar fields make filtered search document-aware.
What each end of the pipeline actually provides
Docling (IBM's open-source converter, also served via docling-serve) returns the DoclingDocument: a typed texts/tables/pictures/groups tree in which section_header items carry an explicit numeric level, tables are cell grids, and repeating page furniture is pre-isolated in the furniture group with page_header/page_footer labels. What it doesn't provide: retrieval units — its HybridChunker packs the tree into token windows, which controls length, not meaning. Details: The Optimal Chunker for Docling.
Milvus provides collections with a fixed schema, scalar fields whose boolean filter expressions apply during the ANN search, partition keys that hash entities into partitions automatically, a spectrum of index types (HNSW, IVF variants, DiskANN for corpora that outgrow RAM), and — on recent versions — sparse vectors with built-in BM25 fused in one hybrid request. What it doesn't provide: any opinion about entities. It retrieves the nearest neighbors of whatever you embedded. Details: The Optimal Chunks for the Best Retrieval in Milvus.
The naive wiring, and where it breaks
The common recipe — export_to_markdown() → RecursiveCharacterTextSplitter → embed → insert into a text-plus-vector collection — breaks in a pair-specific way: Milvus fixes its schema at collection creation, so chunk-time decisions crystallize into the collection.
- The schema is declared once. Flatten first and there is nothing to put in
depthorpagescalar fields — Docling's explicitsection_headerlevels became#glyphs before pymilvus ever ran. Filter expressions and partition routing exist in the API but have nothing to act on in your data. - Correcting it is not a migration, it's a rebuild. Adding the missing hierarchy later means dropping the collection, re-chunking, re-embedding every entity, and rebuilding the HNSW/IVF/DiskANN index — at exactly the corpus sizes people choose Milvus for.
- Overlap compounds at scale. Splitter overlap embeds every boundary span twice; in a store sized for hundreds of millions of entities, that is a materially larger index build and top-k results that repeat each other.
- Docling's pre-isolated furniture gets re-flattened into the text stream, so page headers and footers become entities and pollute every search.
The pipeline, end to end
pip install docling poma pymilvusimport json
import os
from docling.document_converter import DocumentConverter
from poma import PrimeCut
from poma.utils import unpack_poma_archive
from pymilvus import DataType, MilvusClient
# 1. Docling — your existing conversion, unchanged.
converter = DocumentConverter()
doc = converter.convert("contract.pdf").document
with open("contract.docling.json", "w") as f:
json.dump(doc.export_to_dict(), f)
# 2. The missing link — DoclingDocument tree in, chunks + chunksets out.
poma = PrimeCut() # reads POMA_API_KEY
result = poma.ingest(
"contract.docling.json", # Docling shape auto-detected
download_dir="store",
filename="contract.poma", # keep the archive — page numbers live in its chunk records
)
# A chunkset's structural metadata comes from its leaf chunk (cs.chunks is in
# document order, so the last index is the leaf). depth is on the SDK objects;
# page only exists on the archive's chunk records.
depth_of = {c.chunk_index: c.depth for c in result.chunks}
page_of = {
c["chunk_index"]: c.get("page")
for c in unpack_poma_archive(poma_archive_path="store/contract.poma")["chunks"]
}
# 3. Milvus — the schema is declared once, so declare the hierarchy now.
milvus = MilvusClient(uri=os.environ["MILVUS_URI"], token=os.environ["MILVUS_TOKEN"])
schema = MilvusClient.create_schema(auto_id=True)
schema.add_field("pk", DataType.INT64, is_primary=True)
schema.add_field("vector", DataType.FLOAT_VECTOR, dim=384)
schema.add_field("text", DataType.VARCHAR, max_length=65535)
schema.add_field("file_id", DataType.VARCHAR, max_length=128, is_partition_key=True)
schema.add_field("page", DataType.INT64)
schema.add_field("depth", DataType.INT64)
schema.add_field("chunkset_index", DataType.INT64)
index_params = milvus.prepare_index_params()
index_params.add_index(field_name="vector", index_type="HNSW", metric_type="COSINE")
milvus.create_collection(
collection_name="docling_chunksets",
schema=schema,
index_params=index_params,
)
# 4. Embed each chunkset's normalized to_embed text and insert.
milvus.insert(
collection_name="docling_chunksets",
data=[
{
"vector": embed(cs.to_embed), # your embedding model
"text": cs.to_embed,
"file_id": cs.file_id,
"page": page_of[cs.chunks[-1]],
"depth": depth_of[cs.chunks[-1]],
"chunkset_index": cs.chunkset_index,
}
for cs in result.chunksets
],
)
# 5. Filtered ANN: one document, top-level structure only.
hits = milvus.search(
collection_name="docling_chunksets",
data=[embed("What are the early termination conditions?")],
filter='file_id == "contract.pdf" and depth <= 3',
limit=5,
output_fields=["text", "file_id", "page", "depth", "chunkset_index"],
)POMA validates the payload up front (a corrupted or mislabeled upload 422s immediately, never a silently degraded collection), honors the furniture Docling already isolated, keeps tables as HTML, and accepts docling-serve md_content envelopes via a markdown passthrough. To force or suppress detection, set external_ocr_source to "docling" or "none". On recent Milvus versions, add a SPARSE_FLOAT_VECTOR field with BM25 on the same schema and issue a hybrid request — the chunkset design does not change, only the query does.
Retrieved chunksets merge client-side into a deduplicated, prompt-ready cheatsheet — on the reference legal-document benchmark, 337 tokens of context versus 1,542 for a recursive-splitter baseline, with zero information loss (methodology).
Metadata mapping: POMA fields → Milvus primitives
| POMA chunk field | Milvus primitive | What it enables |
|---|---|---|
to_embed | FLOAT_VECTOR (HNSW/IVF/DiskANN) + optional sparse BM25 field | dense + exact-term hybrid retrieval |
file_id | VARCHAR scalar field, is_partition_key=True | per-document filters + automatic partition routing |
page (on the archive's chunk records; take the chunkset's leaf chunk) | INT64 scalar field | page-cited answers, page-range filter expressions |
depth of the leaf chunk (from Docling section_header level) | INT64 scalar field | depth <= N filters applied during the ANN search |
chunkset_index | INT64 scalar field | stable ordering at assembly time |
| chunkset lineage | .poma archive / document store, keyed by file_id | cheatsheet assembly client-side |
Frequently asked questions
How do I get Docling output into Milvus for RAG?
Save export_to_dict() as JSON, run PrimeCut().ingest() on it (auto-detected via schema_name), create a collection with a dense vector field plus file_id/page/depth/chunkset_index scalar fields, embed each chunkset's to_embed, and insert with pymilvus. Filtered ANN then scopes queries by document, page, or depth.
What Milvus scalar fields should Docling chunks carry?
file_id (VARCHAR), page, depth, chunkset_index (INT64), next to the vector and a VARCHAR text field. depth and chunkset_index come off the SDK objects; page comes from the archive's chunk records, read from the chunkset's leaf chunk. depth comes straight from Docling's explicit section_header levels, so depth <= 2 filters act on detected structure. Declare file_id or tenant_id as the partition key.
What breaks if I flatten Docling markdown before inserting into Milvus?
The schema is fixed at collection creation: flattening discards heading levels and page provenance, so depth and page fields have nothing to hold, and filters and partition routing are inert. Fixing it later means dropping the collection, re-chunking, re-embedding, and rebuilding the index.
Should a Docling-to-Milvus pipeline use hybrid dense and sparse search?
Yes, on versions with sparse vectors and built-in BM25. Docling preserves exact tokens with high fidelity, and dense embeddings blur exactly those. One hybrid request fuses both rankings; only the query changes, not the chunkset design.
How do I isolate multiple documents or tenants in one Milvus collection of Docling output?
Declare file_id — or a tenant_id field — with is_partition_key=True. Milvus hashes entities into partitions and routes filtered queries to only the relevant ones. POMA writes file_id on every chunk and chunkset, so the key is already in the data.
Related recipes
Same parser, different store: Docling → Qdrant · Docling → pgvector · Docling → Chroma
Same store, different parser: Mistral OCR → Milvus · LlamaParse → Milvus · Unstructured.io → Milvus
Foundations: The Optimal Chunker for Docling · The Optimal Chunks for Milvus · All pipeline recipes · RAG chunking guide