Skip to content

Marker to Milvus: Chunking for RAG ​

The short answer: Marker gives you fast, local PDF→structure; Milvus gives you filtered ANN at scale — partitions, scalar-field filters, and dense+sparse hybrid search. Wired together naively — flatten the markdown, split, embed, insert — the pairing underdelivers: page identity is gone before the first scalar field is written, and the collection fills with near-duplicate overlap vectors. The missing link is POMA: PrimeCut().ingest() consumes the saved Marker JSON (the Document block tree auto-routes), emits hierarchy-preserving chunksets, and those map cleanly onto Milvus's primitives — scalar fields for hierarchy, a partition key for scoping, dense plus BM25 sparse vectors for hybrid retrieval.

What each end of the pipeline actually provides ​

Marker (by Datalab / VikParuchuri) is the fast open-source PDF→markdown favourite for local pipelines. With the JSON renderer (--output_format json) it returns a Document block tree: a root block_type: "Document" with Page children, per-block HTML content, tables as full <table> elements with row and column spans, and a side-channel images dict of {name: base64}. What it doesn't provide: cross-page hierarchy or retrieval units — display markdown and retrieval units are different artifacts. Details: The Optimal Chunker for Marker.

Milvus provides collections with typed scalar fields and filtered search, partitions with automatic partition-key routing, index types from HNSW to IVF to DiskANN, and — in recent versions (sparse vectors since 2.4, built-in BM25 in 2.5) — dense+sparse hybrid search in one collection, self-hosted or on Zilliz Cloud. What it doesn't provide: any opinion about what a row should contain. It retrieves nearest neighbors of whatever you embedded. Details: The Optimal Chunks for the Best Retrieval in Milvus.

The naive wiring, and where it breaks ​

The common recipe — rendered.markdown → RecursiveCharacterTextSplitter → embed → client.insert(...) — breaks in a pair-specific way:

  • The page scalar field is unfillable. Marker's markdown output is one flat string with no page boundaries, so the very field Milvus filtered search is built around has nothing to hold. Filters degrade to file_id only; page-cited answers are impossible.
  • Span-heavy tables collide with max_length. Marker's JSON output carries tables as complete HTML with row/colspans — long strings by design. A VARCHAR field sized for prose fragments either rejects the insert or forces truncation that silently amputates the table's tail rows.
  • Overlap pollutes the index. Splitter overlap embeds every boundary span twice; whether the index is HNSW, IVF, or DiskANN, top-k results fill with near-duplicates of the same hit, crowding out the passage that actually answers.
  • Dangling ![](name) refs from Marker's side-channel images dict get embedded as noise — the figures themselves never reach the collection.

The pipeline, end to end ​

bash
pip install marker-pdf poma pymilvus sentence-transformers
marker_single contract.pdf --output_format json --output_dir out/
python
import os
from pymilvus import (
    MilvusClient, DataType, Function, FunctionType,
    AnnSearchRequest, RRFRanker,
)
from sentence_transformers import SentenceTransformer
from poma import PrimeCut

# 1. The missing link — raw Marker JSON in, chunks + chunksets out.
poma = PrimeCut()  # reads POMA_API_KEY
result = poma.ingest("out/contract/contract.json")  # Document block tree auto-detected

# 2. Collection: scalar fields for hierarchy, partition key for scoping,
#    dense + BM25 sparse vectors for hybrid search.
client = MilvusClient(uri=os.environ["MILVUS_URI"], token=os.environ.get("MILVUS_TOKEN", ""))

schema = client.create_schema(auto_id=True)
schema.add_field("id", DataType.INT64, is_primary=True)
schema.add_field("file_id", DataType.VARCHAR, max_length=256, is_partition_key=True)
schema.add_field("page", DataType.INT64)
schema.add_field("depth", DataType.INT64)
schema.add_field("chunk_index", DataType.INT64)
schema.add_field("to_embed", DataType.VARCHAR, max_length=65535, enable_analyzer=True)
schema.add_field("dense", DataType.FLOAT_VECTOR, dim=384)
schema.add_field("sparse", DataType.SPARSE_FLOAT_VECTOR)
schema.add_function(Function(
    name="bm25",
    input_field_names=["to_embed"],
    output_field_names=["sparse"],
    function_type=FunctionType.BM25,
))

index_params = client.prepare_index_params()
index_params.add_index("dense", index_type="HNSW", metric_type="COSINE")
index_params.add_index("sparse", index_type="SPARSE_INVERTED_INDEX", metric_type="BM25")
client.create_collection("marker_chunksets", schema=schema, index_params=index_params)

# 3. Embed and insert — one row per chunkset, hierarchy in scalar fields.
model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
rows = []
for cs, emb in zip(result.chunksets,
                   model.encode([cs.to_embed for cs in result.chunksets])):
    leaf = cs.chunks[-1]  # the leaf chunk; ancestors are its breadcrumbs
    rows.append({
        "file_id": leaf.file_id,
        "page": leaf.page,
        "depth": leaf.depth,
        "chunk_index": leaf.chunk_index,
        "to_embed": cs.to_embed,
        "dense": emb,
    })
client.insert("marker_chunksets", rows)

# 4. Hybrid retrieval — dense + BM25 sparse, fused by RRF, scalar-filtered.
query = "What are the early termination conditions?"
hits = client.hybrid_search(
    "marker_chunksets",
    reqs=[
        AnnSearchRequest(data=[model.encode(query).tolist()],
                         anns_field="dense", param={}, limit=10),
        AnnSearchRequest(data=[query], anns_field="sparse", param={}, limit=10),
    ],
    ranker=RRFRanker(),
    limit=5,
    filter='file_id == "contract"',
    output_fields=["to_embed", "page", "depth", "chunk_index"],
)

Deduplicate and merge the retrieved chunksets into a cheatsheet — one prompt-ready context block — before handing them to the LLM. POMA validates the upload shape up front (a corrupted or mislabeled payload 422s immediately), splices Marker's side-channel image bytes into described text, strips running headers and footers, and rebuilds the cross-page heading tree before chunking. Bare markdown shapes ({markdown, images, metadata}) are not auto-routed — declare external_ocr_source="marker", or "none" to opt out of detection.

The payoff, measured on our reference legal-document benchmark: 337 tokens of retrieved context instead of 1,542 for a recursive character splitter, zero information loss. Methodology: Document Ingestion & Chunking for RAG.

Metadata mapping: POMA fields → Milvus primitives ​

POMA chunk fieldMilvus primitiveWhat it enables
to_embedFLOAT_VECTOR (HNSW) + SPARSE_FLOAT_VECTOR via BM25 functiondense + lexical hybrid in one hybrid_search
file_idVARCHAR scalar field, partition keyautomatic per-document routing, scoped search
page (from Marker's Page blocks)INT64 scalar fieldfilter expressions, page-cited answers
depthINT64 scalar fieldfilter or re-rank by hierarchy level
chunk_indexINT64 scalar fieldstable ordering at assembly time
chunkset lineageto_embed carries the root-to-leaf text itselfself-explanatory hits, client-side cheatsheet assembly

Choosing the index — and why it matters less than the chunks ​

HNSW is the default choice for chunkset corpora: memory-resident, latency-sensitive workloads are its home ground. IVF variants reduce memory at the cost of recall tuning; DiskANN serves corpora too large for RAM. But no index can compensate for what you feed it — overlap-based splitting puts near-identical vectors into every one of them, and the graph faithfully returns the duplicates. Overlap-free chunksets fix this at the source: every vector in the collection represents a distinct, self-explanatory passage.

Frequently asked questions ​

How do I get Marker output into Milvus for RAG? ​

Run Marker with the JSON renderer, hand the saved result to PrimeCut().ingest() (auto-detected, hierarchy rebuilt), embed each chunkset's to_embed, and insert rows with file_id, page, depth, and chunk_index scalar fields next to the dense vector. Retrieve with hybrid_search plus a scalar filter.

Which Milvus index should I use for Marker chunksets — HNSW, IVF, or DiskANN? ​

Start with HNSW for memory-resident, latency-sensitive corpora; IVF for lower memory; DiskANN for corpora beyond RAM. The upstream choice matters more: overlap-free chunksets keep near-duplicate vectors out of whichever index you pick.

How should I use Milvus partitions for Marker-parsed documents? ​

Declare file_id (or a tenant field) as the partition key — Milvus routes each document's chunksets automatically, and filtered searches touch only the relevant partitions. Ideal for the batch-converted local document sets Marker users typically have.

Does Milvus BM25 hybrid search work with Marker output? ​

Yes — recent Milvus versions (sparse since 2.4, built-in BM25 in 2.5) derive a BM25 sparse vector from the text field via a declared function. Parsed documents are full of exact tokens that dense embeddings blur; RRF fusion catches both signals.

Why do Marker tables get truncated or rejected when inserting into Milvus? ​

Milvus VARCHAR fields need a declared max_length, and Marker's HTML tables with row/colspans are long by design. Size the field generously and keep tables whole — POMA never cuts a table across chunks, so the grid arrives intact.

Same parser, different store: Marker → Qdrant · Marker → pgvector · Marker → Chroma

Same store, different parser: Mistral OCR → Milvus · Docling → Milvus · Textract → Milvus

Foundations: The Optimal Chunker for Marker · The Optimal Chunks for Milvus · All pipeline recipes · RAG chunking guide