Source: http://www.poma-ai.com/docs/pipelines/marker-to-milvus

# The Missing Link Between Marker and Optimal Retrieval in Milvus

<ByAuthor />

**The short answer:** Marker gives you fast, local PDF→structure; Milvus gives you filtered ANN at scale — partitions, scalar-field filters, and dense+sparse hybrid search. Wired together naively — flatten the markdown, split, embed, insert — the pairing underdelivers: page identity is gone before the first scalar field is written, and the collection fills with near-duplicate overlap vectors. The missing link is POMA: `PrimeCut().ingest()` consumes the saved Marker JSON (the Document block tree auto-routes), emits hierarchy-preserving [chunksets](/learn/chunking/chunksets), and those map cleanly onto Milvus's primitives — scalar fields for hierarchy, a partition key for scoping, dense plus BM25 sparse vectors for hybrid retrieval.

## What each end of the pipeline actually provides

**Marker** (by Datalab / VikParuchuri) is the fast open-source PDF→markdown favourite for local pipelines. With the JSON renderer (`--output_format json`) it returns a Document block tree: a root `block_type: "Document"` with Page children, per-block HTML content, tables as full `<table>` elements with row and column spans, and a side-channel `images` dict of `{name: base64}`. What it doesn't provide: cross-page hierarchy or retrieval units — display markdown and retrieval units are different artifacts. Details: [The Optimal Chunker for Marker](/optimal-chunker-marker).

**Milvus** provides collections with typed scalar fields and filtered search, partitions with automatic partition-key routing, index types from HNSW to IVF to DiskANN, and — in recent versions (sparse vectors since 2.4, built-in BM25 in 2.5) — dense+sparse hybrid search in one collection, self-hosted or on Zilliz Cloud. What it doesn't provide: any opinion about what a row should contain. It retrieves nearest neighbors of whatever you embedded. Details: [The Optimal Chunks for the Best Retrieval in Milvus](/optimal-chunks-milvus).

## The naive wiring, and where it breaks

The common recipe — `rendered.markdown` → `RecursiveCharacterTextSplitter` → embed → `client.insert(...)` — breaks in a pair-specific way:

- **The `page` scalar field is unfillable.** Marker's markdown output is one flat string with no page boundaries, so the very field Milvus filtered search is built around has nothing to hold. Filters degrade to `file_id` only; page-cited answers are impossible.
- **Span-heavy tables collide with `max_length`.** Marker's JSON output carries tables as complete HTML with row/colspans — long strings by design. A VARCHAR field sized for prose fragments either rejects the insert or forces truncation that silently amputates the table's tail rows.
- **Overlap pollutes the index.** Splitter overlap embeds every boundary span twice; whether the index is HNSW, IVF, or DiskANN, top-k results fill with near-duplicates of the same hit, crowding out the passage that actually answers.
- **Dangling `![](name)` refs** from Marker's side-channel images dict get embedded as noise — the figures themselves never reach the collection.

## The pipeline, end to end

```bash
pip install marker-pdf poma pymilvus sentence-transformers
marker_single contract.pdf --output_format json --output_dir out/
```

```python
import os
from pymilvus import (
    MilvusClient, DataType, Function, FunctionType,
    AnnSearchRequest, RRFRanker,
)
from sentence_transformers import SentenceTransformer
from poma import PrimeCut

# 1. The missing link — raw Marker JSON in, chunks + chunksets out.
poma = PrimeCut()  # reads POMA_API_KEY
result = poma.ingest("out/contract/contract.json")  # Document block tree auto-detected

# 2. Collection: scalar fields for hierarchy, partition key for scoping,
#    dense + BM25 sparse vectors for hybrid search.
client = MilvusClient(uri=os.environ["MILVUS_URI"], token=os.environ.get("MILVUS_TOKEN", ""))

schema = client.create_schema(auto_id=True)
schema.add_field("id", DataType.INT64, is_primary=True)
schema.add_field("file_id", DataType.VARCHAR, max_length=256, is_partition_key=True)
schema.add_field("page", DataType.INT64)
schema.add_field("depth", DataType.INT64)
schema.add_field("chunk_index", DataType.INT64)
schema.add_field("to_embed", DataType.VARCHAR, max_length=65535, enable_analyzer=True)
schema.add_field("dense", DataType.FLOAT_VECTOR, dim=384)
schema.add_field("sparse", DataType.SPARSE_FLOAT_VECTOR)
schema.add_function(Function(
    name="bm25",
    input_field_names=["to_embed"],
    output_field_names=["sparse"],
    function_type=FunctionType.BM25,
))

index_params = client.prepare_index_params()
index_params.add_index("dense", index_type="HNSW", metric_type="COSINE")
index_params.add_index("sparse", index_type="SPARSE_INVERTED_INDEX", metric_type="BM25")
client.create_collection("marker_chunksets", schema=schema, index_params=index_params)

# 3. Embed and insert — one row per chunkset, hierarchy in scalar fields.
model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
rows = []
for cs, emb in zip(result.chunksets,
                   model.encode([cs.to_embed for cs in result.chunksets])):
    leaf = cs.chunks[-1]  # the leaf chunk; ancestors are its breadcrumbs
    rows.append({
        "file_id": leaf.file_id,
        "page": leaf.page,
        "depth": leaf.depth,
        "chunk_index": leaf.chunk_index,
        "to_embed": cs.to_embed,
        "dense": emb,
    })
client.insert("marker_chunksets", rows)

# 4. Hybrid retrieval — dense + BM25 sparse, fused by RRF, scalar-filtered.
query = "What are the early termination conditions?"
hits = client.hybrid_search(
    "marker_chunksets",
    reqs=[
        AnnSearchRequest(data=[model.encode(query).tolist()],
                         anns_field="dense", param={}, limit=10),
        AnnSearchRequest(data=[query], anns_field="sparse", param={}, limit=10),
    ],
    ranker=RRFRanker(),
    limit=5,
    filter='file_id == "contract"',
    output_fields=["to_embed", "page", "depth", "chunk_index"],
)
```

Deduplicate and merge the retrieved chunksets into a cheatsheet — one prompt-ready context block — before handing them to the LLM. POMA validates the upload shape up front (a corrupted or mislabeled payload 422s immediately), splices Marker's side-channel image bytes into described text, strips running headers and footers, and rebuilds the cross-page heading tree before chunking. Bare markdown shapes (`{markdown, images, metadata}`) are not auto-routed — declare `external_ocr_source="marker"`, or `"none"` to opt out of detection.

The payoff, measured on our reference legal-document benchmark: **337 tokens** of retrieved context instead of **1,542** for a recursive character splitter, zero information loss. Methodology: [Document Ingestion & Chunking for RAG](/document-ingestion-chunking-rag).

## Metadata mapping: POMA fields → Milvus primitives

| POMA chunk field | Milvus primitive | What it enables |
| --- | --- | --- |
| `to_embed` | `FLOAT_VECTOR` (HNSW) + `SPARSE_FLOAT_VECTOR` via BM25 function | dense + lexical hybrid in one `hybrid_search` |
| `file_id` | `VARCHAR` scalar field, **partition key** | automatic per-document routing, scoped search |
| `page` (from Marker's Page blocks) | `INT64` scalar field | filter expressions, page-cited answers |
| `depth` | `INT64` scalar field | filter or re-rank by hierarchy level |
| `chunk_index` | `INT64` scalar field | stable ordering at assembly time |
| chunkset lineage | `to_embed` carries the root-to-leaf text itself | self-explanatory hits, client-side cheatsheet assembly |

## Choosing the index — and why it matters less than the chunks

HNSW is the default choice for chunkset corpora: memory-resident, latency-sensitive workloads are its home ground. IVF variants reduce memory at the cost of recall tuning; DiskANN serves corpora too large for RAM. But no index can compensate for what you feed it — overlap-based splitting puts near-identical vectors into every one of them, and the graph faithfully returns the duplicates. Overlap-free chunksets fix this at the source: every vector in the collection represents a distinct, self-explanatory passage.

## Frequently asked questions

### How do I get Marker output into Milvus for RAG?

Run Marker with the JSON renderer, hand the saved result to `PrimeCut().ingest()` (auto-detected, hierarchy rebuilt), embed each chunkset's `to_embed`, and insert rows with `file_id`, `page`, `depth`, and `chunk_index` scalar fields next to the dense vector. Retrieve with `hybrid_search` plus a scalar filter.

### Which Milvus index should I use for Marker chunksets — HNSW, IVF, or DiskANN?

Start with HNSW for memory-resident, latency-sensitive corpora; IVF for lower memory; DiskANN for corpora beyond RAM. The upstream choice matters more: overlap-free chunksets keep near-duplicate vectors out of whichever index you pick.

### How should I use Milvus partitions for Marker-parsed documents?

Declare `file_id` (or a tenant field) as the partition key — Milvus routes each document's chunksets automatically, and filtered searches touch only the relevant partitions. Ideal for the batch-converted local document sets Marker users typically have.

### Does Milvus BM25 hybrid search work with Marker output?

Yes — recent Milvus versions (sparse since 2.4, built-in BM25 in 2.5) derive a BM25 sparse vector from the text field via a declared function. Parsed documents are full of exact tokens that dense embeddings blur; RRF fusion catches both signals.

### Why do Marker tables get truncated or rejected when inserting into Milvus?

Milvus VARCHAR fields need a declared `max_length`, and Marker's HTML tables with row/colspans are long by design. Size the field generously and keep tables whole — POMA never cuts a table across chunks, so the grid arrives intact.

## Related recipes

Same parser, different store: [Marker → Qdrant](/pipelines/marker-to-qdrant) · [Marker → pgvector](/pipelines/marker-to-pgvector) · [Marker → Chroma](/pipelines/marker-to-chroma)

Same store, different parser: [Mistral OCR → Milvus](/pipelines/mistral-ocr-to-milvus) · [Docling → Milvus](/pipelines/docling-to-milvus) · [Textract → Milvus](/pipelines/textract-to-milvus)

Foundations: [The Optimal Chunker for Marker](/optimal-chunker-marker) · [The Optimal Chunks for Milvus](/optimal-chunks-milvus) · [All pipeline recipes](/pipelines/) · [RAG chunking guide](/guides/rag-chunking/)