Skip to content

Milvus Chunking Strategy for RAG: The Optimal Chunks for Retrieval ​

The short answer: the optimal chunks for Milvus are chunksets — self-explanatory units that carry their full heading lineage — embedded from their to_embed text, stored with hierarchy metadata (file_id, page, depth) as scalar fields for filtered search, and queried hybrid dense + sparse where your Milvus version supports it. Milvus scales vector search further than almost anything else; what decides your retrieval quality is what you put into the collection. POMA's PrimeCut produces exactly this shape, and retrieved chunksets merge client-side into prompt-ready cheatsheets.

This page covers what Milvus does brilliantly, what it cannot do for you, and the collection design that gets the most out of it.

Milvus gives you search at serious scale — over whatever you feed it ​

Milvus's primitives are built for collections that keep growing:

  • Collections and partitions — a partition key hashes entities into partitions automatically, so multi-tenant and per-corpus isolation come without a collection per tenant.
  • Scalar fields with filtered search — boolean expressions (file_id == "...", depth <= 2) are applied during the ANN search, not as a post-filter afterthought.
  • A spectrum of index types — HNSW for memory-resident speed, IVF variants for smaller footprints, DiskANN for corpora that outgrow RAM.
  • Sparse vectors and BM25 hybrid — recent Milvus versions host sparse term vectors next to dense embeddings and fuse both in one hybrid request.
  • Zilliz Cloud — the same API, managed.

None of these primitives, however, know anything about your documents. Milvus retrieves the nearest neighbors of what you embedded. If what you embedded is a context-free fragment — a paragraph cut loose from the section that gives it meaning — Milvus will faithfully retrieve context-free fragments, at any scale, with excellent latency. The ceiling on retrieval quality is set before the first entity is inserted.

What "optimal chunks" means for Milvus, concretely ​

Four properties separate a Milvus collection that answers questions from one that returns trivia:

  1. Every entity is self-explanatory. A chunkset is a root-to-leaf path through the document — Master Services Agreement → Termination clauses → Early termination → "Either party may…". Retrieved alone, it still tells the LLM (and the human reading citations) exactly where it sits. A bare 512-token window does not. See POMA chunksets.
  2. No overlap, no near-duplicates. Overlap is a patch for splitters that cut mid-thought — and it embeds every overlapped span twice. That inflates the entity count (and with it HNSW build time, IVF training, or DiskANN's disk footprint) and fills top-k with near-identical hits. Chunksets carry context structurally, so overlap is unnecessary and your limit=5 returns five different candidates.
  3. Hierarchy in scalar fields. file_id, chunkset_index, page, depth as scalar fields turn Milvus filter expressions into document-aware operations: scope a query to one contract, exclude appendices by depth, cite pages in the answer. Promote file_id (or tenant_id) to the partition key and the same field also drives partition routing.
  4. Hybrid where it counts. Legal, financial, and technical corpora are full of exact tokens (§ numbers, SKUs, defined terms) that dense embeddings smear. On recent Milvus versions, a sparse BM25 vector in the same collection catches them, and one hybrid request fuses both result sets.

Chunksets into Milvus with pymilvus ​

There is no shortcut integration to hide here — the point is that POMA's output maps one-to-one onto a clean Milvus schema:

python
import os

from poma import PrimeCut
from poma.utils import unpack_poma_archive
from pymilvus import DataType, MilvusClient

# 1. Chunk any document — PDF, DOCX, HTML, or a bring-your-own-OCR result JSON.
#    Keep the .poma archive: page numbers live on its chunk records.
client = PrimeCut()  # reads POMA_API_KEY
result = client.ingest("contract.pdf", download_dir="store", filename="contract.poma")

# Hierarchy lookups by chunk index. The SDK exposes depth; page comes from
# the archive's chunk records.
depth_of = {c.chunk_index: c.depth for c in result.chunks}
archive = unpack_poma_archive(poma_archive_path="store/contract.poma")
# -1 when the source has no pages (Markdown, text)
page_of = {c["chunk_index"]: (c.get("page") if c.get("page") is not None else -1) for c in archive["chunks"]}

# 2. A chunkset collection: one dense vector + hierarchy scalar fields.
milvus = MilvusClient(
    uri=os.environ["MILVUS_URI"],
    token=os.environ["MILVUS_TOKEN"],
)

schema = MilvusClient.create_schema(auto_id=True)
schema.add_field("pk", DataType.INT64, is_primary=True)
schema.add_field("vector", DataType.FLOAT_VECTOR, dim=384)
schema.add_field("text", DataType.VARCHAR, max_length=65535)
schema.add_field("file_id", DataType.VARCHAR, max_length=128)
schema.add_field("page", DataType.INT64)
schema.add_field("depth", DataType.INT64)
schema.add_field("chunkset_index", DataType.INT64)

index_params = milvus.prepare_index_params()
index_params.add_index(field_name="vector", index_type="HNSW", metric_type="COSINE")

milvus.create_collection(
    collection_name="chunksets",
    schema=schema,
    index_params=index_params,
)

# 3. Embed each chunkset's normalized to_embed text and insert.
#    `cs.chunks` holds chunk indices in document order; the last is the leaf.
milvus.insert(
    collection_name="chunksets",
    data=[
        {
            "vector": embed(cs.to_embed),  # your embedding model
            "text": cs.to_embed,
            "file_id": cs.file_id,
            "chunkset_index": cs.chunkset_index,
            "page": page_of[cs.chunks[-1]],
            "depth": depth_of[cs.chunks[-1]],
        }
        for cs in result.chunksets
    ],
)

# 4. Filtered search: scope the ANN query to one document.
hits = milvus.search(
    collection_name="chunksets",
    data=[embed("What does the contract say about early termination?")],
    filter='file_id == "contract.pdf"',
    limit=5,
    output_fields=["text", "file_id", "chunkset_index", "page", "depth"],
)

On recent Milvus versions, add a SPARSE_FLOAT_VECTOR field with a BM25 function on the same schema and issue a hybrid request that fuses dense and sparse rankings — the chunkset design does not change, only the query does. For multi-tenant deployments, declare file_id (or a tenant_id field) with is_partition_key=True and Milvus routes filtered queries to the right partitions automatically.

The last step is client-side: retrieved chunksets share ancestors, so deduplicate the shared lineage and merge the hits into one coherent context block — a cheatsheet — before it goes into the prompt. On one reference legal document that assembly meant 337 tokens of context versus 1,542 for a recursive-splitter baseline, with nothing lost. One document is an illustration, not a benchmark; the ingestion guide has the broader numbers.

Chunk shapes in Milvus, compared ​

PropertyFixed 512-token chunks + overlapSemantic chunksPOMA chunksets
Entity is self-explanatory when retrieved alone✗Sometimes✓ (by construction)
Near-duplicate vectors in the indexMany (overlap)FewNone
Hierarchy in scalar fields for filtered searchManualManual✓ (file_id, page, depth)
Partition-key multi-tenancy fits naturallyRetrofitRetrofit✓ (file_id/tenant_id scalar)
Tables survive embedding✗ (cut mid-row)Fragile✓ (HTML, never cut)
Fits DiskANN-scale corpora without bloatIndex inflated by overlapModerate✓ (overlap-free, fewer entities)
Post-retrieval assemblyConcatenate raw hitsConcatenate raw hitsCheatsheets (deduped lineage)

Frequently asked questions ​

What is the optimal chunk size for Milvus? ​

There is no universal token count — the optimal unit is an entity that is self-explanatory without its neighbors. Chunksets achieve that at any size, which is why they beat fixed 512-token windows: every entity can be retrieved alone and still be understood. If you must fix a size, start at 512 tokens — but the size knob cannot fix missing context.

What scalar fields should I add to a Milvus collection for RAG? ​

file_id, chunkset_index, page, depth, plus a VARCHAR field for the chunk text. page comes from the chunk records in the .poma archive (the chunkset’s leaf chunk) — the SDK objects expose depth but not page. Milvus applies filter expressions like file_id == "contract-001" or depth <= 2 during the ANN search itself, enabling per-document scoping, page citations, and depth-aware re-ranking without a second lookup.

Should I use hybrid search in Milvus for RAG? ​

For most document corpora, yes. Recent Milvus versions host sparse BM25-style vectors next to dense embeddings in the same collection and fuse both in one hybrid request. Sparse catches exact terms dense embeddings blur; dense catches paraphrases. One to_embed text feeds both.

How do I make a Milvus collection multi-tenant? ​

Declare a scalar field (tenant_id, or file_id for per-document isolation) as the partition key at collection creation. Milvus hashes entities into partitions automatically and routes filtered queries to only the relevant ones — isolation and lower latency without a collection per tenant.

Which Milvus index type should I use for RAG — HNSW, IVF, or DiskANN? ​

HNSW for memory-resident speed and recall; IVF variants for smaller footprints; DiskANN when the corpus outgrows RAM and NVMe-resident indexing is worth a modest latency cost. Overlap-free chunksets keep the entity count down, which makes every one of the three cheaper.

How do I turn Milvus search results into prompt-ready context? ​

Merge, don't concatenate: deduplicate the shared ancestor lineage across retrieved chunksets and assemble one coherent block — a cheatsheet — client-side. On one reference legal document that assembly answered with 337 tokens of context instead of 1,542.One document is an illustration, not a benchmark; the ingestion guide has the broader numbers.

Retrieval with vektoria's assemble() ​

POMA's poma.vektoria package keeps the collection content-free: Milvus stores only (id, vector, {file_id, chunkset_index}), while the actual chunk content lives on a volume (a path, s3://, or gs:// URL). After your normal Milvus search, hand the raw result to assemble():

python
from poma.vektoria import assemble

res = client.search("poma", data=[qv], limit=10, output_fields=["file_id", "chunkset_index"])
context = assemble(res, volume="s3://your-bucket/poma")  # -> [{"file_id", "content"}, ...]

assemble() auto-detects Milvus's result shape (one of 12 supported stores) and returns deduplicated cheatsheets. The one requirement: output_fields=["file_id", "chunkset_index"] on the search, or Milvus won't return them for assemble() to work with.

Feed Milvus from the parser you already run ​

Pair this page with your ingestion side — the same chunksets, straight from your OCR/parsing tool's raw output:

Running a different store? The same chunk design applies: Azure AI Search · Chroma · Elasticsearch · FAISS · LanceDB · MongoDB Atlas · OpenSearch · pgvector · Pinecone · Qdrant · Redis · Turbopuffer · Vespa · Weaviate — or browse all pipeline recipes.

Fundamentals: RAG chunking guide · RAG architecture guide · POMA chunksets.