Unstructured.io to Milvus: Chunking for RAG
The short answer: Unstructured.io gives you a typed, ordered element list; Milvus gives you filtered vector search at serious scale — partitions with automatic key routing, scalar filter expressions, and your pick of HNSW, IVF, or DiskANN. Wired together naively — one entity per element, or flatten-and-split — the pipeline underuses both, because nothing in between rebuilds hierarchy or populates the scalar fields Milvus filters on. The missing link is POMA: PrimeCut().ingest() consumes the raw elements_to_json output (auto-detected), emits hierarchy-preserving chunksets, and each chunkset lands in Milvus with file_id, chunkset_index, page, and depth as scalar fields ready for filter expressions and partition-key routing.
What each end of the pipeline actually provides
Unstructured.io (the open-source unstructured library and the hosted API) returns a flat, ordered list of typed elements — Title, NarrativeText, ListItem, Table, Image — each with its text, a stable element_id, and metadata such as page_number, text_as_html for tables, and optionally image_base64. What it doesn't provide: hierarchy (the list is flat — nothing records which Title nests under which) or retrieval units; the built-in by_title and basic strategies emit flat fragments. Details: The Optimal Chunker for Unstructured.io.
Milvus provides collections with typed schemas, partitions with automatic partition-key routing for multi-tenancy, scalar fields queried through filter expressions (file_id == "...", page <= 10), index types from in-memory HNSW through IVF to disk-resident DiskANN, and — in recent versions — sparse vectors with built-in BM25 for hybrid search. What it doesn't provide: any opinion about what an entity should contain. Empty scalar fields filter nothing; fragment vectors retrieve fragments. Details: The Optimal Chunks for the Best Retrieval in Milvus.
The naive wiring, and where it breaks
The tempting recipe — one Milvus entity per element (element_id even looks like a primary key), or join all element texts and split — breaks in a pair-specific way:
- The fixed schema fights the element list. A Milvus
VARCHARfield declares amax_lengthup front, but Unstructured elements vary from a three-wordTitleto atext_as_htmltable that dwarfs any budget you picked. Per-element inserts either fail on the long tail or silently truncate tables mid-row — precisely the elements that hold the numbers your users ask about. - Furniture becomes vectors.
Header,Footer, andPageNumberelements recur on every page with near-identical text. Embedded per element, they become dozens of near-duplicate vectors that match everything vaguely and crowd top-k in every partition of the collection. - The scalar fields end up empty. Flatten-and-split destroys
metadata.page_numberand the element typing at its first step, sopageanddepthhave nothing to hold — and Milvus's headline feature, filtered search during ANN, has nothing to filter on.
POMA's chunksets fix all three at the source: furniture is dropped, tables are spliced whole, and every chunkset is a self-explanatory root-to-leaf unit carrying its hierarchy metadata, with no overlap. On a notoriously hard reference legal document, chunksets plus cheatsheet assembly delivered the answer in 337 tokens of retrieved context versus 1,542 for a recursive character splitter, with zero information loss (methodology).
The pipeline, end to end
pip install 'unstructured[pdf]' poma pymilvus sentence-transformersimport os
from poma import PrimeCut
from poma.utils import unpack_poma_archive
from pymilvus import DataType, MilvusClient
from sentence_transformers import SentenceTransformer
from unstructured.partition.pdf import partition_pdf
from unstructured.staging.base import elements_to_json
# 1. Unstructured — your existing call, unchanged.
elements = partition_pdf(
filename="contract.pdf",
strategy="hi_res",
infer_table_structure=True, # populates metadata.text_as_html
extract_image_block_types=["Image"],
extract_image_block_to_payload=True, # inline base64 → POMA describes figures
)
elements_to_json(elements, filename="contract.unstructured.json")
# 2. The missing link — raw element list in, chunks + chunksets out.
poma = PrimeCut() # reads POMA_API_KEY
result = poma.ingest( # Unstructured shape auto-detected
"contract.unstructured.json",
download_dir="store",
filename="contract.poma", # keep the archive — page numbers live in it
)
# Depth comes off the SDK objects; page is on the archive's chunk records only.
depth_of = {c.chunk_index: c.depth for c in result.chunks}
page_of = {
c["chunk_index"]: c.get("page")
for c in unpack_poma_archive(poma_archive_path="store/contract.poma")["chunks"]
}
# 3. Milvus — a schema whose scalar fields mirror the hierarchy metadata.
client = MilvusClient(uri=os.environ["MILVUS_URI"])
schema = MilvusClient.create_schema()
schema.add_field("pk", DataType.INT64, is_primary=True, auto_id=True)
schema.add_field("vector", DataType.FLOAT_VECTOR, dim=384)
schema.add_field("text", DataType.VARCHAR, max_length=65535)
schema.add_field("file_id", DataType.VARCHAR, max_length=256, is_partition_key=True)
schema.add_field("page", DataType.INT64)
schema.add_field("depth", DataType.INT64)
schema.add_field("chunkset_index", DataType.INT64)
index_params = client.prepare_index_params()
index_params.add_index(field_name="vector", index_type="HNSW", metric_type="COSINE")
client.create_collection("contracts", schema=schema, index_params=index_params)
# 4. Embed chunksets and insert — hierarchy metadata rides along as scalars.
model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
client.insert(
collection_name="contracts",
data=[
{
"vector": model.encode(cs.to_embed).tolist(),
"text": cs.to_embed,
"file_id": cs.file_id,
"chunkset_index": cs.chunkset_index,
# chunk indices are in document order; the last one is the leaf
"page": page_of[cs.chunks[-1]],
"depth": depth_of[cs.chunks[-1]],
}
for cs in result.chunksets
],
)
# 5. Filtered search — scope to one document, then assemble the cheatsheet.
query = "What are the early termination conditions?"
hits = client.search(
collection_name="contracts",
data=[model.encode(query).tolist()],
limit=5,
filter='file_id == "contract.unstructured.json"',
output_fields=["text", "chunkset_index", "page", "depth"],
)
ordered = sorted(hits[0], key=lambda h: h["entity"]["chunkset_index"])
cheatsheet = "\n\n".join(dict.fromkeys(h["entity"]["text"] for h in ordered))POMA validates the payload shape up front (fingerprint: elements carrying type + element_id; a corrupted or mislabeled upload 422s immediately), groups elements by page_number, splices each Table's text_as_html, describes inline image_base64 figures (disk-bound image_path refs are neutralized and counted in content_metadata), drops Header/Footer/PageNumber furniture, and rebuilds the cross-page heading tree before chunking. To force or suppress detection, set external_ocr_source to "unstructured" or "none".
Note that the single text field with a generous max_length holds because chunkset text is normalized prose plus spliced HTML — POMA has already bounded the units. On recent Milvus versions, add a SPARSE_FLOAT_VECTOR field with a BM25 function to the same schema and issue a hybrid request that fuses dense and sparse rankings; the chunkset design does not change, only the query does.
Metadata mapping: POMA fields → Milvus primitives
| POMA chunk field | Milvus primitive | What it enables |
|---|---|---|
to_embed | FLOAT_VECTOR field (add a sparse/BM25 field for hybrid) | paraphrase + exact-term retrieval |
file_id | VARCHAR scalar field, partition key | per-document filters, automatic multi-tenant routing |
page of the chunkset's leaf chunk (from the archive's chunk records, filled from Unstructured metadata.page_number) | INT64 scalar field | page-cited answers, page <= N filter expressions |
depth of the leaf chunk | INT64 scalar field | filter/re-rank by hierarchy level |
chunkset_index | INT64 scalar field | stable ordering at cheatsheet assembly |
chunkset text (with spliced text_as_html) | VARCHAR field in output_fields | assembly without a second store |
Frequently asked questions
How do I get Unstructured.io elements into Milvus for RAG?
Save the element list with elements_to_json, run PrimeCut().ingest() on the raw JSON (auto-detected, hierarchy rebuilt), embed each chunkset's to_embed, and insert entities whose scalar fields carry file_id, chunkset_index, page, and depth — page and depth taken from the chunkset's leaf chunk, with page read out of the .poma archive's chunk records. Retrieve with a filtered search and assemble cheatsheets client-side.
Why not insert one Milvus entity per Unstructured element?
Milvus's fixed VARCHAR max_length collides with text_as_html tables — inserts fail or truncate mid-row — and recurring Header/Footer/PageNumber elements become near-identical vectors crowding top-k in every partition. Chunksets drop the furniture and splice tables whole.
Can Milvus filter expressions use Unstructured's page numbers?
Yes — POMA carries metadata.page_number through to every chunk record in the .poma archive; read it back with unpack_poma_archive and write the chunkset's leaf-chunk page into an INT64 scalar field, where page <= 10 filters during the ANN search itself. Flatten-and-split destroys the page number before it ever reaches the schema.
Should file_id be a Milvus partition key for Unstructured-parsed corpora?
Yes, when one collection serves many documents or tenants: declare it with is_partition_key=True and filtered searches only touch matching partitions. POMA populates file_id on every chunkset, so the key always has a value to route on.
Does Milvus hybrid BM25 search help with Unstructured.io content?
Yes — recent Milvus versions fuse sparse BM25 and dense rankings in one hybrid request. Part numbers, clause references, and values inside text_as_html tables are exact tokens dense embeddings blur, and POMA's intact-HTML splicing keeps them present for BM25 to match.
Related recipes
Same parser, different store: Unstructured.io → Qdrant · Unstructured.io → pgvector · Unstructured.io → Chroma
Same store, different parser: Mistral OCR → Milvus · Marker → Milvus · Azure Document Intelligence → Milvus
Foundations: The Optimal Chunker for Unstructured.io · The Optimal Chunks for Milvus · All pipeline recipes · RAG chunking guide