Skip to content

Unstructured.io to Milvus: Chunking for RAG ​

The short answer: Unstructured.io gives you a typed, ordered element list; Milvus gives you filtered vector search at serious scale — partitions with automatic key routing, scalar filter expressions, and your pick of HNSW, IVF, or DiskANN. Wired together naively — one entity per element, or flatten-and-split — the pipeline underuses both, because nothing in between rebuilds hierarchy or populates the scalar fields Milvus filters on. The missing link is POMA: PrimeCut().ingest() consumes the raw elements_to_json output (auto-detected), emits hierarchy-preserving chunksets, and each chunkset lands in Milvus with file_id, chunkset_index, page, and depth as scalar fields ready for filter expressions and partition-key routing.

What each end of the pipeline actually provides ​

Unstructured.io (the open-source unstructured library and the hosted API) returns a flat, ordered list of typed elements — Title, NarrativeText, ListItem, Table, Image — each with its text, a stable element_id, and metadata such as page_number, text_as_html for tables, and optionally image_base64. What it doesn't provide: hierarchy (the list is flat — nothing records which Title nests under which) or retrieval units; the built-in by_title and basic strategies emit flat fragments. Details: The Optimal Chunker for Unstructured.io.

Milvus provides collections with typed schemas, partitions with automatic partition-key routing for multi-tenancy, scalar fields queried through filter expressions (file_id == "...", page <= 10), index types from in-memory HNSW through IVF to disk-resident DiskANN, and — in recent versions — sparse vectors with built-in BM25 for hybrid search. What it doesn't provide: any opinion about what an entity should contain. Empty scalar fields filter nothing; fragment vectors retrieve fragments. Details: The Optimal Chunks for the Best Retrieval in Milvus.

The naive wiring, and where it breaks ​

The tempting recipe — one Milvus entity per element (element_id even looks like a primary key), or join all element texts and split — breaks in a pair-specific way:

  • The fixed schema fights the element list. A Milvus VARCHAR field declares a max_length up front, but Unstructured elements vary from a three-word Title to a text_as_html table that dwarfs any budget you picked. Per-element inserts either fail on the long tail or silently truncate tables mid-row — precisely the elements that hold the numbers your users ask about.
  • Furniture becomes vectors. Header, Footer, and PageNumber elements recur on every page with near-identical text. Embedded per element, they become dozens of near-duplicate vectors that match everything vaguely and crowd top-k in every partition of the collection.
  • The scalar fields end up empty. Flatten-and-split destroys metadata.page_number and the element typing at its first step, so page and depth have nothing to hold — and Milvus's headline feature, filtered search during ANN, has nothing to filter on.

POMA's chunksets fix all three at the source: furniture is dropped, tables are spliced whole, and every chunkset is a self-explanatory root-to-leaf unit carrying its hierarchy metadata, with no overlap. On a notoriously hard reference legal document, chunksets plus cheatsheet assembly delivered the answer in 337 tokens of retrieved context versus 1,542 for a recursive character splitter, with zero information loss (methodology).

The pipeline, end to end ​

bash
pip install 'unstructured[pdf]' poma pymilvus sentence-transformers
python
import os
from poma import PrimeCut
from poma.utils import unpack_poma_archive
from pymilvus import DataType, MilvusClient
from sentence_transformers import SentenceTransformer
from unstructured.partition.pdf import partition_pdf
from unstructured.staging.base import elements_to_json

# 1. Unstructured — your existing call, unchanged.
elements = partition_pdf(
    filename="contract.pdf",
    strategy="hi_res",
    infer_table_structure=True,           # populates metadata.text_as_html
    extract_image_block_types=["Image"],
    extract_image_block_to_payload=True,  # inline base64 → POMA describes figures
)
elements_to_json(elements, filename="contract.unstructured.json")

# 2. The missing link — raw element list in, chunks + chunksets out.
poma = PrimeCut()  # reads POMA_API_KEY
result = poma.ingest(  # Unstructured shape auto-detected
    "contract.unstructured.json",
    download_dir="store",
    filename="contract.poma",  # keep the archive — page numbers live in it
)

# Depth comes off the SDK objects; page is on the archive's chunk records only.
depth_of = {c.chunk_index: c.depth for c in result.chunks}
page_of = {
    c["chunk_index"]: c.get("page")
    for c in unpack_poma_archive(poma_archive_path="store/contract.poma")["chunks"]
}

# 3. Milvus — a schema whose scalar fields mirror the hierarchy metadata.
client = MilvusClient(uri=os.environ["MILVUS_URI"])
schema = MilvusClient.create_schema()
schema.add_field("pk", DataType.INT64, is_primary=True, auto_id=True)
schema.add_field("vector", DataType.FLOAT_VECTOR, dim=384)
schema.add_field("text", DataType.VARCHAR, max_length=65535)
schema.add_field("file_id", DataType.VARCHAR, max_length=256, is_partition_key=True)
schema.add_field("page", DataType.INT64)
schema.add_field("depth", DataType.INT64)
schema.add_field("chunkset_index", DataType.INT64)

index_params = client.prepare_index_params()
index_params.add_index(field_name="vector", index_type="HNSW", metric_type="COSINE")
client.create_collection("contracts", schema=schema, index_params=index_params)

# 4. Embed chunksets and insert — hierarchy metadata rides along as scalars.
model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
client.insert(
    collection_name="contracts",
    data=[
        {
            "vector": model.encode(cs.to_embed).tolist(),
            "text": cs.to_embed,
            "file_id": cs.file_id,
            "chunkset_index": cs.chunkset_index,
            # chunk indices are in document order; the last one is the leaf
            "page": page_of[cs.chunks[-1]],
            "depth": depth_of[cs.chunks[-1]],
        }
        for cs in result.chunksets
    ],
)

# 5. Filtered search — scope to one document, then assemble the cheatsheet.
query = "What are the early termination conditions?"
hits = client.search(
    collection_name="contracts",
    data=[model.encode(query).tolist()],
    limit=5,
    filter='file_id == "contract.unstructured.json"',
    output_fields=["text", "chunkset_index", "page", "depth"],
)
ordered = sorted(hits[0], key=lambda h: h["entity"]["chunkset_index"])
cheatsheet = "\n\n".join(dict.fromkeys(h["entity"]["text"] for h in ordered))

POMA validates the payload shape up front (fingerprint: elements carrying type + element_id; a corrupted or mislabeled upload 422s immediately), groups elements by page_number, splices each Table's text_as_html, describes inline image_base64 figures (disk-bound image_path refs are neutralized and counted in content_metadata), drops Header/Footer/PageNumber furniture, and rebuilds the cross-page heading tree before chunking. To force or suppress detection, set external_ocr_source to "unstructured" or "none".

Note that the single text field with a generous max_length holds because chunkset text is normalized prose plus spliced HTML — POMA has already bounded the units. On recent Milvus versions, add a SPARSE_FLOAT_VECTOR field with a BM25 function to the same schema and issue a hybrid request that fuses dense and sparse rankings; the chunkset design does not change, only the query does.

Metadata mapping: POMA fields → Milvus primitives ​

POMA chunk fieldMilvus primitiveWhat it enables
to_embedFLOAT_VECTOR field (add a sparse/BM25 field for hybrid)paraphrase + exact-term retrieval
file_idVARCHAR scalar field, partition keyper-document filters, automatic multi-tenant routing
page of the chunkset's leaf chunk (from the archive's chunk records, filled from Unstructured metadata.page_number)INT64 scalar fieldpage-cited answers, page <= N filter expressions
depth of the leaf chunkINT64 scalar fieldfilter/re-rank by hierarchy level
chunkset_indexINT64 scalar fieldstable ordering at cheatsheet assembly
chunkset text (with spliced text_as_html)VARCHAR field in output_fieldsassembly without a second store

Frequently asked questions ​

How do I get Unstructured.io elements into Milvus for RAG? ​

Save the element list with elements_to_json, run PrimeCut().ingest() on the raw JSON (auto-detected, hierarchy rebuilt), embed each chunkset's to_embed, and insert entities whose scalar fields carry file_id, chunkset_index, page, and depth — page and depth taken from the chunkset's leaf chunk, with page read out of the .poma archive's chunk records. Retrieve with a filtered search and assemble cheatsheets client-side.

Why not insert one Milvus entity per Unstructured element? ​

Milvus's fixed VARCHAR max_length collides with text_as_html tables — inserts fail or truncate mid-row — and recurring Header/Footer/PageNumber elements become near-identical vectors crowding top-k in every partition. Chunksets drop the furniture and splice tables whole.

Can Milvus filter expressions use Unstructured's page numbers? ​

Yes — POMA carries metadata.page_number through to every chunk record in the .poma archive; read it back with unpack_poma_archive and write the chunkset's leaf-chunk page into an INT64 scalar field, where page <= 10 filters during the ANN search itself. Flatten-and-split destroys the page number before it ever reaches the schema.

Should file_id be a Milvus partition key for Unstructured-parsed corpora? ​

Yes, when one collection serves many documents or tenants: declare it with is_partition_key=True and filtered searches only touch matching partitions. POMA populates file_id on every chunkset, so the key always has a value to route on.

Does Milvus hybrid BM25 search help with Unstructured.io content? ​

Yes — recent Milvus versions fuse sparse BM25 and dense rankings in one hybrid request. Part numbers, clause references, and values inside text_as_html tables are exact tokens dense embeddings blur, and POMA's intact-HTML splicing keeps them present for BM25 to match.

Same parser, different store: Unstructured.io → Qdrant · Unstructured.io → pgvector · Unstructured.io → Chroma

Same store, different parser: Mistral OCR → Milvus · Marker → Milvus · Azure Document Intelligence → Milvus

Foundations: The Optimal Chunker for Unstructured.io · The Optimal Chunks for Milvus · All pipeline recipes · RAG chunking guide