Source: http://www.poma-ai.com/docs/pipelines/unstructured-to-milvus

# The Missing Link Between Unstructured.io and Optimal Retrieval in Milvus

<ByAuthor />

**The short answer:** Unstructured.io gives you a typed, ordered element list; Milvus gives you filtered vector search at serious scale — partitions with automatic key routing, scalar filter expressions, and your pick of HNSW, IVF, or DiskANN. Wired together naively — one entity per element, or flatten-and-split — the pipeline underuses both, because nothing in between rebuilds hierarchy or populates the scalar fields Milvus filters on. The missing link is POMA: `PrimeCut().ingest()` consumes the raw `elements_to_json` output (auto-detected), emits hierarchy-preserving [chunksets](/learn/chunking/chunksets), and each chunkset lands in Milvus with `file_id`, `page`, `depth`, and `chunk_index` as scalar fields ready for filter expressions and partition-key routing.

## What each end of the pipeline actually provides

**Unstructured.io** (the open-source `unstructured` library and the hosted API) returns a flat, ordered list of typed elements — `Title`, `NarrativeText`, `ListItem`, `Table`, `Image` — each with its `text`, a stable `element_id`, and `metadata` such as `page_number`, `text_as_html` for tables, and optionally `image_base64`. What it doesn't provide: hierarchy (the list is flat — nothing records which `Title` nests under which) or retrieval units; the built-in `by_title` and `basic` strategies emit flat fragments. Details: [The Optimal Chunker for Unstructured.io](/optimal-chunker-unstructured).

**Milvus** provides collections with typed schemas, partitions with automatic partition-key routing for multi-tenancy, scalar fields queried through filter expressions (`file_id == "..."`, `page <= 10`), index types from in-memory HNSW through IVF to disk-resident DiskANN, and — in recent versions — sparse vectors with built-in BM25 for hybrid search. What it doesn't provide: any opinion about what an entity should contain. Empty scalar fields filter nothing; fragment vectors retrieve fragments. Details: [The Optimal Chunks for the Best Retrieval in Milvus](/optimal-chunks-milvus).

## The naive wiring, and where it breaks

The tempting recipe — one Milvus entity per element (`element_id` even looks like a primary key), or join all element texts and split — breaks in a pair-specific way:

- **The fixed schema fights the element list.** A Milvus `VARCHAR` field declares a `max_length` up front, but Unstructured elements vary from a three-word `Title` to a `text_as_html` table that dwarfs any budget you picked. Per-element inserts either fail on the long tail or silently truncate tables mid-row — precisely the elements that hold the numbers your users ask about.
- **Furniture becomes vectors.** `Header`, `Footer`, and `PageNumber` elements recur on every page with near-identical text. Embedded per element, they become dozens of near-duplicate vectors that match everything vaguely and crowd top-k in every partition of the collection.
- **The scalar fields end up empty.** Flatten-and-split destroys `metadata.page_number` and the element typing at its first step, so `page` and `depth` have nothing to hold — and Milvus's headline feature, filtered search during ANN, has nothing to filter on.

POMA's chunksets fix all three at the source: furniture is dropped, tables are spliced whole, and every chunkset is a self-explanatory root-to-leaf unit carrying its hierarchy metadata, with no overlap. On a notoriously hard reference legal document, chunksets plus cheatsheet assembly delivered the answer in 337 tokens of retrieved context versus 1,542 for a recursive character splitter, with zero information loss ([methodology](/document-ingestion-chunking-rag)).

## The pipeline, end to end

```bash
pip install 'unstructured[pdf]' poma pymilvus sentence-transformers
```

```python
import os
from poma import PrimeCut
from pymilvus import DataType, MilvusClient
from sentence_transformers import SentenceTransformer
from unstructured.partition.pdf import partition_pdf
from unstructured.staging.base import elements_to_json

# 1. Unstructured — your existing call, unchanged.
elements = partition_pdf(
    filename="contract.pdf",
    strategy="hi_res",
    infer_table_structure=True,           # populates metadata.text_as_html
    extract_image_block_types=["Image"],
    extract_image_block_to_payload=True,  # inline base64 → POMA describes figures
)
elements_to_json(elements, filename="contract.unstructured.json")

# 2. The missing link — raw element list in, chunks + chunksets out.
poma = PrimeCut()  # reads POMA_API_KEY
result = poma.ingest("contract.unstructured.json")  # Unstructured shape auto-detected

# 3. Milvus — a schema whose scalar fields mirror the hierarchy metadata.
client = MilvusClient(uri=os.environ["MILVUS_URI"])
schema = MilvusClient.create_schema()
schema.add_field("pk", DataType.INT64, is_primary=True, auto_id=True)
schema.add_field("vector", DataType.FLOAT_VECTOR, dim=384)
schema.add_field("text", DataType.VARCHAR, max_length=65535)
schema.add_field("file_id", DataType.VARCHAR, max_length=256, is_partition_key=True)
schema.add_field("page", DataType.INT64)
schema.add_field("depth", DataType.INT64)
schema.add_field("chunk_index", DataType.INT64)

index_params = client.prepare_index_params()
index_params.add_index(field_name="vector", index_type="HNSW", metric_type="COSINE")
client.create_collection("contracts", schema=schema, index_params=index_params)

# 4. Embed chunksets and insert — hierarchy metadata rides along as scalars.
model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
client.insert(
    collection_name="contracts",
    data=[
        {
            "vector": model.encode(cs.to_embed).tolist(),
            "text": cs.to_embed,
            "file_id": cs.file_id,
            "page": cs.page,
            "depth": cs.depth,
            "chunk_index": cs.chunk_index,
        }
        for cs in result.chunksets
    ],
)

# 5. Filtered search — scope to one document, then assemble the cheatsheet.
query = "What are the early termination conditions?"
hits = client.search(
    collection_name="contracts",
    data=[model.encode(query).tolist()],
    limit=5,
    filter='file_id == "contract.unstructured.json"',
    output_fields=["text", "page", "depth", "chunk_index"],
)
ordered = sorted(hits[0], key=lambda h: h["entity"]["chunk_index"])
cheatsheet = "\n\n".join(dict.fromkeys(h["entity"]["text"] for h in ordered))
```

POMA validates the payload shape up front (fingerprint: elements carrying `type` + `element_id`; a corrupted or mislabeled upload 422s immediately), groups elements by `page_number`, splices each `Table`'s `text_as_html`, describes inline `image_base64` figures (disk-bound `image_path` refs are neutralized and counted in `content_metadata`), drops `Header`/`Footer`/`PageNumber` furniture, and rebuilds the cross-page heading tree before chunking. To force or suppress detection, set `external_ocr_source` to `"unstructured"` or `"none"`.

Note that the single `text` field with a generous `max_length` holds because chunkset text is normalized prose plus spliced HTML — POMA has already bounded the units. On recent Milvus versions, add a `SPARSE_FLOAT_VECTOR` field with a BM25 function to the same schema and issue a hybrid request that fuses dense and sparse rankings; the chunkset design does not change, only the query does.

## Metadata mapping: POMA fields → Milvus primitives

| POMA chunk field | Milvus primitive | What it enables |
| --- | --- | --- |
| `to_embed` | `FLOAT_VECTOR` field (add a sparse/BM25 field for hybrid) | paraphrase + exact-term retrieval |
| `file_id` | `VARCHAR` scalar field, **partition key** | per-document filters, automatic multi-tenant routing |
| `page` (from Unstructured `metadata.page_number`) | `INT64` scalar field | page-cited answers, `page <= N` filter expressions |
| `depth` | `INT64` scalar field | filter/re-rank by hierarchy level |
| `chunk_index` | `INT64` scalar field | stable ordering at cheatsheet assembly |
| chunkset text (with spliced `text_as_html`) | `VARCHAR` field in `output_fields` | assembly without a second store |

## Frequently asked questions

### How do I get Unstructured.io elements into Milvus for RAG?

Save the element list with `elements_to_json`, run `PrimeCut().ingest()` on the raw JSON (auto-detected, hierarchy rebuilt), embed each chunkset's `to_embed`, and insert entities whose scalar fields carry `file_id`, `page`, `depth`, and `chunk_index`. Retrieve with a filtered search and assemble cheatsheets client-side.

### Why not insert one Milvus entity per Unstructured element?

Milvus's fixed `VARCHAR` `max_length` collides with `text_as_html` tables — inserts fail or truncate mid-row — and recurring `Header`/`Footer`/`PageNumber` elements become near-identical vectors crowding top-k in every partition. Chunksets drop the furniture and splice tables whole.

### Can Milvus filter expressions use Unstructured's page numbers?

Yes — POMA carries `metadata.page_number` through to each chunkset, so it lands in an `INT64` scalar field where `page <= 10` filters during the ANN search itself. Flatten-and-split destroys the page number before it ever reaches the schema.

### Should file_id be a Milvus partition key for Unstructured-parsed corpora?

Yes, when one collection serves many documents or tenants: declare it with `is_partition_key=True` and filtered searches only touch matching partitions. POMA populates `file_id` on every chunkset, so the key always has a value to route on.

### Does Milvus hybrid BM25 search help with Unstructured.io content?

Yes — recent Milvus versions fuse sparse BM25 and dense rankings in one hybrid request. Part numbers, clause references, and values inside `text_as_html` tables are exact tokens dense embeddings blur, and POMA's intact-HTML splicing keeps them present for BM25 to match.

## Related recipes

Same parser, different store: [Unstructured.io → Qdrant](/pipelines/unstructured-to-qdrant) · [Unstructured.io → pgvector](/pipelines/unstructured-to-pgvector) · [Unstructured.io → Chroma](/pipelines/unstructured-to-chroma)

Same store, different parser: [Mistral OCR → Milvus](/pipelines/mistral-ocr-to-milvus) · [Marker → Milvus](/pipelines/marker-to-milvus) · [Azure Document Intelligence → Milvus](/pipelines/azure-document-intelligence-to-milvus)

Foundations: [The Optimal Chunker for Unstructured.io](/optimal-chunker-unstructured) · [The Optimal Chunks for Milvus](/optimal-chunks-milvus) · [All pipeline recipes](/pipelines/) · [RAG chunking guide](/guides/rag-chunking/)