Azure Document Intelligence to Milvus: Chunking for RAG
The short answer: Azure Document Intelligence gives you an excellent layout analysis; Milvus gives you vector search that scales further than almost anything else. Wired together naively — flatten, split, embed, insert — they still produce mediocre RAG, because nothing in between rebuilds the document's hierarchy or fills the scalar fields Milvus filters on. The missing link is POMA: PrimeCut().ingest() consumes the raw analyze result (auto-detected, markdown or JSON mode), emits hierarchy-preserving chunksets, and each one lands as a Milvus entity with file_id, page, and depth scalar fields driving filtered — and optionally partition-routed — search.
What each end of the pipeline actually provides
Azure Document Intelligence (prebuilt-layout) returns one of two shapes: markdown mode — a single reading-order markdown string with inline HTML tables, PageBreak delimiters, and PageHeader/PageFooter/PageNumber furniture comments — or JSON mode, paragraphs[] ordered by span offset with role tags plus tables[] as cell grids. Multi-column de-interleaving happens server-side. What it doesn't provide: cross-page hierarchy (roles classify paragraphs, they don't nest them), retrieval units, or figure bytes. Details: The Optimal Chunker for Azure Document Intelligence.
Milvus provides collections with partitions (a partition key hashes entities automatically — natural multi-tenancy), scalar fields whose filter expressions apply during the ANN search, a spectrum of index types (HNSW, IVF variants, DiskANN for corpora that outgrow RAM), and — on recent versions — sparse vectors with built-in BM25 for hybrid search. What it doesn't provide: any opinion about what an entity should contain. Details: The Optimal Chunks for the Best Retrieval in Milvus.
The naive wiring, and where it breaks
The common recipe — flatten the analyze result to text, run a fixed-size splitter, embed, insert (vector, text) entities — breaks in a pair-specific way:
- Page furniture becomes entities. Azure's markdown mode repeats
PageHeader,PageFooter, andPageNumbercomments on every page. Flattened and split, a 300-page filing embeds its running header hundreds of times — hundreds of near-identical vectors in the HNSW graph, and on the BM25 sparse side of a hybrid search, boilerplate tokens ("Confidential", the company name) repeated per page skew the term statistics of exactly the lane meant to catch rare exact terms. - The scalar fields stay empty. Milvus's distinguishing primitive — boolean filter expressions applied during the ANN search — has nothing to act on when entities carry only a vector and a text blob. No
file_idscoping, no page citations, no depth filters, and the partition key has no field to hash on. - The pipeline is shape-blind. Teams write a splitter for markdown mode's
contentstring, then feed it a JSON-mode result already sitting in blob storage (or vice versa) — and get one giant unsplit string or a crash, depending on the day.
The pipeline, end to end
pip install azure-ai-documentintelligence poma pymilvusimport json
import os
from azure.ai.documentintelligence import DocumentIntelligenceClient
from azure.core.credentials import AzureKeyCredential
from poma import PrimeCut
from poma.utils import unpack_poma_archive
from pymilvus import DataType, MilvusClient
# 1. Your existing Azure Document Intelligence call — unchanged.
di = DocumentIntelligenceClient(
endpoint=os.environ["AZURE_DI_ENDPOINT"],
credential=AzureKeyCredential(os.environ["AZURE_DI_KEY"]),
)
with open("contract.pdf", "rb") as f:
poller = di.begin_analyze_document(
"prebuilt-layout",
body=f,
output_content_format="markdown", # omit for classic JSON mode — both work
)
with open("contract.azure-di.json", "w") as f:
json.dump(poller.result().as_dict(), f)
# 2. The missing link — raw analyze result in, chunks + chunksets out.
poma = PrimeCut() # reads POMA_API_KEY
result = poma.ingest( # Azure shape auto-detected
"contract.azure-di.json",
download_dir="store",
filename="contract.poma", # keep the archive — page numbers live in it
)
# Depth comes off the SDK objects; page is on the archive's chunk records only.
depth_of = {c.chunk_index: c.depth for c in result.chunks}
page_of = {
c["chunk_index"]: c.get("page")
for c in unpack_poma_archive(poma_archive_path="store/contract.poma")["chunks"]
}
# 3. A chunkset collection: dense vector + hierarchy scalar fields.
milvus = MilvusClient(
uri=os.environ["MILVUS_URI"],
token=os.environ["MILVUS_TOKEN"],
)
schema = MilvusClient.create_schema(auto_id=True)
schema.add_field("pk", DataType.INT64, is_primary=True)
schema.add_field("vector", DataType.FLOAT_VECTOR, dim=384)
schema.add_field("text", DataType.VARCHAR, max_length=65535)
schema.add_field("file_id", DataType.VARCHAR, max_length=128, is_partition_key=True)
schema.add_field("page", DataType.INT64)
schema.add_field("depth", DataType.INT64)
schema.add_field("chunkset_index", DataType.INT64)
index_params = milvus.prepare_index_params()
index_params.add_index(field_name="vector", index_type="HNSW", metric_type="COSINE")
milvus.create_collection(
collection_name="chunksets",
schema=schema,
index_params=index_params,
)
milvus.insert(
collection_name="chunksets",
data=[
{
"vector": embed(cs.to_embed), # your embedding model
"text": cs.to_embed,
"file_id": cs.file_id,
"chunkset_index": cs.chunkset_index,
# chunk indices are in document order; the last one is the leaf
"page": page_of[cs.chunks[-1]],
"depth": depth_of[cs.chunks[-1]],
}
for cs in result.chunksets
],
)
# 4. Filtered search — the expression is applied during the ANN search,
# and the partition key routes it to only the relevant partitions.
hits = milvus.search(
collection_name="chunksets",
data=[embed("What does the contract say about early termination?")],
filter='file_id == "contract.azure-di.json"',
limit=5,
output_fields=["text", "file_id", "chunkset_index", "page", "depth"],
)On recent Milvus versions, add a SPARSE_FLOAT_VECTOR field with a BM25 function to the same schema and issue a hybrid request that fuses dense and sparse rankings — the chunkset design doesn't change, only the query does. And because POMA stripped the page furniture before chunking, the BM25 term statistics reflect real content, not repeated headers.
The last step is client-side assembly: retrieved chunksets share ancestors, so deduplicate the shared lineage and merge the hits into one coherent context block — a cheatsheet — before it goes into the prompt. On our reference legal-document benchmark that assembly meant 337 tokens of context versus 1,542 for a recursive-splitter baseline, with zero information loss — methodology here.
POMA validates the payload up front (a corrupted or mislabeled upload 422s immediately, never a silently degraded index), keeps Azure's HTML tables whole, and counts Azure's byte-less figures as offloaded content in content_metadata — visible loss, never silent. To force or suppress detection, set external_ocr_source to "azure_di" or "none".
Metadata mapping: POMA fields → Milvus primitives
| POMA chunk field | Milvus primitive | What it enables |
|---|---|---|
to_embed | FLOAT_VECTOR field (+ optional SPARSE_FLOAT_VECTOR BM25) | dense + exact-term hybrid retrieval |
to_embed text | VARCHAR field (text) | hits arrive with their content, no second lookup |
file_id | VARCHAR scalar, is_partition_key=True | per-document scoping + partition routing |
page of the chunkset's leaf chunk (from the archive's chunk records, filled from PageBreak / span offsets) | INT64 scalar, filter expression | page-cited answers, page <= 20 filters |
depth of the leaf chunk (from Azure roles + rebuilt tree) | INT64 scalar, filter expression | depth <= 2 structural filters, re-ranking |
chunkset_index | INT64 scalar | stable ordering at assembly time |
Frequently asked questions
How do I get Azure Document Intelligence results into Milvus for RAG?
Save the raw analyze result JSON, run PrimeCut().ingest() on it (auto-detected, hierarchy rebuilt), then embed each chunkset's to_embed and insert entities with file_id, chunkset_index, page, and depth scalar fields — page and depth taken from the chunkset's leaf chunk, with page read out of the .poma archive's chunk records. Filtered ANN search plus client-side cheatsheet assembly completes the loop.
What scalar fields should Azure Document Intelligence chunks carry in Milvus?
file_id, chunkset_index, page, depth, plus a VARCHAR field for the text. Filter expressions like file_id == "contract-001" or depth <= 2 apply during the ANN search itself. page and depth belong to the chunkset's leaf chunk: depth is on the SDK's chunk objects, and page — recovered from Azure's PageBreak delimiters or span offsets — is on every chunk record in the .poma archive.
Do Azure Document Intelligence page headers pollute Milvus hybrid BM25 search?
In naive pipelines, yes: markdown mode repeats header/footer/page-number furniture on every page, and once embedded it skews both the HNSW graph and the sparse BM25 term statistics toward boilerplate. POMA drops the furniture comments before chunking, so both lanes index only real content.
Does markdown mode versus JSON mode matter when loading Azure Document Intelligence into Milvus?
No — POMA auto-detects and handles both, and the resulting chunks and chunksets are identical in shape either way. Your Milvus schema, filter expressions, and retrieval code never change.
How do I isolate documents or tenants in one Milvus collection for Azure Document Intelligence content?
Declare file_id (or a tenant_id) as the partition key at collection creation. Milvus hashes entities into partitions automatically and routes filtered queries to only the relevant ones — POMA writes file_id on every chunkset, so isolation comes for free.
Related recipes
Same parser, different store: Azure Document Intelligence → pgvector · Azure Document Intelligence → Pinecone · Azure Document Intelligence → Chroma
Same store, different parser: Mistral OCR → Milvus · Unstructured.io → Milvus · Textract → Milvus
Foundations: The Optimal Chunker for Azure Document Intelligence · The Optimal Chunks for Milvus · All pipeline recipes · RAG chunking guide