Azure Document Intelligence to Elasticsearch: Chunking for RAG
The short answer: Azure Document Intelligence's prebuilt-layout model recovers reading order and tables better than almost any parser on the market; Elasticsearch fuses k-NN and two decades of mature BM25 in one request. Wired together with a fixed-size splitter, the pair still produces mediocre RAG, because nothing rebuilds the cross-page heading hierarchy Azure discards or keeps the index content-free. The missing link is POMA: PrimeCut().ingest() consumes the raw analyze result (auto-detected), emits chunksets into a portable .poma archive, and poma.vektoria indexes content-free documents into Elasticsearch while your document content lives on a volume. Retrieval runs Elasticsearch's own knn query, then assemble() reconstructs prompt-ready context.
What each end of the pipeline actually provides
Azure Document Intelligence returns one of two shapes: markdown mode (outputContentFormat="markdown") — one reading-order string with inline HTML tables and <!-- PageBreak --> delimiters — or JSON mode's paragraphs[], ordered by span offset with role fields for headings. Both de-interleave multi-column layouts server-side. What it never provides: cross-page hierarchy, chunking, or figure bytes. Details: The Optimal Chunker for Azure Document Intelligence.
Elasticsearch provides HNSW-backed dense_vector fields, native BM25 on every analyzed text field, a single request combining knn with bool/match, and rank (RRF) fusion. What it doesn't provide: any opinion about what a document should contain, or where the actual text lives — vektoria's contract keeps the index holding only (embedding, file_id, chunkset_index). Details: Elasticsearch Chunking Strategy for RAG.
The naive wiring, and where it breaks
The common shortcut — split Azure's content string on <!-- PageBreak -->, run a fixed-size splitter over each page, embed, bulk-index the raw text into an analyzed field — breaks in a pair-specific way. The PageHeader/PageFooter/PageNumber furniture comments Azure leaves inline survive the split and land inside the same text field Elasticsearch's BM25 scores. A query for a real defined term now competes against every document carrying the repeated header string, and rank's RRF fusion dutifully merges those lexical false-positives in with the vector hits. Compounding this, Azure never returns cropped figure bytes — a naive pipeline has no image content to index for a figure at all, so that content is simply absent, with nothing in the mapping to flag that it was ever supposed to be there.
The pipeline, end to end
pip install azure-ai-documentintelligence elasticsearchimport json
import os
from azure.ai.documentintelligence import DocumentIntelligenceClient
from azure.core.credentials import AzureKeyCredential
from poma import PrimeCut
from poma.vektoria import assemble, records_from_archive, Volume
from poma.embeddings import get_embedder
from elasticsearch import Elasticsearch
# 1. Azure Document Intelligence — your existing call, unchanged.
di = DocumentIntelligenceClient(
endpoint=os.environ["AZURE_DI_ENDPOINT"],
credential=AzureKeyCredential(os.environ["AZURE_DI_KEY"]),
)
with open("contract.pdf", "rb") as f:
poller = di.begin_analyze_document(
"prebuilt-layout",
body=f,
output_content_format="markdown",
)
analysis = poller.result()
with open("result.azure-di.json", "w") as f:
json.dump(analysis.as_dict(), f)
# 2. The missing link — raw result JSON in, .poma archive out (auto-detected Azure shape).
client = PrimeCut() # reads POMA_API_KEY
client.ingest("result.azure-di.json", download_dir="archives", filename="doc.poma")
embedder = get_embedder("local:BAAI/bge-small-en-v1.5")
vol = Volume("s3://your-bucket/poma")
es = Elasticsearch("http://localhost:9200") # local dev cluster (xpack.security.enabled=false); a secured cluster needs https + basic_auth/api_key + ca_certs
es.indices.create(index="poma", mappings={"properties": {
"embedding": {"type": "dense_vector", "dims": embedder.dims, "index": True, "similarity": "cosine"},
"file_id": {"type": "keyword"},
"chunkset_index": {"type": "integer"},
}}, ignore=400)
# 3. Content-free ingest — Elasticsearch holds only routing metadata; content lives on the volume.
records = records_from_archive("archives/doc.poma") # -> list[Record] (id, text, payload)
# assemble() reads the FULL document (chunks + chunksets) from the volume — write it once,
# not one {"chunks": <indices>, "text"} stub per record (which also overwrote itself each turn).
from poma.utils import unpack_poma_archive
_data = unpack_poma_archive(poma_archive_path="archives/doc.poma")
file_id = records[0].payload["file_id"]
vol.write_doc(file_id, {"file_id": file_id, "chunks": _data["chunks"], "chunksets": _data["chunksets"]})
for r in records:
es.index(index="poma", id=r.id, document={
"embedding": embedder.embed([r.text])[0],
"file_id": r.payload["file_id"],
"chunkset_index": r.payload["chunkset_index"],
})
# 4. Retrieval — Elasticsearch's normal knn query, then assemble.
qv = embedder.embed(["early termination conditions"])[0]
es.indices.refresh(index="poma") # make the just-indexed docs searchable before querying
res = es.search(index="poma", knn={"field": "embedding", "query_vector": qv,
"k": 10, "num_candidates": 100})
context = assemble(res.body, volume=vol) # elasticsearch-py 8+ wraps the response (ObjectApiResponse) — pass the dict body # -> [{"file_id", "content"}, ...]POMA validates the payload shape up front (a mislabeled upload 422s immediately), splits markdown mode on PageBreak while dropping furniture comments, reconstructs JSON mode from span-ordered paragraphs[], and counts every figure as offloaded content since Azure never returns figure bytes. To force or suppress detection, set external_ocr_source to "azure_di" or "none".
Metadata mapping: POMA fields → Elasticsearch primitives
| POMA field | Elasticsearch primitive | What it enables |
|---|---|---|
id (chunkset_uuid) | document _id | stable identity across re-ingests |
to_embed | dense_vector field | knn ranking |
file_id | keyword field | bool.filter scoping to one document |
chunkset_index | integer field, returned in _source | assemble() match key |
page, depth, chunks, text | volume document, not indexed | content-free retrieval, cheatsheet assembly |
Frequently asked questions
How do I get Azure Document Intelligence results into Elasticsearch for RAG?
Save the raw analyze result JSON, hand it to PrimeCut().ingest() (auto-detected, hierarchy rebuilt into a .poma archive), then loop records_from_archive() — write content to a volume, es.index() the vector plus file_id/chunkset_index. Retrieve with a knn query plus assemble().
Why not just bulk-index Azure's markdown content string directly?
Azure's PageHeader/PageFooter/PageBreak comments survive a naive split and land in the analyzed field BM25 scores, so repeated boilerplate competes with real matches. Overlap also doubles near-duplicate documents in the index.
What Elasticsearch mapping fields should Azure Document Intelligence chunksets use?
embedding as dense_vector (similarity: cosine), file_id as keyword, chunkset_index as integer. Page, depth, and chunk text are not stored in the index at all — they live on the volume.
What happens to Azure Document Intelligence figures in an Elasticsearch index?
Nothing reaches the index — Azure never returns cropped figure bytes, so there's no content to map into any field. POMA counts every figure as offloaded content in content_metadata instead of a silent gap.
Does Elasticsearch retrieval need anything special for POMA's assemble() to work?
No extra flag — Elasticsearch returns _source by default. The one gotcha: applying source filtering (_source: false or an excludes list) strips the fields assemble() needs and returns an empty context list.
Related recipes
Same parser, different store: Azure Document Intelligence → Qdrant · Azure Document Intelligence → pgvector · Azure Document Intelligence → OpenSearch
Foundations: The Optimal Chunker for Azure Document Intelligence · Elasticsearch Chunking Strategy for RAG · All pipeline recipes · RAG chunking guide