Azure Document Intelligence to OpenSearch: Chunking for RAG
The short answer: Azure Document Intelligence's prebuilt-layout model recovers reading order and tables better than almost any parser on the market; OpenSearch's k-NN plugin fuses vector and BM25 search through its own hybrid query pipeline. Wired together with a fixed-size splitter, the pair still produces mediocre RAG, because nothing rebuilds the cross-page heading hierarchy Azure discards or keeps the index content-free. The missing link is POMA: PrimeCut().ingest() consumes the raw analyze result (auto-detected), emits chunksets into a portable .poma archive, and poma.vektoria indexes content-free documents into OpenSearch while your document content lives on a volume. Retrieval runs OpenSearch's own knn query, then assemble() reconstructs prompt-ready context.
What each end of the pipeline actually provides
Azure Document Intelligence returns one of two shapes: markdown mode (outputContentFormat="markdown") — one reading-order string with inline HTML tables and <!-- PageBreak --> delimiters — or JSON mode's paragraphs[], ordered by span offset with role fields for headings. Both de-interleave multi-column layouts server-side. What it never provides: cross-page hierarchy, chunking, or figure bytes. Details: The Optimal Chunker for Azure Document Intelligence.
OpenSearch provides HNSW-backed knn_vector fields via nmslib or faiss, native BM25 on every analyzed text field, and a dedicated hybrid query type fused by a search pipeline. What it doesn't provide: any opinion about what a document should contain, or where the actual text lives — vektoria's contract keeps the index holding only (embedding, file_id, chunkset_index). Details: OpenSearch Chunking Strategy for RAG.
The naive wiring, and where it breaks
The common shortcut — split Azure's content string on <!-- PageBreak -->, run a fixed-size splitter over each page, embed, bulk-index the raw text into an analyzed field — breaks in a pair-specific way. The PageHeader/PageFooter/PageNumber furniture comments Azure leaves inline survive the split and land inside the same text field OpenSearch's BM25 scores. OpenSearch's hybrid query type then normalizes and weight-sums that inflated lexical score against the vector score in its search pipeline — unlike Elasticsearch's RRF, this normalization step can let one dominant lexical outlier (a header string repeated on every page) skew the fused ranking further than rank-based fusion would. Compounding this, Azure never returns cropped figure bytes — a naive pipeline has no image content to index for a figure at all, so that content is simply absent, with nothing in the mapping to flag that it was ever supposed to be there.
The pipeline, end to end
pip install azure-ai-documentintelligence opensearch-pyimport json
import os
from azure.ai.documentintelligence import DocumentIntelligenceClient
from azure.core.credentials import AzureKeyCredential
from poma import PrimeCut
from poma.vektoria import assemble, records_from_archive, Volume
from poma.embeddings import get_embedder
from opensearchpy import OpenSearch
# 1. Azure Document Intelligence — your existing call, unchanged.
di = DocumentIntelligenceClient(
endpoint=os.environ["AZURE_DI_ENDPOINT"],
credential=AzureKeyCredential(os.environ["AZURE_DI_KEY"]),
)
with open("contract.pdf", "rb") as f:
poller = di.begin_analyze_document(
"prebuilt-layout",
body=f,
output_content_format="markdown",
)
analysis = poller.result()
with open("result.azure-di.json", "w") as f:
json.dump(analysis.as_dict(), f)
# 2. The missing link — raw result JSON in, .poma archive out (auto-detected Azure shape).
client = PrimeCut() # reads POMA_API_KEY
client.ingest("result.azure-di.json", download_dir="archives", filename="doc.poma")
embedder = get_embedder("local:BAAI/bge-small-en-v1.5")
vol = Volume("s3://your-bucket/poma")
osc = OpenSearch(hosts=[{"host": "localhost", "port": 9200}])
osc.indices.create(index="poma", body={"settings": {"index.knn": True}, # knn_vector fields need index.knn=true, otherwise the mapping is rejected (and ignore=400 hides it)
"mappings": {"properties": {
"embedding": {"type": "knn_vector", "dimension": embedder.dims,
"method": {"name": "hnsw", "space_type": "cosinesimil", "engine": "faiss"}},
"file_id": {"type": "keyword"},
"chunkset_index": {"type": "integer"},
}}}, ignore=400)
# 3. Content-free ingest — OpenSearch holds only routing metadata; content lives on the volume.
records = records_from_archive("archives/doc.poma") # -> list[Record] (id, text, payload)
# assemble() reads the FULL document (chunks + chunksets) from the volume — write it once,
# not one {"chunks": <indices>, "text"} stub per record (which also overwrote itself each turn).
from poma.utils import unpack_poma_archive
_data = unpack_poma_archive(poma_archive_path="archives/doc.poma")
file_id = records[0].payload["file_id"]
vol.write_doc(file_id, {"file_id": file_id, "chunks": _data["chunks"], "chunksets": _data["chunksets"]})
for r in records:
osc.index(index="poma", id=r.id, body={
"embedding": embedder.embed([r.text])[0],
"file_id": r.payload["file_id"],
"chunkset_index": r.payload["chunkset_index"],
})
# 4. Retrieval — OpenSearch's normal knn query, then assemble.
qv = embedder.embed(["early termination conditions"])[0]
osc.indices.refresh(index="poma") # make the just-indexed docs searchable before querying
res = osc.search(index="poma", body={"query": {"knn": {"embedding": {
"vector": qv, "k": 10}}}})
context = assemble(res, volume=vol) # -> [{"file_id", "content"}, ...]POMA validates the payload shape up front (a mislabeled upload 422s immediately), splits markdown mode on PageBreak while dropping furniture comments, reconstructs JSON mode from span-ordered paragraphs[], and counts every figure as offloaded content since Azure never returns figure bytes. To force or suppress detection, set external_ocr_source to "azure_di" or "none".
Metadata mapping: POMA fields → OpenSearch primitives
| POMA field | OpenSearch primitive | What it enables |
|---|---|---|
id (chunkset_uuid) | document _id | stable identity across re-ingests |
to_embed | knn_vector field | k-NN plugin ranking |
file_id | keyword field | filter scoping to one document |
chunkset_index | integer field, returned by default | assemble() match key |
page, depth, chunks, text | volume document, not indexed | content-free retrieval, cheatsheet assembly |
Frequently asked questions
How do I get Azure Document Intelligence results into OpenSearch for RAG?
Save the raw analyze result JSON, hand it to PrimeCut().ingest() (auto-detected, hierarchy rebuilt into a .poma archive), then loop records_from_archive() — write content to a volume, index the vector plus file_id/chunkset_index into a knn_vector mapping. Retrieve with a knn query plus assemble().
Why not just bulk-index Azure's markdown content string directly into OpenSearch?
Azure's PageHeader/PageFooter/PageBreak comments survive a naive split and land in the analyzed field BM25 scores. OpenSearch's hybrid query pipeline then normalizes and weight-sums that inflated lexical score against the vector score, letting boilerplate skew the fused ranking.
Is wiring Azure Document Intelligence to OpenSearch different from wiring it to Elasticsearch?
The Azure Document Intelligence side is identical. OpenSearch differs in field type (knn_vector, not dense_vector), engine (nmslib/faiss), and hybrid mechanism (a search pipeline instead of RRF rank) — POMA's ingest and assemble() adapt to whichever native client you use.
What happens to Azure Document Intelligence figures in an OpenSearch index?
Nothing reaches the index — Azure never returns cropped figure bytes, so there's no content to map into any field. POMA counts every figure as offloaded content in content_metadata instead of a silent gap.
Does OpenSearch retrieval need anything special for POMA's assemble() to work?
No extra flag — a standard knn query body returns indexed fields including file_id and chunkset_index by default, and assemble() auto-detects the OpenSearch result shape without any extra metadata request.
Related recipes
Same parser, different store: Azure Document Intelligence → Chroma · Azure Document Intelligence → Pinecone · Azure Document Intelligence → Elasticsearch
Foundations: The Optimal Chunker for Azure Document Intelligence · OpenSearch Chunking Strategy for RAG · All pipeline recipes · RAG chunking guide