Azure Document Intelligence to Chroma: Chunking for RAG
The short answer: Azure Document Intelligence gives you an excellent layout analysis; Chroma gives you the fastest path from zero to a working retrieval loop. Wired together naively — flatten, split, collection.add — they still produce mediocre RAG, and worse: the corner cut in the prototype notebook is the one that ships. The missing link is POMA: PrimeCut().ingest() consumes the raw analyze result (auto-detected, markdown or JSON mode), emits hierarchy-preserving chunksets, and those land in a local PersistentClient collection with where-filterable hierarchy metadata — while the portable .poma archive keeps the same chunks ready for whatever store production lands on.
What each end of the pipeline actually provides
Azure Document Intelligence (prebuilt-layout) returns one of two shapes: markdown mode — a single reading-order markdown string with inline HTML tables, PageBreak delimiters, and page-furniture comments — or JSON mode, paragraphs[] ordered by span offset with role tags plus tables[] as cell grids. Multi-column de-interleaving happens server-side. What it doesn't provide: cross-page hierarchy (roles classify paragraphs, they don't nest them), retrieval units, or figure bytes. Details: The Optimal Chunker for Azure Document Intelligence.
Chroma provides a local-first, embedded store — chromadb.PersistentClient(path=...) needs no server and the same API scales to client/server later — with collections, where metadata filters, where_document content filters, and default or bring-your-own embedding functions. What it doesn't provide: any opinion about what a document in the collection should contain. Details: The Optimal Chunks for the Best Retrieval in Chroma.
The naive wiring, and where it breaks
The common recipe — take markdown mode's content string, run a fixed-size splitter, collection.add(documents=chunks) with empty metadata — breaks in a pair-specific way:
- The page anchor is a comment, and the splitter eats it. In markdown mode,
<!-- PageBreak -->comments are the only page signal in the entire result. A character splitter treats them as ordinary text and slices through them, so themetadatasyou pass to Chroma can never include apage— andwherefilters, Chroma's main retrieval lever beyond similarity, have nothing to act on. The enterprise question "where in the document does it say that?" becomes unanswerable. - Azure's HTML tables are cut mid-row.
prebuilt-layout's best output — intact tables with merged cells — becomes fragments of<td>soup, each embedded as a separate near-meaningless document in the collection. - The prototype's chunking sticks. Chroma demos get built in an afternoon and funded by Friday; re-chunking later means re-embedding, re-running every evaluation, and explaining the shift. The splitter in the notebook quietly becomes the production chunker — which is exactly why the prototype is the right place to chunk properly once.
The pipeline, end to end
pip install azure-ai-documentintelligence poma chromadbimport json
import os
import chromadb
from azure.ai.documentintelligence import DocumentIntelligenceClient
from azure.core.credentials import AzureKeyCredential
from poma import PrimeCut
from poma.utils import unpack_poma_archive
# 1. Your existing Azure Document Intelligence call — unchanged.
di = DocumentIntelligenceClient(
endpoint=os.environ["AZURE_DI_ENDPOINT"],
credential=AzureKeyCredential(os.environ["AZURE_DI_KEY"]),
)
with open("contract.pdf", "rb") as f:
poller = di.begin_analyze_document(
"prebuilt-layout",
body=f,
output_content_format="markdown", # omit for classic JSON mode — both work
)
with open("contract.azure-di.json", "w") as f:
json.dump(poller.result().as_dict(), f)
# 2. The missing link — raw analyze result in, chunks + chunksets out.
poma = PrimeCut() # reads POMA_API_KEY
result = poma.ingest( # Azure shape auto-detected
"contract.azure-di.json",
download_dir="store",
filename="contract.poma", # keep the archive — page numbers live in it
)
# Depth comes off the SDK objects; page is on the archive's chunk records only.
depth_of = {c.chunk_index: c.depth for c in result.chunks}
page_of = {
c["chunk_index"]: c.get("page")
for c in unpack_poma_archive(poma_archive_path="store/contract.poma")["chunks"]
}
# 3. A durable local collection — no server, same API as client/server.
chroma = chromadb.PersistentClient(path="./chroma")
collection = chroma.get_or_create_collection("contracts")
# 4. Add chunksets: Chroma embeds `documents` (default embedding function,
# or bring your own); hierarchy lands as where-filterable metadata.
leaves = [cs.chunks[-1] for cs in result.chunksets] # document order — last is the leaf
collection.add(
documents=[cs.to_embed for cs in result.chunksets],
metadatas=[
{
"file_id": cs.file_id,
"chunkset_index": cs.chunkset_index,
"page": page_of[leaf],
"depth": depth_of[leaf],
}
for cs, leaf in zip(result.chunksets, leaves)
],
ids=[f"{cs.file_id}:{cs.chunkset_index}" for cs in result.chunksets],
)
# 5. Query, scoped by the metadata the naive pipeline never had.
hits = collection.query(
query_texts=["What does the contract say about early termination?"],
n_results=5,
where={"file_id": result.chunksets[0].file_id},
)After retrieval, merge the hits instead of concatenating them: chunksets from the same section share ancestor headings, and deduplicating that shared lineage assembles one coherent context block — a cheatsheet. On our reference legal-document benchmark that assembly meant 337 tokens of context versus 1,542 for a recursive-splitter baseline, with zero information loss — methodology here.
POMA validates the payload up front (a corrupted or mislabeled upload 422s immediately, never a silently degraded index), handles both Azure shapes, drops the PageHeader/PageFooter/PageNumber furniture, keeps HTML tables whole, and counts Azure's byte-less figures as offloaded content in content_metadata — visible loss, never silent. To force or suppress detection, set external_ocr_source to "azure_di" or "none".
And because every ingest also produces a portable .poma archive, the collection you just built is a projection, not a commitment: when production lands on Qdrant, pgvector, or Milvus, you re-upsert the same chunksets with the same metadata — no second Azure analysis, no re-chunking, no re-evaluation.
Metadata mapping: POMA fields → Chroma primitives
| POMA chunk field | Chroma primitive | What it enables |
|---|---|---|
to_embed | documents= (embedded by default or custom embedding function) | similarity retrieval without managing vectors yourself |
file_id | scalar metadata, where={"file_id": ...} | scope queries to one document |
page of the chunkset's leaf chunk (from the archive's chunk records, filled from PageBreak / span offsets) | scalar metadata (int), where={"page": {"$lte": ...}} | page-cited answers, page-range filters |
depth of the leaf chunk (from Azure roles + rebuilt tree) | scalar metadata (int) | exclude deep appendix material via where |
chunkset_index | id construction ({file_id}:{chunkset_index}) / scalar metadata | stable ordering at assembly time |
| chunkset text | where_document={"$contains": ...} | exact-phrase conditions on content |
Frequently asked questions
How do I get Azure Document Intelligence output into Chroma for RAG?
Save the raw analyze result JSON, run PrimeCut().ingest() on it (auto-detected, hierarchy rebuilt), then collection.add the chunksets to a PersistentClient collection — to_embed as documents, file_id/page/depth as metadata. Chroma embeds them; where filters scope retrieval.
How do I keep Azure Document Intelligence page numbers filterable in Chroma?
Write them at ingest time. Markdown mode's only page anchors are the PageBreak comments, which a fixed-size splitter destroys. POMA splits on those delimiters (or reads span offsets in JSON mode) and records page on every chunk record in the .poma archive; read it back with unpack_poma_archive, attach the leaf chunk's page to the chunkset, and where={"page": {"$lte": 20}} works out of the box.
Do Azure Document Intelligence tables survive in a Chroma collection?
Yes — POMA keeps markdown mode's inline HTML tables intact, converts JSON mode's cell grids to HTML, and never cuts a table mid-row, so an embedded chunkset contains the whole table. A character splitter slices them into fragments that embed as noise.
Can I move a Chroma prototype to another vector database without re-running the Azure analysis?
Yes, twice over: the analysis is never re-run (BYOCR works from your saved JSON), and the .poma archive makes the chunking portable — moving to Qdrant, pgvector, or Milvus is a re-upsert of the same chunksets, not a re-chunking project.
Do I need to run a server to test an Azure Document Intelligence RAG pipeline with Chroma?
No — chromadb.PersistentClient(path=...) gives you a durable on-disk collection with no server process. One script and one afternoon tell you what proper chunking does for your retrieval; the same API scales to client/server later.
Related recipes
Same parser, different store: Azure Document Intelligence → Qdrant · Azure Document Intelligence → pgvector · Azure Document Intelligence → Milvus
Same store, different parser: Mistral OCR → Chroma · LlamaParse → Chroma · Marker → Chroma
Foundations: The Optimal Chunker for Azure Document Intelligence · The Optimal Chunks for Chroma · All pipeline recipes · RAG chunking guide