Skip to content

Azure Document Intelligence to Chroma: Chunking for RAG ​

The short answer: Azure Document Intelligence gives you an excellent layout analysis; Chroma gives you the fastest path from zero to a working retrieval loop. Wired together naively — flatten, split, collection.add — they still produce mediocre RAG, and worse: the corner cut in the prototype notebook is the one that ships. The missing link is POMA: PrimeCut().ingest() consumes the raw analyze result (auto-detected, markdown or JSON mode), emits hierarchy-preserving chunksets, and those land in a local PersistentClient collection with where-filterable hierarchy metadata — while the portable .poma archive keeps the same chunks ready for whatever store production lands on.

What each end of the pipeline actually provides ​

Azure Document Intelligence (prebuilt-layout) returns one of two shapes: markdown mode — a single reading-order markdown string with inline HTML tables, PageBreak delimiters, and page-furniture comments — or JSON mode, paragraphs[] ordered by span offset with role tags plus tables[] as cell grids. Multi-column de-interleaving happens server-side. What it doesn't provide: cross-page hierarchy (roles classify paragraphs, they don't nest them), retrieval units, or figure bytes. Details: The Optimal Chunker for Azure Document Intelligence.

Chroma provides a local-first, embedded store — chromadb.PersistentClient(path=...) needs no server and the same API scales to client/server later — with collections, where metadata filters, where_document content filters, and default or bring-your-own embedding functions. What it doesn't provide: any opinion about what a document in the collection should contain. Details: The Optimal Chunks for the Best Retrieval in Chroma.

The naive wiring, and where it breaks ​

The common recipe — take markdown mode's content string, run a fixed-size splitter, collection.add(documents=chunks) with empty metadata — breaks in a pair-specific way:

  • The page anchor is a comment, and the splitter eats it. In markdown mode, <!-- PageBreak --> comments are the only page signal in the entire result. A character splitter treats them as ordinary text and slices through them, so the metadatas you pass to Chroma can never include a page — and where filters, Chroma's main retrieval lever beyond similarity, have nothing to act on. The enterprise question "where in the document does it say that?" becomes unanswerable.
  • Azure's HTML tables are cut mid-row. prebuilt-layout's best output — intact tables with merged cells — becomes fragments of <td> soup, each embedded as a separate near-meaningless document in the collection.
  • The prototype's chunking sticks. Chroma demos get built in an afternoon and funded by Friday; re-chunking later means re-embedding, re-running every evaluation, and explaining the shift. The splitter in the notebook quietly becomes the production chunker — which is exactly why the prototype is the right place to chunk properly once.

The pipeline, end to end ​

bash
pip install azure-ai-documentintelligence poma chromadb
python
import json
import os

import chromadb
from azure.ai.documentintelligence import DocumentIntelligenceClient
from azure.core.credentials import AzureKeyCredential
from poma import PrimeCut
from poma.utils import unpack_poma_archive

# 1. Your existing Azure Document Intelligence call — unchanged.
di = DocumentIntelligenceClient(
    endpoint=os.environ["AZURE_DI_ENDPOINT"],
    credential=AzureKeyCredential(os.environ["AZURE_DI_KEY"]),
)
with open("contract.pdf", "rb") as f:
    poller = di.begin_analyze_document(
        "prebuilt-layout",
        body=f,
        output_content_format="markdown",  # omit for classic JSON mode — both work
    )
with open("contract.azure-di.json", "w") as f:
    json.dump(poller.result().as_dict(), f)

# 2. The missing link — raw analyze result in, chunks + chunksets out.
poma = PrimeCut()  # reads POMA_API_KEY
result = poma.ingest(  # Azure shape auto-detected
    "contract.azure-di.json",
    download_dir="store",
    filename="contract.poma",  # keep the archive — page numbers live in it
)

# Depth comes off the SDK objects; page is on the archive's chunk records only.
depth_of = {c.chunk_index: c.depth for c in result.chunks}
page_of = {
    c["chunk_index"]: c.get("page")
    for c in unpack_poma_archive(poma_archive_path="store/contract.poma")["chunks"]
}

# 3. A durable local collection — no server, same API as client/server.
chroma = chromadb.PersistentClient(path="./chroma")
collection = chroma.get_or_create_collection("contracts")

# 4. Add chunksets: Chroma embeds `documents` (default embedding function,
#    or bring your own); hierarchy lands as where-filterable metadata.
leaves = [cs.chunks[-1] for cs in result.chunksets]  # document order — last is the leaf
collection.add(
    documents=[cs.to_embed for cs in result.chunksets],
    metadatas=[
        {
            "file_id": cs.file_id,
            "chunkset_index": cs.chunkset_index,
            "page": page_of[leaf],
            "depth": depth_of[leaf],
        }
        for cs, leaf in zip(result.chunksets, leaves)
    ],
    ids=[f"{cs.file_id}:{cs.chunkset_index}" for cs in result.chunksets],
)

# 5. Query, scoped by the metadata the naive pipeline never had.
hits = collection.query(
    query_texts=["What does the contract say about early termination?"],
    n_results=5,
    where={"file_id": result.chunksets[0].file_id},
)

After retrieval, merge the hits instead of concatenating them: chunksets from the same section share ancestor headings, and deduplicating that shared lineage assembles one coherent context block — a cheatsheet. On our reference legal-document benchmark that assembly meant 337 tokens of context versus 1,542 for a recursive-splitter baseline, with zero information loss — methodology here.

POMA validates the payload up front (a corrupted or mislabeled upload 422s immediately, never a silently degraded index), handles both Azure shapes, drops the PageHeader/PageFooter/PageNumber furniture, keeps HTML tables whole, and counts Azure's byte-less figures as offloaded content in content_metadata — visible loss, never silent. To force or suppress detection, set external_ocr_source to "azure_di" or "none".

And because every ingest also produces a portable .poma archive, the collection you just built is a projection, not a commitment: when production lands on Qdrant, pgvector, or Milvus, you re-upsert the same chunksets with the same metadata — no second Azure analysis, no re-chunking, no re-evaluation.

Metadata mapping: POMA fields → Chroma primitives ​

POMA chunk fieldChroma primitiveWhat it enables
to_embeddocuments= (embedded by default or custom embedding function)similarity retrieval without managing vectors yourself
file_idscalar metadata, where={"file_id": ...}scope queries to one document
page of the chunkset's leaf chunk (from the archive's chunk records, filled from PageBreak / span offsets)scalar metadata (int), where={"page": {"$lte": ...}}page-cited answers, page-range filters
depth of the leaf chunk (from Azure roles + rebuilt tree)scalar metadata (int)exclude deep appendix material via where
chunkset_indexid construction ({file_id}:{chunkset_index}) / scalar metadatastable ordering at assembly time
chunkset textwhere_document={"$contains": ...}exact-phrase conditions on content

Frequently asked questions ​

How do I get Azure Document Intelligence output into Chroma for RAG? ​

Save the raw analyze result JSON, run PrimeCut().ingest() on it (auto-detected, hierarchy rebuilt), then collection.add the chunksets to a PersistentClient collection — to_embed as documents, file_id/page/depth as metadata. Chroma embeds them; where filters scope retrieval.

How do I keep Azure Document Intelligence page numbers filterable in Chroma? ​

Write them at ingest time. Markdown mode's only page anchors are the PageBreak comments, which a fixed-size splitter destroys. POMA splits on those delimiters (or reads span offsets in JSON mode) and records page on every chunk record in the .poma archive; read it back with unpack_poma_archive, attach the leaf chunk's page to the chunkset, and where={"page": {"$lte": 20}} works out of the box.

Do Azure Document Intelligence tables survive in a Chroma collection? ​

Yes — POMA keeps markdown mode's inline HTML tables intact, converts JSON mode's cell grids to HTML, and never cuts a table mid-row, so an embedded chunkset contains the whole table. A character splitter slices them into fragments that embed as noise.

Can I move a Chroma prototype to another vector database without re-running the Azure analysis? ​

Yes, twice over: the analysis is never re-run (BYOCR works from your saved JSON), and the .poma archive makes the chunking portable — moving to Qdrant, pgvector, or Milvus is a re-upsert of the same chunksets, not a re-chunking project.

Do I need to run a server to test an Azure Document Intelligence RAG pipeline with Chroma? ​

No — chromadb.PersistentClient(path=...) gives you a durable on-disk collection with no server process. One script and one afternoon tell you what proper chunking does for your retrieval; the same API scales to client/server later.

Same parser, different store: Azure Document Intelligence → Qdrant · Azure Document Intelligence → pgvector · Azure Document Intelligence → Milvus

Same store, different parser: Mistral OCR → Chroma · LlamaParse → Chroma · Marker → Chroma

Foundations: The Optimal Chunker for Azure Document Intelligence · The Optimal Chunks for Chroma · All pipeline recipes · RAG chunking guide