Skip to content

Azure Document Intelligence to Pinecone: Chunking for RAG ​

The short answer: Azure Document Intelligence's prebuilt-layout produces a superb layout analysis; Pinecone gives you serverless vector search with namespaces and metadata filtering. Wired together naively — flatten, split, embed, stuff the text into metadata — the pipeline fails twice: the split destroys the hierarchy, and Azure's large HTML tables collide with Pinecone's ~40 KB metadata budget. The missing link is POMA: PrimeCut().ingest() consumes the raw analyze result (markdown or JSON mode, auto-detected) and emits hierarchy-preserving chunksets whose compact file_id/page/depth fields map cleanly onto Pinecone metadata, with the full text living in the portable .poma archive.

What each end of the pipeline actually provides ​

Azure Document Intelligence (prebuilt-layout) returns two shapes: markdown mode — one reading-order markdown string with inline HTML tables, <!-- PageBreak --> delimiters, and page-furniture comments — or classic JSON mode, a paragraphs[] array in span-offset order with title/sectionHeading roles plus tables[] cell grids. Multi-column de-interleaving happens server-side. What it doesn't provide: cross-page hierarchy, retrieval units, or figure bytes. Details: The Optimal Chunker for Azure Document Intelligence.

Pinecone provides serverless indexes, namespaces for hard partitioning, metadata filtering ($eq, $in, $gte, …), and sparse-dense hybrid support — with a metadata budget of roughly 40 KB per vector. What it doesn't provide: any opinion about what a vector should represent, or a place to keep long passages. It retrieves nearest neighbors of whatever you embedded. Details: The Optimal Chunks for the Best Retrieval in Pinecone.

The naive wiring, and where it breaks ​

The common recipe — flatten the analyze result to text, run a fixed-size splitter, embed, and upsert each fragment with its full text copied into Pinecone metadata (so query responses can return the passage) — breaks in a pair-specific way:

  • Azure's tables meet Pinecone's metadata cap. prebuilt-layout emits tables as inline HTML strings, and on table-heavy pages — rate cards, spec sheets, financial statements — a single fragment carrying its <table> markup can approach or blow past the ~40 KB metadata budget. The upsert is rejected or the pipeline starts truncating table markup mid-cell: silent content loss in the store that was supposed to return your evidence.
  • PageBreak delimiters and paragraph roles are flattened away before the splitter runs, so there is no page field to filter on and no heading lineage in what gets embedded. A retrieved fragment can't cite a page and doesn't know which section it came from.
  • Overlap wastes a metered resource. Splitter overlap embeds every boundary span twice — in a serverless index that means paying to store and query near-duplicate records that crowd top-k with redundant matches.

The fix is not a bigger metadata budget. It's separating concerns: embed compact, self-explanatory units; keep only filterable scalars in metadata; store the full text where full text belongs.

The pipeline, end to end ​

bash
pip install azure-ai-documentintelligence pinecone poma
python
import json
import os

from azure.ai.documentintelligence import DocumentIntelligenceClient
from azure.core.credentials import AzureKeyCredential
from pinecone import Pinecone
from poma import PrimeCut
from poma.utils import unpack_poma_archive

# 1. Azure Document Intelligence — your existing call, unchanged.
di = DocumentIntelligenceClient(
    endpoint=os.environ["AZURE_DI_ENDPOINT"],
    credential=AzureKeyCredential(os.environ["AZURE_DI_KEY"]),
)
with open("contract.pdf", "rb") as f:
    poller = di.begin_analyze_document(
        "prebuilt-layout",
        body=f,
        output_content_format="markdown",  # omit for classic JSON mode — POMA handles both
    )
analysis = poller.result()
with open("contract.azure-di.json", "w") as f:
    json.dump(analysis.as_dict(), f)

# 2. The missing link — raw analyze result in, chunks + chunksets out.
poma = PrimeCut()  # reads POMA_API_KEY
result = poma.ingest(  # Azure shape auto-detected
    "contract.azure-di.json",
    download_dir="store",
    filename="contract.poma",  # keep the archive — page numbers live in it
)

# Depth comes off the SDK objects; page is on the archive's chunk records only.
depth_of = {c.chunk_index: c.depth for c in result.chunks}
page_of = {
    c["chunk_index"]: c.get("page")
    for c in unpack_poma_archive(poma_archive_path="store/contract.poma")["chunks"]
}

# 3. Embed chunksets and upsert — compact metadata only, text stays in the .poma archive.
def embed(text: str) -> list[float]:
    ...  # your embedding model — return a dense vector

pc = Pinecone(api_key=os.environ["PINECONE_API_KEY"])
index = pc.Index("contracts")
index.upsert(
    vectors=[
        {
            "id": f"{cs.file_id}:{cs.chunkset_index}",
            "values": embed(cs.to_embed),
            "metadata": {
                "file_id": cs.file_id,
                "chunkset_index": cs.chunkset_index,
                # chunk indices are in document order; the last one is the leaf
                "page": page_of[cs.chunks[-1]],
                "depth": depth_of[cs.chunks[-1]],
            },
        }
        for cs in result.chunksets
    ],
    namespace="tenant-a",  # one namespace per tenant or corpus
)

# 4. Query scoped to namespace + document, then assemble context from IDs.
matches = index.query(
    vector=embed("What are the early termination conditions?"),
    top_k=5,
    namespace="tenant-a",
    filter={"file_id": {"$eq": result.chunksets[0].file_id}},
    include_metadata=True,
)
# Resolve match IDs against the .poma archive (or your doc store) and
# merge the retrieved chunksets into one deduplicated cheatsheet.

POMA validates the payload shape up front (a corrupted or mislabeled upload 422s immediately). Markdown mode is split on PageBreak comments with page-furniture comments dropped; JSON mode is reconstructed from paragraphs[] in span-offset order, roles mapped to heading levels, cell grids converted to HTML. Figures — which Azure never returns as bytes — are counted as offloaded in content_metadata, visible loss rather than silent. To force or suppress detection, set external_ocr_source to "azure_di" or "none".

Measured on our reference legal-document benchmark, chunksets plus cheatsheet assembly answer the same query with 337 tokens of retrieved context instead of 1,542 for a recursive character splitter, with zero information loss — methodology in Document Ingestion & Chunking for RAG.

Metadata mapping: POMA fields → Pinecone primitives ​

POMA chunk fieldPinecone primitiveWhat it enables
to_embeddense vector values (+ optional sparse values for hybrid)paraphrase + exact-term retrieval
file_idmetadata string, filter $eq/$inscope queries to one document
page of the chunkset's leaf chunk (from the archive's chunk records, filled from PageBreak delimiters or span-to-page mapping)metadata number, filter $eq/$gte/$ltepage-cited answers, page-range filters
depth of the leaf chunk (from title/sectionHeading roles, rebuilt cross-page)metadata numberfilter/re-rank by hierarchy level
chunkset_indexmetadata number, vector ID ({file_id}:{chunkset_index})stable ordering at assembly time
chunkset full textnot metadata (~40 KB cap) — .poma archive or doc store, keyed by vector IDcheatsheet assembly without truncation
tenant / corpusnamespacehard partitioning, zero cross-tenant scans

Frequently asked questions ​

How do I get Azure Document Intelligence results into Pinecone for RAG? ​

Save the raw analyze result (either mode), run PrimeCut().ingest() on it (auto-detected, hierarchy rebuilt), embed each chunkset's to_embed, and upsert with compact file_id/chunkset_index/page/depth metadata into a per-tenant namespace — page and depth taken from the chunkset's leaf chunk, with page read out of the .poma archive's chunk records. Assemble cheatsheets from the retrieved IDs.

How do I stay under Pinecone's metadata limit with Azure Document Intelligence chunks? ​

Don't copy chunk text into metadata — prebuilt-layout's inline HTML tables can run tens of kilobytes and collide with the ~40 KB budget. Store only compact scalars in metadata; keep full text in the .poma archive or your document store, keyed by vector ID.

Should I use Pinecone namespaces for Azure Document Intelligence documents? ​

Yes — one namespace per tenant, department, or corpus. Queries are namespace-scoped with zero cross-talk; within a namespace, filter on file_id for per-document scoping and page for page ranges.

Does sparse-dense hybrid search in Pinecone help with Azure Document Intelligence content? ​

Yes. Contracts, invoices, and spec sheets are full of exact tokens (clause numbers, part codes, defined terms) that dense embeddings blur; sparse-dense records carry both signals. Overlap-free chunksets keep top-k free of near-duplicates.

How do I keep Azure Document Intelligence page numbers in Pinecone? ​

Not by splitting the markdown — that flattens the PageBreak comments away. POMA records a page on every chunk record in the .poma archive (delimiters in markdown mode, span-to-page mapping in JSON mode); read it back with unpack_poma_archive, store the leaf chunk's page as a numeric page metadata field, and filter with $eq/$gte/$lte.

Same parser, different store: Azure Document Intelligence → Qdrant · Azure Document Intelligence → Weaviate · Azure Document Intelligence → Milvus

Same store, different parser: Mistral OCR → Pinecone · Textract → Pinecone · Unstructured.io → Pinecone

Foundations: The Optimal Chunker for Azure Document Intelligence · The Optimal Chunks for Pinecone · All pipeline recipes · RAG chunking guide