Source: http://www.poma-ai.com/docs/pipelines/azure-document-intelligence-to-pinecone

# The Missing Link Between Azure Document Intelligence and Optimal Retrieval in Pinecone

<ByAuthor />

**The short answer:** Azure Document Intelligence's `prebuilt-layout` produces a superb layout analysis; Pinecone gives you serverless vector search with namespaces and metadata filtering. Wired together naively — flatten, split, embed, stuff the text into metadata — the pipeline fails twice: the split destroys the hierarchy, and Azure's large HTML tables collide with Pinecone's ~40 KB metadata budget. The missing link is POMA: `PrimeCut().ingest()` consumes the raw analyze result (markdown or JSON mode, auto-detected) and emits hierarchy-preserving [chunksets](/learn/chunking/chunksets) whose compact `file_id`/`page`/`depth` fields map cleanly onto Pinecone metadata, with the full text living in the portable `.poma` archive.

## What each end of the pipeline actually provides

**Azure Document Intelligence** (`prebuilt-layout`) returns two shapes: markdown mode — one reading-order markdown string with inline HTML tables, `<!-- PageBreak -->` delimiters, and page-furniture comments — or classic JSON mode, a `paragraphs[]` array in span-offset order with `title`/`sectionHeading` roles plus `tables[]` cell grids. Multi-column de-interleaving happens server-side. What it doesn't provide: cross-page hierarchy, retrieval units, or figure bytes. Details: [The Optimal Chunker for Azure Document Intelligence](/optimal-chunker-azure-document-intelligence).

**Pinecone** provides serverless indexes, namespaces for hard partitioning, metadata filtering (`$eq`, `$in`, `$gte`, …), and sparse-dense hybrid support — with a metadata budget of roughly 40 KB per vector. What it doesn't provide: any opinion about what a vector should represent, or a place to keep long passages. It retrieves nearest neighbors of whatever you embedded. Details: [The Optimal Chunks for the Best Retrieval in Pinecone](/optimal-chunks-pinecone).

## The naive wiring, and where it breaks

The common recipe — flatten the analyze result to text, run a fixed-size splitter, embed, and upsert each fragment with its full text copied into Pinecone metadata (so query responses can return the passage) — breaks in a pair-specific way:

- **Azure's tables meet Pinecone's metadata cap.** `prebuilt-layout` emits tables as inline HTML strings, and on table-heavy pages — rate cards, spec sheets, financial statements — a single fragment carrying its `<table>` markup can approach or blow past the ~40 KB metadata budget. The upsert is rejected or the pipeline starts truncating table markup mid-cell: silent content loss in the store that was supposed to return your evidence.
- **`PageBreak` delimiters and paragraph roles are flattened away** before the splitter runs, so there is no `page` field to filter on and no heading lineage in what gets embedded. A retrieved fragment can't cite a page and doesn't know which section it came from.
- **Overlap wastes a metered resource.** Splitter overlap embeds every boundary span twice — in a serverless index that means paying to store and query near-duplicate records that crowd top-k with redundant matches.

The fix is not a bigger metadata budget. It's separating concerns: embed compact, self-explanatory units; keep only filterable scalars in metadata; store the full text where full text belongs.

## The pipeline, end to end

```bash
pip install azure-ai-documentintelligence pinecone poma
```

```python
import json
import os

from azure.ai.documentintelligence import DocumentIntelligenceClient
from azure.core.credentials import AzureKeyCredential
from pinecone import Pinecone
from poma import PrimeCut

# 1. Azure Document Intelligence — your existing call, unchanged.
di = DocumentIntelligenceClient(
    endpoint=os.environ["AZURE_DI_ENDPOINT"],
    credential=AzureKeyCredential(os.environ["AZURE_DI_KEY"]),
)
with open("contract.pdf", "rb") as f:
    poller = di.begin_analyze_document(
        "prebuilt-layout",
        body=f,
        output_content_format="markdown",  # omit for classic JSON mode — POMA handles both
    )
analysis = poller.result()
with open("contract.azure-di.json", "w") as f:
    json.dump(analysis.as_dict(), f)

# 2. The missing link — raw analyze result in, chunks + chunksets out.
poma = PrimeCut()  # reads POMA_API_KEY
result = poma.ingest("contract.azure-di.json")  # Azure shape auto-detected

# 3. Embed chunksets and upsert — compact metadata only, text stays in the .poma archive.
def embed(text: str) -> list[float]:
    ...  # your embedding model — return a dense vector

pc = Pinecone(api_key=os.environ["PINECONE_API_KEY"])
index = pc.Index("contracts")
index.upsert(
    vectors=[
        {
            "id": f"{cs.file_id}:{i}",
            "values": embed(cs.to_embed),
            "metadata": {
                "file_id": cs.file_id,
                "page": cs.page,
                "depth": cs.depth,
                "chunk_index": cs.chunk_index,
            },
        }
        for i, cs in enumerate(result.chunksets)
    ],
    namespace="tenant-a",  # one namespace per tenant or corpus
)

# 4. Query scoped to namespace + document, then assemble context from IDs.
matches = index.query(
    vector=embed("What are the early termination conditions?"),
    top_k=5,
    namespace="tenant-a",
    filter={"file_id": {"$eq": result.chunksets[0].file_id}},
    include_metadata=True,
)
# Resolve match IDs against the .poma archive (or your doc store) and
# merge the retrieved chunksets into one deduplicated cheatsheet.
```

POMA validates the payload shape up front (a corrupted or mislabeled upload 422s immediately). Markdown mode is split on `PageBreak` comments with page-furniture comments dropped; JSON mode is reconstructed from `paragraphs[]` in span-offset order, roles mapped to heading levels, cell grids converted to HTML. Figures — which Azure never returns as bytes — are counted as offloaded in `content_metadata`, visible loss rather than silent. To force or suppress detection, set `external_ocr_source` to `"azure_di"` or `"none"`.

Measured on our reference legal-document benchmark, chunksets plus cheatsheet assembly answer the same query with **337 tokens** of retrieved context instead of **1,542** for a recursive character splitter, with zero information loss — methodology in [Document Ingestion & Chunking for RAG](/document-ingestion-chunking-rag).

## Metadata mapping: POMA fields → Pinecone primitives

| POMA chunk field | Pinecone primitive | What it enables |
| --- | --- | --- |
| `to_embed` | dense vector `values` (+ optional sparse values for hybrid) | paraphrase + exact-term retrieval |
| `file_id` | metadata string, filter `$eq`/`$in` | scope queries to one document |
| `page` (from `PageBreak` delimiters or span-to-page mapping) | metadata number, filter `$eq`/`$gte`/`$lte` | page-cited answers, page-range filters |
| `depth` (from `title`/`sectionHeading` roles, rebuilt cross-page) | metadata number | filter/re-rank by hierarchy level |
| `chunk_index` | metadata number | stable ordering at assembly time |
| chunkset full text | **not metadata** (~40 KB cap) — `.poma` archive or doc store, keyed by vector ID | cheatsheet assembly without truncation |
| tenant / corpus | namespace | hard partitioning, zero cross-tenant scans |

## Frequently asked questions

### How do I get Azure Document Intelligence results into Pinecone for RAG?

Save the raw analyze result (either mode), run `PrimeCut().ingest()` on it (auto-detected, hierarchy rebuilt), embed each chunkset's `to_embed`, and upsert with compact `file_id`/`page`/`depth`/`chunk_index` metadata into a per-tenant namespace. Assemble cheatsheets from the retrieved IDs.

### How do I stay under Pinecone's metadata limit with Azure Document Intelligence chunks?

Don't copy chunk text into metadata — prebuilt-layout's inline HTML tables can run tens of kilobytes and collide with the ~40 KB budget. Store only compact scalars in metadata; keep full text in the `.poma` archive or your document store, keyed by vector ID.

### Should I use Pinecone namespaces for Azure Document Intelligence documents?

Yes — one namespace per tenant, department, or corpus. Queries are namespace-scoped with zero cross-talk; within a namespace, filter on `file_id` for per-document scoping and `page` for page ranges.

### Does sparse-dense hybrid search in Pinecone help with Azure Document Intelligence content?

Yes. Contracts, invoices, and spec sheets are full of exact tokens (clause numbers, part codes, defined terms) that dense embeddings blur; sparse-dense records carry both signals. Overlap-free chunksets keep top-k free of near-duplicates.

### How do I keep Azure Document Intelligence page numbers in Pinecone?

Not by splitting the markdown — that flattens the `PageBreak` comments away. POMA recovers a page for every chunk (delimiters in markdown mode, span-to-page mapping in JSON mode); store it as a numeric `page` metadata field and filter with `$eq`/`$gte`/`$lte`.

## Related recipes

Same parser, different store: [Azure Document Intelligence → Qdrant](/pipelines/azure-document-intelligence-to-qdrant) · [Azure Document Intelligence → Weaviate](/pipelines/azure-document-intelligence-to-weaviate) · [Azure Document Intelligence → Milvus](/pipelines/azure-document-intelligence-to-milvus)

Same store, different parser: [Mistral OCR → Pinecone](/pipelines/mistral-ocr-to-pinecone) · [Textract → Pinecone](/pipelines/textract-to-pinecone) · [Unstructured.io → Pinecone](/pipelines/unstructured-to-pinecone)

Foundations: [The Optimal Chunker for Azure Document Intelligence](/optimal-chunker-azure-document-intelligence) · [The Optimal Chunks for Pinecone](/optimal-chunks-pinecone) · [All pipeline recipes](/pipelines/) · [RAG chunking guide](/guides/rag-chunking/)