Source: http://www.poma-ai.com/docs/pipelines/azure-document-intelligence-to-lancedb

# The Missing Link Between Azure Document Intelligence and Optimal Retrieval in LanceDB

<ByAuthor />

**The short answer:** Azure Document Intelligence's `prebuilt-layout` model gives you excellent reading order and table extraction; LanceDB gives you an embedded, Arrow-native table that needs no server to operate. Wired together naively — flatten, fixed-split, `create_table` — the pipeline still produces mediocre RAG, because nothing in between rebuilds the document's hierarchy or keeps metadata consistent across documents captured in Azure's two different output modes. The missing link is POMA: `PrimeCut().ingest()` consumes the raw analyze result (auto-detected, either shape), emits hierarchy-preserving [chunksets](/learn/chunking/chunksets), and `vektoria` keeps the LanceDB table content-free while `assemble()` reconstructs prompt-ready context from a volume.

## What each end of the pipeline actually provides

**Azure Document Intelligence** returns one of two shapes: markdown mode's single reading-order string with inline HTML tables and `<!-- PageBreak -->` delimiters, or JSON mode's `paragraphs[]` (ordered by span offset, role-tagged) plus `tables[]` cell grids. Neither shape links a page-41 heading back to the page-3 title that opened its chapter, and neither decides what a retrieval unit should be. Details: [The Optimal Chunker for Azure Document Intelligence](/optimal-chunker-azure-document-intelligence).

**LanceDB** provides an embedded, Arrow/Lance-backed table — a `vector` column alongside ordinary scalar columns, filterable with a SQL-like `.where(...)` predicate, with an optional full-text index for hybrid queries. What it doesn't provide: schema flexibility across writes — `create_table` infers its Arrow schema from the first batch of rows you hand it. Details: [LanceDB Chunking Strategy for RAG](/optimal-chunks-lancedb).

## The naive wiring, and where it breaks

The common recipe — flatten Azure's `content` string (or `paragraphs[]`) with a fixed-size splitter, embed, `create_table`/append rows — breaks in a way specific to this pair:

- **Azure's two shapes become LanceDB's schema-mismatch risk.** Azure Document Intelligence emits either markdown mode or JSON mode, and a hand-rolled pipeline that maps each to metadata slightly differently per document — `page` as an `int` recovered from a `PageBreak` comment for one document, a `string` parsed loosely from a paragraph role for another — hands LanceDB inconsistent column types across batches. `create_table` accepts the first batch's inferred schema and silently commits to it; the next `add()` call with a differently-typed `page` column fails at append time, often long after the first document's pipeline "worked."
- **Overlap still crowds the table, even without a schema problem.** Splitter overlap (typically 10–20%) duplicates every boundary span into the table and the vector index built over it — worse on Azure's table-heavy documents, where a table sliced mid-row by the splitter gets embedded twice, once per half, and both rows rank near-identically at query time.
- **Azure's missing figure bytes are easy to lose track of.** Azure Document Intelligence never returns cropped figure bytes, only regions with metadata; a hand-rolled pipeline that flattens straight to text has no place to record that a figure existed at all, so the gap is invisible instead of counted.

## The pipeline, end to end

```bash
pip install azure-ai-documentintelligence lancedb
```

```python
import json
import os

import lancedb
from azure.ai.documentintelligence import DocumentIntelligenceClient
from azure.core.credentials import AzureKeyCredential
from poma import PrimeCut
from poma.vektoria import assemble, records_from_archive, Volume
from poma.embeddings import get_embedder

# 1. Azure Document Intelligence — your existing call, unchanged.
di = DocumentIntelligenceClient(
    endpoint=os.environ["AZURE_DI_ENDPOINT"],
    credential=AzureKeyCredential(os.environ["AZURE_DI_KEY"]),
)
with open("document.pdf", "rb") as f:
    poller = di.begin_analyze_document(
        "prebuilt-layout",
        body=f,
        output_content_format="markdown",  # omit for classic JSON mode — POMA handles both
    )
analysis = poller.result()
with open("result.azure-di.json", "w") as f:
    json.dump(analysis.as_dict(), f)

# 2. The missing link — raw result JSON in, content-free archive out.
client = PrimeCut()  # reads POMA_API_KEY
client.ingest("result.azure-di.json", download_dir="archives", filename="doc.poma")

embedder = get_embedder("local:BAAI/bge-small-en-v1.5")
vol = Volume("s3://your-bucket/poma")
db = lancedb.connect("s3://your-bucket/lancedb")

# 3. Content-free ingest — LanceDB holds only routing metadata; content lives on the volume.
records = records_from_archive("archives/doc.poma")  # -> list[Record] (id, text, payload)
rows = []
for r in records:
    vol.write_doc(r.payload["file_id"], {"file_id": r.payload["file_id"],
                                          "chunks": r.payload["chunks"], "text": r.text})
    rows.append({
        "id": r.id,
        "vector": embedder.embed([r.text])[0],
        "file_id": r.payload["file_id"],
        "chunkset_index": r.payload["chunkset_index"],
    })

tbl = db.create_table("poma", data=rows, exist_ok=True)

# 4. Vector search retrieval, then assemble prompt-ready context.
qv = embedder.embed(["early termination conditions"])[0]
results = tbl.search(qv).where("file_id = 'document.pdf'").limit(10).to_list()
context = assemble(results, volume=vol)  # -> [{"file_id", "content"}, ...]
```

Because the analysis already ran, POMA charges only for the downstream structure and chunking value — the OCR front-end is skipped. LanceDB never stores document content: `vektoria`'s `Volume` holds it, and `assemble()` deduplicates ancestor lineage across retrieved chunksets into one prompt-ready cheatsheet — the discipline behind our reference legal-document benchmark answering the same query with **337 tokens** instead of **1,542** for a recursive-splitter baseline. [Methodology here](/document-ingestion-chunking-rag).

## Metadata mapping: POMA fields → LanceDB primitives

| POMA chunk field | LanceDB primitive | What it enables |
| --- | --- | --- |
| `to_embed` | `vector` column | ANN `search()` |
| `file_id` | scalar column | `.where("file_id = '...'")` scope |
| `page` (from Azure `PageBreak`/span offsets) | volume document, not a column | content-free retrieval, cheatsheet assembly |
| `depth` | volume document, not a column | content-free retrieval, cheatsheet assembly |
| `chunkset_index` | scalar column, returned by default | `assemble()` dedupe key |
| chunk content | not stored in the table — lives on `Volume` | content-free table, smaller storage |

## Frequently asked questions

### How do I get Azure Document Intelligence results into LanceDB for RAG?

Run `begin_analyze_document` with `prebuilt-layout` as you already do, and save the raw analyze result JSON. Hand that unmodified file to `PrimeCut`, which auto-detects the Azure shape and rebuilds the cross-page heading hierarchy into chunks and chunksets. Embed each chunkset's `to_embed` text, write it as a row in an Arrow-backed table with `file_id`/`chunkset_index` columns, and retrieve through `vektoria`'s `assemble()`, which fetches content from a volume and returns prompt-ready cheatsheets.

### What happens to Azure Document Intelligence's figures when indexed in LanceDB?

Azure Document Intelligence never returns cropped figure bytes in the analyze result, so POMA counts every figure as offloaded content in `content_metadata` before anything reaches LanceDB — from the `figures[]` array in JSON mode, or from neutralized image references in markdown mode. The loss is visible in the ingest result, never a silent gap in your LanceDB table.

### What columns should Azure Document Intelligence chunks map to in a LanceDB table?

A `vector` column sized to your embedder, plus `file_id` and `chunkset_index` as ordinary scalar columns. `page` and `depth` — recovered from Azure's `PageBreak` comments or paragraph span offsets — stay on the volume document with the rest of the chunkset content, not as separate columns. `create_table` infers the schema from your first batch of rows, so every document's rows need the same column names and types.

### Why might a LanceDB table reject rows from a mixed batch of Azure Document Intelligence documents?

`create_table` infers its Arrow schema from the first batch you write. Azure Document Intelligence returns two different shapes — markdown mode and JSON mode — and a hand-rolled pipeline that maps them to metadata differently per document (say, `page` as an `int` for one and a string parsed from a comment for another) hands LanceDB inconsistent column types across batches, which surfaces as an append-time schema mismatch. POMA normalizes both Azure shapes into the same chunk fields before anything reaches LanceDB, so every row shares one schema regardless of capture mode.

### Does LanceDB need any extra query flags to work with POMA's assemble()?

No. LanceDB returns all columns by default from a `search().to_list()` call, so `file_id` and `chunkset_index` arrive on every hit without an extra projection, metadata flag, or dialect setting — unlike Redis or MongoDB Atlas, which both require an explicit ask at query time.

## Related recipes

Same parser, different store: [Azure Document Intelligence → Qdrant](/pipelines/azure-document-intelligence-to-qdrant) · [Azure Document Intelligence → Pinecone](/pipelines/azure-document-intelligence-to-pinecone) · [Azure Document Intelligence → Redis](/pipelines/azure-document-intelligence-to-redis) · [Azure Document Intelligence → MongoDB Atlas](/pipelines/azure-document-intelligence-to-mongodb-atlas)

Foundations: [The Optimal Chunker for Azure Document Intelligence](/optimal-chunker-azure-document-intelligence) · [LanceDB Chunking Strategy for RAG](/optimal-chunks-lancedb) · [All pipeline recipes](/pipelines/) · [RAG chunking guide](/guides/rag-chunking/)