Source: http://www.poma-ai.com/docs/pipelines/azure-document-intelligence-to-pgvector

# The Missing Link Between Azure Document Intelligence and Optimal Retrieval in pgvector

<ByAuthor />

**The short answer:** Azure Document Intelligence gives you an excellent layout analysis; pgvector gives you ANN search inside the database you already run. Wired together naively — flatten, split, embed, insert into one table — they still produce mediocre RAG, because the structure the layout model recovered never reaches the relational engine built to hold it. The missing link is POMA: `PrimeCut().ingest()` consumes the raw analyze result (auto-detected, markdown or JSON mode), emits hierarchy-preserving [chunksets](/learn/chunking/chunksets), and those land in Postgres as two joined tables — so one SQL statement runs the vector search *and* reconstructs each hit's full lineage.

## What each end of the pipeline actually provides

**Azure Document Intelligence** (`prebuilt-layout`) returns one of two shapes: markdown mode — a single reading-order markdown string with inline HTML tables, `PageBreak` delimiters, and page-furniture comments — or JSON mode, `paragraphs[]` ordered by span offset with `role` tags (`title`, `sectionHeading`, …) plus `tables[]` as cell grids. Multi-column de-interleaving happens server-side. What it doesn't provide: cross-page hierarchy (roles classify paragraphs, they don't nest them), retrieval units, or figure bytes. Details: [The Optimal Chunker for Azure Document Intelligence](/optimal-chunker-azure-document-intelligence).

**pgvector** provides the `vector` column type, HNSW and IVFFlat indexes, distance operators that drop into `ORDER BY`, `tsvector` full-text search in the same engine, and transactions. What it doesn't provide: any opinion about what a row should contain. It orders rows by whatever distance you ask for, over whatever you inserted. Details: [The Optimal Chunks for the Best Retrieval in pgvector](/optimal-chunks-pgvector).

## The naive wiring, and where it breaks

The common recipe — take `content` (or flatten `paragraphs[]` to text), run a recursive character splitter, embed, `INSERT` into a single `(id, text, embedding)` table — breaks in a pair-specific way:

- **Azure's structure is flattened at the door of the one store that could model it.** Span-ordered, role-tagged paragraphs collapse into an untyped text column. `role` could have been a `depth` int, `PageBreak` delimiters a `page` column, the heading tree a join table — Postgres is a relational database, and the naive pipeline hands it a blob.
- **Page anchors vanish.** In markdown mode the `PageBreak` comments are the only page signal; a character splitter destroys them, so no column can ever say which page a chunk came from, and page-cited answers become impossible.
- **Overlap bloats the HNSW index.** Splitter overlap embeds every boundary span twice — a fatter table, a slower index build, and `LIMIT 5` results where hits 2 and 3 are near-duplicates of hit 1.
- **Loop inserts without a transaction** leave a half-ingested filing when the pipeline crashes mid-document — a silently partial corpus that Postgres would have prevented for free.

## The pipeline, end to end

```bash
pip install azure-ai-documentintelligence poma "psycopg[binary]"
```

The schema mirrors the document's real hierarchy — chunks and chunksets as two joined tables (full design discussion on the [pgvector chunks page](/optimal-chunks-pgvector)):

```sql
CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE chunks (
    id          bigserial PRIMARY KEY,
    file_id     text NOT NULL,
    chunk_index int  NOT NULL,
    depth       int  NOT NULL,   -- from Azure roles + POMA's rebuilt heading tree
    page        int,             -- from PageBreak splits or span offsets
    content     text NOT NULL,   -- Azure's HTML tables arrive intact, never cut
    UNIQUE (file_id, chunk_index)
);

CREATE TABLE chunksets (
    id        bigserial PRIMARY KEY,
    file_id   text NOT NULL,
    to_embed  text NOT NULL,
    embedding vector(384) NOT NULL,
    tsv       tsvector GENERATED ALWAYS AS (to_tsvector('english', to_embed)) STORED
);

CREATE TABLE chunkset_members (
    chunkset_id bigint NOT NULL REFERENCES chunksets(id) ON DELETE CASCADE,
    chunk_id    bigint NOT NULL REFERENCES chunks(id)    ON DELETE CASCADE,
    position    int    NOT NULL, -- 0 = document root … last = the leaf
    PRIMARY KEY (chunkset_id, chunk_id)
);

CREATE INDEX ON chunksets USING hnsw (embedding vector_cosine_ops);
CREATE INDEX ON chunksets USING gin (tsv);
```

```python
import json
import os

import psycopg
from azure.ai.documentintelligence import DocumentIntelligenceClient
from azure.core.credentials import AzureKeyCredential
from poma import PrimeCut

# 1. Your existing Azure Document Intelligence call — unchanged.
di = DocumentIntelligenceClient(
    endpoint=os.environ["AZURE_DI_ENDPOINT"],
    credential=AzureKeyCredential(os.environ["AZURE_DI_KEY"]),
)
with open("contract.pdf", "rb") as f:
    poller = di.begin_analyze_document(
        "prebuilt-layout",
        body=f,
        output_content_format="markdown",  # omit for classic JSON mode — both work
    )
with open("contract.azure-di.json", "w") as f:
    json.dump(poller.result().as_dict(), f)

# 2. The missing link — raw analyze result in, chunks + chunksets out.
poma = PrimeCut()  # reads POMA_API_KEY
result = poma.ingest("contract.azure-di.json")  # Azure shape auto-detected

# 3. One transaction: the document lands atomically, or not at all.
with psycopg.connect() as conn, conn.transaction():
    chunk_ids = {}
    for chunk in result.chunks:
        row = conn.execute(
            "INSERT INTO chunks (file_id, chunk_index, depth, page, content) "
            "VALUES (%s, %s, %s, %s, %s) RETURNING id",
            (chunk.file_id, chunk.chunk_index, chunk.depth, chunk.page, chunk.content),
        ).fetchone()
        chunk_ids[chunk.chunk_index] = row[0]

    for cs in result.chunksets:
        row = conn.execute(
            "INSERT INTO chunksets (file_id, to_embed, embedding) "
            "VALUES (%s, %s, %s) RETURNING id",
            (cs.file_id, cs.to_embed, embed(cs.to_embed)),  # your embedding model
        ).fetchone()
        for position, chunk_index in enumerate(cs.chunks):
            conn.execute(
                "INSERT INTO chunkset_members (chunkset_id, chunk_id, position) "
                "VALUES (%s, %s, %s)",
                (row[0], chunk_ids[chunk_index], position),
            )
```

Retrieval is where the pairing pays off — ANN search and full lineage reconstruction in a single round trip:

```sql
WITH hits AS (
    SELECT id, embedding <=> %(query_vec)s AS distance
    FROM chunksets
    WHERE file_id = %(file_id)s          -- ordinary indexed predicate
    ORDER BY embedding <=> %(query_vec)s
    LIMIT 5
)
SELECT h.distance, m.position, c.depth, c.page, c.content
FROM hits h
JOIN chunkset_members m ON m.chunkset_id = h.id
JOIN chunks c           ON c.id = m.chunk_id
ORDER BY h.distance, m.position;
```

Each hit comes back as ordered rows from root to leaf — every ancestor heading with its page and depth — the raw material for cheatsheet assembly: deduplicate the shared lineage across hits and merge into one prompt-ready block. On our reference legal-document benchmark that meant **337 tokens** of context versus **1,542** for a recursive-splitter baseline, with zero information loss — [methodology here](/document-ingestion-chunking-rag).

POMA validates the payload up front (a corrupted or mislabeled upload 422s immediately, never a silently degraded index), handles both Azure shapes, drops the `PageHeader`/`PageFooter`/`PageNumber` furniture, keeps HTML tables whole, and counts Azure's byte-less figures as offloaded content in `content_metadata` — visible loss, never silent. To force or suppress detection, set `external_ocr_source` to `"azure_di"` or `"none"`.

## Metadata mapping: POMA fields → Postgres primitives

| POMA chunk field | Postgres primitive | What it enables |
| --- | --- | --- |
| `to_embed` | `vector(384)` column + generated `tsvector` (GIN) | ANN + keyword hybrid in one engine |
| `file_id` | `text` column, indexed `WHERE` predicate | scope queries to one document |
| `page` (from `PageBreak` / span offsets) | `int` column | page-cited answers, `BETWEEN` range filters |
| `depth` (from Azure roles + rebuilt tree) | `int` column | `WHERE depth <= 2` structural filters |
| `chunk_index` | `UNIQUE (file_id, chunk_index)` | stable ordering at assembly time |
| chunkset lineage | `chunkset_members` join table | full ancestry in the same query as the ANN search |

## Frequently asked questions

### How do I get Azure Document Intelligence output into pgvector for RAG?

Save the raw analyze result JSON, run `PrimeCut().ingest()` on it (auto-detected, hierarchy rebuilt), then insert chunks and chunksets as two joined tables in one transaction, embedding each chunkset's `to_embed` into a `vector` column. Retrieval is one SQL statement: ANN search plus a lineage `JOIN`.

### What Postgres schema fits Azure Document Intelligence results?

`chunks` (file_id, chunk_index, depth, page, content) and `chunksets` (to_embed, `vector` column, generated `tsvector`), linked by a membership table. The structure `prebuilt-layout` recovered becomes typed, indexed columns instead of one flattened text blob.

### Can I keep Azure Document Intelligence page numbers queryable in Postgres?

Yes — POMA recovers `page` from the `PageBreak` comments (markdown mode) or span offsets (JSON mode) and emits it per chunk. As an `int` column it's an ordinary indexed predicate, and the lineage `JOIN` returns pages for citation. Flatten-and-split pipelines destroy this anchor.

### Should I use tsvector hybrid search for Azure Document Intelligence content in pgvector?

Yes: contracts, filings, and forms are full of exact tokens (clause numbers, invoice IDs, defined terms) that dense embeddings blur. A generated `tsvector` over `to_embed` with a GIN index gives the keyword lane in the same engine; fuse with reciprocal rank fusion in plain SQL.

### Why ingest Azure Document Intelligence chunks in a single Postgres transaction?

Because a half-ingested filing silently returns partial answers. Postgres makes atomicity free — wrap chunk, chunkset, and membership inserts in one transaction and the document lands whole or not at all, a guarantee standalone vector stores generally can't make across related records.

## Related recipes

Same parser, different store: [Azure Document Intelligence → Qdrant](/pipelines/azure-document-intelligence-to-qdrant) · [Azure Document Intelligence → Milvus](/pipelines/azure-document-intelligence-to-milvus) · [Azure Document Intelligence → Weaviate](/pipelines/azure-document-intelligence-to-weaviate)

Same store, different parser: [Mistral OCR → pgvector](/pipelines/mistral-ocr-to-pgvector) · [Textract → pgvector](/pipelines/textract-to-pgvector) · [Docling → pgvector](/pipelines/docling-to-pgvector)

Foundations: [The Optimal Chunker for Azure Document Intelligence](/optimal-chunker-azure-document-intelligence) · [The Optimal Chunks for pgvector](/optimal-chunks-pgvector) · [All pipeline recipes](/pipelines/) · [RAG chunking guide](/guides/rag-chunking/)