Source: http://www.poma-ai.com/docs/pipelines/azure-document-intelligence-to-redis

# The Missing Link Between Azure Document Intelligence and Optimal Retrieval in Redis

<ByAuthor />

**The short answer:** Azure Document Intelligence's `prebuilt-layout` model gives you excellent reading order and table extraction; Redis gives you an in-memory `VECTOR` field with `TAG`/`NUMERIC` filters as fast as the vector search itself. Wired together naively — flatten, fixed-split, `HSET` — the pipeline still produces mediocre RAG, because nothing in between rebuilds the document's hierarchy or preserves its page anchors. The missing link is POMA: `PrimeCut().ingest()` consumes the raw analyze result (auto-detected, either shape), emits hierarchy-preserving [chunksets](/learn/chunking/chunksets), and `vektoria` keeps the Redis index content-free while `assemble()` reconstructs prompt-ready context from a volume.

## What each end of the pipeline actually provides

**Azure Document Intelligence** returns one of two shapes: markdown mode's single reading-order string with inline HTML tables and `<!-- PageBreak -->` delimiters, or JSON mode's `paragraphs[]` (ordered by span offset, role-tagged) plus `tables[]` cell grids. Neither shape links a page-41 heading back to the page-3 title that opened its chapter, and neither decides what a retrieval unit should be. Details: [The Optimal Chunker for Azure Document Intelligence](/optimal-chunker-azure-document-intelligence).

**Redis** (via the RediSearch module) provides a `VECTOR` field for `HNSW`/`FLAT` ANN search plus `TAG`/`NUMERIC` fields that filter at the same speed as the vector search — all of it in memory. What it doesn't provide: any opinion about what belongs in that vector, or a ceiling on how large the index grows when a chunking strategy duplicates content. Details: [Redis Chunking Strategy for RAG](/optimal-chunks-redis).

## The naive wiring, and where it breaks

The common recipe — flatten Azure's `content` string (or `paragraphs[]`) with a fixed-size splitter, embed, `HSET` each fragment into Redis — breaks in a way specific to this pair:

- **Azure's page anchors and Redis's `NUMERIC` filter go to waste together.** Azure only exposes page boundaries through `<!-- PageBreak -->` comments (markdown mode) or span offsets against `paragraphs[]` (JSON mode). A naive flatten-then-split loses that boundary before any page number is attached to a chunk. Redis's `NUMERIC` field can filter by page cheaply — but only if something upstream preserved which page a chunk came from, and a character-count splitter has no reason to check.
- **Overlap is a direct RAM cost here, not an abstract one.** RediSearch's `HNSW`/`FLAT` structures live entirely in memory. Splitter overlap (typically 10–20%) duplicates every boundary span into that index, and it's worse on Azure's table-heavy documents: a table sliced mid-row by the splitter gets embedded — and stored — twice, once per half.
- **The retrieval side has its own trap.** Even chunked correctly, a `KNN` query missing the `AS vector_score` alias, `.return_fields(...)`, or `.dialect(2)` returns a result `assemble()` cannot extract metadata from. The search runs and looks successful; the metadata just never comes back, so `assemble()` returns an empty list.

## The pipeline, end to end

```bash
pip install azure-ai-documentintelligence redis
```

```python
import json
import os

import numpy as np
from azure.ai.documentintelligence import DocumentIntelligenceClient
from azure.core.credentials import AzureKeyCredential
from redis import Redis
from redis.commands.search.field import VectorField, TagField, NumericField
from redis.commands.search.query import Query
from redis.commands.search.index_definition import IndexDefinition, IndexType
from poma import PrimeCut
from poma.vektoria import assemble, records_from_archive, Volume
from poma.embeddings import get_embedder

# 1. Azure Document Intelligence — your existing call, unchanged.
di = DocumentIntelligenceClient(
    endpoint=os.environ["AZURE_DI_ENDPOINT"],
    credential=AzureKeyCredential(os.environ["AZURE_DI_KEY"]),
)
with open("document.pdf", "rb") as f:
    poller = di.begin_analyze_document(
        "prebuilt-layout",
        body=f,
        output_content_format="markdown",  # omit for classic JSON mode — POMA handles both
    )
analysis = poller.result()
with open("result.azure-di.json", "w") as f:
    json.dump(analysis.as_dict(), f)

# 2. The missing link — raw result JSON in, content-free archive out.
client = PrimeCut()  # reads POMA_API_KEY
client.ingest("result.azure-di.json", download_dir="archives", filename="doc.poma")

embedder = get_embedder("local:BAAI/bge-small-en-v1.5")
vol = Volume("s3://your-bucket/poma")
r = Redis(host="localhost", port=6379)
r.ft("poma_idx").create_index(
    fields=[
        VectorField("embedding", "HNSW", {"TYPE": "FLOAT32", "DIM": embedder.dims,
                                           "DISTANCE_METRIC": "COSINE"}),
        TagField("file_id"),
        NumericField("chunkset_index"),
        NumericField("page"),
    ],
    definition=IndexDefinition(prefix=["poma:"], index_type=IndexType.HASH),
)

# 3. Content-free ingest — Redis holds only routing metadata; content lives on the volume.
records = records_from_archive("archives/doc.poma")  # -> list[Record] (id, text, payload)
for rec in records:
    vol.write_doc(rec.payload["file_id"], {"file_id": rec.payload["file_id"],
                                            "chunks": rec.payload["chunks"], "text": rec.text})
    vector = np.asarray(embedder.embed([rec.text])[0], dtype=np.float32).tobytes()
    r.hset(f"poma:{rec.id}", mapping={
        "embedding": vector, "file_id": rec.payload["file_id"],
        "chunkset_index": rec.payload["chunkset_index"],
    })

# 4. KNN retrieval — the three-part contract Redis requires, then assemble.
qv = np.asarray(embedder.embed(["early termination conditions"])[0], dtype=np.float32).tobytes()
q = (Query("*=>[KNN 10 @embedding $vec AS vector_score]").sort_by("vector_score")
     .return_fields("file_id", "chunkset_index", "vector_score").dialect(2))
res = r.ft("poma_idx").search(q, query_params={"vec": qv})
context = assemble(res, volume=vol)  # -> [{"file_id", "content"}, ...]
```

Because the analysis already ran, POMA charges only for the downstream structure and chunking value — the OCR front-end is skipped. The vector DB never stores document content: `vektoria`'s `Volume` holds it, and `assemble()` deduplicates ancestor lineage across retrieved chunksets into one prompt-ready cheatsheet — the discipline behind our reference legal-document benchmark answering the same query with **337 tokens** instead of **1,542** for a recursive-splitter baseline. [Methodology here](/document-ingestion-chunking-rag).

## Metadata mapping: POMA fields → Redis primitives

| POMA chunk field | Redis primitive | What it enables |
| --- | --- | --- |
| `to_embed` | `VECTOR` field (`HNSW`, `FLOAT32`, `COSINE`) | KNN similarity search |
| `file_id` | `TAG` field | exact per-document scoping |
| `page` (from Azure `PageBreak`/span offsets) | `NUMERIC` field | page-range filters, page-cited answers |
| `depth` | `NUMERIC` field | hierarchy-level filter/re-rank |
| `chunkset_index` | returned via `.return_fields(...)` | `assemble()` dedupe key |
| chunk content | not stored in Redis — lives on `Volume` | content-free index, smaller RAM footprint |

## Frequently asked questions

### How do I get Azure Document Intelligence results into Redis for RAG?

Run `begin_analyze_document` with `prebuilt-layout` as you already do, and save the raw analyze result JSON. Hand that unmodified file to `PrimeCut`, which auto-detects the Azure shape and rebuilds the cross-page heading hierarchy into chunks and chunksets. Embed each chunkset's `to_embed` text, write it to a RediSearch `VECTOR` field alongside `file_id`/`chunkset_index`, and retrieve through `vektoria`'s `assemble()`, which fetches content from a volume and returns prompt-ready cheatsheets.

### Why does my Redis KNN query return nothing after ingesting Azure Document Intelligence chunks?

This is the one documented Redis gotcha, and it applies regardless of source: the `KNN` clause must alias its score `AS vector_score`, the query needs `.return_fields("file_id", "chunkset_index", "vector_score")`, and `.dialect(2)` must be set. Skip any of the three and RediSearch returns a shape `assemble()` cannot read metadata from — the search succeeds, but `assemble()` gets nothing back.

### What Redis fields should Azure Document Intelligence chunks carry?

`file_id` as a `TAG` field for exact per-document scoping, `page` and `depth` as `NUMERIC` fields recovered from Azure's `PageBreak` comments (markdown mode) or paragraph span offsets (JSON mode), and `chunkset_index` returned via `.return_fields(...)` so `assemble()` can deduplicate ancestor lineage. The embedding itself lives in a `VECTOR` field sized to your embedder's dimensions with the `COSINE` distance metric.

### Do Azure Document Intelligence's tables survive being embedded into Redis?

Yes, if you chunk the raw result instead of a flattened string. Azure Document Intelligence returns tables as inline HTML (markdown mode) or cell grids with merged-cell spans (JSON mode), and POMA converts both into HTML that stays intact inside a chunk's content — no row gets split across two Redis hashes the way a fixed-size splitter would cut it.

### Should I preserve Azure Document Intelligence's page breaks when indexing into Redis?

Yes. Azure Document Intelligence's `PageBreak` comments (markdown mode) or paragraph span offsets (JSON mode) are the only reliable page anchors in the result, and losing them at flattening means the `page` `NUMERIC` field in your Redis index has nothing to filter on. POMA preserves per-page anchoring through chunking, so every chunkset in Redis can still answer which page it came from.

## Related recipes

Same parser, different store: [Azure Document Intelligence → Qdrant](/pipelines/azure-document-intelligence-to-qdrant) · [Azure Document Intelligence → Pinecone](/pipelines/azure-document-intelligence-to-pinecone) · [Azure Document Intelligence → MongoDB Atlas](/pipelines/azure-document-intelligence-to-mongodb-atlas) · [Azure Document Intelligence → LanceDB](/pipelines/azure-document-intelligence-to-lancedb)

Foundations: [The Optimal Chunker for Azure Document Intelligence](/optimal-chunker-azure-document-intelligence) · [Redis Chunking Strategy for RAG](/optimal-chunks-redis) · [All pipeline recipes](/pipelines/) · [RAG chunking guide](/guides/rag-chunking/)