Skip to content

Azure Document Intelligence to Redis: Chunking for RAG ​

The short answer: Azure Document Intelligence's prebuilt-layout model gives you excellent reading order and table extraction; Redis gives you an in-memory VECTOR field with TAG/NUMERIC filters as fast as the vector search itself. Wired together naively — flatten, fixed-split, HSET — the pipeline still produces mediocre RAG, because nothing in between rebuilds the document's hierarchy or preserves its page anchors. The missing link is POMA: PrimeCut().ingest() consumes the raw analyze result (auto-detected, either shape), emits hierarchy-preserving chunksets, and vektoria keeps the Redis index content-free while assemble() reconstructs prompt-ready context from a volume.

What each end of the pipeline actually provides ​

Azure Document Intelligence returns one of two shapes: markdown mode's single reading-order string with inline HTML tables and <!-- PageBreak --> delimiters, or JSON mode's paragraphs[] (ordered by span offset, role-tagged) plus tables[] cell grids. Neither shape links a page-41 heading back to the page-3 title that opened its chapter, and neither decides what a retrieval unit should be. Details: The Optimal Chunker for Azure Document Intelligence.

Redis (via the RediSearch module) provides a VECTOR field for HNSW/FLAT ANN search plus TAG/NUMERIC fields that filter at the same speed as the vector search — all of it in memory. What it doesn't provide: any opinion about what belongs in that vector, or a ceiling on how large the index grows when a chunking strategy duplicates content. Details: Redis Chunking Strategy for RAG.

The naive wiring, and where it breaks ​

The common recipe — flatten Azure's content string (or paragraphs[]) with a fixed-size splitter, embed, HSET each fragment into Redis — breaks in a way specific to this pair:

  • Azure's page anchors and Redis's NUMERIC filter go to waste together. Azure only exposes page boundaries through <!-- PageBreak --> comments (markdown mode) or span offsets against paragraphs[] (JSON mode). A naive flatten-then-split loses that boundary before any page number is attached to a chunk. Redis's NUMERIC field can filter by page cheaply — but only if something upstream preserved which page a chunk came from, and a character-count splitter has no reason to check.
  • Overlap is a direct RAM cost here, not an abstract one. RediSearch's HNSW/FLAT structures live entirely in memory. Splitter overlap (typically 10–20%) duplicates every boundary span into that index, and it's worse on Azure's table-heavy documents: a table sliced mid-row by the splitter gets embedded — and stored — twice, once per half.
  • The retrieval side has its own trap. Even chunked correctly, a KNN query missing the AS vector_score alias, .return_fields(...), or .dialect(2) returns a result assemble() cannot extract metadata from. The search runs and looks successful; the metadata just never comes back, so assemble() returns an empty list.

The pipeline, end to end ​

bash
pip install azure-ai-documentintelligence redis
python
import json
import os

import numpy as np
from azure.ai.documentintelligence import DocumentIntelligenceClient
from azure.core.credentials import AzureKeyCredential
from redis import Redis
from redis.commands.search.field import VectorField, TagField, NumericField
from redis.commands.search.query import Query
from redis.commands.search.index_definition import IndexDefinition, IndexType
from poma import PrimeCut
from poma.vektoria import assemble, records_from_archive, Volume
from poma.embeddings import get_embedder

# 1. Azure Document Intelligence — your existing call, unchanged.
di = DocumentIntelligenceClient(
    endpoint=os.environ["AZURE_DI_ENDPOINT"],
    credential=AzureKeyCredential(os.environ["AZURE_DI_KEY"]),
)
with open("document.pdf", "rb") as f:
    poller = di.begin_analyze_document(
        "prebuilt-layout",
        body=f,
        output_content_format="markdown",  # omit for classic JSON mode — POMA handles both
    )
analysis = poller.result()
with open("result.azure-di.json", "w") as f:
    json.dump(analysis.as_dict(), f)

# 2. The missing link — raw result JSON in, content-free archive out.
client = PrimeCut()  # reads POMA_API_KEY
client.ingest("result.azure-di.json", download_dir="archives", filename="doc.poma")

embedder = get_embedder("local:BAAI/bge-small-en-v1.5")
vol = Volume("s3://your-bucket/poma")
r = Redis(host="localhost", port=6379)
r.ft("poma_idx").create_index(
    fields=[
        VectorField("embedding", "HNSW", {"TYPE": "FLOAT32", "DIM": embedder.dims,
                                           "DISTANCE_METRIC": "COSINE"}),
        TagField("file_id"),
        NumericField("chunkset_index"),
        NumericField("page"),
    ],
    definition=IndexDefinition(prefix=["poma:"], index_type=IndexType.HASH),
)

# 3. Content-free ingest — Redis holds only routing metadata; content lives on the volume.
records = records_from_archive("archives/doc.poma")  # -> list[Record] (id, text, payload)
# assemble() reads the FULL document (chunks + chunksets) from the volume — write it once,
# not one {"chunks": <indices>, "text"} stub per record (which also overwrote itself each turn).
from poma.utils import unpack_poma_archive
_data = unpack_poma_archive(poma_archive_path="archives/doc.poma")
file_id = records[0].payload["file_id"]
vol.write_doc(file_id, {"file_id": file_id, "chunks": _data["chunks"], "chunksets": _data["chunksets"]})
for rec in records:
    vector = np.asarray(embedder.embed([rec.text])[0], dtype=np.float32).tobytes()
    r.hset(f"poma:{rec.id}", mapping={
        "embedding": vector, "file_id": rec.payload["file_id"],
        "chunkset_index": rec.payload["chunkset_index"],
    })

# 4. KNN retrieval — the three-part contract Redis requires, then assemble.
qv = np.asarray(embedder.embed(["early termination conditions"])[0], dtype=np.float32).tobytes()
q = (Query("*=>[KNN 10 @embedding $vec AS vector_score]").sort_by("vector_score")
     .return_fields("file_id", "chunkset_index", "vector_score").dialect(2))
res = r.ft("poma_idx").search(q, query_params={"vec": qv})
context = assemble(res, volume=vol)  # -> [{"file_id", "content"}, ...]

Because the analysis already ran, POMA charges only for the downstream structure and chunking value — the OCR front-end is skipped. The vector DB never stores document content: vektoria's Volume holds it, and assemble() deduplicates ancestor lineage across retrieved chunksets into one prompt-ready cheatsheet — the discipline behind our reference legal-document benchmark answering the same query with 337 tokens instead of 1,542 for a recursive-splitter baseline. Methodology here.

Metadata mapping: POMA fields → Redis primitives ​

POMA chunk fieldRedis primitiveWhat it enables
to_embedVECTOR field (HNSW, FLOAT32, COSINE)KNN similarity search
file_idTAG fieldexact per-document scoping
page (from Azure PageBreak/span offsets)NUMERIC fieldpage-range filters, page-cited answers
depthNUMERIC fieldhierarchy-level filter/re-rank
chunkset_indexreturned via .return_fields(...)assemble() dedupe key
chunk contentnot stored in Redis — lives on Volumecontent-free index, smaller RAM footprint

Frequently asked questions ​

How do I get Azure Document Intelligence results into Redis for RAG? ​

Run begin_analyze_document with prebuilt-layout as you already do, and save the raw analyze result JSON. Hand that unmodified file to PrimeCut, which auto-detects the Azure shape and rebuilds the cross-page heading hierarchy into chunks and chunksets. Embed each chunkset's to_embed text, write it to a RediSearch VECTOR field alongside file_id/chunkset_index, and retrieve through vektoria's assemble(), which fetches content from a volume and returns prompt-ready cheatsheets.

Why does my Redis KNN query return nothing after ingesting Azure Document Intelligence chunks? ​

This is the one documented Redis gotcha, and it applies regardless of source: the KNN clause must alias its score AS vector_score, the query needs .return_fields("file_id", "chunkset_index", "vector_score"), and .dialect(2) must be set. Skip any of the three and RediSearch returns a shape assemble() cannot read metadata from — the search succeeds, but assemble() gets nothing back.

What Redis fields should Azure Document Intelligence chunks carry? ​

file_id as a TAG field for exact per-document scoping, page and depth as NUMERIC fields recovered from Azure's PageBreak comments (markdown mode) or paragraph span offsets (JSON mode), and chunkset_index returned via .return_fields(...) so assemble() can deduplicate ancestor lineage. The embedding itself lives in a VECTOR field sized to your embedder's dimensions with the COSINE distance metric.

Do Azure Document Intelligence's tables survive being embedded into Redis? ​

Yes, if you chunk the raw result instead of a flattened string. Azure Document Intelligence returns tables as inline HTML (markdown mode) or cell grids with merged-cell spans (JSON mode), and POMA converts both into HTML that stays intact inside a chunk's content — no row gets split across two Redis hashes the way a fixed-size splitter would cut it.

Should I preserve Azure Document Intelligence's page breaks when indexing into Redis? ​

Yes. Azure Document Intelligence's PageBreak comments (markdown mode) or paragraph span offsets (JSON mode) are the only reliable page anchors in the result, and losing them at flattening means the page NUMERIC field in your Redis index has nothing to filter on. POMA preserves per-page anchoring through chunking: page lands on every chunk record in the .poma archive, so read it back with unpack_poma_archive and write the chunkset's leaf-chunk page into the NUMERIC field.

Same parser, different store: Azure Document Intelligence → Qdrant · Azure Document Intelligence → Pinecone · Azure Document Intelligence → MongoDB Atlas · Azure Document Intelligence → LanceDB

Foundations: The Optimal Chunker for Azure Document Intelligence · Redis Chunking Strategy for RAG · All pipeline recipes · RAG chunking guide