Azure Document Intelligence to Redis: Chunking for RAG
The short answer: Azure Document Intelligence's prebuilt-layout model gives you excellent reading order and table extraction; Redis gives you an in-memory VECTOR field with TAG/NUMERIC filters as fast as the vector search itself. Wired together naively — flatten, fixed-split, HSET — the pipeline still produces mediocre RAG, because nothing in between rebuilds the document's hierarchy or preserves its page anchors. The missing link is POMA: PrimeCut().ingest() consumes the raw analyze result (auto-detected, either shape), emits hierarchy-preserving chunksets, and vektoria keeps the Redis index content-free while assemble() reconstructs prompt-ready context from a volume.
What each end of the pipeline actually provides
Azure Document Intelligence returns one of two shapes: markdown mode's single reading-order string with inline HTML tables and <!-- PageBreak --> delimiters, or JSON mode's paragraphs[] (ordered by span offset, role-tagged) plus tables[] cell grids. Neither shape links a page-41 heading back to the page-3 title that opened its chapter, and neither decides what a retrieval unit should be. Details: The Optimal Chunker for Azure Document Intelligence.
Redis (via the RediSearch module) provides a VECTOR field for HNSW/FLAT ANN search plus TAG/NUMERIC fields that filter at the same speed as the vector search — all of it in memory. What it doesn't provide: any opinion about what belongs in that vector, or a ceiling on how large the index grows when a chunking strategy duplicates content. Details: Redis Chunking Strategy for RAG.
The naive wiring, and where it breaks
The common recipe — flatten Azure's content string (or paragraphs[]) with a fixed-size splitter, embed, HSET each fragment into Redis — breaks in a way specific to this pair:
- Azure's page anchors and Redis's
NUMERICfilter go to waste together. Azure only exposes page boundaries through<!-- PageBreak -->comments (markdown mode) or span offsets againstparagraphs[](JSON mode). A naive flatten-then-split loses that boundary before any page number is attached to a chunk. Redis'sNUMERICfield can filter by page cheaply — but only if something upstream preserved which page a chunk came from, and a character-count splitter has no reason to check. - Overlap is a direct RAM cost here, not an abstract one. RediSearch's
HNSW/FLATstructures live entirely in memory. Splitter overlap (typically 10–20%) duplicates every boundary span into that index, and it's worse on Azure's table-heavy documents: a table sliced mid-row by the splitter gets embedded — and stored — twice, once per half. - The retrieval side has its own trap. Even chunked correctly, a
KNNquery missing theAS vector_scorealias,.return_fields(...), or.dialect(2)returns a resultassemble()cannot extract metadata from. The search runs and looks successful; the metadata just never comes back, soassemble()returns an empty list.
The pipeline, end to end
pip install azure-ai-documentintelligence redisimport json
import os
import numpy as np
from azure.ai.documentintelligence import DocumentIntelligenceClient
from azure.core.credentials import AzureKeyCredential
from redis import Redis
from redis.commands.search.field import VectorField, TagField, NumericField
from redis.commands.search.query import Query
from redis.commands.search.index_definition import IndexDefinition, IndexType
from poma import PrimeCut
from poma.vektoria import assemble, records_from_archive, Volume
from poma.embeddings import get_embedder
# 1. Azure Document Intelligence — your existing call, unchanged.
di = DocumentIntelligenceClient(
endpoint=os.environ["AZURE_DI_ENDPOINT"],
credential=AzureKeyCredential(os.environ["AZURE_DI_KEY"]),
)
with open("document.pdf", "rb") as f:
poller = di.begin_analyze_document(
"prebuilt-layout",
body=f,
output_content_format="markdown", # omit for classic JSON mode — POMA handles both
)
analysis = poller.result()
with open("result.azure-di.json", "w") as f:
json.dump(analysis.as_dict(), f)
# 2. The missing link — raw result JSON in, content-free archive out.
client = PrimeCut() # reads POMA_API_KEY
client.ingest("result.azure-di.json", download_dir="archives", filename="doc.poma")
embedder = get_embedder("local:BAAI/bge-small-en-v1.5")
vol = Volume("s3://your-bucket/poma")
r = Redis(host="localhost", port=6379)
r.ft("poma_idx").create_index(
fields=[
VectorField("embedding", "HNSW", {"TYPE": "FLOAT32", "DIM": embedder.dims,
"DISTANCE_METRIC": "COSINE"}),
TagField("file_id"),
NumericField("chunkset_index"),
NumericField("page"),
],
definition=IndexDefinition(prefix=["poma:"], index_type=IndexType.HASH),
)
# 3. Content-free ingest — Redis holds only routing metadata; content lives on the volume.
records = records_from_archive("archives/doc.poma") # -> list[Record] (id, text, payload)
# assemble() reads the FULL document (chunks + chunksets) from the volume — write it once,
# not one {"chunks": <indices>, "text"} stub per record (which also overwrote itself each turn).
from poma.utils import unpack_poma_archive
_data = unpack_poma_archive(poma_archive_path="archives/doc.poma")
file_id = records[0].payload["file_id"]
vol.write_doc(file_id, {"file_id": file_id, "chunks": _data["chunks"], "chunksets": _data["chunksets"]})
for rec in records:
vector = np.asarray(embedder.embed([rec.text])[0], dtype=np.float32).tobytes()
r.hset(f"poma:{rec.id}", mapping={
"embedding": vector, "file_id": rec.payload["file_id"],
"chunkset_index": rec.payload["chunkset_index"],
})
# 4. KNN retrieval — the three-part contract Redis requires, then assemble.
qv = np.asarray(embedder.embed(["early termination conditions"])[0], dtype=np.float32).tobytes()
q = (Query("*=>[KNN 10 @embedding $vec AS vector_score]").sort_by("vector_score")
.return_fields("file_id", "chunkset_index", "vector_score").dialect(2))
res = r.ft("poma_idx").search(q, query_params={"vec": qv})
context = assemble(res, volume=vol) # -> [{"file_id", "content"}, ...]Because the analysis already ran, POMA charges only for the downstream structure and chunking value — the OCR front-end is skipped. The vector DB never stores document content: vektoria's Volume holds it, and assemble() deduplicates ancestor lineage across retrieved chunksets into one prompt-ready cheatsheet — the discipline behind our reference legal-document benchmark answering the same query with 337 tokens instead of 1,542 for a recursive-splitter baseline. Methodology here.
Metadata mapping: POMA fields → Redis primitives
| POMA chunk field | Redis primitive | What it enables |
|---|---|---|
to_embed | VECTOR field (HNSW, FLOAT32, COSINE) | KNN similarity search |
file_id | TAG field | exact per-document scoping |
page (from Azure PageBreak/span offsets) | NUMERIC field | page-range filters, page-cited answers |
depth | NUMERIC field | hierarchy-level filter/re-rank |
chunkset_index | returned via .return_fields(...) | assemble() dedupe key |
| chunk content | not stored in Redis — lives on Volume | content-free index, smaller RAM footprint |
Frequently asked questions
How do I get Azure Document Intelligence results into Redis for RAG?
Run begin_analyze_document with prebuilt-layout as you already do, and save the raw analyze result JSON. Hand that unmodified file to PrimeCut, which auto-detects the Azure shape and rebuilds the cross-page heading hierarchy into chunks and chunksets. Embed each chunkset's to_embed text, write it to a RediSearch VECTOR field alongside file_id/chunkset_index, and retrieve through vektoria's assemble(), which fetches content from a volume and returns prompt-ready cheatsheets.
Why does my Redis KNN query return nothing after ingesting Azure Document Intelligence chunks?
This is the one documented Redis gotcha, and it applies regardless of source: the KNN clause must alias its score AS vector_score, the query needs .return_fields("file_id", "chunkset_index", "vector_score"), and .dialect(2) must be set. Skip any of the three and RediSearch returns a shape assemble() cannot read metadata from — the search succeeds, but assemble() gets nothing back.
What Redis fields should Azure Document Intelligence chunks carry?
file_id as a TAG field for exact per-document scoping, page and depth as NUMERIC fields recovered from Azure's PageBreak comments (markdown mode) or paragraph span offsets (JSON mode), and chunkset_index returned via .return_fields(...) so assemble() can deduplicate ancestor lineage. The embedding itself lives in a VECTOR field sized to your embedder's dimensions with the COSINE distance metric.
Do Azure Document Intelligence's tables survive being embedded into Redis?
Yes, if you chunk the raw result instead of a flattened string. Azure Document Intelligence returns tables as inline HTML (markdown mode) or cell grids with merged-cell spans (JSON mode), and POMA converts both into HTML that stays intact inside a chunk's content — no row gets split across two Redis hashes the way a fixed-size splitter would cut it.
Should I preserve Azure Document Intelligence's page breaks when indexing into Redis?
Yes. Azure Document Intelligence's PageBreak comments (markdown mode) or paragraph span offsets (JSON mode) are the only reliable page anchors in the result, and losing them at flattening means the page NUMERIC field in your Redis index has nothing to filter on. POMA preserves per-page anchoring through chunking: page lands on every chunk record in the .poma archive, so read it back with unpack_poma_archive and write the chunkset's leaf-chunk page into the NUMERIC field.
Related recipes
Same parser, different store: Azure Document Intelligence → Qdrant · Azure Document Intelligence → Pinecone · Azure Document Intelligence → MongoDB Atlas · Azure Document Intelligence → LanceDB
Foundations: The Optimal Chunker for Azure Document Intelligence · Redis Chunking Strategy for RAG · All pipeline recipes · RAG chunking guide