AWS Textract to Redis: Chunking for RAG
The short answer: AWS Textract gives you an excellent, forms-and-tables-aware layout graph; Redis gives you a fast, in-memory vector index with exact-match filters as first-class citizens. Wired together naively — flatten Blocks[], split, embed — they still produce mediocre RAG, because nothing in between rebuilds Textract's own reading order or shapes the result for Redis's query contract. The missing link is POMA: PrimeCut().ingest() consumes the raw AnalyzeDocument response, follows the LAYOUT spine, and emits hierarchy-preserving chunksets; vektoria's records_from_archive and assemble() keep the RediSearch index content-free and turn a properly-formed KNN query into prompt-ready context.
What each end of the pipeline actually provides
AWS Textract (AnalyzeDocument with the LAYOUT feature) returns a flat Blocks[] graph: LAYOUT_TITLE/LAYOUT_SECTION_HEADER blocks in multi-column-aware reading order, structured TABLE blocks with no Id link to their LAYOUT_TABLE region, and LAYOUT_FIGURE regions with no image bytes at all. What it doesn't provide: cross-page hierarchy, a table-to-layout link, or any retrieval unit whatsoever. Details: The Optimal Chunker for AWS Textract.
Redis (RediSearch) provides a VECTOR field for the embedding plus TAG/NUMERIC fields for exact-match and range filters, all served from memory at the same speed as the vector search itself. What it doesn't provide: any opinion about what a hash should hold, or automatic metadata return — a KNN query has to explicitly ask for the fields it wants back. Details: Redis Chunking Strategy for RAG.
The naive wiring, and where it breaks
The common recipe — flatten Blocks[] to text, split into fixed windows, HSET one hash per chunk — breaks in a pair-specific way:
- Without the
LAYOUTfeature honored during flattening, Textract's two-column reports interleave into scrambled prose — position-sortedLINEblocks, not reading order. Whatever text you split next, Redis will happily rank the scramble. - Skip
TAG/NUMERICfields forfile_idandchunkset_indexand every filter degrades to a full sequential scan across a memory-resident index — the one place RediSearch is supposed to be fast. - Even correctly-chunked text fails silently at the query layer. A
KNNclause missing theAS vector_scorealias,.return_fields(...), or.dialect(2)returns a non-empty search result that POMA'sassemble()cannot extract metadata from — it comes back an empty list, indistinguishable from "no relevant chunks" unless you already know to check the query syntax.
The pipeline, end to end
pip install boto3 redisimport json
import boto3
import numpy as np
from redis import Redis
from redis.commands.search.field import VectorField, TagField, NumericField
from redis.commands.search.query import Query
from redis.commands.search.index_definition import IndexDefinition, IndexType
from poma import PrimeCut
from poma.vektoria import assemble, records_from_archive, Volume
from poma.embeddings import get_embedder
# 1. Your existing Textract call — LAYOUT is required; add TABLES for cell grids.
textract = boto3.client("textract")
with open("contract.png", "rb") as f:
response = textract.analyze_document(
Document={"Bytes": f.read()},
FeatureTypes=["LAYOUT", "TABLES"],
)
with open("contract.textract.json", "w") as f:
json.dump(response, f)
# 2. The missing link — raw Textract result in, a portable .poma archive out.
client = PrimeCut() # reads POMA_API_KEY
client.ingest("contract.textract.json", download_dir="archives", filename="contract.poma")
embedder = get_embedder("local:BAAI/bge-small-en-v1.5")
vol = Volume("s3://your-bucket/poma")
r = Redis(host="localhost", port=6379)
r.ft("poma_idx").create_index(
fields=[
VectorField("embedding", "HNSW", {"TYPE": "FLOAT32", "DIM": embedder.dims,
"DISTANCE_METRIC": "COSINE"}),
TagField("file_id"),
NumericField("chunkset_index"),
],
definition=IndexDefinition(prefix=["poma:"], index_type=IndexType.HASH),
)
# 3. Content-free ingest — Redis holds only routing metadata; content lives on the volume.
records = records_from_archive("archives/contract.poma") # -> list[Record] (id, text, payload)
# assemble() reads the FULL document (chunks + chunksets) from the volume — write it once,
# not one {"chunks": <indices>, "text"} stub per record (which also overwrote itself each turn).
from poma.utils import unpack_poma_archive
_data = unpack_poma_archive(poma_archive_path="archives/contract.poma")
file_id = records[0].payload["file_id"]
vol.write_doc(file_id, {"file_id": file_id, "chunks": _data["chunks"], "chunksets": _data["chunksets"]})
for rec in records:
vector = np.asarray(embedder.embed([rec.text])[0], dtype=np.float32).tobytes()
r.hset(f"poma:{rec.id}", mapping={
"embedding": vector, "file_id": rec.payload["file_id"],
"chunkset_index": rec.payload["chunkset_index"],
})
# 4. KNN retrieval — Redis's three-part contract, then assemble.
qv = np.asarray(embedder.embed(["early termination conditions"])[0], dtype=np.float32).tobytes()
q = (Query("*=>[KNN 10 @embedding $vec AS vector_score]").sort_by("vector_score")
.return_fields("file_id", "chunkset_index", "vector_score").dialect(2))
res = r.ft("poma_idx").search(q, query_params={"vec": qv})
context = assemble(res, volume=vol) # -> [{"file_id", "content"}, ...]Textract's own quirks are handled before a single record reaches Redis: TABLE blocks are spliced into their LAYOUT_TABLE region by bounding-box geometry and rendered as HTML with rowspan/colspan, SELECTION_ELEMENT checkboxes survive as ☒/☐, and offloaded figures are counted, never silently dropped. A payload missing LAYOUT blocks 422s up front rather than shipping scrambled reading order.
Metadata mapping: POMA fields → Redis primitives
| POMA chunk field | Redis primitive | What it enables |
|---|---|---|
to_embed | VECTOR field (HNSW, FLOAT32) | KNN vector search |
file_id | TAG field | exact-match per-document scoping, filter-then-search hybrid |
chunkset_index | NUMERIC field (part of hash key) | assemble() dedup id, deterministic routing |
page (Textract layout-derived) | NUMERIC field | page-range filters, page-cited answers |
depth | NUMERIC field | hierarchy-aware filter/re-rank |
| chunkset content | volume (Volume.write_doc), not in Redis | content-free index, cheatsheet assembly |
Frequently asked questions
How do I get AWS Textract results into Redis for RAG?
Run AnalyzeDocument with the LAYOUT feature as you already do, save the raw JSON, and hand it to PrimeCut, which auto-detects the Textract shape and rebuilds the LAYOUT reading order into chunks and chunksets. records_from_archive then gives you deterministic ids and embeddable text for a RediSearch VECTOR field with TAG/NUMERIC metadata.
Why does my Redis KNN query return nothing after indexing Textract chunks?
Redis's one documented gotcha: the KNN clause needs AS vector_score, .return_fields(...), and .dialect(2). Miss any of the three and assemble() gets a result shape it can't read metadata from — a real match comes back as an empty context list.
Do Textract's LAYOUT_FIGURE regions and checkboxes survive the trip to Redis?
Textract never returns cropped image bytes, so figures are counted as offloaded content rather than embedded — visible in your ingest report, not silent. SELECTION_ELEMENT checkboxes survive as ☒/☐ marks in the chunk text, so form state reaches the retrieved cheatsheet.
What Redis fields should AWS Textract chunks carry?
A VECTOR field for the embedding, a TAG field for file_id, and NUMERIC fields for page and chunkset_index. That gives exact-match and range filtering at KNN speed, without a post-filter scan.
Is Textract's forms and table extraction wasted if I flatten Blocks before indexing in Redis?
Largely, yes — the LAYOUT sequence, table splicing, and checkbox state are exactly what a naive flatten discards before a vector ever reaches Redis. A structure-aware chunker keeps that extraction value intact into the VECTOR field.
Related recipes
Same parser, different store: Textract → MongoDB Atlas · Textract → LanceDB · Textract → Qdrant
Foundations: The Optimal Chunker for AWS Textract · Redis Chunking Strategy for RAG · All pipeline recipes · RAG architecture guide