Skip to content

AWS Textract to Redis: Chunking for RAG ​

The short answer: AWS Textract gives you an excellent, forms-and-tables-aware layout graph; Redis gives you a fast, in-memory vector index with exact-match filters as first-class citizens. Wired together naively — flatten Blocks[], split, embed — they still produce mediocre RAG, because nothing in between rebuilds Textract's own reading order or shapes the result for Redis's query contract. The missing link is POMA: PrimeCut().ingest() consumes the raw AnalyzeDocument response, follows the LAYOUT spine, and emits hierarchy-preserving chunksets; vektoria's records_from_archive and assemble() keep the RediSearch index content-free and turn a properly-formed KNN query into prompt-ready context.

What each end of the pipeline actually provides ​

AWS Textract (AnalyzeDocument with the LAYOUT feature) returns a flat Blocks[] graph: LAYOUT_TITLE/LAYOUT_SECTION_HEADER blocks in multi-column-aware reading order, structured TABLE blocks with no Id link to their LAYOUT_TABLE region, and LAYOUT_FIGURE regions with no image bytes at all. What it doesn't provide: cross-page hierarchy, a table-to-layout link, or any retrieval unit whatsoever. Details: The Optimal Chunker for AWS Textract.

Redis (RediSearch) provides a VECTOR field for the embedding plus TAG/NUMERIC fields for exact-match and range filters, all served from memory at the same speed as the vector search itself. What it doesn't provide: any opinion about what a hash should hold, or automatic metadata return — a KNN query has to explicitly ask for the fields it wants back. Details: Redis Chunking Strategy for RAG.

The naive wiring, and where it breaks ​

The common recipe — flatten Blocks[] to text, split into fixed windows, HSET one hash per chunk — breaks in a pair-specific way:

  • Without the LAYOUT feature honored during flattening, Textract's two-column reports interleave into scrambled prose — position-sorted LINE blocks, not reading order. Whatever text you split next, Redis will happily rank the scramble.
  • Skip TAG/NUMERIC fields for file_id and chunkset_index and every filter degrades to a full sequential scan across a memory-resident index — the one place RediSearch is supposed to be fast.
  • Even correctly-chunked text fails silently at the query layer. A KNN clause missing the AS vector_score alias, .return_fields(...), or .dialect(2) returns a non-empty search result that POMA's assemble() cannot extract metadata from — it comes back an empty list, indistinguishable from "no relevant chunks" unless you already know to check the query syntax.

The pipeline, end to end ​

bash
pip install boto3 redis
python
import json

import boto3
import numpy as np
from redis import Redis
from redis.commands.search.field import VectorField, TagField, NumericField
from redis.commands.search.query import Query
from redis.commands.search.index_definition import IndexDefinition, IndexType
from poma import PrimeCut
from poma.vektoria import assemble, records_from_archive, Volume
from poma.embeddings import get_embedder

# 1. Your existing Textract call — LAYOUT is required; add TABLES for cell grids.
textract = boto3.client("textract")
with open("contract.png", "rb") as f:
    response = textract.analyze_document(
        Document={"Bytes": f.read()},
        FeatureTypes=["LAYOUT", "TABLES"],
    )
with open("contract.textract.json", "w") as f:
    json.dump(response, f)

# 2. The missing link — raw Textract result in, a portable .poma archive out.
client = PrimeCut()  # reads POMA_API_KEY
client.ingest("contract.textract.json", download_dir="archives", filename="contract.poma")

embedder = get_embedder("local:BAAI/bge-small-en-v1.5")
vol = Volume("s3://your-bucket/poma")
r = Redis(host="localhost", port=6379)

r.ft("poma_idx").create_index(
    fields=[
        VectorField("embedding", "HNSW", {"TYPE": "FLOAT32", "DIM": embedder.dims,
                                           "DISTANCE_METRIC": "COSINE"}),
        TagField("file_id"),
        NumericField("chunkset_index"),
    ],
    definition=IndexDefinition(prefix=["poma:"], index_type=IndexType.HASH),
)

# 3. Content-free ingest — Redis holds only routing metadata; content lives on the volume.
records = records_from_archive("archives/contract.poma")  # -> list[Record] (id, text, payload)
# assemble() reads the FULL document (chunks + chunksets) from the volume — write it once,
# not one {"chunks": <indices>, "text"} stub per record (which also overwrote itself each turn).
from poma.utils import unpack_poma_archive
_data = unpack_poma_archive(poma_archive_path="archives/contract.poma")
file_id = records[0].payload["file_id"]
vol.write_doc(file_id, {"file_id": file_id, "chunks": _data["chunks"], "chunksets": _data["chunksets"]})
for rec in records:
    vector = np.asarray(embedder.embed([rec.text])[0], dtype=np.float32).tobytes()
    r.hset(f"poma:{rec.id}", mapping={
        "embedding": vector, "file_id": rec.payload["file_id"],
        "chunkset_index": rec.payload["chunkset_index"],
    })

# 4. KNN retrieval — Redis's three-part contract, then assemble.
qv = np.asarray(embedder.embed(["early termination conditions"])[0], dtype=np.float32).tobytes()
q = (Query("*=>[KNN 10 @embedding $vec AS vector_score]").sort_by("vector_score")
     .return_fields("file_id", "chunkset_index", "vector_score").dialect(2))
res = r.ft("poma_idx").search(q, query_params={"vec": qv})
context = assemble(res, volume=vol)  # -> [{"file_id", "content"}, ...]

Textract's own quirks are handled before a single record reaches Redis: TABLE blocks are spliced into their LAYOUT_TABLE region by bounding-box geometry and rendered as HTML with rowspan/colspan, SELECTION_ELEMENT checkboxes survive as ☒/☐, and offloaded figures are counted, never silently dropped. A payload missing LAYOUT blocks 422s up front rather than shipping scrambled reading order.

Metadata mapping: POMA fields → Redis primitives ​

POMA chunk fieldRedis primitiveWhat it enables
to_embedVECTOR field (HNSW, FLOAT32)KNN vector search
file_idTAG fieldexact-match per-document scoping, filter-then-search hybrid
chunkset_indexNUMERIC field (part of hash key)assemble() dedup id, deterministic routing
page (Textract layout-derived)NUMERIC fieldpage-range filters, page-cited answers
depthNUMERIC fieldhierarchy-aware filter/re-rank
chunkset contentvolume (Volume.write_doc), not in Rediscontent-free index, cheatsheet assembly

Frequently asked questions ​

How do I get AWS Textract results into Redis for RAG? ​

Run AnalyzeDocument with the LAYOUT feature as you already do, save the raw JSON, and hand it to PrimeCut, which auto-detects the Textract shape and rebuilds the LAYOUT reading order into chunks and chunksets. records_from_archive then gives you deterministic ids and embeddable text for a RediSearch VECTOR field with TAG/NUMERIC metadata.

Why does my Redis KNN query return nothing after indexing Textract chunks? ​

Redis's one documented gotcha: the KNN clause needs AS vector_score, .return_fields(...), and .dialect(2). Miss any of the three and assemble() gets a result shape it can't read metadata from — a real match comes back as an empty context list.

Do Textract's LAYOUT_FIGURE regions and checkboxes survive the trip to Redis? ​

Textract never returns cropped image bytes, so figures are counted as offloaded content rather than embedded — visible in your ingest report, not silent. SELECTION_ELEMENT checkboxes survive as ☒/☐ marks in the chunk text, so form state reaches the retrieved cheatsheet.

What Redis fields should AWS Textract chunks carry? ​

A VECTOR field for the embedding, a TAG field for file_id, and NUMERIC fields for page and chunkset_index. That gives exact-match and range filtering at KNN speed, without a post-filter scan.

Is Textract's forms and table extraction wasted if I flatten Blocks before indexing in Redis? ​

Largely, yes — the LAYOUT sequence, table splicing, and checkbox state are exactly what a naive flatten discards before a vector ever reaches Redis. A structure-aware chunker keeps that extraction value intact into the VECTOR field.

Same parser, different store: Textract → MongoDB Atlas · Textract → LanceDB · Textract → Qdrant

Foundations: The Optimal Chunker for AWS Textract · Redis Chunking Strategy for RAG · All pipeline recipes · RAG architecture guide