Source: http://www.poma-ai.com/docs/pipelines/textract-to-redis

# The Missing Link Between AWS Textract and Optimal Retrieval in Redis

<ByAuthor />

**The short answer:** AWS Textract gives you an excellent, forms-and-tables-aware layout graph; Redis gives you a fast, in-memory vector index with exact-match filters as first-class citizens. Wired together naively — flatten `Blocks[]`, split, embed — they still produce mediocre RAG, because nothing in between rebuilds Textract's own reading order or shapes the result for Redis's query contract. The missing link is POMA: `PrimeCut().ingest()` consumes the raw `AnalyzeDocument` response, follows the `LAYOUT` spine, and emits hierarchy-preserving [chunksets](/learn/chunking/chunksets); `vektoria`'s `records_from_archive` and `assemble()` keep the RediSearch index content-free and turn a properly-formed `KNN` query into prompt-ready context.

## What each end of the pipeline actually provides

**AWS Textract** (`AnalyzeDocument` with the `LAYOUT` feature) returns a flat `Blocks[]` graph: `LAYOUT_TITLE`/`LAYOUT_SECTION_HEADER` blocks in multi-column-aware reading order, structured `TABLE` blocks with no `Id` link to their `LAYOUT_TABLE` region, and `LAYOUT_FIGURE` regions with no image bytes at all. What it doesn't provide: cross-page hierarchy, a table-to-layout link, or any retrieval unit whatsoever. Details: [The Optimal Chunker for AWS Textract](/optimal-chunker-textract).

**Redis** (RediSearch) provides a `VECTOR` field for the embedding plus `TAG`/`NUMERIC` fields for exact-match and range filters, all served from memory at the same speed as the vector search itself. What it doesn't provide: any opinion about what a hash should hold, or automatic metadata return — a `KNN` query has to explicitly ask for the fields it wants back. Details: [Redis Chunking Strategy for RAG](/optimal-chunks-redis).

## The naive wiring, and where it breaks

The common recipe — flatten `Blocks[]` to text, split into fixed windows, `HSET` one hash per chunk — breaks in a pair-specific way:

- **Without the `LAYOUT` feature honored during flattening**, Textract's two-column reports interleave into scrambled prose — position-sorted `LINE` blocks, not reading order. Whatever text you split next, Redis will happily rank the scramble.
- **Skip `TAG`/`NUMERIC` fields for `file_id` and `chunkset_index`** and every filter degrades to a full sequential scan across a memory-resident index — the one place RediSearch is supposed to be fast.
- **Even correctly-chunked text fails silently at the query layer.** A `KNN` clause missing the `AS vector_score` alias, `.return_fields(...)`, or `.dialect(2)` returns a non-empty search result that POMA's `assemble()` cannot extract metadata from — it comes back an empty list, indistinguishable from "no relevant chunks" unless you already know to check the query syntax.

## The pipeline, end to end

```bash
pip install boto3 redis
```

```python
import json

import boto3
import numpy as np
from redis import Redis
from redis.commands.search.field import VectorField, TagField, NumericField
from redis.commands.search.query import Query
from redis.commands.search.index_definition import IndexDefinition, IndexType
from poma import PrimeCut
from poma.vektoria import assemble, records_from_archive, Volume
from poma.embeddings import get_embedder

# 1. Your existing Textract call — LAYOUT is required; add TABLES for cell grids.
textract = boto3.client("textract")
with open("contract.png", "rb") as f:
    response = textract.analyze_document(
        Document={"Bytes": f.read()},
        FeatureTypes=["LAYOUT", "TABLES"],
    )
with open("contract.textract.json", "w") as f:
    json.dump(response, f)

# 2. The missing link — raw Textract result in, a portable .poma archive out.
client = PrimeCut()  # reads POMA_API_KEY
client.ingest("contract.textract.json", download_dir="archives", filename="contract.poma")

embedder = get_embedder("local:BAAI/bge-small-en-v1.5")
vol = Volume("s3://your-bucket/poma")
r = Redis(host="localhost", port=6379)

r.ft("poma_idx").create_index(
    fields=[
        VectorField("embedding", "HNSW", {"TYPE": "FLOAT32", "DIM": embedder.dims,
                                           "DISTANCE_METRIC": "COSINE"}),
        TagField("file_id"),
        NumericField("chunkset_index"),
    ],
    definition=IndexDefinition(prefix=["poma:"], index_type=IndexType.HASH),
)

# 3. Content-free ingest — Redis holds only routing metadata; content lives on the volume.
records = records_from_archive("archives/contract.poma")  # -> list[Record] (id, text, payload)
for rec in records:
    vol.write_doc(rec.payload["file_id"], {"file_id": rec.payload["file_id"],
                                            "chunks": rec.payload["chunks"], "text": rec.text})
    vector = np.asarray(embedder.embed([rec.text])[0], dtype=np.float32).tobytes()
    r.hset(f"poma:{rec.id}", mapping={
        "embedding": vector, "file_id": rec.payload["file_id"],
        "chunkset_index": rec.payload["chunkset_index"],
    })

# 4. KNN retrieval — Redis's three-part contract, then assemble.
qv = np.asarray(embedder.embed(["early termination conditions"])[0], dtype=np.float32).tobytes()
q = (Query("*=>[KNN 10 @embedding $vec AS vector_score]").sort_by("vector_score")
     .return_fields("file_id", "chunkset_index", "vector_score").dialect(2))
res = r.ft("poma_idx").search(q, query_params={"vec": qv})
context = assemble(res, volume=vol)  # -> [{"file_id", "content"}, ...]
```

Textract's own quirks are handled before a single record reaches Redis: `TABLE` blocks are spliced into their `LAYOUT_TABLE` region by bounding-box geometry and rendered as HTML with `rowspan`/`colspan`, `SELECTION_ELEMENT` checkboxes survive as ☒/☐, and offloaded figures are counted, never silently dropped. A payload missing `LAYOUT` blocks 422s up front rather than shipping scrambled reading order.

## Metadata mapping: POMA fields → Redis primitives

| POMA chunk field | Redis primitive | What it enables |
| --- | --- | --- |
| `to_embed` | `VECTOR` field (`HNSW`, `FLOAT32`) | KNN vector search |
| `file_id` | `TAG` field | exact-match per-document scoping, filter-then-search hybrid |
| `chunkset_index` | `NUMERIC` field (part of hash key) | `assemble()` dedup id, deterministic routing |
| `page` (Textract layout-derived) | `NUMERIC` field | page-range filters, page-cited answers |
| `depth` | `NUMERIC` field | hierarchy-aware filter/re-rank |
| chunkset content | volume (`Volume.write_doc`), not in Redis | content-free index, cheatsheet assembly |

## Frequently asked questions

### How do I get AWS Textract results into Redis for RAG?

Run `AnalyzeDocument` with the `LAYOUT` feature as you already do, save the raw JSON, and hand it to `PrimeCut`, which auto-detects the Textract shape and rebuilds the `LAYOUT` reading order into chunks and chunksets. `records_from_archive` then gives you deterministic ids and embeddable text for a RediSearch `VECTOR` field with `TAG`/`NUMERIC` metadata.

### Why does my Redis KNN query return nothing after indexing Textract chunks?

Redis's one documented gotcha: the `KNN` clause needs `AS vector_score`, `.return_fields(...)`, and `.dialect(2)`. Miss any of the three and `assemble()` gets a result shape it can't read metadata from — a real match comes back as an empty context list.

### Do Textract's LAYOUT_FIGURE regions and checkboxes survive the trip to Redis?

Textract never returns cropped image bytes, so figures are counted as offloaded content rather than embedded — visible in your ingest report, not silent. `SELECTION_ELEMENT` checkboxes survive as ☒/☐ marks in the chunk text, so form state reaches the retrieved cheatsheet.

### What Redis fields should AWS Textract chunks carry?

A `VECTOR` field for the embedding, a `TAG` field for `file_id`, and `NUMERIC` fields for `page` and `chunkset_index`. That gives exact-match and range filtering at KNN speed, without a post-filter scan.

### Is Textract's forms and table extraction wasted if I flatten Blocks before indexing in Redis?

Largely, yes — the LAYOUT sequence, table splicing, and checkbox state are exactly what a naive flatten discards before a vector ever reaches Redis. A structure-aware chunker keeps that extraction value intact into the `VECTOR` field.

## Related recipes

Same parser, different store: [Textract → MongoDB Atlas](/pipelines/textract-to-mongodb-atlas) · [Textract → LanceDB](/pipelines/textract-to-lancedb) · [Textract → Qdrant](/pipelines/textract-to-qdrant)

Foundations: [The Optimal Chunker for AWS Textract](/optimal-chunker-textract) · [Redis Chunking Strategy for RAG](/optimal-chunks-redis) · [All pipeline recipes](/pipelines/) · [RAG architecture guide](/guides/rag-architecture/)