Source: http://www.poma-ai.com/docs/pipelines/textract-to-turbopuffer

# The Missing Link Between AWS Textract and Optimal Retrieval in Turbopuffer

<ByAuthor />

**The short answer:** Textract's `AnalyzeDocument` with `LAYOUT` gives you multi-column-aware reading order and structured tables; Turbopuffer gives you a namespace-per-tenant store with native hybrid ANN+BM25 over object storage. Wired together naively — flatten `Blocks[]`, split, embed — they still produce mediocre RAG, because nothing in between rebuilds the document's hierarchy or keeps the row small. The missing link is POMA: `PrimeCut().ingest()` consumes the raw Textract response, emits hierarchy-preserving [chunksets](/learn/chunking/chunksets) into a `.poma` archive, and `poma.vektoria` writes **content-free** rows to Turbopuffer — `id`, `vector`, `file_id`, `chunkset_index` — while the actual text lives on a volume. Retrieval runs your normal Turbopuffer query, then `assemble()` fetches matching content and returns prompt-ready cheatsheets.

## What each end of the pipeline actually provides

**AWS Textract** (`AnalyzeDocument`) returns a flat `Blocks[]` graph. Without `LAYOUT`, only `PAGE`/`LINE`/`WORD` blocks exist and reading order isn't reliably reconstructible; with `LAYOUT`, the `LAYOUT_*` sequence encodes multi-column-aware reading order, `LAYOUT_TITLE`/`LAYOUT_SECTION_HEADER` mark headings, and `TABLES` adds structured `TABLE`/`MERGED_CELL` blocks spliced to `LAYOUT_TABLE` regions by geometry (no Id link exists between them). What it doesn't provide: markdown, cross-page hierarchy, or retrieval units. Details: [The Optimal Chunker for AWS Textract](/optimal-chunker-textract).

**Turbopuffer** provides namespaces (the tenant/collection boundary, cheap to create per corpus), an inline write-time schema, and native hybrid search — `rank_by` over dense `ANN`, `SparseKNN`, and `BM25`, fused server-side via `rerank_by=("RRF",)`. Storage is object-storage-backed with a caching compute layer. What it doesn't provide: any opinion about what a row should contain, or a size limit that forgives dumping raw text into a filterable field. Details: [Turbopuffer Chunking Strategy for RAG](/optimal-chunks-turbopuffer).

## The naive wiring, and where it breaks

The common recipe — iterate `LINE` blocks (or even `LAYOUT_*` blocks) into one string per document, split, embed, `ns.write(...)` — breaks in a pair-specific way. Flattening `Blocks[]` throws away the `LAYOUT_TABLE`-to-`TABLE` splice, so a table that Textract computed as a clean grid arrives as loose words in the row's text. To make that text filterable — a natural instinct once you've paid for the storage — teams mark the flattened blob as a filterable attribute, and promptly hit Turbopuffer's documented 4 KiB filterable-value cap on any page with a sizeable table or dense paragraph. The workaround (marking it non-filterable) then leaves you with no per-document scoping short of a full namespace scan, since the pipeline never adopted Turbopuffer's namespace-per-tenant pattern in the first place. Two Textract capabilities and one Turbopuffer constraint, defeated by the same shortcut.

## The pipeline, end to end

```bash
pip install boto3 turbopuffer
```

```python
import json
import os

import boto3
import turbopuffer
from poma import PrimeCut
from poma.vektoria import assemble, records_from_archive, Volume
from poma.embeddings import get_embedder

# 1. Your existing Textract call — LAYOUT is required; add TABLES for cell grids.
textract = boto3.client("textract")
with open("contract.png", "rb") as f:
    response = textract.analyze_document(
        Document={"Bytes": f.read()},
        FeatureTypes=["LAYOUT", "TABLES"],
    )
with open("contract.textract.json", "w") as f:
    json.dump(response, f)

# 2. The missing link — raw result JSON in, .poma archive out (the volume's source of truth).
client = PrimeCut()  # reads POMA_API_KEY
client.ingest("contract.textract.json", download_dir="archives", filename="contract.poma")

embedder = get_embedder("local:BAAI/bge-small-en-v1.5")
vol = Volume("s3://your-bucket/poma")
tpuf = turbopuffer.Turbopuffer(api_key=os.environ["TURBOPUFFER_API_KEY"], region="gcp-us-central1")
ns = tpuf.namespace("contracts")

# 3. Content-free ingest — Turbopuffer holds only routing attributes; content lives on the volume.
records = records_from_archive("archives/contract.poma")  # -> list[Record] (id, text, payload)
rows = []
for r in records:
    vol.write_doc(r.payload["file_id"], {"file_id": r.payload["file_id"],
                                          "chunks": r.payload["chunks"], "text": r.text})
    rows.append({
        "id": r.id,
        "vector": embedder.embed([r.text])[0],
        "file_id": r.payload["file_id"],
        "chunkset_index": r.payload["chunkset_index"],
    })

ns.write(
    upsert_rows=rows,
    distance_metric="cosine_distance",
    schema={"file_id": {"type": "string"}, "chunkset_index": {"type": "int"}},
)

# 4. Retrieval — Turbopuffer's verified query shape, then assemble.
qv = embedder.embed(["early termination conditions"])[0]
res = ns.query(rank_by=("vector", "ANN", qv), top_k=10,
               include_attributes=["file_id", "chunkset_index"])
context = assemble(res, volume=vol)  # -> [{"file_id", "content"}, ...]
```

POMA validates the Textract payload up front — a response missing `LAYOUT` blocks 422s immediately rather than shipping scrambled reading order — and rebuilds the cross-page heading tree before chunking. Retrieved chunksets share ancestor lineage across a document; `assemble()` deduplicates that lineage into one prompt-ready cheatsheet, the same discipline that answers our reference legal-document benchmark with **337 tokens** instead of **1,542** for a recursive-splitter baseline. [Methodology here](/document-ingestion-chunking-rag).

## Metadata mapping: POMA fields → Turbopuffer primitives

| POMA field (vektoria `Record`) | Turbopuffer primitive | What it enables |
| --- | --- | --- |
| `r.text` (chunkset `to_embed`) | `vector` in the row | dense ANN via `rank_by=("vector", "ANN", ...)` |
| `r.id` (`chunkset_uuid`) | row `id` | deterministic identity — re-ingest overwrites, never duplicates |
| `payload["file_id"]` | string attribute, filterable | scope queries to one document; required by `assemble()` |
| `payload["chunkset_index"]` | int attribute, filterable | maps a hit back to volume content; required by `assemble()` |
| `payload["chunks"]` / `text` | written to the **volume**, not the row | full chunkset content (incl. spliced Textract tables) reconstructed at retrieval, never indexed |

## Frequently asked questions

### How do I get AWS Textract output into Turbopuffer for RAG?

Run `AnalyzeDocument` with `LAYOUT` (and `TABLES` for grids) as you already do, save the raw response, and hand it to `PrimeCut.ingest`, which auto-detects the Textract shape and writes a `.poma` archive. Read that archive with `records_from_archive`, write each record's content to a volume, and upsert content-free rows — `id`, `vector`, `file_id`, `chunkset_index` — into a Turbopuffer namespace. Retrieval runs your normal query, then `assemble()` fetches content from the volume.

### Why not just flatten Textract's Blocks[] and embed the whole page into one Turbopuffer row?

Flattening loses the `LAYOUT_*` reading order Textract already computed — multi-column pages interleave into scrambled prose — and Textract's structured `TABLE` grids melt into row-by-row text with no splice back to their `LAYOUT_TABLE` region. One oversized row per page also strains Turbopuffer's 4 KiB filterable-attribute cap if that text ends up in a filterable field, and it has no chunkset boundaries to make hits self-explanatory.

### What Turbopuffer attributes should Textract chunksets carry, and why keep content off the row?

Each row needs the deterministic id, the embedding vector, and two compact filterable attributes: `file_id` and `chunkset_index`. The chunkset's actual text and member chunks stay on a volume, written once via `Volume.write_doc` — keeping rows small and comfortably under Turbopuffer's 4 KiB filterable-value cap while `assemble()` reconstructs full content from the volume at query time.

### Does Turbopuffer's 4 KiB filterable-attribute cap cause problems with Textract tables?

Only if you index table content directly as a filterable attribute — a Textract table spliced into HTML with `rowspan`/`colspan` can exceed 4 KiB for large grids. Vektoria's pattern avoids this by never putting chunk content on the row at all: `file_id` and `chunkset_index` are the only filterable attributes, well under the cap, and the HTML table lives on the volume with the rest of the chunkset.

### Do I need extra fields at query time for assemble() to work with Turbopuffer?

Yes — request `include_attributes=["file_id", "chunkset_index"]` on the query. Turbopuffer doesn't return attributes by default unless asked, and without them `assemble()` has no way to look up the matching content on the volume, so it returns nothing for a query that otherwise had hits.

## Related recipes

Same parser, different store: [Textract → Vespa](/pipelines/textract-to-vespa) · [Textract → Elasticsearch](/pipelines/textract-to-elasticsearch) · [Textract → OpenSearch](/pipelines/textract-to-opensearch)

Also available: [Textract → Qdrant](/pipelines/textract-to-qdrant) · [Textract → Pinecone](/pipelines/textract-to-pinecone) · [Textract → Weaviate](/pipelines/textract-to-weaviate)

Foundations: [The Optimal Chunker for AWS Textract](/optimal-chunker-textract) · [Turbopuffer Chunking Strategy for RAG](/optimal-chunks-turbopuffer) · [All pipeline recipes](/pipelines/) · [RAG chunking guide](/guides/rag-chunking/)