Source: http://www.poma-ai.com/docs/pipelines/textract-to-lancedb

# The Missing Link Between AWS Textract and Optimal Retrieval in LanceDB

<ByAuthor />

**The short answer:** AWS Textract gives you an excellent, forms-and-tables-aware layout graph; LanceDB gives you an embedded, Arrow-native table that scales from a laptop to object storage with no schema rewrite. Wired together naively — flatten `Blocks[]`, split, embed — they still produce mediocre RAG, and LanceDB will not tell you: it has no query-time metadata gotcha to fail loudly against. The missing link is POMA: `PrimeCut().ingest()` consumes the raw `AnalyzeDocument` response, follows the `LAYOUT` spine, and emits hierarchy-preserving [chunksets](/learn/chunking/chunksets); `vektoria`'s `records_from_archive` and `assemble()` keep the table content-free and turn a normal `search()` call into prompt-ready context.

## What each end of the pipeline actually provides

**AWS Textract** (`AnalyzeDocument` with the `LAYOUT` feature) returns a flat `Blocks[]` graph: `LAYOUT_TITLE`/`LAYOUT_SECTION_HEADER` blocks in multi-column-aware reading order, structured `TABLE` blocks with no `Id` link to their `LAYOUT_TABLE` region, and `LAYOUT_FIGURE` regions with no image bytes at all. What it doesn't provide: cross-page hierarchy, a table-to-layout link, or any retrieval unit whatsoever. Details: [The Optimal Chunker for AWS Textract](/optimal-chunker-textract).

**LanceDB** provides an Arrow/Lance-backed table you can query with `.search(qv).where(...)`, embedded locally or directly against object storage, with every scalar column returned by default. What it doesn't provide: any opinion about what a row should hold, or any error when that row is garbage — a bad chunking decision ranks confidently, same as a good one. Details: [LanceDB Chunking Strategy for RAG](/optimal-chunks-lancedb).

## The naive wiring, and where it breaks

The common recipe — flatten `Blocks[]` to text, split into fixed windows, append rows to a table — breaks in a pair-specific way, and LanceDB won't flag it:

- **Flatten `Blocks[]` without honoring `LAYOUT`** and Textract's multi-column reports interleave into scrambled prose before a single row is ever written.
- **Skip `file_id` as a proper scalar column** and `.where(...)` filtering degrades from an indexed predicate to a full table scan the moment your corpus grows past a handful of documents.
- **Unlike Redis or MongoDB Atlas, LanceDB returns every column by default** — there is no missing-`return_fields` or missing-`$project` error to catch a scrambled-prose mistake at the query layer. A `search()` call over garbage rows succeeds, ranks the garbage confidently, and `assemble()` dutifully serves it as context. The only thing standing between a two-column Textract report and a silently broken RAG answer is whether `LAYOUT` was requested at ingest time.

## The pipeline, end to end

```bash
pip install boto3 lancedb
```

```python
import json

import boto3
import lancedb
from poma import PrimeCut
from poma.vektoria import assemble, records_from_archive, Volume
from poma.embeddings import get_embedder

# 1. Your existing Textract call — LAYOUT is required; add TABLES for cell grids.
textract = boto3.client("textract")
with open("contract.png", "rb") as f:
    response = textract.analyze_document(
        Document={"Bytes": f.read()},
        FeatureTypes=["LAYOUT", "TABLES"],
    )
with open("contract.textract.json", "w") as f:
    json.dump(response, f)

# 2. The missing link — raw Textract result in, a portable .poma archive out.
client = PrimeCut()  # reads POMA_API_KEY
client.ingest("contract.textract.json", download_dir="archives", filename="contract.poma")

embedder = get_embedder("local:BAAI/bge-small-en-v1.5")
vol = Volume("s3://your-bucket/poma")
db = lancedb.connect("s3://your-bucket/lancedb")

# 3. Content-free ingest — the table holds only routing metadata; content lives on the volume.
records = records_from_archive("archives/contract.poma")  # -> list[Record] (id, text, payload)
rows = []
for rec in records:
    vol.write_doc(rec.payload["file_id"], {"file_id": rec.payload["file_id"],
                                            "chunks": rec.payload["chunks"], "text": rec.text})
    rows.append({
        "id": rec.id,
        "vector": embedder.embed([rec.text])[0],
        "file_id": rec.payload["file_id"],
        "chunkset_index": rec.payload["chunkset_index"],
    })

tbl = db.create_table("poma", data=rows, exist_ok=True)

# 4. Vector search retrieval, then assemble prompt-ready context.
qv = embedder.embed(["early termination conditions"])[0]
results = tbl.search(qv).where("file_id = 'contract.pdf'").limit(10).to_list()
context = assemble(results, volume=vol)  # -> [{"file_id", "content"}, ...]
```

Textract's own quirks are handled before a single row reaches Lance: `TABLE` blocks are spliced into their `LAYOUT_TABLE` region by bounding-box geometry and rendered as HTML with `rowspan`/`colspan`, `SELECTION_ELEMENT` checkboxes survive as ☒/☐, and offloaded figures are counted, never silently dropped. A payload missing `LAYOUT` blocks 422s up front rather than shipping scrambled reading order.

## Metadata mapping: POMA fields → LanceDB primitives

| POMA chunk field | LanceDB primitive | What it enables |
| --- | --- | --- |
| `to_embed` | `vector` column | ANN via `.search(qv)` |
| `file_id` | scalar column (string) | `.where("file_id = '...'")` per-document scoping |
| `chunkset_index` | scalar column | `assemble()` dedup id, deterministic routing |
| `page` (Textract layout-derived) | volume document, not a column | content-free retrieval, cheatsheet assembly |
| `depth` | volume document, not a column | content-free retrieval, cheatsheet assembly |
| chunkset content | volume (`Volume.write_doc`), not in the table | content-free table, cheatsheet assembly |

## Frequently asked questions

### How do I get AWS Textract results into LanceDB for RAG?

Run `AnalyzeDocument` with the `LAYOUT` feature as you already do, save the raw JSON, and hand it to `PrimeCut`, which auto-detects the Textract shape and rebuilds the reading order into chunks and chunksets. `records_from_archive` gives you deterministic ids and embeddable text to write as rows alongside `file_id` and `chunkset_index` columns.

### What happens if I flatten Textract's Blocks before writing rows to LanceDB?

Nothing stops it. LanceDB returns every column by default, so there's no missing-metadata error to surface the mistake. Scrambled, position-sorted text from a multi-column report gets embedded, written, and ranked with complete confidence — the only signal is the answer quality itself.

### How are Textract's checkboxes and offloaded figures represented in a LanceDB row?

`SELECTION_ELEMENT` checkboxes survive as ☒/☐ marks directly in the text column. `LAYOUT_FIGURE` regions have no bytes, so each is counted as offloaded content in `content_metadata` rather than written as a row — visible in your ingest report, not silently missing.

### Does LanceDB need any extra query flag to retrieve Textract chunk metadata?

No. Unlike Redis or MongoDB Atlas, `search().to_list()` returns every column by default, including `file_id` and `chunkset_index` — `assemble()` reads it with no extra flag. That's also why a flattening mistake goes uncaught elsewhere in the pipeline.

### What LanceDB columns should AWS Textract chunksets carry?

A `vector` column sized to your embedder, plus `file_id` and `chunkset_index` as scalar columns — `page` and `depth` stay on the volume document alongside the full chunkset content, not as separate columns. `create_table` infers this schema from your first batch of rows, so keep the same keys across every record.

## Related recipes

Same parser, different store: [Textract → Redis](/pipelines/textract-to-redis) · [Textract → MongoDB Atlas](/pipelines/textract-to-mongodb-atlas) · [Textract → Chroma](/pipelines/textract-to-chroma)

Foundations: [The Optimal Chunker for AWS Textract](/optimal-chunker-textract) · [LanceDB Chunking Strategy for RAG](/optimal-chunks-lancedb) · [All pipeline recipes](/pipelines/) · [RAG architecture guide](/guides/rag-architecture/)