Source: http://www.poma-ai.com/docs/pipelines/textract-to-opensearch

# The Missing Link Between AWS Textract and Optimal Retrieval in OpenSearch

<ByAuthor />

**The short answer:** Textract's `AnalyzeDocument` with `LAYOUT` gives you multi-column-aware reading order and structured tables; OpenSearch gives you its own k-NN plugin plus a dedicated hybrid search pipeline. Wired together naively — flatten `Blocks[]`, split, embed — they still produce mediocre RAG, because nothing in between rebuilds the document's hierarchy or gives OpenSearch's hybrid fusion a real lexical signal to work with. The missing link is POMA: `PrimeCut().ingest()` consumes the raw Textract response, emits hierarchy-preserving [chunksets](/learn/chunking/chunksets) into a `.poma` archive, and `poma.vektoria` indexes **content-free** documents — `knn_vector` embedding, `file_id`, `chunkset_index` — while the actual text lives on a volume. Retrieval runs your normal `knn` query, then `assemble()` fetches matching content and returns prompt-ready cheatsheets.

## What each end of the pipeline actually provides

**AWS Textract** (`AnalyzeDocument`) returns a flat `Blocks[]` graph. Without `LAYOUT`, only `PAGE`/`LINE`/`WORD` blocks exist and reading order isn't reliably reconstructible; with `LAYOUT`, the `LAYOUT_*` sequence is Textract's own multi-column-aware reading order, `LAYOUT_TITLE`/`LAYOUT_SECTION_HEADER` mark headings, and `TABLES` adds structured `TABLE`/`MERGED_CELL` blocks spliced to `LAYOUT_TABLE` regions by geometry (no Id link exists between them). What it doesn't provide: markdown, cross-page hierarchy, or retrieval units. Details: [The Optimal Chunker for AWS Textract](/optimal-chunker-textract).

**OpenSearch** forked from Elasticsearch 7.10 and built its own `knn_vector` field type (HNSW via `nmslib` or `faiss`), native BM25 on analyzed text fields, and a dedicated `hybrid` query type fused through a search pipeline doing normalization and weighted-sum scoring — distinct from Elasticsearch's RRF `rank`. What it doesn't provide: any opinion about what a document should represent, or a lexical signal if you never indexed one. Details: [OpenSearch Chunking Strategy for RAG](/optimal-chunks-opensearch).

## The naive wiring, and where it breaks

The common recipe — flatten `Blocks[]` into one string per page, index it with only a `knn_vector` field, wire up a `hybrid` query, expect Elasticsearch-style RRF behavior — breaks in a pair-specific way. OpenSearch's hybrid mechanism isn't RRF: it's a search pipeline that normalizes and weight-sums a lexical sub-query's score against the vector sub-query's score. If the flattened Textract text was never indexed into an analyzed field (or was collapsed into the same field as routing metadata), the lexical sub-query has nothing meaningful to score, and the pipeline's weighted sum degrades toward vector-only ranking no matter how the weights are tuned. The same shortcut that skips Textract's `LAYOUT_TABLE`-to-`TABLE` splice — leaving table cells melted into the flattened blob — is what starved the lexical half of the hybrid pipeline in the first place.

## The pipeline, end to end

```bash
pip install boto3 opensearch-py
```

```python
import json

import boto3
from poma import PrimeCut
from poma.vektoria import assemble, records_from_archive, Volume
from poma.embeddings import get_embedder
from opensearchpy import OpenSearch

# 1. Your existing Textract call — LAYOUT is required; add TABLES for cell grids.
textract = boto3.client("textract")
with open("contract.png", "rb") as f:
    response = textract.analyze_document(
        Document={"Bytes": f.read()},
        FeatureTypes=["LAYOUT", "TABLES"],
    )
with open("contract.textract.json", "w") as f:
    json.dump(response, f)

# 2. The missing link — raw result JSON in, .poma archive out (the volume's source of truth).
client = PrimeCut()  # reads POMA_API_KEY
client.ingest("contract.textract.json", download_dir="archives", filename="contract.poma")

embedder = get_embedder("local:BAAI/bge-small-en-v1.5")
vol = Volume("s3://your-bucket/poma")
osc = OpenSearch(hosts=[{"host": "localhost", "port": 9200}])

osc.indices.create(index="poma", body={"mappings": {"properties": {
    "embedding": {"type": "knn_vector", "dimension": embedder.dims,
                  "method": {"name": "hnsw", "space_type": "cosinesimil", "engine": "faiss"}},
    "file_id": {"type": "keyword"},
    "chunkset_index": {"type": "integer"},
}}}, ignore=400)

# 3. Content-free ingest — OpenSearch holds only routing metadata; content lives on the volume.
records = records_from_archive("archives/contract.poma")  # -> list[Record] (id, text, payload)
for r in records:
    vol.write_doc(r.payload["file_id"], {"file_id": r.payload["file_id"],
                                          "chunks": r.payload["chunks"], "text": r.text})
    osc.index(index="poma", id=r.id, body={
        "embedding": embedder.embed([r.text])[0],
        "file_id": r.payload["file_id"],
        "chunkset_index": r.payload["chunkset_index"],
    })

# 4. Retrieval — OpenSearch's verified knn shape, then assemble.
qv = embedder.embed(["early termination conditions"])[0]
res = osc.search(index="poma", knn={"field": "embedding", "query_vector": qv,
                                     "k": 10, "num_candidates": 100})
context = assemble(res, volume=vol)  # -> [{"file_id", "content"}, ...]
```

POMA validates the Textract payload up front — a response missing `LAYOUT` blocks 422s immediately rather than shipping scrambled reading order — and rebuilds the cross-page heading tree before chunking. Retrieved chunksets share ancestor lineage across a document; `assemble()` deduplicates that lineage into one prompt-ready cheatsheet, the same discipline that answers our reference legal-document benchmark with **337 tokens** instead of **1,542** for a recursive-splitter baseline. [Methodology here](/document-ingestion-chunking-rag).

## Metadata mapping: POMA fields → OpenSearch primitives

| POMA field (vektoria `Record`) | OpenSearch primitive | What it enables |
| --- | --- | --- |
| `r.text` (chunkset `to_embed`) | `knn_vector` field | k-NN via the `knn` search clause |
| `r.id` (`chunkset_uuid`) | document `_id` | deterministic identity — re-index overwrites, never duplicates |
| `payload["file_id"]` | `keyword` field | exact-match filtering scoped to one document; returned by default for `assemble()` |
| `payload["chunkset_index"]` | `integer` field | maps a hit back to volume content; returned by default for `assemble()` |
| `payload["chunks"]` / `text` | written to the **volume**, not the mapping | full chunkset content (incl. spliced Textract tables) reconstructed at retrieval, never indexed |

## Frequently asked questions

### How do I get AWS Textract output into OpenSearch for RAG?

Run `AnalyzeDocument` with `LAYOUT` (and `TABLES`), save the response, and feed it to `PrimeCut.ingest`, which auto-detects the Textract shape and writes a `.poma` archive. Read it with `records_from_archive`, write each record's content to a volume, and index content-free documents — an embedding in a `knn_vector` field plus `file_id` and `chunkset_index` — into OpenSearch. Query with the `knn` clause, then hand the result to `assemble()`.

### Why does flattening Textract's Blocks[] complicate OpenSearch's hybrid search pipeline?

OpenSearch's hybrid query type fuses a lexical and a vector sub-query through a search pipeline that normalizes and weight-sums their scores — a different mechanism from Elasticsearch's RRF. A flattened `Blocks[]` blob with no chunkset boundaries and no separate text field gives the lexical half of that pipeline nothing meaningful to score, so the fused result degrades toward vector-only ranking regardless of the pipeline configuration.

### What OpenSearch mapping should Textract chunksets use?

`embedding` as `knn_vector` (dimension matching your embedder, method name `hnsw`, engine `faiss` or `nmslib`), `file_id` as `keyword`, `chunkset_index` as `integer` — one document per chunkset. As with Elasticsearch, chunkset content itself is not indexed; it's written to the volume vektoria manages, and OpenSearch returns `file_id` and `chunkset_index` in the document body by default.

### Is OpenSearch's k-NN plugin the same as Elasticsearch's dense_vector for Textract content?

Conceptually, not literally. OpenSearch forked from Elasticsearch at 7.10 and built its own `knn_vector` field and k-NN plugin (`nmslib` or `faiss` engines) rather than tracking Elasticsearch's `dense_vector` work, and its hybrid mechanism is a dedicated search pipeline rather than RRF. The vektoria ingest pattern for Textract chunksets is identical either way; only the field type and hybrid setup differ.

### What happens to Textract tables and checkboxes when they reach OpenSearch?

Nothing OpenSearch-specific — PrimeCut does that work before ingest. Textract's `TABLE` blocks are spliced into HTML by geometry against their `LAYOUT_TABLE` region, and `SELECTION_ELEMENT` checkboxes become selected/unselected marks in the chunkset text. OpenSearch only ever stores the embedding and routing metadata; the table HTML and checkbox state live in the chunkset content on the volume, fetched by `assemble()` after retrieval.

## Related recipes

Same parser, different store: [Textract → Turbopuffer](/pipelines/textract-to-turbopuffer) · [Textract → Vespa](/pipelines/textract-to-vespa) · [Textract → Elasticsearch](/pipelines/textract-to-elasticsearch)

Also available: [Textract → Pinecone](/pipelines/textract-to-pinecone) · [Textract → Weaviate](/pipelines/textract-to-weaviate) · [Textract → pgvector](/pipelines/textract-to-pgvector)

Foundations: [The Optimal Chunker for AWS Textract](/optimal-chunker-textract) · [OpenSearch Chunking Strategy for RAG](/optimal-chunks-opensearch) · [All pipeline recipes](/pipelines/) · [RAG architecture guide](/guides/rag-architecture/)