Source: http://www.poma-ai.com/docs/pipelines/textract-to-mongodb-atlas

# The Missing Link Between AWS Textract and Optimal Retrieval in MongoDB Atlas

<ByAuthor />

**The short answer:** AWS Textract gives you an excellent, forms-and-tables-aware layout graph; MongoDB Atlas gives you vector search folded directly into the aggregation pipeline you already run. Wired together naively — flatten `Blocks[]`, split, embed — they still produce mediocre RAG, because nothing in between rebuilds Textract's own reading order or projects the metadata Atlas needs at query time. The missing link is POMA: `PrimeCut().ingest()` consumes the raw `AnalyzeDocument` response, follows the `LAYOUT` spine, and emits hierarchy-preserving [chunksets](/learn/chunking/chunksets); `vektoria`'s `records_from_archive` and `assemble()` keep the collection content-free and turn a properly-projected `$vectorSearch` result into prompt-ready context.

## What each end of the pipeline actually provides

**AWS Textract** (`AnalyzeDocument` with the `LAYOUT` feature) returns a flat `Blocks[]` graph: `LAYOUT_TITLE`/`LAYOUT_SECTION_HEADER` blocks in multi-column-aware reading order, structured `TABLE` blocks with no `Id` link to their `LAYOUT_TABLE` region, and `LAYOUT_FIGURE` regions with no image bytes at all. What it doesn't provide: cross-page hierarchy, a table-to-layout link, or any retrieval unit whatsoever. Details: [The Optimal Chunker for AWS Textract](/optimal-chunker-textract).

**MongoDB Atlas** provides `$vectorSearch` as an aggregation stage, compound hybrid search via Atlas Search, and ordinary document fields you can `$match` alongside the vector query — all inside the cluster you already operate. What it doesn't provide: any field in the aggregation output you didn't explicitly project. Details: [MongoDB Atlas Chunking Strategy for RAG](/optimal-chunks-mongodb-atlas).

## The naive wiring, and where it breaks

The common recipe — flatten `Blocks[]` to text, split into fixed windows, `insert_one` per chunk — breaks in a pair-specific way:

- **Flatten `Blocks[]` without honoring `LAYOUT`** and Textract's multi-column reports interleave into scrambled prose before they ever reach Mongo — no aggregation stage fixes reading order after the fact.
- **Insert one document per fixed-size window with no metadata field**, and `file_id`/`page` scoping is impossible without a full collection scan — Atlas ranks whatever you wrote, nothing more.
- **Even correctly-chunked text fails silently at the query layer.** A `$vectorSearch` stage with no follow-up `$project` of the metadata field returns a non-empty hit list that POMA's `assemble()` cannot read `file_id` or `chunkset_index` from — it comes back an empty list, indistinguishable from "no relevant chunks" unless you already know to check the aggregation pipeline.

## The pipeline, end to end

```bash
pip install boto3 pymongo
```

```python
import json
import os

import boto3
from pymongo import MongoClient
from poma import PrimeCut
from poma.vektoria import assemble, records_from_archive, Volume
from poma.embeddings import get_embedder

# 1. Your existing Textract call — LAYOUT is required; add TABLES for cell grids.
textract = boto3.client("textract")
with open("contract.png", "rb") as f:
    response = textract.analyze_document(
        Document={"Bytes": f.read()},
        FeatureTypes=["LAYOUT", "TABLES"],
    )
with open("contract.textract.json", "w") as f:
    json.dump(response, f)

# 2. The missing link — raw Textract result in, a portable .poma archive out.
client = PrimeCut()  # reads POMA_API_KEY
client.ingest("contract.textract.json", download_dir="archives", filename="contract.poma")

embedder = get_embedder("local:BAAI/bge-small-en-v1.5")
vol = Volume("s3://your-bucket/poma")
mongo = MongoClient(os.environ["MONGODB_ATLAS_URI"])
col = mongo["poma"]["chunksets"]
# Atlas vector index (numDimensions, similarity) is created once via the Atlas UI/API,
# named "poma_vec" here — not through the pymongo driver itself.

# 3. Content-free ingest — MongoDB holds only routing metadata; content lives on the volume.
records = records_from_archive("archives/contract.poma")  # -> list[Record] (id, text, payload)
for rec in records:
    vol.write_doc(rec.payload["file_id"], {"file_id": rec.payload["file_id"],
                                            "chunks": rec.payload["chunks"], "text": rec.text})
    col.insert_one({
        "_id": rec.id,
        "embedding": embedder.embed([rec.text])[0],
        "metadata": {"file_id": rec.payload["file_id"], "chunkset_index": rec.payload["chunkset_index"]},
    })

# 4. $vectorSearch retrieval — $project the metadata explicitly, then assemble.
qv = embedder.embed(["early termination conditions"])[0]
hits = list(col.aggregate([
    {"$vectorSearch": {"index": "poma_vec", "path": "embedding",
                       "queryVector": qv, "numCandidates": 200, "limit": 10}},
    {"$project": {"_id": 1, "metadata": 1, "score": {"$meta": "vectorSearchScore"}}},
]))
context = assemble(hits, volume=vol)  # -> [{"file_id", "content"}, ...]
```

Textract's own quirks are handled before a single document reaches Atlas: `TABLE` blocks are spliced into their `LAYOUT_TABLE` region by bounding-box geometry and rendered as HTML with `rowspan`/`colspan`, `SELECTION_ELEMENT` checkboxes survive as ☒/☐, and offloaded figures are counted, never silently dropped. A payload missing `LAYOUT` blocks 422s up front rather than shipping scrambled reading order.

## Metadata mapping: POMA fields → MongoDB Atlas primitives

| POMA chunk field | MongoDB Atlas primitive | What it enables |
| --- | --- | --- |
| `to_embed` | `embedding` field, covered by the Atlas Vector Search index | `$vectorSearch` ANN |
| `file_id` | `metadata.file_id` document field | `$match` per-document scoping |
| `chunkset_index` | `metadata.chunkset_index` document field | `assemble()` dedup id, deterministic routing |
| `page` (Textract layout-derived) | `metadata.page` document field | page-cited answers, `$match` range filters |
| `depth` | `metadata.depth` document field | hierarchy-aware filter/re-rank |
| chunkset content | volume (`Volume.write_doc`), not in Mongo | content-free documents, cheatsheet assembly |

## Frequently asked questions

### How do I get AWS Textract results into MongoDB Atlas for RAG?

Run `AnalyzeDocument` with the `LAYOUT` feature as you already do, save the raw JSON, and hand it to `PrimeCut`, which auto-detects the Textract shape and rebuilds the reading order into chunks and chunksets. `records_from_archive` gives you deterministic ids and embeddable text to insert as documents covered by an Atlas Vector Search index.

### Why does POMA's assemble() return nothing from my MongoDB Atlas Textract index?

Atlas's one documented gotcha: `$vectorSearch` must be followed by a `$project` that explicitly includes the `metadata` field. Skip it and a real hit list comes back with a score but no `file_id`/`chunkset_index` — `assemble()` returns an empty list.

### How are Textract's spliced tables represented once they land in MongoDB Atlas?

Textract has no Id link between a `LAYOUT_TABLE` region and its `TABLE` block, so POMA splices them by geometry before chunking — the cell grid arrives as HTML with `rowspan`/`colspan` already inline in the chunkset text stored in the collection.

### What Atlas document fields should AWS Textract chunks carry?

An embedding field covered by the vector index, plus a `metadata` object with `file_id`, `chunkset_index`, `page`, and `depth` — one object keeps the required `$project` a single inclusion.

### Should I switch from Textract to another OCR to fix MongoDB Atlas retrieval quality?

Usually not. Atlas ranks whatever you indexed; weak retrieval is almost always upstream, in how Blocks becomes chunks. A BYOCR chunker closes that gap without touching your Textract contract or your cluster.

## Related recipes

Same parser, different store: [Textract → Redis](/pipelines/textract-to-redis) · [Textract → LanceDB](/pipelines/textract-to-lancedb) · [Textract → Qdrant](/pipelines/textract-to-qdrant)

Foundations: [The Optimal Chunker for AWS Textract](/optimal-chunker-textract) · [MongoDB Atlas Chunking Strategy for RAG](/optimal-chunks-mongodb-atlas) · [All pipeline recipes](/pipelines/) · [RAG architecture guide](/guides/rag-architecture/)