Skip to content

AWS Textract to MongoDB Atlas: Chunking for RAG ​

The short answer: AWS Textract gives you an excellent, forms-and-tables-aware layout graph; MongoDB Atlas gives you vector search folded directly into the aggregation pipeline you already run. Wired together naively — flatten Blocks[], split, embed — they still produce mediocre RAG, because nothing in between rebuilds Textract's own reading order or projects the metadata Atlas needs at query time. The missing link is POMA: PrimeCut().ingest() consumes the raw AnalyzeDocument response, follows the LAYOUT spine, and emits hierarchy-preserving chunksets; vektoria's records_from_archive and assemble() keep the collection content-free and turn a properly-projected $vectorSearch result into prompt-ready context.

What each end of the pipeline actually provides ​

AWS Textract (AnalyzeDocument with the LAYOUT feature) returns a flat Blocks[] graph: LAYOUT_TITLE/LAYOUT_SECTION_HEADER blocks in multi-column-aware reading order, structured TABLE blocks with no Id link to their LAYOUT_TABLE region, and LAYOUT_FIGURE regions with no image bytes at all. What it doesn't provide: cross-page hierarchy, a table-to-layout link, or any retrieval unit whatsoever. Details: The Optimal Chunker for AWS Textract.

MongoDB Atlas provides $vectorSearch as an aggregation stage, compound hybrid search via Atlas Search, and ordinary document fields you can $match alongside the vector query — all inside the cluster you already operate. What it doesn't provide: any field in the aggregation output you didn't explicitly project. Details: MongoDB Atlas Chunking Strategy for RAG.

The naive wiring, and where it breaks ​

The common recipe — flatten Blocks[] to text, split into fixed windows, insert_one per chunk — breaks in a pair-specific way:

  • Flatten Blocks[] without honoring LAYOUT and Textract's multi-column reports interleave into scrambled prose before they ever reach Mongo — no aggregation stage fixes reading order after the fact.
  • Insert one document per fixed-size window with no metadata field, and file_id/page scoping is impossible without a full collection scan — Atlas ranks whatever you wrote, nothing more.
  • Even correctly-chunked text fails silently at the query layer. A $vectorSearch stage with no follow-up $project of the metadata field returns a non-empty hit list that POMA's assemble() cannot read file_id or chunkset_index from — it comes back an empty list, indistinguishable from "no relevant chunks" unless you already know to check the aggregation pipeline.

The pipeline, end to end ​

bash
pip install boto3 pymongo
python
import json
import os

import boto3
from pymongo import MongoClient
from poma import PrimeCut
from poma.vektoria import assemble, records_from_archive, Volume
from poma.embeddings import get_embedder

# 1. Your existing Textract call — LAYOUT is required; add TABLES for cell grids.
textract = boto3.client("textract")
with open("contract.png", "rb") as f:
    response = textract.analyze_document(
        Document={"Bytes": f.read()},
        FeatureTypes=["LAYOUT", "TABLES"],
    )
with open("contract.textract.json", "w") as f:
    json.dump(response, f)

# 2. The missing link — raw Textract result in, a portable .poma archive out.
client = PrimeCut()  # reads POMA_API_KEY
client.ingest("contract.textract.json", download_dir="archives", filename="contract.poma")

embedder = get_embedder("local:BAAI/bge-small-en-v1.5")
vol = Volume("s3://your-bucket/poma")
mongo = MongoClient(os.environ["MONGODB_ATLAS_URI"])
col = mongo["poma"]["chunksets"]
# Atlas vector index (numDimensions, similarity) is created once via the Atlas UI/API,
# named "poma_vec" here — not through the pymongo driver itself.

# 3. Content-free ingest — MongoDB holds only routing metadata; content lives on the volume.
records = records_from_archive("archives/contract.poma")  # -> list[Record] (id, text, payload)
for rec in records:
    vol.write_doc(rec.payload["file_id"], {"file_id": rec.payload["file_id"],
                                            "chunks": rec.payload["chunks"], "text": rec.text})
    col.insert_one({
        "_id": rec.id,
        "embedding": embedder.embed([rec.text])[0],
        "metadata": {"file_id": rec.payload["file_id"], "chunkset_index": rec.payload["chunkset_index"]},
    })

# 4. $vectorSearch retrieval — $project the metadata explicitly, then assemble.
qv = embedder.embed(["early termination conditions"])[0]
hits = list(col.aggregate([
    {"$vectorSearch": {"index": "poma_vec", "path": "embedding",
                       "queryVector": qv, "numCandidates": 200, "limit": 10}},
    {"$project": {"_id": 1, "metadata": 1, "score": {"$meta": "vectorSearchScore"}}},
]))
context = assemble(hits, volume=vol)  # -> [{"file_id", "content"}, ...]

Textract's own quirks are handled before a single document reaches Atlas: TABLE blocks are spliced into their LAYOUT_TABLE region by bounding-box geometry and rendered as HTML with rowspan/colspan, SELECTION_ELEMENT checkboxes survive as ☒/☐, and offloaded figures are counted, never silently dropped. A payload missing LAYOUT blocks 422s up front rather than shipping scrambled reading order.

Metadata mapping: POMA fields → MongoDB Atlas primitives ​

POMA chunk fieldMongoDB Atlas primitiveWhat it enables
to_embedembedding field, covered by the Atlas Vector Search index$vectorSearch ANN
file_idmetadata.file_id document field$match per-document scoping
chunkset_indexmetadata.chunkset_index document fieldassemble() dedup id, deterministic routing
page (Textract layout-derived)metadata.page document fieldpage-cited answers, $match range filters
depthmetadata.depth document fieldhierarchy-aware filter/re-rank
chunkset contentvolume (Volume.write_doc), not in Mongocontent-free documents, cheatsheet assembly

Frequently asked questions ​

How do I get AWS Textract results into MongoDB Atlas for RAG? ​

Run AnalyzeDocument with the LAYOUT feature as you already do, save the raw JSON, and hand it to PrimeCut, which auto-detects the Textract shape and rebuilds the reading order into chunks and chunksets. records_from_archive gives you deterministic ids and embeddable text to insert as documents covered by an Atlas Vector Search index.

Why does POMA's assemble() return nothing from my MongoDB Atlas Textract index? ​

Atlas's one documented gotcha: $vectorSearch must be followed by a $project that explicitly includes the metadata field. Skip it and a real hit list comes back with a score but no file_id/chunkset_index — assemble() returns an empty list.

How are Textract's spliced tables represented once they land in MongoDB Atlas? ​

Textract has no Id link between a LAYOUT_TABLE region and its TABLE block, so POMA splices them by geometry before chunking — the cell grid arrives as HTML with rowspan/colspan already inline in the chunkset text stored in the collection.

What Atlas document fields should AWS Textract chunks carry? ​

An embedding field covered by the vector index, plus a metadata object with file_id, chunkset_index, page, and depth — one object keeps the required $project a single inclusion.

Should I switch from Textract to another OCR to fix MongoDB Atlas retrieval quality? ​

Usually not. Atlas ranks whatever you indexed; weak retrieval is almost always upstream, in how Blocks becomes chunks. A BYOCR chunker closes that gap without touching your Textract contract or your cluster.

Same parser, different store: Textract → Redis · Textract → LanceDB · Textract → Qdrant

Foundations: The Optimal Chunker for AWS Textract · MongoDB Atlas Chunking Strategy for RAG · All pipeline recipes · RAG architecture guide