Skip to content

AWS Textract to Chroma: Chunking for RAG ​

The short answer: AWS Textract gives you a LAYOUT-annotated block graph; Chroma gives you the fastest path from zero to working retrieval — an embedded, local-first store with metadata filters. Wired together naively — flatten Blocks[], split, add — they produce a prototype that lies to you: fine on single-column demo files, scrambled on the multi-column documents Textract is actually for. The missing link is POMA: PrimeCut().ingest() consumes the raw AnalyzeDocument JSON (auto-detected), emits hierarchy-preserving chunksets, and those go into a PersistentClient collection with hierarchy metadata for where-filters — while the portable .poma archive keeps the same chunks valid for whatever database the prototype graduates to.

What each end of the pipeline actually provides ​

AWS Textract (AnalyzeDocument with FeatureTypes: ["LAYOUT", "TABLES"]) returns a flat Blocks[] graph: LAYOUT_* blocks in multi-column-aware reading order, LAYOUT_TITLE and LAYOUT_SECTION_HEADER marking structure, structured TABLE grids with MERGED_CELL spans. What it doesn't provide: markdown, cross-page hierarchy, or retrieval units — and without the LAYOUT feature, not even reliable reading order. Details: The Optimal Chunker for AWS Textract.

Chroma runs embedded in your process or as a lightweight server, persists to a local directory via PersistentClient, embeds added documents with a default embedding function (or one you bring), and filters queries with where (metadata) and where_document (content) clauses. What it doesn't provide: any opinion about what a document entry should contain. It retrieves nearest neighbors of whatever you added — including scrambled Block fragments, if that's what the prototype glue produced. Details: The Optimal Chunks for the Best Retrieval in Chroma.

The naive wiring, and where it breaks ​

The common recipe — iterate LINE blocks, join with newlines, RecursiveCharacterTextSplitter, collection.add(...) — breaks in a pair-specific way:

  • The prototype passes on the wrong corpus. Position-sorting LINE blocks happens to work on the clean, single-column memo used for the demo. On the two-column reports and forms Textract is bought for, it interleaves the columns into scrambled prose — and Chroma's default embedding function embeds it without complaint. Nothing errors; retrieval just quietly degrades in production. This is the most dangerous failure mode a prototyping store can have: false confidence.
  • The chunking decision sticks. Chroma's whole point is speed to first result, so the improvised flatten-and-split lands in glue code — and the chunking decision made at prototype time is the one that ships. When the project graduates to a hosted database, the units either port as-is or get re-invented, invalidating every retrieval evaluation the prototype produced.
  • Metadata never exists. Flattening destroys Textract's page numbers and heading roles before add() is called, so where={"page": ...} and depth filters — Chroma's retrieval primitives — have nothing to work with.

The pipeline, end to end ​

bash
pip install boto3 poma chromadb
python
import json
import os

import boto3
import chromadb
from poma import PrimeCut
from poma.utils import unpack_poma_archive

# 1. Textract — your existing call, unchanged. LAYOUT is required.
textract = boto3.client("textract")
with open("contract.png", "rb") as f:
    response = textract.analyze_document(
        Document={"Bytes": f.read()},
        FeatureTypes=["LAYOUT", "TABLES"],
    )
with open("contract.textract.json", "w") as f:
    json.dump(response, f)

# 2. The missing link — raw Blocks[] in, chunks + chunksets out.
poma = PrimeCut()  # reads POMA_API_KEY
result = poma.ingest(
    "contract.textract.json",  # Textract shape auto-detected
    download_dir="store",
    filename="contract.poma",  # keep the archive — page numbers live in its chunk records
)

# A chunkset's structural metadata comes from its leaf chunk (cs.chunks is in
# document order, so the last index is the leaf). depth is on the SDK objects;
# page only exists on the archive's chunk records.
depth_of = {c.chunk_index: c.depth for c in result.chunks}
page_of = {
    c["chunk_index"]: c.get("page")
    for c in unpack_poma_archive(poma_archive_path="store/contract.poma")["chunks"]
}

# 3. Chroma — persistent local collection, hierarchy in the metadata.
chroma = chromadb.PersistentClient(path="./chroma-data")
collection = chroma.get_or_create_collection(
    name="contracts",
    metadata={"hnsw:space": "cosine"},
)

collection.add(
    ids=[f"{cs.file_id}:{cs.chunkset_index}" for cs in result.chunksets],
    documents=[cs.to_embed for cs in result.chunksets],  # embedded by the
    # collection's embedding function — default model or bring your own
    metadatas=[
        {
            "file_id": cs.file_id,
            "chunkset_index": cs.chunkset_index,
            "page": page_of[cs.chunks[-1]],
            "depth": depth_of[cs.chunks[-1]],
        }
        for cs in result.chunksets
    ],
)

# 4. Filtered query — scope to one document, cite pages from metadata.
hits = collection.query(
    query_texts=["What are the early termination conditions?"],
    n_results=3,
    where={"file_id": result.chunksets[0].file_id},
)

POMA validates the payload up front: a Textract result missing LAYOUT blocks fails with a clear 422 telling you to re-run with FeatureTypes including "LAYOUT" — the scramble never reaches your collection. Tables are spliced by geometry and arrive as HTML with rowspan/colspan; figures (Textract returns no crops) are counted as offloaded content in content_metadata. To force or suppress detection, set external_ocr_source to "textract" or "none".

Deduplicate and merge the retrieved chunksets in chunkset_index order and you have a cheatsheet — one prompt-ready context block. On our reference legal document, that assembly answers the same query with 337 tokens of retrieved context instead of 1,542 for a recursive character splitter, with zero information loss. Methodology: Document Ingestion & Chunking for RAG.

Prototype honestly, port cleanly ​

Chroma is where chunking decisions get made — usually implicitly, inside glue code, on unrepresentative sample files. Two habits keep the prototype honest:

  • Chunk outside the store. The .poma archive — chunks and chunksets derived from your Textract results — is the portable source of truth. The Chroma collection is a disposable projection of it: rebuild locally, hand a teammate the archive, or load the identical units into the production database later. Retrieval evaluations done on the prototype stay valid because the units don't change, only the store does.
  • Prototype on the hard documents. Two-column layouts, merged-cell tables, checkbox forms — the inputs that justify Textract in the first place. POMA's LAYOUT-spine reading order and geometry-spliced HTML tables behave identically at prototype scale and production scale, so what you measure locally is what you ship.

Metadata mapping: POMA fields → Chroma primitives ​

POMA chunk fieldChroma primitiveWhat it enables
to_embeddocuments entry, embedded by the collection's embedding functionsemantic retrieval; where_document content filters
file_idmetadata key (string), where={"file_id": ...}scope queries to one document
page (on the archive's chunk records; take the chunkset's leaf chunk)metadata key (int), where with $gte/$ltepage-cited answers, page-range filters
depth of the leaf chunkmetadata key (int)filter or re-rank by hierarchy level
chunkset_indexmetadata key (int)stable ordering at assembly time
chunkset lineageroot-to-leaf text inside the document entry; .poma archive alongsideself-explanatory hits, portability to any other store

Frequently asked questions ​

How do I get AWS Textract results into Chroma for RAG? ​

Save the raw AnalyzeDocument JSON (with LAYOUT in the feature list), run PrimeCut().ingest() on it (auto-detected, hierarchy rebuilt), then collection.add() each chunkset's to_embed as a document with file_id/chunkset_index/page/depth metadata on a PersistentClient collection. Query with where-filters and assemble cheatsheets.

What metadata should Textract chunks carry in a Chroma collection? ​

file_id (string), chunkset_index (int), page (int), and depth (int) — all scalar, as Chroma requires. file_id, chunkset_index, and depth come off the SDK's chunkset and chunk objects; page comes from the .poma archive's chunk records, read from the chunkset's leaf chunk. They enable document scoping, page-range filters via $gte/$lte, hierarchy filtering, and stable assembly order.

Why does my Textract-to-Chroma prototype work on memos but fail on two-column PDFs? ​

Position-sorting LINE blocks works on single-column memos and silently interleaves multi-column layouts. Chroma embeds the scrambled prose without complaint, so nothing errors — retrieval just degrades. POMA follows the LAYOUT block sequence instead and 422s on payloads without LAYOUT.

Can I move a Chroma prototype built on Textract output to another vector database later? ​

Yes — the .poma archive is the portable source of truth. The identical chunks and chunksets load into Qdrant, pgvector, Milvus, or any other store, so prototype retrieval evaluations stay valid after the move.

Should I use Chroma's default embedding function for Textract chunksets? ​

For prototyping, yes — zero wiring. Chunksets make even a small default model useful because to_embed is normalized and self-explanatory. Swapping in your own embedding function later changes nothing about the units or metadata.

Same parser, different store: Textract → Qdrant · Textract → pgvector · Textract → Milvus

Same store, different parser: Mistral OCR → Chroma · Docling → Chroma · LlamaParse → Chroma

Foundations: The Optimal Chunker for AWS Textract · The Optimal Chunks for Chroma · All pipeline recipes · RAG chunking guide