AWS Textract to Pinecone: Chunking for RAG
The short answer: AWS Textract gives you a rich block graph — layout regions in reading order, structured tables, form state; Pinecone gives you serverless vector search with namespaces and metadata filtering. Wired together naively they fail twice: flattening Blocks[] to LINE text scrambles multi-column reading order before anything is embedded, and dragging Textract's enormous per-block metadata along blows Pinecone's ~40 KB per-vector metadata cap. The missing link is POMA: PrimeCut().ingest() consumes the raw AnalyzeDocument response (auto-detected), reads the LAYOUT spine, and emits hierarchy-preserving chunksets whose compact file_id/page/depth fields map cleanly onto Pinecone's metadata filters.
What each end of the pipeline actually provides
AWS Textract (AnalyzeDocument with FeatureTypes=["LAYOUT", "TABLES"]) returns a flat Blocks[] array: LAYOUT_* blocks in multi-column-aware reading order, heading roles, structured TABLE/CELL/MERGED_CELL grids, and checkbox state — every block wrapped in geometry, confidence scores, Ids, and Relationships arrays. What it doesn't provide: markdown, cross-page hierarchy, or anything remotely sized for a vector store's metadata field. Details: The Optimal Chunker for AWS Textract.
Pinecone provides serverless indexes, namespaces that physically partition an index (the natural per-tenant boundary), metadata filtering with $eq/$in/$gte operators, and sparse-dense hybrid support. What it doesn't provide: any opinion about what a vector should contain — and it enforces a hard constraint the Textract side keeps tripping: metadata of roughly 40 KB per vector, maximum. Details: The Optimal Chunks for the Best Retrieval in Pinecone.
The naive wiring, and where it breaks
The common recipe — flatten LINE blocks to text, split, embed, attach "everything we might need later" as metadata — breaks in a pair-specific way:
- Raw Block metadata is enormous, and Pinecone's cap is not. A single Textract page yields hundreds of blocks, each with bounding boxes, polygons, confidences, Ids, and relationship arrays. Attaching source blocks to each vector — a tempting way to keep provenance — exceeds the ~40 KB per-vector metadata limit almost immediately, and the upsert fails. The usual retreat is to strip metadata entirely, which throws away provenance and filtering in one move.
- Reading order is scrambled before embedding. Position-sorted LINE text interleaves multi-column pages into prose no human wrote; the embedding encodes the garble and Pinecone faithfully indexes it.
- Hierarchy and page numbers are melted away, so retrieved fragments arrive without lineage, and no metadata filter can scope by page because the page was lost at the join.
- Overlap pollutes top-k. Splitter overlap embeds boundary spans twice, so near-duplicate vectors crowd out the passage that actually answers the question.
POMA resolves the tension: the heavy Block graph is consumed once, upstream, and what reaches Pinecone is exactly what belongs there — one embedded chunkset per vector, plus four compact metadata fields. A Textract payload without LAYOUT blocks fails transparently with a 422 telling you to re-run with FeatureTypes including "LAYOUT", rather than silently indexing scrambled text.
The pipeline, end to end
pip install boto3 pinecone sentence-transformers pomaimport json
import os
import boto3
from pinecone import Pinecone, ServerlessSpec
from poma import PrimeCut
from sentence_transformers import SentenceTransformer
# 1. Textract — your existing call; LAYOUT is required, TABLES recommended.
textract = boto3.client("textract")
with open("contract.png", "rb") as f:
response = textract.analyze_document(
Document={"Bytes": f.read()},
FeatureTypes=["LAYOUT", "TABLES"],
)
with open("contract.textract.json", "w") as f:
json.dump(response, f)
# 2. The missing link — raw Blocks[] in, chunks + chunksets out.
poma = PrimeCut() # reads POMA_API_KEY
result = poma.ingest("contract.textract.json") # Textract shape auto-detected
# 3. Embed chunksets; upsert with compact metadata — never raw Blocks.
model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
pc = Pinecone(api_key=os.environ["PINECONE_API_KEY"])
if not pc.has_index("contracts"):
pc.create_index(
name="contracts",
dimension=384,
metric="cosine",
spec=ServerlessSpec(cloud="aws", region="us-east-1"),
)
index = pc.Index("contracts")
records, lookup = [], {}
for i, chunkset in enumerate(result.chunksets):
leaf = chunkset.chunks[-1] # deepest chunk in the root-to-leaf path
vector_id = f"{leaf.file_id}:{i}"
lookup[vector_id] = chunkset
records.append({
"id": vector_id,
"values": model.encode(chunkset.to_embed).tolist(),
"metadata": {
"file_id": leaf.file_id,
"page": leaf.page,
"depth": leaf.depth,
"chunk_index": leaf.chunk_index,
},
})
index.upsert(vectors=records, namespace="acme-corp") # one namespace per tenant
# 4. Query, then map IDs back to full chunksets for context assembly.
question = "Which termination clauses require written notice?"
hits = index.query(
vector=model.encode(question).tolist(),
top_k=5,
namespace="acme-corp",
filter={"file_id": {"$eq": leaf.file_id}},
include_metadata=True,
)
retrieved = [lookup[m["id"]] for m in hits["matches"]]
context = "\n\n".join(dict.fromkeys(cs.to_embed for cs in retrieved))In production, replace the in-memory lookup with your document store or the portable .poma archive keyed by vector ID — the point is that full chunkset text lives outside the index, and Pinecone metadata stays compact. Upstream, POMA validates the payload shape (a corrupted or mislabeled upload 422s immediately), splices TABLE blocks into LAYOUT_TABLE regions by geometry, drops header/footer/page-number furniture, keeps checkbox state, and counts byte-less LAYOUT_FIGURE regions as offloaded content in content_metadata. To force or suppress detection, set external_ocr_source to "textract" or "none". Pinecone's sparse-dense hybrid support slots in the same way: add a sparse encoding of to_embed alongside the dense values, and exact-token queries (clause numbers, form labels) stop depending on the embedding alone.
Measured on our reference legal-document benchmark, this pipeline answers the same query with 337 tokens of retrieved context instead of 1,542 for a recursive character splitter, with zero information loss. Methodology: Document Ingestion & Chunking for RAG.
Metadata mapping: POMA fields → Pinecone primitives
| POMA chunk field | Pinecone primitive | What it enables |
|---|---|---|
to_embed | dense vector values (+ optional sparse values for hybrid) | paraphrase + exact-term retrieval |
file_id | metadata field, filter with $eq/$in | scope queries to one document |
page | metadata field, filter with $gte/$lte | page-cited answers, page-range filters |
depth | metadata field, filter with $eq/$lte | hierarchy-aware retrieval |
chunk_index | metadata field | stable ordering at assembly time |
| tenant / corpus boundary | namespace | physical partitioning, clean per-tenant deletes |
| full chunkset text | not in Pinecone — doc store / .poma archive keyed by vector ID | context assembly under the ~40 KB metadata cap |
Frequently asked questions
How do I get AWS Textract results into Pinecone for RAG?
Call AnalyzeDocument with FeatureTypes including LAYOUT (add TABLES for grids), save the raw JSON, and run PrimeCut().ingest() on it — auto-detected, LAYOUT reading order followed, hierarchy rebuilt. Embed each chunkset's to_embed and upsert to a serverless index with compact file_id/page/depth/chunk_index metadata, keeping full text in your doc store or the .poma archive.
Why does Pinecone reject my Textract vectors with a metadata size error?
Raw Textract Block metadata is enormous — geometry, polygons, confidences, Ids, Relationships — and a page produces hundreds of blocks. Attaching source blocks to vectors blows the ~40 KB per-vector cap almost immediately. Chunk first, store only compact retrieval fields, and keep provenance in the archive.
What Pinecone metadata should Textract chunks carry?
file_id, page, depth, chunk_index — all filterable with $eq/$in/$gte. Optionally a trimmed preview, never the full text or source blocks.
How should I organize Textract documents across Pinecone namespaces?
One namespace per tenant or corpus: physical partitioning, no cross-tenant scans, clean deletes. Use metadata filters (file_id, page ranges, depth) for finer scoping within a namespace.
Where does the full chunkset text live if not in Pinecone metadata?
In your document store or the portable .poma archive, keyed by vector ID. Pinecone returns IDs and scores; you map them back to full chunksets and assemble a deduplicated context block — complete root-to-leaf context for the LLM, compact metadata for the index.
Related recipes
Same parser, different store: Textract → Qdrant · Textract → Weaviate · Textract → Milvus
Same store, different parser: Mistral OCR → Pinecone · LlamaParse → Pinecone · Azure Document Intelligence → Pinecone
Foundations: The Optimal Chunker for AWS Textract · The Optimal Chunks for Pinecone · All pipeline recipes · RAG chunking guide