Unstructured.io to Pinecone: Chunking for RAG
The short answer: Unstructured.io gives you excellent typed elements; Pinecone gives you excellent serverless vector search with namespaces and metadata filtering. Wired together naively — chunk_by_title, dump element metadata, upsert — the pipeline hits Pinecone's ~40 KB metadata cap and still retrieves context-free fragments. The missing link is POMA: PrimeCut().ingest() consumes the raw elements_to_json output (auto-detected), emits hierarchy-preserving chunksets, and you upsert their to_embed vectors with compact file_id/page/depth metadata that filters cleanly and never approaches the cap.
What each end of the pipeline actually provides
Unstructured.io (partition_pdf or the hosted API) returns a flat, ordered list of typed elements — Title, NarrativeText, ListItem, Table, Image — each with a stable element_id and a metadata dict carrying page_number, text_as_html for tables, and optionally image_base64. What it doesn't provide: hierarchy. The list is flat, and the built-in by_title and basic chunkers emit fragments that know their nearest Title at best. Details: The Optimal Chunker for Unstructured.io.
Pinecone provides serverless indexes, namespaces for hard partitioning, metadata filtering with $eq/$in/$gte-style operators, and sparse-dense hybrid support — with a metadata budget of roughly 40 KB per vector. What it doesn't provide: any opinion about what a vector should represent. It returns the nearest IDs to whatever you embedded. Details: The Optimal Chunks for the Best Retrieval in Pinecone.
The naive wiring, and where it breaks
The common recipe — chunk_by_title(elements) → embed each fragment → upsert with element.metadata.to_dict() attached — breaks in a pair-specific way:
- Element metadata dumps blow the ~40 KB cap. Unstructured's per-element metadata is generous: coordinates, emphasized-text spans,
text_as_html, sometimes inlineimage_base64. Attach it wholesale and a single large table's HTML — or one base64 image — exceeds the metadata budget on its own. Upserts fail, or you start truncating fields ad hoc and lose track of what survived. - What fits is still the wrong thing. Even a trimmed metadata dict describes the element — its box on the page — not the fragment's place in the document. No ancestor path exists in a flat element list, so no metadata mapping can add one.
- Namespaces and filters go unused. Everything lands in the default namespace with unfilterable blob metadata, so per-corpus isolation and per-document scoping — Pinecone's actual strengths — never engage.
The fix is to decide before the upsert what a vector represents (a self-explanatory chunkset) and what metadata earns its bytes (compact scalars you filter on).
The pipeline, end to end
pip install unstructured pinecone sentence-transformers pomaimport os
from unstructured.partition.pdf import partition_pdf
from unstructured.staging.base import elements_to_json
from pinecone import Pinecone
from sentence_transformers import SentenceTransformer
from poma import PrimeCut
from poma.utils import unpack_poma_archive
# 1. Your existing Unstructured call — unchanged.
elements = partition_pdf(
filename="contract.pdf",
strategy="hi_res",
infer_table_structure=True, # populates metadata.text_as_html
extract_image_block_types=["Image"],
extract_image_block_to_payload=True, # inline base64 → POMA can describe figures
)
elements_to_json(elements, filename="contract.unstructured.json")
# 2. The missing link — raw element list in, chunks + chunksets out.
poma = PrimeCut() # reads POMA_API_KEY
result = poma.ingest( # Unstructured shape auto-detected
"contract.unstructured.json",
download_dir="store",
filename="contract.poma", # keep the archive — page numbers live in it
)
# Depth comes off the SDK objects; page is on the archive's chunk records only.
depth_of = {c.chunk_index: c.depth for c in result.chunks}
page_of = {
c["chunk_index"]: c.get("page")
for c in unpack_poma_archive(poma_archive_path="store/contract.poma")["chunks"]
}
# 3. Pinecone — chunkset vectors with compact, filterable metadata.
model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
pc = Pinecone(api_key=os.environ["PINECONE_API_KEY"])
index = pc.Index("contracts")
vectors = []
for cs in result.chunksets:
leaf = cs.chunks[-1] # document order — the last chunk index is the leaf
vectors.append({
"id": f"{cs.file_id}:{cs.chunkset_index}",
"values": model.encode(cs.to_embed).tolist(),
"metadata": {
"file_id": cs.file_id,
"chunkset_index": cs.chunkset_index,
"page": page_of[leaf],
"depth": depth_of[leaf],
},
})
index.upsert(vectors=vectors, namespace="contracts")
# 4. Query the namespace, then assemble context from the .poma archive by ID.
hits = index.query(
vector=model.encode("What are the early termination conditions?").tolist(),
top_k=5,
namespace="contracts",
filter={"file_id": {"$eq": result.chunksets[0].file_id}},
include_metadata=True,
)
retrieved_ids = [m["id"] for m in hits["matches"]] # → look up full chunksets, build cheatsheetPOMA validates the payload shape up front (a corrupted or mislabeled upload 422s immediately), splices text_as_html for tables, describes inline images, drops Header/Footer/PageNumber elements as page furniture, and rebuilds the cross-page heading tree before chunking. To force or suppress detection, set external_ocr_source to "unstructured" or "none". The final step — deduplicating and merging the retrieved chunksets into one prompt-ready cheatsheet — runs client-side from the .poma archive or your document store.
On our reference legal-document benchmark, this pipeline answers the same query with 337 tokens of retrieved context instead of 1,542 for a recursive character splitter, with zero information loss — methodology in Document Ingestion & Chunking for RAG.
Metadata mapping: POMA fields → Pinecone primitives
| POMA chunk field | Pinecone primitive | What it enables |
|---|---|---|
to_embed | dense vector values (+ optional sparse values for hybrid) | paraphrase + exact-term retrieval |
file_id | metadata field, filtered with $eq/$in | scope queries to one document |
page of the chunkset's leaf chunk (from the archive's chunk records, filled from metadata.page_number) | metadata field, filtered with $gte/$lte | page-cited answers, page-range filters |
depth of the leaf chunk | metadata field | filter/re-rank by hierarchy level |
chunkset_index | encoded in the vector ID ({file_id}:{chunkset_index}) | stable ordering at assembly time |
| corpus / tenant | namespace | hard isolation, smaller search space |
The chunkset text deliberately stays out of metadata: the ~40 KB budget is for filterable scalars, and the full retrieval unit lives in the .poma archive, addressable by ID.
Frequently asked questions
How do I get Unstructured.io elements into Pinecone for RAG?
Save the element list with elements_to_json, run PrimeCut().ingest() on it (auto-detected, hierarchy rebuilt), embed each chunkset's to_embed, and upsert to a namespace with file_id/chunkset_index/page/depth metadata — page and depth taken from the chunkset's leaf chunk, with page read out of the .poma archive's chunk records. Retrieve IDs, then assemble cheatsheets from the .poma archive.
Why does Unstructured element metadata blow Pinecone's metadata limit?
Pinecone caps metadata at roughly 40 KB per vector, and Unstructured's per-element dict — coordinates, text_as_html, emphasized spans, sometimes image_base64 — dumps easily past it: one large table's HTML or one base64 image can exceed the cap alone. Store compact scalars only.
What Pinecone metadata should Unstructured-derived chunks carry?
file_id, chunkset_index (also encoded in the ID), plus the leaf chunk's page and depth — small scalars that $eq/$in/$gte filters handle efficiently. depth is on the SDK's chunk objects; page is on the archive's chunk records, so read it back with unpack_poma_archive. Keep the chunkset text out of metadata; look the full unit up by ID at assembly time.
How should I use Pinecone namespaces with Unstructured documents?
One namespace per tenant or corpus: queries run against exactly one namespace, so you get hard isolation and a smaller search space for free. Keep file_id metadata for per-document scoping inside the namespace.
Does Pinecone sparse-dense hybrid help with Unstructured content?
Yes — parsed documents are full of exact tokens (clause numbers, part codes, defined terms) that dense embeddings blur. Hybrid works best on clean units: POMA's overlap-free chunksets avoid the near-duplicate vectors that overlap-based splitters push into top-k.
Related recipes
Same parser, different store: Unstructured → Qdrant · Unstructured → Weaviate · Unstructured → Milvus
Same store, different parser: Mistral OCR → Pinecone · Textract → Pinecone · Azure Document Intelligence → Pinecone
Foundations: The Optimal Chunker for Unstructured.io · The Optimal Chunks for Pinecone · All pipeline recipes · RAG chunking guide