Source: http://www.poma-ai.com/docs/pipelines/unstructured-to-pinecone

# The Missing Link Between Unstructured.io and Optimal Retrieval in Pinecone

<ByAuthor />

**The short answer:** Unstructured.io gives you excellent typed elements; Pinecone gives you excellent serverless vector search with namespaces and metadata filtering. Wired together naively — `chunk_by_title`, dump element metadata, upsert — the pipeline hits Pinecone's ~40 KB metadata cap and still retrieves context-free fragments. The missing link is POMA: `PrimeCut().ingest()` consumes the raw `elements_to_json` output (auto-detected), emits hierarchy-preserving [chunksets](/learn/chunking/chunksets), and you upsert their `to_embed` vectors with compact `file_id`/`page`/`depth` metadata that filters cleanly and never approaches the cap.

## What each end of the pipeline actually provides

**Unstructured.io** (`partition_pdf` or the hosted API) returns a flat, ordered list of typed elements — `Title`, `NarrativeText`, `ListItem`, `Table`, `Image` — each with a stable `element_id` and a metadata dict carrying `page_number`, `text_as_html` for tables, and optionally `image_base64`. What it doesn't provide: hierarchy. The list is flat, and the built-in `by_title` and `basic` chunkers emit fragments that know their nearest `Title` at best. Details: [The Optimal Chunker for Unstructured.io](/optimal-chunker-unstructured).

**Pinecone** provides serverless indexes, namespaces for hard partitioning, metadata filtering with `$eq`/`$in`/`$gte`-style operators, and sparse-dense hybrid support — with a metadata budget of roughly 40 KB per vector. What it doesn't provide: any opinion about what a vector should represent. It returns the nearest IDs to whatever you embedded. Details: [The Optimal Chunks for the Best Retrieval in Pinecone](/optimal-chunks-pinecone).

## The naive wiring, and where it breaks

The common recipe — `chunk_by_title(elements)` → embed each fragment → upsert with `element.metadata.to_dict()` attached — breaks in a pair-specific way:

- **Element metadata dumps blow the ~40 KB cap.** Unstructured's per-element metadata is generous: coordinates, emphasized-text spans, `text_as_html`, sometimes inline `image_base64`. Attach it wholesale and a single large table's HTML — or one base64 image — exceeds the metadata budget on its own. Upserts fail, or you start truncating fields ad hoc and lose track of what survived.
- **What fits is still the wrong thing.** Even a trimmed metadata dict describes the *element* — its box on the page — not the fragment's place in the document. No ancestor path exists in a flat element list, so no metadata mapping can add one.
- **Namespaces and filters go unused.** Everything lands in the default namespace with unfilterable blob metadata, so per-corpus isolation and per-document scoping — Pinecone's actual strengths — never engage.

The fix is to decide *before* the upsert what a vector represents (a self-explanatory chunkset) and what metadata earns its bytes (compact scalars you filter on).

## The pipeline, end to end

```bash
pip install unstructured pinecone sentence-transformers poma
```

```python
import os
from unstructured.partition.pdf import partition_pdf
from unstructured.staging.base import elements_to_json
from pinecone import Pinecone
from sentence_transformers import SentenceTransformer
from poma import PrimeCut

# 1. Your existing Unstructured call — unchanged.
elements = partition_pdf(
    filename="contract.pdf",
    strategy="hi_res",
    infer_table_structure=True,            # populates metadata.text_as_html
    extract_image_block_types=["Image"],
    extract_image_block_to_payload=True,   # inline base64 → POMA can describe figures
)
elements_to_json(elements, filename="contract.unstructured.json")

# 2. The missing link — raw element list in, chunks + chunksets out.
poma = PrimeCut()  # reads POMA_API_KEY
result = poma.ingest("contract.unstructured.json")  # Unstructured shape auto-detected

# 3. Pinecone — chunkset vectors with compact, filterable metadata.
model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
pc = Pinecone(api_key=os.environ["PINECONE_API_KEY"])
index = pc.Index("contracts")

vectors = []
for i, cs in enumerate(result.chunksets):
    vectors.append({
        "id": f"{cs.file_id}:{i}",
        "values": model.encode(cs.to_embed).tolist(),
        "metadata": {"file_id": cs.file_id, "page": cs.page, "depth": cs.depth},
    })
index.upsert(vectors=vectors, namespace="contracts")

# 4. Query the namespace, then assemble context from the .poma archive by ID.
hits = index.query(
    vector=model.encode("What are the early termination conditions?").tolist(),
    top_k=5,
    namespace="contracts",
    filter={"file_id": {"$eq": result.chunksets[0].file_id}},
    include_metadata=True,
)
retrieved_ids = [m["id"] for m in hits["matches"]]  # → look up full chunksets, build cheatsheet
```

POMA validates the payload shape up front (a corrupted or mislabeled upload 422s immediately), splices `text_as_html` for tables, describes inline images, drops `Header`/`Footer`/`PageNumber` elements as page furniture, and rebuilds the cross-page heading tree before chunking. To force or suppress detection, set `external_ocr_source` to `"unstructured"` or `"none"`. The final step — deduplicating and merging the retrieved chunksets into one prompt-ready cheatsheet — runs client-side from the `.poma` archive or your document store.

On our reference legal-document benchmark, this pipeline answers the same query with **337 tokens** of retrieved context instead of **1,542** for a recursive character splitter, with zero information loss — methodology in [Document Ingestion & Chunking for RAG](/document-ingestion-chunking-rag).

## Metadata mapping: POMA fields → Pinecone primitives

| POMA chunk field | Pinecone primitive | What it enables |
| --- | --- | --- |
| `to_embed` | dense vector values (+ optional sparse values for hybrid) | paraphrase + exact-term retrieval |
| `file_id` | metadata field, filtered with `$eq`/`$in` | scope queries to one document |
| `page` (from `metadata.page_number`) | metadata field, filtered with `$gte`/`$lte` | page-cited answers, page-range filters |
| `depth` | metadata field | filter/re-rank by hierarchy level |
| `chunk_index` | encoded in the vector ID | stable ordering at assembly time |
| corpus / tenant | **namespace** | hard isolation, smaller search space |

The chunkset *text* deliberately stays out of metadata: the ~40 KB budget is for filterable scalars, and the full retrieval unit lives in the `.poma` archive, addressable by ID.

## Frequently asked questions

### How do I get Unstructured.io elements into Pinecone for RAG?

Save the element list with `elements_to_json`, run `PrimeCut().ingest()` on it (auto-detected, hierarchy rebuilt), embed each chunkset's `to_embed`, and upsert to a namespace with `file_id`/`page`/`depth` metadata. Retrieve IDs, then assemble cheatsheets from the `.poma` archive.

### Why does Unstructured element metadata blow Pinecone's metadata limit?

Pinecone caps metadata at roughly 40 KB per vector, and Unstructured's per-element dict — coordinates, `text_as_html`, emphasized spans, sometimes `image_base64` — dumps easily past it: one large table's HTML or one base64 image can exceed the cap alone. Store compact scalars only.

### What Pinecone metadata should Unstructured-derived chunks carry?

`file_id`, `page`, `depth`, and `chunk_index` (in the ID) — small scalars that `$eq`/`$in`/`$gte` filters handle efficiently. Keep the chunkset text out of metadata; look the full unit up by ID at assembly time.

### How should I use Pinecone namespaces with Unstructured documents?

One namespace per tenant or corpus: queries run against exactly one namespace, so you get hard isolation and a smaller search space for free. Keep `file_id` metadata for per-document scoping inside the namespace.

### Does Pinecone sparse-dense hybrid help with Unstructured content?

Yes — parsed documents are full of exact tokens (clause numbers, part codes, defined terms) that dense embeddings blur. Hybrid works best on clean units: POMA's overlap-free chunksets avoid the near-duplicate vectors that overlap-based splitters push into top-k.

## Related recipes

Same parser, different store: [Unstructured → Qdrant](/pipelines/unstructured-to-qdrant) · [Unstructured → Weaviate](/pipelines/unstructured-to-weaviate) · [Unstructured → Milvus](/pipelines/unstructured-to-milvus)

Same store, different parser: [Mistral OCR → Pinecone](/pipelines/mistral-ocr-to-pinecone) · [Textract → Pinecone](/pipelines/textract-to-pinecone) · [Azure Document Intelligence → Pinecone](/pipelines/azure-document-intelligence-to-pinecone)

Foundations: [The Optimal Chunker for Unstructured.io](/optimal-chunker-unstructured) · [The Optimal Chunks for Pinecone](/optimal-chunks-pinecone) · [All pipeline recipes](/pipelines/) · [RAG chunking guide](/guides/rag-chunking/)