Source: http://www.poma-ai.com/docs/optimal-chunks-pinecone

# The Optimal Chunks for the Best Retrieval in Pinecone

<ByAuthor />

**The short answer:** the optimal chunks for Pinecone are **chunksets** — self-explanatory units that carry their full heading lineage — embedded from their `to_embed` text and upserted with **compact hierarchy metadata** (`file_id`, `page`, `depth`), comfortably under Pinecone's ~40 KB per-vector metadata limit. Use one **namespace per tenant or corpus**, keep the full chunkset text in your document store or the `.poma` archive, and rebuild retrieved IDs into prompt-ready cheatsheets after the query. Pinecone's serverless search is excellent; what decides your retrieval quality is what you put *into* it.

This page covers what Pinecone does brilliantly, what it cannot do for you, and the record/metadata design that gets the most out of it.

## Pinecone gives you serverless scale — over whatever you feed it

Pinecone's primitives are built for running RAG in production without operating anything:

- **Serverless indexes** — storage and compute scale independently; you pay for what you use, not for idle pods.
- **Namespaces** — physical partitions within an index, a natural fit for one-namespace-per-tenant or per-corpus isolation.
- **Metadata filtering** — `$eq`, `$in`, `$gte` and friends scope queries to a document, a page range, or a hierarchy depth at query time.
- **Sparse-dense vectors** — a term-weight sparse vector on the same record as the dense embedding, for hybrid retrieval in one query.
- **Integrated inference** — hosted embedding and reranking models, so you can send text and let Pinecone produce the vectors.

None of these primitives, however, know anything about your documents. Pinecone retrieves the nearest neighbors of what you embedded. If what you embedded is a context-free fragment — a paragraph cut loose from the section that gives it meaning — Pinecone will faithfully retrieve context-free fragments, at any scale, with excellent latency. **The ceiling on retrieval quality is set before the first vector is upserted.**

## What "optimal chunks" means for Pinecone, concretely

Four properties separate a Pinecone index that answers questions from one that returns trivia:

1. **Every record is self-explanatory.** A chunkset is a root-to-leaf path through the document — `Master Services Agreement → Termination clauses → Early termination → "Either party may…"`. Retrieved alone, it still tells the LLM (and the human reading citations) exactly where it sits. A bare 512-token window does not. See [POMA chunksets](/learn/chunking/chunksets).
2. **No overlap, no near-duplicates.** Overlap is a patch for splitters that cut mid-thought — and it embeds every overlapped span twice. In a serverless index you pay for those duplicate records twice, and at query time near-identical hits crowd out diverse results. Chunksets carry context structurally, so overlap is simply unnecessary and your `top_k=5` returns five *different* candidates.
3. **Compact metadata, text elsewhere.** Pinecone caps metadata at roughly 40 KB per vector — a constraint, but a productive one. Map POMA's hierarchy fields (`file_id`, `page`, `depth`, `chunk_index`) into metadata for filtering, and keep the full chunkset text in your document store or the portable `.poma` archive, keyed by vector ID. You get document-scoped filters and page citations without ever brushing the cap.
4. **Namespaces for isolation, filters for scoping.** One namespace per tenant or corpus keeps queries physically partitioned; `filter={"file_id": {"$eq": ...}}` narrows to a single document inside it. Two mechanisms, two jobs — don't emulate one with the other.

## The pattern: embed `to_embed`, keep the text in your doc store

POMA has no first-party Pinecone helper — none is needed. The pattern is three steps with the genuine Pinecone Python client:

```bash
pip install poma pinecone
```

```python
import os

from pinecone import Pinecone
from poma import PrimeCut

# 1. Chunk any document — PDF, DOCX, HTML, or a bring-your-own-OCR result JSON.
poma = PrimeCut()  # reads POMA_API_KEY
result = poma.ingest("contract.pdf")

# 2. Embed each chunkset's to_embed text and upsert with compact metadata.
pc = Pinecone(api_key=os.environ["PINECONE_API_KEY"])
index = pc.Index("contracts")

records = []
for cs in result.chunksets:
    records.append({
        "id": f"{cs.file_id}:{cs.chunk_index}",
        "values": embed(cs.to_embed),  # your embedding model
        "metadata": {
            "file_id": cs.file_id,
            "page": cs.page,
            "depth": cs.depth,
            "chunk_index": cs.chunk_index,
        },
    })
index.upsert(vectors=records, namespace="acme-corp")

# 3. Query with a filter, then rebuild cheatsheets from the returned IDs.
matches = index.query(
    vector=embed("What does the contract say about early termination?"),
    top_k=5,
    namespace="acme-corp",
    filter={"file_id": {"$eq": result.chunksets[0].file_id}},
    include_metadata=True,
)
hit_ids = [m["id"] for m in matches["matches"]]
# Look up the full chunksets by ID in your doc store or the .poma archive,
# deduplicate the shared heading lineage, and assemble one context block.
```

The record IDs are the join key: `{file_id}:{chunk_index}` points straight back to the chunkset in the `.poma` archive or whatever document store you keep. That lookup-then-merge step — deduplicating the ancestors that retrieved chunksets share — is what POMA calls a **cheatsheet**, and it is where the token savings land. On our reference legal-document benchmark that meant **337 tokens** of context versus **1,542** for a recursive-splitter baseline, with zero information loss — [methodology here](/document-ingestion-chunking-rag).

If you use Pinecone's integrated inference, the same design holds: send `chunkset.to_embed` as the text field and let the hosted model embed it — the unit you embed matters more than where the embedding runs.

## Chunk shapes in Pinecone, compared

| Property | Fixed 512-token chunks + overlap | Semantic chunks | **POMA chunksets** |
| --- | --- | --- | --- |
| Record is self-explanatory when retrieved alone | ✗ | Sometimes | **✓ (by construction)** |
| Near-duplicate vectors in the index | Many (overlap) | Few | **None** |
| Filterable hierarchy metadata (`file_id`, `page`, `depth`) | Manual | Manual | **✓ (mapped from chunk fields)** |
| Stays clear of the ~40 KB metadata cap | Risky (text stuffed in metadata) | Risky | **✓ (compact fields; text in doc store / `.poma`)** |
| Tables survive embedding | ✗ (cut mid-row) | Fragile | **✓ (HTML, never cut)** |
| Sparse-dense hybrid ready | DIY on fragments | DIY | **✓ (dense + sparse from one `to_embed` string)** |
| Post-retrieval assembly | Concatenate raw hits | Concatenate raw hits | **Cheatsheets (rebuilt from retrieved IDs)** |

## Frequently asked questions

### What is the optimal chunk size for Pinecone?

There is no universal token count — the optimal unit is a chunk that is self-explanatory without its neighbors. Chunksets achieve that at any size, which is why they beat fixed 512-token windows: every vector can be retrieved alone and still be understood. If you must fix a size, start at 512 tokens — but the size knob cannot fix missing context.

### What metadata should I store with Pinecone vectors for RAG?

Compact hierarchy fields: `file_id`, `page`, `depth`, `chunk_index`. They power Pinecone's filters (`$eq`, `$in`, `$gte`) for per-document scoping, page citations, and depth-aware exclusion. Keep metadata small — the cap is roughly 40 KB per vector — and keep full chunkset text in your doc store or the `.poma` archive.

### Should I store the full chunk text in Pinecone metadata?

Usually not. The ~40 KB cap makes full lineage text a liability for deep documents. Store the ID plus compact fields in Pinecone; keep the authoritative text outside, keyed by vector ID. A short snippet for debugging is fine.

### How should I use Pinecone namespaces for RAG?

One namespace per tenant or corpus — physical partitioning, so queries never touch other tenants' vectors. Inside a namespace, metadata filters (`file_id`, `page`, `depth`) do the fine-grained scoping. Many documents per namespace; filters select the document.

### Does Pinecone support hybrid search for RAG chunks?

Yes — sparse-dense vectors put a term-weight sparse vector on the same record as the dense embedding, catching exact tokens (statute numbers, SKUs, defined terms) that dense embeddings blur. Compute both from the same `to_embed` string so both signals describe the identical unit.

### How do I turn Pinecone matches into prompt-ready context?

Merge, don't concatenate: look up the full chunksets by their retrieved IDs in your doc store or `.poma` archive, deduplicate the shared ancestor lineage, and assemble one coherent block — a cheatsheet.

## Retrieval with vektoria's assemble()

POMA's `poma.vektoria` package keeps the index content-free: Pinecone stores only `(id, vector, {file_id, chunkset_index})`, while the actual chunk content lives on a **volume** (a path, `s3://`, or `gs://` URL). After your normal Pinecone query, hand the raw result to `assemble()`:

```python
from poma.vektoria import assemble

resp = index.query(vector=qv, top_k=10, include_metadata=True)  # include_metadata is required
context = assemble(resp, volume="s3://your-bucket/poma")  # -> [{"file_id", "content"}, ...]
```

`assemble()` auto-detects Pinecone's result shape (one of 12 supported stores) and returns deduplicated cheatsheets — no reconstruction code on your side. The one requirement: `include_metadata=True` on the query, or Pinecone won't return `file_id`/`chunkset_index` for `assemble()` to work with.

## Feed Pinecone from the parser you already run

Pair this page with your ingestion side — the same chunksets, straight from your OCR/parsing tool's raw output:

- [Mistral OCR → Pinecone](/pipelines/mistral-ocr-to-pinecone)
- [LlamaParse → Pinecone](/pipelines/llamaparse-to-pinecone)
- [Azure Document Intelligence → Pinecone](/pipelines/azure-document-intelligence-to-pinecone)
- [Unstructured.io → Pinecone](/pipelines/unstructured-to-pinecone)
- [Docling → Pinecone](/pipelines/docling-to-pinecone)
- [Marker → Pinecone](/pipelines/marker-to-pinecone)
- [AWS Textract → Pinecone](/pipelines/textract-to-pinecone)

Running a different store? The same chunk design applies: [Qdrant](/optimal-chunks-qdrant) · [Weaviate](/optimal-chunks-weaviate) · [pgvector](/optimal-chunks-pgvector) · [Milvus](/optimal-chunks-milvus) · [Chroma](/optimal-chunks-chroma).

Fundamentals: [RAG chunking guide](/guides/rag-chunking/) · [RAG architecture guide](/guides/rag-architecture/) · [POMA chunksets](/learn/chunking/chunksets).