Source: http://www.poma-ai.com/docs/pipelines/docling-to-pinecone

# The Missing Link Between Docling and Optimal Retrieval in Pinecone

<ByAuthor />

**The short answer:** Docling hands you a typed DoclingDocument tree — explicit heading levels, cell-grid tables, page furniture already isolated. Pinecone hands you serverless vector search with namespaces, metadata filtering, and sparse-dense hybrid support. The naive wiring — `export_to_markdown()`, split, embed, upsert — wastes the first and misuses the second: structure is flattened away, and the usual "fix" of stuffing full text and heading paths into metadata collides with Pinecone's ~40 KB per-vector cap. The missing link is POMA: `PrimeCut().ingest()` parses the tree natively into [chunksets](/learn/chunking/chunksets), you embed each chunkset's `to_embed` text, and Pinecone stores compact `{file_id, page, depth}` metadata while the full text stays in the portable `.poma` archive.

## What each end of the pipeline actually provides

**Docling** (IBM's open-source converter, also served via docling-serve) returns the DoclingDocument: a typed `texts`/`tables`/`pictures`/`groups` tree where `section_header` items carry an explicit numeric `level`, tables are cell grids, and repeating page furniture sits pre-classified in the `furniture` group with `page_header`/`page_footer` labels. What it doesn't decide: what an embedding vector should represent. Details: [The Optimal Chunker for Docling](/optimal-chunker-docling).

**Pinecone** provides serverless indexes, namespaces as hard partitions (a natural fit for one namespace per tenant or corpus), metadata filtering with `$eq`/`$in`/`$gte` operators, and sparse-dense hybrid support. What it doesn't provide: any repair of what you embed. It also enforces a real constraint the naive pipeline trips over — a metadata budget of roughly 40 KB per vector. Details: [The Optimal Chunks for the Best Retrieval in Pinecone](/optimal-chunks-pinecone).

## The naive wiring, and where it breaks

The common recipe — `doc.export_to_markdown()` → `RecursiveCharacterTextSplitter` → embed → `index.upsert(...)` — breaks in a pair-specific way, in two stages:

1. **The flattening stage discards Docling's structure.** Explicit `section_header` levels are reduced to `#` glyphs the splitter ignores; the `furniture` group's page headers, footers, and page numbers pour back into the text stream and become per-page noise vectors; overlap embeds every boundary span twice, inflating serverless storage with near-duplicates.
2. **The repair attempt hits the metadata cap.** Teams notice the retrieved fragments arrive context-free, so they stuff the fix into Pinecone metadata: the full chunk text, the reconstructed heading path, sometimes a serialized table for good measure. On a deeply nested Docling section — a level-4 clause under three ancestor headings, next to an HTML table — that payload runs straight into Pinecone's ~40 KB per-vector metadata limit. Now the pipeline truncates the very context it was trying to preserve, silently, on exactly the documents where hierarchy matters most.

The fix is architectural, not cosmetic: make the retrieval unit self-explanatory *before* it reaches Pinecone, and store coordinates rather than content in metadata. That is what chunksets are — root-to-leaf units whose `to_embed` text already carries the ancestor breadcrumbs — so the metadata can stay three scalars small.

## The pipeline, end to end

```bash
pip install docling poma pinecone sentence-transformers
```

```python
import json
import os
from docling.document_converter import DocumentConverter
from pinecone import Pinecone, ServerlessSpec
from poma import PrimeCut
from sentence_transformers import SentenceTransformer

# 1. Docling conversion — your existing call, unchanged.
converter = DocumentConverter()
doc = converter.convert("contract.pdf").document
with open("contract.docling.json", "w") as f:
    json.dump(doc.export_to_dict(), f)

# 2. The missing link — DoclingDocument tree in, chunks + chunksets out.
poma = PrimeCut()  # reads POMA_API_KEY
result = poma.ingest("contract.docling.json")  # DoclingDocument shape auto-detected

# 3. Embed chunksets; upsert with compact metadata (well under the 40 KB cap).
model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
pc = Pinecone(api_key=os.environ["PINECONE_API_KEY"])
if not pc.has_index("contracts"):
    pc.create_index(
        name="contracts",
        dimension=384,
        metric="cosine",
        spec=ServerlessSpec(cloud="aws", region="us-east-1"),
    )
index = pc.Index("contracts")

index.upsert(
    vectors=[
        {
            "id": f"{cs.file_id}:{i}",
            "values": model.encode(cs.to_embed).tolist(),
            "metadata": {"file_id": cs.file_id, "page": cs.page, "depth": cs.depth},
        }
        for i, cs in enumerate(result.chunksets)
    ],
    namespace="legal",  # one namespace per tenant/corpus
)

# 4. Query the namespace, join IDs back to full chunkset text.
query = "What are the early termination conditions?"
hits = index.query(
    vector=model.encode(query).tolist(),
    top_k=5,
    namespace="legal",
    include_metadata=True,
)
retrieved = [result.chunksets[int(m["id"].split(":", 1)[1])] for m in hits["matches"]]
context = "\n\n".join(dict.fromkeys(cs.to_embed for cs in retrieved))  # dedupe = cheatsheet step
```

POMA validates the payload up front — auto-detection fingerprints `schema_name == "DoclingDocument"`, and a corrupted or mislabeled upload 422s immediately rather than silently degrading the index. A docling-serve envelope carrying only `md_content` falls back to a markdown passthrough. To force or suppress detection, set `external_ocr_source` to `"docling"` or `"none"`. In production, the ID-to-text join runs against the portable `.poma` archive or your document store instead of an in-memory result, and the final dedupe-and-merge is the cheatsheet assembly step.

The payoff is measurable: on our reference legal-document benchmark, chunksets plus cheatsheet assembly answered the same query with **337 tokens** of retrieved context instead of **1,542** for a recursive character splitter, with zero information loss — methodology in [Document Ingestion & Chunking for RAG](/document-ingestion-chunking-rag).

## Metadata mapping: POMA fields → Pinecone primitives

| POMA chunk field | Pinecone primitive | What it enables |
| --- | --- | --- |
| `to_embed` | dense vector `values` (+ optional sparse values for hybrid) | paraphrase + exact-term retrieval |
| `file_id` | metadata field, filter with `$eq`/`$in` | scope queries to one document |
| `page` (from Docling item provenance) | metadata field, filter with `$gte`/`$lte` | page-cited answers, page-range filters |
| `depth` (from explicit `section_header` levels) | metadata field | filter/re-rank by hierarchy level |
| `chunk_index` | encoded in the vector ID | stable ordering, ID-to-text join |
| chunkset full text | **not** metadata — `.poma` archive / doc store | stays clear of the ~40 KB cap |

Tenant or corpus boundaries map to **namespaces**; document boundaries map to the `file_id` filter within a namespace.

## Frequently asked questions

### How do I get Docling output into Pinecone for RAG?

Save `export_to_dict()` as JSON, run `PrimeCut().ingest()` on it (auto-detected, tree parsed natively), embed each chunkset's `to_embed`, and upsert with `{file_id, page, depth}` metadata into a namespace. Join returned IDs back to full text for the prompt.

### How do I keep Docling chunk metadata under Pinecone's 40 KB limit?

Store coordinates, not content: three compact scalars in metadata, full chunkset text in the `.poma` archive or your document store, vector IDs as the join key. Deep heading paths and HTML tables never touch the cap.

### Should each Docling document get its own Pinecone namespace?

Namespaces for tenants or corpora — they are hard partitions queries never cross. Documents are scoped inside a namespace with a `file_id` `$eq` filter, keeping cross-document search available.

### Does sparse-dense hybrid search in Pinecone help with Docling-parsed documents?

Yes — clause numbers, part codes, and defined terms are exact tokens dense embeddings blur. Attach a sparse representation alongside each dense vector; chunkset `to_embed` text includes heading breadcrumbs, so the sparse side matches section titles too.

### Why not export Docling to markdown and split it before upserting to Pinecone?

Flattening collapses explicit heading levels to glyphs, re-injects the pre-isolated `furniture` noise, and overlap multiplies near-duplicate vectors. The usual metadata "fix" then collides with the 40 KB cap. Parse the tree; embed chunksets.

## Related recipes

Same parser, different store: [Docling → Qdrant](/pipelines/docling-to-qdrant) · [Docling → Weaviate](/pipelines/docling-to-weaviate) · [Docling → Milvus](/pipelines/docling-to-milvus)

Same store, different parser: [Mistral OCR → Pinecone](/pipelines/mistral-ocr-to-pinecone) · [Unstructured → Pinecone](/pipelines/unstructured-to-pinecone) · [Textract → Pinecone](/pipelines/textract-to-pinecone)

Foundations: [The Optimal Chunker for Docling](/optimal-chunker-docling) · [The Optimal Chunks for Pinecone](/optimal-chunks-pinecone) · [All pipeline recipes](/pipelines/) · [RAG chunking guide](/guides/rag-chunking/)