Source: http://www.poma-ai.com/docs/pipelines/marker-to-pinecone

# The Missing Link Between Marker and Optimal Retrieval in Pinecone

<ByAuthor />

**The short answer:** Marker gives you fast local PDF parsing with real recovered structure; Pinecone gives you serverless vector search with namespaces and metadata filtering. Wired together naively, the pipeline drops Marker's figures, loses its page numbers, and — the pair-specific trap — blows Pinecone's ~40 KB per-vector metadata cap on the first big HTML table. The missing link is POMA: `PrimeCut().ingest()` consumes the saved Marker JSON (auto-detected), splices the side-channel images back in, and emits hierarchy-preserving [chunksets](/learn/chunking/chunksets) whose compact `file_id`/`page`/`depth` fields map cleanly onto Pinecone metadata, with the full text living in the `.poma` archive instead of the index.

## What each end of the pipeline actually provides

**Marker** (by Datalab / VikParuchuri) runs on your own hardware and produces structured output: with the JSON renderer (`--output_format json`), a Document block tree with Page children, per-block HTML, and full `<table>` elements with row and column spans intact. What it doesn't provide: cross-page hierarchy, retrieval units, or inline images — figure bytes sit in a side-channel `images` dict with only `![](name)` references in the text. Details: [The Optimal Chunker for Marker](/optimal-chunker-marker).

**Pinecone** provides serverless indexes, namespaces as hard partitions (a query runs inside exactly one), metadata filtering with `$eq`/`$in`/`$gte`-style operators, and sparse-dense hybrid support. What it doesn't provide: storage for your full documents — metadata is capped at roughly 40 KB per vector, which is a filtering budget, not a document store. Details: [The Optimal Chunks for the Best Retrieval in Pinecone](/optimal-chunks-pinecone).

## The naive wiring, and where it breaks

The common recipe — split `rendered.markdown` with a character splitter, embed, upsert each fragment with its full text stuffed into metadata — breaks in a way specific to this pair:

- **Marker's tables detonate the metadata cap.** Marker preserves tables as full HTML with row and column spans — which is precisely why a wide contract table or a spliced data-URI figure pushes a single vector's metadata past Pinecone's ~40 KB limit. The upsert batch is rejected, usually deep into ingestion, and the "fix" most pipelines reach for — truncating the text — silently amputates the very tables Marker worked to preserve.
- **Every figure vanishes.** Markdown-only pipelines embed dangling `![](name)` references while the bytes stay behind in the `images` dict; the resulting index has no figures to retrieve.
- **Page numbers never become metadata.** Marker's markdown output has no page boundaries, so `{"page": {"$gte": ...}}` filters are impossible and no answer can cite a page.
- **Overlap pollutes top-k.** Splitter overlap embeds boundary spans twice; Pinecone then returns near-duplicate hits that crowd out the passage that actually answers the question.

## The pipeline, end to end

The division of labor that respects both tools: Pinecone stores vectors and *compact* metadata; the `.poma` archive (or your document store) keeps the full chunk and chunkset text for assembly after retrieval.

```bash
pip install poma pinecone sentence-transformers

# Your existing Marker run — unchanged. The JSON renderer keeps the
# Document block tree that POMA auto-detects, pages included.
marker_single contract.pdf --output_format json --output_dir out/
```

```python
import os
from pinecone import Pinecone
from poma import PrimeCut
from sentence_transformers import SentenceTransformer

# 1. The missing link — raw Marker JSON in, chunks + chunksets out.
#    Side-channel image bytes are spliced into their ![](name) refs
#    and described into searchable text before chunking.
poma = PrimeCut()  # reads POMA_API_KEY
result = poma.ingest("out/contract/contract.json")  # block tree auto-detected

# 2. Embed chunksets; upsert with compact metadata only.
model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
pc = Pinecone(api_key=os.environ["PINECONE_API_KEY"])
index = pc.Index("contracts")

vectors = []
for i, cs in enumerate(result.chunksets):
    vectors.append({
        "id": f"{cs.file_id}:{i}",
        "values": model.encode(cs.to_embed).tolist(),
        "metadata": {  # scalars only — full text stays in the .poma archive
            "file_id": cs.file_id,
            "page": cs.page,
            "depth": cs.depth,
            "chunkset_index": i,
        },
    })
index.upsert(vectors=vectors, namespace="acme-corp")  # namespace = tenant

# 3. Retrieve, then assemble cheatsheets from the returned IDs.
hits = index.query(
    vector=model.encode("What are the early termination conditions?").tolist(),
    top_k=5,
    namespace="acme-corp",
    filter={"file_id": {"$eq": result.chunksets[0].file_id}},
    include_metadata=True,
)
retrieved = [int(m["metadata"]["chunkset_index"]) for m in hits["matches"]]
# Look up those chunksets in the .poma archive and merge them into one
# deduplicated, prompt-ready cheatsheet — hierarchy included.
```

POMA validates the payload up front — the JSON Document block tree is the strict auto-route fingerprint; a corrupted or mislabeled upload 422s immediately. Bare markdown shapes (`{markdown, images, metadata}`, the deprecated `{output, format}` envelope) need an explicit `external_ocr_source="marker"`. Because chunksets are overlap-free and cheatsheets deduplicate at assembly time, retrieved context stays lean: on our reference legal document, **337 tokens** versus **1,542** for a recursive character splitter, with zero information loss — methodology in [Document Ingestion & Chunking for RAG](/document-ingestion-chunking-rag).

## Metadata mapping: POMA fields → Pinecone primitives

| POMA chunk field | Pinecone primitive | What it enables |
| --- | --- | --- |
| `to_embed` | dense vector `values` (+ optional sparse values for hybrid) | paraphrase + exact-term retrieval |
| `file_id` | metadata field, `{"file_id": {"$eq": ...}}` | scope queries to one document |
| `page` (from Marker's Page blocks, JSON renderer) | numeric metadata field, `$gte`/`$lte` | page-cited answers, page-range filters |
| `depth` | numeric metadata field | filter/re-rank by hierarchy level |
| `chunk_index` / chunkset index | metadata field + vector ID | ID-based lookup in the `.poma` archive |
| tenant / corpus | **namespace** | hard isolation; one query per namespace |
| full chunkset text | **not stored in Pinecone** — `.poma` archive / doc store | stays under the ~40 KB metadata cap |

## Frequently asked questions

### How do I get Marker output into Pinecone for RAG?

Run `marker_single --output_format json`, hand the saved result to `PrimeCut().ingest()` (auto-detected, images spliced, hierarchy rebuilt), embed each chunkset's `to_embed` text, and upsert with compact `file_id`/`page`/`depth`/`chunk_index` metadata. Retrieve from the namespace, then assemble cheatsheets from the returned IDs.

### Why does my Pinecone upsert fail with metadata size errors on Marker output?

Pinecone caps metadata at roughly 40 KB per vector, and Marker's full HTML tables (row and column spans included) or spliced data-URI figures overflow it if you stuff whole chunk text into metadata. Store only scalar metadata in Pinecone; keep the full text in the `.poma` archive or your document store, keyed by vector ID.

### Should each Marker document get its own Pinecone namespace?

No — a query runs inside exactly one namespace, so per-document namespaces break corpus-wide questions. Use one namespace per tenant or corpus and scope to documents with a `file_id` metadata filter.

### How do I keep Marker page numbers filterable in Pinecone?

Use the JSON renderer (markdown output has no page boundaries) and a chunker that reads the Page blocks. POMA emits `page` per chunk; written as a numeric metadata field, it supports `{"page": {"$gte": 30, "$lte": 45}}` filters and page-cited answers.

### Does sparse-dense hybrid search in Pinecone help with Marker-parsed documents?

Yes. Locally parsed contracts and manuals are full of exact identifiers that dense embeddings blur; attach a sparse representation of `to_embed` alongside the dense vector and both paraphrase and exact-term queries hit. Overlap-free chunksets keep top-k free of near-duplicates either way.

## Related recipes

Same parser, different store: [Marker → Qdrant](/pipelines/marker-to-qdrant) · [Marker → Weaviate](/pipelines/marker-to-weaviate) · [Marker → Milvus](/pipelines/marker-to-milvus)

Same store, different parser: [Mistral OCR → Pinecone](/pipelines/mistral-ocr-to-pinecone) · [Textract → Pinecone](/pipelines/textract-to-pinecone) · [LlamaParse → Pinecone](/pipelines/llamaparse-to-pinecone)

Foundations: [The Optimal Chunker for Marker](/optimal-chunker-marker) · [The Optimal Chunks for Pinecone](/optimal-chunks-pinecone) · [All pipeline recipes](/pipelines/) · [RAG chunking guide](/guides/rag-chunking/)