Marker to Pinecone: Chunking for RAG
The short answer: Marker gives you fast local PDF parsing with real recovered structure; Pinecone gives you serverless vector search with namespaces and metadata filtering. Wired together naively, the pipeline drops Marker's figures, loses its page numbers, and — the pair-specific trap — blows Pinecone's ~40 KB per-vector metadata cap on the first big HTML table. The missing link is POMA: PrimeCut().ingest() consumes the saved Marker JSON (auto-detected), splices the side-channel images back in, and emits hierarchy-preserving chunksets whose compact file_id/page/depth fields map cleanly onto Pinecone metadata, with the full text living in the .poma archive instead of the index.
What each end of the pipeline actually provides
Marker (by Datalab / VikParuchuri) runs on your own hardware and produces structured output: with the JSON renderer (--output_format json), a Document block tree with Page children, per-block HTML, and full <table> elements with row and column spans intact. What it doesn't provide: cross-page hierarchy, retrieval units, or inline images — figure bytes sit in a side-channel images dict with only  references in the text. Details: The Optimal Chunker for Marker.
Pinecone provides serverless indexes, namespaces as hard partitions (a query runs inside exactly one), metadata filtering with $eq/$in/$gte-style operators, and sparse-dense hybrid support. What it doesn't provide: storage for your full documents — metadata is capped at roughly 40 KB per vector, which is a filtering budget, not a document store. Details: The Optimal Chunks for the Best Retrieval in Pinecone.
The naive wiring, and where it breaks
The common recipe — split rendered.markdown with a character splitter, embed, upsert each fragment with its full text stuffed into metadata — breaks in a way specific to this pair:
- Marker's tables detonate the metadata cap. Marker preserves tables as full HTML with row and column spans — which is precisely why a wide contract table or a spliced data-URI figure pushes a single vector's metadata past Pinecone's ~40 KB limit. The upsert batch is rejected, usually deep into ingestion, and the "fix" most pipelines reach for — truncating the text — silently amputates the very tables Marker worked to preserve.
- Every figure vanishes. Markdown-only pipelines embed dangling
references while the bytes stay behind in theimagesdict; the resulting index has no figures to retrieve. - Page numbers never become metadata. Marker's markdown output has no page boundaries, so
{"page": {"$gte": ...}}filters are impossible and no answer can cite a page. - Overlap pollutes top-k. Splitter overlap embeds boundary spans twice; Pinecone then returns near-duplicate hits that crowd out the passage that actually answers the question.
The pipeline, end to end
The division of labor that respects both tools: Pinecone stores vectors and compact metadata; the .poma archive (or your document store) keeps the full chunk and chunkset text for assembly after retrieval.
pip install poma pinecone sentence-transformers
# Your existing Marker run — unchanged. The JSON renderer keeps the
# Document block tree that POMA auto-detects, pages included.
marker_single contract.pdf --output_format json --output_dir out/import os
from pinecone import Pinecone
from poma import PrimeCut
from poma.utils import unpack_poma_archive
from sentence_transformers import SentenceTransformer
# 1. The missing link — raw Marker JSON in, chunks + chunksets out.
# Side-channel image bytes are spliced into their  refs
# and described into searchable text before chunking.
poma = PrimeCut() # reads POMA_API_KEY
result = poma.ingest(
"out/contract/contract.json", # block tree auto-detected
download_dir="store",
filename="contract.poma", # keep the archive — page numbers live in its chunk records
)
# A chunkset's structural metadata comes from its leaf chunk (cs.chunks is in
# document order, so the last index is the leaf). depth is on the SDK objects;
# page only exists on the archive's chunk records.
depth_of = {c.chunk_index: c.depth for c in result.chunks}
# -1 when the source has no pages (Markdown, text)
page_of = {
c["chunk_index"]: (c.get("page") if c.get("page") is not None else -1)
for c in unpack_poma_archive(poma_archive_path="store/contract.poma")["chunks"]
}
# 2. Embed chunksets; upsert with compact metadata only.
model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
pc = Pinecone(api_key=os.environ["PINECONE_API_KEY"])
index = pc.Index("contracts")
vectors = []
for cs in result.chunksets:
vectors.append({
"id": f"{cs.file_id}:{cs.chunkset_index}",
"values": model.encode(cs.to_embed).tolist(),
"metadata": { # scalars only — full text stays in the .poma archive
"file_id": cs.file_id,
"chunkset_index": cs.chunkset_index,
"page": page_of[cs.chunks[-1]],
"depth": depth_of[cs.chunks[-1]],
},
})
index.upsert(vectors=vectors, namespace="acme-corp") # namespace = tenant
# 3. Retrieve, then assemble cheatsheets from the returned IDs.
hits = index.query(
vector=model.encode("What are the early termination conditions?").tolist(),
top_k=5,
namespace="acme-corp",
filter={"file_id": {"$eq": result.chunksets[0].file_id}},
include_metadata=True,
)
retrieved = [int(m["metadata"]["chunkset_index"]) for m in hits["matches"]]
# Look up those chunksets in the .poma archive and merge them into one
# deduplicated, prompt-ready cheatsheet — hierarchy included.POMA validates the payload up front — the JSON Document block tree is the strict auto-route fingerprint; a corrupted or mislabeled upload 422s immediately. Bare markdown shapes ({markdown, images, metadata}, the deprecated {output, format} envelope) need an explicit external_ocr_source="marker". Because chunksets are overlap-free and cheatsheets deduplicate at assembly time, retrieved context stays lean: on our reference legal document, 337 tokens versus 1,542 for a recursive character splitter, with zero information loss — methodology in Document Ingestion & Chunking for RAG.
Metadata mapping: POMA fields → Pinecone primitives
| POMA chunk field | Pinecone primitive | What it enables |
|---|---|---|
to_embed | dense vector values (+ optional sparse values for hybrid) | paraphrase + exact-term retrieval |
file_id | metadata field, {"file_id": {"$eq": ...}} | scope queries to one document |
page (on the archive's chunk records, from Marker's Page blocks, JSON renderer; take the chunkset's leaf chunk) | numeric metadata field, $gte/$lte | page-cited answers, page-range filters |
depth of the leaf chunk | numeric metadata field | filter/re-rank by hierarchy level |
chunkset_index | metadata field + vector ID ({file_id}:{chunkset_index}) | ID-based lookup in the .poma archive |
| tenant / corpus | namespace | hard isolation; one query per namespace |
| full chunkset text | not stored in Pinecone — .poma archive / doc store | stays under the ~40 KB metadata cap |
Frequently asked questions
How do I get Marker output into Pinecone for RAG?
Run marker_single --output_format json, hand the saved result to PrimeCut().ingest() (auto-detected, images spliced, hierarchy rebuilt), embed each chunkset's to_embed text, and upsert with compact file_id/chunkset_index/page/depth metadata (depth off the SDK objects, page from the archive's chunk records for the chunkset's leaf chunk). Retrieve from the namespace, then assemble cheatsheets from the returned IDs.
Why does my Pinecone upsert fail with metadata size errors on Marker output?
Pinecone caps metadata at roughly 40 KB per vector, and Marker's full HTML tables (row and column spans included) or spliced data-URI figures overflow it if you stuff whole chunk text into metadata. Store only scalar metadata in Pinecone; keep the full text in the .poma archive or your document store, keyed by vector ID.
Should each Marker document get its own Pinecone namespace?
No — a query runs inside exactly one namespace, so per-document namespaces break corpus-wide questions. Use one namespace per tenant or corpus and scope to documents with a file_id metadata filter.
How do I keep Marker page numbers filterable in Pinecone?
Use the JSON renderer (markdown output has no page boundaries) and a chunker that reads the Page blocks. POMA records page on every chunk in the .poma archive; read it for the chunkset's leaf chunk and write it as a numeric metadata field. That supports {"page": {"$gte": 30, "$lte": 45}} filters and page-cited answers.
Does sparse-dense hybrid search in Pinecone help with Marker-parsed documents?
Yes. Locally parsed contracts and manuals are full of exact identifiers that dense embeddings blur; attach a sparse representation of to_embed alongside the dense vector and both paraphrase and exact-term queries hit. Overlap-free chunksets keep top-k free of near-duplicates either way.
Related recipes
Same parser, different store: Marker → Qdrant · Marker → Weaviate · Marker → Milvus
Same store, different parser: Mistral OCR → Pinecone · Textract → Pinecone · LlamaParse → Pinecone
Foundations: The Optimal Chunker for Marker · The Optimal Chunks for Pinecone · All pipeline recipes · RAG chunking guide