Source: http://www.poma-ai.com/docs/pipelines/marker-to-turbopuffer

# The Missing Link Between Marker and Optimal Retrieval in Turbopuffer

<ByAuthor />

**The short answer:** Marker gives you fast, local PDF parsing with real recovered structure — headings, lists, and full HTML tables inside its JSON block tree. Turbopuffer gives you a namespace-per-tenant vector store with native hybrid ANN+BM25 ranking at object-storage scale. Wired together naively, the pipeline drops every figure and risks blowing past Turbopuffer's filterable-attribute cap on wide tables before a single row is written. The missing link is POMA: `PrimeCut().ingest()` turns the saved Marker JSON into hierarchy-preserving [chunksets](/learn/chunking/chunksets), and `poma.vektoria` writes them to Turbopuffer as content-free rows — `id`, `vector`, and two routing attributes — while the actual chunkset text lives on a volume. Retrieval runs Turbopuffer's own query, then `assemble()` reconstructs prompt-ready cheatsheets.

## What each end of the pipeline actually provides

**Marker** (by Datalab / VikParuchuri) runs on your own hardware and turns PDFs into clean structured output. With the JSON renderer (`--output_format json`) you get a Document block tree — Page children, per-block HTML, and full `<table>` elements with row and column spans intact. What it doesn't provide: cross-page hierarchy, retrieval units, or inline images — figure bytes live in a side-channel `images` dict keyed by name, with only `![](name)` references left in the text. Details: [The Optimal Chunker for Marker](/optimal-chunker-marker).

**Turbopuffer** provides namespaces (the tenant/collection boundary, created implicitly on first write), typed row attributes declared inline on write, and native hybrid search — `ANN`, `SparseKNN`, and `BM25` ranking fused server-side via `rerank_by=("RRF",)`. Storage/compute are separated: durable state lives in object storage, so cold namespaces pay a small extra read before hot data is cached. What it doesn't provide: any opinion about what a row should contain, or a document store — that's the `poma.vektoria` volume's job. Details: [Turbopuffer Chunking Strategy for RAG](/optimal-chunks-turbopuffer).

## The naive wiring, and where it breaks

The common recipe — run Marker's default markdown output, `RecursiveCharacterTextSplitter`, embed, `ns.write(upsert_rows=[...])` with the raw chunk text as a row attribute — breaks in ways specific to this pair:

- **Every figure vanishes before Turbopuffer sees a row.** Marker's images live in the side-channel `images` dict; the markdown carries only `![](name)` references. A markdown-only pipeline embeds those dangling references as literal text, so the chart that answers "what does the payment schedule look like" was never in the vector at all.
- **Wide tables risk the filterable-attribute cap.** Marker's JSON renderer preserves full `<table>` elements with colspans — genuinely useful structure. Writing that raw HTML straight into a Turbopuffer attribute you intend to filter on courts the documented 4 KiB filterable-value limit; a naive splitter that cuts the table mid-row loses the span structure anyway, so neither path keeps the table usable.
- **Page numbers never make it into any filter**, because Marker's markdown output has no page boundaries — only the JSON renderer's Page blocks carry them, and only if your chunker actually reads them.
- **Overlap inflates the namespace.** Splitter overlap embeds every boundary span twice, growing the SPFresh index and crowding `top_k` with near-duplicates of the same passage.

## The pipeline, end to end

```bash
pip install turbopuffer

# Your existing Marker run — unchanged. The JSON renderer keeps the
# Document block tree that POMA auto-detects, tables and pages included.
marker_single contract.pdf --output_format json --output_dir out/
```

```python
import os
from poma import PrimeCut
from poma.vektoria import assemble, records_from_archive, Volume
from poma.embeddings import get_embedder
import turbopuffer

# 1. Chunk the raw Marker JSON and keep the .poma archive — the volume's source of truth.
client = PrimeCut()  # reads POMA_API_KEY
client.ingest("out/contract/contract.json", download_dir="archives", filename="contract.poma")

embedder = get_embedder("local:BAAI/bge-small-en-v1.5")
vol = Volume("s3://your-bucket/poma")
tpuf = turbopuffer.Turbopuffer(api_key=os.environ["TURBOPUFFER_API_KEY"], region="gcp-us-central1")
ns = tpuf.namespace("contracts")

# 2. Content-free ingest — the namespace holds only routing attributes; content lives on the volume.
records = records_from_archive("archives/contract.poma")  # -> list[Record] (id, text, payload)
rows = []
for r in records:
    vol.write_doc(r.payload["file_id"], {"file_id": r.payload["file_id"],
                                          "chunks": r.payload["chunks"], "text": r.text})
    rows.append({
        "id": r.id,
        "vector": embedder.embed([r.text])[0],
        "file_id": r.payload["file_id"],
        "chunkset_index": r.payload["chunkset_index"],
    })
ns.write(
    upsert_rows=rows,
    distance_metric="cosine_distance",
    schema={"file_id": {"type": "string"}, "chunkset_index": {"type": "int"}},
)

# 3. Retrieval — Turbopuffer's normal ANN query, then assemble.
qv = embedder.embed(["What are the early termination conditions?"])[0]
res = ns.query(rank_by=("vector", "ANN", qv), top_k=10,
               include_attributes=["file_id", "chunkset_index"])
context = assemble(res, volume=vol)  # -> [{"file_id", "content"}, ...]
print(context[0]["content"])
```

`records_from_archive` reads the `.poma` archive PrimeCut just wrote and returns one `Record` per chunkset: `.id` (deterministic `chunkset_uuid(file_id, chunkset_index)`), `.text` (the `to_embed` string), and `.payload` (`file_id`, `chunkset_index`, `chunks`). Because the same `id` scheme is used on every write, re-ingesting a document overwrites the same rows rather than duplicating them. On the reference legal-document benchmark this discipline answers with **337 tokens** of retrieved context instead of **1,542** for a recursive-splitter baseline — [methodology here](/document-ingestion-chunking-rag).

## Metadata mapping: POMA fields → Turbopuffer primitives

| POMA field | Where it lives | Turbopuffer role |
| --- | --- | --- |
| `to_embed` | embedded, written as `vector` | dense ANN ranking |
| `file_id` | row attribute (`string`) | filterable, scopes queries to one document |
| `chunkset_index` | row attribute (`int`) + volume key | forms the deterministic `id`; must be requested via `include_attributes` for `assemble()` to resolve content |
| `page`, `depth`, `chunks` (from Marker's Page blocks and block tree) | volume only (`record.payload`) | reconstructed into cheatsheet content, never stored as a Turbopuffer row attribute |
| chunkset lineage | volume document | deduplicated and merged into one cheatsheet by `assemble()` |

## Frequently asked questions

### How do I get Marker output into Turbopuffer for RAG?

Run Marker with the JSON renderer (`marker_single --output_format json`), hand the saved result to `PrimeCut().ingest()`, and keep the `.poma` archive. Turn each chunkset into a `Record` with `records_from_archive`, embed `record.text`, and `ns.write()` a content-free row (`id`, `vector`, `file_id`, `chunkset_index`) per record. Retrieve with `ns.query()` and pass the result to `assemble()`.

### Why not just split Marker's markdown and write it into a Turbopuffer namespace directly?

Marker's markdown has no page boundaries and leaves image bytes in a side-channel dict, so a naive splitter embeds dangling `![](name)` refs and drops every figure. Writing raw chunk text (including wide HTML tables) into a filterable attribute also risks the 4 KiB filterable-value cap. The missing link keeps Marker's tables and images intact upstream and writes only compact routing attributes downstream.

### What Turbopuffer attributes should Marker chunksets carry?

`file_id` and `chunkset_index` as typed row attributes, plus the vector. `records_from_archive` emits exactly these two routing fields per chunkset — the actual text and hierarchy stay on the volume.

### Do Marker's extracted images and tables survive the trip to Turbopuffer?

Yes, upstream of Turbopuffer. PrimeCut splices Marker's side-channel `images` dict back into `![](name)` references and describes each figure, and keeps full HTML tables intact through the chunk layer. Turbopuffer itself never stores images or tables — only a vector and two routing attributes — and the reconstructed content comes back from the volume via `assemble()`.

### Does Turbopuffer retrieval need extra fields for POMA's assemble() to work?

Yes — `ns.query()` must explicitly pass `include_attributes=["file_id", "chunkset_index"]`, or the returned rows carry only scores and vectors and `assemble()` has nothing to match against the volume.

## Related recipes

Same parser, different store: [Marker → Qdrant](/pipelines/marker-to-qdrant) · [Marker → Pinecone](/pipelines/marker-to-pinecone) · [Marker → Weaviate](/pipelines/marker-to-weaviate)

Foundations: [The Optimal Chunker for Marker](/optimal-chunker-marker) · [Turbopuffer Chunking Strategy for RAG](/optimal-chunks-turbopuffer) · [All pipeline recipes](/pipelines/) · [RAG chunking guide](/guides/rag-chunking/)