Source: http://www.poma-ai.com/docs/pipelines/marker-to-lancedb

# The Missing Link Between Marker and Optimal Retrieval in LanceDB

<ByAuthor />

**The short answer:** Marker gives you fast, local PDF parsing with real recovered structure; LanceDB gives you an embedded, Arrow-native table with no server to operate. Wired together naively — flatten the markdown, split, embed, write whatever columns happen to be at hand — the pipeline drops every figure before LanceDB sees a row, and the table's schema locks in around whatever the first batch of documents happened to include. The missing link is POMA: `PrimeCut().ingest()` consumes the saved Marker JSON (the Document block tree auto-detects), splices the side-channel image bytes back in, and rebuilds the cross-page hierarchy into [chunksets](/learn/chunking/chunksets). `vektoria` keeps the table content-free — a vector plus routing columns only — and `assemble()` turns a plain `search().to_list()` result into prompt-ready context.

## What each end of the pipeline actually provides

**Marker** (by Datalab / VikParuchuri) runs on your own hardware and turns PDFs into clean structured output. With the JSON renderer (`--output_format json`) you get a Document block tree — Page children, per-block HTML, full `<table>` elements with row and column spans. What it doesn't provide: cross-page hierarchy, retrieval units, or inline images — figure bytes live in a side-channel `images` dict keyed by name, with only `![](name)` references left in the text. Details: [The Optimal Chunker for Marker](/optimal-chunker-marker).

**LanceDB** is an embedded, Arrow/Lance-backed table format, run in-process against a local path or directly against `s3://` — no server, `tbl.search(qv).where(...).to_list()` for retrieval. What it doesn't provide: any opinion about what a row should represent, or forgiveness for a schema that `create_table()` locks in from whatever the first batch of rows contains. Details: [LanceDB Chunking Strategy for RAG](/optimal-chunks-lancedb).

## The naive wiring, and where it breaks

The common recipe — run Marker's default markdown output, `RecursiveCharacterTextSplitter`, embed, write rows with `create_table()` on whatever fields the first document happened to produce — breaks in a way specific to this pair:

- **Every figure vanishes before LanceDB.** Marker's images live in the side-channel `images` dict; the markdown carries only `![](name)` references. A markdown-only pipeline embeds those dangling references as literal text, so the chart that answers a question was never written as a row at all.
- **The first batch fixes the schema for every batch after it.** `create_table(name, data)` infers columns from the first write. Ingest one document through Marker's default markdown (no page data) before a second document run through the JSON renderer, and the table never gets a `page` column — LanceDB has no way to retroactively add what wasn't there at creation.
- **Page numbers depend entirely on renderer choice.** Marker's default markdown output has no page boundaries; only the JSON renderer's Page blocks give you something to put in a `page` column in the first place.
- **Overlap inflates the table.** Splitter overlap embeds every boundary span twice, and since LanceDB tables are frequently object-storage-backed, that duplication is a direct storage cost with no server around to flag it.

## The pipeline, end to end

```bash
# 1. Marker — your existing run, unchanged. The JSON renderer keeps the
#    Document block tree that POMA auto-detects, pages included.
marker_single contract.pdf --output_format json --output_dir out/
```

```python
import lancedb
from poma import PrimeCut
from poma.vektoria import assemble, records_from_archive, Volume
from poma.embeddings import get_embedder

# 2. The missing link — raw Marker JSON in, .poma archive out (chunks +
#    chunksets, image bytes spliced back into their references).
client = PrimeCut()  # reads POMA_API_KEY
client.ingest("out/contract/contract.json", download_dir="archives", filename="contract.poma")

embedder = get_embedder("local:BAAI/bge-small-en-v1.5")
vol = Volume("s3://your-bucket/poma")
db = lancedb.connect("s3://your-bucket/lancedb")

# 3. Content-free ingest — the table holds only routing metadata; content lives on the volume.
records = records_from_archive("archives/contract.poma")  # -> list[Record] (id, text, payload)
rows = []
for r in records:
    vol.write_doc(r.payload["file_id"], {"file_id": r.payload["file_id"],
                                          "chunks": r.payload["chunks"], "text": r.text})
    rows.append({
        "id": r.id,
        "vector": embedder.embed([r.text])[0],
        "file_id": r.payload["file_id"],
        "chunkset_index": r.payload["chunkset_index"],
    })

tbl = db.create_table("poma", data=rows, exist_ok=True)

# 4. Vector search retrieval, then assemble prompt-ready context.
qv = embedder.embed(["early termination conditions"])[0]
results = tbl.search(qv).where("file_id = 'contract.pdf'").limit(10).to_list()
context = assemble(results, volume=vol)  # -> [{"file_id", "content"}, ...]
```

POMA validates the payload shape up front — the JSON Document block tree is the strict auto-route fingerprint, and a corrupted or mislabeled upload 422s immediately instead of degrading the index. If what you saved is one of Marker's bare markdown shapes, declare it explicitly with `external_ocr_source="marker"`. On our reference legal document, this chunking answers the same query with **337 tokens** of retrieved context instead of **1,542** for a recursive character splitter — methodology in [Document Ingestion & Chunking for RAG](/document-ingestion-chunking-rag).

## Metadata mapping: POMA fields → LanceDB primitives

| POMA chunk field | LanceDB primitive | What it enables |
| --- | --- | --- |
| `to_embed` | `vector` column | similarity search over the embedded chunkset |
| `file_id` | scalar column | `.where("file_id = '...'")` scoping to one document |
| `chunkset_index` | scalar column | `assemble()` lookup key, stable ordering |
| `page` (JSON renderer only) | volume document by default; optional scalar column if added at `create_table()` time | content-free retrieval by default, or direct `.where()` page filters if denormalized |
| chunkset content | not in the table — lives on a `Volume` | small table, reassembled at retrieval |

## Frequently asked questions

### How do I get Marker output into LanceDB for RAG?

Run Marker with the JSON renderer and save the result; hand it to `PrimeCut().ingest()`, which auto-detects the Document block tree, splices images back in, and writes a `.poma` archive of chunks and chunksets. Pull records with `records_from_archive()`, embed each record's `to_embed` text, and write one row per record — a `vector` column plus `file_id` and `chunkset_index` scalar columns — with `create_table()`. Query with `tbl.search(qv).to_list()` and hand the result to `assemble()`.

### Do Marker's extracted images make it into a LanceDB table?

Only if something reunites them first. Marker parks figure bytes in a side-channel `images` dict and leaves bare `![](name)` references in the markdown, so a pipeline that embeds only the markdown writes table rows with zero figures represented. POMA splices the bytes back into their references as data URIs and describes each figure into searchable text before `embedder.embed()` ever runs, so the figure becomes a real row. References without bytes are neutralized and counted, never silently dropped.

### Why would a page column go missing from my LanceDB table after ingesting Marker output?

`create_table(name, data)` infers the table's schema from the very first batch of rows you write. If that first document came from Marker's default markdown output (no JSON renderer, no page boundaries), the rows in that batch carry no page field, and the schema locks in without one. A later document processed with the JSON renderer will have page data in its records, but nothing to write it into — the column was never created.

### What LanceDB columns should Marker chunks carry?

A `vector` column sized to your embedder, plus `file_id` and `chunkset_index` as scalar columns — the same fields `records_from_archive()` hands you after ingesting Marker's JSON block tree. Include a `page` column from the first batch onward if you're running the JSON renderer, since `create_table()` fixes the schema at creation time.

### Does Marker's JSON renderer matter for a LanceDB pipeline?

Yes. The JSON renderer's Document block tree is what POMA auto-detects and what preserves per-page structure — the only source for a page scalar column in your LanceDB rows. Marker's default markdown output loses page boundaries entirely, and because LanceDB infers its schema from the first write, mixing renderer choices across documents is exactly how a page column ends up defined for some rows and absent for others.

## Related recipes

Same parser, different store: [Marker → Qdrant](/pipelines/marker-to-qdrant) · [Marker → Chroma](/pipelines/marker-to-chroma) · [Marker → pgvector](/pipelines/marker-to-pgvector)

Foundations: [The Optimal Chunker for Marker](/optimal-chunker-marker) · [LanceDB Chunking Strategy for RAG](/optimal-chunks-lancedb) · [All pipeline recipes](/pipelines/) · [RAG chunking guide](/guides/rag-chunking/)