Marker to LanceDB: Chunking for RAG
The short answer: Marker gives you fast, local PDF parsing with real recovered structure; LanceDB gives you an embedded, Arrow-native table with no server to operate. Wired together naively — flatten the markdown, split, embed, write whatever columns happen to be at hand — the pipeline drops every figure before LanceDB sees a row, and the table's schema locks in around whatever the first batch of documents happened to include. The missing link is POMA: PrimeCut().ingest() consumes the saved Marker JSON (the Document block tree auto-detects), splices the side-channel image bytes back in, and rebuilds the cross-page hierarchy into chunksets. vektoria keeps the table content-free — a vector plus routing columns only — and assemble() turns a plain search().to_list() result into prompt-ready context.
What each end of the pipeline actually provides
Marker (by Datalab / VikParuchuri) runs on your own hardware and turns PDFs into clean structured output. With the JSON renderer (--output_format json) you get a Document block tree — Page children, per-block HTML, full <table> elements with row and column spans. What it doesn't provide: cross-page hierarchy, retrieval units, or inline images — figure bytes live in a side-channel images dict keyed by name, with only  references left in the text. Details: The Optimal Chunker for Marker.
LanceDB is an embedded, Arrow/Lance-backed table format, run in-process against a local path or directly against s3:// — no server, tbl.search(qv).where(...).to_list() for retrieval. What it doesn't provide: any opinion about what a row should represent, or forgiveness for a schema that create_table() locks in from whatever the first batch of rows contains. Details: LanceDB Chunking Strategy for RAG.
The naive wiring, and where it breaks
The common recipe — run Marker's default markdown output, RecursiveCharacterTextSplitter, embed, write rows with create_table() on whatever fields the first document happened to produce — breaks in a way specific to this pair:
- Every figure vanishes before LanceDB. Marker's images live in the side-channel
imagesdict; the markdown carries onlyreferences. A markdown-only pipeline embeds those dangling references as literal text, so the chart that answers a question was never written as a row at all. - The first batch fixes the schema for every batch after it.
create_table(name, data)infers columns from the first write. Ingest one document through Marker's default markdown (no page data) before a second document run through the JSON renderer, and the table never gets apagecolumn — LanceDB has no way to retroactively add what wasn't there at creation. - Page numbers depend entirely on renderer choice. Marker's default markdown output has no page boundaries; only the JSON renderer's Page blocks give you something to put in a
pagecolumn in the first place. - Overlap inflates the table. Splitter overlap embeds every boundary span twice, and since LanceDB tables are frequently object-storage-backed, that duplication is a direct storage cost with no server around to flag it.
The pipeline, end to end
# 1. Marker — your existing run, unchanged. The JSON renderer keeps the
# Document block tree that POMA auto-detects, pages included.
marker_single contract.pdf --output_format json --output_dir out/import lancedb
from poma import PrimeCut
from poma.vektoria import assemble, records_from_archive, Volume
from poma.embeddings import get_embedder
# 2. The missing link — raw Marker JSON in, .poma archive out (chunks +
# chunksets, image bytes spliced back into their references).
client = PrimeCut() # reads POMA_API_KEY
client.ingest("out/contract/contract.json", download_dir="archives", filename="contract.poma")
embedder = get_embedder("local:BAAI/bge-small-en-v1.5")
vol = Volume("s3://your-bucket/poma")
db = lancedb.connect("s3://your-bucket/lancedb")
# 3. Content-free ingest — the table holds only routing metadata; content lives on the volume.
records = records_from_archive("archives/contract.poma") # -> list[Record] (id, text, payload)
rows = []
# assemble() reads the FULL document (chunks + chunksets) from the volume — write it once,
# not one {"chunks": <indices>, "text"} stub per record (which also overwrote itself each turn).
from poma.utils import unpack_poma_archive
_data = unpack_poma_archive(poma_archive_path="archives/contract.poma")
file_id = records[0].payload["file_id"]
vol.write_doc(file_id, {"file_id": file_id, "chunks": _data["chunks"], "chunksets": _data["chunksets"]})
for r in records:
rows.append({
"id": r.id,
"vector": embedder.embed([r.text])[0],
"file_id": r.payload["file_id"],
"chunkset_index": r.payload["chunkset_index"],
})
tbl = db.create_table("poma", data=rows, exist_ok=True)
# 4. Vector search retrieval, then assemble prompt-ready context.
qv = embedder.embed(["early termination conditions"])[0]
results = tbl.search(qv).where(f"file_id = '{file_id}'").limit(10).to_list() # POMA assigns an opaque file_id, not the file name
context = assemble(results, volume=vol) # -> [{"file_id", "content"}, ...]POMA validates the payload shape up front — the JSON Document block tree is the strict auto-route fingerprint, and a corrupted or mislabeled upload 422s immediately instead of degrading the index. If what you saved is one of Marker's bare markdown shapes, declare it explicitly with external_ocr_source="marker". On our reference legal document, this chunking answers the same query with 337 tokens of retrieved context instead of 1,542 for a recursive character splitter — methodology in Document Ingestion & Chunking for RAG.
Metadata mapping: POMA fields → LanceDB primitives
| POMA chunk field | LanceDB primitive | What it enables |
|---|---|---|
to_embed | vector column | similarity search over the embedded chunkset |
file_id | scalar column | .where("file_id = '...'") scoping to one document |
chunkset_index | scalar column | assemble() lookup key, stable ordering |
page (JSON renderer only) | volume document by default; optional scalar column if added at create_table() time | content-free retrieval by default, or direct .where() page filters if denormalized |
| chunkset content | not in the table — lives on a Volume | small table, reassembled at retrieval |
Frequently asked questions
How do I get Marker output into LanceDB for RAG?
Run Marker with the JSON renderer and save the result; hand it to PrimeCut().ingest(), which auto-detects the Document block tree, splices images back in, and writes a .poma archive of chunks and chunksets. Pull records with records_from_archive(), embed each record's to_embed text, and write one row per record — a vector column plus file_id and chunkset_index scalar columns — with create_table(). Query with tbl.search(qv).to_list() and hand the result to assemble().
Do Marker's extracted images make it into a LanceDB table?
Only if something reunites them first. Marker parks figure bytes in a side-channel images dict and leaves bare  references in the markdown, so a pipeline that embeds only the markdown writes table rows with zero figures represented. POMA splices the bytes back into their references as data URIs and describes each figure into searchable text before embedder.embed() ever runs, so the figure becomes a real row. References without bytes are neutralized and counted, never silently dropped.
Why would a page column go missing from my LanceDB table after ingesting Marker output?
create_table(name, data) infers the table's schema from the very first batch of rows you write. If that first document came from Marker's default markdown output (no JSON renderer, no page boundaries), the rows in that batch carry no page field, and the schema locks in without one. A later document processed with the JSON renderer will have page data in its records, but nothing to write it into — the column was never created.
What LanceDB columns should Marker chunks carry?
A vector column sized to your embedder, plus file_id and chunkset_index as scalar columns — the same fields records_from_archive() hands you after ingesting Marker's JSON block tree. Include a page column from the first batch onward if you're running the JSON renderer, since create_table() fixes the schema at creation time.
Does Marker's JSON renderer matter for a LanceDB pipeline?
Yes. The JSON renderer's Document block tree is what POMA auto-detects and what preserves per-page structure — the only source for a page scalar column in your LanceDB rows. Marker's default markdown output loses page boundaries entirely, and because LanceDB infers its schema from the first write, mixing renderer choices across documents is exactly how a page column ends up defined for some rows and absent for others.
Related recipes
Same parser, different store: Marker → Qdrant · Marker → Chroma · Marker → pgvector
Foundations: The Optimal Chunker for Marker · LanceDB Chunking Strategy for RAG · All pipeline recipes · RAG chunking guide