Source: http://www.poma-ai.com/docs/pipelines/llamaparse-to-lancedb

# The Missing Link Between LlamaParse and Optimal Retrieval in LanceDB

<ByAuthor />

**The short answer:** LlamaParse gives you excellent per-page markdown; LanceDB gives you an embedded, Arrow-native table with no server to operate. Wired together naively — split each page, embed, `create_table` — they still produce mediocre RAG, because nothing in between rebuilds cross-page hierarchy or accounts for how LanceDB infers its schema. The missing link is POMA: `PrimeCut().ingest()` consumes the raw LlamaParse JSON (auto-detected), emits hierarchy-preserving [chunksets](/learn/chunking/chunksets) as a portable `.poma` archive, and `vektoria`'s `records_from_archive()` + the `lancedb` client write content-free rows — `vector`, `file_id`, `chunkset_index` — while chunkset content lives on a volume. Retrieval runs LanceDB's own `search()` call, then `assemble()` reassembles prompt-ready context.

## What each end of the pipeline actually provides

**LlamaParse** returns `pages[]`, each with a `page` number, an `md` string (markdown, tables inline), a `text` flattening, and an `images` list whose bytes stay server-side — a saved result JSON carries only `![](name)` references. What it doesn't provide: cross-page hierarchy (a `##` heading on page 41 has no link to the `#` chapter that opened on page 3) or retrieval units. Details: [The Optimal Chunker for LlamaParse](/optimal-chunker-llamaparse).

**LanceDB** provides an Arrow/Lance-backed table you can run embedded or against object storage, with `.where(...)` SQL-like filtering and an optional full-text index for hybrid queries. What it doesn't provide: a schema you declare up front — `create_table(name, data)` infers it from whatever rows arrive in the first batch. That inference is a deliberate simplicity trade-off — no separate DDL step — but it shifts the burden onto whoever writes the first batch to make sure it's representative. Details: [LanceDB Chunking Strategy for RAG](/optimal-chunks-lancedb).

## The naive wiring, and where it breaks

The common recipe — split each `pages[].md` entry on its own, embed, `create_table("poma", data=rows)` — breaks in a pair-specific way that compounds two quirks at once:

- **Splitting per page bakes in LlamaParse's independence.** LlamaParse never links headings across pages, so treating each page's `md` as a standalone unit locks in orphan fragments as your table's permanent row shape — there is no later step that recovers the chapter a page-41 clause belongs to.
- **LanceDB's schema is inferred from whichever rows arrive first.** `create_table` fixes its Arrow schema from the first batch it sees. A naive per-page pipeline processes pages in document order, and a cover page or an image-only page — exactly the kind of page LlamaParse's dead `![](name)` references show up on — can produce a row with a null or oddly-typed field before any "normal" text row exists. That locks in a schema the rest of the document's rows don't cleanly fit.
- **The failure surfaces downstream, not at write time.** A later batch with a populated `file_id` or a different `chunkset_index` type against the locked-in schema either raises an append error or gets silently coerced — neither of which points back to the actual cause: the first batch wasn't representative because the ingest pipeline had no hierarchy rebuild to normalize it.
- **Debugging it backwards is slow.** Because the schema mismatch surfaces on whatever row happens to violate it — not on the row that actually caused it — teams typically spend more time bisecting the ingest batch than they would have spent running the hierarchy rebuild up front.

## The pipeline, end to end

```bash
pip install llama-parse lancedb
```

```python
import json
import os
import lancedb
from llama_parse import LlamaParse
from poma import PrimeCut
from poma.vektoria import assemble, records_from_archive, Volume
from poma.embeddings import get_embedder

# 1. Your existing LlamaParse call — unchanged. Save the raw result JSON.
parser = LlamaParse(api_key=os.environ["LLAMA_CLOUD_API_KEY"])
json_result = parser.get_json_result("contract.pdf")
with open("contract.llamaparse.json", "w") as f:
    json.dump(json_result[0], f)

# 2. The missing link — raw LlamaParse JSON in, a portable .poma archive out.
client = PrimeCut()  # reads POMA_API_KEY
client.ingest("contract.llamaparse.json", download_dir="archives", filename="contract.poma")

embedder = get_embedder("local:BAAI/bge-small-en-v1.5")
vol = Volume("s3://your-bucket/poma")
db = lancedb.connect("s3://your-bucket/lancedb")

# 3. Content-free ingest — the table holds only routing metadata; content lives on the volume.
records = records_from_archive("archives/contract.poma")  # -> list[Record] (id, text, payload)
rows = []
for r in records:
    vol.write_doc(r.payload["file_id"], {"file_id": r.payload["file_id"],
                                          "chunks": r.payload["chunks"], "text": r.text})
    rows.append({
        "id": r.id,
        "vector": embedder.embed([r.text])[0],
        "file_id": r.payload["file_id"],
        "chunkset_index": r.payload["chunkset_index"],
    })

tbl = db.create_table("poma", data=rows, exist_ok=True)

# 4. Vector search retrieval, then assemble prompt-ready context.
qv = embedder.embed(["early termination conditions"])[0]
results = tbl.search(qv).where("file_id = 'contract.pdf'").limit(10).to_list()
context = assemble(results, volume=vol)  # -> [{"file_id", "content"}, ...]
```

Because `records_from_archive()` emits one uniform `Record` per chunkset — same fields, same types, every time — the whole batch you pass to `create_table` is representative from the first row, which is exactly what sidesteps the schema-inference failure mode above. PrimeCut also validates the payload shape up front (a mislabeled upload 422s immediately), prefers `md` over `text`, neutralizes LlamaParse's dead image references into visible, counted markers, and rebuilds the cross-page heading tree before chunking. To force or suppress detection, set `external_ocr_source` to `"llamaparse"` or `"none"`.

## Metadata mapping: POMA fields → LanceDB primitives

| POMA chunk field | LanceDB primitive | What it enables |
| --- | --- | --- |
| `to_embed` | `vector` column | dense retrieval via `search()` |
| `file_id` | scalar column | `.where("file_id = '...'")` filtering |
| `chunkset_index` | scalar column | deterministic id + lineage for `assemble()` |
| chunkset content (`text`, `chunks`) | volume, not a table column | keeps the table small and object-storage-friendly |

## Frequently asked questions

### How do I get LlamaParse results into LanceDB for RAG?

Save the raw result JSON, run `PrimeCut().ingest()` on it (auto-detected, archived to `.poma`), then write `records_from_archive()` output as rows via `create_table`. Retrieve with `search().to_list()`, then `assemble(results, volume=vol)`.

### Why not just split LlamaParse's md per page and write rows into LanceDB directly?

Each LlamaParse page's `md` is independent, so splitting per page bakes orphan fragments into the table permanently. It also risks LanceDB's schema-inference gotcha: `create_table` fixes its schema from the first batch, and an unrepresentative first page can lock in a bad shape.

### What table schema does LanceDB need for LlamaParse chunksets?

One row per chunkset: a `vector` column sized to your embedder, plus `file_id` and `chunkset_index` as scalar columns. `create_table(name, data)` infers the schema from the first batch you pass it.

### Why did my LanceDB ingest fail partway through when loading LlamaParse content?

`create_table` locks its Arrow schema to the first batch. A naive pipeline processing raw LlamaParse pages in document order can hit a null or oddly-typed field on an early cover or image-only page before a normal text row exists, so a later, fully-populated batch fails to append or gets coerced against the wrong schema.

### What happens to LlamaParse's offloaded images in a LanceDB table?

PrimeCut neutralizes LlamaParse's dead `![](name)` references into visible, counted markers before chunking, so no row in the table represents an unresolved image link as real content. Each neutralized reference is also counted in `content_metadata`, so the loss stays visible and quantified rather than silently vanishing into the table.

## Related recipes

Same parser, different store: [LlamaParse → Redis](/pipelines/llamaparse-to-redis) · [LlamaParse → MongoDB Atlas](/pipelines/llamaparse-to-mongodb-atlas) · [LlamaParse → Chroma](/pipelines/llamaparse-to-chroma)

Foundations: [The Optimal Chunker for LlamaParse](/optimal-chunker-llamaparse) · [LanceDB Chunking Strategy for RAG](/optimal-chunks-lancedb) · [All pipeline recipes](/pipelines/) · [RAG architecture guide](/guides/rag-architecture/)