Source: http://www.poma-ai.com/docs/pipelines/mistral-ocr-to-lancedb

# The Missing Link Between Mistral OCR and Optimal Retrieval in LanceDB

<ByAuthor />

**The short answer:** Mistral OCR gives you excellent per-page markdown; LanceDB gives you an embedded, Arrow-native vector table with no server to operate. Wired together naively — flatten, split, embed, insert — they still produce mediocre RAG, and a naive row shape can also break LanceDB's schema inference across documents. The missing link is POMA: `PrimeCut().ingest()` consumes the raw `/v1/ocr` JSON, emits hierarchy-preserving [chunksets](/learn/chunking/chunksets), and `vektoria`'s content-free ingest pattern writes a fixed, small row shape into LanceDB while `assemble()` reconstructs prompt-ready context from a volume at query time.

## What each end of the pipeline actually provides

**Mistral OCR** (`/v1/ocr`, `mistral-ocr-latest`) returns `pages[]`, each with an `index` and a `markdown` string — headings, lists, and tables recovered *within* each page, image bytes inline when you set `include_image_base64`, in an `images` array whose length varies page to page. What it doesn't provide: cross-page hierarchy (a heading on page 41 has no machine link to the chapter that opened on page 3) or retrieval units. Details: [The Optimal Chunker for Mistral OCR](/optimal-chunker-mistral-ocr).

**LanceDB** provides an embedded, Arrow/Lance-backed table — local or `s3://`-backed, no server — with a `vector` column, SQL-like `.where(...)` filtering on scalar columns, and an optional full-text index for hybrid queries. What it doesn't provide: any opinion about what a row should hold, and its table schema is inferred once, from the first batch written. Details: [LanceDB Chunking Strategy for RAG](/optimal-chunks-lancedb).

## The naive wiring, and where it breaks

The common recipe — `"\n".join(p["markdown"] for p in pages)` → `RecursiveCharacterTextSplitter` → embed → build a row dict straight from each page's raw fields → `create_table(...)` — breaks in a pair-specific way:

- **Mistral's page indices vanish at the join**, so no scalar column can carry a page number, and `.where(...)` has only `file_id` left to filter on.
- **Overlap adds near-duplicate rows** that inflate both the table and the vector index built over it, for no retrieval benefit.
- **LanceDB's schema is fixed at `create_table`, inferred from the first batch — and Mistral's own output shape is exactly what breaks it.** If a naive pipeline keeps Mistral's raw `images` field (or any other page-specific key) on the row, the column set differs between a page with two images and a page with none, or between one document and the next. LanceDB either rejects the mismatched `add()` call or silently coerces the data against a schema that no longer describes it — a failure mode unique to LanceDB's Arrow-first design, and one a content-free row shape (only `id`, `vector`, `file_id`, `chunkset_index`) sidesteps completely, since those four fields never vary across pages or documents.

## The pipeline, end to end

```bash
pip install mistralai lancedb
```

```python
import os
from mistralai import Mistral
import lancedb
from poma import PrimeCut
from poma.vektoria import assemble, records_from_archive, Volume
from poma.embeddings import get_embedder

# 1. Mistral OCR — your existing call, unchanged. Save the raw result JSON.
mistral = Mistral(api_key=os.environ["MISTRAL_API_KEY"])
ocr = mistral.ocr.process(
    model="mistral-ocr-latest",
    document={"type": "document_url", "document_url": "https://example.com/contract.pdf"},
    include_image_base64=True,
)
with open("contract.mistral-ocr.json", "w") as f:
    f.write(ocr.model_dump_json())

# 2. The missing link — raw result JSON in, a portable .poma archive out.
client = PrimeCut()  # reads POMA_API_KEY
client.ingest("contract.mistral-ocr.json", download_dir="archives", filename="contract.poma")

embedder = get_embedder("local:BAAI/bge-small-en-v1.5")
vol = Volume("s3://your-bucket/poma")
db = lancedb.connect("s3://your-bucket/lancedb")

# 3. Content-free ingest — a fixed row shape, so schema inference never breaks across documents.
records = records_from_archive("archives/contract.poma")  # -> list[Record] (id, text, payload)
rows = []
for r in records:
    vol.write_doc(r.payload["file_id"], {"file_id": r.payload["file_id"],
                                          "chunks": r.payload["chunks"], "text": r.text})
    rows.append({
        "id": r.id,
        "vector": embedder.embed([r.text])[0],
        "file_id": r.payload["file_id"],
        "chunkset_index": r.payload["chunkset_index"],
    })

tbl = db.create_table("poma", data=rows, exist_ok=True)

# 4. Vector search retrieval — metadata returned by default, then assemble.
qv = embedder.embed(["early termination conditions"])[0]
results = tbl.search(qv).where("file_id = 'contract.pdf'").limit(10).to_list()
context = assemble(results, volume=vol)  # -> [{"file_id", "content"}, ...]
```

POMA validates the Mistral payload shape up front (a corrupted or mislabeled upload 422s immediately), handles all three Mistral image cases, strips running headers/footers, and rebuilds the cross-page heading tree before chunking. Retrieved chunksets share ancestor lineage across a document — `assemble()` deduplicates that lineage into one prompt-ready cheatsheet, the same discipline that answers our reference legal-document benchmark with **337 tokens** instead of **1,542** for a recursive-splitter baseline. [Methodology here](/document-ingestion-chunking-rag).

## Metadata mapping: POMA fields → LanceDB primitives

| POMA chunk field | LanceDB primitive | What it enables |
| --- | --- | --- |
| `to_embed` | `vector` column | ANN search via `tbl.search(...)` |
| `file_id` | scalar column | `.where("file_id = '...'")` scoping |
| `page` (from Mistral `pages[].index`) | volume document, not a column | content-free retrieval, cheatsheet assembly |
| `depth` | volume document, not a column | content-free retrieval, cheatsheet assembly |
| `chunkset_index` | scalar column, returned by default | `assemble()` dedup and ordering |
| chunkset lineage (content) | not stored in the table — lives on the `Volume` | content-free table, no schema churn from raw page fields |

## Frequently asked questions

### How do I get Mistral OCR results into LanceDB for RAG?

Save the raw `/v1/ocr` JSON, run `PrimeCut().ingest()` on it, then pull `records_from_archive(...)`, write each chunkset's content to a volume, embed its `to_embed` text, and `create_table`/`add` rows (`id`, `vector`, `file_id`, `chunkset_index`). Retrieve with `tbl.search(...).to_list()` and `assemble()`.

### Does POMA's assemble() need any special flag to read LanceDB results after a Mistral OCR ingest?

No — LanceDB returns every column by default on `.to_list()`, including `file_id` and `chunkset_index`, so `assemble(results, volume=VOL)` works with no extra request. LanceDB is the one new-DB pair here with no metadata-return gotcha.

### What happens if I insert raw Mistral OCR page dicts straight into a LanceDB table?

LanceDB infers a table's schema from the first batch written. Mistral's `images` array varies in length page to page and document to document, so a row shape built from raw page dicts changes columns across inserts, and later `add()` calls fail or coerce against a stale schema. A fixed, content-free row shape avoids this.

### Why not just split Mistral OCR's markdown and add it to a LanceDB table directly?

Concatenation discards page indices and heading levels, and splitter overlap adds near-duplicate rows that inflate the table and its vector index for no retrieval benefit. LanceDB then ranks context-free fragments exactly as fast as it would rank real chunksets.

### Is LanceDB a good fit for prototyping a Mistral OCR RAG pipeline before scaling it?

Yes — its embedded, Arrow/Lance-backed design needs no server, and tables scale to object storage without a schema rewrite, so a pipeline built with the content-free chunkset row shape from day one carries into a larger deployment unchanged.

## Related recipes

Same parser, different store: [Mistral OCR → Chroma](/pipelines/mistral-ocr-to-chroma) · [Mistral OCR → Qdrant](/pipelines/mistral-ocr-to-qdrant) · [Mistral OCR → pgvector](/pipelines/mistral-ocr-to-pgvector)

Foundations: [The Optimal Chunker for Mistral OCR](/optimal-chunker-mistral-ocr) · [LanceDB Chunking Strategy for RAG](/optimal-chunks-lancedb) · [All pipeline recipes](/pipelines/) · [RAG chunking guide](/guides/rag-chunking/) · [POMA chunksets](/learn/chunking/chunksets)