Source: http://www.poma-ai.com/docs/pipelines/docling-to-lancedb

# The Missing Link Between Docling and Optimal Retrieval in LanceDB

<ByAuthor />

**The short answer:** Docling gives you an explicit, typed document tree; LanceDB gives you an embedded, Arrow-native table with no server to operate. Wired together the common way — flatten to markdown, split, embed, one row per fragment — both tools end up worse than either alone, because nothing rebuilds cross-page hierarchy or keeps the table content-free. The missing link is POMA: `PrimeCut().ingest()` consumes the DoclingDocument tree directly, emits hierarchy-preserving [chunksets](/learn/chunking/chunksets), and `vektoria` writes them into LanceDB as content-free rows — vector plus `{file_id, chunkset_index}` only — while the actual text lives on a volume. Retrieval runs LanceDB's own `search()`, then `assemble()` reconstructs prompt-ready cheatsheets.

## What each end of the pipeline actually provides

**Docling** (IBM's open-source converter, also served via docling-serve) returns a typed **DoclingDocument**: `texts`/`tables`/`pictures`/`groups`, with `section_header` items carrying an explicit numeric `level` and repeating page furniture pre-isolated into a `furniture` group labeled `page_header`/`page_footer`. What it doesn't provide is a retrieval unit — its own `HybridChunker` packs tree items into token windows sized for an embedding model, which controls length, not whether a retrieved passage carries the heading path that explains it. Details: [The Optimal Chunker for Docling](/optimal-chunker-docling).

**LanceDB** provides an Arrow/Lance-backed table you `create_table` from a list of dicts — embedded in-process or object-storage-backed, no server, `.where(...)` for SQL-like scalar filtering alongside vector search. What it doesn't provide is any opinion about what a row should represent. It ranks whatever vector and columns you wrote. Details: [LanceDB Chunking Strategy for RAG](/optimal-chunks-lancedb).

## The naive wiring, and where it breaks

The common recipe — `doc.export_to_markdown()` → `RecursiveCharacterTextSplitter` → one table row per fragment — breaks in a pair-specific way:

- **Docling's `section_header` levels are discarded at export.** The tree already told you a passage lives under *Termination clauses* under *Master Services Agreement*; flattening to markdown reduces that to `##` glyphs the splitter ignores, cutting wherever the character count lands.
- **Furniture Docling already isolated re-enters the text stream.** The `furniture` group and `page_header`/`page_footer` labels exist so downstream tools skip re-detecting running noise — flattening throws that classification away, and page numbers end up embedded and indexed.
- **Overlap inflates the same Arrow table LanceDB indexes.** Splitter overlap duplicates every boundary span into rows `create_table` writes, growing both the table on disk or object storage and the vector index built over it, with top-k crowded by near-duplicate rows instead of the passage that actually answers the query.
- **A markdown-fragment table has no clean scalar shape.** Rows written from arbitrary splitter output typically carry only `{text, vector}`, so `.where("file_id = '...'")` filtering — the thing that would let you scope a query to one document cheaply — isn't available at all, and every query scans the whole table's vectors instead of a pre-filtered subset. This is specific to Docling-sourced pipelines that skip PrimeCut: the DoclingDocument tree already carries per-item structure that would populate those scalar columns for free, and flattening throws that opportunity away before the table is even created.

## The pipeline, end to end

```bash
pip install docling lancedb
```

```python
import json
import lancedb
from docling.document_converter import DocumentConverter
from poma import PrimeCut
from poma.vektoria import assemble, records_from_archive, Volume
from poma.embeddings import get_embedder

# 1. Your existing Docling conversion — unchanged. Save the tree as JSON.
converter = DocumentConverter()
doc = converter.convert("contract.pdf").document
with open("contract.docling.json", "w") as f:
    json.dump(doc.export_to_dict(), f)

# 2. The missing link — raw DoclingDocument JSON in, a portable .poma archive out.
client = PrimeCut()  # reads POMA_API_KEY
client.ingest("contract.docling.json", download_dir="archives", filename="contract.poma")

embedder = get_embedder("local:BAAI/bge-small-en-v1.5")
vol = Volume("s3://your-bucket/poma")
db = lancedb.connect("s3://your-bucket/lancedb")

# 3. Content-free ingest — the table holds only routing metadata; content lives on the volume.
records = records_from_archive("archives/contract.poma")  # -> list[Record] (id, text, payload)
rows = []
for r in records:
    vol.write_doc(r.payload["file_id"], {"file_id": r.payload["file_id"],
                                          "chunks": r.payload["chunks"], "text": r.text})
    rows.append({
        "id": r.id,
        "vector": embedder.embed([r.text])[0],
        "file_id": r.payload["file_id"],
        "chunkset_index": r.payload["chunkset_index"],
    })

tbl = db.create_table("poma", data=rows, exist_ok=True)

# 4. Vector search retrieval, then assemble prompt-ready context.
qv = embedder.embed(["early termination conditions"])[0]
results = tbl.search(qv).where("file_id = 'contract.pdf'").limit(10).to_list()
context = assemble(results, volume=vol)  # -> [{"file_id", "content"}, ...]
```

`records_from_archive` walks the `.poma` archive produced by ingest — the same archive that's the volume's source of truth — and returns one `Record` per chunkset: a deterministic `id`, the `to_embed` text, and a `payload` carrying `file_id`, `chunkset_index`, and the member chunk list. No `push()` connector exists for LanceDB yet; ingest here is the native `lancedb` client, shaped exactly the way vektoria's `assemble()` already expects on the way back out — and unlike Redis or MongoDB Atlas, no extra query-time flag is needed for metadata to come back.

## Metadata mapping: POMA fields → LanceDB primitives

| POMA record field | LanceDB primitive | What it enables |
| --- | --- | --- |
| `id` (`chunkset_uuid`) | `id` scalar column | deterministic key — safe re-ingest, no duplicate rows |
| `text` (`to_embed`) | `vector` column | the `search()` similarity query itself |
| `payload["file_id"]` | `file_id` scalar column | `.where("file_id = '...'")` scoping to one document |
| `payload["chunkset_index"]` | `chunkset_index` scalar column | routes a hit back to its content on the volume |
| chunkset content | volume (`Volume.write_doc`), never the table | smaller table; citations reconstructed at retrieval |

## Frequently asked questions

### How do I get Docling output into LanceDB for RAG?

Export the DoclingDocument tree with `export_to_dict` and save it as JSON. `PrimeCut().ingest()` auto-detects the shape and rebuilds hierarchy into chunksets; write each one's content to a `vektoria` `Volume`, embed its `to_embed` text, and add a row with the vector plus `file_id`/`chunkset_index` as scalar columns. Query with `search()`, then `assemble()`.

### Why not just export Docling to markdown and embed it into LanceDB directly?

`export_to_markdown()` collapses explicit `section_header` levels and pre-isolated furniture into display text a splitter must re-guess, and its overlap writes duplicate rows into the same Arrow table LanceDB indexes — inflating both the table and the vector index with no retrieval benefit.

### What LanceDB columns should Docling chunksets carry?

A `vector` column sized to your embedder, plus `file_id` and `chunkset_index` as scalar columns. Docling's hierarchy is resolved upstream by PrimeCut, so it lives in the chunkset text on the volume, not a separate table column.

### Does POMA assemble need any extra flag to read LanceDB results from Docling chunksets?

No. LanceDB returns all columns by default from `search().to_list()`, so `assemble()` reads `file_id`/`chunkset_index` off every hit with no extra request — the one DB pair in this series with no metadata-return gotcha.

### Does LanceDB support hybrid search for Docling-parsed documents?

Yes — `create_fts_index(...)` on the chunkset's `to_embed` text adds a lexical signal alongside `.where(...)` scalar filtering, useful for Docling-sourced documents where part numbers and defined terms are exact tokens dense embeddings alone tend to blur.

## Related recipes

Same parser, different store: [Docling → Chroma](/pipelines/docling-to-chroma) · [Docling → pgvector](/pipelines/docling-to-pgvector) · [Docling → Milvus](/pipelines/docling-to-milvus)

Foundations: [The Optimal Chunker for Docling](/optimal-chunker-docling) · [LanceDB Chunking Strategy for RAG](/optimal-chunks-lancedb) · [All pipeline recipes](/pipelines/) · [RAG architecture guide](/guides/rag-architecture/)