Source: http://www.poma-ai.com/docs/pipelines/unstructured-to-lancedb

# The Missing Link Between Unstructured.io and Optimal Retrieval in LanceDB

<ByAuthor />

**The short answer:** Unstructured.io gives you a typed, ordered element list; LanceDB gives you an embedded, Arrow-backed table with SQL-style filtering and no server to operate. Wired together naively — join element text, split, embed, write rows — they still produce mediocre RAG, because nothing in between rebuilds the hierarchy Unstructured's typing implies but never states. The missing link is POMA: `PrimeCut().ingest()` consumes the raw element list (auto-detected), emits hierarchy-preserving [chunksets](/learn/chunking/chunksets), and `vektoria`'s content-free ingest pattern writes them into a LanceDB table with the columns retrieval needs. Retrieval comes back through `assemble()` as prompt-ready cheatsheets.

## What each end of the pipeline actually provides

**Unstructured.io** (`partition_pdf` or the hosted API) returns a flat, ordered list of typed elements — `Title`, `NarrativeText`, `ListItem`, `Table`, `Image`, plus page furniture types — each carrying a stable `element_id` and metadata such as `page_number` and `text_as_html`. What it doesn't provide: any record of which `Title` nests under which, or a retrieval-ready unit. Details: [The Optimal Chunker for Unstructured.io](/optimal-chunker-unstructured).

**LanceDB** provides an Arrow/Lance columnar table — `vector` as one column among ordinary scalar columns, `.where(...)` for SQL-like predicates, and an optional `create_fts_index(...)` for a lexical signal — embedded in-process or backed by object storage. What it doesn't provide: any opinion about what a row's text should represent, or filtering of what you write into the full-text index. Details: [LanceDB Chunking Strategy for RAG](/optimal-chunks-lancedb).

## The naive wiring, and where it breaks

The common recipe — write one LanceDB row per raw Unstructured element instead of per chunkset — breaks in a pair-specific way. Unstructured's element stream includes `Header`, `Footer`, and `PageNumber` types that are meant to be dropped, and it also includes elements whose image bytes were never captured, which POMA (or a hand-rolled equivalent) replaces with a visible `[IMG-{block_id}]` marker so the loss is never silent. Row-per-element ingestion turns both into ordinary table rows: the furniture pollutes the vector index the way overlap would, and — LanceDB's specific angle — if you then build a `create_fts_index(...)` for hybrid search over that same text column, the lexical index treats the literal bracketed marker as a real token, so a query that happens to share vocabulary with `[IMG-` fragments surfaces placeholder rows instead of content.

Row-per-chunkset ingestion avoids both: furniture is dropped before any row is written, and image markers only ever appear as counted metadata, never as embedded or full-text-indexed content.

## The pipeline, end to end

```bash
pip install 'unstructured[pdf]' lancedb
```

```python
from unstructured.partition.pdf import partition_pdf
from unstructured.staging.base import elements_to_json
import lancedb
from poma import PrimeCut
from poma.vektoria import assemble, records_from_archive, Volume
from poma.embeddings import get_embedder

# 1. Your existing Unstructured call — unchanged.
elements = partition_pdf(
    filename="contract.pdf",
    strategy="hi_res",
    infer_table_structure=True,
    extract_image_block_types=["Image"],
    extract_image_block_to_payload=True,
)
elements_to_json(elements, filename="elements.json")

# 2. The missing link — raw element list in, a .poma archive out.
client = PrimeCut()  # reads POMA_API_KEY
client.ingest("elements.json", download_dir="archives", filename="doc.poma")

embedder = get_embedder("local:BAAI/bge-small-en-v1.5")
vol = Volume("s3://your-bucket/poma")
db = lancedb.connect("s3://your-bucket/lancedb")

# 3. Content-free ingest — table holds only routing metadata; content lives on the volume.
records = records_from_archive("archives/doc.poma")  # -> list[Record] (id, text, payload)
rows = []
for r in records:
    vol.write_doc(r.payload["file_id"], {"file_id": r.payload["file_id"],
                                          "chunks": r.payload["chunks"], "text": r.text})
    rows.append({
        "id": r.id,
        "vector": embedder.embed([r.text])[0],
        "file_id": r.payload["file_id"],
        "chunkset_index": r.payload["chunkset_index"],
    })

tbl = db.create_table("poma", data=rows, exist_ok=True)

# 4. Vector search retrieval, then assemble prompt-ready context.
qv = embedder.embed(["early termination conditions"])[0]
results = tbl.search(qv).where("file_id = 'contract.pdf'").limit(10).to_list()
context = assemble(results, volume=vol)  # -> [{"file_id", "content"}, ...]
```

POMA validates the element list's shape up front (a corrupted or mislabeled upload 422s immediately), drops `Header`/`Footer`/`PageNumber` elements before any row is written, splices `Table` elements' `text_as_html` so rows survive on the volume, and rebuilds the cross-page heading tree before chunking. To force or suppress detection, set `external_ocr_source` to `"unstructured"` or `"none"`.

## Metadata mapping: POMA fields → LanceDB primitives

| POMA chunk field | LanceDB primitive | What it enables |
| --- | --- | --- |
| `to_embed` | `vector` column | ANN search over chunkset text |
| `file_id` | scalar column | `.where("file_id = '...'")` filtering |
| `chunkset_index` | scalar column | required by `assemble()`; returned by default |
| `page` (from Unstructured `metadata.page_number`) | optional scalar column | page-cited answers |
| `depth` (rebuilt from flat `Title` elements) | optional scalar column | hierarchy-aware `.where(...)` filtering |
| `[IMG-{block_id}]` offload markers (Unstructured) | excluded from any `create_fts_index(...)` build; kept only as counted metadata | keeps the lexical hybrid index from matching placeholder tokens as content |

## Frequently asked questions

### How do I get Unstructured.io elements into LanceDB for RAG?

Save the element list with `elements_to_json`, run `PrimeCut().ingest()` (auto-detected, hierarchy rebuilt), and write each chunkset as a row with a `vector` column plus `file_id`/`chunkset_index`. Retrieve with `search().to_list()` and hand the result to `assemble()`.

### Why not just embed Unstructured's raw element list directly into LanceDB?

Row-per-element ingestion admits `Header`/`Footer`/`PageNumber` noise and, for images with no captured bytes, the literal `[IMG-{block_id}]` placeholder marker as ordinary table content — polluting both vector similarity and any full-text index built over the same column.

### What LanceDB columns should Unstructured chunk data carry?

A `vector` column and scalar columns for `file_id`/`chunkset_index` at minimum, returned by default. Add `page` (from `page_number`) and `depth` as optional scalar columns for citation and hierarchy filtering.

### Do images from Unstructured survive into LanceDB retrieval?

Yes, when `extract_image_block_to_payload=true` keeps the bytes inline — POMA describes the figure and it's embedded into the row's vector like any other chunkset text. Images offloaded via `image_path` become a visible, counted marker instead of silent loss.

### Does LanceDB need any extra query flag for POMA assemble() to work, the way Redis or MongoDB Atlas do?

No — `search().to_list()` returns every column by default, so `assemble()` reads `file_id`/`chunkset_index` without a projection, `return_fields` call, or dialect setting, unlike Redis or MongoDB Atlas.

## Related recipes

Same parser, different store: [Unstructured.io → Chroma](/pipelines/unstructured-to-chroma) · [Unstructured.io → pgvector](/pipelines/unstructured-to-pgvector) · [Unstructured.io → Milvus](/pipelines/unstructured-to-milvus)

Foundations: [The Optimal Chunker for Unstructured.io](/optimal-chunker-unstructured) · [LanceDB Chunking Strategy for RAG](/optimal-chunks-lancedb) · [All pipeline recipes](/pipelines/) · [RAG chunking guide](/guides/rag-chunking/)