Unstructured.io to LanceDB: Chunking for RAG
The short answer: Unstructured.io gives you a typed, ordered element list; LanceDB gives you an embedded, Arrow-backed table with SQL-style filtering and no server to operate. Wired together naively — join element text, split, embed, write rows — they still produce mediocre RAG, because nothing in between rebuilds the hierarchy Unstructured's typing implies but never states. The missing link is POMA: PrimeCut().ingest() consumes the raw element list (auto-detected), emits hierarchy-preserving chunksets, and vektoria's content-free ingest pattern writes them into a LanceDB table with the columns retrieval needs. Retrieval comes back through assemble() as prompt-ready cheatsheets.
What each end of the pipeline actually provides
Unstructured.io (partition_pdf or the hosted API) returns a flat, ordered list of typed elements — Title, NarrativeText, ListItem, Table, Image, plus page furniture types — each carrying a stable element_id and metadata such as page_number and text_as_html. What it doesn't provide: any record of which Title nests under which, or a retrieval-ready unit. Details: The Optimal Chunker for Unstructured.io.
LanceDB provides an Arrow/Lance columnar table — vector as one column among ordinary scalar columns, .where(...) for SQL-like predicates, and an optional create_fts_index(...) for a lexical signal — embedded in-process or backed by object storage. What it doesn't provide: any opinion about what a row's text should represent, or filtering of what you write into the full-text index. Details: LanceDB Chunking Strategy for RAG.
The naive wiring, and where it breaks
The common recipe — write one LanceDB row per raw Unstructured element instead of per chunkset — breaks in a pair-specific way. Unstructured's element stream includes Header, Footer, and PageNumber types that are meant to be dropped, and it also includes elements whose image bytes were never captured, which POMA (or a hand-rolled equivalent) replaces with a visible [IMG-{block_id}] marker so the loss is never silent. Row-per-element ingestion turns both into ordinary table rows: the furniture pollutes the vector index the way overlap would, and — LanceDB's specific angle — if you then build a create_fts_index(...) for hybrid search over that same text column, the lexical index treats the literal bracketed marker as a real token, so a query that happens to share vocabulary with [IMG- fragments surfaces placeholder rows instead of content.
Row-per-chunkset ingestion avoids both: furniture is dropped before any row is written, and image markers only ever appear as counted metadata, never as embedded or full-text-indexed content.
The pipeline, end to end
pip install 'unstructured[pdf]' lancedbfrom unstructured.partition.pdf import partition_pdf
from unstructured.staging.base import elements_to_json
import lancedb
from poma import PrimeCut
from poma.vektoria import assemble, records_from_archive, Volume
from poma.embeddings import get_embedder
# 1. Your existing Unstructured call — unchanged.
elements = partition_pdf(
filename="contract.pdf",
strategy="hi_res",
infer_table_structure=True,
extract_image_block_types=["Image"],
extract_image_block_to_payload=True,
)
elements_to_json(elements, filename="elements.json")
# 2. The missing link — raw element list in, a .poma archive out.
client = PrimeCut() # reads POMA_API_KEY
client.ingest("elements.json", download_dir="archives", filename="doc.poma")
embedder = get_embedder("local:BAAI/bge-small-en-v1.5")
vol = Volume("s3://your-bucket/poma")
db = lancedb.connect("s3://your-bucket/lancedb")
# 3. Content-free ingest — table holds only routing metadata; content lives on the volume.
records = records_from_archive("archives/doc.poma") # -> list[Record] (id, text, payload)
rows = []
# assemble() reads the FULL document (chunks + chunksets) from the volume — write it once,
# not one {"chunks": <indices>, "text"} stub per record (which also overwrote itself each turn).
from poma.utils import unpack_poma_archive
_data = unpack_poma_archive(poma_archive_path="archives/doc.poma")
file_id = records[0].payload["file_id"]
vol.write_doc(file_id, {"file_id": file_id, "chunks": _data["chunks"], "chunksets": _data["chunksets"]})
for r in records:
rows.append({
"id": r.id,
"vector": embedder.embed([r.text])[0],
"file_id": r.payload["file_id"],
"chunkset_index": r.payload["chunkset_index"],
})
tbl = db.create_table("poma", data=rows, exist_ok=True)
# 4. Vector search retrieval, then assemble prompt-ready context.
qv = embedder.embed(["early termination conditions"])[0]
results = tbl.search(qv).where(f"file_id = '{file_id}'").limit(10).to_list() # POMA assigns an opaque file_id, not the file name
context = assemble(results, volume=vol) # -> [{"file_id", "content"}, ...]POMA validates the element list's shape up front (a corrupted or mislabeled upload 422s immediately), drops Header/Footer/PageNumber elements before any row is written, splices Table elements' text_as_html so rows survive on the volume, and rebuilds the cross-page heading tree before chunking. To force or suppress detection, set external_ocr_source to "unstructured" or "none".
Metadata mapping: POMA fields → LanceDB primitives
| POMA chunk field | LanceDB primitive | What it enables |
|---|---|---|
to_embed | vector column | ANN search over chunkset text |
file_id | scalar column | .where("file_id = '...'") filtering |
chunkset_index | scalar column | required by assemble(); returned by default |
page (from Unstructured metadata.page_number) | optional scalar column | page-cited answers |
depth (rebuilt from flat Title elements) | optional scalar column | hierarchy-aware .where(...) filtering |
[IMG-{block_id}] offload markers (Unstructured) | excluded from any create_fts_index(...) build; kept only as counted metadata | keeps the lexical hybrid index from matching placeholder tokens as content |
Frequently asked questions
How do I get Unstructured.io elements into LanceDB for RAG?
Save the element list with elements_to_json, run PrimeCut().ingest() (auto-detected, hierarchy rebuilt), and write each chunkset as a row with a vector column plus file_id/chunkset_index. Retrieve with search().to_list() and hand the result to assemble().
Why not just embed Unstructured's raw element list directly into LanceDB?
Row-per-element ingestion admits Header/Footer/PageNumber noise and, for images with no captured bytes, the literal [IMG-{block_id}] placeholder marker as ordinary table content — polluting both vector similarity and any full-text index built over the same column.
What LanceDB columns should Unstructured chunk data carry?
A vector column and scalar columns for file_id/chunkset_index at minimum, returned by default. Add page (from page_number) and depth as optional scalar columns for citation and hierarchy filtering.
Do images from Unstructured survive into LanceDB retrieval?
Yes, when extract_image_block_to_payload=true keeps the bytes inline — POMA describes the figure and it's embedded into the row's vector like any other chunkset text. Images offloaded via image_path become a visible, counted marker instead of silent loss.
Does LanceDB need any extra query flag for POMA assemble() to work, the way Redis or MongoDB Atlas do?
No — search().to_list() returns every column by default, so assemble() reads file_id/chunkset_index without a projection, return_fields call, or dialect setting, unlike Redis or MongoDB Atlas.
Related recipes
Same parser, different store: Unstructured.io → Chroma · Unstructured.io → pgvector · Unstructured.io → Milvus
Foundations: The Optimal Chunker for Unstructured.io · LanceDB Chunking Strategy for RAG · All pipeline recipes · RAG chunking guide