Unstructured.io to Turbopuffer: Chunking for RAG
The short answer: Unstructured.io gives you a typed, ordered element list; Turbopuffer gives you a namespace-per-tenant vector store with native hybrid ANN+BM25. Wired together with chunk_by_title and a direct row write, they still lose the document's hierarchy and, in Turbopuffer's content-free design, they lose it twice — once in the chunker, once in a namespace that was never told what a row means. The missing link is POMA: PrimeCut().ingest() consumes the raw element list, emits hierarchy-preserving chunksets into a portable .poma archive, and poma.vektoria writes only (id, vector, {file_id, chunkset_index}) to Turbopuffer while the actual content lives on a volume. Retrieval runs Turbopuffer's own query, then assemble() reassembles prompt-ready cheatsheets.
What each end of the pipeline actually provides
Unstructured.io (partition_pdf or the hosted API) returns a flat, ordered list of typed elements — Title, NarrativeText, ListItem, Table, Image — each with element_id, text, and metadata (page_number, text_as_html for tables, optionally image_base64). Element order is document order; nothing records which Title nests under which. Details: The Optimal Chunker for Unstructured.io.
Turbopuffer provides a namespace per tenant or corpus, an inline write-time schema, and native hybrid ranking (ANN, SparseKNN, BM25, server-side RRF via multi_query()). It has documented caps — 4 KiB per filterable attribute value, 8 MiB per attribute overall — and it ranks whatever you wrote, nothing more. Details: Turbopuffer Chunking Strategy for RAG.
The naive wiring, and where it breaks
The common recipe — run chunk_by_title, embed each fragment, ns.write() the fragment text straight into a filterable attribute — breaks in a pair-specific way:
chunk_by_titlefragments carry no lineage above their nearestTitle. ATableelement'stext_as_htmlthat spans a character-limit boundary gets split into two fragments, each written as its own Turbopuffer row with the samefile_idbut nothing linking them as one table.- Filterable attributes have an 8 MiB cap, but filterable values cap at 4 KiB. A naive pipeline that marks the full fragment text filterable (to do substring or exact-match filtering later) can hit that cap on a table-heavy fragment, or silently make filtering slow on ones that fit.
- Overlap, if the splitter still applies one, inflates the namespace. Duplicate spans mean a larger index built over the same object storage, and
top_kcrowded with near-duplicate rows.
The pipeline, end to end
pip install unstructured turbopufferimport os
from unstructured.partition.pdf import partition_pdf
from unstructured.staging.base import elements_to_json
from poma import PrimeCut
from poma.vektoria import assemble, records_from_archive, Volume
from poma.embeddings import get_embedder
import turbopuffer
# 1. Your existing Unstructured call — unchanged.
elements = partition_pdf(
filename="contract.pdf",
strategy="hi_res",
infer_table_structure=True, # populates metadata.text_as_html
extract_image_block_types=["Image"],
extract_image_block_to_payload=True, # inline base64 → POMA can describe figures
)
elements_to_json(elements, filename="contract.unstructured.json")
# 2. The missing link — raw element list in, a portable .poma archive out.
client = PrimeCut() # reads POMA_API_KEY
client.ingest("contract.unstructured.json", download_dir="archives", filename="contract.poma")
embedder = get_embedder("local:BAAI/bge-small-en-v1.5")
vol = Volume("s3://your-bucket/poma")
tpuf = turbopuffer.Turbopuffer(api_key=os.environ["TURBOPUFFER_API_KEY"], region="gcp-us-central1")
ns = tpuf.namespace("contracts")
# 3. Content-free ingest — the namespace holds only routing metadata; content lives on the volume.
records = records_from_archive("archives/contract.poma") # -> list[Record] (id, text, payload)
rows = []
for r in records:
vol.write_doc(r.payload["file_id"], {"file_id": r.payload["file_id"],
"chunks": r.payload["chunks"], "text": r.text})
rows.append({
"id": r.id,
"vector": embedder.embed([r.text])[0],
"file_id": r.payload["file_id"],
"chunkset_index": r.payload["chunkset_index"],
})
ns.write(
upsert_rows=rows,
distance_metric="cosine_distance",
schema={"file_id": {"type": "string"}, "chunkset_index": {"type": "uint"}},
)
# 4. Retrieval — Turbopuffer's normal ANN query, then assemble.
qv = embedder.embed(["early termination conditions"])[0]
res = ns.query(rank_by=("vector", "ANN", qv), top_k=10,
include_attributes=["file_id", "chunkset_index"])
context = assemble(res, volume=vol) # -> [{"file_id", "content"}, ...]include_attributes=["file_id", "chunkset_index"] is not optional here — without it, Turbopuffer returns bare rows and assemble() has nothing to fetch. Every chunkset id is deterministic (chunkset_uuid(file_id, chunkset_index)), so re-ingesting the same document lands on the same rows across every connector, not just Turbopuffer.
Metadata mapping: POMA fields → Turbopuffer primitives
| POMA field (Unstructured-sourced) | Turbopuffer primitive | Notes |
|---|---|---|
chunkset id (chunkset_uuid) | id (string) | row identity, same id on re-ingest |
to_embed (tables spliced as text_as_html) | vector | embedded before write, never stored as text |
file_id | attribute, type: string | requested back via include_attributes for assemble() |
chunkset_index | attribute, type: uint | requested back via include_attributes for assemble() |
page (from metadata.page_number), depth, chunk_index, full chunkset text | volume document | content-free — never a Turbopuffer attribute, fetched by assemble() |
Frequently asked questions
How do I get Unstructured.io elements into Turbopuffer for RAG?
Save the element list with elements_to_json, ingest it with PrimeCut, and keep the .poma archive. Build Turbopuffer rows (id, vector, file_id, chunkset_index) from records_from_archive, write chunkset content to a volume, then retrieve with a normal ns.query() plus assemble(res, volume=vol).
Why does chunk_by_title break in a namespace-per-tenant Turbopuffer design?
chunk_by_title fragments carry no lineage above their nearest Title, so a Table element split across two fragments becomes two Turbopuffer rows with no field linking them. A namespace-filtered query can return one half without the other.
What Turbopuffer attributes should Unstructured chunks carry?
file_id and chunkset_index, both compact and requested via include_attributes at query time. Page, depth, chunk index, and the full text stay off the row and live on the volume — the 4 KiB filterable-value cap never comes into play.
Do Unstructured's inline images survive into Turbopuffer retrieval?
Yes, if extract_image_block_to_payload=true was set — POMA describes the figure and the description becomes ordinary embedded, volume-stored text. Disk-based image_path images are neutralized and counted, never silently dropped.
Does Turbopuffer's hybrid ANN+BM25 help with Unstructured's typed elements?
Only if you add a full_text_search-enabled text attribute to the schema — the content-free pattern above embeds vectors without writing chunkset text into Turbopuffer. Add it and fuse with rerank_by=("RRF",) via multi_query() for exact-term hybrid on clause numbers and defined terms.
Related recipes
Same parser, different store: Unstructured.io → Vespa · Unstructured.io → Elasticsearch · Unstructured.io → Qdrant
Foundations: The Optimal Chunker for Unstructured.io · Turbopuffer Chunking Strategy for RAG · All pipeline recipes · RAG chunking guide