Skip to content

Unstructured.io to Turbopuffer: Chunking for RAG ​

The short answer: Unstructured.io gives you a typed, ordered element list; Turbopuffer gives you a namespace-per-tenant vector store with native hybrid ANN+BM25. Wired together with chunk_by_title and a direct row write, they still lose the document's hierarchy and, in Turbopuffer's content-free design, they lose it twice — once in the chunker, once in a namespace that was never told what a row means. The missing link is POMA: PrimeCut().ingest() consumes the raw element list, emits hierarchy-preserving chunksets into a portable .poma archive, and poma.vektoria writes only (id, vector, {file_id, chunkset_index}) to Turbopuffer while the actual content lives on a volume. Retrieval runs Turbopuffer's own query, then assemble() reassembles prompt-ready cheatsheets.

What each end of the pipeline actually provides ​

Unstructured.io (partition_pdf or the hosted API) returns a flat, ordered list of typed elements — Title, NarrativeText, ListItem, Table, Image — each with element_id, text, and metadata (page_number, text_as_html for tables, optionally image_base64). Element order is document order; nothing records which Title nests under which. Details: The Optimal Chunker for Unstructured.io.

Turbopuffer provides a namespace per tenant or corpus, an inline write-time schema, and native hybrid ranking (ANN, SparseKNN, BM25, server-side RRF via multi_query()). It has documented caps — 4 KiB per filterable attribute value, 8 MiB per attribute overall — and it ranks whatever you wrote, nothing more. Details: Turbopuffer Chunking Strategy for RAG.

The naive wiring, and where it breaks ​

The common recipe — run chunk_by_title, embed each fragment, ns.write() the fragment text straight into a filterable attribute — breaks in a pair-specific way:

  • chunk_by_title fragments carry no lineage above their nearest Title. A Table element's text_as_html that spans a character-limit boundary gets split into two fragments, each written as its own Turbopuffer row with the same file_id but nothing linking them as one table.
  • Filterable attributes have an 8 MiB cap, but filterable values cap at 4 KiB. A naive pipeline that marks the full fragment text filterable (to do substring or exact-match filtering later) can hit that cap on a table-heavy fragment, or silently make filtering slow on ones that fit.
  • Overlap, if the splitter still applies one, inflates the namespace. Duplicate spans mean a larger index built over the same object storage, and top_k crowded with near-duplicate rows.

The pipeline, end to end ​

bash
pip install unstructured turbopuffer
python
import os
from unstructured.partition.pdf import partition_pdf
from unstructured.staging.base import elements_to_json
from poma import PrimeCut
from poma.vektoria import assemble, records_from_archive, Volume
from poma.embeddings import get_embedder
import turbopuffer

# 1. Your existing Unstructured call — unchanged.
elements = partition_pdf(
    filename="contract.pdf",
    strategy="hi_res",
    infer_table_structure=True,            # populates metadata.text_as_html
    extract_image_block_types=["Image"],
    extract_image_block_to_payload=True,   # inline base64 → POMA can describe figures
)
elements_to_json(elements, filename="contract.unstructured.json")

# 2. The missing link — raw element list in, a portable .poma archive out.
client = PrimeCut()  # reads POMA_API_KEY
client.ingest("contract.unstructured.json", download_dir="archives", filename="contract.poma")

embedder = get_embedder("local:BAAI/bge-small-en-v1.5")
vol = Volume("s3://your-bucket/poma")
tpuf = turbopuffer.Turbopuffer(api_key=os.environ["TURBOPUFFER_API_KEY"], region="gcp-us-central1")
ns = tpuf.namespace("contracts")

# 3. Content-free ingest — the namespace holds only routing metadata; content lives on the volume.
records = records_from_archive("archives/contract.poma")  # -> list[Record] (id, text, payload)
rows = []
for r in records:
    vol.write_doc(r.payload["file_id"], {"file_id": r.payload["file_id"],
                                          "chunks": r.payload["chunks"], "text": r.text})
    rows.append({
        "id": r.id,
        "vector": embedder.embed([r.text])[0],
        "file_id": r.payload["file_id"],
        "chunkset_index": r.payload["chunkset_index"],
    })
ns.write(
    upsert_rows=rows,
    distance_metric="cosine_distance",
    schema={"file_id": {"type": "string"}, "chunkset_index": {"type": "uint"}},
)

# 4. Retrieval — Turbopuffer's normal ANN query, then assemble.
qv = embedder.embed(["early termination conditions"])[0]
res = ns.query(rank_by=("vector", "ANN", qv), top_k=10,
               include_attributes=["file_id", "chunkset_index"])
context = assemble(res, volume=vol)  # -> [{"file_id", "content"}, ...]

include_attributes=["file_id", "chunkset_index"] is not optional here — without it, Turbopuffer returns bare rows and assemble() has nothing to fetch. Every chunkset id is deterministic (chunkset_uuid(file_id, chunkset_index)), so re-ingesting the same document lands on the same rows across every connector, not just Turbopuffer.

Metadata mapping: POMA fields → Turbopuffer primitives ​

POMA field (Unstructured-sourced)Turbopuffer primitiveNotes
chunkset id (chunkset_uuid)id (string)row identity, same id on re-ingest
to_embed (tables spliced as text_as_html)vectorembedded before write, never stored as text
file_idattribute, type: stringrequested back via include_attributes for assemble()
chunkset_indexattribute, type: uintrequested back via include_attributes for assemble()
page (from metadata.page_number), depth, chunk_index, full chunkset textvolume documentcontent-free — never a Turbopuffer attribute, fetched by assemble()

Frequently asked questions ​

How do I get Unstructured.io elements into Turbopuffer for RAG? ​

Save the element list with elements_to_json, ingest it with PrimeCut, and keep the .poma archive. Build Turbopuffer rows (id, vector, file_id, chunkset_index) from records_from_archive, write chunkset content to a volume, then retrieve with a normal ns.query() plus assemble(res, volume=vol).

Why does chunk_by_title break in a namespace-per-tenant Turbopuffer design? ​

chunk_by_title fragments carry no lineage above their nearest Title, so a Table element split across two fragments becomes two Turbopuffer rows with no field linking them. A namespace-filtered query can return one half without the other.

What Turbopuffer attributes should Unstructured chunks carry? ​

file_id and chunkset_index, both compact and requested via include_attributes at query time. Page, depth, chunk index, and the full text stay off the row and live on the volume — the 4 KiB filterable-value cap never comes into play.

Do Unstructured's inline images survive into Turbopuffer retrieval? ​

Yes, if extract_image_block_to_payload=true was set — POMA describes the figure and the description becomes ordinary embedded, volume-stored text. Disk-based image_path images are neutralized and counted, never silently dropped.

Does Turbopuffer's hybrid ANN+BM25 help with Unstructured's typed elements? ​

Only if you add a full_text_search-enabled text attribute to the schema — the content-free pattern above embeds vectors without writing chunkset text into Turbopuffer. Add it and fuse with rerank_by=("RRF",) via multi_query() for exact-term hybrid on clause numbers and defined terms.

Same parser, different store: Unstructured.io → Vespa · Unstructured.io → Elasticsearch · Unstructured.io → Qdrant

Foundations: The Optimal Chunker for Unstructured.io · Turbopuffer Chunking Strategy for RAG · All pipeline recipes · RAG chunking guide