Source: http://www.poma-ai.com/docs/pipelines/unstructured-to-turbopuffer

# The Missing Link Between Unstructured.io and Optimal Retrieval in Turbopuffer

<ByAuthor />

**The short answer:** Unstructured.io gives you a typed, ordered element list; Turbopuffer gives you a namespace-per-tenant vector store with native hybrid ANN+BM25. Wired together with `chunk_by_title` and a direct row write, they still lose the document's hierarchy and, in Turbopuffer's content-free design, they lose it twice — once in the chunker, once in a namespace that was never told what a row means. The missing link is POMA: `PrimeCut().ingest()` consumes the raw element list, emits hierarchy-preserving [chunksets](/learn/chunking/chunksets) into a portable `.poma` archive, and `poma.vektoria` writes only `(id, vector, {file_id, chunkset_index})` to Turbopuffer while the actual content lives on a volume. Retrieval runs Turbopuffer's own query, then `assemble()` reassembles prompt-ready cheatsheets.

## What each end of the pipeline actually provides

**Unstructured.io** (`partition_pdf` or the hosted API) returns a flat, ordered list of typed elements — `Title`, `NarrativeText`, `ListItem`, `Table`, `Image` — each with `element_id`, `text`, and `metadata` (`page_number`, `text_as_html` for tables, optionally `image_base64`). Element order is document order; nothing records which `Title` nests under which. Details: [The Optimal Chunker for Unstructured.io](/optimal-chunker-unstructured).

**Turbopuffer** provides a **namespace** per tenant or corpus, an inline write-time schema, and native hybrid ranking (`ANN`, `SparseKNN`, `BM25`, server-side RRF via `multi_query()`). It has documented caps — 4 KiB per filterable attribute value, 8 MiB per attribute overall — and it ranks whatever you wrote, nothing more. Details: [Turbopuffer Chunking Strategy for RAG](/optimal-chunks-turbopuffer).

## The naive wiring, and where it breaks

The common recipe — run `chunk_by_title`, embed each fragment, `ns.write()` the fragment text straight into a filterable attribute — breaks in a pair-specific way:

- **`chunk_by_title` fragments carry no lineage above their nearest `Title`.** A `Table` element's `text_as_html` that spans a character-limit boundary gets split into two fragments, each written as its own Turbopuffer row with the same `file_id` but nothing linking them as one table.
- **Filterable attributes have an 8 MiB cap, but filterable *values* cap at 4 KiB.** A naive pipeline that marks the full fragment text filterable (to do substring or exact-match filtering later) can hit that cap on a table-heavy fragment, or silently make filtering slow on ones that fit.
- **Overlap, if the splitter still applies one, inflates the namespace.** Duplicate spans mean a larger index built over the same object storage, and `top_k` crowded with near-duplicate rows.

## The pipeline, end to end

```bash
pip install unstructured turbopuffer
```

```python
import os
from unstructured.partition.pdf import partition_pdf
from unstructured.staging.base import elements_to_json
from poma import PrimeCut
from poma.vektoria import assemble, records_from_archive, Volume
from poma.embeddings import get_embedder
import turbopuffer

# 1. Your existing Unstructured call — unchanged.
elements = partition_pdf(
    filename="contract.pdf",
    strategy="hi_res",
    infer_table_structure=True,            # populates metadata.text_as_html
    extract_image_block_types=["Image"],
    extract_image_block_to_payload=True,   # inline base64 → POMA can describe figures
)
elements_to_json(elements, filename="contract.unstructured.json")

# 2. The missing link — raw element list in, a portable .poma archive out.
client = PrimeCut()  # reads POMA_API_KEY
client.ingest("contract.unstructured.json", download_dir="archives", filename="contract.poma")

embedder = get_embedder("local:BAAI/bge-small-en-v1.5")
vol = Volume("s3://your-bucket/poma")
tpuf = turbopuffer.Turbopuffer(api_key=os.environ["TURBOPUFFER_API_KEY"], region="gcp-us-central1")
ns = tpuf.namespace("contracts")

# 3. Content-free ingest — the namespace holds only routing metadata; content lives on the volume.
records = records_from_archive("archives/contract.poma")  # -> list[Record] (id, text, payload)
rows = []
for r in records:
    vol.write_doc(r.payload["file_id"], {"file_id": r.payload["file_id"],
                                          "chunks": r.payload["chunks"], "text": r.text})
    rows.append({
        "id": r.id,
        "vector": embedder.embed([r.text])[0],
        "file_id": r.payload["file_id"],
        "chunkset_index": r.payload["chunkset_index"],
    })
ns.write(
    upsert_rows=rows,
    distance_metric="cosine_distance",
    schema={"file_id": {"type": "string"}, "chunkset_index": {"type": "uint"}},
)

# 4. Retrieval — Turbopuffer's normal ANN query, then assemble.
qv = embedder.embed(["early termination conditions"])[0]
res = ns.query(rank_by=("vector", "ANN", qv), top_k=10,
               include_attributes=["file_id", "chunkset_index"])
context = assemble(res, volume=vol)  # -> [{"file_id", "content"}, ...]
```

`include_attributes=["file_id", "chunkset_index"]` is not optional here — without it, Turbopuffer returns bare rows and `assemble()` has nothing to fetch. Every chunkset id is deterministic (`chunkset_uuid(file_id, chunkset_index)`), so re-ingesting the same document lands on the same rows across every connector, not just Turbopuffer.

## Metadata mapping: POMA fields → Turbopuffer primitives

| POMA field (Unstructured-sourced) | Turbopuffer primitive | Notes |
| --- | --- | --- |
| chunkset id (`chunkset_uuid`) | `id` (string) | row identity, same id on re-ingest |
| `to_embed` (tables spliced as `text_as_html`) | `vector` | embedded before write, never stored as text |
| `file_id` | attribute, `type: string` | requested back via `include_attributes` for `assemble()` |
| `chunkset_index` | attribute, `type: uint` | requested back via `include_attributes` for `assemble()` |
| `page` (from `metadata.page_number`), `depth`, `chunk_index`, full chunkset text | volume document | content-free — never a Turbopuffer attribute, fetched by `assemble()` |

## Frequently asked questions

### How do I get Unstructured.io elements into Turbopuffer for RAG?

Save the element list with `elements_to_json`, ingest it with `PrimeCut`, and keep the `.poma` archive. Build Turbopuffer rows (`id`, `vector`, `file_id`, `chunkset_index`) from `records_from_archive`, write chunkset content to a volume, then retrieve with a normal `ns.query()` plus `assemble(res, volume=vol)`.

### Why does chunk_by_title break in a namespace-per-tenant Turbopuffer design?

`chunk_by_title` fragments carry no lineage above their nearest `Title`, so a `Table` element split across two fragments becomes two Turbopuffer rows with no field linking them. A namespace-filtered query can return one half without the other.

### What Turbopuffer attributes should Unstructured chunks carry?

`file_id` and `chunkset_index`, both compact and requested via `include_attributes` at query time. Page, depth, chunk index, and the full text stay off the row and live on the volume — the 4 KiB filterable-value cap never comes into play.

### Do Unstructured's inline images survive into Turbopuffer retrieval?

Yes, if `extract_image_block_to_payload=true` was set — POMA describes the figure and the description becomes ordinary embedded, volume-stored text. Disk-based `image_path` images are neutralized and counted, never silently dropped.

### Does Turbopuffer's hybrid ANN+BM25 help with Unstructured's typed elements?

Only if you add a `full_text_search`-enabled text attribute to the schema — the content-free pattern above embeds vectors without writing chunkset text into Turbopuffer. Add it and fuse with `rerank_by=("RRF",)` via `multi_query()` for exact-term hybrid on clause numbers and defined terms.

## Related recipes

Same parser, different store: [Unstructured.io → Vespa](/pipelines/unstructured-to-vespa) · [Unstructured.io → Elasticsearch](/pipelines/unstructured-to-elasticsearch) · [Unstructured.io → Qdrant](/pipelines/unstructured-to-qdrant)

Foundations: [The Optimal Chunker for Unstructured.io](/optimal-chunker-unstructured) · [Turbopuffer Chunking Strategy for RAG](/optimal-chunks-turbopuffer) · [All pipeline recipes](/pipelines/) · [RAG chunking guide](/guides/rag-chunking/)