Unstructured.io to Vespa: Chunking for RAG
The short answer: Unstructured.io gives you a typed, ordered element list; Vespa gives you one system that natively fuses BM25 and ANN over documents you define. Wired together with chunk_by_title and a flat document schema, they still lose the hierarchy Unstructured recovered, and Vespa's rank profile has no field to recover it from. The missing link is POMA: PrimeCut().ingest() consumes the raw element list, emits hierarchy-preserving chunksets into a portable .poma archive, and poma.vektoria feeds Vespa only (id, embedding, {file_id, chunkset_index}) while the actual content lives on a volume. Retrieval runs Vespa's own YQL query, then assemble() reassembles prompt-ready cheatsheets.
What each end of the pipeline actually provides
Unstructured.io (partition_pdf or the hosted API) returns a flat, ordered list of typed elements — Title, NarrativeText, ListItem, Table, Image — each with element_id, text, and metadata (page_number, text_as_html for tables, optionally image_base64). Element order is document order; nothing records which Title nests under which. Details: The Optimal Chunker for Unstructured.io.
Vespa stores vectors as ordinary tensor fields alongside scalar attributes in the same document type, with index: enable-bm25 for native lexical scoring and a rank profile that fuses both in one query. It has no separate "vector store" object, and its ranking is only as good as the fields you declared and populated. Details: Vespa Chunking Strategy for RAG.
The naive wiring, and where it breaks
The common recipe — run chunk_by_title, feed one Vespa document per fragment with just text and embedding — breaks in a pair-specific way:
chunk_by_titlefragments carry no lineage above their nearestTitle. ATableelement split across a character-limit boundary becomes two Vespa documents that share afile_id(if you even declared one) but nothing else — nodepth, no ordering field, no way for a rank profile to know they were one table.- Vespa's schema is declared upfront, and naive pipelines under-declare it. Skipping
file_id/hierarchy attributes becausechunk_by_titlenever surfaced them means queries can't filter by document or section — every fragment from every tenant sits in one undifferentiated pool. - Multi-phase ranking scores documents independently.
closeness(field, embedding) * (1 + bm25(text))ranks each fragment on its own merits; it cannot reconstruct a table's missing half because Vespa was never told the two fragments were related.
The pipeline, end to end
pip install unstructured pyvespafrom unstructured.partition.pdf import partition_pdf
from unstructured.staging.base import elements_to_json
from poma import PrimeCut
from poma.vektoria import assemble, records_from_archive, Volume
from poma.embeddings import get_embedder
from vespa.package import ApplicationPackage, Field, Schema, Document, HNSW, RankProfile
from vespa.deployment import VespaDocker
# 1. Your existing Unstructured call — unchanged.
elements = partition_pdf(
filename="contract.pdf",
strategy="hi_res",
infer_table_structure=True, # populates metadata.text_as_html
extract_image_block_types=["Image"],
extract_image_block_to_payload=True, # inline base64 → POMA can describe figures
)
elements_to_json(elements, filename="contract.unstructured.json")
# 2. The missing link — raw element list in, a portable .poma archive out.
client = PrimeCut() # reads POMA_API_KEY
client.ingest("contract.unstructured.json", download_dir="archives", filename="contract.poma")
embedder = get_embedder("local:BAAI/bge-small-en-v1.5")
vol = Volume("s3://your-bucket/poma")
# 3. One document type carrying only the embedding and the routing attributes.
package = ApplicationPackage(
name="contracts",
schema=[Schema(
name="chunkset",
document=Document(fields=[
Field(name="file_id", type="string", indexing=["attribute"]),
Field(name="chunkset_index", type="int", indexing=["attribute"]),
Field(
name="embedding", type=f"tensor<float>(x[{embedder.dims}])",
indexing=["attribute", "index"],
ann=HNSW(distance_metric="angular"),
),
]),
rank_profiles=[RankProfile(
name="poma_rank",
inputs=[("query(q)", f"tensor<float>(x[{embedder.dims}])")],
first_phase="closeness(field, embedding)",
)],
)],
)
app = VespaDocker().deploy(application_package=package)
records = records_from_archive("archives/contract.poma") # -> list[Record] (id, text, payload)
docs = []
# assemble() reads the FULL document (chunks + chunksets) from the volume — write it once,
# not one {"chunks": <indices>, "text"} stub per record (which also overwrote itself each turn).
from poma.utils import unpack_poma_archive
_data = unpack_poma_archive(poma_archive_path="archives/contract.poma")
file_id = records[0].payload["file_id"]
vol.write_doc(file_id, {"file_id": file_id, "chunks": _data["chunks"], "chunksets": _data["chunksets"]})
for r in records:
docs.append({"id": r.id, "fields": {
"file_id": r.payload["file_id"],
"chunkset_index": r.payload["chunkset_index"],
"embedding": embedder.embed([r.text])[0],
}})
app.feed_iterable(docs, schema="chunkset", namespace="contracts")
# 4. Retrieval — select the routing fields explicitly in YQL, then assemble.
qv = embedder.embed(["early termination conditions"])[0]
resp = app.query(
yql="select file_id, chunkset_index from chunkset where " # the page deploys schema `chunkset`, not `poma`
"{targetHits:100}nearestNeighbor(embedding, q) limit 10",
ranking="poma_rank",
body={"input.query(q)": qv},
)
context = assemble(resp, volume=vol) # -> [{"file_id", "content"}, ...]Selecting file_id, chunkset_index explicitly in the YQL select clause is not optional — Vespa's select * would work too, but assemble() only needs the routing fields, and explicit selection is Vespa's documented way to keep the query lean. Every chunkset id is deterministic, so re-feeding the same document lands on the same document ids.
Metadata mapping: POMA fields → Vespa primitives
| POMA field (Unstructured-sourced) | Vespa primitive | Notes |
|---|---|---|
chunkset id (chunkset_uuid) | document id | stable across re-feeds |
to_embed (tables spliced as text_as_html) | tensor<float>(x[N]) field, HNSW-indexed | embedded before feed, never stored as text |
file_id | attribute field, string | selected explicitly in YQL for assemble() |
chunkset_index | attribute field, int | selected explicitly in YQL for assemble() |
page (from metadata.page_number), depth, chunk_index, full chunkset text | volume document | content-free — never a Vespa field, fetched by assemble() |
Frequently asked questions
How do I get Unstructured.io elements into Vespa for RAG?
Save the element list with elements_to_json, ingest it with PrimeCut, and keep the .poma archive. Declare a chunkset document type with file_id/chunkset_index attributes and an embedding tensor, feed it via feed_iterable, write chunkset content to a volume, then retrieve with a YQL query that selects the routing fields plus assemble(resp, volume=vol).
Why does chunk_by_title fragmentation cause problems in a Vespa schema?
chunk_by_title fragments carry no lineage above their nearest Title, so a Table element split across a boundary becomes two documents with nothing linking them. Vespa's multi-phase ranking scores each independently and cannot recover the missing half.
What Vespa schema fields should Unstructured chunks carry?
file_id and chunkset_index as attribute fields, plus a tensor field with an HNSW index for the embedding. Page, depth, chunk index, and full text stay off the document and live on a volume.
Do Unstructured's tables (text_as_html) survive into Vespa's BM25 field?
In this content-free pattern, chunkset text is embedded and volume-stored, not indexed as BM25 — Vespa only sees the vector and routing attributes. Add a string field with index: enable-bm25 populated from the same to_embed text if you need native lexical ranking too.
Should I use Vespa streaming mode for Unstructured-parsed multi-tenant documents?
Streaming mode's grouped document IDs fit small per-tenant corpora queried one tenant at a time; indexed mode fits queries spanning many tenants. Vespa's guidance is not to mix both modes in one content cluster.
Related recipes
Same parser, different store: Unstructured.io → Turbopuffer · Unstructured.io → OpenSearch · Unstructured.io → Weaviate
Foundations: The Optimal Chunker for Unstructured.io · Vespa Chunking Strategy for RAG · All pipeline recipes · RAG chunking guide