Skip to content

Unstructured.io to Vespa: Chunking for RAG ​

The short answer: Unstructured.io gives you a typed, ordered element list; Vespa gives you one system that natively fuses BM25 and ANN over documents you define. Wired together with chunk_by_title and a flat document schema, they still lose the hierarchy Unstructured recovered, and Vespa's rank profile has no field to recover it from. The missing link is POMA: PrimeCut().ingest() consumes the raw element list, emits hierarchy-preserving chunksets into a portable .poma archive, and poma.vektoria feeds Vespa only (id, embedding, {file_id, chunkset_index}) while the actual content lives on a volume. Retrieval runs Vespa's own YQL query, then assemble() reassembles prompt-ready cheatsheets.

What each end of the pipeline actually provides ​

Unstructured.io (partition_pdf or the hosted API) returns a flat, ordered list of typed elements — Title, NarrativeText, ListItem, Table, Image — each with element_id, text, and metadata (page_number, text_as_html for tables, optionally image_base64). Element order is document order; nothing records which Title nests under which. Details: The Optimal Chunker for Unstructured.io.

Vespa stores vectors as ordinary tensor fields alongside scalar attributes in the same document type, with index: enable-bm25 for native lexical scoring and a rank profile that fuses both in one query. It has no separate "vector store" object, and its ranking is only as good as the fields you declared and populated. Details: Vespa Chunking Strategy for RAG.

The naive wiring, and where it breaks ​

The common recipe — run chunk_by_title, feed one Vespa document per fragment with just text and embedding — breaks in a pair-specific way:

  • chunk_by_title fragments carry no lineage above their nearest Title. A Table element split across a character-limit boundary becomes two Vespa documents that share a file_id (if you even declared one) but nothing else — no depth, no ordering field, no way for a rank profile to know they were one table.
  • Vespa's schema is declared upfront, and naive pipelines under-declare it. Skipping file_id/hierarchy attributes because chunk_by_title never surfaced them means queries can't filter by document or section — every fragment from every tenant sits in one undifferentiated pool.
  • Multi-phase ranking scores documents independently. closeness(field, embedding) * (1 + bm25(text)) ranks each fragment on its own merits; it cannot reconstruct a table's missing half because Vespa was never told the two fragments were related.

The pipeline, end to end ​

bash
pip install unstructured pyvespa
python
from unstructured.partition.pdf import partition_pdf
from unstructured.staging.base import elements_to_json
from poma import PrimeCut
from poma.vektoria import assemble, records_from_archive, Volume
from poma.embeddings import get_embedder
from vespa.package import ApplicationPackage, Field, Schema, Document, HNSW, RankProfile
from vespa.deployment import VespaDocker

# 1. Your existing Unstructured call — unchanged.
elements = partition_pdf(
    filename="contract.pdf",
    strategy="hi_res",
    infer_table_structure=True,            # populates metadata.text_as_html
    extract_image_block_types=["Image"],
    extract_image_block_to_payload=True,   # inline base64 → POMA can describe figures
)
elements_to_json(elements, filename="contract.unstructured.json")

# 2. The missing link — raw element list in, a portable .poma archive out.
client = PrimeCut()  # reads POMA_API_KEY
client.ingest("contract.unstructured.json", download_dir="archives", filename="contract.poma")

embedder = get_embedder("local:BAAI/bge-small-en-v1.5")
vol = Volume("s3://your-bucket/poma")

# 3. One document type carrying only the embedding and the routing attributes.
package = ApplicationPackage(
    name="contracts",
    schema=[Schema(
        name="chunkset",
        document=Document(fields=[
            Field(name="file_id", type="string", indexing=["attribute"]),
            Field(name="chunkset_index", type="int", indexing=["attribute"]),
            Field(
                name="embedding", type=f"tensor<float>(x[{embedder.dims}])",
                indexing=["attribute", "index"],
                ann=HNSW(distance_metric="angular"),
            ),
        ]),
        rank_profiles=[RankProfile(
            name="poma_rank",
            inputs=[("query(q)", f"tensor<float>(x[{embedder.dims}])")],
            first_phase="closeness(field, embedding)",
        )],
    )],
)
app = VespaDocker().deploy(application_package=package)

records = records_from_archive("archives/contract.poma")  # -> list[Record] (id, text, payload)
docs = []
# assemble() reads the FULL document (chunks + chunksets) from the volume — write it once,
# not one {"chunks": <indices>, "text"} stub per record (which also overwrote itself each turn).
from poma.utils import unpack_poma_archive
_data = unpack_poma_archive(poma_archive_path="archives/contract.poma")
file_id = records[0].payload["file_id"]
vol.write_doc(file_id, {"file_id": file_id, "chunks": _data["chunks"], "chunksets": _data["chunksets"]})
for r in records:
    docs.append({"id": r.id, "fields": {
        "file_id": r.payload["file_id"],
        "chunkset_index": r.payload["chunkset_index"],
        "embedding": embedder.embed([r.text])[0],
    }})
app.feed_iterable(docs, schema="chunkset", namespace="contracts")

# 4. Retrieval — select the routing fields explicitly in YQL, then assemble.
qv = embedder.embed(["early termination conditions"])[0]
resp = app.query(
    yql="select file_id, chunkset_index from chunkset where "  # the page deploys schema `chunkset`, not `poma`
        "{targetHits:100}nearestNeighbor(embedding, q) limit 10",
    ranking="poma_rank",
    body={"input.query(q)": qv},
)
context = assemble(resp, volume=vol)  # -> [{"file_id", "content"}, ...]

Selecting file_id, chunkset_index explicitly in the YQL select clause is not optional — Vespa's select * would work too, but assemble() only needs the routing fields, and explicit selection is Vespa's documented way to keep the query lean. Every chunkset id is deterministic, so re-feeding the same document lands on the same document ids.

Metadata mapping: POMA fields → Vespa primitives ​

POMA field (Unstructured-sourced)Vespa primitiveNotes
chunkset id (chunkset_uuid)document idstable across re-feeds
to_embed (tables spliced as text_as_html)tensor<float>(x[N]) field, HNSW-indexedembedded before feed, never stored as text
file_idattribute field, stringselected explicitly in YQL for assemble()
chunkset_indexattribute field, intselected explicitly in YQL for assemble()
page (from metadata.page_number), depth, chunk_index, full chunkset textvolume documentcontent-free — never a Vespa field, fetched by assemble()

Frequently asked questions ​

How do I get Unstructured.io elements into Vespa for RAG? ​

Save the element list with elements_to_json, ingest it with PrimeCut, and keep the .poma archive. Declare a chunkset document type with file_id/chunkset_index attributes and an embedding tensor, feed it via feed_iterable, write chunkset content to a volume, then retrieve with a YQL query that selects the routing fields plus assemble(resp, volume=vol).

Why does chunk_by_title fragmentation cause problems in a Vespa schema? ​

chunk_by_title fragments carry no lineage above their nearest Title, so a Table element split across a boundary becomes two documents with nothing linking them. Vespa's multi-phase ranking scores each independently and cannot recover the missing half.

What Vespa schema fields should Unstructured chunks carry? ​

file_id and chunkset_index as attribute fields, plus a tensor field with an HNSW index for the embedding. Page, depth, chunk index, and full text stay off the document and live on a volume.

Do Unstructured's tables (text_as_html) survive into Vespa's BM25 field? ​

In this content-free pattern, chunkset text is embedded and volume-stored, not indexed as BM25 — Vespa only sees the vector and routing attributes. Add a string field with index: enable-bm25 populated from the same to_embed text if you need native lexical ranking too.

Should I use Vespa streaming mode for Unstructured-parsed multi-tenant documents? ​

Streaming mode's grouped document IDs fit small per-tenant corpora queried one tenant at a time; indexed mode fits queries spanning many tenants. Vespa's guidance is not to mix both modes in one content cluster.

Same parser, different store: Unstructured.io → Turbopuffer · Unstructured.io → OpenSearch · Unstructured.io → Weaviate

Foundations: The Optimal Chunker for Unstructured.io · Vespa Chunking Strategy for RAG · All pipeline recipes · RAG chunking guide