Source: http://www.poma-ai.com/docs/pipelines/unstructured-to-vespa

# The Missing Link Between Unstructured.io and Optimal Retrieval in Vespa

<ByAuthor />

**The short answer:** Unstructured.io gives you a typed, ordered element list; Vespa gives you one system that natively fuses BM25 and ANN over documents you define. Wired together with `chunk_by_title` and a flat document schema, they still lose the hierarchy Unstructured recovered, and Vespa's rank profile has no field to recover it from. The missing link is POMA: `PrimeCut().ingest()` consumes the raw element list, emits hierarchy-preserving [chunksets](/learn/chunking/chunksets) into a portable `.poma` archive, and `poma.vektoria` feeds Vespa only `(id, embedding, {file_id, chunkset_index})` while the actual content lives on a volume. Retrieval runs Vespa's own YQL query, then `assemble()` reassembles prompt-ready cheatsheets.

## What each end of the pipeline actually provides

**Unstructured.io** (`partition_pdf` or the hosted API) returns a flat, ordered list of typed elements — `Title`, `NarrativeText`, `ListItem`, `Table`, `Image` — each with `element_id`, `text`, and `metadata` (`page_number`, `text_as_html` for tables, optionally `image_base64`). Element order is document order; nothing records which `Title` nests under which. Details: [The Optimal Chunker for Unstructured.io](/optimal-chunker-unstructured).

**Vespa** stores vectors as ordinary `tensor` fields alongside scalar attributes in the same document type, with `index: enable-bm25` for native lexical scoring and a rank profile that fuses both in one query. It has no separate "vector store" object, and its ranking is only as good as the fields you declared and populated. Details: [Vespa Chunking Strategy for RAG](/optimal-chunks-vespa).

## The naive wiring, and where it breaks

The common recipe — run `chunk_by_title`, feed one Vespa document per fragment with just `text` and `embedding` — breaks in a pair-specific way:

- **`chunk_by_title` fragments carry no lineage above their nearest `Title`.** A `Table` element split across a character-limit boundary becomes two Vespa documents that share a `file_id` (if you even declared one) but nothing else — no `depth`, no ordering field, no way for a rank profile to know they were one table.
- **Vespa's schema is declared upfront, and naive pipelines under-declare it.** Skipping `file_id`/hierarchy attributes because `chunk_by_title` never surfaced them means queries can't filter by document or section — every fragment from every tenant sits in one undifferentiated pool.
- **Multi-phase ranking scores documents independently.** `closeness(field, embedding) * (1 + bm25(text))` ranks each fragment on its own merits; it cannot reconstruct a table's missing half because Vespa was never told the two fragments were related.

## The pipeline, end to end

```bash
pip install unstructured pyvespa
```

```python
from unstructured.partition.pdf import partition_pdf
from unstructured.staging.base import elements_to_json
from poma import PrimeCut
from poma.vektoria import assemble, records_from_archive, Volume
from poma.embeddings import get_embedder
from vespa.package import ApplicationPackage, Field, Schema, Document, HNSW, RankProfile
from vespa.deployment import VespaDocker

# 1. Your existing Unstructured call — unchanged.
elements = partition_pdf(
    filename="contract.pdf",
    strategy="hi_res",
    infer_table_structure=True,            # populates metadata.text_as_html
    extract_image_block_types=["Image"],
    extract_image_block_to_payload=True,   # inline base64 → POMA can describe figures
)
elements_to_json(elements, filename="contract.unstructured.json")

# 2. The missing link — raw element list in, a portable .poma archive out.
client = PrimeCut()  # reads POMA_API_KEY
client.ingest("contract.unstructured.json", download_dir="archives", filename="contract.poma")

embedder = get_embedder("local:BAAI/bge-small-en-v1.5")
vol = Volume("s3://your-bucket/poma")

# 3. One document type carrying only the embedding and the routing attributes.
package = ApplicationPackage(
    name="contracts",
    schema=[Schema(
        name="chunkset",
        document=Document(fields=[
            Field(name="file_id", type="string", indexing=["attribute"]),
            Field(name="chunkset_index", type="int", indexing=["attribute"]),
            Field(
                name="embedding", type=f"tensor<float>(x[{embedder.dims}])",
                indexing=["attribute", "index"],
                ann=HNSW(distance_metric="angular"),
            ),
        ]),
        rank_profiles=[RankProfile(
            name="poma_rank",
            inputs=[("query(q)", f"tensor<float>(x[{embedder.dims}])")],
            first_phase="closeness(field, embedding)",
        )],
    )],
)
app = VespaDocker().deploy(application_package=package)

records = records_from_archive("archives/contract.poma")  # -> list[Record] (id, text, payload)
docs = []
for r in records:
    vol.write_doc(r.payload["file_id"], {"file_id": r.payload["file_id"],
                                          "chunks": r.payload["chunks"], "text": r.text})
    docs.append({"id": r.id, "fields": {
        "file_id": r.payload["file_id"],
        "chunkset_index": r.payload["chunkset_index"],
        "embedding": embedder.embed([r.text])[0],
    }})
app.feed_iterable(docs, schema="chunkset", namespace="contracts")

# 4. Retrieval — select the routing fields explicitly in YQL, then assemble.
qv = embedder.embed(["early termination conditions"])[0]
resp = app.query(
    yql="select file_id, chunkset_index from poma where "
        "{targetHits:100}nearestNeighbor(embedding, q) limit 10",
    ranking="poma_rank",
    body={"input.query(q)": qv},
)
context = assemble(resp, volume=vol)  # -> [{"file_id", "content"}, ...]
```

Selecting `file_id, chunkset_index` explicitly in the YQL `select` clause is not optional — Vespa's `select *` would work too, but `assemble()` only needs the routing fields, and explicit selection is Vespa's documented way to keep the query lean. Every chunkset id is deterministic, so re-feeding the same document lands on the same document ids.

## Metadata mapping: POMA fields → Vespa primitives

| POMA field (Unstructured-sourced) | Vespa primitive | Notes |
| --- | --- | --- |
| chunkset id (`chunkset_uuid`) | document `id` | stable across re-feeds |
| `to_embed` (tables spliced as `text_as_html`) | `tensor<float>(x[N])` field, HNSW-indexed | embedded before feed, never stored as text |
| `file_id` | `attribute` field, `string` | selected explicitly in YQL for `assemble()` |
| `chunkset_index` | `attribute` field, `int` | selected explicitly in YQL for `assemble()` |
| `page` (from `metadata.page_number`), `depth`, `chunk_index`, full chunkset text | volume document | content-free — never a Vespa field, fetched by `assemble()` |

## Frequently asked questions

### How do I get Unstructured.io elements into Vespa for RAG?

Save the element list with `elements_to_json`, ingest it with `PrimeCut`, and keep the `.poma` archive. Declare a `chunkset` document type with `file_id`/`chunkset_index` attributes and an embedding tensor, feed it via `feed_iterable`, write chunkset content to a volume, then retrieve with a YQL query that selects the routing fields plus `assemble(resp, volume=vol)`.

### Why does chunk_by_title fragmentation cause problems in a Vespa schema?

`chunk_by_title` fragments carry no lineage above their nearest `Title`, so a `Table` element split across a boundary becomes two documents with nothing linking them. Vespa's multi-phase ranking scores each independently and cannot recover the missing half.

### What Vespa schema fields should Unstructured chunks carry?

`file_id` and `chunkset_index` as attribute fields, plus a `tensor` field with an HNSW index for the embedding. Page, depth, chunk index, and full text stay off the document and live on a volume.

### Do Unstructured's tables (text_as_html) survive into Vespa's BM25 field?

In this content-free pattern, chunkset text is embedded and volume-stored, not indexed as BM25 — Vespa only sees the vector and routing attributes. Add a string field with `index: enable-bm25` populated from the same `to_embed` text if you need native lexical ranking too.

### Should I use Vespa streaming mode for Unstructured-parsed multi-tenant documents?

Streaming mode's grouped document IDs fit small per-tenant corpora queried one tenant at a time; indexed mode fits queries spanning many tenants. Vespa's guidance is not to mix both modes in one content cluster.

## Related recipes

Same parser, different store: [Unstructured.io → Turbopuffer](/pipelines/unstructured-to-turbopuffer) · [Unstructured.io → OpenSearch](/pipelines/unstructured-to-opensearch) · [Unstructured.io → Weaviate](/pipelines/unstructured-to-weaviate)

Foundations: [The Optimal Chunker for Unstructured.io](/optimal-chunker-unstructured) · [Vespa Chunking Strategy for RAG](/optimal-chunks-vespa) · [All pipeline recipes](/pipelines/) · [RAG chunking guide](/guides/rag-chunking/)