Source: http://www.poma-ai.com/docs/optimal-chunks-vespa

# Vespa Chunking Strategy for RAG: The Optimal Chunks for Retrieval

<ByAuthor />

**The short answer:** the optimal chunks for Vespa are **chunksets** — self-explanatory units that carry their full heading lineage — modeled as one **document type** with `file_id`/`page`/`depth` attribute fields for filtering, a BM25-enabled text field, and a `tensor` field carrying the embedding with its own HNSW index. Because Vespa stores vectors and scalar attributes in the same document, there's no separate vector-store round trip — one YQL query fuses ANN and BM25 in a single ranking pass. POMA's PrimeCut emits exactly the fields this schema needs from any document.

## Vespa: one system, not a vector store bolted onto a database

Vespa's design predates the current wave of vector databases by over a decade, and it shows in what's native rather than bolted on:

- **Tensor fields alongside scalar fields** — `tensor<float>(x[384])` lives in the same document as `file_id`, `page`, and text — no second system to keep in sync.
- **Native BM25** — `index: enable-bm25` on any string field, no separate full-text engine.
- **Multi-phase ranking** — a cheap first phase scores every match, an expensive second phase re-ranks the top candidates, all inside one rank profile you write.
- **Tiered execution modes** — `indexed` (in-memory inverted + HNSW) for large shared corpora, `streaming` (per-group scan, no persistent index) purpose-built for many small per-tenant partitions.

None of this decides what a document should represent. Vespa ranks whatever fields you populated, however you defined the rank profile. A schema whose text field holds a context-free character-count fragment will rank context-free fragments — correctly, and at the scale Vespa runs at Yahoo and Spotify.

## What "optimal chunks" means for Vespa, concretely

1. **Every document is self-explanatory.** A chunkset is a root-to-leaf path — `Master Services Agreement → Termination clauses → Early termination → "Either party may…"`. Retrieved alone, it still tells the LLM exactly where it sits. See [POMA chunksets](/learn/chunking/chunksets).
2. **Hierarchy as attribute fields, not a blob.** `file_id`, `page`, `depth` as `attribute` fields turn Vespa's structured YQL predicates into document-aware filters — scope to one contract, exclude by depth, cite pages.
3. **Choose streaming vs. indexed on tenancy shape.** Streaming mode's grouped IDs suit many small per-tenant document sets; indexed mode suits a large corpus queried across tenants. Don't mix both in one content cluster.
4. **No overlap, no redundant index entries.** Overlap duplicates spans into both the HNSW graph and the BM25 inverted index. Chunksets carry context structurally, so overlap is unnecessary.
5. **One rank profile fuses both signals.** `closeness(field, embedding) * (1 + bm25(text))` (or an explicit `global-phase` re-rank) — designed once, applied consistently, no client-side score merging.

## The pipeline: PrimeCut to Vespa

```bash
pip install pyvespa
```

```python
from poma import PrimeCut
from vespa.package import ApplicationPackage, Field, Schema, Document, HNSW, RankProfile
from vespa.deployment import VespaDocker

# 1. Chunk any document — PDF, DOCX, HTML, or a bring-your-own-OCR result JSON.
client = PrimeCut()  # reads POMA_API_KEY
result = client.ingest("contract.pdf")

# 2. One document type carrying both the embedding and the filterable hierarchy.
package = ApplicationPackage(
    name="contracts",
    schema=[Schema(
        name="chunkset",
        document=Document(fields=[
            Field(name="file_id", type="string", indexing=["attribute"]),
            Field(name="depth", type="int", indexing=["attribute"]),
            Field(name="text", type="string", indexing=["index"], index="enable-bm25"),
            Field(
                name="embedding", type="tensor<float>(x[384])",
                indexing=["attribute", "index"],
                ann=HNSW(distance_metric="angular"),
            ),
        ]),
        rank_profiles=[RankProfile(
            name="fusion",
            inputs=[("query(q)", "tensor<float>(x[384])")],
            first_phase="closeness(field, embedding) * (1 + bm25(text))",
        )],
    )],
)
app = VespaDocker().deploy(application_package=package)

chunkset_docs = [
    {"id": f"{cs.file_id}:{cs.chunkset_index}", "fields": {
        "file_id": cs.file_id, "depth": len(cs.chunks), "text": cs.to_embed,
        "embedding": embed(cs.to_embed),  # your embedding model
    }}
    for cs in result.chunksets
]
app.feed_iterable(chunkset_docs, schema="chunkset", namespace="contracts")

# 3. One YQL query, both signals fused in the rank profile.
response = app.query(
    yql="select * from sources * where userQuery() or "
        "({targetHits:1000}nearestNeighbor(embedding,q))",
    query="early termination conditions",
    ranking="fusion",
    body={"input.query(q)": "embed(early termination conditions)"},
)
```

Merge retrieved chunksets' shared ancestor lineage into one cheatsheet before prompting — the discipline that answers the reference legal-document benchmark with **337 tokens** instead of **1,542** for a recursive-splitter baseline. [Methodology here](/document-ingestion-chunking-rag).

## Chunk shapes in Vespa, compared

| Property | Fixed 512-token chunks + overlap | Semantic chunks | **POMA chunksets** |
| --- | --- | --- | --- |
| Document is self-explanatory alone | ✗ | Sometimes | **✓ (by construction)** |
| Hierarchy as attribute fields | Manual | Manual | **✓ (`file_id`, `depth`)** |
| Redundant index entries (HNSW + BM25) | Many (overlap) | Few | **None** |
| Hybrid BM25 + ANN in one rank profile | DIY client fusion | DIY client fusion | **Native multi-phase ranking** |
| Tenancy model fits per-tenant corpora | Manual sharding | Manual sharding | **✓ streaming mode grouped IDs** |

## Frequently asked questions

### What is the optimal chunk size for Vespa?

There's no universal token count — chunksets are self-explanatory at any size, which is why they beat fixed 512-token windows. Start at 512 tokens if you must fix a size, but the size knob cannot fix missing context.

### How should I model chunks and chunksets in a Vespa schema?

One document type with `file_id`/`page`/`depth` attribute fields, a BM25-enabled text field, and a `tensor` field with an HNSW index — vectors and scalars in the same document, no second system.

### Does Vespa support hybrid vector and keyword search natively?

Yes — one YQL query combines `userQuery()` and `nearestNeighbor()`, fused by a rank profile using multi-phase ranking rather than client-side score merging.

### Should I use streaming mode or indexed mode for a multi-tenant RAG corpus in Vespa?

Streaming mode's grouped IDs suit many small per-tenant partitions; indexed mode suits a large shared corpus. Vespa's guidance is not to mix both in one content cluster.

### How does chunk overlap affect a Vespa deployment?

It duplicates spans into both the HNSW graph and the BM25 inverted index, inflating memory either way. Chunksets need no overlap.

### Is Vespa worth the setup complexity for a RAG project?

If you need one system natively fusing BM25, ANN, and custom ranking at large scale, yes. If you want the fastest time to a first query, start with a hosted API-first store and migrate later using the same chunkset shape.

## Retrieval with vektoria's assemble()

POMA's `poma.vektoria` package keeps the document content-free: Vespa stores only `(id, vector, {file_id, chunkset_index})`, while the actual chunk content lives on a **volume** (a path, `s3://`, or `gs://` URL). After your normal Vespa query, hand the raw result to `assemble()`:

```python
from poma.vektoria import assemble

resp = sess.query(yql="select file_id, chunkset_index from poma where "
                      "{targetHits:100}nearestNeighbor(embedding, q) limit 10",
                  ranking="poma_rank", body={"input.query(q)": qv})
context = assemble(resp, volume="s3://your-bucket/poma")  # -> [{"file_id", "content"}, ...]
```

`assemble()` auto-detects Vespa's result shape (one of 12 supported stores) and returns deduplicated cheatsheets. The one requirement: explicitly `select` `file_id`/`chunkset_index` in the YQL, or Vespa won't return them for `assemble()` to work with.

## Feed Vespa from the parser you already run

- [Mistral OCR → Vespa](/pipelines/mistral-ocr-to-vespa)
- [LlamaParse → Vespa](/pipelines/llamaparse-to-vespa)
- [Azure Document Intelligence → Vespa](/pipelines/azure-document-intelligence-to-vespa)
- [Unstructured.io → Vespa](/pipelines/unstructured-to-vespa)
- [Docling → Vespa](/pipelines/docling-to-vespa)
- [Marker → Vespa](/pipelines/marker-to-vespa)
- [AWS Textract → Vespa](/pipelines/textract-to-vespa)
- [PaddleOCR-VL → Vespa](/pipelines/paddleocr-vl-to-vespa)

Running a different store? [Qdrant](/optimal-chunks-qdrant) · [Pinecone](/optimal-chunks-pinecone) · [Weaviate](/optimal-chunks-weaviate) · [pgvector](/optimal-chunks-pgvector) · [Milvus](/optimal-chunks-milvus) · [Chroma](/optimal-chunks-chroma) · [Turbopuffer](/optimal-chunks-turbopuffer).

Fundamentals: [RAG chunking guide](/guides/rag-chunking/) · [RAG architecture guide](/guides/rag-architecture/) · [POMA chunksets](/learn/chunking/chunksets).