Vespa Chunking Strategy for RAG: The Optimal Chunks for Retrieval
The short answer: the optimal chunks for Vespa are chunksets — self-explanatory units that carry their full heading lineage — modeled as one document type with file_id/page/depth attribute fields for filtering, a BM25-enabled text field, and a tensor field carrying the embedding with its own HNSW index. Because Vespa stores vectors and scalar attributes in the same document, there's no separate vector-store round trip — one YQL query fuses ANN and BM25 in a single ranking pass. POMA's PrimeCut emits exactly the fields this schema needs from any document.
Vespa: one system, not a vector store bolted onto a database
Vespa's design predates the current wave of vector databases by over a decade, and it shows in what's native rather than bolted on:
- Tensor fields alongside scalar fields —
tensor<float>(x[384])lives in the same document asfile_id,page, and text — no second system to keep in sync. - Native BM25 —
index: enable-bm25on any string field, no separate full-text engine. - Multi-phase ranking — a cheap first phase scores every match, an expensive second phase re-ranks the top candidates, all inside one rank profile you write.
- Tiered execution modes —
indexed(in-memory inverted + HNSW) for large shared corpora,streaming(per-group scan, no persistent index) purpose-built for many small per-tenant partitions.
None of this decides what a document should represent. Vespa ranks whatever fields you populated, however you defined the rank profile. A schema whose text field holds a context-free character-count fragment will rank context-free fragments — correctly, and at the scale Vespa runs at Yahoo and Spotify.
What "optimal chunks" means for Vespa, concretely
- Every document is self-explanatory. A chunkset is a root-to-leaf path —
Master Services Agreement → Termination clauses → Early termination → "Either party may…". Retrieved alone, it still tells the LLM exactly where it sits. See POMA chunksets. - Hierarchy as attribute fields, not a blob.
file_id,page,depthasattributefields turn Vespa's structured YQL predicates into document-aware filters — scope to one contract, exclude by depth, cite pages. - Choose streaming vs. indexed on tenancy shape. Streaming mode's grouped IDs suit many small per-tenant document sets; indexed mode suits a large corpus queried across tenants. Don't mix both in one content cluster.
- No overlap, no redundant index entries. Overlap duplicates spans into both the HNSW graph and the BM25 inverted index. Chunksets carry context structurally, so overlap is unnecessary.
- One rank profile fuses both signals.
closeness(field, embedding) * (1 + bm25(text))(or an explicitglobal-phasere-rank) — designed once, applied consistently, no client-side score merging.
The pipeline: PrimeCut to Vespa
pip install pyvespafrom poma import PrimeCut
from vespa.package import ApplicationPackage, Field, Schema, Document, HNSW, RankProfile
from vespa.deployment import VespaDocker
# 1. Chunk any document — PDF, DOCX, HTML, or a bring-your-own-OCR result JSON.
client = PrimeCut() # reads POMA_API_KEY
result = client.ingest("contract.pdf")
# 2. One document type carrying both the embedding and the filterable hierarchy.
package = ApplicationPackage(
name="contracts",
schema=[Schema(
name="chunkset",
document=Document(fields=[
Field(name="file_id", type="string", indexing=["attribute"]),
Field(name="depth", type="int", indexing=["attribute"]),
Field(name="text", type="string", indexing=["index"], index="enable-bm25"),
Field(
name="embedding", type="tensor<float>(x[384])",
indexing=["attribute", "index"],
ann=HNSW(distance_metric="angular"),
),
]),
rank_profiles=[RankProfile(
name="fusion",
inputs=[("query(q)", "tensor<float>(x[384])")],
first_phase="closeness(field, embedding) * (1 + bm25(text))",
)],
)],
)
app = VespaDocker().deploy(application_package=package)
chunkset_docs = [
{"id": f"{cs.file_id}:{cs.chunkset_index}", "fields": {
"file_id": cs.file_id, "depth": len(cs.chunks), "text": cs.to_embed,
"embedding": embed(cs.to_embed), # your embedding model
}}
for cs in result.chunksets
]
app.feed_iterable(chunkset_docs, schema="chunkset", namespace="contracts")
# 3. One YQL query, both signals fused in the rank profile.
response = app.query(
yql="select * from sources * where userQuery() or "
"({targetHits:1000}nearestNeighbor(embedding,q))",
query="early termination conditions",
ranking="fusion",
body={"input.query(q)": embed("early termination conditions")}, # Vespa-side embed() needs an embedder component in services.xml; pass the vector from the same model used at feed time
)Merge retrieved chunksets' shared ancestor lineage into one cheatsheet before prompting — the discipline that answered one reference legal document with 337 tokens instead of 1,542 for a recursive-splitter baseline. One document is an illustration, not a benchmark; the ingestion guide has the broader numbers.
Chunk shapes in Vespa, compared
| Property | Fixed 512-token chunks + overlap | Semantic chunks | POMA chunksets |
|---|---|---|---|
| Document is self-explanatory alone | ✗ | Sometimes | ✓ (by construction) |
| Hierarchy as attribute fields | Manual | Manual | ✓ (file_id, depth) |
| Redundant index entries (HNSW + BM25) | Many (overlap) | Few | None |
| Hybrid BM25 + ANN in one rank profile | DIY client fusion | DIY client fusion | Native multi-phase ranking |
| Tenancy model fits per-tenant corpora | Manual sharding | Manual sharding | ✓ streaming mode grouped IDs |
Frequently asked questions
What is the optimal chunk size for Vespa?
There's no universal token count — chunksets are self-explanatory at any size, which is why they beat fixed 512-token windows. Start at 512 tokens if you must fix a size, but the size knob cannot fix missing context.
How should I model chunks and chunksets in a Vespa schema?
One document type with file_id/page/depth attribute fields, a BM25-enabled text field, and a tensor field with an HNSW index — vectors and scalars in the same document, no second system.
Does Vespa support hybrid vector and keyword search natively?
Yes — one YQL query combines userQuery() and nearestNeighbor(), fused by a rank profile using multi-phase ranking rather than client-side score merging.
Should I use streaming mode or indexed mode for a multi-tenant RAG corpus in Vespa?
Streaming mode's grouped IDs suit many small per-tenant partitions; indexed mode suits a large shared corpus. Vespa's guidance is not to mix both in one content cluster.
How does chunk overlap affect a Vespa deployment?
It duplicates spans into both the HNSW graph and the BM25 inverted index, inflating memory either way. Chunksets need no overlap.
Is Vespa worth the setup complexity for a RAG project?
If you need one system natively fusing BM25, ANN, and custom ranking at large scale, yes. If you want the fastest time to a first query, start with a hosted API-first store and migrate later using the same chunkset shape.
Retrieval with vektoria's assemble()
POMA's poma.vektoria package keeps the document content-free: Vespa stores only (id, vector, {file_id, chunkset_index}), while the actual chunk content lives on a volume (a path, s3://, or gs:// URL). After your normal Vespa query, hand the raw result to assemble():
from poma.vektoria import assemble
resp = sess.query(yql="select file_id, chunkset_index from poma where "
"{targetHits:100}nearestNeighbor(embedding, q) limit 10",
ranking="poma_rank", body={"input.query(q)": qv})
context = assemble(resp, volume="s3://your-bucket/poma") # -> [{"file_id", "content"}, ...]assemble() auto-detects Vespa's result shape (one of 12 supported stores) and returns deduplicated cheatsheets. The one requirement: explicitly select file_id/chunkset_index in the YQL, or Vespa won't return them for assemble() to work with.
Feed Vespa from the parser you already run
- AWS Textract → Vespa
- Azure Document Intelligence → Vespa
- Docling → Vespa
- LlamaParse → Vespa
- Marker → Vespa
- Mistral OCR → Vespa
- PaddleOCR-VL → Vespa
- Unstructured.io → Vespa
Running a different store? The same chunk design applies: Azure AI Search · Chroma · Elasticsearch · FAISS · LanceDB · Milvus · MongoDB Atlas · OpenSearch · pgvector · Pinecone · Qdrant · Redis · Turbopuffer · Weaviate — or browse all pipeline recipes.
Fundamentals: RAG chunking guide · RAG architecture guide · POMA chunksets.