Skip to content

Vespa Chunking Strategy for RAG: The Optimal Chunks for Retrieval ​

The short answer: the optimal chunks for Vespa are chunksets — self-explanatory units that carry their full heading lineage — modeled as one document type with file_id/page/depth attribute fields for filtering, a BM25-enabled text field, and a tensor field carrying the embedding with its own HNSW index. Because Vespa stores vectors and scalar attributes in the same document, there's no separate vector-store round trip — one YQL query fuses ANN and BM25 in a single ranking pass. POMA's PrimeCut emits exactly the fields this schema needs from any document.

Vespa: one system, not a vector store bolted onto a database ​

Vespa's design predates the current wave of vector databases by over a decade, and it shows in what's native rather than bolted on:

  • Tensor fields alongside scalar fields — tensor<float>(x[384]) lives in the same document as file_id, page, and text — no second system to keep in sync.
  • Native BM25 — index: enable-bm25 on any string field, no separate full-text engine.
  • Multi-phase ranking — a cheap first phase scores every match, an expensive second phase re-ranks the top candidates, all inside one rank profile you write.
  • Tiered execution modes — indexed (in-memory inverted + HNSW) for large shared corpora, streaming (per-group scan, no persistent index) purpose-built for many small per-tenant partitions.

None of this decides what a document should represent. Vespa ranks whatever fields you populated, however you defined the rank profile. A schema whose text field holds a context-free character-count fragment will rank context-free fragments — correctly, and at the scale Vespa runs at Yahoo and Spotify.

What "optimal chunks" means for Vespa, concretely ​

  1. Every document is self-explanatory. A chunkset is a root-to-leaf path — Master Services Agreement → Termination clauses → Early termination → "Either party may…". Retrieved alone, it still tells the LLM exactly where it sits. See POMA chunksets.
  2. Hierarchy as attribute fields, not a blob. file_id, page, depth as attribute fields turn Vespa's structured YQL predicates into document-aware filters — scope to one contract, exclude by depth, cite pages.
  3. Choose streaming vs. indexed on tenancy shape. Streaming mode's grouped IDs suit many small per-tenant document sets; indexed mode suits a large corpus queried across tenants. Don't mix both in one content cluster.
  4. No overlap, no redundant index entries. Overlap duplicates spans into both the HNSW graph and the BM25 inverted index. Chunksets carry context structurally, so overlap is unnecessary.
  5. One rank profile fuses both signals. closeness(field, embedding) * (1 + bm25(text)) (or an explicit global-phase re-rank) — designed once, applied consistently, no client-side score merging.

The pipeline: PrimeCut to Vespa ​

bash
pip install pyvespa
python
from poma import PrimeCut
from vespa.package import ApplicationPackage, Field, Schema, Document, HNSW, RankProfile
from vespa.deployment import VespaDocker

# 1. Chunk any document — PDF, DOCX, HTML, or a bring-your-own-OCR result JSON.
client = PrimeCut()  # reads POMA_API_KEY
result = client.ingest("contract.pdf")

# 2. One document type carrying both the embedding and the filterable hierarchy.
package = ApplicationPackage(
    name="contracts",
    schema=[Schema(
        name="chunkset",
        document=Document(fields=[
            Field(name="file_id", type="string", indexing=["attribute"]),
            Field(name="depth", type="int", indexing=["attribute"]),
            Field(name="text", type="string", indexing=["index"], index="enable-bm25"),
            Field(
                name="embedding", type="tensor<float>(x[384])",
                indexing=["attribute", "index"],
                ann=HNSW(distance_metric="angular"),
            ),
        ]),
        rank_profiles=[RankProfile(
            name="fusion",
            inputs=[("query(q)", "tensor<float>(x[384])")],
            first_phase="closeness(field, embedding) * (1 + bm25(text))",
        )],
    )],
)
app = VespaDocker().deploy(application_package=package)

chunkset_docs = [
    {"id": f"{cs.file_id}:{cs.chunkset_index}", "fields": {
        "file_id": cs.file_id, "depth": len(cs.chunks), "text": cs.to_embed,
        "embedding": embed(cs.to_embed),  # your embedding model
    }}
    for cs in result.chunksets
]
app.feed_iterable(chunkset_docs, schema="chunkset", namespace="contracts")

# 3. One YQL query, both signals fused in the rank profile.
response = app.query(
    yql="select * from sources * where userQuery() or "
        "({targetHits:1000}nearestNeighbor(embedding,q))",
    query="early termination conditions",
    ranking="fusion",
    body={"input.query(q)": embed("early termination conditions")},  # Vespa-side embed() needs an embedder component in services.xml; pass the vector from the same model used at feed time
)

Merge retrieved chunksets' shared ancestor lineage into one cheatsheet before prompting — the discipline that answered one reference legal document with 337 tokens instead of 1,542 for a recursive-splitter baseline. One document is an illustration, not a benchmark; the ingestion guide has the broader numbers.

Chunk shapes in Vespa, compared ​

PropertyFixed 512-token chunks + overlapSemantic chunksPOMA chunksets
Document is self-explanatory alone✗Sometimes✓ (by construction)
Hierarchy as attribute fieldsManualManual✓ (file_id, depth)
Redundant index entries (HNSW + BM25)Many (overlap)FewNone
Hybrid BM25 + ANN in one rank profileDIY client fusionDIY client fusionNative multi-phase ranking
Tenancy model fits per-tenant corporaManual shardingManual sharding✓ streaming mode grouped IDs

Frequently asked questions ​

What is the optimal chunk size for Vespa? ​

There's no universal token count — chunksets are self-explanatory at any size, which is why they beat fixed 512-token windows. Start at 512 tokens if you must fix a size, but the size knob cannot fix missing context.

How should I model chunks and chunksets in a Vespa schema? ​

One document type with file_id/page/depth attribute fields, a BM25-enabled text field, and a tensor field with an HNSW index — vectors and scalars in the same document, no second system.

Does Vespa support hybrid vector and keyword search natively? ​

Yes — one YQL query combines userQuery() and nearestNeighbor(), fused by a rank profile using multi-phase ranking rather than client-side score merging.

Should I use streaming mode or indexed mode for a multi-tenant RAG corpus in Vespa? ​

Streaming mode's grouped IDs suit many small per-tenant partitions; indexed mode suits a large shared corpus. Vespa's guidance is not to mix both in one content cluster.

How does chunk overlap affect a Vespa deployment? ​

It duplicates spans into both the HNSW graph and the BM25 inverted index, inflating memory either way. Chunksets need no overlap.

Is Vespa worth the setup complexity for a RAG project? ​

If you need one system natively fusing BM25, ANN, and custom ranking at large scale, yes. If you want the fastest time to a first query, start with a hosted API-first store and migrate later using the same chunkset shape.

Retrieval with vektoria's assemble() ​

POMA's poma.vektoria package keeps the document content-free: Vespa stores only (id, vector, {file_id, chunkset_index}), while the actual chunk content lives on a volume (a path, s3://, or gs:// URL). After your normal Vespa query, hand the raw result to assemble():

python
from poma.vektoria import assemble

resp = sess.query(yql="select file_id, chunkset_index from poma where "
                      "{targetHits:100}nearestNeighbor(embedding, q) limit 10",
                  ranking="poma_rank", body={"input.query(q)": qv})
context = assemble(resp, volume="s3://your-bucket/poma")  # -> [{"file_id", "content"}, ...]

assemble() auto-detects Vespa's result shape (one of 12 supported stores) and returns deduplicated cheatsheets. The one requirement: explicitly select file_id/chunkset_index in the YQL, or Vespa won't return them for assemble() to work with.

Feed Vespa from the parser you already run ​

Running a different store? The same chunk design applies: Azure AI Search · Chroma · Elasticsearch · FAISS · LanceDB · Milvus · MongoDB Atlas · OpenSearch · pgvector · Pinecone · Qdrant · Redis · Turbopuffer · Weaviate — or browse all pipeline recipes.

Fundamentals: RAG chunking guide · RAG architecture guide · POMA chunksets.