Skip to content

Redis Chunking Strategy for RAG: The Optimal Chunks for Retrieval ​

The short answer: the optimal chunks for Redis are chunksets — self-explanatory units that carry their full heading lineage — indexed via RediSearch's FT.CREATE with a VECTOR field for the embedding and TAG/NUMERIC fields for file_id/page/depth, queried with the KNN syntax that aliases its score and returns the metadata fields explicitly. Redis's ranking is only as good as what you indexed; POMA's PrimeCut emits exactly the fields this index needs, and its vektoria package keeps the index content-free so retrieval assembles clean context from a volume instead of duplicating document bodies into RAM.

Redis: an in-memory vector index with exact-match filters as a first-class citizen ​

RediSearch's vector support inherits Redis's core strength — everything lives in memory, so lookups are fast and filters are cheap:

  • VECTOR fields — HNSW or FLAT algorithm, TYPE FLOAT32, DIM and DISTANCE_METRIC declared per field in the schema.
  • TAG/NUMERIC fields — exact-match and range filters that run at the same speed as the vector search, not a post-filter scan.
  • Filter-then-search hybrid — a TAG or text predicate can prefix the KNN clause in one query string, narrowing the vector search to a pre-filtered candidate set.
  • In-memory scale — fast, but every field you index costs RAM; content-free indexing matters more here than in disk-backed stores.

None of this decides what a document should represent. Redis ranks whatever vector and fields you indexed. A VECTOR field holding a context-free character-count fragment will rank context-free fragments — correctly, and very fast.

What "optimal chunks" means for Redis, concretely ​

  1. One hash/document per chunkset, self-explanatory alone. A chunkset is a root-to-leaf path — Master Services Agreement → Termination clauses → Early termination → "Either party may…". See POMA chunksets.
  2. Hierarchy as TAG/NUMERIC fields. file_id (TAG), page/depth (NUMERIC) turn the filter half of a hybrid query into document-aware scoping without a post-filter scan.
  3. The three-part retrieval contract. Alias the score AS vector_score, call .return_fields("file_id", "chunkset_index", "vector_score"), and set .dialect(2) — miss any of the three and POMA's assemble() gets an empty result from a non-empty search.
  4. No overlap. Overlap duplicates spans into the HNSW/FLAT structure, and since Redis indexes live in memory, that duplication cost is more directly a RAM cost than in disk-backed stores. Chunksets need no overlap.
  5. Content-free index. Store {file_id, chunkset_index} in RediSearch and the actual content on a volume — smaller in-memory footprint, citations reconstructed at retrieval time.

The pipeline: PrimeCut to Redis ​

bash
pip install redis
python
import numpy as np
from redis import Redis
from redis.commands.search.field import VectorField, TagField, NumericField
from redis.commands.search.query import Query
from redis.commands.search.index_definition import IndexDefinition, IndexType
from poma import PrimeCut
from poma.vektoria import assemble, records_from_archive, Volume
from poma.embeddings import get_embedder

# 1. Chunk any document and keep the .poma archive — the volume's source of truth.
client = PrimeCut()  # reads POMA_API_KEY
client.ingest("contract.pdf", download_dir="archives", filename="contract.poma")

embedder = get_embedder("local:BAAI/bge-small-en-v1.5")
vol = Volume("s3://your-bucket/poma")
r = Redis(host="localhost", port=6379)

r.ft("poma_idx").create_index(
    fields=[
        VectorField("embedding", "HNSW", {"TYPE": "FLOAT32", "DIM": embedder.dims,
                                           "DISTANCE_METRIC": "COSINE"}),
        TagField("file_id"),
        NumericField("chunkset_index"),
    ],
    definition=IndexDefinition(prefix=["poma:"], index_type=IndexType.HASH),
)

# 2. Content-free ingest — vector index holds only routing metadata; content lives on the volume.
records = records_from_archive("archives/contract.poma")  # -> list[Record] (id, text, payload)
# assemble() reads the FULL document (chunks + chunksets) from the volume — write it once,
# not one {"chunks": <indices>, "text"} stub per record (which also overwrote itself each turn).
from poma.utils import unpack_poma_archive
_data = unpack_poma_archive(poma_archive_path="archives/contract.poma")
file_id = records[0].payload["file_id"]
vol.write_doc(file_id, {"file_id": file_id, "chunks": _data["chunks"], "chunksets": _data["chunksets"]})
for rec in records:
    vector = np.asarray(embedder.embed([rec.text])[0], dtype=np.float32).tobytes()
    r.hset(f"poma:{rec.id}", mapping={
        "embedding": vector, "file_id": rec.payload["file_id"],
        "chunkset_index": rec.payload["chunkset_index"],
    })

# 3. KNN retrieval — the three-part contract Redis requires, then assemble.
qv = np.asarray(embedder.embed(["early termination conditions"])[0], dtype=np.float32).tobytes()
q = (Query("*=>[KNN 10 @embedding $vec AS vector_score]").sort_by("vector_score")
     .return_fields("file_id", "chunkset_index", "vector_score").dialect(2))
res = r.ft("poma_idx").search(q, query_params={"vec": qv})
context = assemble(res, volume=vol)  # -> [{"file_id", "content"}, ...]

Retrieved chunksets share ancestor lineage across a document — assemble() deduplicates that lineage into one prompt-ready cheatsheet, the same discipline that answered one reference legal document with 337 tokens instead of 1,542 for a recursive-splitter baseline. One document is an illustration, not a benchmark; the ingestion guide has the broader numbers.

Chunk shapes in Redis, compared ​

PropertyFixed 512-token chunks + overlapSemantic chunksPOMA chunksets
Document is self-explanatory alone✗Sometimes✓ (by construction)
Hierarchy as TAG/NUMERIC fieldsManualManual✓ (file_id, page, depth)
Redundant HNSW/FLAT entries (RAM cost)Many (overlap)FewNone
Content-free index + volumeDIYDIY✓ (vektoria assemble/Volume)
Metadata returned on hitsRequires return_fields+dialect(2)Requires sameSame requirement — documented, not silent

Frequently asked questions ​

There's no universal token count — chunksets are self-explanatory at any size, which is why they beat fixed 512-token windows. Start at 512 tokens if you must fix a size, but the size knob cannot fix missing context.

How do I define a Redis index for chunksets? ​

FT.CREATE with a VECTOR field (HNSW/FLAT, FLOAT32, dims matching your embedder, COSINE) plus TAG for file_id and NUMERIC for page/depth.

Why does my Redis KNN query return nothing through POMA assemble? ​

The score must be aliased AS vector_score, the query needs .return_fields("file_id", "chunkset_index", "vector_score"), and .dialect(2) must be set. Miss any of the three and assemble() gets nothing back from a non-empty search.

Yes, via filter-then-search: a TAG or text predicate prefixes the KNN clause in one query string, narrowing candidates before the vector search — distinct from BM25-fusion hybrid in other stores.

How does chunk overlap affect a Redis vector index? ​

It duplicates spans into the HNSW/FLAT structure — a direct RAM cost since Redis indexes live in memory. Chunksets need no overlap.

Feed Redis from the parser you already run ​

Running a different store? The same chunk design applies: Azure AI Search · Chroma · Elasticsearch · FAISS · LanceDB · Milvus · MongoDB Atlas · OpenSearch · pgvector · Pinecone · Qdrant · Turbopuffer · Vespa · Weaviate — or browse all pipeline recipes.

Fundamentals: RAG chunking guide · RAG architecture guide · POMA chunksets.