Redis Chunking Strategy for RAG: The Optimal Chunks for Retrieval
The short answer: the optimal chunks for Redis are chunksets — self-explanatory units that carry their full heading lineage — indexed via RediSearch's FT.CREATE with a VECTOR field for the embedding and TAG/NUMERIC fields for file_id/page/depth, queried with the KNN syntax that aliases its score and returns the metadata fields explicitly. Redis's ranking is only as good as what you indexed; POMA's PrimeCut emits exactly the fields this index needs, and its vektoria package keeps the index content-free so retrieval assembles clean context from a volume instead of duplicating document bodies into RAM.
Redis: an in-memory vector index with exact-match filters as a first-class citizen
RediSearch's vector support inherits Redis's core strength — everything lives in memory, so lookups are fast and filters are cheap:
VECTORfields —HNSWorFLATalgorithm,TYPE FLOAT32,DIMandDISTANCE_METRICdeclared per field in the schema.TAG/NUMERICfields — exact-match and range filters that run at the same speed as the vector search, not a post-filter scan.- Filter-then-search hybrid — a
TAGor text predicate can prefix theKNNclause in one query string, narrowing the vector search to a pre-filtered candidate set. - In-memory scale — fast, but every field you index costs RAM; content-free indexing matters more here than in disk-backed stores.
None of this decides what a document should represent. Redis ranks whatever vector and fields you indexed. A VECTOR field holding a context-free character-count fragment will rank context-free fragments — correctly, and very fast.
What "optimal chunks" means for Redis, concretely
- One hash/document per chunkset, self-explanatory alone. A chunkset is a root-to-leaf path —
Master Services Agreement → Termination clauses → Early termination → "Either party may…". See POMA chunksets. - Hierarchy as TAG/NUMERIC fields.
file_id(TAG),page/depth(NUMERIC) turn the filter half of a hybrid query into document-aware scoping without a post-filter scan. - The three-part retrieval contract. Alias the score
AS vector_score, call.return_fields("file_id", "chunkset_index", "vector_score"), and set.dialect(2)— miss any of the three and POMA'sassemble()gets an empty result from a non-empty search. - No overlap. Overlap duplicates spans into the HNSW/FLAT structure, and since Redis indexes live in memory, that duplication cost is more directly a RAM cost than in disk-backed stores. Chunksets need no overlap.
- Content-free index. Store
{file_id, chunkset_index}in RediSearch and the actual content on a volume — smaller in-memory footprint, citations reconstructed at retrieval time.
The pipeline: PrimeCut to Redis
pip install redisimport numpy as np
from redis import Redis
from redis.commands.search.field import VectorField, TagField, NumericField
from redis.commands.search.query import Query
from redis.commands.search.index_definition import IndexDefinition, IndexType
from poma import PrimeCut
from poma.vektoria import assemble, records_from_archive, Volume
from poma.embeddings import get_embedder
# 1. Chunk any document and keep the .poma archive — the volume's source of truth.
client = PrimeCut() # reads POMA_API_KEY
client.ingest("contract.pdf", download_dir="archives", filename="contract.poma")
embedder = get_embedder("local:BAAI/bge-small-en-v1.5")
vol = Volume("s3://your-bucket/poma")
r = Redis(host="localhost", port=6379)
r.ft("poma_idx").create_index(
fields=[
VectorField("embedding", "HNSW", {"TYPE": "FLOAT32", "DIM": embedder.dims,
"DISTANCE_METRIC": "COSINE"}),
TagField("file_id"),
NumericField("chunkset_index"),
],
definition=IndexDefinition(prefix=["poma:"], index_type=IndexType.HASH),
)
# 2. Content-free ingest — vector index holds only routing metadata; content lives on the volume.
records = records_from_archive("archives/contract.poma") # -> list[Record] (id, text, payload)
# assemble() reads the FULL document (chunks + chunksets) from the volume — write it once,
# not one {"chunks": <indices>, "text"} stub per record (which also overwrote itself each turn).
from poma.utils import unpack_poma_archive
_data = unpack_poma_archive(poma_archive_path="archives/contract.poma")
file_id = records[0].payload["file_id"]
vol.write_doc(file_id, {"file_id": file_id, "chunks": _data["chunks"], "chunksets": _data["chunksets"]})
for rec in records:
vector = np.asarray(embedder.embed([rec.text])[0], dtype=np.float32).tobytes()
r.hset(f"poma:{rec.id}", mapping={
"embedding": vector, "file_id": rec.payload["file_id"],
"chunkset_index": rec.payload["chunkset_index"],
})
# 3. KNN retrieval — the three-part contract Redis requires, then assemble.
qv = np.asarray(embedder.embed(["early termination conditions"])[0], dtype=np.float32).tobytes()
q = (Query("*=>[KNN 10 @embedding $vec AS vector_score]").sort_by("vector_score")
.return_fields("file_id", "chunkset_index", "vector_score").dialect(2))
res = r.ft("poma_idx").search(q, query_params={"vec": qv})
context = assemble(res, volume=vol) # -> [{"file_id", "content"}, ...]Retrieved chunksets share ancestor lineage across a document — assemble() deduplicates that lineage into one prompt-ready cheatsheet, the same discipline that answered one reference legal document with 337 tokens instead of 1,542 for a recursive-splitter baseline. One document is an illustration, not a benchmark; the ingestion guide has the broader numbers.
Chunk shapes in Redis, compared
| Property | Fixed 512-token chunks + overlap | Semantic chunks | POMA chunksets |
|---|---|---|---|
| Document is self-explanatory alone | ✗ | Sometimes | ✓ (by construction) |
| Hierarchy as TAG/NUMERIC fields | Manual | Manual | ✓ (file_id, page, depth) |
| Redundant HNSW/FLAT entries (RAM cost) | Many (overlap) | Few | None |
| Content-free index + volume | DIY | DIY | ✓ (vektoria assemble/Volume) |
| Metadata returned on hits | Requires return_fields+dialect(2) | Requires same | Same requirement — documented, not silent |
Frequently asked questions
What is the optimal chunk size for Redis vector search?
There's no universal token count — chunksets are self-explanatory at any size, which is why they beat fixed 512-token windows. Start at 512 tokens if you must fix a size, but the size knob cannot fix missing context.
How do I define a Redis index for chunksets?
FT.CREATE with a VECTOR field (HNSW/FLAT, FLOAT32, dims matching your embedder, COSINE) plus TAG for file_id and NUMERIC for page/depth.
Why does my Redis KNN query return nothing through POMA assemble?
The score must be aliased AS vector_score, the query needs .return_fields("file_id", "chunkset_index", "vector_score"), and .dialect(2) must be set. Miss any of the three and assemble() gets nothing back from a non-empty search.
Does Redis support hybrid vector and keyword search?
Yes, via filter-then-search: a TAG or text predicate prefixes the KNN clause in one query string, narrowing candidates before the vector search — distinct from BM25-fusion hybrid in other stores.
How does chunk overlap affect a Redis vector index?
It duplicates spans into the HNSW/FLAT structure — a direct RAM cost since Redis indexes live in memory. Chunksets need no overlap.
Feed Redis from the parser you already run
- AWS Textract → Redis
- Azure Document Intelligence → Redis
- Docling → Redis
- LlamaParse → Redis
- Marker → Redis
- Mistral OCR → Redis
- PaddleOCR-VL → Redis
- Unstructured.io → Redis
Running a different store? The same chunk design applies: Azure AI Search · Chroma · Elasticsearch · FAISS · LanceDB · Milvus · MongoDB Atlas · OpenSearch · pgvector · Pinecone · Qdrant · Turbopuffer · Vespa · Weaviate — or browse all pipeline recipes.
Fundamentals: RAG chunking guide · RAG architecture guide · POMA chunksets.