Source: http://www.poma-ai.com/docs/optimal-chunks-redis

# Redis Chunking Strategy for RAG: The Optimal Chunks for Retrieval

<ByAuthor />

**The short answer:** the optimal chunks for Redis are **chunksets** — self-explanatory units that carry their full heading lineage — indexed via RediSearch's `FT.CREATE` with a `VECTOR` field for the embedding and `TAG`/`NUMERIC` fields for `file_id`/`page`/`depth`, queried with the `KNN` syntax that aliases its score and returns the metadata fields explicitly. Redis's ranking is only as good as what you indexed; POMA's PrimeCut emits exactly the fields this index needs, and its `vektoria` package keeps the index content-free so retrieval assembles clean context from a volume instead of duplicating document bodies into RAM.

## Redis: an in-memory vector index with exact-match filters as a first-class citizen

RediSearch's vector support inherits Redis's core strength — everything lives in memory, so lookups are fast and filters are cheap:

- **`VECTOR` fields** — `HNSW` or `FLAT` algorithm, `TYPE FLOAT32`, `DIM` and `DISTANCE_METRIC` declared per field in the schema.
- **`TAG`/`NUMERIC` fields** — exact-match and range filters that run at the same speed as the vector search, not a post-filter scan.
- **Filter-then-search hybrid** — a `TAG` or text predicate can prefix the `KNN` clause in one query string, narrowing the vector search to a pre-filtered candidate set.
- **In-memory scale** — fast, but every field you index costs RAM; content-free indexing matters more here than in disk-backed stores.

None of this decides what a document should represent. Redis ranks whatever vector and fields you indexed. A `VECTOR` field holding a context-free character-count fragment will rank context-free fragments — correctly, and very fast.

## What "optimal chunks" means for Redis, concretely

1. **One hash/document per chunkset, self-explanatory alone.** A chunkset is a root-to-leaf path — `Master Services Agreement → Termination clauses → Early termination → "Either party may…"`. See [POMA chunksets](/learn/chunking/chunksets).
2. **Hierarchy as TAG/NUMERIC fields.** `file_id` (`TAG`), `page`/`depth` (`NUMERIC`) turn the filter half of a hybrid query into document-aware scoping without a post-filter scan.
3. **The three-part retrieval contract.** Alias the score `AS vector_score`, call `.return_fields("file_id", "chunkset_index", "vector_score")`, and set `.dialect(2)` — miss any of the three and POMA's `assemble()` gets an empty result from a non-empty search.
4. **No overlap.** Overlap duplicates spans into the HNSW/FLAT structure, and since Redis indexes live in memory, that duplication cost is more directly a RAM cost than in disk-backed stores. Chunksets need no overlap.
5. **Content-free index.** Store `{file_id, chunkset_index}` in RediSearch and the actual content on a volume — smaller in-memory footprint, citations reconstructed at retrieval time.

## The pipeline: PrimeCut to Redis

```bash
pip install redis
```

```python
import numpy as np
from redis import Redis
from redis.commands.search.field import VectorField, TagField, NumericField
from redis.commands.search.query import Query
from redis.commands.search.index_definition import IndexDefinition, IndexType
from poma import PrimeCut
from poma.vektoria import assemble, records_from_archive, Volume
from poma.embeddings import get_embedder

# 1. Chunk any document and keep the .poma archive — the volume's source of truth.
client = PrimeCut()  # reads POMA_API_KEY
client.ingest("contract.pdf", download_dir="archives", filename="contract.poma")

embedder = get_embedder("local:BAAI/bge-small-en-v1.5")
vol = Volume("s3://your-bucket/poma")
r = Redis(host="localhost", port=6379)

r.ft("poma_idx").create_index(
    fields=[
        VectorField("embedding", "HNSW", {"TYPE": "FLOAT32", "DIM": embedder.dims,
                                           "DISTANCE_METRIC": "COSINE"}),
        TagField("file_id"),
        NumericField("chunkset_index"),
    ],
    definition=IndexDefinition(prefix=["poma:"], index_type=IndexType.HASH),
)

# 2. Content-free ingest — vector index holds only routing metadata; content lives on the volume.
records = records_from_archive("archives/contract.poma")  # -> list[Record] (id, text, payload)
for rec in records:
    vol.write_doc(rec.payload["file_id"], {"file_id": rec.payload["file_id"],
                                            "chunks": rec.payload["chunks"], "text": rec.text})
    vector = np.asarray(embedder.embed([rec.text])[0], dtype=np.float32).tobytes()
    r.hset(f"poma:{rec.id}", mapping={
        "embedding": vector, "file_id": rec.payload["file_id"],
        "chunkset_index": rec.payload["chunkset_index"],
    })

# 3. KNN retrieval — the three-part contract Redis requires, then assemble.
qv = np.asarray(embedder.embed(["early termination conditions"])[0], dtype=np.float32).tobytes()
q = (Query("*=>[KNN 10 @embedding $vec AS vector_score]").sort_by("vector_score")
     .return_fields("file_id", "chunkset_index", "vector_score").dialect(2))
res = r.ft("poma_idx").search(q, query_params={"vec": qv})
context = assemble(res, volume=vol)  # -> [{"file_id", "content"}, ...]
```

Retrieved chunksets share ancestor lineage across a document — `assemble()` deduplicates that lineage into one prompt-ready cheatsheet, the same discipline that answers our reference legal-document benchmark with **337 tokens** instead of **1,542** for a recursive-splitter baseline. [Methodology here](/document-ingestion-chunking-rag).

## Chunk shapes in Redis, compared

| Property | Fixed 512-token chunks + overlap | Semantic chunks | **POMA chunksets** |
| --- | --- | --- | --- |
| Document is self-explanatory alone | ✗ | Sometimes | **✓ (by construction)** |
| Hierarchy as TAG/NUMERIC fields | Manual | Manual | **✓ (`file_id`, `page`, `depth`)** |
| Redundant HNSW/FLAT entries (RAM cost) | Many (overlap) | Few | **None** |
| Content-free index + volume | DIY | DIY | **✓ (`vektoria` `assemble`/`Volume`)** |
| Metadata returned on hits | Requires `return_fields`+`dialect(2)` | Requires same | **Same requirement — documented, not silent** |

## Frequently asked questions

### What is the optimal chunk size for Redis vector search?

There's no universal token count — chunksets are self-explanatory at any size, which is why they beat fixed 512-token windows. Start at 512 tokens if you must fix a size, but the size knob cannot fix missing context.

### How do I define a Redis index for chunksets?

`FT.CREATE` with a `VECTOR` field (HNSW/FLAT, `FLOAT32`, dims matching your embedder, `COSINE`) plus `TAG` for `file_id` and `NUMERIC` for `page`/`depth`.

### Why does my Redis KNN query return nothing through POMA assemble?

The score must be aliased `AS vector_score`, the query needs `.return_fields("file_id", "chunkset_index", "vector_score")`, and `.dialect(2)` must be set. Miss any of the three and `assemble()` gets nothing back from a non-empty search.

### Does Redis support hybrid vector and keyword search?

Yes, via filter-then-search: a `TAG` or text predicate prefixes the `KNN` clause in one query string, narrowing candidates before the vector search — distinct from BM25-fusion hybrid in other stores.

### How does chunk overlap affect a Redis vector index?

It duplicates spans into the HNSW/FLAT structure — a direct RAM cost since Redis indexes live in memory. Chunksets need no overlap.

## Feed Redis from the parser you already run

- [Mistral OCR → Redis](/pipelines/mistral-ocr-to-redis)
- [LlamaParse → Redis](/pipelines/llamaparse-to-redis)
- [Azure Document Intelligence → Redis](/pipelines/azure-document-intelligence-to-redis)
- [Unstructured.io → Redis](/pipelines/unstructured-to-redis)
- [Docling → Redis](/pipelines/docling-to-redis)
- [Marker → Redis](/pipelines/marker-to-redis)
- [AWS Textract → Redis](/pipelines/textract-to-redis)
- [PaddleOCR-VL → Redis](/pipelines/paddleocr-vl-to-redis)

Running a different store? Browse [all 13 vector databases](/pipelines/).

Fundamentals: [RAG chunking guide](/guides/rag-chunking/) · [RAG architecture guide](/guides/rag-architecture/) · [POMA chunksets](/learn/chunking/chunksets).