LlamaParse to Redis: Chunking for RAG
The short answer: LlamaParse gives you excellent per-page markdown; Redis gives you a fast, in-memory vector index with exact-match filters as a first-class citizen. Wired together naively — concatenate pages, split, embed, HSET — they still produce mediocre RAG, because nothing in between rebuilds the document's hierarchy or respects Redis's three-part retrieval contract. The missing link is POMA: PrimeCut().ingest() consumes the raw LlamaParse JSON (auto-detected), emits hierarchy-preserving chunksets as a portable .poma archive, and vektoria's records_from_archive() + your native Redis client write content-free hashes — embedding, file_id, chunkset_index — while the chunkset content itself lives on a volume. Retrieval runs Redis's own KNN query, then assemble() reassembles prompt-ready context.
What each end of the pipeline actually provides
LlamaParse returns pages[], each with a page number, an md string (markdown, tables inline), a text flattening, and an images list whose bytes stay server-side — a saved result JSON carries only  references. What it doesn't provide: cross-page hierarchy (a ## heading on page 41 has no link to the # chapter that opened on page 3) or retrieval units. Details: The Optimal Chunker for LlamaParse.
Redis (via RediSearch) provides a VECTOR field for HNSW/FLAT ANN search alongside TAG/NUMERIC fields that filter at the same speed as the vector search — all in memory. What it doesn't provide: any opinion about what belongs in that vector, or a metadata-return default you can skip configuring. It ranks whatever you indexed, and its KNN query syntax has to be built correctly or metadata simply doesn't come back. Details: Redis Chunking Strategy for RAG.
The naive wiring, and where it breaks
The common recipe — "\n".join(p["md"] for p in pages) → fixed-size splitter → embed → HSET into a hash with only a VECTOR field — breaks in a pair-specific way that compounds two quirks at once:
- LlamaParse's dead image references get embedded as if they were content. Without PrimeCut's neutralization pass, a naive splitter treats
as ordinary text, and that fragment gets embedded and written into Redis's RAM-residentHNSW/FLATstructure — a permanent, unremovable garbage vector sitting in a memory-priced index. - The KNN query never declares the three-part contract. A naive
redis-pyquery without.return_fields("file_id", "chunkset_index", "vector_score"), theAS vector_scorealias, and.dialect(2)executes and finds hits, butassemble()gets a result shape with no metadata to extract from — it returns empty. - The two failures look identical from the outside. An empty
assemble()result could mean the index is full of garbage image-reference vectors crowding out real hits, or it could mean the query contract is wrong and real hits never had a chance to return metadata — you cannot tell which without fixing the query contract first, since a broken query returns empty regardless of what's actually indexed.
The pipeline, end to end
pip install llama-parse redisimport json
import os
import numpy as np
from llama_parse import LlamaParse
from redis import Redis
from redis.commands.search.field import VectorField, TagField, NumericField
from redis.commands.search.query import Query
from redis.commands.search.index_definition import IndexDefinition, IndexType
from poma import PrimeCut
from poma.vektoria import assemble, records_from_archive, Volume
from poma.embeddings import get_embedder
# 1. Your existing LlamaParse call — unchanged. Save the raw result JSON.
parser = LlamaParse(api_key=os.environ["LLAMA_CLOUD_API_KEY"])
json_result = parser.get_json_result("contract.pdf")
with open("contract.llamaparse.json", "w") as f:
json.dump(json_result[0], f)
# 2. The missing link — raw LlamaParse JSON in, a portable .poma archive out.
client = PrimeCut() # reads POMA_API_KEY
client.ingest("contract.llamaparse.json", download_dir="archives", filename="contract.poma")
embedder = get_embedder("local:BAAI/bge-small-en-v1.5")
vol = Volume("s3://your-bucket/poma")
r = Redis(host="localhost", port=6379)
r.ft("poma_idx").create_index(
fields=[
VectorField("embedding", "HNSW", {"TYPE": "FLOAT32", "DIM": embedder.dims,
"DISTANCE_METRIC": "COSINE"}),
TagField("file_id"),
NumericField("chunkset_index"),
],
definition=IndexDefinition(prefix=["poma:"], index_type=IndexType.HASH),
)
# 3. Content-free ingest — RediSearch holds only routing metadata; content lives on the volume.
records = records_from_archive("archives/contract.poma") # -> list[Record] (id, text, payload)
# assemble() reads the FULL document (chunks + chunksets) from the volume — write it once,
# not one {"chunks": <indices>, "text"} stub per record (which also overwrote itself each turn).
from poma.utils import unpack_poma_archive
_data = unpack_poma_archive(poma_archive_path="archives/contract.poma")
file_id = records[0].payload["file_id"]
vol.write_doc(file_id, {"file_id": file_id, "chunks": _data["chunks"], "chunksets": _data["chunksets"]})
for rec in records:
vector = np.asarray(embedder.embed([rec.text])[0], dtype=np.float32).tobytes()
r.hset(f"poma:{rec.id}", mapping={
"embedding": vector, "file_id": rec.payload["file_id"],
"chunkset_index": rec.payload["chunkset_index"],
})
# 4. KNN retrieval — the three-part contract Redis requires, then assemble.
qv = np.asarray(embedder.embed(["early termination conditions"])[0], dtype=np.float32).tobytes()
q = (Query("*=>[KNN 10 @embedding $vec AS vector_score]").sort_by("vector_score")
.return_fields("file_id", "chunkset_index", "vector_score").dialect(2))
res = r.ft("poma_idx").search(q, query_params={"vec": qv})
context = assemble(res, volume=vol) # -> [{"file_id", "content"}, ...]PrimeCut validates the payload shape up front (a mislabeled upload 422s immediately), prefers md over text, neutralizes LlamaParse's dead image references into visible, counted markers, strips running headers/footers, and rebuilds the cross-page heading tree before chunking. To force or suppress detection, set external_ocr_source to "llamaparse" or "none".
Metadata mapping: POMA fields → Redis primitives
| POMA chunk field | Redis primitive | What it enables |
|---|---|---|
to_embed | VECTOR field (HNSW/FLAT, FLOAT32) | dense KNN retrieval |
file_id | TAG field | per-document scoping, filter-then-search hybrid |
chunkset_index | NUMERIC field | deterministic id + lineage for assemble() |
chunkset content (text, chunks) | volume, not a Redis field | keeps the RAM-resident index small; content fetched at assemble time |
Frequently asked questions
How do I get LlamaParse results into Redis for RAG?
Save the raw result JSON, run PrimeCut().ingest() on it (auto-detected, archived to .poma), then write records_from_archive() output as Redis hashes with a VECTOR field plus TAG/NUMERIC metadata. Retrieve with a KNN query and assemble(res, volume=vol).
Why not just split LlamaParse's md and embed it into Redis directly?
LlamaParse's md has no cross-page hierarchy on its own, and overlap-based splitting duplicates spans into RediSearch's in-memory index. Redis then retrieves context-free fragments, and you still have to hand-build the file_id/chunkset_index fields chunking should have given you for free.
What Redis fields do I need to declare for LlamaParse chunksets?
A VECTOR field (HNSW/FLAT, FLOAT32, dims matching your embedder, COSINE), a TAG field for file_id, and a NUMERIC field for chunkset_index — declared in FT.CREATE before any writes.
Why does my Redis KNN query return nothing after ingesting LlamaParse content?
The documented RETRIEVAL.md gotcha: alias the score AS vector_score, call .return_fields("file_id", "chunkset_index", "vector_score"), and set .dialect(2). Miss any of the three and assemble() gets nothing back from a non-empty search.
What happens to LlamaParse's offloaded images when they land in Redis?
PrimeCut neutralizes LlamaParse's dead  references into visible, counted markers before chunking, so no garbage "image reference" vectors ever occupy space in RediSearch's RAM-resident index.
Related recipes
Same parser, different store: LlamaParse → MongoDB Atlas · LlamaParse → LanceDB · LlamaParse → Qdrant
Foundations: The Optimal Chunker for LlamaParse · Redis Chunking Strategy for RAG · All pipeline recipes · RAG architecture guide