Skip to content

LlamaParse to Redis: Chunking for RAG ​

The short answer: LlamaParse gives you excellent per-page markdown; Redis gives you a fast, in-memory vector index with exact-match filters as a first-class citizen. Wired together naively — concatenate pages, split, embed, HSET — they still produce mediocre RAG, because nothing in between rebuilds the document's hierarchy or respects Redis's three-part retrieval contract. The missing link is POMA: PrimeCut().ingest() consumes the raw LlamaParse JSON (auto-detected), emits hierarchy-preserving chunksets as a portable .poma archive, and vektoria's records_from_archive() + your native Redis client write content-free hashes — embedding, file_id, chunkset_index — while the chunkset content itself lives on a volume. Retrieval runs Redis's own KNN query, then assemble() reassembles prompt-ready context.

What each end of the pipeline actually provides ​

LlamaParse returns pages[], each with a page number, an md string (markdown, tables inline), a text flattening, and an images list whose bytes stay server-side — a saved result JSON carries only ![](name) references. What it doesn't provide: cross-page hierarchy (a ## heading on page 41 has no link to the # chapter that opened on page 3) or retrieval units. Details: The Optimal Chunker for LlamaParse.

Redis (via RediSearch) provides a VECTOR field for HNSW/FLAT ANN search alongside TAG/NUMERIC fields that filter at the same speed as the vector search — all in memory. What it doesn't provide: any opinion about what belongs in that vector, or a metadata-return default you can skip configuring. It ranks whatever you indexed, and its KNN query syntax has to be built correctly or metadata simply doesn't come back. Details: Redis Chunking Strategy for RAG.

The naive wiring, and where it breaks ​

The common recipe — "\n".join(p["md"] for p in pages) → fixed-size splitter → embed → HSET into a hash with only a VECTOR field — breaks in a pair-specific way that compounds two quirks at once:

  • LlamaParse's dead image references get embedded as if they were content. Without PrimeCut's neutralization pass, a naive splitter treats ![](img_p12_3.png) as ordinary text, and that fragment gets embedded and written into Redis's RAM-resident HNSW/FLAT structure — a permanent, unremovable garbage vector sitting in a memory-priced index.
  • The KNN query never declares the three-part contract. A naive redis-py query without .return_fields("file_id", "chunkset_index", "vector_score"), the AS vector_score alias, and .dialect(2) executes and finds hits, but assemble() gets a result shape with no metadata to extract from — it returns empty.
  • The two failures look identical from the outside. An empty assemble() result could mean the index is full of garbage image-reference vectors crowding out real hits, or it could mean the query contract is wrong and real hits never had a chance to return metadata — you cannot tell which without fixing the query contract first, since a broken query returns empty regardless of what's actually indexed.

The pipeline, end to end ​

bash
pip install llama-parse redis
python
import json
import os
import numpy as np
from llama_parse import LlamaParse
from redis import Redis
from redis.commands.search.field import VectorField, TagField, NumericField
from redis.commands.search.query import Query
from redis.commands.search.index_definition import IndexDefinition, IndexType
from poma import PrimeCut
from poma.vektoria import assemble, records_from_archive, Volume
from poma.embeddings import get_embedder

# 1. Your existing LlamaParse call — unchanged. Save the raw result JSON.
parser = LlamaParse(api_key=os.environ["LLAMA_CLOUD_API_KEY"])
json_result = parser.get_json_result("contract.pdf")
with open("contract.llamaparse.json", "w") as f:
    json.dump(json_result[0], f)

# 2. The missing link — raw LlamaParse JSON in, a portable .poma archive out.
client = PrimeCut()  # reads POMA_API_KEY
client.ingest("contract.llamaparse.json", download_dir="archives", filename="contract.poma")

embedder = get_embedder("local:BAAI/bge-small-en-v1.5")
vol = Volume("s3://your-bucket/poma")
r = Redis(host="localhost", port=6379)

r.ft("poma_idx").create_index(
    fields=[
        VectorField("embedding", "HNSW", {"TYPE": "FLOAT32", "DIM": embedder.dims,
                                           "DISTANCE_METRIC": "COSINE"}),
        TagField("file_id"),
        NumericField("chunkset_index"),
    ],
    definition=IndexDefinition(prefix=["poma:"], index_type=IndexType.HASH),
)

# 3. Content-free ingest — RediSearch holds only routing metadata; content lives on the volume.
records = records_from_archive("archives/contract.poma")  # -> list[Record] (id, text, payload)
# assemble() reads the FULL document (chunks + chunksets) from the volume — write it once,
# not one {"chunks": <indices>, "text"} stub per record (which also overwrote itself each turn).
from poma.utils import unpack_poma_archive
_data = unpack_poma_archive(poma_archive_path="archives/contract.poma")
file_id = records[0].payload["file_id"]
vol.write_doc(file_id, {"file_id": file_id, "chunks": _data["chunks"], "chunksets": _data["chunksets"]})
for rec in records:
    vector = np.asarray(embedder.embed([rec.text])[0], dtype=np.float32).tobytes()
    r.hset(f"poma:{rec.id}", mapping={
        "embedding": vector, "file_id": rec.payload["file_id"],
        "chunkset_index": rec.payload["chunkset_index"],
    })

# 4. KNN retrieval — the three-part contract Redis requires, then assemble.
qv = np.asarray(embedder.embed(["early termination conditions"])[0], dtype=np.float32).tobytes()
q = (Query("*=>[KNN 10 @embedding $vec AS vector_score]").sort_by("vector_score")
     .return_fields("file_id", "chunkset_index", "vector_score").dialect(2))
res = r.ft("poma_idx").search(q, query_params={"vec": qv})
context = assemble(res, volume=vol)  # -> [{"file_id", "content"}, ...]

PrimeCut validates the payload shape up front (a mislabeled upload 422s immediately), prefers md over text, neutralizes LlamaParse's dead image references into visible, counted markers, strips running headers/footers, and rebuilds the cross-page heading tree before chunking. To force or suppress detection, set external_ocr_source to "llamaparse" or "none".

Metadata mapping: POMA fields → Redis primitives ​

POMA chunk fieldRedis primitiveWhat it enables
to_embedVECTOR field (HNSW/FLAT, FLOAT32)dense KNN retrieval
file_idTAG fieldper-document scoping, filter-then-search hybrid
chunkset_indexNUMERIC fielddeterministic id + lineage for assemble()
chunkset content (text, chunks)volume, not a Redis fieldkeeps the RAM-resident index small; content fetched at assemble time

Frequently asked questions ​

How do I get LlamaParse results into Redis for RAG? ​

Save the raw result JSON, run PrimeCut().ingest() on it (auto-detected, archived to .poma), then write records_from_archive() output as Redis hashes with a VECTOR field plus TAG/NUMERIC metadata. Retrieve with a KNN query and assemble(res, volume=vol).

Why not just split LlamaParse's md and embed it into Redis directly? ​

LlamaParse's md has no cross-page hierarchy on its own, and overlap-based splitting duplicates spans into RediSearch's in-memory index. Redis then retrieves context-free fragments, and you still have to hand-build the file_id/chunkset_index fields chunking should have given you for free.

What Redis fields do I need to declare for LlamaParse chunksets? ​

A VECTOR field (HNSW/FLAT, FLOAT32, dims matching your embedder, COSINE), a TAG field for file_id, and a NUMERIC field for chunkset_index — declared in FT.CREATE before any writes.

Why does my Redis KNN query return nothing after ingesting LlamaParse content? ​

The documented RETRIEVAL.md gotcha: alias the score AS vector_score, call .return_fields("file_id", "chunkset_index", "vector_score"), and set .dialect(2). Miss any of the three and assemble() gets nothing back from a non-empty search.

What happens to LlamaParse's offloaded images when they land in Redis? ​

PrimeCut neutralizes LlamaParse's dead ![](name) references into visible, counted markers before chunking, so no garbage "image reference" vectors ever occupy space in RediSearch's RAM-resident index.

Same parser, different store: LlamaParse → MongoDB Atlas · LlamaParse → LanceDB · LlamaParse → Qdrant

Foundations: The Optimal Chunker for LlamaParse · Redis Chunking Strategy for RAG · All pipeline recipes · RAG architecture guide