Source: http://www.poma-ai.com/docs/pipelines/llamaparse-to-redis

# The Missing Link Between LlamaParse and Optimal Retrieval in Redis

<ByAuthor />

**The short answer:** LlamaParse gives you excellent per-page markdown; Redis gives you a fast, in-memory vector index with exact-match filters as a first-class citizen. Wired together naively — concatenate pages, split, embed, `HSET` — they still produce mediocre RAG, because nothing in between rebuilds the document's hierarchy or respects Redis's three-part retrieval contract. The missing link is POMA: `PrimeCut().ingest()` consumes the raw LlamaParse JSON (auto-detected), emits hierarchy-preserving [chunksets](/learn/chunking/chunksets) as a portable `.poma` archive, and `vektoria`'s `records_from_archive()` + your native Redis client write content-free hashes — `embedding`, `file_id`, `chunkset_index` — while the chunkset content itself lives on a volume. Retrieval runs Redis's own KNN query, then `assemble()` reassembles prompt-ready context.

## What each end of the pipeline actually provides

**LlamaParse** returns `pages[]`, each with a `page` number, an `md` string (markdown, tables inline), a `text` flattening, and an `images` list whose bytes stay server-side — a saved result JSON carries only `![](name)` references. What it doesn't provide: cross-page hierarchy (a `##` heading on page 41 has no link to the `#` chapter that opened on page 3) or retrieval units. Details: [The Optimal Chunker for LlamaParse](/optimal-chunker-llamaparse).

**Redis** (via RediSearch) provides a `VECTOR` field for HNSW/FLAT ANN search alongside `TAG`/`NUMERIC` fields that filter at the same speed as the vector search — all in memory. What it doesn't provide: any opinion about what belongs in that vector, or a metadata-return default you can skip configuring. It ranks whatever you indexed, and its KNN query syntax has to be built correctly or metadata simply doesn't come back. Details: [Redis Chunking Strategy for RAG](/optimal-chunks-redis).

## The naive wiring, and where it breaks

The common recipe — `"\n".join(p["md"] for p in pages)` → fixed-size splitter → embed → `HSET` into a hash with only a `VECTOR` field — breaks in a pair-specific way that compounds two quirks at once:

- **LlamaParse's dead image references get embedded as if they were content.** Without PrimeCut's neutralization pass, a naive splitter treats `![](img_p12_3.png)` as ordinary text, and that fragment gets embedded and written into Redis's RAM-resident `HNSW`/`FLAT` structure — a permanent, unremovable garbage vector sitting in a memory-priced index.
- **The KNN query never declares the three-part contract.** A naive `redis-py` query without `.return_fields("file_id", "chunkset_index", "vector_score")`, the `AS vector_score` alias, and `.dialect(2)` executes and finds hits, but `assemble()` gets a result shape with no metadata to extract from — it returns empty.
- **The two failures look identical from the outside.** An empty `assemble()` result could mean the index is full of garbage image-reference vectors crowding out real hits, or it could mean the query contract is wrong and real hits never had a chance to return metadata — you cannot tell which without fixing the query contract first, since a broken query returns empty regardless of what's actually indexed.

## The pipeline, end to end

```bash
pip install llama-parse redis
```

```python
import json
import os
import numpy as np
from llama_parse import LlamaParse
from redis import Redis
from redis.commands.search.field import VectorField, TagField, NumericField
from redis.commands.search.query import Query
from redis.commands.search.index_definition import IndexDefinition, IndexType
from poma import PrimeCut
from poma.vektoria import assemble, records_from_archive, Volume
from poma.embeddings import get_embedder

# 1. Your existing LlamaParse call — unchanged. Save the raw result JSON.
parser = LlamaParse(api_key=os.environ["LLAMA_CLOUD_API_KEY"])
json_result = parser.get_json_result("contract.pdf")
with open("contract.llamaparse.json", "w") as f:
    json.dump(json_result[0], f)

# 2. The missing link — raw LlamaParse JSON in, a portable .poma archive out.
client = PrimeCut()  # reads POMA_API_KEY
client.ingest("contract.llamaparse.json", download_dir="archives", filename="contract.poma")

embedder = get_embedder("local:BAAI/bge-small-en-v1.5")
vol = Volume("s3://your-bucket/poma")
r = Redis(host="localhost", port=6379)

r.ft("poma_idx").create_index(
    fields=[
        VectorField("embedding", "HNSW", {"TYPE": "FLOAT32", "DIM": embedder.dims,
                                           "DISTANCE_METRIC": "COSINE"}),
        TagField("file_id"),
        NumericField("chunkset_index"),
    ],
    definition=IndexDefinition(prefix=["poma:"], index_type=IndexType.HASH),
)

# 3. Content-free ingest — RediSearch holds only routing metadata; content lives on the volume.
records = records_from_archive("archives/contract.poma")  # -> list[Record] (id, text, payload)
for rec in records:
    vol.write_doc(rec.payload["file_id"], {"file_id": rec.payload["file_id"],
                                            "chunks": rec.payload["chunks"], "text": rec.text})
    vector = np.asarray(embedder.embed([rec.text])[0], dtype=np.float32).tobytes()
    r.hset(f"poma:{rec.id}", mapping={
        "embedding": vector, "file_id": rec.payload["file_id"],
        "chunkset_index": rec.payload["chunkset_index"],
    })

# 4. KNN retrieval — the three-part contract Redis requires, then assemble.
qv = np.asarray(embedder.embed(["early termination conditions"])[0], dtype=np.float32).tobytes()
q = (Query("*=>[KNN 10 @embedding $vec AS vector_score]").sort_by("vector_score")
     .return_fields("file_id", "chunkset_index", "vector_score").dialect(2))
res = r.ft("poma_idx").search(q, query_params={"vec": qv})
context = assemble(res, volume=vol)  # -> [{"file_id", "content"}, ...]
```

PrimeCut validates the payload shape up front (a mislabeled upload 422s immediately), prefers `md` over `text`, neutralizes LlamaParse's dead image references into visible, counted markers, strips running headers/footers, and rebuilds the cross-page heading tree before chunking. To force or suppress detection, set `external_ocr_source` to `"llamaparse"` or `"none"`.

## Metadata mapping: POMA fields → Redis primitives

| POMA chunk field | Redis primitive | What it enables |
| --- | --- | --- |
| `to_embed` | `VECTOR` field (HNSW/FLAT, FLOAT32) | dense KNN retrieval |
| `file_id` | `TAG` field | per-document scoping, filter-then-search hybrid |
| `chunkset_index` | `NUMERIC` field | deterministic id + lineage for `assemble()` |
| chunkset content (`text`, `chunks`) | volume, not a Redis field | keeps the RAM-resident index small; content fetched at assemble time |

## Frequently asked questions

### How do I get LlamaParse results into Redis for RAG?

Save the raw result JSON, run `PrimeCut().ingest()` on it (auto-detected, archived to `.poma`), then write `records_from_archive()` output as Redis hashes with a `VECTOR` field plus `TAG`/`NUMERIC` metadata. Retrieve with a KNN query and `assemble(res, volume=vol)`.

### Why not just split LlamaParse's md and embed it into Redis directly?

LlamaParse's md has no cross-page hierarchy on its own, and overlap-based splitting duplicates spans into RediSearch's in-memory index. Redis then retrieves context-free fragments, and you still have to hand-build the `file_id`/`chunkset_index` fields chunking should have given you for free.

### What Redis fields do I need to declare for LlamaParse chunksets?

A `VECTOR` field (HNSW/FLAT, `FLOAT32`, dims matching your embedder, `COSINE`), a `TAG` field for `file_id`, and a `NUMERIC` field for `chunkset_index` — declared in `FT.CREATE` before any writes.

### Why does my Redis KNN query return nothing after ingesting LlamaParse content?

The documented RETRIEVAL.md gotcha: alias the score `AS vector_score`, call `.return_fields("file_id", "chunkset_index", "vector_score")`, and set `.dialect(2)`. Miss any of the three and `assemble()` gets nothing back from a non-empty search.

### What happens to LlamaParse's offloaded images when they land in Redis?

PrimeCut neutralizes LlamaParse's dead `![](name)` references into visible, counted markers before chunking, so no garbage "image reference" vectors ever occupy space in RediSearch's RAM-resident index.

## Related recipes

Same parser, different store: [LlamaParse → MongoDB Atlas](/pipelines/llamaparse-to-mongodb-atlas) · [LlamaParse → LanceDB](/pipelines/llamaparse-to-lancedb) · [LlamaParse → Qdrant](/pipelines/llamaparse-to-qdrant)

Foundations: [The Optimal Chunker for LlamaParse](/optimal-chunker-llamaparse) · [Redis Chunking Strategy for RAG](/optimal-chunks-redis) · [All pipeline recipes](/pipelines/) · [RAG architecture guide](/guides/rag-architecture/)