Source: http://www.poma-ai.com/docs/pipelines/mistral-ocr-to-redis

# The Missing Link Between Mistral OCR and Optimal Retrieval in Redis

<ByAuthor />

**The short answer:** Mistral OCR gives you excellent per-page markdown; Redis gives you excellent in-memory vector search with exact-match filters as a first-class citizen. Wired together naively — flatten, split, embed, hash — they still produce mediocre RAG, because nothing in between rebuilds the document's hierarchy or respects Redis's query contract. The missing link is POMA: `PrimeCut().ingest()` consumes the raw `/v1/ocr` JSON, emits hierarchy-preserving [chunksets](/learn/chunking/chunksets), and `vektoria`'s content-free ingest pattern writes only routing metadata into RediSearch while `assemble()` reconstructs prompt-ready context from a volume at query time.

## What each end of the pipeline actually provides

**Mistral OCR** (`/v1/ocr`, `mistral-ocr-latest`) returns `pages[]`, each with an `index` and a `markdown` string — headings, lists, and tables recovered *within* each page, image bytes inline when you set `include_image_base64`. What it doesn't provide: cross-page hierarchy (a heading on page 41 has no machine link to the chapter that opened on page 3) or retrieval units. Details: [The Optimal Chunker for Mistral OCR](/optimal-chunker-mistral-ocr).

**Redis** (via RediSearch) provides a `VECTOR` field for HNSW/FLAT ANN search, `TAG`/`NUMERIC` fields for exact-match and range filters that run at the same speed as the vector search, and a filter-then-search hybrid syntax — all served from memory. What it doesn't provide: any opinion about what belongs in that vector. It ranks whatever you hashed, including a context-free fragment, exactly as fast as it would rank a self-explanatory chunkset. Details: [Redis Chunking Strategy for RAG](/optimal-chunks-redis).

## The naive wiring, and where it breaks

The common recipe — `"\n".join(p["markdown"] for p in pages)` → `RecursiveCharacterTextSplitter` → embed → `r.hset(...)` per chunk — breaks in a pair-specific way:

- **Mistral's page indices vanish at the join**, so no `NUMERIC` field can carry a page number, and Redis's fast range/exact-match filters — its actual differentiator over disk-backed stores — have nothing to work with but `file_id`.
- **Overlap inflates the in-memory index directly.** Every duplicated span becomes a duplicated HNSW/FLAT entry, and because RediSearch's vector index lives entirely in RAM, that duplication is immediate memory cost, not deferred disk cost the way it would be in Qdrant or pgvector.
- **The KNN query looks like it works, and silently doesn't.** A port that does bother to embed and search still tends to write `Query("*=>[KNN 10 @embedding $vec]")` without aliasing the score `AS vector_score`, without `.return_fields(...)`, and without `.dialect(2)`. RediSearch returns real hits — the search itself succeeds — but the result carries none of the fields POMA's `assemble()` needs, so retrieval returns an empty context list from a non-empty search. Nothing throws; the failure looks like a bad query, not a missing three-line contract.

## The pipeline, end to end

```bash
pip install mistralai redis
```

```python
import os
import numpy as np
from mistralai import Mistral
from redis import Redis
from redis.commands.search.field import VectorField, TagField, NumericField
from redis.commands.search.query import Query
from redis.commands.search.index_definition import IndexDefinition, IndexType
from poma import PrimeCut
from poma.vektoria import assemble, records_from_archive, Volume
from poma.embeddings import get_embedder

# 1. Mistral OCR — your existing call, unchanged. Save the raw result JSON.
mistral = Mistral(api_key=os.environ["MISTRAL_API_KEY"])
ocr = mistral.ocr.process(
    model="mistral-ocr-latest",
    document={"type": "document_url", "document_url": "https://example.com/contract.pdf"},
    include_image_base64=True,
)
with open("contract.mistral-ocr.json", "w") as f:
    f.write(ocr.model_dump_json())

# 2. The missing link — raw result JSON in, a portable .poma archive out.
client = PrimeCut()  # reads POMA_API_KEY
client.ingest("contract.mistral-ocr.json", download_dir="archives", filename="contract.poma")

embedder = get_embedder("local:BAAI/bge-small-en-v1.5")
vol = Volume("s3://your-bucket/poma")
r = Redis(host="localhost", port=6379)

r.ft("poma_idx").create_index(
    fields=[
        VectorField("embedding", "HNSW", {"TYPE": "FLOAT32", "DIM": embedder.dims,
                                           "DISTANCE_METRIC": "COSINE"}),
        TagField("file_id"),
        NumericField("chunkset_index"),
    ],
    definition=IndexDefinition(prefix=["poma:"], index_type=IndexType.HASH),
)

# 3. Content-free ingest — RediSearch holds only routing metadata; content lives on the volume.
records = records_from_archive("archives/contract.poma")  # -> list[Record] (id, text, payload)
for rec in records:
    vol.write_doc(rec.payload["file_id"], {"file_id": rec.payload["file_id"],
                                            "chunks": rec.payload["chunks"], "text": rec.text})
    vector = np.asarray(embedder.embed([rec.text])[0], dtype=np.float32).tobytes()
    r.hset(f"poma:{rec.id}", mapping={
        "embedding": vector, "file_id": rec.payload["file_id"],
        "chunkset_index": rec.payload["chunkset_index"],
    })

# 4. KNN retrieval — the three-part contract Redis requires, then assemble.
qv = np.asarray(embedder.embed(["early termination conditions"])[0], dtype=np.float32).tobytes()
q = (Query("*=>[KNN 10 @embedding $vec AS vector_score]").sort_by("vector_score")
     .return_fields("file_id", "chunkset_index", "vector_score").dialect(2))
res = r.ft("poma_idx").search(q, query_params={"vec": qv})
context = assemble(res, volume=vol)  # -> [{"file_id", "content"}, ...]
```

POMA validates the Mistral payload shape up front (a corrupted or mislabeled upload 422s immediately), handles all three Mistral image cases, strips running headers/footers, and rebuilds the cross-page heading tree before chunking. Retrieved chunksets share ancestor lineage across a document — `assemble()` deduplicates that lineage into one prompt-ready cheatsheet, the same discipline that answers our reference legal-document benchmark with **337 tokens** instead of **1,542** for a recursive-splitter baseline. [Methodology here](/document-ingestion-chunking-rag).

## Metadata mapping: POMA fields → Redis primitives

| POMA chunk field | Redis primitive | What it enables |
| --- | --- | --- |
| `to_embed` | `VECTOR` field (HNSW/FLAT, FLOAT32) | KNN vector search over chunkset text |
| `file_id` | `TAG` field | exact-match scoping to one document |
| `page` (from Mistral `pages[].index`) | volume document, not a field | content-free retrieval, cheatsheet assembly |
| `depth` | volume document, not a field | content-free retrieval, cheatsheet assembly |
| `chunkset_index` | `NUMERIC` field, must appear in `.return_fields(...)` | `assemble()` dedup and ordering |
| chunkset lineage (content) | not stored in Redis — lives on the `Volume` | content-free index, smaller RAM footprint |

## Frequently asked questions

### How do I get Mistral OCR results into Redis for RAG?

Save the raw `/v1/ocr` JSON, run `PrimeCut().ingest()` on it, then pull `records_from_archive(...)`, write each chunkset's content to a volume, embed its `to_embed` text, and `hset` the vector plus `file_id`/`chunkset_index` into a RediSearch `VECTOR`-indexed hash. Retrieve with a KNN query and `assemble()`.

### Why does my Redis KNN query return nothing through POMA's assemble() after a Mistral OCR ingest?

The KNN clause needs `AS vector_score`, `.return_fields("file_id", "chunkset_index", "vector_score")`, and `.dialect(2)`. Miss any of the three and RediSearch still returns hits, but `assemble()` can't find `file_id`/`chunkset_index` on them and returns an empty context list.

### What Redis fields should Mistral OCR chunksets carry?

A `VECTOR` field for the embedding and a `NUMERIC`/`TAG` pair for `file_id`/`chunkset_index` — enough for a single KNN query to pre-filter on document scope. `page` (from Mistral's `pages[].index`) and `depth` stay on the volume document with the rest of the chunkset content, not as separate Redis fields.

### Why not just split Mistral OCR's markdown and hash it into Redis directly?

Concatenation discards page indices and heading levels, and splitter overlap duplicates spans directly into Redis's in-memory HNSW/FLAT structure — immediate RAM cost, not deferred disk cost. Redis then ranks context-free fragments exactly as fast as it would rank real chunksets.

### Does Redis's in-memory design change how I should index Mistral OCR content?

Yes — keep the index content-free. Store only the vector, `file_id`, and `chunkset_index` per hash, and let chunkset text and described images live on a volume. Every byte you don't hash into Redis is RAM you don't pay for.

## Related recipes

Same parser, different store: [Mistral OCR → Qdrant](/pipelines/mistral-ocr-to-qdrant) · [Mistral OCR → Pinecone](/pipelines/mistral-ocr-to-pinecone) · [Mistral OCR → Weaviate](/pipelines/mistral-ocr-to-weaviate)

Foundations: [The Optimal Chunker for Mistral OCR](/optimal-chunker-mistral-ocr) · [Redis Chunking Strategy for RAG](/optimal-chunks-redis) · [All pipeline recipes](/pipelines/) · [RAG chunking guide](/guides/rag-chunking/) · [POMA chunksets](/learn/chunking/chunksets)