Source: http://www.poma-ai.com/docs/pipelines/marker-to-redis

# The Missing Link Between Marker and Optimal Retrieval in Redis

<ByAuthor />

**The short answer:** Marker gives you fast, local PDF parsing with real recovered structure; Redis gives you an in-memory vector index with exact-match filters as a first-class citizen. Wired together naively — flatten the markdown, split, embed, HSET only a vector field — the pipeline drops figures, loses page numbers, and often can't even be queried correctly once RediSearch's KNN contract is missed. The missing link is POMA: `PrimeCut().ingest()` consumes the saved Marker JSON (the Document block tree auto-detects), splices the side-channel image bytes back in, and rebuilds the cross-page hierarchy into [chunksets](/learn/chunking/chunksets). `vektoria` keeps Redis content-free — vectors and routing metadata only — and `assemble()` turns a correctly-shaped KNN result into prompt-ready context.

## What each end of the pipeline actually provides

**Marker** (by Datalab / VikParuchuri) runs on your own hardware and turns PDFs into clean structured output. With the JSON renderer (`--output_format json`) you get a Document block tree — Page children, per-block HTML, full `<table>` elements with row and column spans. What it doesn't provide: cross-page hierarchy, retrieval units, or inline images — figure bytes live in a side-channel `images` dict keyed by name, with only `![](name)` references left in the text. Details: [The Optimal Chunker for Marker](/optimal-chunker-marker).

**Redis** (via the RediSearch module) provides a `VECTOR` field (`HNSW`/`FLAT`, `FLOAT32`, `COSINE`) alongside `TAG`/`NUMERIC` fields that filter at the same speed as the vector search itself, all served from memory. What it doesn't provide: any opinion about what the vector should represent, or any tolerance for a malformed query — get the KNN clause's three-part contract wrong and RediSearch hands back metadata-free hits from a search that otherwise succeeded. Details: [Redis Chunking Strategy for RAG](/optimal-chunks-redis).

## The naive wiring, and where it breaks

The common recipe — run Marker's default markdown output, `RecursiveCharacterTextSplitter`, embed, `HSET` with only an `embedding` field, then a bare `"*=>[KNN 10 @embedding $vec]"` query — breaks in a way specific to this pair:

- **Every figure vanishes before Redis.** Marker's images live in the side-channel `images` dict; the markdown carries only `![](name)` references. A markdown-only pipeline embeds those dangling references as literal text, so the chart that answers a question was never written as a vector at all.
- **No `TAG`/`NUMERIC` fields exist to scope or cite from.** Marker's default markdown has no page boundaries, and a naive HSET writes only the embedding — so Redis's fast in-memory filtering primitive sits unused even though it costs nothing extra to populate.
- **The KNN query itself is malformed.** Without aliasing the score `AS vector_score`, calling `.return_fields(...)`, and setting `.dialect(2)`, RediSearch returns a result shape `assemble()` cannot parse — it comes back empty from a non-empty search. Two failures stack here: content that was never written, and content that was written but can't be read back through `vektoria` because the query shape is wrong.
- **Overlap inflates the index.** Splitter overlap embeds every boundary span twice, and because Redis holds everything in memory, that duplication is a direct RAM cost, not a disk one.

## The pipeline, end to end

```bash
# 1. Marker — your existing run, unchanged. The JSON renderer keeps the
#    Document block tree that POMA auto-detects, pages included.
marker_single contract.pdf --output_format json --output_dir out/
```

```python
import numpy as np
from redis import Redis
from redis.commands.search.field import VectorField, TagField, NumericField
from redis.commands.search.query import Query
from redis.commands.search.index_definition import IndexDefinition, IndexType
from poma import PrimeCut
from poma.vektoria import assemble, records_from_archive, Volume
from poma.embeddings import get_embedder

# 2. The missing link — raw Marker JSON in, .poma archive out (chunks +
#    chunksets, image bytes spliced back into their references).
client = PrimeCut()  # reads POMA_API_KEY
client.ingest("out/contract/contract.json", download_dir="archives", filename="contract.poma")

embedder = get_embedder("local:BAAI/bge-small-en-v1.5")
vol = Volume("s3://your-bucket/poma")
r = Redis(host="localhost", port=6379)

r.ft("poma_idx").create_index(
    fields=[
        VectorField("embedding", "HNSW", {"TYPE": "FLOAT32", "DIM": embedder.dims,
                                           "DISTANCE_METRIC": "COSINE"}),
        TagField("file_id"),
        NumericField("chunkset_index"),
    ],
    definition=IndexDefinition(prefix=["poma:"], index_type=IndexType.HASH),
)

# 3. Content-free ingest — Redis holds only routing metadata; content lives on the volume.
records = records_from_archive("archives/contract.poma")  # -> list[Record] (id, text, payload)
for rec in records:
    vol.write_doc(rec.payload["file_id"], {"file_id": rec.payload["file_id"],
                                            "chunks": rec.payload["chunks"], "text": rec.text})
    vector = np.asarray(embedder.embed([rec.text])[0], dtype=np.float32).tobytes()
    r.hset(f"poma:{rec.id}", mapping={
        "embedding": vector, "file_id": rec.payload["file_id"],
        "chunkset_index": rec.payload["chunkset_index"],
    })

# 4. KNN retrieval — the three-part contract Redis requires, then assemble.
qv = np.asarray(embedder.embed(["early termination conditions"])[0], dtype=np.float32).tobytes()
q = (Query("*=>[KNN 10 @embedding $vec AS vector_score]").sort_by("vector_score")
     .return_fields("file_id", "chunkset_index", "vector_score").dialect(2))
res = r.ft("poma_idx").search(q, query_params={"vec": qv})
context = assemble(res, volume=vol)  # -> [{"file_id", "content"}, ...]
```

POMA validates the payload shape up front — the JSON Document block tree is the strict auto-route fingerprint, and a corrupted or mislabeled upload 422s immediately instead of degrading the index. If what you saved is one of Marker's bare markdown shapes, declare it explicitly with `external_ocr_source="marker"`. On our reference legal document, this chunking answers the same query with **337 tokens** of retrieved context instead of **1,542** for a recursive character splitter — methodology in [Document Ingestion & Chunking for RAG](/document-ingestion-chunking-rag).

## Metadata mapping: POMA fields → Redis primitives

| POMA chunk field | Redis primitive | What it enables |
| --- | --- | --- |
| `to_embed` | `VECTOR` field (HNSW/FLAT, FLOAT32, COSINE) | similarity search over the embedded chunkset |
| `file_id` | `TAG` field | exact-match scoping to one document |
| `chunkset_index` | `NUMERIC` field | stable ordering, range filters, `assemble()` lookup key |
| `page` (JSON renderer only) | `NUMERIC` field | page-cited answers, page-range filters |
| chunkset content | not in Redis — lives on a `Volume` | RAM-efficient content-free index, reassembled at retrieval |

## Frequently asked questions

### How do I get Marker output into Redis for RAG?

Run Marker with the JSON renderer and save the result; hand it to `PrimeCut().ingest()`, which auto-detects the Document block tree, splices images back in, and writes a `.poma` archive of chunks and chunksets. Pull records with `records_from_archive()`, embed each record's `to_embed` text, and `HSET` them into a RediSearch index with a `VECTOR` field plus `TAG`/`NUMERIC` metadata. Query with the KNN syntax and hand the result to `assemble()` for prompt-ready context.

### Do Marker's extracted images make it into a Redis vector index?

Only if something reunites them first. Marker parks figure bytes in a side-channel `images` dict and leaves bare `![](name)` references in the markdown, so a pipeline that embeds only the markdown writes a Redis index with zero figures in it. POMA splices the bytes back into their references as data URIs and describes each figure into searchable text before `embedder.embed()` ever runs, so the figure becomes a real vector. References without bytes are neutralized and counted, never silently dropped.

### Why does my Redis KNN query return nothing after ingesting Marker output through POMA?

The ingest side is rarely the problem — it's almost always the query. RediSearch's KNN clause must alias its score `AS vector_score`, the query must call `.return_fields("file_id", "chunkset_index", "vector_score")`, and `.dialect(2)` must be set. Miss any of the three and RediSearch returns a result shape `assemble()` cannot read metadata from, so it comes back empty even though the HNSW search succeeded and Marker's chunksets are sitting in the index correctly.

### What Redis fields should Marker chunks carry?

A `VECTOR` field for the embedding (HNSW or FLAT, FLOAT32, `DIM` matching your embedder, COSINE), a `TAG` field for `file_id`, and a `NUMERIC` field for `chunkset_index` — the same fields `records_from_archive()` hands you after ingesting Marker's JSON block tree. Page numbers only exist if you ran Marker's JSON renderer; the default markdown output has no page boundaries to carry into a `NUMERIC` field at all.

### Does Marker's JSON renderer matter for a Redis pipeline?

Yes. The JSON renderer's Document block tree is what POMA auto-detects and what preserves per-page structure; Marker's default markdown output loses page boundaries entirely, so there is no page metadata to write into a Redis `NUMERIC` field even if you wanted it. The JSON renderer also keeps full HTML tables intact instead of collapsing them into one flat string, which matters once that content is deduplicated into a cheatsheet at retrieval time.

## Related recipes

Same parser, different store: [Marker → Qdrant](/pipelines/marker-to-qdrant) · [Marker → Pinecone](/pipelines/marker-to-pinecone) · [Marker → Chroma](/pipelines/marker-to-chroma)

Foundations: [The Optimal Chunker for Marker](/optimal-chunker-marker) · [Redis Chunking Strategy for RAG](/optimal-chunks-redis) · [All pipeline recipes](/pipelines/) · [RAG chunking guide](/guides/rag-chunking/)