Marker to Redis: Chunking for RAG
The short answer: Marker gives you fast, local PDF parsing with real recovered structure; Redis gives you an in-memory vector index with exact-match filters as a first-class citizen. Wired together naively — flatten the markdown, split, embed, HSET only a vector field — the pipeline drops figures, loses page numbers, and often can't even be queried correctly once RediSearch's KNN contract is missed. The missing link is POMA: PrimeCut().ingest() consumes the saved Marker JSON (the Document block tree auto-detects), splices the side-channel image bytes back in, and rebuilds the cross-page hierarchy into chunksets. vektoria keeps Redis content-free — vectors and routing metadata only — and assemble() turns a correctly-shaped KNN result into prompt-ready context.
What each end of the pipeline actually provides
Marker (by Datalab / VikParuchuri) runs on your own hardware and turns PDFs into clean structured output. With the JSON renderer (--output_format json) you get a Document block tree — Page children, per-block HTML, full <table> elements with row and column spans. What it doesn't provide: cross-page hierarchy, retrieval units, or inline images — figure bytes live in a side-channel images dict keyed by name, with only  references left in the text. Details: The Optimal Chunker for Marker.
Redis (via the RediSearch module) provides a VECTOR field (HNSW/FLAT, FLOAT32, COSINE) alongside TAG/NUMERIC fields that filter at the same speed as the vector search itself, all served from memory. What it doesn't provide: any opinion about what the vector should represent, or any tolerance for a malformed query — get the KNN clause's three-part contract wrong and RediSearch hands back metadata-free hits from a search that otherwise succeeded. Details: Redis Chunking Strategy for RAG.
The naive wiring, and where it breaks
The common recipe — run Marker's default markdown output, RecursiveCharacterTextSplitter, embed, HSET with only an embedding field, then a bare "*=>[KNN 10 @embedding $vec]" query — breaks in a way specific to this pair:
- Every figure vanishes before Redis. Marker's images live in the side-channel
imagesdict; the markdown carries onlyreferences. A markdown-only pipeline embeds those dangling references as literal text, so the chart that answers a question was never written as a vector at all. - No
TAG/NUMERICfields exist to scope or cite from. Marker's default markdown has no page boundaries, and a naive HSET writes only the embedding — so Redis's fast in-memory filtering primitive sits unused even though it costs nothing extra to populate. - The KNN query itself is malformed. Without aliasing the score
AS vector_score, calling.return_fields(...), and setting.dialect(2), RediSearch returns a result shapeassemble()cannot parse — it comes back empty from a non-empty search. Two failures stack here: content that was never written, and content that was written but can't be read back throughvektoriabecause the query shape is wrong. - Overlap inflates the index. Splitter overlap embeds every boundary span twice, and because Redis holds everything in memory, that duplication is a direct RAM cost, not a disk one.
The pipeline, end to end
# 1. Marker — your existing run, unchanged. The JSON renderer keeps the
# Document block tree that POMA auto-detects, pages included.
marker_single contract.pdf --output_format json --output_dir out/import numpy as np
from redis import Redis
from redis.commands.search.field import VectorField, TagField, NumericField
from redis.commands.search.query import Query
from redis.commands.search.index_definition import IndexDefinition, IndexType
from poma import PrimeCut
from poma.vektoria import assemble, records_from_archive, Volume
from poma.embeddings import get_embedder
# 2. The missing link — raw Marker JSON in, .poma archive out (chunks +
# chunksets, image bytes spliced back into their references).
client = PrimeCut() # reads POMA_API_KEY
client.ingest("out/contract/contract.json", download_dir="archives", filename="contract.poma")
embedder = get_embedder("local:BAAI/bge-small-en-v1.5")
vol = Volume("s3://your-bucket/poma")
r = Redis(host="localhost", port=6379)
r.ft("poma_idx").create_index(
fields=[
VectorField("embedding", "HNSW", {"TYPE": "FLOAT32", "DIM": embedder.dims,
"DISTANCE_METRIC": "COSINE"}),
TagField("file_id"),
NumericField("chunkset_index"),
],
definition=IndexDefinition(prefix=["poma:"], index_type=IndexType.HASH),
)
# 3. Content-free ingest — Redis holds only routing metadata; content lives on the volume.
records = records_from_archive("archives/contract.poma") # -> list[Record] (id, text, payload)
# assemble() reads the FULL document (chunks + chunksets) from the volume — write it once,
# not one {"chunks": <indices>, "text"} stub per record (which also overwrote itself each turn).
from poma.utils import unpack_poma_archive
_data = unpack_poma_archive(poma_archive_path="archives/contract.poma")
file_id = records[0].payload["file_id"]
vol.write_doc(file_id, {"file_id": file_id, "chunks": _data["chunks"], "chunksets": _data["chunksets"]})
for rec in records:
vector = np.asarray(embedder.embed([rec.text])[0], dtype=np.float32).tobytes()
r.hset(f"poma:{rec.id}", mapping={
"embedding": vector, "file_id": rec.payload["file_id"],
"chunkset_index": rec.payload["chunkset_index"],
})
# 4. KNN retrieval — the three-part contract Redis requires, then assemble.
qv = np.asarray(embedder.embed(["early termination conditions"])[0], dtype=np.float32).tobytes()
q = (Query("*=>[KNN 10 @embedding $vec AS vector_score]").sort_by("vector_score")
.return_fields("file_id", "chunkset_index", "vector_score").dialect(2))
res = r.ft("poma_idx").search(q, query_params={"vec": qv})
context = assemble(res, volume=vol) # -> [{"file_id", "content"}, ...]POMA validates the payload shape up front — the JSON Document block tree is the strict auto-route fingerprint, and a corrupted or mislabeled upload 422s immediately instead of degrading the index. If what you saved is one of Marker's bare markdown shapes, declare it explicitly with external_ocr_source="marker". On our reference legal document, this chunking answers the same query with 337 tokens of retrieved context instead of 1,542 for a recursive character splitter — methodology in Document Ingestion & Chunking for RAG.
Metadata mapping: POMA fields → Redis primitives
| POMA chunk field | Redis primitive | What it enables |
|---|---|---|
to_embed | VECTOR field (HNSW/FLAT, FLOAT32, COSINE) | similarity search over the embedded chunkset |
file_id | TAG field | exact-match scoping to one document |
chunkset_index | NUMERIC field | stable ordering, range filters, assemble() lookup key |
page (JSON renderer only) | NUMERIC field | page-cited answers, page-range filters |
| chunkset content | not in Redis — lives on a Volume | RAM-efficient content-free index, reassembled at retrieval |
Frequently asked questions
How do I get Marker output into Redis for RAG?
Run Marker with the JSON renderer and save the result; hand it to PrimeCut().ingest(), which auto-detects the Document block tree, splices images back in, and writes a .poma archive of chunks and chunksets. Pull records with records_from_archive(), embed each record's to_embed text, and HSET them into a RediSearch index with a VECTOR field plus TAG/NUMERIC metadata. Query with the KNN syntax and hand the result to assemble() for prompt-ready context.
Do Marker's extracted images make it into a Redis vector index?
Only if something reunites them first. Marker parks figure bytes in a side-channel images dict and leaves bare  references in the markdown, so a pipeline that embeds only the markdown writes a Redis index with zero figures in it. POMA splices the bytes back into their references as data URIs and describes each figure into searchable text before embedder.embed() ever runs, so the figure becomes a real vector. References without bytes are neutralized and counted, never silently dropped.
Why does my Redis KNN query return nothing after ingesting Marker output through POMA?
The ingest side is rarely the problem — it's almost always the query. RediSearch's KNN clause must alias its score AS vector_score, the query must call .return_fields("file_id", "chunkset_index", "vector_score"), and .dialect(2) must be set. Miss any of the three and RediSearch returns a result shape assemble() cannot read metadata from, so it comes back empty even though the HNSW search succeeded and Marker's chunksets are sitting in the index correctly.
What Redis fields should Marker chunks carry?
A VECTOR field for the embedding (HNSW or FLAT, FLOAT32, DIM matching your embedder, COSINE), a TAG field for file_id, and a NUMERIC field for chunkset_index — the same fields records_from_archive() hands you after ingesting Marker's JSON block tree. Page numbers only exist if you ran Marker's JSON renderer; the default markdown output has no page boundaries to carry into a NUMERIC field at all.
Does Marker's JSON renderer matter for a Redis pipeline?
Yes. The JSON renderer's Document block tree is what POMA auto-detects and what preserves per-page structure; Marker's default markdown output loses page boundaries entirely, so there is no page metadata to write into a Redis NUMERIC field even if you wanted it. The JSON renderer also keeps full HTML tables intact instead of collapsing them into one flat string, which matters once that content is deduplicated into a cheatsheet at retrieval time.
Related recipes
Same parser, different store: Marker → Qdrant · Marker → Pinecone · Marker → Chroma
Foundations: The Optimal Chunker for Marker · Redis Chunking Strategy for RAG · All pipeline recipes · RAG chunking guide