Source: http://www.poma-ai.com/docs/pipelines/docling-to-redis

# The Missing Link Between Docling and Optimal Retrieval in Redis

<ByAuthor />

**The short answer:** Docling gives you an explicit, typed document tree; Redis gives you a fast in-memory vector index with exact-match filters. Wired together the common way — flatten to markdown, split, embed, one field per hash — both tools end up worse than either alone, because nothing rebuilds cross-page hierarchy or keeps the index content-free. The missing link is POMA: `PrimeCut().ingest()` consumes the DoclingDocument tree directly (or the JSON export), emits hierarchy-preserving [chunksets](/learn/chunking/chunksets), and `vektoria` writes them into Redis as content-free vectors — `{file_id, chunkset_index}` only — while the actual text lives on a volume. Retrieval runs Redis's own KNN query, then `assemble()` reconstructs prompt-ready cheatsheets.

## What each end of the pipeline actually provides

**Docling** (IBM's open-source converter, also served via docling-serve) returns a typed **DoclingDocument**: `texts`/`tables`/`pictures`/`groups`, with `section_header` items carrying an explicit numeric `level` and repeating page furniture pre-isolated into a `furniture` group labeled `page_header`/`page_footer`. What it doesn't provide is a retrieval unit — its own `HybridChunker` packs tree items into token windows sized for an embedding model, which controls length, not whether a retrieved passage carries the heading path that explains it. Details: [The Optimal Chunker for Docling](/optimal-chunker-docling).

**Redis** (via the RediSearch module) provides `FT.CREATE` with a `VECTOR` field (`HNSW`/`FLAT`, `TYPE FLOAT32`, `DIM`, `DISTANCE_METRIC`) plus `TAG`/`NUMERIC` fields that filter at the same speed as the vector search itself — genuinely fast, entirely in memory. What it doesn't provide is any opinion about what belongs in that `VECTOR` field. It ranks whatever you wrote. Details: [Redis Chunking Strategy for RAG](/optimal-chunks-redis).

## The naive wiring, and where it breaks

The common recipe — `doc.export_to_markdown()` → `RecursiveCharacterTextSplitter` → one Redis hash per fragment with a single `text` field — breaks in a pair-specific way:

- **Docling's `section_header` levels are discarded at export.** The tree already told you a passage lives under *Termination clauses* under *Master Services Agreement*; flattening to markdown reduces that to `##` glyphs, then the splitter ignores them and cuts wherever the character count lands.
- **Furniture Docling already isolated re-enters the text stream.** The `furniture` group and `page_header`/`page_footer` labels exist specifically so downstream tools don't have to re-detect running noise — flattening throws that classification away, and page numbers end up embedded and indexed.
- **Overlap is a direct RAM cost in Redis.** Splitter overlap duplicates every boundary span into the `HNSW`/`FLAT` structure RediSearch builds — and because that structure lives entirely in memory, redundant near-duplicate vectors cost RAM you're paying for on every node, not just crowding top-k.
- **The retrieval query silently returns nothing.** Even with perfect hierarchy, a Redis `KNN` clause that doesn't alias the score `AS vector_score`, call `.return_fields(...)`, and set `.dialect(2)` hands `assemble()` a shape it can't parse metadata from — the search succeeds, but the pipeline appears to return zero results downstream, and this failure mode looks identical whether the vectors came from Docling, Mistral OCR, or anything else, which makes it easy to misdiagnose as a Docling parsing problem when it isn't.

## The pipeline, end to end

```bash
pip install docling redis
```

```python
import json
import numpy as np
from docling.document_converter import DocumentConverter
from redis import Redis
from redis.commands.search.field import VectorField, TagField, NumericField
from redis.commands.search.query import Query
from redis.commands.search.index_definition import IndexDefinition, IndexType
from poma import PrimeCut
from poma.vektoria import assemble, records_from_archive, Volume
from poma.embeddings import get_embedder

# 1. Your existing Docling conversion — unchanged. Save the tree as JSON.
converter = DocumentConverter()
doc = converter.convert("contract.pdf").document
with open("contract.docling.json", "w") as f:
    json.dump(doc.export_to_dict(), f)

# 2. The missing link — raw DoclingDocument JSON in, a portable .poma archive out.
client = PrimeCut()  # reads POMA_API_KEY
client.ingest("contract.docling.json", download_dir="archives", filename="contract.poma")

embedder = get_embedder("local:BAAI/bge-small-en-v1.5")
vol = Volume("s3://your-bucket/poma")
r = Redis(host="localhost", port=6379)

r.ft("poma_idx").create_index(
    fields=[
        VectorField("embedding", "HNSW", {"TYPE": "FLOAT32", "DIM": embedder.dims,
                                           "DISTANCE_METRIC": "COSINE"}),
        TagField("file_id"),
        NumericField("chunkset_index"),
    ],
    definition=IndexDefinition(prefix=["poma:"], index_type=IndexType.HASH),
)

# 3. Content-free ingest — Redis holds only routing metadata; content lives on the volume.
records = records_from_archive("archives/contract.poma")  # -> list[Record] (id, text, payload)
for rec in records:
    vol.write_doc(rec.payload["file_id"], {"file_id": rec.payload["file_id"],
                                            "chunks": rec.payload["chunks"], "text": rec.text})
    vector = np.asarray(embedder.embed([rec.text])[0], dtype=np.float32).tobytes()
    r.hset(f"poma:{rec.id}", mapping={
        "embedding": vector, "file_id": rec.payload["file_id"],
        "chunkset_index": rec.payload["chunkset_index"],
    })

# 4. KNN retrieval — the three-part contract Redis requires, then assemble.
qv = np.asarray(embedder.embed(["early termination conditions"])[0], dtype=np.float32).tobytes()
q = (Query("*=>[KNN 10 @embedding $vec AS vector_score]").sort_by("vector_score")
     .return_fields("file_id", "chunkset_index", "vector_score").dialect(2))
res = r.ft("poma_idx").search(q, query_params={"vec": qv})
context = assemble(res, volume=vol)  # -> [{"file_id", "content"}, ...]
```

`records_from_archive` walks the `.poma` archive produced by ingest — the same archive that's the volume's source of truth — and returns one `Record` per chunkset: a deterministic `id`, the `to_embed` text, and a `payload` carrying `file_id`, `chunkset_index`, and the member chunk list. No `push()` connector exists for Redis yet; ingest here is the native `redis` client, shaped exactly the way vektoria's `assemble()` already expects on the way back out.

## Metadata mapping: POMA fields → Redis primitives

| POMA record field | Redis primitive | What it enables |
| --- | --- | --- |
| `id` (`chunkset_uuid`) | hash key suffix (`poma:{id}`) | deterministic key — safe re-ingest, no duplicate vectors |
| `text` (`to_embed`) | `VECTOR` field (`embedding`) | the KNN similarity search itself |
| `payload["file_id"]` | `TAG` field | exact-match scoping to one document in the same query |
| `payload["chunkset_index"]` | `NUMERIC` field | routes a hit back to its content on the volume |
| chunkset content | volume (`Volume.write_doc`), never Redis | small in-memory hashes; citations reconstructed at retrieval |

## Frequently asked questions

### How do I get Docling output into Redis for RAG?

Export the DoclingDocument tree with `export_to_dict` and save it as JSON. `PrimeCut().ingest()` auto-detects the shape and rebuilds hierarchy into chunksets; write each one's content to a `vektoria` `Volume`, embed its `to_embed` text, and upsert a content-free vector keyed by `file_id`/`chunkset_index` into a RediSearch `VECTOR` field. Retrieve with a normal KNN query, then `assemble()`.

### Why not just export Docling to markdown and embed it into Redis directly?

`export_to_markdown()` flattens explicit `section_header` levels and pre-isolated furniture into display text a splitter must re-guess, and overlap duplicates spans into Redis's `HNSW`/`FLAT` structure — a direct RAM cost, since RediSearch indexes live entirely in memory.

### What Redis fields should Docling chunksets carry?

A `VECTOR` field for the embedding, a `TAG` field for `file_id`, and a `NUMERIC` field for `chunkset_index` — the minimum vektoria's content-free contract needs. Docling's hierarchy is already resolved upstream by PrimeCut, so it lives in the chunkset text, not a separate index field.

### Why does my Redis KNN query return nothing after ingesting Docling chunksets?

Alias the score `AS vector_score`, call `.return_fields("file_id", "chunkset_index", ...)`, and set `.dialect(2)`. Miss any of the three and `assemble()` gets an unparseable result shape back — regardless of which parser produced the vectors.

### Does Redis support hybrid search for Docling-parsed documents?

Yes, via filter-then-search: a `TAG` or text predicate prefixes the `KNN` clause in one query string. This helps with Docling-sourced contracts and technical documents, which carry exact tokens dense embeddings alone tend to blur.

## Related recipes

Same parser, different store: [Docling → Qdrant](/pipelines/docling-to-qdrant) · [Docling → Pinecone](/pipelines/docling-to-pinecone) · [Docling → Chroma](/pipelines/docling-to-chroma)

Foundations: [The Optimal Chunker for Docling](/optimal-chunker-docling) · [Redis Chunking Strategy for RAG](/optimal-chunks-redis) · [All pipeline recipes](/pipelines/) · [RAG architecture guide](/guides/rag-architecture/)