Source: http://www.poma-ai.com/docs/pipelines/unstructured-to-redis

# The Missing Link Between Unstructured.io and Optimal Retrieval in Redis

<ByAuthor />

**The short answer:** Unstructured.io gives you a typed, ordered element list; Redis gives you an in-memory `VECTOR` field with `TAG`/`NUMERIC` filters that run as fast as the vector search itself. Wired together naively — join the element texts, split, embed, write hashes — they still produce mediocre RAG, because nothing in between rebuilds the hierarchy Unstructured's typing implies but never states. The missing link is POMA: `PrimeCut().ingest()` consumes the raw element list (auto-detected), emits hierarchy-preserving [chunksets](/learn/chunking/chunksets), and `vektoria`'s content-free ingest pattern writes them into a RediSearch index with the metadata fields the KNN query needs. Retrieval comes back through `assemble()` as prompt-ready cheatsheets.

## What each end of the pipeline actually provides

**Unstructured.io** (`partition_pdf` or the hosted API) returns a flat, ordered list of typed elements — `Title`, `NarrativeText`, `ListItem`, `Table`, `Image`, plus page furniture types — each carrying a stable `element_id` and metadata such as `page_number` and `text_as_html`. What it doesn't provide: any record of which `Title` nests under which, or a retrieval-ready unit. Details: [The Optimal Chunker for Unstructured.io](/optimal-chunker-unstructured).

**Redis** (via the RediSearch module) provides a `VECTOR` field on a hash or JSON document, `HNSW` or `FLAT` indexing, and `TAG`/`NUMERIC` fields that filter at the same speed as the KNN search — all held in memory. What it doesn't provide: any opinion about what belongs in that field. It ranks whatever vector you wrote, including a context-free fragment, exactly as fast as it would rank a self-explanatory chunkset. Details: [Redis Chunking Strategy for RAG](/optimal-chunks-redis).

## The naive wiring, and where it breaks

The common recipe — concatenate every element's `text`, run a character splitter, embed, `hset` into Redis — breaks in a pair-specific way. Concatenating Unstructured's flat element stream re-admits the `Header`, `Footer`, and `PageNumber` noise the typing already isolated, and because Redis's `VECTOR` field lives entirely in RAM, every noise vector is a permanent memory cost, not a disk one — there is no cheap tier to demote it to. `element_id` and `page_number` vanish at the join, so no `TAG`/`NUMERIC` field can scope by document or cite a page.

Worse, even a correctly filtered ingest can look broken at query time for an unrelated reason: Redis's KNN syntax requires the score aliased `AS vector_score`, `.return_fields(...)` naming the metadata fields, and `.dialect(2)` — omit any one and `assemble()` receives a result shape it cannot read, returning empty from a perfectly healthy index. The two failure modes compound: a team debugging "empty retrieval" is as likely to be chasing noisy ingest as a missing query flag, and the symptom looks identical either way.

## The pipeline, end to end

```bash
pip install 'unstructured[pdf]' redis
```

```python
import numpy as np
from unstructured.partition.pdf import partition_pdf
from unstructured.staging.base import elements_to_json
from redis import Redis
from redis.commands.search.field import VectorField, TagField, NumericField
from redis.commands.search.query import Query
from redis.commands.search.index_definition import IndexDefinition, IndexType
from poma import PrimeCut
from poma.vektoria import assemble, records_from_archive, Volume
from poma.embeddings import get_embedder

# 1. Your existing Unstructured call — unchanged.
elements = partition_pdf(
    filename="contract.pdf",
    strategy="hi_res",
    infer_table_structure=True,
    extract_image_block_types=["Image"],
    extract_image_block_to_payload=True,
)
elements_to_json(elements, filename="elements.json")

# 2. The missing link — raw element list in, a .poma archive out.
client = PrimeCut()  # reads POMA_API_KEY
client.ingest("elements.json", download_dir="archives", filename="doc.poma")

embedder = get_embedder("local:BAAI/bge-small-en-v1.5")
vol = Volume("s3://your-bucket/poma")
r = Redis(host="localhost", port=6379)

r.ft("poma_idx").create_index(
    fields=[
        VectorField("embedding", "HNSW", {"TYPE": "FLOAT32", "DIM": embedder.dims,
                                           "DISTANCE_METRIC": "COSINE"}),
        TagField("file_id"),
        NumericField("chunkset_index"),
    ],
    definition=IndexDefinition(prefix=["poma:"], index_type=IndexType.HASH),
)

# 3. Content-free ingest — vector index holds only routing metadata; content lives on the volume.
records = records_from_archive("archives/doc.poma")  # -> list[Record] (id, text, payload)
for rec in records:
    vol.write_doc(rec.payload["file_id"], {"file_id": rec.payload["file_id"],
                                            "chunks": rec.payload["chunks"], "text": rec.text})
    vector = np.asarray(embedder.embed([rec.text])[0], dtype=np.float32).tobytes()
    r.hset(f"poma:{rec.id}", mapping={
        "embedding": vector, "file_id": rec.payload["file_id"],
        "chunkset_index": rec.payload["chunkset_index"],
    })

# 4. KNN retrieval — the three-part Redis contract, then assemble.
qv = np.asarray(embedder.embed(["early termination conditions"])[0], dtype=np.float32).tobytes()
q = (Query("*=>[KNN 10 @embedding $vec AS vector_score]").sort_by("vector_score")
     .return_fields("file_id", "chunkset_index", "vector_score").dialect(2))
res = r.ft("poma_idx").search(q, query_params={"vec": qv})
context = assemble(res, volume=vol)  # -> [{"file_id", "content"}, ...]
```

POMA validates the element list's shape up front (a corrupted or mislabeled upload 422s immediately), drops `Header`/`Footer`/`PageNumber` elements before any hash is written, splices `Table` elements' `text_as_html` so rows never reach Redis mid-cut, and rebuilds the cross-page heading tree before chunking. To force or suppress detection, set `external_ocr_source` to `"unstructured"` or `"none"`.

## Metadata mapping: POMA fields → Redis primitives

| POMA chunk field | Redis primitive | What it enables |
| --- | --- | --- |
| `to_embed` | `VECTOR` field (`HNSW`, `FLOAT32`, `COSINE`) | KNN search over chunkset text |
| `file_id` | `TAG` field | scope the KNN query to one document |
| `chunkset_index` | `NUMERIC` field | required by `assemble()`; stable ordering |
| `page` (from Unstructured `metadata.page_number`) | `NUMERIC` field (optional) | page-cited answers, page-range filters |
| `depth` (rebuilt from flat `Title` elements) | `NUMERIC` field (optional) | hierarchy-aware filter or re-rank |
| `element_id` (Unstructured's per-element identity) | not written to Redis; kept in the volume payload | traceability back to the exact source element |

## Frequently asked questions

### How do I get Unstructured.io elements into Redis for RAG?

Save the element list with `elements_to_json`, run `PrimeCut().ingest()` on it (auto-detected, hierarchy rebuilt), then write each chunkset as a Redis hash with an `embedding` `VECTOR` field plus `file_id`/`chunkset_index`. Retrieve with a `KNN` query and hand the result to `assemble()`.

### Why does my Redis KNN query return nothing after ingesting Unstructured elements?

It's Redis's contract, not an Unstructured issue: alias the score `AS vector_score`, call `.return_fields("file_id", "chunkset_index", "vector_score")`, and set `.dialect(2)`. Miss any one and `assemble()` gets nothing back from a healthy index.

### What Redis fields should Unstructured chunk data carry?

At minimum a `VECTOR` field, `TAG` for `file_id`, and `NUMERIC` for `chunkset_index`. Add `page` (from `page_number`) and `depth` as optional `NUMERIC` fields for citation and hierarchy filtering.

### Do Header, Footer, and PageNumber elements from Unstructured end up in my Redis index?

Not through PrimeCut — they're dropped as page furniture before chunking. They only reappear if you bypass PrimeCut and hand-roll a join-then-split yourself.

### Do images described from Unstructured survive into Redis retrieval?

Yes, when `extract_image_block_to_payload=true` keeps the bytes inline — POMA describes the figure and it's embedded like any other chunkset text. Images offloaded via `image_path` are neutralized and counted, never silently dropped.

## Related recipes

Same parser, different store: [Unstructured.io → Qdrant](/pipelines/unstructured-to-qdrant) · [Unstructured.io → Pinecone](/pipelines/unstructured-to-pinecone) · [Unstructured.io → Weaviate](/pipelines/unstructured-to-weaviate)

Foundations: [The Optimal Chunker for Unstructured.io](/optimal-chunker-unstructured) · [Redis Chunking Strategy for RAG](/optimal-chunks-redis) · [All pipeline recipes](/pipelines/) · [RAG chunking guide](/guides/rag-chunking/)