Unstructured.io to Redis: Chunking for RAG
The short answer: Unstructured.io gives you a typed, ordered element list; Redis gives you an in-memory VECTOR field with TAG/NUMERIC filters that run as fast as the vector search itself. Wired together naively — join the element texts, split, embed, write hashes — they still produce mediocre RAG, because nothing in between rebuilds the hierarchy Unstructured's typing implies but never states. The missing link is POMA: PrimeCut().ingest() consumes the raw element list (auto-detected), emits hierarchy-preserving chunksets, and vektoria's content-free ingest pattern writes them into a RediSearch index with the metadata fields the KNN query needs. Retrieval comes back through assemble() as prompt-ready cheatsheets.
What each end of the pipeline actually provides
Unstructured.io (partition_pdf or the hosted API) returns a flat, ordered list of typed elements — Title, NarrativeText, ListItem, Table, Image, plus page furniture types — each carrying a stable element_id and metadata such as page_number and text_as_html. What it doesn't provide: any record of which Title nests under which, or a retrieval-ready unit. Details: The Optimal Chunker for Unstructured.io.
Redis (via the RediSearch module) provides a VECTOR field on a hash or JSON document, HNSW or FLAT indexing, and TAG/NUMERIC fields that filter at the same speed as the KNN search — all held in memory. What it doesn't provide: any opinion about what belongs in that field. It ranks whatever vector you wrote, including a context-free fragment, exactly as fast as it would rank a self-explanatory chunkset. Details: Redis Chunking Strategy for RAG.
The naive wiring, and where it breaks
The common recipe — concatenate every element's text, run a character splitter, embed, hset into Redis — breaks in a pair-specific way. Concatenating Unstructured's flat element stream re-admits the Header, Footer, and PageNumber noise the typing already isolated, and because Redis's VECTOR field lives entirely in RAM, every noise vector is a permanent memory cost, not a disk one — there is no cheap tier to demote it to. element_id and page_number vanish at the join, so no TAG/NUMERIC field can scope by document or cite a page.
Worse, even a correctly filtered ingest can look broken at query time for an unrelated reason: Redis's KNN syntax requires the score aliased AS vector_score, .return_fields(...) naming the metadata fields, and .dialect(2) — omit any one and assemble() receives a result shape it cannot read, returning empty from a perfectly healthy index. The two failure modes compound: a team debugging "empty retrieval" is as likely to be chasing noisy ingest as a missing query flag, and the symptom looks identical either way.
The pipeline, end to end
pip install 'unstructured[pdf]' redisimport numpy as np
from unstructured.partition.pdf import partition_pdf
from unstructured.staging.base import elements_to_json
from redis import Redis
from redis.commands.search.field import VectorField, TagField, NumericField
from redis.commands.search.query import Query
from redis.commands.search.index_definition import IndexDefinition, IndexType
from poma import PrimeCut
from poma.vektoria import assemble, records_from_archive, Volume
from poma.embeddings import get_embedder
# 1. Your existing Unstructured call — unchanged.
elements = partition_pdf(
filename="contract.pdf",
strategy="hi_res",
infer_table_structure=True,
extract_image_block_types=["Image"],
extract_image_block_to_payload=True,
)
elements_to_json(elements, filename="elements.json")
# 2. The missing link — raw element list in, a .poma archive out.
client = PrimeCut() # reads POMA_API_KEY
client.ingest("elements.json", download_dir="archives", filename="doc.poma")
embedder = get_embedder("local:BAAI/bge-small-en-v1.5")
vol = Volume("s3://your-bucket/poma")
r = Redis(host="localhost", port=6379)
r.ft("poma_idx").create_index(
fields=[
VectorField("embedding", "HNSW", {"TYPE": "FLOAT32", "DIM": embedder.dims,
"DISTANCE_METRIC": "COSINE"}),
TagField("file_id"),
NumericField("chunkset_index"),
],
definition=IndexDefinition(prefix=["poma:"], index_type=IndexType.HASH),
)
# 3. Content-free ingest — vector index holds only routing metadata; content lives on the volume.
records = records_from_archive("archives/doc.poma") # -> list[Record] (id, text, payload)
# assemble() reads the FULL document (chunks + chunksets) from the volume — write it once,
# not one {"chunks": <indices>, "text"} stub per record (which also overwrote itself each turn).
from poma.utils import unpack_poma_archive
_data = unpack_poma_archive(poma_archive_path="archives/doc.poma")
file_id = records[0].payload["file_id"]
vol.write_doc(file_id, {"file_id": file_id, "chunks": _data["chunks"], "chunksets": _data["chunksets"]})
for rec in records:
vector = np.asarray(embedder.embed([rec.text])[0], dtype=np.float32).tobytes()
r.hset(f"poma:{rec.id}", mapping={
"embedding": vector, "file_id": rec.payload["file_id"],
"chunkset_index": rec.payload["chunkset_index"],
})
# 4. KNN retrieval — the three-part Redis contract, then assemble.
qv = np.asarray(embedder.embed(["early termination conditions"])[0], dtype=np.float32).tobytes()
q = (Query("*=>[KNN 10 @embedding $vec AS vector_score]").sort_by("vector_score")
.return_fields("file_id", "chunkset_index", "vector_score").dialect(2))
res = r.ft("poma_idx").search(q, query_params={"vec": qv})
context = assemble(res, volume=vol) # -> [{"file_id", "content"}, ...]POMA validates the element list's shape up front (a corrupted or mislabeled upload 422s immediately), drops Header/Footer/PageNumber elements before any hash is written, splices Table elements' text_as_html so rows never reach Redis mid-cut, and rebuilds the cross-page heading tree before chunking. To force or suppress detection, set external_ocr_source to "unstructured" or "none".
Metadata mapping: POMA fields → Redis primitives
| POMA chunk field | Redis primitive | What it enables |
|---|---|---|
to_embed | VECTOR field (HNSW, FLOAT32, COSINE) | KNN search over chunkset text |
file_id | TAG field | scope the KNN query to one document |
chunkset_index | NUMERIC field | required by assemble(); stable ordering |
page (from Unstructured metadata.page_number) | NUMERIC field (optional) | page-cited answers, page-range filters |
depth (rebuilt from flat Title elements) | NUMERIC field (optional) | hierarchy-aware filter or re-rank |
element_id (Unstructured's per-element identity) | not written to Redis; kept in the volume payload | traceability back to the exact source element |
Frequently asked questions
How do I get Unstructured.io elements into Redis for RAG?
Save the element list with elements_to_json, run PrimeCut().ingest() on it (auto-detected, hierarchy rebuilt), then write each chunkset as a Redis hash with an embedding VECTOR field plus file_id/chunkset_index. Retrieve with a KNN query and hand the result to assemble().
Why does my Redis KNN query return nothing after ingesting Unstructured elements?
It's Redis's contract, not an Unstructured issue: alias the score AS vector_score, call .return_fields("file_id", "chunkset_index", "vector_score"), and set .dialect(2). Miss any one and assemble() gets nothing back from a healthy index.
What Redis fields should Unstructured chunk data carry?
At minimum a VECTOR field, TAG for file_id, and NUMERIC for chunkset_index. Add page (from page_number) and depth as optional NUMERIC fields for citation and hierarchy filtering.
Do Header, Footer, and PageNumber elements from Unstructured end up in my Redis index?
Not through PrimeCut — they're dropped as page furniture before chunking. They only reappear if you bypass PrimeCut and hand-roll a join-then-split yourself.
Do images described from Unstructured survive into Redis retrieval?
Yes, when extract_image_block_to_payload=true keeps the bytes inline — POMA describes the figure and it's embedded like any other chunkset text. Images offloaded via image_path are neutralized and counted, never silently dropped.
Related recipes
Same parser, different store: Unstructured.io → Qdrant · Unstructured.io → Pinecone · Unstructured.io → Weaviate
Foundations: The Optimal Chunker for Unstructured.io · Redis Chunking Strategy for RAG · All pipeline recipes · RAG chunking guide