Skip to content

Unstructured.io to Redis: Chunking for RAG ​

The short answer: Unstructured.io gives you a typed, ordered element list; Redis gives you an in-memory VECTOR field with TAG/NUMERIC filters that run as fast as the vector search itself. Wired together naively — join the element texts, split, embed, write hashes — they still produce mediocre RAG, because nothing in between rebuilds the hierarchy Unstructured's typing implies but never states. The missing link is POMA: PrimeCut().ingest() consumes the raw element list (auto-detected), emits hierarchy-preserving chunksets, and vektoria's content-free ingest pattern writes them into a RediSearch index with the metadata fields the KNN query needs. Retrieval comes back through assemble() as prompt-ready cheatsheets.

What each end of the pipeline actually provides ​

Unstructured.io (partition_pdf or the hosted API) returns a flat, ordered list of typed elements — Title, NarrativeText, ListItem, Table, Image, plus page furniture types — each carrying a stable element_id and metadata such as page_number and text_as_html. What it doesn't provide: any record of which Title nests under which, or a retrieval-ready unit. Details: The Optimal Chunker for Unstructured.io.

Redis (via the RediSearch module) provides a VECTOR field on a hash or JSON document, HNSW or FLAT indexing, and TAG/NUMERIC fields that filter at the same speed as the KNN search — all held in memory. What it doesn't provide: any opinion about what belongs in that field. It ranks whatever vector you wrote, including a context-free fragment, exactly as fast as it would rank a self-explanatory chunkset. Details: Redis Chunking Strategy for RAG.

The naive wiring, and where it breaks ​

The common recipe — concatenate every element's text, run a character splitter, embed, hset into Redis — breaks in a pair-specific way. Concatenating Unstructured's flat element stream re-admits the Header, Footer, and PageNumber noise the typing already isolated, and because Redis's VECTOR field lives entirely in RAM, every noise vector is a permanent memory cost, not a disk one — there is no cheap tier to demote it to. element_id and page_number vanish at the join, so no TAG/NUMERIC field can scope by document or cite a page.

Worse, even a correctly filtered ingest can look broken at query time for an unrelated reason: Redis's KNN syntax requires the score aliased AS vector_score, .return_fields(...) naming the metadata fields, and .dialect(2) — omit any one and assemble() receives a result shape it cannot read, returning empty from a perfectly healthy index. The two failure modes compound: a team debugging "empty retrieval" is as likely to be chasing noisy ingest as a missing query flag, and the symptom looks identical either way.

The pipeline, end to end ​

bash
pip install 'unstructured[pdf]' redis
python
import numpy as np
from unstructured.partition.pdf import partition_pdf
from unstructured.staging.base import elements_to_json
from redis import Redis
from redis.commands.search.field import VectorField, TagField, NumericField
from redis.commands.search.query import Query
from redis.commands.search.index_definition import IndexDefinition, IndexType
from poma import PrimeCut
from poma.vektoria import assemble, records_from_archive, Volume
from poma.embeddings import get_embedder

# 1. Your existing Unstructured call — unchanged.
elements = partition_pdf(
    filename="contract.pdf",
    strategy="hi_res",
    infer_table_structure=True,
    extract_image_block_types=["Image"],
    extract_image_block_to_payload=True,
)
elements_to_json(elements, filename="elements.json")

# 2. The missing link — raw element list in, a .poma archive out.
client = PrimeCut()  # reads POMA_API_KEY
client.ingest("elements.json", download_dir="archives", filename="doc.poma")

embedder = get_embedder("local:BAAI/bge-small-en-v1.5")
vol = Volume("s3://your-bucket/poma")
r = Redis(host="localhost", port=6379)

r.ft("poma_idx").create_index(
    fields=[
        VectorField("embedding", "HNSW", {"TYPE": "FLOAT32", "DIM": embedder.dims,
                                           "DISTANCE_METRIC": "COSINE"}),
        TagField("file_id"),
        NumericField("chunkset_index"),
    ],
    definition=IndexDefinition(prefix=["poma:"], index_type=IndexType.HASH),
)

# 3. Content-free ingest — vector index holds only routing metadata; content lives on the volume.
records = records_from_archive("archives/doc.poma")  # -> list[Record] (id, text, payload)
# assemble() reads the FULL document (chunks + chunksets) from the volume — write it once,
# not one {"chunks": <indices>, "text"} stub per record (which also overwrote itself each turn).
from poma.utils import unpack_poma_archive
_data = unpack_poma_archive(poma_archive_path="archives/doc.poma")
file_id = records[0].payload["file_id"]
vol.write_doc(file_id, {"file_id": file_id, "chunks": _data["chunks"], "chunksets": _data["chunksets"]})
for rec in records:
    vector = np.asarray(embedder.embed([rec.text])[0], dtype=np.float32).tobytes()
    r.hset(f"poma:{rec.id}", mapping={
        "embedding": vector, "file_id": rec.payload["file_id"],
        "chunkset_index": rec.payload["chunkset_index"],
    })

# 4. KNN retrieval — the three-part Redis contract, then assemble.
qv = np.asarray(embedder.embed(["early termination conditions"])[0], dtype=np.float32).tobytes()
q = (Query("*=>[KNN 10 @embedding $vec AS vector_score]").sort_by("vector_score")
     .return_fields("file_id", "chunkset_index", "vector_score").dialect(2))
res = r.ft("poma_idx").search(q, query_params={"vec": qv})
context = assemble(res, volume=vol)  # -> [{"file_id", "content"}, ...]

POMA validates the element list's shape up front (a corrupted or mislabeled upload 422s immediately), drops Header/Footer/PageNumber elements before any hash is written, splices Table elements' text_as_html so rows never reach Redis mid-cut, and rebuilds the cross-page heading tree before chunking. To force or suppress detection, set external_ocr_source to "unstructured" or "none".

Metadata mapping: POMA fields → Redis primitives ​

POMA chunk fieldRedis primitiveWhat it enables
to_embedVECTOR field (HNSW, FLOAT32, COSINE)KNN search over chunkset text
file_idTAG fieldscope the KNN query to one document
chunkset_indexNUMERIC fieldrequired by assemble(); stable ordering
page (from Unstructured metadata.page_number)NUMERIC field (optional)page-cited answers, page-range filters
depth (rebuilt from flat Title elements)NUMERIC field (optional)hierarchy-aware filter or re-rank
element_id (Unstructured's per-element identity)not written to Redis; kept in the volume payloadtraceability back to the exact source element

Frequently asked questions ​

How do I get Unstructured.io elements into Redis for RAG? ​

Save the element list with elements_to_json, run PrimeCut().ingest() on it (auto-detected, hierarchy rebuilt), then write each chunkset as a Redis hash with an embedding VECTOR field plus file_id/chunkset_index. Retrieve with a KNN query and hand the result to assemble().

Why does my Redis KNN query return nothing after ingesting Unstructured elements? ​

It's Redis's contract, not an Unstructured issue: alias the score AS vector_score, call .return_fields("file_id", "chunkset_index", "vector_score"), and set .dialect(2). Miss any one and assemble() gets nothing back from a healthy index.

What Redis fields should Unstructured chunk data carry? ​

At minimum a VECTOR field, TAG for file_id, and NUMERIC for chunkset_index. Add page (from page_number) and depth as optional NUMERIC fields for citation and hierarchy filtering.

Not through PrimeCut — they're dropped as page furniture before chunking. They only reappear if you bypass PrimeCut and hand-roll a join-then-split yourself.

Do images described from Unstructured survive into Redis retrieval? ​

Yes, when extract_image_block_to_payload=true keeps the bytes inline — POMA describes the figure and it's embedded like any other chunkset text. Images offloaded via image_path are neutralized and counted, never silently dropped.

Same parser, different store: Unstructured.io → Qdrant · Unstructured.io → Pinecone · Unstructured.io → Weaviate

Foundations: The Optimal Chunker for Unstructured.io · Redis Chunking Strategy for RAG · All pipeline recipes · RAG chunking guide