Skip to content

Unstructured.io to Qdrant: Chunking for RAG ​

The short answer: Unstructured.io gives you excellent typed elements; Qdrant gives you excellent hybrid vector search. Wired together naively — chunk_by_title, embed, upsert — they still produce mediocre RAG, because a flat element list never contained the ancestor hierarchy that retrieval needs. The missing link is POMA: PrimeCut().ingest() consumes the raw elements_to_json output (auto-detected), emits hierarchy-preserving chunksets, and PomaQdrant.upsert_poma_points() writes them as hybrid dense+sparse points that carry the chunkset's lineage in the payload. Retrieval comes back as assembled, prompt-ready cheatsheets.

What each end of the pipeline actually provides ​

Unstructured.io (partition_pdf or the hosted API) returns a flat, ordered list of typed elements — Title, NarrativeText, ListItem, Table, Image — each with a stable element_id and metadata such as page_number and text_as_html for tables. What it doesn't provide: nesting. Two Title elements sit side by side in the stream whether one is a chapter and the other a sub-subsection, and the built-in by_title and basic chunkers emit flat fragments bounded by character limits. Details: The Optimal Chunker for Unstructured.io.

Qdrant provides named dense + sparse vectors on one point, payload indexes for filtered ANN, and HNSW with quantization options. What it doesn't provide: any opinion about what a point should contain. It retrieves nearest neighbors of whatever you embedded — including context-free fragments, if that's what you gave it. Details: The Optimal Chunks for the Best Retrieval in Qdrant.

The naive wiring, and where it breaks ​

The common recipe — chunk_by_title(elements) → embed each fragment → client.upsert(...) — breaks in a pair-specific way:

  • chunk_by_title fragments carry element metadata but no ancestor path. Each fragment knows its page_number, its filename, and at best its nearest Title — because that's all a flat element list contains. Nothing records that the fragment's section lives inside a chapter inside a part.
  • Qdrant's hybrid search can't recover lineage that was never stored. BM25 sparse vectors and dense vectors improve ranking; the winning point still arrives at the LLM as a bare paragraph. No query-time trick reconstructs a heading tree out of payloads that never held one.
  • Long sections get re-split by max_characters, so the tail fragments lose even the nearest-title link — the one piece of context by_title promised.

The result: Qdrant performs exactly as designed, faithfully retrieving fragments that answer out of context. The fix belongs between the two tools, not in either of them.

The pipeline, end to end ​

bash
pip install unstructured 'poma[qdrant]'
python
import os
from unstructured.partition.pdf import partition_pdf
from unstructured.staging.base import elements_to_json
from poma import PrimeCut
from poma.integrations.qdrant import PomaQdrant

# 1. Your existing Unstructured call — unchanged.
elements = partition_pdf(
    filename="contract.pdf",
    strategy="hi_res",
    infer_table_structure=True,            # populates metadata.text_as_html
    extract_image_block_types=["Image"],
    extract_image_block_to_payload=True,   # inline base64 → POMA can describe figures
)
elements_to_json(elements, filename="contract.unstructured.json")

# 2. The missing link — raw element list in, chunks + chunksets out.
poma = PrimeCut()  # reads POMA_API_KEY
result = poma.ingest("contract.unstructured.json")  # Unstructured shape auto-detected

# 3. Qdrant — hybrid points with hierarchy payloads, one call.
qdrant = PomaQdrant(
    url=os.environ["QDRANT_URL"],
    api_key=os.environ["QDRANT_API_KEY"],
    cloud_inference=False,  # True needs Qdrant Cloud's inference service; False embeds locally via fastembed and works on OSS/local Qdrant too
    collection_name="contracts",
    dense_model="sentence-transformers/all-MiniLM-L6-v2",
    sparse_model="Qdrant/bm25",
    dense_size=384,
    auto_create_collection=True,
)
qdrant.upsert_poma_points(result)

# 4. Retrieve prompt-ready context.
cheatsheets = qdrant.get_cheatsheets(
    query="What are the early termination conditions?",
    limit=3,
)
print(cheatsheets[0]["content"])

POMA validates the payload shape up front (a corrupted or mislabeled upload 422s immediately), splices text_as_html for tables, describes inline images, drops Header/Footer/PageNumber elements as page furniture, restores under-marked headings, and rebuilds the cross-page heading tree before chunking. To force or suppress detection, set external_ocr_source to "unstructured" or "none".

On our reference legal-document benchmark, this pipeline answers the same query with 337 tokens of retrieved context instead of 1,542 for a recursive character splitter, with zero information loss — methodology in Document Ingestion & Chunking for RAG. Overlap-free chunksets also mean no near-duplicate vectors crowding Qdrant's top-k.

Metadata mapping: POMA fields → Qdrant primitives ​

One point per chunkset. upsert_poma_points writes exactly this payload: {"chunkset_index", "chunks", "file_id", "text"} plus "chunk_details" (the chunkset's chunk records — chunk_index, content, depth, file_id, code) while store_chunk_details stays at its default True.

POMA fieldQdrant primitiveWhat it enables
to_embeddense vector + BM25 sparse vector (named vectors, one point)paraphrase + exact-term hybrid retrieval
text (a copy of to_embed)payload fieldread back what was embedded
file_idpayload field (no index is created for you — add one with create_payload_index)scope queries to one document
chunkset_indexpayload fieldidentifies the point: the id is a UUIDv5 of file_id + chunkset_index
chunks (chunk indices)payload fieldjoins the point back to its chunk records
chunk lineage (chunk_index, content, depth, code)payload (chunk_details)cheatsheet assembly without a second store
page (from metadata.page_number)not written by PomaQdrant — the SDK's PomaChunk drops it; it survives on the archive's chunk recordspage-cited answers, page-range filters — see below

get_cheatsheets reads only that payload back: it groups each hit's chunkset_index + chunks by file_id and assembles the text from the hit's chunk_details (or from a chunk_data you pass instead).

Page numbers are in the .poma archive, so keep it (poma.ingest(..., download_dir="store", filename="contract.poma")) and attach them to the same points:

python
from poma.utils import unpack_poma_archive
from poma.integrations.qdrant.qdrant_poma_utils import chunk_uuid_string

page_of = {c["chunk_index"]: c.get("page") for c in unpack_poma_archive(poma_archive_path="store/contract.poma")["chunks"]}
for cs in result.chunksets:
    leaf = cs.chunks[-1]  # chunk indices are in document order; the last one is the leaf
    qdrant.set_payload(collection_name="contracts", payload={"page": page_of[leaf]}, points=[chunk_uuid_string(cs.file_id, cs.chunkset_index)])

Frequently asked questions ​

How do I get Unstructured.io elements into Qdrant for RAG? ​

Save the element list with elements_to_json, run PrimeCut().ingest() on it (auto-detected, hierarchy rebuilt), then PomaQdrant.upsert_poma_points(result) — hybrid points whose payload carries file_id, chunkset_index, the chunkset's chunk indices, the embedded text, and chunk_details (chunk_index, content, depth) for assembly. Retrieve with get_cheatsheets(query=...).

Can Qdrant hybrid search make up for chunk_by_title's flat fragments? ​

No. Hybrid search improves ranking, not content. A chunk_by_title fragment carries page_number and its nearest Title but no ancestor path — the flat element list never had one — and Qdrant can't recover lineage that was never stored. POMA chunksets carry lineage inside the retrieval unit itself.

What Qdrant payload fields should Unstructured-derived chunks carry? ​

PomaQdrant writes file_id, chunkset_index, chunks, text and chunk_details (the chunk records with chunk_index, content and depth) by default. It does not write page, and it creates no payload indexes: add a payload index on file_id yourself for per-document scoping, and if you need page-cited answers or page-range filters, read page (from metadata.page_number) off the archive's chunk records with unpack_poma_archive and attach it with client.set_payload on the same point ids.

Do Unstructured's tables and images survive the trip to Qdrant? ​

Tables yes — text_as_html is spliced, never cut mid-row. Images yes, if partitioned with extract_image_block_to_payload=true: POMA describes each figure and the description becomes a searchable point. image_path images on disk are neutralized and counted in content_metadata — visible loss, never silent.

Should I store Unstructured's element metadata in Qdrant payloads? ​

Only the fields you filter on. PomaQdrant already writes file_id, chunkset_index, chunks, text and the chunkset lineage in chunk_details; add a payload index on file_id yourself, and attach page from the archive with client.set_payload only if you actually filter on it. Coordinates, emphasized-text spans, and other per-element metadata are ingestion-time detail — leave them in the .poma archive.

Same parser, different store: Unstructured → Pinecone · Unstructured → Weaviate · Unstructured → pgvector

Same store, different parser: Mistral OCR → Qdrant · LlamaParse → Qdrant · Docling → Qdrant

Foundations: The Optimal Chunker for Unstructured.io · The Optimal Chunks for Qdrant · All pipeline recipes · RAG chunking guide