Unstructured.io to Qdrant: Chunking for RAG
The short answer: Unstructured.io gives you excellent typed elements; Qdrant gives you excellent hybrid vector search. Wired together naively — chunk_by_title, embed, upsert — they still produce mediocre RAG, because a flat element list never contained the ancestor hierarchy that retrieval needs. The missing link is POMA: PrimeCut().ingest() consumes the raw elements_to_json output (auto-detected), emits hierarchy-preserving chunksets, and PomaQdrant.upsert_poma_points() writes them as hybrid dense+sparse points that carry the chunkset's lineage in the payload. Retrieval comes back as assembled, prompt-ready cheatsheets.
What each end of the pipeline actually provides
Unstructured.io (partition_pdf or the hosted API) returns a flat, ordered list of typed elements — Title, NarrativeText, ListItem, Table, Image — each with a stable element_id and metadata such as page_number and text_as_html for tables. What it doesn't provide: nesting. Two Title elements sit side by side in the stream whether one is a chapter and the other a sub-subsection, and the built-in by_title and basic chunkers emit flat fragments bounded by character limits. Details: The Optimal Chunker for Unstructured.io.
Qdrant provides named dense + sparse vectors on one point, payload indexes for filtered ANN, and HNSW with quantization options. What it doesn't provide: any opinion about what a point should contain. It retrieves nearest neighbors of whatever you embedded — including context-free fragments, if that's what you gave it. Details: The Optimal Chunks for the Best Retrieval in Qdrant.
The naive wiring, and where it breaks
The common recipe — chunk_by_title(elements) → embed each fragment → client.upsert(...) — breaks in a pair-specific way:
chunk_by_titlefragments carry element metadata but no ancestor path. Each fragment knows itspage_number, itsfilename, and at best its nearestTitle— because that's all a flat element list contains. Nothing records that the fragment's section lives inside a chapter inside a part.- Qdrant's hybrid search can't recover lineage that was never stored. BM25 sparse vectors and dense vectors improve ranking; the winning point still arrives at the LLM as a bare paragraph. No query-time trick reconstructs a heading tree out of payloads that never held one.
- Long sections get re-split by
max_characters, so the tail fragments lose even the nearest-title link — the one piece of contextby_titlepromised.
The result: Qdrant performs exactly as designed, faithfully retrieving fragments that answer out of context. The fix belongs between the two tools, not in either of them.
The pipeline, end to end
pip install unstructured 'poma[qdrant]'import os
from unstructured.partition.pdf import partition_pdf
from unstructured.staging.base import elements_to_json
from poma import PrimeCut
from poma.integrations.qdrant import PomaQdrant
# 1. Your existing Unstructured call — unchanged.
elements = partition_pdf(
filename="contract.pdf",
strategy="hi_res",
infer_table_structure=True, # populates metadata.text_as_html
extract_image_block_types=["Image"],
extract_image_block_to_payload=True, # inline base64 → POMA can describe figures
)
elements_to_json(elements, filename="contract.unstructured.json")
# 2. The missing link — raw element list in, chunks + chunksets out.
poma = PrimeCut() # reads POMA_API_KEY
result = poma.ingest("contract.unstructured.json") # Unstructured shape auto-detected
# 3. Qdrant — hybrid points with hierarchy payloads, one call.
qdrant = PomaQdrant(
url=os.environ["QDRANT_URL"],
api_key=os.environ["QDRANT_API_KEY"],
cloud_inference=False, # True needs Qdrant Cloud's inference service; False embeds locally via fastembed and works on OSS/local Qdrant too
collection_name="contracts",
dense_model="sentence-transformers/all-MiniLM-L6-v2",
sparse_model="Qdrant/bm25",
dense_size=384,
auto_create_collection=True,
)
qdrant.upsert_poma_points(result)
# 4. Retrieve prompt-ready context.
cheatsheets = qdrant.get_cheatsheets(
query="What are the early termination conditions?",
limit=3,
)
print(cheatsheets[0]["content"])POMA validates the payload shape up front (a corrupted or mislabeled upload 422s immediately), splices text_as_html for tables, describes inline images, drops Header/Footer/PageNumber elements as page furniture, restores under-marked headings, and rebuilds the cross-page heading tree before chunking. To force or suppress detection, set external_ocr_source to "unstructured" or "none".
On our reference legal-document benchmark, this pipeline answers the same query with 337 tokens of retrieved context instead of 1,542 for a recursive character splitter, with zero information loss — methodology in Document Ingestion & Chunking for RAG. Overlap-free chunksets also mean no near-duplicate vectors crowding Qdrant's top-k.
Metadata mapping: POMA fields → Qdrant primitives
One point per chunkset. upsert_poma_points writes exactly this payload: {"chunkset_index", "chunks", "file_id", "text"} plus "chunk_details" (the chunkset's chunk records — chunk_index, content, depth, file_id, code) while store_chunk_details stays at its default True.
| POMA field | Qdrant primitive | What it enables |
|---|---|---|
to_embed | dense vector + BM25 sparse vector (named vectors, one point) | paraphrase + exact-term hybrid retrieval |
text (a copy of to_embed) | payload field | read back what was embedded |
file_id | payload field (no index is created for you — add one with create_payload_index) | scope queries to one document |
chunkset_index | payload field | identifies the point: the id is a UUIDv5 of file_id + chunkset_index |
chunks (chunk indices) | payload field | joins the point back to its chunk records |
chunk lineage (chunk_index, content, depth, code) | payload (chunk_details) | cheatsheet assembly without a second store |
page (from metadata.page_number) | not written by PomaQdrant — the SDK's PomaChunk drops it; it survives on the archive's chunk records | page-cited answers, page-range filters — see below |
get_cheatsheets reads only that payload back: it groups each hit's chunkset_index + chunks by file_id and assembles the text from the hit's chunk_details (or from a chunk_data you pass instead).
Page numbers are in the .poma archive, so keep it (poma.ingest(..., download_dir="store", filename="contract.poma")) and attach them to the same points:
from poma.utils import unpack_poma_archive
from poma.integrations.qdrant.qdrant_poma_utils import chunk_uuid_string
page_of = {c["chunk_index"]: c.get("page") for c in unpack_poma_archive(poma_archive_path="store/contract.poma")["chunks"]}
for cs in result.chunksets:
leaf = cs.chunks[-1] # chunk indices are in document order; the last one is the leaf
qdrant.set_payload(collection_name="contracts", payload={"page": page_of[leaf]}, points=[chunk_uuid_string(cs.file_id, cs.chunkset_index)])Frequently asked questions
How do I get Unstructured.io elements into Qdrant for RAG?
Save the element list with elements_to_json, run PrimeCut().ingest() on it (auto-detected, hierarchy rebuilt), then PomaQdrant.upsert_poma_points(result) — hybrid points whose payload carries file_id, chunkset_index, the chunkset's chunk indices, the embedded text, and chunk_details (chunk_index, content, depth) for assembly. Retrieve with get_cheatsheets(query=...).
Can Qdrant hybrid search make up for chunk_by_title's flat fragments?
No. Hybrid search improves ranking, not content. A chunk_by_title fragment carries page_number and its nearest Title but no ancestor path — the flat element list never had one — and Qdrant can't recover lineage that was never stored. POMA chunksets carry lineage inside the retrieval unit itself.
What Qdrant payload fields should Unstructured-derived chunks carry?
PomaQdrant writes file_id, chunkset_index, chunks, text and chunk_details (the chunk records with chunk_index, content and depth) by default. It does not write page, and it creates no payload indexes: add a payload index on file_id yourself for per-document scoping, and if you need page-cited answers or page-range filters, read page (from metadata.page_number) off the archive's chunk records with unpack_poma_archive and attach it with client.set_payload on the same point ids.
Do Unstructured's tables and images survive the trip to Qdrant?
Tables yes — text_as_html is spliced, never cut mid-row. Images yes, if partitioned with extract_image_block_to_payload=true: POMA describes each figure and the description becomes a searchable point. image_path images on disk are neutralized and counted in content_metadata — visible loss, never silent.
Should I store Unstructured's element metadata in Qdrant payloads?
Only the fields you filter on. PomaQdrant already writes file_id, chunkset_index, chunks, text and the chunkset lineage in chunk_details; add a payload index on file_id yourself, and attach page from the archive with client.set_payload only if you actually filter on it. Coordinates, emphasized-text spans, and other per-element metadata are ingestion-time detail — leave them in the .poma archive.
Related recipes
Same parser, different store: Unstructured → Pinecone · Unstructured → Weaviate · Unstructured → pgvector
Same store, different parser: Mistral OCR → Qdrant · LlamaParse → Qdrant · Docling → Qdrant
Foundations: The Optimal Chunker for Unstructured.io · The Optimal Chunks for Qdrant · All pipeline recipes · RAG chunking guide