Source: http://www.poma-ai.com/docs/pipelines/unstructured-to-qdrant

# The Missing Link Between Unstructured.io and Optimal Retrieval in Qdrant

<ByAuthor />

**The short answer:** Unstructured.io gives you excellent typed elements; Qdrant gives you excellent hybrid vector search. Wired together naively — `chunk_by_title`, embed, upsert — they still produce mediocre RAG, because a flat element list never contained the ancestor hierarchy that retrieval needs. The missing link is POMA: `PrimeCut().ingest()` consumes the raw `elements_to_json` output (auto-detected), emits hierarchy-preserving [chunksets](/learn/chunking/chunksets), and `PomaQdrant.upsert_poma_points()` writes them as hybrid dense+sparse points with indexed hierarchy payloads. Retrieval comes back as assembled, prompt-ready cheatsheets.

## What each end of the pipeline actually provides

**Unstructured.io** (`partition_pdf` or the hosted API) returns a flat, ordered list of typed elements — `Title`, `NarrativeText`, `ListItem`, `Table`, `Image` — each with a stable `element_id` and metadata such as `page_number` and `text_as_html` for tables. What it doesn't provide: nesting. Two `Title` elements sit side by side in the stream whether one is a chapter and the other a sub-subsection, and the built-in `by_title` and `basic` chunkers emit flat fragments bounded by character limits. Details: [The Optimal Chunker for Unstructured.io](/optimal-chunker-unstructured).

**Qdrant** provides named dense + sparse vectors on one point, payload indexes for filtered ANN, and HNSW with quantization options. What it doesn't provide: any opinion about what a point should contain. It retrieves nearest neighbors of whatever you embedded — including context-free fragments, if that's what you gave it. Details: [The Optimal Chunks for the Best Retrieval in Qdrant](/optimal-chunks-qdrant).

## The naive wiring, and where it breaks

The common recipe — `chunk_by_title(elements)` → embed each fragment → `client.upsert(...)` — breaks in a pair-specific way:

- **`chunk_by_title` fragments carry element metadata but no ancestor path.** Each fragment knows its `page_number`, its `filename`, and at best its nearest `Title` — because that's all a flat element list contains. Nothing records that the fragment's section lives inside a chapter inside a part.
- **Qdrant's hybrid search can't recover lineage that was never stored.** BM25 sparse vectors and dense vectors improve *ranking*; the winning point still arrives at the LLM as a bare paragraph. No query-time trick reconstructs a heading tree out of payloads that never held one.
- **Long sections get re-split by `max_characters`**, so the tail fragments lose even the nearest-title link — the one piece of context `by_title` promised.

The result: Qdrant performs exactly as designed, faithfully retrieving fragments that answer out of context. The fix belongs between the two tools, not in either of them.

## The pipeline, end to end

```bash
pip install unstructured 'poma[qdrant]'
```

```python
import os
from unstructured.partition.pdf import partition_pdf
from unstructured.staging.base import elements_to_json
from poma import PrimeCut
from poma.integrations.qdrant import PomaQdrant

# 1. Your existing Unstructured call — unchanged.
elements = partition_pdf(
    filename="contract.pdf",
    strategy="hi_res",
    infer_table_structure=True,            # populates metadata.text_as_html
    extract_image_block_types=["Image"],
    extract_image_block_to_payload=True,   # inline base64 → POMA can describe figures
)
elements_to_json(elements, filename="contract.unstructured.json")

# 2. The missing link — raw element list in, chunks + chunksets out.
poma = PrimeCut()  # reads POMA_API_KEY
result = poma.ingest("contract.unstructured.json")  # Unstructured shape auto-detected

# 3. Qdrant — hybrid points with hierarchy payloads, one call.
qdrant = PomaQdrant(
    url=os.environ["QDRANT_URL"],
    api_key=os.environ["QDRANT_API_KEY"],
    cloud_inference=True,
    collection_name="contracts",
    dense_model="sentence-transformers/all-MiniLM-L6-v2",
    sparse_model="Qdrant/bm25",
    dense_size=384,
    auto_create_collection=True,
)
qdrant.upsert_poma_points(result)

# 4. Retrieve prompt-ready context.
cheatsheets = qdrant.get_cheatsheets(
    query="What are the early termination conditions?",
    limit=3,
)
print(cheatsheets[0]["content"])
```

POMA validates the payload shape up front (a corrupted or mislabeled upload 422s immediately), splices `text_as_html` for tables, describes inline images, drops `Header`/`Footer`/`PageNumber` elements as page furniture, restores under-marked headings, and rebuilds the cross-page heading tree before chunking. To force or suppress detection, set `external_ocr_source` to `"unstructured"` or `"none"`.

On our reference legal-document benchmark, this pipeline answers the same query with **337 tokens** of retrieved context instead of **1,542** for a recursive character splitter, with zero information loss — methodology in [Document Ingestion & Chunking for RAG](/document-ingestion-chunking-rag). Overlap-free chunksets also mean no near-duplicate vectors crowding Qdrant's top-k.

## Metadata mapping: POMA fields → Qdrant primitives

| POMA chunk field | Qdrant primitive | What it enables |
| --- | --- | --- |
| `to_embed` | dense vector + BM25 sparse vector (named vectors, one point) | paraphrase + exact-term hybrid retrieval |
| `file_id` | payload field, **payload-indexed** | scope queries to one document |
| `page` (from `metadata.page_number`) | payload field, **payload-indexed** | page-cited answers, page-range filters |
| `depth` | payload field | filter/re-rank by hierarchy level |
| `chunk_index` | payload field | stable ordering at assembly time |
| chunkset lineage | payload (`chunk_details`) | cheatsheet assembly without a second store |

## Frequently asked questions

### How do I get Unstructured.io elements into Qdrant for RAG?

Save the element list with `elements_to_json`, run `PrimeCut().ingest()` on it (auto-detected, hierarchy rebuilt), then `PomaQdrant.upsert_poma_points(result)` — hybrid points with `file_id`/`page`/`depth` payloads. Retrieve with `get_cheatsheets(query=...)`.

### Can Qdrant hybrid search make up for chunk_by_title's flat fragments?

No. Hybrid search improves ranking, not content. A `chunk_by_title` fragment carries `page_number` and its nearest `Title` but no ancestor path — the flat element list never had one — and Qdrant can't recover lineage that was never stored. POMA chunksets carry lineage inside the retrieval unit itself.

### What Qdrant payload fields should Unstructured-derived chunks carry?

`file_id`, `page` (from `metadata.page_number`), `depth`, `chunk_index` plus content; payload-index `file_id` and `page`. `PomaQdrant` writes these by default.

### Do Unstructured's tables and images survive the trip to Qdrant?

Tables yes — `text_as_html` is spliced, never cut mid-row. Images yes, if partitioned with `extract_image_block_to_payload=true`: POMA describes each figure and the description becomes a searchable point. `image_path` images on disk are neutralized and counted in `content_metadata` — visible loss, never silent.

### Should I store Unstructured's element metadata in Qdrant payloads?

Only the fields you filter on: `file_id`, `page`, `depth`, `chunk_index`, plus the chunkset lineage for cheatsheet assembly. Coordinates, emphasized-text spans, and other per-element metadata are ingestion-time detail — leave them in the `.poma` archive.

## Related recipes

Same parser, different store: [Unstructured → Pinecone](/pipelines/unstructured-to-pinecone) · [Unstructured → Weaviate](/pipelines/unstructured-to-weaviate) · [Unstructured → pgvector](/pipelines/unstructured-to-pgvector)

Same store, different parser: [Mistral OCR → Qdrant](/pipelines/mistral-ocr-to-qdrant) · [LlamaParse → Qdrant](/pipelines/llamaparse-to-qdrant) · [Docling → Qdrant](/pipelines/docling-to-qdrant)

Foundations: [The Optimal Chunker for Unstructured.io](/optimal-chunker-unstructured) · [The Optimal Chunks for Qdrant](/optimal-chunks-qdrant) · [All pipeline recipes](/pipelines/) · [RAG chunking guide](/guides/rag-chunking/)