Source: http://www.poma-ai.com/docs/pipelines/mistral-ocr-to-qdrant

# The Missing Link Between Mistral OCR and Optimal Retrieval in Qdrant

<ByAuthor />

**The short answer:** Mistral OCR gives you excellent per-page markdown; Qdrant gives you excellent hybrid vector search. Wired together naively — flatten, split, embed — they still produce mediocre RAG, because nothing in between rebuilds the document's hierarchy. The missing link is POMA: `PrimeCut().ingest()` consumes the raw `/v1/ocr` JSON (auto-detected), emits hierarchy-preserving [chunksets](/learn/chunking/chunksets), and `PomaQdrant.upsert_poma_points()` writes them as hybrid dense+sparse points with indexed hierarchy payloads. Retrieval comes back as assembled, prompt-ready cheatsheets.

## What each end of the pipeline actually provides

**Mistral OCR** (`/v1/ocr`, `mistral-ocr-latest`) returns `pages[]`, each with an `index` and a `markdown` string — headings, lists, and tables recovered *within* each page, image bytes inline when you set `include_image_base64`. What it doesn't provide: cross-page hierarchy (page 41's `## Termination clauses` has no link to page 3's `# Master Services Agreement`) or retrieval units. Details: [The Optimal Chunker for Mistral OCR](/optimal-chunker-mistral-ocr).

**Qdrant** provides named dense + sparse vectors on one point, payload indexes for filtered ANN, and HNSW with quantization options. What it doesn't provide: any opinion about what a point should contain. It retrieves nearest neighbors of whatever you embedded — including context-free fragments, if that's what you gave it. Details: [The Optimal Chunks for the Best Retrieval in Qdrant](/optimal-chunks-qdrant).

## The naive wiring, and where it breaks

The common recipe — `"\n".join(p["markdown"] for p in pages)` → `RecursiveCharacterTextSplitter` → embed → `client.upsert(...)` — breaks in a pair-specific way:

- **Mistral's page indices vanish at the join**, so no Qdrant payload can cite a page, and per-document filters are all you have left.
- **Overlap inflates the collection.** Splitter overlap (typically 10–20%) embeds every boundary span twice. In Qdrant that means a measurably larger HNSW graph and — worse — top-k results where hits 2 and 3 are near-duplicates of hit 1, crowding out the passage that actually answers the question.
- **Mistral's heading levels are discarded**, so retrieved fragments arrive without lineage and the LLM answers out of context — the failure OCR quality can't fix.

## The pipeline, end to end

```bash
pip install mistralai 'poma[qdrant]'
```

```python
import os
from mistralai import Mistral
from poma import PrimeCut
from poma.integrations.qdrant import PomaQdrant

# 1. Mistral OCR — your existing call, unchanged.
mistral = Mistral(api_key=os.environ["MISTRAL_API_KEY"])
ocr = mistral.ocr.process(
    model="mistral-ocr-latest",
    document={"type": "document_url", "document_url": "https://example.com/contract.pdf"},
    include_image_base64=True,
)
with open("contract.mistral-ocr.json", "w") as f:
    f.write(ocr.model_dump_json())

# 2. The missing link — raw result JSON in, chunks + chunksets out.
poma = PrimeCut()  # reads POMA_API_KEY
result = poma.ingest("contract.mistral-ocr.json")  # Mistral shape auto-detected

# 3. Qdrant — hybrid points with hierarchy payloads, one call.
qdrant = PomaQdrant(
    url=os.environ["QDRANT_URL"],
    api_key=os.environ["QDRANT_API_KEY"],
    cloud_inference=True,
    collection_name="contracts",
    dense_model="sentence-transformers/all-MiniLM-L6-v2",
    sparse_model="Qdrant/bm25",
    dense_size=384,
    auto_create_collection=True,
)
qdrant.upsert_poma_points(result)

# 4. Retrieve prompt-ready context.
cheatsheets = qdrant.get_cheatsheets(
    query="What are the early termination conditions?",
    limit=3,
)
print(cheatsheets[0]["content"])
```

POMA validates the payload shape up front (a corrupted or mislabeled upload 422s immediately), handles all three Mistral image cases (base64 → described; annotation → spliced; neither → visible marker, counted in `content_metadata`), strips running headers/footers, and rebuilds the cross-page heading tree before chunking. To force or suppress detection, set `external_ocr_source` to `"mistral"` or `"none"`.

## Metadata mapping: POMA fields → Qdrant primitives

| POMA chunk field | Qdrant primitive | What it enables |
| --- | --- | --- |
| `to_embed` | dense vector + BM25 sparse vector (named vectors, one point) | paraphrase + exact-term hybrid retrieval |
| `file_id` | payload field, **payload-indexed** | scope queries to one document |
| `page` (from Mistral `pages[].index`) | payload field, **payload-indexed** | page-cited answers, page-range filters |
| `depth` | payload field | filter/re-rank by hierarchy level |
| `chunk_index` | payload field | stable ordering at assembly time |
| chunkset lineage | payload (`chunk_details`) | cheatsheet assembly without a second store |

## Frequently asked questions

### How do I get Mistral OCR results into Qdrant for RAG?

Save the raw `/v1/ocr` JSON, run `PrimeCut().ingest()` on it (auto-detected, hierarchy rebuilt), then `PomaQdrant.upsert_poma_points(result)` — hybrid points with `file_id`/`page`/`depth` payloads. Retrieve with `get_cheatsheets(query=...)`.

### Why not just split Mistral's markdown and embed it into Qdrant directly?

Splitting discards Mistral's page indices and heading levels, and overlap fills Qdrant with near-duplicate vectors. Qdrant then retrieves context-free fragments — the pipeline underuses both tools.

### What Qdrant payload fields should Mistral OCR chunks carry?

`file_id`, `page`, `depth`, `chunk_index` plus content; payload-index `file_id` and `page`. `PomaQdrant` writes these by default.

### Do images in the Mistral OCR result survive the trip to Qdrant?

Yes — call `/v1/ocr` with `include_image_base64`, and POMA describes each figure so it becomes a searchable point. Images without bytes or annotation become visible, counted markers — never silent loss.

### Should the Qdrant collection use hybrid search for Mistral OCR content?

Yes: OCR'd documents are full of exact tokens (clause numbers, IDs, defined terms) that dense embeddings blur. `PomaQdrant` writes BM25 sparse vectors by default and fuses both at query time.

## Related recipes

Same parser, different store: [Mistral OCR → Pinecone](/pipelines/mistral-ocr-to-pinecone) · [Mistral OCR → pgvector](/pipelines/mistral-ocr-to-pgvector) · [Mistral OCR → Weaviate](/pipelines/mistral-ocr-to-weaviate)

Same store, different parser: [LlamaParse → Qdrant](/pipelines/llamaparse-to-qdrant) · [Textract → Qdrant](/pipelines/textract-to-qdrant) · [Docling → Qdrant](/pipelines/docling-to-qdrant)

Foundations: [The Optimal Chunker for Mistral OCR](/optimal-chunker-mistral-ocr) · [The Optimal Chunks for Qdrant](/optimal-chunks-qdrant) · [All pipeline recipes](/pipelines/) · [RAG architecture guide](/guides/rag-architecture/)