Skip to content

Mistral OCR to Milvus: Chunking for RAG ​

The short answer: Mistral OCR gives you excellent per-page markdown; Milvus gives you filtered vector search at serious scale — partitions, scalar filter expressions, and your pick of HNSW, IVF, or DiskANN. Wired together naively — flatten, split, embed, insert — the pipeline underuses both, because nothing in between rebuilds the document hierarchy or populates the scalar fields Milvus filters on. The missing link is POMA: PrimeCut().ingest() consumes the raw /v1/ocr JSON (auto-detected), emits hierarchy-preserving chunksets, and each chunkset lands in Milvus with file_id, page (read from the archive's chunk records), depth, and chunkset_index as scalar fields ready for filter expressions and partition-key routing.

What each end of the pipeline actually provides ​

Mistral OCR (/v1/ocr, mistral-ocr-latest) returns pages[], each with an index and a markdown string — headings, lists, and tables recovered within each page, image bytes inline when you set include_image_base64. What it doesn't provide: cross-page hierarchy (page 41's ## Termination clauses has no link to page 3's # Master Services Agreement) or retrieval units. Details: The Optimal Chunker for Mistral OCR.

Milvus provides collections with typed schemas, partitions with automatic partition-key routing for multi-tenancy, scalar fields queried through filter expressions (file_id == "...", page <= 10), index types from in-memory HNSW through IVF to disk-resident DiskANN, and — in recent versions — sparse vectors with built-in BM25 for hybrid search. What it doesn't provide: any opinion about what a row should contain. Empty scalar fields filter nothing; fragment vectors retrieve fragments. Details: The Optimal Chunks for the Best Retrieval in Milvus.

The naive wiring, and where it breaks ​

The common recipe — "\n".join(p["markdown"] for p in pages) → RecursiveCharacterTextSplitter → embed → client.insert(...) — breaks in a pair-specific way:

  • The scalar fields end up empty. A flat splitter emits anonymous text windows: no page (Mistral's pages[].index vanished at the join), no depth (heading levels discarded), often not even a reliable document ID. You define file_id, page, and depth in the schema, but there is nothing to put in them — so Milvus's headline feature, filtered search, has nothing to filter on, and the partition key can't route multi-tenant traffic.
  • Overlap bloats the index. Splitter overlap (typically 10–20%) embeds every boundary span twice. That inflates vector count across the board, and it hits DiskANN hardest: a disk-resident index pays for near-duplicate vectors in build time, disk footprint, and top-k lists where hits 2 and 3 restate hit 1.
  • Retrieved fragments arrive without lineage, so the LLM answers out of context — the failure neither Mistral's OCR quality nor Milvus's recall can fix.

POMA's chunksets fix all three at the source: every chunkset is a self-explanatory root-to-leaf unit carrying its hierarchy metadata, with no overlap. On a notoriously hard reference legal document, chunksets plus cheatsheet assembly delivered the answer in 337 tokens of retrieved context versus 1,542 for a recursive character splitter, with zero information loss (methodology).

The pipeline, end to end ​

bash
pip install mistralai poma pymilvus sentence-transformers
python
import os
from mistralai import Mistral
from poma import PrimeCut
from poma.utils import unpack_poma_archive
from pymilvus import DataType, MilvusClient
from sentence_transformers import SentenceTransformer

# 1. Mistral OCR — your existing call, unchanged.
mistral = Mistral(api_key=os.environ["MISTRAL_API_KEY"])
ocr = mistral.ocr.process(
    model="mistral-ocr-latest",
    document={"type": "document_url", "document_url": "https://example.com/contract.pdf"},
    include_image_base64=True,
)
with open("contract.mistral-ocr.json", "w") as f:
    f.write(ocr.model_dump_json())

# 2. The missing link — raw result JSON in, chunks + chunksets out.
poma = PrimeCut()  # reads POMA_API_KEY
result = poma.ingest(
    "contract.mistral-ocr.json",  # Mistral shape auto-detected
    download_dir="store",
    filename="contract.poma",     # keep the archive — it carries the page numbers
)

# Page numbers live on the archive's chunk records; the SDK's typed objects
# expose depth but not page. Look both up by chunk index.
archive = unpack_poma_archive(poma_archive_path="store/contract.poma")
page_of = {c["chunk_index"]: (c.get("page") if c.get("page") is not None else -1)
           for c in archive["chunks"]}  # INT64 fields cannot take NULL
depth_of = {c.chunk_index: c.depth for c in result.chunks}

# 3. Milvus — a schema whose scalar fields mirror the hierarchy metadata.
client = MilvusClient(uri=os.environ["MILVUS_URI"])
schema = MilvusClient.create_schema()
schema.add_field("pk", DataType.INT64, is_primary=True, auto_id=True)
schema.add_field("vector", DataType.FLOAT_VECTOR, dim=384)
schema.add_field("text", DataType.VARCHAR, max_length=65535)
schema.add_field("file_id", DataType.VARCHAR, max_length=256, is_partition_key=True)
schema.add_field("page", DataType.INT64)
schema.add_field("depth", DataType.INT64)
schema.add_field("chunkset_index", DataType.INT64)

index_params = client.prepare_index_params()
index_params.add_index(field_name="vector", index_type="HNSW", metric_type="COSINE")
client.create_collection("contracts", schema=schema, index_params=index_params)

# 4. Embed chunksets and insert — hierarchy metadata rides along as scalars.
model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
client.insert(
    collection_name="contracts",
    data=[
        {
            "vector": model.encode(cs.to_embed).tolist(),
            "text": cs.to_embed,
            "file_id": cs.file_id,
            # cs.chunks holds chunk indices in document order; the last is the leaf.
            "page": page_of[cs.chunks[-1]],
            "depth": depth_of[cs.chunks[-1]],
            "chunkset_index": cs.chunkset_index,
        }
        for cs in result.chunksets
    ],
)

# 5. Filtered search — scope to one document, then assemble the cheatsheet.
query = "What are the early termination conditions?"
hits = client.search(
    collection_name="contracts",
    data=[model.encode(query).tolist()],
    limit=5,
    filter='file_id == "contract.mistral-ocr.json"',
    output_fields=["text", "page", "depth", "chunkset_index"],
)
ordered = sorted(hits[0], key=lambda h: h["entity"]["chunkset_index"])
cheatsheet = "\n\n".join(dict.fromkeys(h["entity"]["text"] for h in ordered))

POMA validates the payload shape up front (a corrupted or mislabeled upload 422s immediately), handles all three Mistral image cases (base64 → described; annotation → spliced; neither → visible marker, counted in content_metadata), strips running headers/footers, and rebuilds the cross-page heading tree before chunking. To force or suppress detection, set external_ocr_source to "mistral" or "none". The final step is the cheatsheet: retrieved chunksets are deduplicated (shared ancestor headings appear once) and merged into a single prompt-ready block — Milvus returns the rows, your client assembles them.

Metadata mapping: POMA fields → Milvus primitives ​

POMA chunk fieldMilvus primitiveWhat it enables
to_embedFLOAT_VECTOR field (add a sparse/BM25 field for hybrid)paraphrase + exact-term retrieval
file_idVARCHAR scalar field, partition keyper-document filters, automatic multi-tenant routing
page (on the archive's chunk records, derived from Mistral pages[].index; take the chunkset's leaf chunk)INT64 scalar fieldpage-cited answers, page <= N filter expressions
depth (of the chunkset's leaf chunk)INT64 scalar fieldfilter/re-rank by hierarchy level
chunkset_indexINT64 scalar fieldstable ordering at cheatsheet assembly
chunkset textVARCHAR field in output_fieldsassembly without a second store

Frequently asked questions ​

How do I get Mistral OCR results into Milvus for RAG? ​

Save the raw /v1/ocr JSON, run PrimeCut().ingest() on it (auto-detected, hierarchy rebuilt), embed each chunkset's to_embed, and insert rows whose scalar fields carry file_id, page, depth, and chunkset_index — page comes from the archive's chunk records (the chunkset's leaf chunk), depth from the matching typed chunk. Retrieve with a filtered search and assemble cheatsheets client-side.

What scalar fields should a Milvus collection have for Mistral OCR chunks? ​

file_id (VARCHAR, the natural partition-key candidate), page (INT64, recorded on every chunk in the .poma archive and derived from Mistral's pages[].index), depth (INT64), and chunkset_index (INT64) alongside the vector and text. They power filter expressions and ordering at assembly time.

Should I use a Milvus partition key for multi-tenant Mistral OCR content? ​

Yes — declare file_id (or a tenant ID) as the partition key so filtered searches only touch matching partitions. POMA populates file_id on every chunkset, so the key always has a value to route on.

Which Milvus index type fits Mistral OCR chunksets: HNSW, IVF, or DiskANN? ​

All three work; choose by corpus size and latency. Chunking controls vector count: overlap-free chunksets embed each span once, which keeps any index smaller and matters most for DiskANN's disk footprint.

Does Milvus hybrid sparse plus BM25 search help with Mistral OCR documents? ​

Yes — OCR'd documents carry exact tokens (clause numbers, IDs, part codes) that dense embeddings blur. Recent Milvus versions store sparse BM25 vectors next to dense ones and fuse both at query time.

Same parser, different store: Mistral OCR → Qdrant · Mistral OCR → Chroma · Mistral OCR → Weaviate

Same store, different parser: Unstructured.io → Milvus · Marker → Milvus · Azure Document Intelligence → Milvus

Foundations: The Optimal Chunker for Mistral OCR · The Optimal Chunks for Milvus · All pipeline recipes · RAG chunking guide