Skip to content

Mistral OCR to Chroma: Chunking for RAG ​

The short answer: Mistral OCR gives you excellent per-page markdown; Chroma gives you the fastest path from zero to working retrieval — an embedded, persistent vector store with metadata filtering and pluggable embedding functions. Wired together naively — flatten, split, embed — the prototype works just well enough to ship its worst decision everywhere. The missing link is POMA: PrimeCut().ingest() consumes the raw /v1/ocr JSON (auto-detected), emits hierarchy-preserving chunksets, and each one lands in Chroma with file_id, page (read from the archive's chunk records), depth, and chunkset_index metadata ready for where filters — with the .poma archive as a portable source of truth when the prototype graduates.

What each end of the pipeline actually provides ​

Mistral OCR (/v1/ocr, mistral-ocr-latest) returns pages[], each with an index and a markdown string — headings, lists, and tables recovered within each page, image bytes inline when you set include_image_base64. What it doesn't provide: cross-page hierarchy (page 41's ## Termination clauses has no link to page 3's # Master Services Agreement) or retrieval units. Details: The Optimal Chunker for Mistral OCR.

Chroma provides a local-first store that runs embedded or client/server, collections with default or bring-your-own embedding functions, where filters over metadata, and where_document filters over content. What it doesn't provide: any opinion about what a document entry should contain, or metadata it wasn't given. where={"file_id": ...} matches nothing if no entry carries a file_id. Details: The Optimal Chunks for the Best Retrieval in Chroma.

The naive wiring, and where it breaks ​

The common recipe — "\n".join(p["markdown"] for p in pages) → RecursiveCharacterTextSplitter → collection.add(documents=windows) — breaks in a pair-specific way:

  • The prototype decision gets baked in. Chroma is where RAG pipelines are born: three lines to a working demo, so the chunking chosen in hour one is rarely revisited. When the prototype graduates to a production database, the flat splitter travels with it — the one component that was never production quality gets ported everywhere.
  • where filters have nothing to match. Flat text windows carry no metadata. Mistral's pages[].index vanished at the join and heading levels were discarded, so there is no file_id, page, or depth to filter on — Chroma's filtering, its main lever beyond plain similarity, sits unused.
  • Retrieved fragments arrive without lineage, so the LLM answers out of context — a failure that looks like "Chroma isn't good enough" and prompts a database migration that fixes nothing, because the fragments port along.

POMA's chunksets fix this at the source: every chunkset is a self-explanatory root-to-leaf unit carrying its hierarchy metadata, with no overlap. On a notoriously hard reference legal document, chunksets plus cheatsheet assembly delivered the answer in 337 tokens of retrieved context versus 1,542 for a recursive character splitter, with zero information loss (methodology).

The pipeline, end to end ​

bash
pip install mistralai poma chromadb
python
import os
import chromadb
from mistralai import Mistral
from poma import PrimeCut
from poma.utils import unpack_poma_archive

# 1. Mistral OCR — your existing call, unchanged.
mistral = Mistral(api_key=os.environ["MISTRAL_API_KEY"])
ocr = mistral.ocr.process(
    model="mistral-ocr-latest",
    document={"type": "document_url", "document_url": "https://example.com/contract.pdf"},
    include_image_base64=True,
)
with open("contract.mistral-ocr.json", "w") as f:
    f.write(ocr.model_dump_json())

# 2. The missing link — raw result JSON in, chunks + chunksets out.
poma = PrimeCut()  # reads POMA_API_KEY
result = poma.ingest(
    "contract.mistral-ocr.json",  # Mistral shape auto-detected
    download_dir="store",
    filename="contract.poma",     # keep the archive — it carries the page numbers
)

# Page numbers live on the archive's chunk records; the SDK's typed objects
# expose depth but not page. Look both up by chunk index.
archive = unpack_poma_archive(poma_archive_path="store/contract.poma")
page_of = {c["chunk_index"]: (c.get("page") if c.get("page") is not None else -1)
           for c in archive["chunks"]}  # Chroma metadata values cannot be None
depth_of = {c.chunk_index: c.depth for c in result.chunks}

# 3. Chroma — persistent local store; the collection's embedding
#    function (default or your own) embeds the documents on add.
chroma = chromadb.PersistentClient(path="./chroma")
collection = chroma.get_or_create_collection(name="contracts")

# 4. Add chunksets — documents, stable IDs, where-filterable metadata.
#    cs.chunks holds chunk indices in document order; the last one is the leaf.
collection.add(
    ids=[f"{cs.file_id}:{cs.chunkset_index}" for cs in result.chunksets],
    documents=[cs.to_embed for cs in result.chunksets],
    metadatas=[
        {
            "file_id": cs.file_id,
            "page": page_of[cs.chunks[-1]],
            "depth": depth_of[cs.chunks[-1]],
            "chunkset_index": cs.chunkset_index,
        }
        for cs in result.chunksets
    ],
)

# 5. Query with a metadata filter, then assemble the cheatsheet.
res = collection.query(
    query_texts=["What are the early termination conditions?"],
    n_results=5,
    where={"file_id": "contract.mistral-ocr.json"},
)
pairs = sorted(
    zip(res["documents"][0], res["metadatas"][0]),
    key=lambda p: p[1]["chunkset_index"],
)
cheatsheet = "\n\n".join(dict.fromkeys(doc for doc, _ in pairs))

POMA validates the payload shape up front (a corrupted or mislabeled upload 422s immediately), handles all three Mistral image cases (base64 → described; annotation → spliced; neither → visible marker, counted in content_metadata), strips running headers/footers, and rebuilds the cross-page heading tree before chunking. To force or suppress detection, set external_ocr_source to "mistral" or "none". The final step is the cheatsheet: retrieved chunksets are deduplicated (shared ancestor headings appear once) and merged into a single prompt-ready block.

Prototype today, port tomorrow ​

The .poma archive that PrimeCut produces is the portable source of truth for the document — chunks, chunksets, and hierarchy metadata, independent of any database. Loading it into Chroma is one loop; loading the same archive into Qdrant, Milvus, Weaviate, pgvector, or Pinecone is the same loop against a different client, because file_id/page/depth map onto every store's filtering primitive. The prototype and the production system disagree only about infrastructure, never about content — no re-OCR, no re-chunk, no drift between what the demo retrieved and what production retrieves.

Metadata mapping: POMA fields → Chroma primitives ​

POMA chunk fieldChroma primitiveWhat it enables
to_embeddocuments entry, embedded by the collection's embedding functionsemantic retrieval; where_document content filters
file_idmetadatas key, where={"file_id": ...}scope queries to one document
page (on the archive's chunk records, derived from Mistral pages[].index; take the chunkset's leaf chunk)metadatas key, where with range operatorspage-cited answers, page-range filters
depth (of the chunkset's leaf chunk)metadatas keyfilter/re-rank by hierarchy level
chunkset_indexmetadatas key + part of the entry idstable IDs, ordering at cheatsheet assembly
chunkset lineage.poma archive (portable source of truth)identical reload into any other vector DB

Frequently asked questions ​

How do I get Mistral OCR results into Chroma for RAG? ​

Save the raw /v1/ocr JSON, run PrimeCut().ingest() on it (auto-detected, hierarchy rebuilt), then collection.add() with each chunkset's to_embed as the document, a stable {file_id}:{chunkset_index} ID, and file_id/page/depth/chunkset_index metadata — page comes from the archive's chunk records (the chunkset's leaf chunk), depth from the matching typed chunk. Query with where filters and assemble cheatsheets client-side.

What metadata should Chroma store for Mistral OCR chunks so where filters work? ​

file_id (per-document scoping), page (recorded on every chunk in the .poma archive, derived from Mistral's pages[].index), depth, and chunkset_index. where filters only match metadata that exists — flat splitter windows carry none of it.

Can I keep Chroma's default embedding function with POMA chunksets? ​

Yes — pass to_embed as the document and Chroma embeds it with the collection's configured function, default or your own. You can swap embedding functions later without redoing the chunking.

Will a Chroma prototype built on Mistral OCR chunks port to a production vector database? ​

The chunking ports: the .poma archive reloads into Qdrant, Milvus, Weaviate, pgvector, or Pinecone without re-OCR or re-chunking. A naive prototype chunking also ports — fragments stay fragments — which is why it should be production quality from day one.

How do I restrict a Chroma query to one Mistral OCR document? ​

collection.query(query_texts=[...], where={"file_id": "contract.mistral-ocr.json"}). Combine with where_document for exact-term constraints or range conditions on page.

Same parser, different store: Mistral OCR → Qdrant · Mistral OCR → Milvus · Mistral OCR → pgvector

Same store, different parser: LlamaParse → Chroma · Docling → Chroma · Marker → Chroma

Foundations: The Optimal Chunker for Mistral OCR · The Optimal Chunks for Chroma · All pipeline recipes · RAG chunking guide