Mistral OCR to Chroma: Chunking for RAG
The short answer: Mistral OCR gives you excellent per-page markdown; Chroma gives you the fastest path from zero to working retrieval — an embedded, persistent vector store with metadata filtering and pluggable embedding functions. Wired together naively — flatten, split, embed — the prototype works just well enough to ship its worst decision everywhere. The missing link is POMA: PrimeCut().ingest() consumes the raw /v1/ocr JSON (auto-detected), emits hierarchy-preserving chunksets, and each one lands in Chroma with file_id, page (read from the archive's chunk records), depth, and chunkset_index metadata ready for where filters — with the .poma archive as a portable source of truth when the prototype graduates.
What each end of the pipeline actually provides
Mistral OCR (/v1/ocr, mistral-ocr-latest) returns pages[], each with an index and a markdown string — headings, lists, and tables recovered within each page, image bytes inline when you set include_image_base64. What it doesn't provide: cross-page hierarchy (page 41's ## Termination clauses has no link to page 3's # Master Services Agreement) or retrieval units. Details: The Optimal Chunker for Mistral OCR.
Chroma provides a local-first store that runs embedded or client/server, collections with default or bring-your-own embedding functions, where filters over metadata, and where_document filters over content. What it doesn't provide: any opinion about what a document entry should contain, or metadata it wasn't given. where={"file_id": ...} matches nothing if no entry carries a file_id. Details: The Optimal Chunks for the Best Retrieval in Chroma.
The naive wiring, and where it breaks
The common recipe — "\n".join(p["markdown"] for p in pages) → RecursiveCharacterTextSplitter → collection.add(documents=windows) — breaks in a pair-specific way:
- The prototype decision gets baked in. Chroma is where RAG pipelines are born: three lines to a working demo, so the chunking chosen in hour one is rarely revisited. When the prototype graduates to a production database, the flat splitter travels with it — the one component that was never production quality gets ported everywhere.
wherefilters have nothing to match. Flat text windows carry no metadata. Mistral'spages[].indexvanished at the join and heading levels were discarded, so there is nofile_id,page, ordepthto filter on — Chroma's filtering, its main lever beyond plain similarity, sits unused.- Retrieved fragments arrive without lineage, so the LLM answers out of context — a failure that looks like "Chroma isn't good enough" and prompts a database migration that fixes nothing, because the fragments port along.
POMA's chunksets fix this at the source: every chunkset is a self-explanatory root-to-leaf unit carrying its hierarchy metadata, with no overlap. On a notoriously hard reference legal document, chunksets plus cheatsheet assembly delivered the answer in 337 tokens of retrieved context versus 1,542 for a recursive character splitter, with zero information loss (methodology).
The pipeline, end to end
pip install mistralai poma chromadbimport os
import chromadb
from mistralai import Mistral
from poma import PrimeCut
from poma.utils import unpack_poma_archive
# 1. Mistral OCR — your existing call, unchanged.
mistral = Mistral(api_key=os.environ["MISTRAL_API_KEY"])
ocr = mistral.ocr.process(
model="mistral-ocr-latest",
document={"type": "document_url", "document_url": "https://example.com/contract.pdf"},
include_image_base64=True,
)
with open("contract.mistral-ocr.json", "w") as f:
f.write(ocr.model_dump_json())
# 2. The missing link — raw result JSON in, chunks + chunksets out.
poma = PrimeCut() # reads POMA_API_KEY
result = poma.ingest(
"contract.mistral-ocr.json", # Mistral shape auto-detected
download_dir="store",
filename="contract.poma", # keep the archive — it carries the page numbers
)
# Page numbers live on the archive's chunk records; the SDK's typed objects
# expose depth but not page. Look both up by chunk index.
archive = unpack_poma_archive(poma_archive_path="store/contract.poma")
page_of = {c["chunk_index"]: (c.get("page") if c.get("page") is not None else -1)
for c in archive["chunks"]} # Chroma metadata values cannot be None
depth_of = {c.chunk_index: c.depth for c in result.chunks}
# 3. Chroma — persistent local store; the collection's embedding
# function (default or your own) embeds the documents on add.
chroma = chromadb.PersistentClient(path="./chroma")
collection = chroma.get_or_create_collection(name="contracts")
# 4. Add chunksets — documents, stable IDs, where-filterable metadata.
# cs.chunks holds chunk indices in document order; the last one is the leaf.
collection.add(
ids=[f"{cs.file_id}:{cs.chunkset_index}" for cs in result.chunksets],
documents=[cs.to_embed for cs in result.chunksets],
metadatas=[
{
"file_id": cs.file_id,
"page": page_of[cs.chunks[-1]],
"depth": depth_of[cs.chunks[-1]],
"chunkset_index": cs.chunkset_index,
}
for cs in result.chunksets
],
)
# 5. Query with a metadata filter, then assemble the cheatsheet.
res = collection.query(
query_texts=["What are the early termination conditions?"],
n_results=5,
where={"file_id": "contract.mistral-ocr.json"},
)
pairs = sorted(
zip(res["documents"][0], res["metadatas"][0]),
key=lambda p: p[1]["chunkset_index"],
)
cheatsheet = "\n\n".join(dict.fromkeys(doc for doc, _ in pairs))POMA validates the payload shape up front (a corrupted or mislabeled upload 422s immediately), handles all three Mistral image cases (base64 → described; annotation → spliced; neither → visible marker, counted in content_metadata), strips running headers/footers, and rebuilds the cross-page heading tree before chunking. To force or suppress detection, set external_ocr_source to "mistral" or "none". The final step is the cheatsheet: retrieved chunksets are deduplicated (shared ancestor headings appear once) and merged into a single prompt-ready block.
Prototype today, port tomorrow
The .poma archive that PrimeCut produces is the portable source of truth for the document — chunks, chunksets, and hierarchy metadata, independent of any database. Loading it into Chroma is one loop; loading the same archive into Qdrant, Milvus, Weaviate, pgvector, or Pinecone is the same loop against a different client, because file_id/page/depth map onto every store's filtering primitive. The prototype and the production system disagree only about infrastructure, never about content — no re-OCR, no re-chunk, no drift between what the demo retrieved and what production retrieves.
Metadata mapping: POMA fields → Chroma primitives
| POMA chunk field | Chroma primitive | What it enables |
|---|---|---|
to_embed | documents entry, embedded by the collection's embedding function | semantic retrieval; where_document content filters |
file_id | metadatas key, where={"file_id": ...} | scope queries to one document |
page (on the archive's chunk records, derived from Mistral pages[].index; take the chunkset's leaf chunk) | metadatas key, where with range operators | page-cited answers, page-range filters |
depth (of the chunkset's leaf chunk) | metadatas key | filter/re-rank by hierarchy level |
chunkset_index | metadatas key + part of the entry id | stable IDs, ordering at cheatsheet assembly |
| chunkset lineage | .poma archive (portable source of truth) | identical reload into any other vector DB |
Frequently asked questions
How do I get Mistral OCR results into Chroma for RAG?
Save the raw /v1/ocr JSON, run PrimeCut().ingest() on it (auto-detected, hierarchy rebuilt), then collection.add() with each chunkset's to_embed as the document, a stable {file_id}:{chunkset_index} ID, and file_id/page/depth/chunkset_index metadata — page comes from the archive's chunk records (the chunkset's leaf chunk), depth from the matching typed chunk. Query with where filters and assemble cheatsheets client-side.
What metadata should Chroma store for Mistral OCR chunks so where filters work?
file_id (per-document scoping), page (recorded on every chunk in the .poma archive, derived from Mistral's pages[].index), depth, and chunkset_index. where filters only match metadata that exists — flat splitter windows carry none of it.
Can I keep Chroma's default embedding function with POMA chunksets?
Yes — pass to_embed as the document and Chroma embeds it with the collection's configured function, default or your own. You can swap embedding functions later without redoing the chunking.
Will a Chroma prototype built on Mistral OCR chunks port to a production vector database?
The chunking ports: the .poma archive reloads into Qdrant, Milvus, Weaviate, pgvector, or Pinecone without re-OCR or re-chunking. A naive prototype chunking also ports — fragments stay fragments — which is why it should be production quality from day one.
How do I restrict a Chroma query to one Mistral OCR document?
collection.query(query_texts=[...], where={"file_id": "contract.mistral-ocr.json"}). Combine with where_document for exact-term constraints or range conditions on page.
Related recipes
Same parser, different store: Mistral OCR → Qdrant · Mistral OCR → Milvus · Mistral OCR → pgvector
Same store, different parser: LlamaParse → Chroma · Docling → Chroma · Marker → Chroma
Foundations: The Optimal Chunker for Mistral OCR · The Optimal Chunks for Chroma · All pipeline recipes · RAG chunking guide