Skip to content

LlamaParse to Chroma: Chunking for RAG ​

The short answer: LlamaParse gives you excellent per-page markdown; Chroma gives you the fastest path from zero to a working retrieval prototype — embedded, local-first, batteries included. Wired together naively — one page in, one document out — they produce a prototype that lies to you, because page-sized embeddings blur sections and the chunking decision you make now is the one production inherits. The missing link is POMA: PrimeCut().ingest() consumes the raw LlamaParse result JSON (auto-detected), emits hierarchy-preserving chunksets, and those land in Chroma as small, self-explanatory documents with where-filterable hierarchy metadata — while the .poma archive keeps the whole thing portable to any production store.

What each end of the pipeline actually provides ​

LlamaParse (LlamaIndex's parser) returns pages[], each with a page number, an md string (markdown — headings marked, tables inline), a plain text flattening, and an images list whose bytes stay server-side. What it doesn't provide: cross-page hierarchy (page 41's ## Termination clauses has no link to page 3's # Master Services Agreement) or retrieval units. Details: The Optimal Chunker for LlamaParse.

Chroma provides a PersistentClient that runs embedded or client/server with the same API, collections with default or bring-your-own embedding functions, where filters over scalar metadata, and where_document filters over content. What it doesn't provide: any opinion about what a document should contain — and no escape hatch from a bad chunking decision once your evaluations are built on it. Details: The Optimal Chunks for the Best Retrieval in Chroma.

The naive wiring, and where it breaks ​

LlamaParse's output shape makes the naive wiring seductively easy — pages[].md maps one-to-one onto collection.add, one document per page. That is exactly the trap:

  • Page-sized embeddings blur. A page mixing the end of one section and the start of the next becomes a single vector that represents neither well. n_results=5 returns five walls of text, and sections spanning page breaks are severed mid-thought.
  • Dead image references pollute both search lanes. The saved JSON carries only ![](name) refs (bytes live server-side at LlamaParse). Added naively, they sit inside your documents — embedded as noise, and matchable by where_document={"$contains": ...}, which can hit img_p3_2.png instead of content.
  • No metadata, no filters. Whole-page adds rarely carry file_id or depth, so where filters — Chroma's main retrieval control — have nothing to grip.
  • The prototype's chunking sticks. This is the pair's defining failure: Chroma makes starting so easy that the page-per-document decision hardens into your evaluation baseline. When production lands on another store, you re-chunk and re-evaluate everything — unless the chunks were portable from day one.

The pipeline, end to end ​

bash
pip install llama-parse poma chromadb
python
import json
import os

import chromadb
from llama_parse import LlamaParse

from poma import PrimeCut
from poma.utils import unpack_poma_archive

# 1. LlamaParse — your existing call, unchanged.
parser = LlamaParse(api_key=os.environ["LLAMA_CLOUD_API_KEY"])
json_result = parser.get_json_result("contract.pdf")
with open("contract.llamaparse.json", "w") as f:
    json.dump(json_result[0], f)

# 2. The missing link — raw result JSON in, chunks + chunksets out.
poma = PrimeCut()  # reads POMA_API_KEY
result = poma.ingest(
    "contract.llamaparse.json",  # LlamaParse shape auto-detected
    download_dir="store",
    filename="contract.poma",    # keep the archive — it carries the page numbers
)

# Page numbers live on the archive's chunk records; the SDK's typed objects
# expose depth but not page. Look both up by chunk index.
archive = unpack_poma_archive(poma_archive_path="store/contract.poma")
page_of = {c["chunk_index"]: (c.get("page") if c.get("page") is not None else -1)
           for c in archive["chunks"]}  # Chroma metadata values cannot be None
depth_of = {c.chunk_index: c.depth for c in result.chunks}

# 3. Chroma — durable local collection, default embedding function (or BYO).
chroma = chromadb.PersistentClient(path="./chroma")
collection = chroma.get_or_create_collection("contracts")

# cs.chunks holds chunk indices in document order; the last one is the leaf.
collection.add(
    documents=[cs.to_embed for cs in result.chunksets],
    metadatas=[
        {
            "file_id": cs.file_id,
            "page": page_of[cs.chunks[-1]],
            "depth": depth_of[cs.chunks[-1]],
            "chunkset_index": cs.chunkset_index,
        }
        for cs in result.chunksets
    ],
    ids=[f"{cs.file_id}:{cs.chunkset_index}" for cs in result.chunksets],
)

# 4. Query — metadata and content filters compose.
hits = collection.query(
    query_texts=["What are the early termination conditions?"],
    n_results=5,
    where={"file_id": result.chunksets[0].file_id},
    where_document={"$contains": "termination"},
)

POMA validates the payload up front (a corrupted or mislabeled upload 422s immediately), reads md over text so no structure is lost, neutralizes the dead ![](name) image references and counts them in content_metadata, strips running headers/footers, and rebuilds the cross-page heading tree before chunking. To force or suppress detection, set external_ocr_source to "llamaparse" or "none". After retrieval, merge hits instead of concatenating: chunksets from the same section share ancestor breadcrumbs, and deduplicating that lineage assembles one prompt-ready cheatsheet — 337 tokens of context versus 1,542 for a recursive-splitter baseline on our reference legal-document benchmark, zero information loss (methodology).

Metadata mapping: POMA fields → Chroma primitives ​

POMA chunk fieldChroma primitiveWhat it enables
to_embeddocuments entry, embedded by the collection's embedding functionsemantic query_texts retrieval
file_idmetadatas key → where={"file_id": ...}scope queries to one document
page (on the archive's chunk records, derived from LlamaParse pages[].page; take the chunkset's leaf chunk)metadatas key → where={"page": {"$gte": ...}}page-cited answers, page-range filters
depth (of the chunkset's leaf chunk)metadatas keyfilter/re-rank by hierarchy level
chunkset_indexmetadatas key + part of the entry idstable IDs, ordering at assembly time
chunkset lineagebreadcrumbs inside the document text + the .poma archivecheatsheet assembly; portability to any production store

The last row is the strategic one. Because the .poma archive — not the Chroma directory — is the source of truth, the collection above is a projection. Moving to Qdrant, pgvector, or Milvus later is a re-upsert of the same chunksets with the same metadata: no re-parsing, no re-chunking, no re-running your evaluation suite from scratch.

Frequently asked questions ​

How do I get LlamaParse results into Chroma for RAG? ​

Save the raw JSON result, run PrimeCut().ingest() on it (auto-detected, hierarchy rebuilt), then collection.add the chunksets: to_embed strings as documents, file_id/page/depth/chunkset_index as metadata — page comes from the archive's chunk records (the chunkset's leaf chunk), depth from the matching typed chunk. Query with query_texts plus where and where_document.

Should each LlamaParse page become one Chroma document? ​

No — pages[].md makes it tempting, but page-sized vectors blur every section on the page, n_results=5 returns five walls of text, and sections spanning page breaks are severed. Chunk by heading hierarchy instead; chunksets stay self-explanatory regardless of page boundaries.

What where filters work for LlamaParse content in Chroma? ​

where={"file_id": ...} for document scope, $gte/$in for page ranges and multi-document sets, where_document={"$contains": ...} for content matching — provided the metadata was written at add time. Concatenate-then-split pipelines lose the page numbers and have nothing to filter on.

Do LlamaParse image references end up in the Chroma collection? ​

Naively, yes — the saved JSON's dead ![](name) links get embedded and can even match $contains filters. POMA neutralizes each reference and counts it in content_metadata, so figure loss is visible and quantified, never noise in the collection.

Can I move a Chroma prototype built on LlamaParse output to production without re-chunking? ​

Yes — the .poma archive holds the chunks, chunksets, and metadata as the portable source of truth. Chroma is one projection; Qdrant, Pinecone, Weaviate, pgvector, and Milvus are others. Migration is a re-upsert, and your validated retrieval behavior carries over.

Same parser, different store: LlamaParse → Qdrant · LlamaParse → pgvector · LlamaParse → Milvus

Same store, different parser: Mistral OCR → Chroma · Docling → Chroma · Marker → Chroma

Foundations: The Optimal Chunker for LlamaParse · The Optimal Chunks for Chroma · All pipeline recipes · RAG chunking guide