Source: http://www.poma-ai.com/docs/optimal-chunks-chroma

# Chroma Chunking Strategy for RAG: The Optimal Chunks

<ByAuthor />

**The short answer:** the optimal chunks for Chroma are **chunksets** — self-explanatory units that carry their full heading lineage — added as documents with hierarchy metadata (`file_id`, `page`, `depth`) that Chroma's `where` filters can act on. Chroma gets you from zero to a working retrieval loop faster than any other store; what decides whether that loop actually answers questions is what you `collection.add()` into it. And because the chunking decision you make at prototype time is the one that sticks, make it with chunks that port unchanged to whatever you run in production.

This page covers what Chroma does brilliantly, what Chroma Research's own chunking evaluation established (and what it left out), and the Chroma chunking design — chunks plus metadata — that gets the most out of the store.

## Chroma gives you the fastest path from zero to retrieval — over whatever you feed it

Chroma is the prototyping engineer's vector store, and it earns that reputation:

- **Local-first, embedded** — `chromadb.PersistentClient(path=...)` gives you a durable collection on disk with no server to run; the same API talks to a client/server deployment when you outgrow the laptop.
- **Collections** — a natural unit per corpus or experiment, created idempotently with `get_or_create_collection`.
- **`where` metadata filters + `where_document` content filters** — scope queries by any scalar metadata field, or by full-text conditions on the chunk text itself.
- **Default embedding functions, or bring your own** — pass `documents=` and Chroma embeds them for you; swap in any embedding function when you want control.

None of these primitives, however, know anything about your documents. Chroma retrieves the nearest neighbors of what you embedded. If what you embedded is a context-free fragment — a paragraph cut loose from the section that gives it meaning — Chroma will faithfully retrieve context-free fragments, on your laptop and in production alike. **The ceiling on retrieval quality is set before the first `collection.add()`.**

## What the Chroma Research chunking evaluation measures — and what it leaves out

Chroma's research team published an evaluation of chunking strategies ([trychroma.com/research/evaluating-chunking](https://www.trychroma.com/research/evaluating-chunking)). It is the most useful public reference point when you are picking a splitter, and it is worth reading for what it deliberately does not cover as much as for its results.

**What it compared.** Two off-the-shelf splitters, `RecursiveCharacterTextSplitter` and `TokenTextSplitter`, against semantic approaches: the `KamradtSemanticChunker`, a `KamradtModifiedChunker`, a `ClusterSemanticChunker`, and an `LLMSemanticChunker`. The last three are the paper's own contributions.

**How it scores.** Ground truth is token-level. GPT-4-Turbo generated factual queries paired with the exact excerpts that answer them across five corpora, and a retrieved chunk is judged by which of those tokens it brings back. Three metrics follow: recall, the share of relevant tokens retrieved; precision, the share of retrieved tokens that are relevant; and Intersection over Union, borrowed from computer vision, which also penalises the padding a chunk drags along with the answer.

**What it found.** The cut point matters. The study reports recall varying by up to roughly 9% across configurations, small chunks without overlap scoring best on IoU, and the default parameters of popular splitters underperforming tuned ones. It states that it does not measure how long each method takes to run.

**What it leaves out.** IoU rewards a chunk that contains the answer tokens and little else. It says nothing about whether the retrieved unit is comprehensible on its own — whether the model reading it, or the human reading the citation, can tell which contract, which section, and which page it came from. The evaluation grades **where you cut**, not **what the unit carries**. A 200-token span can score perfectly and still reach the LLM as "Either party may terminate on 30 days' notice," with no indication of which agreement that sentence belongs to. Every strategy in the comparison shares that property, because all of them return a bare span of the source text.

That is the half the rest of this page is about. Cut well *and* keep the lineage attached, and the two decisions stop competing.

## The chunking decision made at prototype time is the one that sticks

Here is the pattern that plays out in almost every RAG project: someone builds a Chroma prototype in an afternoon with a quick fixed-size splitter, the demo works well enough to get funded, and then the splitter — the least examined line in the notebook — becomes load-bearing. Re-chunking later means re-embedding everything, re-running every evaluation, and explaining why retrieval behavior shifted. So nobody does it. The prototype's chunking ships.

That makes the prototype exactly the wrong place to cut corners on chunking, and exactly the right place to fix it once:

1. **Chunk with hierarchy from day one.** POMA's PrimeCut turns a PDF, DOCX, HTML file — or a bring-your-own-OCR result JSON — into **chunksets**: root-to-leaf paths through the document's heading hierarchy, each one self-explanatory when retrieved alone. See [POMA chunksets](/learn/chunking/chunksets).
2. **Keep the source of truth outside the store.** Every ingest produces a portable `.poma` archive containing the chunks, chunksets, and metadata. Chroma holds a projection of it, not the original. When production lands on Qdrant, Pinecone, or pgvector, you re-upsert the same chunksets — the chunking decision, and every evaluation you ran against it, ports unchanged.
3. **Validate on honest chunks.** A prototype evaluated on fragments tells you how well fragments retrieve. A prototype evaluated on chunksets tells you how the production system will behave.

## What "optimal chunks" means for Chroma, concretely

Four properties separate a Chroma collection that answers questions from one that returns trivia:

1. **Every document is self-explanatory.** A chunkset reads like `Master Services Agreement → Termination clauses → Early termination → "Either party may…"`. Retrieved alone, it tells the LLM (and the human reading citations) exactly where it sits. A bare 512-token window does not.
2. **No overlap, no near-duplicates.** Overlap embeds every overlapped span twice, and near-identical vectors crowd out diverse candidates in `n_results`. Chunksets carry context structurally, so overlap is unnecessary and your top-5 returns five *different* passages.
3. **Hierarchy as scalar metadata.** Chroma metadata values are scalars (or homogeneous arrays of them) — which is precisely the shape of `file_id`, `chunkset_index`, `page`, and `depth`. Written at ingest time, they turn `where` filters into document-aware operations: scope a query to one contract, exclude appendices by depth, cite pages in the answer. `where_document` adds content-level conditions on top.
4. **Portable by construction.** The `.poma` archive is the source of truth; Chroma, Qdrant, Pinecone, Weaviate, pgvector, and Milvus are interchangeable projections of it. Your prototype and your production system retrieve the same units.

## From document to a queryable Chroma collection

No special integration needed — chunksets map directly onto Chroma's native API:

```bash
pip install poma chromadb
```

```python
import chromadb

from poma import PrimeCut
from poma.utils import unpack_poma_archive

# 1. Chunk any document — PDF, DOCX, HTML, or a bring-your-own-OCR result JSON.
#    Keep the .poma archive: page numbers live on its chunk records.
client = PrimeCut()  # reads POMA_API_KEY
result = client.ingest("contract.pdf", download_dir="store", filename="contract.poma")

# Look up hierarchy per chunk index. The SDK exposes depth; page comes from
# the archive's chunk records.
depth_of = {c.chunk_index: c.depth for c in result.chunks}
archive = unpack_poma_archive(poma_archive_path="store/contract.poma")
# -1 when the source has no pages (Markdown, text)
page_of = {c["chunk_index"]: (c.get("page") if c.get("page") is not None else -1) for c in archive["chunks"]}

# 2. A durable local collection — same API embedded or client/server.
chroma = chromadb.PersistentClient(path="./chroma")
collection = chroma.get_or_create_collection("contracts")

# 3. Add chunksets: Chroma embeds `documents` with its default embedding
#    function (or bring your own); hierarchy lands as filterable metadata.
#    `cs.chunks` holds chunk indices in document order; the last is the leaf.
collection.add(
    documents=[cs.to_embed for cs in result.chunksets],
    metadatas=[
        {
            "file_id": cs.file_id,
            "chunkset_index": cs.chunkset_index,
            "page": page_of[cs.chunks[-1]],
            "depth": depth_of[cs.chunks[-1]],
        }
        for cs in result.chunksets
    ],
    ids=[f"{cs.file_id}:{cs.chunkset_index}" for cs in result.chunksets],
)

# 4. Query, scoped to one document via a where filter.
hits = collection.query(
    query_texts=["What does the contract say about early termination?"],
    n_results=5,
    where={"file_id": result.chunksets[0].file_id},
)
```

After retrieval, merge the hits instead of concatenating them: chunksets from the same section share ancestor headings, and deduplicating that shared lineage assembles one coherent context block — a **cheatsheet**. The `.poma` archive (or your document store) holds everything needed to rebuild it from the retrieved IDs. On one reference legal document, cheatsheet assembly answered with **337 tokens** of context versus **1,542** for a recursive-splitter baseline, with nothing lost. One document is an illustration, not a benchmark; the [ingestion guide](/document-ingestion-chunking-rag) has the broader numbers.

## ChromaDB chunking in practice: what `collection.add()` actually stores

The API is small enough to hold in your head, and its shape explains why the splitter decides so much. Per [Chroma's add-data reference](https://docs.trychroma.com/docs/collections/add-data), `collection.add()` takes `ids` plus **either** `documents`, **or** `embeddings`, or both; `metadatas` is always optional. Hand it `documents` alone and Chroma embeds them with the collection's embedding function. Hand it both and it stores your vectors without re-embedding. Records whose `ids` already exist are ignored rather than merged — use `update` (or `upsert`) to change one — and every vector in a collection must share a dimensionality.

Three consequences for how you chunk:

1. **The string in `documents` is the whole unit.** It is what gets embedded, what `where_document` full-text conditions run against, and what comes back for the prompt. There is no second field the model sees at answer time. Whatever context the chunk needs has to be *inside* that string. This is why appending the heading lineage to the leaf text is not a formatting nicety — it is the only place it can live.
2. **Metadata is for filtering, not for context.** `metadatas` values are strings, integers, floats, booleans, or arrays of one of those types. Perfect for `file_id`, `chunkset_index`, `page`, and `depth`, which is what `where` clauses act on. But a heading stored only in metadata never reaches the embedding and never reaches the LLM unless you re-inject it yourself.
3. **Ids are your join key back to the source.** `f"{file_id}:{chunkset_index}"` is enough to rebuild a chunkset's neighbours and ancestors from the `.poma` archive after retrieval, which is what makes cheatsheet assembly cheap.

## Chunk shapes in Chroma, compared

| Property | Fixed 512-token chunks + overlap | Semantic chunks | **POMA chunksets** |
| --- | --- | --- | --- |
| Document is self-explanatory when retrieved alone | ✗ | Sometimes | **✓ (by construction)** |
| Near-duplicate vectors in the collection | Many (overlap) | Few | **None** |
| Scalar metadata for `where` filters | Manual | Manual | **✓ (`file_id`, `page`, `depth`)** |
| Tables survive embedding | ✗ (cut mid-row) | Fragile | **✓ (HTML, never cut)** |
| Ports from prototype to production DB unchanged | Re-chunk + re-eval | Re-chunk + re-eval | **✓ (`.poma` archive)** |
| Post-retrieval assembly | Concatenate raw hits | Concatenate raw hits | **Cheatsheets (deduped lineage)** |

## Frequently asked questions

### What is the optimal chunk size for ChromaDB?

There is no universal token count — the optimal unit is a chunk that is self-explanatory without its neighbors. Chunksets achieve that at any size, which is why they beat fixed 512-token windows: every document in the collection can be retrieved alone and still be understood. If you must fix a size, start around 512 tokens — but the size knob cannot restore missing context.

### What metadata should I store with chunks in ChromaDB?

`file_id`, `chunkset_index`, `page`, `depth` — flat scalar fields, which is exactly what Chroma metadata accepts (strings, ints, floats, booleans, or homogeneous arrays of those). `page` comes from the chunk records in the `.poma` archive (the chunkset's leaf chunk); the SDK objects expose `depth` but not `page`. Every one of them is filterable with `where`, enabling per-document scoping, page citations, and depth-aware filtering.

### How do I filter chunks by metadata in ChromaDB?

Pass a `where` clause to `collection.query` — `where={"file_id": "contract-2024"}` scopes to one document; `$in`, `$gte`, `$ne` compose richer filters. `where_document` adds full-text conditions on the chunk content (`$contains`). Both depend on metadata written at ingest time.

### Does chunk overlap hurt a Chroma collection?

Yes. Every overlapped span is embedded twice, and the resulting near-duplicates crowd out diverse candidates in `n_results`. Chunks that carry their heading lineage structurally don't need overlap at all.

### Can I move my Chroma prototype to another vector database without re-chunking?

Yes — if the chunks live outside Chroma as a portable artifact. The `.poma` archive is the source of truth; moving to Qdrant, Pinecone, Weaviate, pgvector, or Milvus is a re-upsert of the same chunksets with the same metadata, not a re-chunking project.

### How do I turn Chroma query results into prompt-ready context?

Merge, don't concatenate: retrieved chunksets share ancestor headings, so deduplicate the shared lineage into one coherent block — a cheatsheet. On one reference legal document that assembly answered with 337 tokens of context instead of 1,542. One document is an illustration, not a benchmark; the [ingestion guide](/document-ingestion-chunking-rag) has the broader numbers.

### What did the Chroma Research chunking evaluation measure?

Chroma Research published an evaluation of chunking strategies that compares `RecursiveCharacterTextSplitter` and `TokenTextSplitter` against semantic approaches, including the `KamradtSemanticChunker`, a `KamradtModifiedChunker`, a `ClusterSemanticChunker` and an `LLMSemanticChunker`. It scores retrieval at the token level: recall is the share of relevant tokens returned, precision the share of returned tokens that are relevant, and Intersection over Union penalises the padding a chunk drags along with the answer. Its findings are about where to cut: small chunks without overlap scored best on IoU, and the default settings of popular splitters underperformed tuned ones. It grades cut points, not whether the retrieved unit is understandable on its own.

### What is the best ChromaDB chunking strategy?

Split on document structure rather than on a character count, and give every unit the headings it sits under. Chroma Research's evaluation shows that the cut point alone moves recall by several points, so a tuned recursive or semantic splitter beats library defaults. What it cannot show is which unit an LLM can read alone, because every strategy it compares returns a bare span. Chunksets supply that half: each unit carries its root-to-leaf heading lineage as text, so the document you add to the collection is self-explanatory at any size and needs no overlap to compensate the cut.

## First-party ingest: vektoria's push()

Chroma is one of only two databases (with pgvector) that has a real first-party `vektoria` ingest connector — one call handles chunking-to-vector in a content-free shape:

```python
from poma.vektoria import push, get_connector
from poma.embeddings import get_embedder

stats = push(
    "contract.poma",
    volume="s3://your-bucket/poma",
    connector=get_connector("chroma", collection="poma", path="./chroma"),
    embedder=get_embedder("local:BAAI/bge-small-en-v1.5"),
)
```

`push()` writes content to the volume first (so a vector id never points at missing content), then embeds and upserts content-free vectors — `(id, vector, {file_id, chunkset_index})` — via the connector. Retrieval is unchanged: run your normal `collection.query(...)`, then `assemble(results, volume=VOL)`.

## Feed Chroma from the parser you already run

Pair this page with your ingestion side — the same chunksets, straight from your OCR/parsing tool's raw output:

- [AWS Textract → Chroma](/pipelines/textract-to-chroma)
- [Azure Document Intelligence → Chroma](/pipelines/azure-document-intelligence-to-chroma)
- [Docling → Chroma](/pipelines/docling-to-chroma)
- [LlamaParse → Chroma](/pipelines/llamaparse-to-chroma)
- [Marker → Chroma](/pipelines/marker-to-chroma)
- [Mistral OCR → Chroma](/pipelines/mistral-ocr-to-chroma)
- [PaddleOCR-VL → Chroma](/pipelines/paddleocr-vl-to-chroma)
- [Unstructured.io → Chroma](/pipelines/unstructured-to-chroma)

Running a different store? The same chunk design applies: [Azure AI Search](/optimal-chunks-azure-ai-search) · [Elasticsearch](/optimal-chunks-elasticsearch) · [FAISS](/optimal-chunks-faiss) · [LanceDB](/optimal-chunks-lancedb) · [Milvus](/optimal-chunks-milvus) · [MongoDB Atlas](/optimal-chunks-mongodb-atlas) · [OpenSearch](/optimal-chunks-opensearch) · [pgvector](/optimal-chunks-pgvector) · [Pinecone](/optimal-chunks-pinecone) · [Qdrant](/optimal-chunks-qdrant) · [Redis](/optimal-chunks-redis) · [Turbopuffer](/optimal-chunks-turbopuffer) · [Vespa](/optimal-chunks-vespa) · [Weaviate](/optimal-chunks-weaviate) — or browse [all pipeline recipes](/pipelines/).

Fundamentals: [RAG chunking guide](/guides/rag-chunking/) · [RAG architecture guide](/guides/rag-architecture/) · [POMA chunksets](/learn/chunking/chunksets).