Source: http://www.poma-ai.com/docs/optimal-chunks-mongodb-atlas

# MongoDB Atlas Chunking Strategy for RAG: The Optimal Chunks for Retrieval

<ByAuthor />

**The short answer:** the optimal chunks for MongoDB Atlas are **chunksets** — self-explanatory units that carry their full heading lineage — stored as documents with an embedding field covered by an Atlas Vector Search index and a `metadata` field carrying `file_id`/`chunkset_index`/`page`/`depth`, queried via `$vectorSearch` with an explicit `$project` of that metadata. Atlas's ranking is only as good as what you indexed; POMA's PrimeCut emits exactly the fields this document shape needs, and its `vektoria` package keeps the index content-free so retrieval assembles clean context from a volume instead of duplicating document bodies in your cluster.

## Atlas Vector Search: vector queries as an aggregation stage, not a separate system

MongoDB Atlas folds vector search directly into the aggregation pipeline you already use:

- **`$vectorSearch` stage** — an Atlas-specific search-index type (`type: "vector"`) on a `path`, with `numCandidates` and `limit` controlling the ANN search.
- **Compound hybrid via `$search`** — Atlas Search's compound query type combines BM25-style text matching with vector results in the same pipeline.
- **Ordinary document fields for filtering** — `file_id`, `page`, `depth` sit as normal fields; a `$match`/`filter` predicate scopes the search alongside the vector stage.
- **Atlas-only** — this is a managed-service feature, not part of self-hosted community MongoDB.

None of this decides what a document should represent. Atlas ranks whatever vector and fields you indexed. A document whose text holds a context-free character-count fragment will rank context-free fragments — correctly, inside the same cluster you already operate.

## What "optimal chunks" means for MongoDB Atlas, concretely

1. **One document per chunkset, self-explanatory alone.** A chunkset is a root-to-leaf path — `Master Services Agreement → Termination clauses → Early termination → "Either party may…"`. See [POMA chunksets](/learn/chunking/chunksets).
2. **Hierarchy as ordinary fields.** `file_id`, `page`, `depth` as normal document fields turn `$match` into document-aware scoping alongside the vector stage.
3. **`$project` the metadata, always.** The aggregation pipeline drops fields you don't explicitly project — a `$vectorSearch` stage with no follow-up `$project` on `metadata` is the one documented way retrieval silently loses `file_id`/`chunkset_index`.
4. **No overlap.** Overlap embeds every overlapped span twice, and Atlas's vector index cost scales with document count like any other collection index. Chunksets need no overlap.
5. **Content-free index.** Store `{file_id, chunkset_index}` in the document's `metadata` field and the actual content on a volume — smaller documents, citations reconstructed at retrieval time.

## The pipeline: PrimeCut to MongoDB Atlas

```bash
pip install pymongo
```

```python
from pymongo import MongoClient
from poma import PrimeCut
from poma.vektoria import assemble, records_from_archive, Volume
from poma.embeddings import get_embedder

# 1. Chunk any document and keep the .poma archive — the volume's source of truth.
client = PrimeCut()  # reads POMA_API_KEY
client.ingest("contract.pdf", download_dir="archives", filename="contract.poma")

embedder = get_embedder("local:BAAI/bge-small-en-v1.5")
vol = Volume("s3://your-bucket/poma")
mongo = MongoClient(os.environ["MONGODB_ATLAS_URI"])
col = mongo["poma"]["chunksets"]

# Atlas vector index (numDimensions, similarity) is created once via the Atlas UI/API,
# named "poma_vec" here — not through the pymongo driver itself.

# 2. Content-free ingest — vector index holds only routing metadata; content lives on the volume.
records = records_from_archive("archives/contract.poma")  # -> list[Record] (id, text, payload)
for r in records:
    vol.write_doc(r.payload["file_id"], {"file_id": r.payload["file_id"],
                                          "chunks": r.payload["chunks"], "text": r.text})
    col.insert_one({
        "_id": r.id,
        "embedding": embedder.embed([r.text])[0],
        "metadata": {"file_id": r.payload["file_id"], "chunkset_index": r.payload["chunkset_index"]},
    })

# 3. $vectorSearch retrieval — $project the metadata explicitly, then assemble.
qv = embedder.embed(["early termination conditions"])[0]
hits = list(col.aggregate([
    {"$vectorSearch": {"index": "poma_vec", "path": "embedding",
                       "queryVector": qv, "numCandidates": 200, "limit": 10}},
    {"$project": {"_id": 1, "metadata": 1, "score": {"$meta": "vectorSearchScore"}}},
]))
context = assemble(hits, volume=vol)  # -> [{"file_id", "content"}, ...]
```

Retrieved chunksets share ancestor lineage across a document — `assemble()` deduplicates that lineage into one prompt-ready cheatsheet, the same discipline that answers our reference legal-document benchmark with **337 tokens** instead of **1,542** for a recursive-splitter baseline. [Methodology here](/document-ingestion-chunking-rag).

## Chunk shapes in MongoDB Atlas, compared

| Property | Fixed 512-token chunks + overlap | Semantic chunks | **POMA chunksets** |
| --- | --- | --- | --- |
| Document is self-explanatory alone | ✗ | Sometimes | **✓ (by construction)** |
| Hierarchy as ordinary fields | Manual | Manual | **✓ (`file_id`, `page`, `depth`)** |
| Redundant vector-index entries | Many (overlap) | Few | **None** |
| Content-free documents + volume | DIY | DIY | **✓ (`vektoria` `assemble`/`Volume`)** |
| Metadata returned on hits | Requires explicit `$project` | Requires same | **Same requirement — documented, not silent** |

## Frequently asked questions

### What is the optimal chunk size for MongoDB Atlas Vector Search?

There's no universal token count — chunksets are self-explanatory at any size, which is why they beat fixed 512-token windows. Start at 512 tokens if you must fix a size, but the size knob cannot fix missing context.

### How do I define a MongoDB Atlas vector index for chunksets?

Via Atlas's search-index API: `type: "vector"`, `path` on your embedding field, `numDimensions` matching your embedder, `similarity: "cosine"`. Add `file_id`/`page` as ordinary fields for `$match` filtering.

### Why does POMA assemble return nothing from my MongoDB Atlas results?

`$vectorSearch` must be followed by a `$project` that explicitly includes `metadata` (holding `file_id`/`chunkset_index`) — the aggregation pipeline drops unprojected fields by default.

### Does MongoDB Atlas support hybrid vector and keyword search?

Yes — Atlas Search's compound query type combines BM25-style text matching with `$vectorSearch` results in the same aggregation pipeline.

### Does this work with self-hosted MongoDB, not just Atlas?

No — `$vectorSearch` and Atlas's vector index type are Atlas-specific; self-hosted community MongoDB doesn't have this feature.

## Feed MongoDB Atlas from the parser you already run

- [Mistral OCR → MongoDB Atlas](/pipelines/mistral-ocr-to-mongodb-atlas)
- [LlamaParse → MongoDB Atlas](/pipelines/llamaparse-to-mongodb-atlas)
- [Azure Document Intelligence → MongoDB Atlas](/pipelines/azure-document-intelligence-to-mongodb-atlas)
- [Unstructured.io → MongoDB Atlas](/pipelines/unstructured-to-mongodb-atlas)
- [Docling → MongoDB Atlas](/pipelines/docling-to-mongodb-atlas)
- [Marker → MongoDB Atlas](/pipelines/marker-to-mongodb-atlas)
- [AWS Textract → MongoDB Atlas](/pipelines/textract-to-mongodb-atlas)
- [PaddleOCR-VL → MongoDB Atlas](/pipelines/paddleocr-vl-to-mongodb-atlas)

Running a different store? Browse [all 13 vector databases](/pipelines/).

Fundamentals: [RAG chunking guide](/guides/rag-chunking/) · [RAG architecture guide](/guides/rag-architecture/) · [POMA chunksets](/learn/chunking/chunksets).