Source: http://www.poma-ai.com/docs/optimal-chunks-faiss

# FAISS Chunking for RAG: The Optimal Chunks for Retrieval

<ByAuthor />

**The short answer:** the optimal chunks for FAISS are **chunksets** — units that carry their full heading lineage inside the text you embed — plus a sidecar that maps the integer ids FAISS returns back to the document, the chunkset index and the chunk indices behind it. FAISS chunking is a stricter problem than chunking for a vector database, because FAISS is a similarity-search library and not a store: it holds vectors and returns row numbers, with no text, no metadata and no filters. Anything the retrieved unit does not carry itself, you have to carry for it.

This page covers what FAISS actually gives you, which index to build for RAG, the sidecar pattern that stands in for metadata filtering, and a complete pipeline in both plain FAISS and LangChain.

## Quick start: a FAISS RAG pipeline in Python

```bash
pip install poma faiss-cpu
```

```python
import faiss
import numpy as np
from poma import PrimeCut
from poma.retrieval import generate_cheatsheets

# 1. Chunk any document — PDF, DOCX, HTML, or a bring-your-own-OCR result JSON.
client = PrimeCut()  # reads POMA_API_KEY
result = client.ingest("contract.pdf")

# 2. Embed the chunkset texts. `to_embed` already carries the heading path.
texts = [cs.to_embed for cs in result.chunksets]
vectors = np.array(embed(texts), dtype="float32")  # your embedding model
faiss.normalize_L2(vectors)  # cosine similarity, computed as inner product

# 3. Build the index. FAISS stores the vectors and nothing else.
index = faiss.IndexFlatIP(vectors.shape[1])
index.add(vectors)
print(index.ntotal)

# 4. The sidecar: row i of the index is chunkset i. Keep the mapping yourself.
sidecar = [
    {"file_id": cs.file_id, "chunkset_index": cs.chunkset_index}
    for cs in result.chunksets
]

# 5. Search. `I` holds row numbers sorted best-first (increasing distance for L2
#    indexes, decreasing similarity for inner-product indexes), `D` the scores.
q = np.array(embed(["early termination conditions"]), dtype="float32")
faiss.normalize_L2(q)
D, I = index.search(q, 5)

# 6. Resolve rows through the sidecar, then merge the hits into one context block.
by_index = {cs.chunkset_index: cs.to_dict() for cs in result.chunksets}
hit_chunksets = [by_index[sidecar[i]["chunkset_index"]] for i in I[0] if i != -1]
cheatsheets = generate_cheatsheets(hit_chunksets, [c.to_dict() for c in result.chunks])
print(cheatsheets[0]["content"])
```

Two details in that snippet are the whole story. Step 2 embeds `to_embed`: the passage together with its chapter and section headings, so a hit is readable without its neighbours. Step 6 merges hits that share ancestors instead of concatenating them, deduplicating the repeated breadcrumb into what POMA calls a **cheatsheet**. On our reference legal document the same question was answered from **337 tokens** of context with chunksets versus **1,542** with a recursive character splitter, with nothing lost. One document is an illustration, not a benchmark; the [ingestion guide](/document-ingestion-chunking-rag) has the broader numbers.

## FAISS is an index library, not a vector database

[FAISS](https://faiss.ai/) describes itself as "a library for efficient similarity search and clustering of dense vectors". That description is exact, and the omissions in it are the design constraints of every FAISS RAG pipeline:

- **No text.** The index never sees the string you embedded. `index.add(xb)` takes a float32 array and nothing else.
- **No metadata.** There is no payload, no document field, no `file_id`. A vector has a row number and coordinates.
- **No filtering.** There is no `where` clause. Restricting a search to one document is something you do before or after the call, not inside it.
- **No persistence layer.** `faiss.write_index(index, path)` and `faiss.read_index(path)` write and read one file. There is no collection, no schema, no migration.
- **Integer ids only.** By default the id of a vector is its insertion ordinal. Indexes that cannot store ids can be wrapped in `IndexIDMap` so that `add_with_ids` accepts your own 64-bit integers, but they are still integers, not strings or objects.

None of that is a shortcoming. FAISS is deliberately the layer below a database, which is why several vector databases embed it, and why it gives you index types and tuning knobs that managed stores hide. It does mean the retrieval quality of a FAISS pipeline is decided entirely by two things you own: what text went into each vector, and what your sidecar can tell you about the row that came back.

## What FAISS chunking has to carry that other stores handle for you

In Qdrant or Chroma, a thin chunk can be partially rescued at query time: the payload still holds the section title, the page, the document id, and the filter can scope the search. In FAISS none of that exists at search time. So the chunk design has to satisfy four properties up front.

1. **Self-explanatory embedded text.** A chunkset is a root-to-leaf path — `Master Services Agreement → Termination clauses → Early termination → "Either party may…"` — and that path is inside `to_embed`. The vector encodes the context, so the nearest-neighbour result is interpretable with nothing attached. See [POMA chunksets](/learn/chunking/chunksets).
2. **A stable, dense id space.** FAISS ids are row numbers, so the natural key is position. Chunksets come out of PrimeCut with a `chunkset_index` per document and a `file_id`, which map onto a list index cleanly and stay stable across rebuilds.
3. **No overlap.** Overlap embeds every overlapped span twice. In FAISS that is doubly expensive: the index is resident in memory, and near-duplicate vectors crowd each other out of a top-`k` that has no metadata diversity to fall back on. Chunksets carry context structurally, so no overlap is needed.
4. **Enough in the sidecar to rebuild an answer.** `{file_id, chunkset_index}` per row is the minimum. The chunkset's `chunks` list (the chunk indices it spans) is what lets you merge overlapping hits into one deduplicated context block rather than pasting five near-identical breadcrumbs into the prompt.

Tables matter here for the same reason. A table cut in half by a character splitter is unrecoverable in FAISS: nothing downstream knows the other half exists. PrimeCut keeps tables whole and keeps page numbers on chunks, so citations survive a round trip through an index that stores integers.

## Choosing a FAISS index for RAG

The index type decides the speed, memory and recall trade-off. It does not decide retrieval quality in the sense that matters for RAG — that was fixed when you chose what to embed — but the wrong choice makes an evaluation hard to read, because you cannot tell a chunking problem from a recall problem.

| Index | What it does | Training | Use it when |
| --- | --- | --- | --- |
| `IndexFlatIP(d)` | Exact search on inner product; cosine if you `normalize_L2` first | None (`is_trained` is true) | Default for RAG. Exact recall, no tuning, no surprises during evaluation |
| `IndexFlatL2(d)` | Exact search on squared L2 distance | None | Your embedder is trained for Euclidean distance rather than cosine |
| `IndexHNSWFlat(d, M)` | Graph-based approximate search, `M` neighbours per node, over flat storage | Builds on add | Exact search is too slow and memory is available. Recall is tunable at search time |
| `IndexIVFFlat(quantizer, d, nlist)` | Inverted file with exact post-verification; a coarse quantizer assigns vectors to `nlist` lists | `train()` on a representative sample | Large corpora where memory and build time matter more than exact recall |
| `IndexIDMap(index)` | Wrapper adding `add_with_ids` to indexes that cannot store ids | Inherits | Your row numbers must survive deletions or partial rebuilds |

Two practical notes. Cosine similarity is not a metric type in FAISS: you use an inner-product index and call `faiss.normalize_L2(x)` on both the stored vectors and the query, exactly as in the quick start. And `index.search` returns `-1` in `I` where fewer than `k` neighbours were found, so filter those out before indexing into your sidecar rather than letting a `-1` silently select the last element of a Python list.

## FAISS metadata filtering: the sidecar pattern

There is no native FAISS metadata filtering. What exists in practice is one of three patterns, in increasing order of effort:

**One index per scope.** If you always search within a single document or tenant, build one index per scope and skip filtering entirely. Cheap, exact, easy to ship and to delete. It stops working once queries span scopes.

**Post-filtering through the sidecar.** Search for more candidates than you need, resolve them through the sidecar, drop the ones that fail the predicate, and truncate to `k`:

```python
D, I = index.search(q, 50)                       # fetch more than you need
rows = [sidecar[i] for i in I[0] if i != -1]
kept = [r for r in rows if r["file_id"] == target_file_id][:5]
```

This is the pattern LangChain's FAISS wrapper implements. Its `similarity_search(query, k, filter=..., fetch_k=20)` searches for `fetch_k` candidates when a filter is present, then applies the filter to the docstore metadata and returns the surviving top `k`. The consequence is worth stating plainly: a selective filter over a small `fetch_k` can return fewer than `k` results, or none. Raise `fetch_k` when the predicate is narrow, and prefer separate indexes when the predicate is a hard partition.

**`IDSelector` with an ID-mapped index.** FAISS can restrict a search to a set of ids through search parameters, which is checked during the scan rather than after it, so `k` is honoured; only `IDSelectorRange` limits the search up front. It requires an index that supports ids and a selector built per query, so it pays off when the allowed set is large and stable, not for ad-hoc predicates.

Whichever you pick, the sidecar is the artifact that makes it possible, and it has to be built at ingest time. That is the same discipline as writing `file_id`, `page` and `depth` as payload fields in a store that has payloads — the difference is only that FAISS will not remind you.

## FAISS RAG with LangChain

If your stack is already LangChain, the wrapper handles the index, the docstore and the id map, and POMA plugs in as the loader and the splitter:

```bash
pip install 'poma[langchain]' faiss-cpu langchain-community
```

```python
from langchain_community.vectorstores import FAISS
from poma import PrimeCut
from poma.integrations.langchain import (
    PomaFileLoader,
    PomaChunksetSplitter,
    PomaCheatsheetRetrieverLC,
)

# 1. Load files into Documents. Each carries `source_path` and `doc_id` metadata —
#    PomaChunksetSplitter needs `source_path` to send the file to PrimeCut.
docs = PomaFileLoader("contracts/").load()

# 2. Split into chunkset Documents: page_content is `to_embed`, and metadata
#    keeps `chunkset`, `chunks`, `chunkset_index` and `doc_id`.
splitter = PomaChunksetSplitter(client=PrimeCut(), verbose=True)
chunkset_docs = splitter.split_documents(docs)

# 3. Build the FAISS store. from_documents embeds, creates an IndexFlatL2 by
#    default, and keeps the Documents in an InMemoryDocstore.
store = FAISS.from_documents(chunkset_docs, embeddings)  # your Embeddings object

# 4a. Plain retrieval — returns the chunkset Documents.
retriever = store.as_retriever(search_kwargs={"k": 5})

# 4b. Or merged context: hits grouped by document, shared lineage deduplicated.
cheatsheet_retriever = PomaCheatsheetRetrieverLC(store, top_k=6)
docs_out = cheatsheet_retriever.invoke("What are the early termination conditions?")
print(docs_out[0].page_content)
```

`FAISS.from_documents` builds an `IndexFlatL2` unless you pass `distance_strategy=DistanceStrategy.MAX_INNER_PRODUCT`, which switches it to `IndexFlatIP`; there is also a `normalize_L2=True` argument, which warns when used outside the Euclidean strategy but still normalizes. One underrated advantage of the docstore approach: because metadata is held in Python objects rather than a typed column, the nested `chunkset` and `chunks` dictionaries survive intact. Stores that require scalar metadata force you to flatten or re-fetch them.

## Persisting a FAISS index and its sidecar

An index in memory is not a pipeline. Persistence is two files that must stay in step:

```python
import json
import faiss

faiss.write_index(index, "contracts.faiss")
with open("contracts.sidecar.json", "w") as f:
    json.dump(sidecar, f)

# Later, in the serving process:
index = faiss.read_index("contracts.faiss")
with open("contracts.sidecar.json") as f:
    sidecar = json.load(f)
assert index.ntotal == len(sidecar)  # the failure you want to catch loudly
```

That assertion is the single most valuable line in a FAISS RAG pipeline. A sidecar out of step with the index does not raise: it returns confidently wrong documents, and the answers look plausible. Version the two artifacts together, rebuild them together, and check the count at load time.

LangChain wraps the same mechanics: `store.save_local("faiss_contracts")` writes `index.faiss` via `faiss.write_index` plus a pickled `index.pkl` holding the docstore and the id map, and `FAISS.load_local(folder, embeddings, allow_dangerous_deserialization=True)` reads them back. That flag is not decoration — the pickle can execute arbitrary code on load, so only set it for files you produced yourself.

Keep the `.poma` archive from the ingest as the third artifact. It holds the chunks, chunksets and hierarchy independent of any index, so an embedder change is a re-embed rather than a re-chunk, and moving to another store is a re-upsert of the same units.

## FAISS vs ChromaDB for RAG

Honestly: they are not competitors, they sit at different layers, and the comparison people actually want is "do I want to do the bookkeeping myself".

FAISS gives you the search itself — the widest range of index types, real control over the memory and recall trade-off, GPU support in the `faiss-gpu` build, and no dependency beyond a file on disk. Chroma gives you a database around a vector index: it stores the documents and their metadata, filters with a `where` clause, persists both without serialization code, and gets you to a working loop in fewer lines. Several databases build on FAISS internally, which is the clearest statement of the relationship.

Pick FAISS when the index is the interesting part, when you want to tune HNSW or IVF against your own recall measurements, or when the deployment is a file you ship. Pick Chroma when metadata filtering and persistence are chores you would rather not own. The chunk design does not change between them, which is the point: chunksets ported to [Chroma](/optimal-chunks-chroma) are the same units, with the sidecar's contents written as metadata instead.

## Frequently asked questions

### Why is FAISS chunking different from chunking for a vector database?

FAISS is a similarity-search library, not a database. It stores vectors and returns row numbers; it has no metadata, no filtering, and no text. Everything a managed store would keep next to the vector has to live either inside the embedded text or in a sidecar structure you maintain yourself. That makes the chunk design load-bearing: a chunk that is not self-explanatory cannot be repaired by a payload field at query time, because there is no payload.

### What is the optimal chunk size for FAISS?

There is no universal token count, and FAISS imposes none: the index only cares about vector dimensionality, not about how much text produced the vector. The useful criterion is whether a retrieved unit is understandable on its own. Chunksets carry their full heading lineage into the embedded text, so they satisfy that at any size. If you must fix a number, start around 512 tokens, but size cannot restore context that the splitter removed.

### Which FAISS index should I use for RAG?

Start with `IndexFlatIP` on L2-normalized vectors, which gives exact cosine search and needs no training. It is fast enough for the corpus sizes most RAG systems actually have. Move to `IndexHNSWFlat(d, M)` when exact search gets slow and you can afford the memory, or to `IndexIVFFlat(quantizer, d, nlist)` when memory matters more and you can run `train()` on a representative sample. Approximate indexes trade recall for speed, so evaluate them against the flat baseline before switching.

### Does FAISS support metadata filtering?

Not in the way a vector database does. FAISS has no metadata layer, so there is no equivalent of a `where` clause on `file_id` or `page`. The common pattern is a sidecar: a Python list or dict mapping the integer id FAISS returns to your own record, which you then filter after the search. LangChain's FAISS wrapper implements exactly this, with an in-memory docstore and a post-filter over `fetch_k` candidates. Post-filtering can return fewer than `k` results, so raise `fetch_k` when a filter is selective.

### How do I build a FAISS RAG pipeline in Python?

Chunk the documents, embed the chunk texts into a float32 array, call `faiss.normalize_L2` if you want cosine, build an index such as `faiss.IndexFlatIP(d)`, and `index.add(vectors)`. At query time embed the question, call `D, I = index.search(q, k)`, and map each id in `I` back to your chunk through the sidecar. `I` holds row numbers sorted best-first: increasing distance for L2 indexes, decreasing similarity for inner-product indexes. It contains `-1` where fewer than `k` neighbours were found, so skip those entries before resolving.

### How do I use FAISS with LangChain for RAG?

Install `faiss-cpu`, produce LangChain Documents, and call `FAISS.from_documents(docs, embeddings)`. The wrapper builds the index, puts each Document in an `InMemoryDocstore`, and keeps an `index_to_docstore_id` map so hits come back as Documents with their metadata intact. POMA's `PomaFileLoader` and `PomaChunksetSplitter` produce those Documents directly, with the chunkset text as `page_content` and the chunkset plus its chunks in metadata, which is what `PomaCheatsheetRetrieverLC` needs to assemble merged context.

### How do I persist a FAISS index and its metadata?

`faiss.write_index(index, "contracts.faiss")` writes the index to one file and `faiss.read_index` loads it back. The sidecar is your responsibility: serialize it separately, as JSON if you want it readable, and version the two together, because a sidecar that no longer matches the index row order silently returns the wrong documents. LangChain's `save_local` writes both for you, `index.faiss` plus a pickled `index.pkl`, and `load_local` requires `allow_dangerous_deserialization=True` because loading a pickle can execute code.

### FAISS vs ChromaDB for RAG: which should I use?

They solve different halves of the problem. FAISS is the search library, with the widest range of index types and the best control over the speed, memory and recall trade-off; several vector databases use it internally. Chroma is a database around a vector index: it stores documents and metadata, filters with a `where` clause, and persists both without you writing serialization code. Choose FAISS when the index itself is the interesting part or when you want no dependency beyond a file, and Chroma when you want the bookkeeping handled. The chunk design is identical either way, so it does not lock the decision in.

Running a different store? The same chunk design applies: [Azure AI Search](/optimal-chunks-azure-ai-search) · [Chroma](/optimal-chunks-chroma) · [Elasticsearch](/optimal-chunks-elasticsearch) · [LanceDB](/optimal-chunks-lancedb) · [Milvus](/optimal-chunks-milvus) · [MongoDB Atlas](/optimal-chunks-mongodb-atlas) · [OpenSearch](/optimal-chunks-opensearch) · [pgvector](/optimal-chunks-pgvector) · [Pinecone](/optimal-chunks-pinecone) · [Qdrant](/optimal-chunks-qdrant) · [Redis](/optimal-chunks-redis) · [Turbopuffer](/optimal-chunks-turbopuffer) · [Vespa](/optimal-chunks-vespa) · [Weaviate](/optimal-chunks-weaviate) — or browse [all pipeline recipes](/pipelines/).

## Continue reading

- [POMA chunksets](/learn/chunking/chunksets) — what a root-to-leaf retrieval unit is and why it needs no overlap
- [Chunking strategies and text splitters](/rag-chunking-strategies-text-splitters) — the full taxonomy, and where fixed-size splitting fails
- [Document ingestion and chunking for RAG](/document-ingestion-chunking-rag) — the 337 vs 1,542 token illustration and its methodology
- [The optimal chunks for Chroma](/optimal-chunks-chroma) — the same design in a store that does have metadata filters
- [RAG architecture guide](/guides/rag-architecture/) — where chunking sits in the rest of the system