Source: http://www.poma-ai.com/docs/optimal-chunks-opensearch

# OpenSearch Chunking: Text Chunking Processor vs Chunksets for RAG

<ByAuthor />

**The short answer:** OpenSearch chunking runs at ingest time, in the `text_chunking` processor — `fixed_token_length`, `fixed_char_length` or `delimiter`, chained into `text_embedding` in the same pipeline. It decides where a passage ends. It does not decide what the passage carries with it. The optimal chunks for OpenSearch are **chunksets** — self-explanatory units that carry their full heading lineage — indexed as documents with a `knn_vector` field for the embedding and `file_id`/`page`/`depth` as filterable keyword/integer fields, with an analyzed text field enabling BM25 alongside the k-NN plugin's search. OpenSearch's ranking is only as good as what you indexed; POMA's PrimeCut emits exactly the fields this mapping needs, and its `vektoria` package keeps the index content-free so retrieval assembles clean context from a volume instead of a bloated document body.

## OpenSearch's own k-NN plugin, forked and independent since 2021

OpenSearch diverged from Elasticsearch at version 7.10 and built its own vector search stack:

- **`knn_vector` fields** — HNSW-backed approximate k-NN via `nmslib` or `faiss`, `space_type: cosinesimil` (or `l2`, `innerproduct`) declared per field.
- **Native BM25** — every analyzed text field participates in relevance scoring without a second system.
- **A dedicated hybrid query type** — OpenSearch's `hybrid` query combines lexical and vector sub-queries, normalized and fused by a **search pipeline** — a distinct mechanism from Elasticsearch's RRF `rank`.
- **Mature filtering** — `bool.filter` on `keyword`/`numeric` fields runs at index speed.

None of this decides what a document should represent. OpenSearch ranks whatever vector and text you indexed. A mapping whose `text` field holds a context-free character-count fragment will rank context-free fragments — correctly, and fast.

## The OpenSearch text chunking processor, parameter by parameter

`text_chunking` is an ingest processor. It takes one long text field and writes an array of shorter passages into another field:

```json
{
  "text_chunking": {
    "field_map": { "<input_field>": "<output_field>" },
    "algorithm": { "<name>": { } }
  }
}
```

`field_map` and `algorithm` are both required. `algorithm` holds at most one key-value pair, and the name defaults to `fixed_token_length`. Two more optional settings sit next to them: `ignore_missing` (default `false`; when `true`, empty fields are dropped from the output instead of yielding an empty list) and the usual `description`/`tag`.

The three algorithms and their parameters, as documented:

| Algorithm | Parameter | Default | Notes |
| --- | --- | --- | --- |
| `fixed_token_length` | `token_limit` | `384` | integers ≥ 1; 384 = 512 × 0.75, so passages fit a 512-token model |
| | `tokenizer` | `standard` | any word tokenizer |
| | `overlap_rate` | `0` | float in 0–0.5; the docs recommend 0–0.2 |
| `fixed_char_length` | `char_limit` | `2048` | integers ≥ 1; 2048 = 512 × 4 chars |
| | `overlap_rate` | `0` | float in 0–0.5 |
| `delimiter` | `delimiter` | `\n\n` | any string, e.g. `\n` for lines or `. ` for sentences |
| all three | `max_chunk_limit` | `100` | excess text is appended to the last chunk; `-1` disables |

Two behaviours worth knowing before you rely on it. Dot paths are not supported for nested fields — write `"field_map": { "foo": { "bar": "bar_chunk" } }`, not `"field_map": { "foo.bar": "foo.bar_chunk" }`. And `max_chunk_limit` does not truncate: when a document produces more passages than the limit, the leftover text is glued onto the final chunk, which quietly produces one oversized passage the embedding model will then truncate.

A minimal pipeline:

```json
PUT _ingest/pipeline/text-chunking-ingest-pipeline
{
  "description": "A text chunking ingest pipeline",
  "processors": [
    {
      "text_chunking": {
        "algorithm": {
          "fixed_token_length": { "token_limit": 384, "overlap_rate": 0.2, "tokenizer": "standard" }
        },
        "field_map": { "passage_text": "passage_chunk" }
      }
    }
  ]
}
```

Test it with `POST _ingest/pipeline/text-chunking-ingest-pipeline/_simulate` before you index anything real — the response shows the exact passage boundaries, including the words duplicated by `overlap_rate`.

## Cascading processors: the OpenSearch chunking strategy for structure

A single `text_chunking` stage cuts on one rule. Chaining stages lets each stage cut inside the output of the last, which is how OpenSearch does structure-aware splitting. Split on paragraphs first, then enforce a token limit inside each paragraph:

```json
PUT _ingest/pipeline/text-chunking-cascade-ingest-pipeline
{
  "description": "A text chunking pipeline with cascaded algorithms",
  "processors": [
    { "text_chunking": { "algorithm": { "delimiter": { "delimiter": "\n\n" } },
                         "field_map": { "passage_text": "passage_chunk1" } } },
    { "text_chunking": { "algorithm": { "fixed_token_length": { "token_limit": 500, "overlap_rate": 0.2, "tokenizer": "standard" } },
                         "field_map": { "passage_chunk1": "passage_chunk2" } } }
  ]
}
```

Add a third stage and you get a recursive effect: paragraphs (`\n\n`), then sentences (`. `), then `fixed_char_length` with a `char_limit` as the backstop. Each level preserves as much structure as the size constraint allows.

Then chain `text_embedding` so the passages become vectors in the same pass:

```json
{
  "text_embedding": {
    "model_id": "<your deployed model id>",
    "field_map": { "passage_chunk": "passage_embedding" }
  }
}
```

`text_embedding` requires a model deployed in OpenSearch and takes `batch_size` (default `1`) and `skip_existing` (default `false`, reuses embeddings when the input text is unchanged). Because the chunk field is an array, the embedding field is mapped as `nested` with a `knn_vector` inside it, and you name only the field, not the full path, in `field_map`.

## What the text chunking processor leaves out

The passages `text_chunking` produces are plain strings. That is the whole shape of the output, and it has two consequences.

1. **No document path travels with the chunk.** Passage 7 of a 40-page agreement does not know it sits under `Master Services Agreement → Termination clauses → Early termination`. Retrieval returns "Either party may…" and nothing in the prompt says which party, which agreement, or which clause.
2. **You cannot filter on depth or page you never put in the index.** The processor writes text, not metadata. There is no `page`, no `depth`, no heading. `bool.filter` can only scope on fields that exist, so document-aware scoping has to come from somewhere upstream.

Cascading fixes boundaries. It does not fix either of these. A chunk that ends cleanly on a paragraph break is still a chunk with no address.

## Indexing chunksets: skip the processor, keep the pipeline

When PrimeCut has already produced chunksets, the splitting is done before the document reaches OpenSearch. Each chunkset is one document, its `to_embed` text is one passage, and its hierarchy is a set of real fields. OpenSearch has no "strategy: none" the way `semantic_text` does — the equivalent is simply to leave `text_chunking` out of the pipeline:

```json
PUT _ingest/pipeline/chunkset-ingest-pipeline
{
  "description": "Chunksets arrive pre-split; embed the text as-is, no text_chunking stage",
  "processors": [
    {
      "text_embedding": {
        "model_id": "<your deployed model id>",
        "field_map": { "text": "embedding" }
      }
    }
  ]
}
```

Index against that pipeline (`POST poma/_doc?pipeline=chunkset-ingest-pipeline`) and OpenSearch embeds one vector per chunkset, with `file_id`, `page`, `depth` and `chunkset_index` sitting beside it as `keyword`/`integer` fields. If you embed client-side instead — the Python path shown below — drop the pipeline entirely and index the vector directly.

This is also the answer for **AWS OpenSearch**: an Amazon OpenSearch Service domain runs the same distribution, so both shapes work there when the domain's engine version includes the processor and the ML plugin. Pre-split chunksets need neither, which makes them the lower-dependency option on a managed domain and on Serverless collections, where the feature surface differs.

## What "optimal chunks" means for OpenSearch, concretely

1. **One document per chunkset, self-explanatory alone.** A chunkset is a root-to-leaf path — `Master Services Agreement → Termination clauses → Early termination → "Either party may…"`. See [POMA chunksets](/learn/chunking/chunksets).
2. **Hierarchy as filterable fields.** `file_id` (keyword), `page` (integer), `depth` (integer) turn `bool.filter` into document-aware scoping.
3. **BM25-eligible text, always.** Keep the chunkset's `to_embed` text in an analyzed field so the `hybrid` query type has both a vector and a lexical signal to fuse.
4. **No overlap.** Overlap duplicates spans into both the HNSW graph and the inverted index. Chunksets need no overlap.
5. **Content-free index.** Store `{file_id, chunkset_index}` in the index and the actual content on a volume — smaller index, faster merges, citations reconstructed at retrieval time.

## The pipeline: PrimeCut to OpenSearch

```bash
pip install opensearch-py
```

```python
from poma import PrimeCut
from poma.vektoria import assemble, records_from_archive, Volume
from poma.embeddings import get_embedder
from opensearchpy import OpenSearch

# 1. Chunk any document and keep the .poma archive — the volume's source of truth.
client = PrimeCut()  # reads POMA_API_KEY
client.ingest("contract.pdf", download_dir="archives", filename="contract.poma")

embedder = get_embedder("local:BAAI/bge-small-en-v1.5")
vol = Volume("s3://your-bucket/poma")
osc = OpenSearch(hosts=[{"host": "localhost", "port": 9200}])

osc.indices.create(index="poma", body={"settings": {"index.knn": True},  # knn_vector fields need index.knn=true, otherwise the mapping is rejected (and ignore=400 hides it)
                                       "mappings": {"properties": {
    "embedding": {"type": "knn_vector", "dimension": embedder.dims,
                  "method": {"name": "hnsw", "space_type": "cosinesimil", "engine": "faiss"}},
    "file_id": {"type": "keyword"},
    "chunkset_index": {"type": "integer"},
    "text": {"type": "text"},
}}}, ignore=400)

# 2. Content-free ingest — vector index holds only routing metadata; content lives on the volume.
records = records_from_archive("archives/contract.poma")  # -> list[Record] (id, text, payload)
# assemble() reads the FULL document (chunks + chunksets) from the volume — write it once,
# not one {"chunks": <indices>, "text"} stub per record (which also overwrote itself each turn).
from poma.utils import unpack_poma_archive
_data = unpack_poma_archive(poma_archive_path="archives/contract.poma")
file_id = records[0].payload["file_id"]
vol.write_doc(file_id, {"file_id": file_id, "chunks": _data["chunks"], "chunksets": _data["chunksets"]})
for r in records:
    osc.index(index="poma", id=r.id, body={
        "embedding": embedder.embed([r.text])[0],
        "file_id": r.payload["file_id"],
        "chunkset_index": r.payload["chunkset_index"],
        "text": r.text,
    })

# 3. k-NN retrieval, then assemble prompt-ready context.
qv = embedder.embed(["early termination conditions"])[0]
osc.indices.refresh(index="poma")  # make the just-indexed docs searchable before querying
res = osc.search(index="poma", body={"query": {"knn": {"embedding": {
    "vector": qv, "k": 10}}}})
context = assemble(res, volume=vol)  # -> [{"file_id", "content"}, ...]
```

Retrieved chunksets share ancestor lineage across a document — `assemble()` deduplicates that lineage into one prompt-ready cheatsheet, the same discipline that answered one reference legal document with **337 tokens** instead of **1,542** for a recursive-splitter baseline. One document is an illustration, not a benchmark; the [ingestion guide](/document-ingestion-chunking-rag) has the broader numbers.

## Chunk shapes in OpenSearch, compared

| Property | Fixed 512-token chunks + overlap | Semantic chunks | **POMA chunksets** |
| --- | --- | --- | --- |
| Document is self-explanatory alone | ✗ | Sometimes | **✓ (by construction)** |
| Hierarchy as filterable fields | Manual | Manual | **✓ (`file_id`, `page`, `depth`)** |
| Redundant HNSW/BM25 entries | Many (overlap) | Few | **None** |
| Content-free index + volume | DIY | DIY | **✓ (`vektoria` `assemble`/`Volume`)** |
| Metadata returned on hits | Default | Default | **Default — no extra flag needed** |

## Frequently asked questions

### How does chunking work in OpenSearch?

Through the `text_chunking` ingest processor. Create a pipeline with `PUT _ingest/pipeline/<name>`, give the processor a `field_map` from an input field to an output field, and pick an `algorithm`: `fixed_token_length` (`token_limit` default `384`, `tokenizer` default `standard`, `overlap_rate` 0–0.5 default `0`), `fixed_char_length` (`char_limit` default `2048`) or `delimiter` (default `\n\n`). All three take `max_chunk_limit`, default `100`, `-1` to disable. The output field holds an array of passages, usually fed straight into `text_embedding`.

### What does the OpenSearch text chunking processor do?

It splits one long text field into an array of shorter passages at ingest. `field_map` names input and output; `algorithm` chooses how to split. Nested fields need a nested `field_map` object, not a dot path. When passages exceed `max_chunk_limit`, the excess text is appended to the last chunk rather than dropped. It adds no metadata: the passages are plain strings with no page, no heading, no path back into the document.

### What is the best OpenSearch chunking strategy for RAG?

The one that yields passages a model can read alone. Inside the processor, cascading a `delimiter` stage on `\n\n` into a `fixed_token_length` stage respects paragraphs before enforcing the token limit, and a third stage gives a recursive effect. That fixes boundaries, not context — the passages still carry no heading lineage and no filterable `page` or `depth`. Chunksets fix the other half: one document per chunkset in a `knn_vector` mapping, hierarchy as real fields, no `text_chunking` stage needed.

### Does AWS OpenSearch Service support chunking?

Amazon OpenSearch Service domains run the same OpenSearch distribution, so `PUT _ingest/pipeline` with `text_chunking` works when the domain's engine version includes the processor and the ML plugin it chains with. Check the version first, and note that Serverless collections have a different feature surface. Indexing chunksets sidesteps the question: the splitting already happened, so the domain only needs a `knn_vector` mapping.

### What is the optimal chunk size for OpenSearch?

There's no universal token count — chunksets are self-explanatory at any size, which is why they beat fixed 512-token windows. Start at 512 tokens if you must fix a size, but the size knob cannot fix missing context.

### How do I map chunksets into an OpenSearch index?

`embedding` as `knn_vector` (dimension matching your embedder, HNSW via `faiss`/`nmslib`), `file_id`/`page` as keyword/integer, `text` as an analyzed field for BM25 — one document per chunkset.

### Is OpenSearch the same as Elasticsearch for vector search?

Conceptually close but not identical — different field type (`knn_vector` vs `dense_vector`), different plugin architecture, and a distinct hybrid mechanism (search pipeline vs RRF `rank`). Check the schema against your specific version.

### How does POMA retrieve context from OpenSearch results?

`vektoria` keeps the index content-free (vector + `file_id`/`chunkset_index` only); content lives on a volume. `assemble(results, volume=VOL)` auto-detects OpenSearch's result shape and returns deduplicated cheatsheets — no extra metadata flag needed.

### How does chunk overlap affect an OpenSearch index?

It duplicates spans into both the HNSW graph and the BM25 inverted index — more memory, top-k crowded with near-duplicates. Chunksets need no overlap.

## Feed OpenSearch from the parser you already run

- [AWS Textract → OpenSearch](/pipelines/textract-to-opensearch)
- [Azure Document Intelligence → OpenSearch](/pipelines/azure-document-intelligence-to-opensearch)
- [Docling → OpenSearch](/pipelines/docling-to-opensearch)
- [LlamaParse → OpenSearch](/pipelines/llamaparse-to-opensearch)
- [Marker → OpenSearch](/pipelines/marker-to-opensearch)
- [Mistral OCR → OpenSearch](/pipelines/mistral-ocr-to-opensearch)
- [PaddleOCR-VL → OpenSearch](/pipelines/paddleocr-vl-to-opensearch)
- [Unstructured.io → OpenSearch](/pipelines/unstructured-to-opensearch)

Running a different store? The same chunk design applies: [Azure AI Search](/optimal-chunks-azure-ai-search) · [Chroma](/optimal-chunks-chroma) · [Elasticsearch](/optimal-chunks-elasticsearch) · [FAISS](/optimal-chunks-faiss) · [LanceDB](/optimal-chunks-lancedb) · [Milvus](/optimal-chunks-milvus) · [MongoDB Atlas](/optimal-chunks-mongodb-atlas) · [pgvector](/optimal-chunks-pgvector) · [Pinecone](/optimal-chunks-pinecone) · [Qdrant](/optimal-chunks-qdrant) · [Redis](/optimal-chunks-redis) · [Turbopuffer](/optimal-chunks-turbopuffer) · [Vespa](/optimal-chunks-vespa) · [Weaviate](/optimal-chunks-weaviate) — or browse [all pipeline recipes](/pipelines/).

Fundamentals: [RAG chunking guide](/guides/rag-chunking/) · [RAG architecture guide](/guides/rag-architecture/) · [POMA chunksets](/learn/chunking/chunksets).