Source: http://www.poma-ai.com/docs/pipelines/marker-to-opensearch

# The Missing Link Between Marker and Optimal Retrieval in OpenSearch

<ByAuthor />

**The short answer:** Marker gives you fast, local PDF parsing with a real Document block tree — per-page blocks, explicit types, and full HTML tables with spans intact, when you run its JSON renderer. OpenSearch gives you its own `knn_vector` field type and k-NN plugin (forked from Elasticsearch in 2021), plus a dedicated hybrid query type for fusing lexical and vector search. Wired together naively, Marker's page boundaries and table structure are gone before a document is indexed, and any lexical layer added on top ranks the resulting garbage tokens without the normalization OpenSearch's hybrid pipeline is built to provide. The missing link is POMA: `PrimeCut().ingest()` turns the saved Marker JSON into hierarchy-preserving [chunksets](/learn/chunking/chunksets), and `poma.vektoria` indexes them as content-free documents — `embedding`, `file_id`, `chunkset_index` — while the actual chunkset text lives on a volume. Retrieval runs OpenSearch's own `knn` query, then `assemble()` reconstructs prompt-ready cheatsheets.

## What each end of the pipeline actually provides

**Marker** (by Datalab / VikParuchuri) runs on your own hardware and turns PDFs into clean structured output. With the JSON renderer (`--output_format json`) you get a Document block tree — Page children, per-block HTML, and full `<table>` elements with row and column spans intact. What it doesn't provide: cross-page hierarchy, retrieval units, or inline images — figure bytes live in a side-channel `images` dict keyed by name, with only `![](name)` references left in the text. Details: [The Optimal Chunker for Marker](/optimal-chunker-marker).

**OpenSearch** provides `knn_vector` fields with HNSW-backed approximate k-NN via `nmslib` or `faiss`, native BM25 on any analyzed text field, and a dedicated `hybrid` query type that fuses lexical and vector sub-queries through a search pipeline's score normalization — a distinct mechanism from Elasticsearch's RRF `rank`, since the two projects diverged at Elasticsearch 7.10 in 2021. What it doesn't provide: any opinion about what a document should represent — a mapping whose fields hold a broken table fragment or a dangling image reference gets ranked exactly as confidently as a clean one. Details: [OpenSearch Chunking Strategy for RAG](/optimal-chunks-opensearch).

## The naive wiring, and where it breaks

The common recipe — concatenate Marker's per-page markdown into one string, `RecursiveCharacterTextSplitter`, embed, `osc.index()` the raw chunk text — breaks in ways specific to this pair:

- **Page boundaries disappear before a single document is indexed.** Marker's markdown output alone has no page delimiters, so a flatten-then-split pipeline can't populate a `page` field to filter or cite by — only the JSON renderer's Page blocks carry that boundary, and only if the ingest path reads them.
- **Tables get cut mid-row.** The JSON renderer's tables arrive as full `<table>` elements with colspans — real structure Marker recovered. A character splitter run over the flattened markdown routinely slices a table in half, leaving raw HTML tag fragments behind.
- **If a hybrid layer is added naively, unnormalized noise wins.** OpenSearch's hybrid query type requires a search pipeline to normalize BM25 and vector scores before combining them. Leftover `<td>`/`<tr>` tag text from a cut table, or dangling `![](name)` references, become unmatched or generic BM25 tokens; without normalization, an unbounded lexical score built on that noise can outrank a properly-scaled vector hit.
- **Every figure vanishes.** Marker's images live in the side-channel `images` dict; the markdown carries only `![](name)` references, which a markdown-only pipeline embeds as literal, unmatched text — the figure itself was never indexed.

## The pipeline, end to end

```bash
pip install opensearch-py

# Your existing Marker run — unchanged. The JSON renderer keeps the
# Document block tree that POMA auto-detects, tables and pages included.
marker_single contract.pdf --output_format json --output_dir out/
```

```python
from poma import PrimeCut
from poma.vektoria import assemble, records_from_archive, Volume
from poma.embeddings import get_embedder
from opensearchpy import OpenSearch

# 1. Chunk the raw Marker JSON and keep the .poma archive — the volume's source of truth.
client = PrimeCut()  # reads POMA_API_KEY
client.ingest("out/contract/contract.json", download_dir="archives", filename="contract.poma")

embedder = get_embedder("local:BAAI/bge-small-en-v1.5")
vol = Volume("s3://your-bucket/poma")
osc = OpenSearch(hosts=[{"host": "localhost", "port": 9200}])

osc.indices.create(index="poma", body={"mappings": {"properties": {
    "embedding": {"type": "knn_vector", "dimension": embedder.dims,
                  "method": {"name": "hnsw", "space_type": "cosinesimil", "engine": "faiss"}},
    "file_id": {"type": "keyword"},
    "chunkset_index": {"type": "integer"},
}}}, ignore=400)

# 2. Content-free ingest — OpenSearch holds only routing metadata; content lives on the volume.
records = records_from_archive("archives/contract.poma")  # -> list[Record] (id, text, payload)
for r in records:
    vol.write_doc(r.payload["file_id"], {"file_id": r.payload["file_id"],
                                          "chunks": r.payload["chunks"], "text": r.text})
    osc.index(index="poma", id=r.id, body={
        "embedding": embedder.embed([r.text])[0],
        "file_id": r.payload["file_id"],
        "chunkset_index": r.payload["chunkset_index"],
    })

# 3. Retrieval — OpenSearch's k-NN plugin query, then assemble.
qv = embedder.embed(["What are the early termination conditions?"])[0]
res = osc.search(index="poma", body={"query": {"knn": {"embedding": {
    "vector": qv, "k": 10}}}})
context = assemble(res, volume=vol)  # -> [{"file_id", "content"}, ...]
```

`records_from_archive` reads the `.poma` archive PrimeCut just wrote and returns one `Record` per chunkset: `.id` (deterministic `chunkset_uuid(file_id, chunkset_index)`), `.text` (the `to_embed` string used to build the embedding), and `.payload` (`file_id`, `chunkset_index`, `chunks`). The same `id` scheme means re-ingesting a document overwrites the same OpenSearch documents rather than duplicating them. On the reference legal-document benchmark this discipline answers with **337 tokens** of retrieved context instead of **1,542** for a recursive-splitter baseline — [methodology here](/document-ingestion-chunking-rag).

## Metadata mapping: POMA fields → OpenSearch primitives

| POMA field | Where it lives | OpenSearch role |
| --- | --- | --- |
| `to_embed` | embedded, written as the `embedding` field | `knn_vector` field for the k-NN plugin's ranking |
| `file_id` | document field (`keyword`) | filterable via `bool.filter`, returned by default |
| `chunkset_index` | document field (`integer`) + volume key | forms the deterministic `id`; also returned by default |
| `page`, `depth`, `chunks` (from Marker's Page blocks and block tree) | volume only (`record.payload`) | reconstructed into cheatsheet content, never indexed as an OpenSearch field |
| chunkset lineage | volume document | deduplicated and merged into one cheatsheet by `assemble()` |

## Frequently asked questions

### How do I get Marker output into OpenSearch for RAG?

Run Marker with the JSON renderer (`marker_single --output_format json`), hand the saved result to `PrimeCut().ingest()`, and keep the `.poma` archive. Turn each chunkset into a `Record` with `records_from_archive`, embed `record.text`, and `osc.index()` a content-free document (`embedding`, `file_id`, `chunkset_index`) per record. Retrieve with the k-NN plugin's `knn` query and pass the response to `assemble()`.

### Why not just split Marker's markdown and index it into OpenSearch directly?

Marker's markdown concatenates every page into one string with no boundaries, and leaves image bytes in a side-channel dict as dangling refs. A generic splitter also cuts the JSON renderer's full HTML tables mid-row — and if you add OpenSearch's hybrid query type on top without a normalization search pipeline, unbounded lexical scores from that leftover table markup can outweigh a properly normalized vector score.

### What OpenSearch mapping fields should Marker chunksets carry?

`embedding` as `knn_vector` (dimension matching your embedder, HNSW via `faiss`/`nmslib`), plus `file_id` (keyword) and `chunkset_index` (integer) as routing fields — the content-free contract `vektoria` writes by default.

### Do Marker's tables and images survive the trip to OpenSearch?

Yes, upstream of OpenSearch. PrimeCut splices Marker's side-channel `images` dict back into its references and describes each figure, and keeps full HTML tables intact through the chunk layer. OpenSearch itself never stores images or tables under this pattern — only the vector and two routing fields — and the reconstructed content comes back from the volume via `assemble()`.

### Does OpenSearch's hybrid query pipeline need extra configuration for Marker-sourced chunksets?

Only if you add lexical search on top of the plain `knn` query shown here. OpenSearch's hybrid query type needs a search pipeline to normalize BM25 and vector scores — a distinct mechanism from Elasticsearch's RRF `rank` — and skipping it lets unnormalized noise from any leftover table markup outrank a properly-scaled vector hit. The plain `knn` retrieval above needs no such step and returns `file_id`/`chunkset_index` by default.

## Related recipes

Same parser, different store: [Marker → Qdrant](/pipelines/marker-to-qdrant) · [Marker → Weaviate](/pipelines/marker-to-weaviate) · [Marker → Chroma](/pipelines/marker-to-chroma)

Foundations: [The Optimal Chunker for Marker](/optimal-chunker-marker) · [OpenSearch Chunking Strategy for RAG](/optimal-chunks-opensearch) · [All pipeline recipes](/pipelines/) · [RAG chunking guide](/guides/rag-chunking/)