Source: http://www.poma-ai.com/docs/pipelines/marker-to-elasticsearch

# The Missing Link Between Marker and Optimal Retrieval in Elasticsearch

<ByAuthor />

**The short answer:** Marker gives you fast, local PDF parsing with a real Document block tree — per-page blocks, explicit types, and full HTML tables with spans intact, when you run its JSON renderer. Elasticsearch gives you mature hybrid search, with `dense_vector` k-NN and native BM25 in a single request. Wired together naively, Marker's page boundaries and table structure are gone before a document is indexed, and Elasticsearch ranks the resulting broken fragments with full confidence. The missing link is POMA: `PrimeCut().ingest()` turns the saved Marker JSON into hierarchy-preserving [chunksets](/learn/chunking/chunksets), and `poma.vektoria` indexes them as content-free documents — `embedding`, `file_id`, `chunkset_index` — while the actual chunkset text lives on a volume. Retrieval runs Elasticsearch's own `knn` query, then `assemble()` reconstructs prompt-ready cheatsheets.

## What each end of the pipeline actually provides

**Marker** (by Datalab / VikParuchuri) runs on your own hardware and turns PDFs into clean structured output. With the JSON renderer (`--output_format json`) you get a Document block tree — Page children, per-block HTML, and full `<table>` elements with row and column spans intact. What it doesn't provide: cross-page hierarchy, retrieval units, or inline images — figure bytes live in a side-channel `images` dict keyed by name, with only `![](name)` references left in the text. Details: [The Optimal Chunker for Marker](/optimal-chunker-marker).

**Elasticsearch** provides `dense_vector` fields with HNSW-backed approximate k-NN, native BM25 on any analyzed text field, a single request combining `knn` with a `bool`/`match` query, and `rank` (reciprocal rank fusion) to merge them server-side. What it doesn't provide: any opinion about what a document should represent — a mapping whose fields hold a broken table fragment or a dangling image reference gets ranked exactly as confidently as a clean one. Details: [Elasticsearch Chunking Strategy for RAG](/optimal-chunks-elasticsearch).

## The naive wiring, and where it breaks

The common recipe — concatenate Marker's per-page markdown into one string, `RecursiveCharacterTextSplitter`, embed, `es.index()` the raw chunk text — breaks in ways specific to this pair:

- **Page boundaries disappear before a single document is indexed.** Marker's markdown output alone has no page delimiters, so a flatten-then-split pipeline can't populate a `page` field to filter or cite by — only the JSON renderer's Page blocks carry that boundary, and only if the ingest path reads them.
- **Tables get cut mid-row.** The JSON renderer's tables arrive as full `<table>` elements with colspans — real structure Marker recovered. A character splitter run over the flattened markdown routinely slices a table in half; Elasticsearch then indexes and ranks half a `<tr>` as if it were complete.
- **Every figure vanishes.** Marker's images live in the side-channel `images` dict; the markdown carries only `![](name)` references, which a markdown-only pipeline embeds as literal, unmatched text — the figure itself was never indexed.
- **Overlap inflates the index.** Splitter overlap duplicates every boundary span into both the HNSW graph and the BM25 inverted index, crowding `knn` results with near-duplicates of the same passage.
- **Heading levels are discarded**, so a retrieved fragment arrives without lineage and the LLM answers out of context — the failure Marker's parsing quality can't fix.

## The pipeline, end to end

```bash
pip install elasticsearch

# Your existing Marker run — unchanged. The JSON renderer keeps the
# Document block tree that POMA auto-detects, tables and pages included.
marker_single contract.pdf --output_format json --output_dir out/
```

```python
from poma import PrimeCut
from poma.vektoria import assemble, records_from_archive, Volume
from poma.embeddings import get_embedder
from elasticsearch import Elasticsearch

# 1. Chunk the raw Marker JSON and keep the .poma archive — the volume's source of truth.
client = PrimeCut()  # reads POMA_API_KEY
client.ingest("out/contract/contract.json", download_dir="archives", filename="contract.poma")

embedder = get_embedder("local:BAAI/bge-small-en-v1.5")
vol = Volume("s3://your-bucket/poma")
es = Elasticsearch("https://localhost:9200")

es.indices.create(index="poma", mappings={"properties": {
    "embedding": {"type": "dense_vector", "dims": embedder.dims, "index": True, "similarity": "cosine"},
    "file_id": {"type": "keyword"},
    "chunkset_index": {"type": "integer"},
}}, ignore=400)

# 2. Content-free ingest — Elasticsearch holds only routing metadata; content lives on the volume.
records = records_from_archive("archives/contract.poma")  # -> list[Record] (id, text, payload)
for r in records:
    vol.write_doc(r.payload["file_id"], {"file_id": r.payload["file_id"],
                                          "chunks": r.payload["chunks"], "text": r.text})
    es.index(index="poma", id=r.id, document={
        "embedding": embedder.embed([r.text])[0],
        "file_id": r.payload["file_id"],
        "chunkset_index": r.payload["chunkset_index"],
    })

# 3. Retrieval — Elasticsearch's normal knn query, then assemble.
qv = embedder.embed(["What are the early termination conditions?"])[0]
res = es.search(index="poma", knn={"field": "embedding", "query_vector": qv,
                                    "k": 10, "num_candidates": 100})
context = assemble(res, volume=vol)  # -> [{"file_id", "content"}, ...]
```

`records_from_archive` reads the `.poma` archive PrimeCut just wrote and returns one `Record` per chunkset: `.id` (deterministic `chunkset_uuid(file_id, chunkset_index)`), `.text` (the `to_embed` string used to build the embedding), and `.payload` (`file_id`, `chunkset_index`, `chunks`). The same `id` scheme means re-ingesting a document overwrites the same Elasticsearch documents rather than duplicating them. On the reference legal-document benchmark this discipline answers with **337 tokens** of retrieved context instead of **1,542** for a recursive-splitter baseline — [methodology here](/document-ingestion-chunking-rag).

## Metadata mapping: POMA fields → Elasticsearch primitives

| POMA field | Where it lives | Elasticsearch role |
| --- | --- | --- |
| `to_embed` | embedded, written as the `embedding` field | `dense_vector` field for `knn` ranking |
| `file_id` | document field (`keyword`) | filterable via `bool.filter`, returned in `_source` by default |
| `chunkset_index` | document field (`integer`) + volume key | forms the deterministic `id`; also returned in `_source` by default |
| `page`, `depth`, `chunks` (from Marker's Page blocks and block tree) | volume only (`record.payload`) | reconstructed into cheatsheet content, never indexed as an Elasticsearch field |
| chunkset lineage | volume document | deduplicated and merged into one cheatsheet by `assemble()` |

## Frequently asked questions

### How do I get Marker output into Elasticsearch for RAG?

Run Marker with the JSON renderer (`marker_single --output_format json`), hand the saved result to `PrimeCut().ingest()`, and keep the `.poma` archive. Turn each chunkset into a `Record` with `records_from_archive`, embed `record.text`, and `es.index()` a content-free document (`embedding`, `file_id`, `chunkset_index`) per record. Retrieve with `es.search(knn=...)` and pass the response to `assemble()`.

### Why not just split Marker's markdown and index it into Elasticsearch directly?

Marker's markdown concatenates every page into one string with no boundaries, and leaves image bytes in a side-channel dict as dangling refs. A generic splitter run over that flat string also cuts the JSON renderer's full HTML tables mid-row, and Elasticsearch ranks the resulting broken fragment with full confidence — nothing in the mapping flags it as truncated.

### What Elasticsearch mapping fields should Marker chunksets carry?

`embedding` as `dense_vector` (dims matching your embedder, `similarity: cosine`), plus `file_id` (keyword) and `chunkset_index` (integer) as routing fields — the content-free contract `vektoria` writes by default.

### Do Marker's tables and images survive the trip to Elasticsearch?

Yes, upstream of Elasticsearch. PrimeCut splices Marker's side-channel `images` dict back into its references and describes each figure, and keeps full HTML tables intact through the chunk layer. Elasticsearch itself never stores images or tables under this pattern — only the vector and two routing fields — and the reconstructed content comes back from the volume via `assemble()`.

### Does Elasticsearch retrieval need extra fields for POMA's assemble() to work?

No extra ask needed — Elasticsearch returns the full `_source` by default on a `knn` search, so `file_id` and `chunkset_index` come back on every hit without an explicit projection.

## Related recipes

Same parser, different store: [Marker → Qdrant](/pipelines/marker-to-qdrant) · [Marker → Pinecone](/pipelines/marker-to-pinecone) · [Marker → pgvector](/pipelines/marker-to-pgvector)

Foundations: [The Optimal Chunker for Marker](/optimal-chunker-marker) · [Elasticsearch Chunking Strategy for RAG](/optimal-chunks-elasticsearch) · [All pipeline recipes](/pipelines/) · [RAG chunking guide](/guides/rag-chunking/)