Source: http://www.poma-ai.com/docs/pipelines/llamaparse-to-elasticsearch

# The Missing Link Between LlamaParse and Optimal Retrieval in Elasticsearch

<ByAuthor />

**The short answer:** LlamaParse gives you excellent per-page markdown; Elasticsearch gives you two decades of full-text search maturity plus `dense_vector` k-NN in the same request. Wired together naively — flatten, split, embed — they still produce mediocre RAG, because nothing in between rebuilds the document's hierarchy or keeps the index content-free. The missing link is POMA: `PrimeCut().ingest()` consumes the raw LlamaParse JSON (auto-detected), emits hierarchy-preserving [chunksets](/learn/chunking/chunksets), and `poma.vektoria` indexes them as content-free documents — vector plus `file_id`/`chunkset_index` only — with chunkset content on a volume, not in `_source`. Retrieval runs a normal `knn` search, then `assemble()` turns the hits into prompt-ready cheatsheets.

## What each end of the pipeline actually provides

**LlamaParse** returns `pages[]`, each with a `page` number, an `md` string (markdown, tables inline — POMA prefers this over `text`, its plain-flattened sibling), and an `images` list. Image bytes are not in the payload: they live server-side at LlamaParse behind a separate `/result/image/{name}` fetch, so a saved result JSON carries only `![](name)` references. What it doesn't provide: cross-page hierarchy or retrieval units. Details: [The Optimal Chunker for LlamaParse](/optimal-chunker-llamaparse).

**Elasticsearch** provides `dense_vector` fields for HNSW-backed k-NN, native BM25 on any analyzed text field, one request combining both via `knn` + `bool`/`match`, and mature `bool.filter` scoping. What it doesn't provide: any opinion about what a document should represent, or whether the text feeding BM25 is a structured chunkset or a dead image reference. Details: [Elasticsearch Chunking Strategy for RAG](/optimal-chunks-elasticsearch).

## The naive wiring, and where it breaks

The common recipe — `"\n".join(p["md"] for p in pages)` → fixed-size splitter → embed → `es.index(...)` per fragment — breaks in a pair-specific way:

- **Embedder choice becomes a one-way door.** A `dense_vector` field's `dims` are fixed at mapping time. Pick an embedder against LlamaParse's `md` text without planning ahead, and switching later means a whole new index and a full reindex — no in-place resize.
- **Splitter overlap inflates the HNSW graph.** Every overlapped span becomes its own document, and `top_k` fills with near-duplicates of the hit that actually answers the query.
- **Dead image references leak into the analyzed text field.** LlamaParse's `![](name)` refs, unneutralized, sit in the field BM25 scores — junk tokens ranking alongside real prose in the same hybrid query.

## The pipeline, end to end

```bash
pip install llama-parse elasticsearch
```

```python
import json
import os
from llama_parse import LlamaParse
from poma import PrimeCut
from poma.vektoria import assemble, records_from_archive, Volume
from poma.embeddings import get_embedder
from elasticsearch import Elasticsearch

# 1. LlamaParse — your existing call, unchanged.
parser = LlamaParse(api_key=os.environ["LLAMA_CLOUD_API_KEY"])
json_result = parser.get_json_result("contract.pdf")
with open("contract.llamaparse.json", "w") as f:
    json.dump(json_result[0], f)

# 2. The missing link — raw LlamaParse JSON in, a portable .poma archive out.
client = PrimeCut()  # reads POMA_API_KEY
client.ingest("contract.llamaparse.json", download_dir="archives", filename="contract.poma")

embedder = get_embedder("local:BAAI/bge-small-en-v1.5")
vol = Volume("s3://your-bucket/poma")
es = Elasticsearch("https://localhost:9200")

es.indices.create(index="poma", mappings={"properties": {
    "embedding": {"type": "dense_vector", "dims": embedder.dims, "index": True, "similarity": "cosine"},
    "file_id": {"type": "keyword"},
    "chunkset_index": {"type": "integer"},
}}, ignore=400)

# 3. Content-free ingest — index holds only routing metadata; content lives on the volume.
records = records_from_archive("archives/contract.poma")  # -> list[Record] (id, text, payload)
for r in records:
    vol.write_doc(r.payload["file_id"], {"file_id": r.payload["file_id"],
                                          "chunks": r.payload["chunks"], "text": r.text})
    es.index(index="poma", id=r.id, document={
        "embedding": embedder.embed([r.text])[0],
        "file_id": r.payload["file_id"],
        "chunkset_index": r.payload["chunkset_index"],
    })

# 4. Retrieval — Elasticsearch returns _source by default, then assemble.
qv = embedder.embed(["early termination conditions"])[0]
res = es.search(index="poma", knn={"field": "embedding", "query_vector": qv,
                                    "k": 10, "num_candidates": 100})
context = assemble(res, volume=vol)  # -> [{"file_id", "content"}, ...]
```

POMA validates the payload shape up front (fingerprinting `pages[]` with `md` + `page`; a corrupted or mislabeled upload 422s immediately), reads `md` over `text`, neutralizes offloaded image refs (counted in `content_metadata`), strips running headers/footers, and rebuilds the cross-page heading tree before chunking — the same treatment described in [The Optimal Chunker for LlamaParse](/optimal-chunker-llamaparse). `records_from_archive` then hands back one `Record` per chunkset, each with a deterministic `id` (`chunkset_uuid(file_id, chunkset_index)`) so the same chunkset always indexes to the same document.

## Metadata mapping: POMA fields → Elasticsearch primitives

| POMA chunk field | Elasticsearch primitive | What it enables |
| --- | --- | --- |
| `to_embed` | `dense_vector` field | `knn` ranking against the query embedding |
| `file_id` | `keyword` field | `bool.filter` scoping per document |
| `chunkset_index` (+ `file_id`) | deterministic document `_id` (`chunkset_uuid`) | same chunkset always indexes to the same document, safe to re-run |
| `chunks` (member list, incl. page/depth) | stored on the volume, not `_source` | reconstructed into cheatsheets by `assemble()` |
| LlamaParse `page` (via chunk lineage) | volume content, never in the mapping | page-cited answers assembled after retrieval, not queried |

## Frequently asked questions

### How do I get LlamaParse results into Elasticsearch for RAG?

Save the raw LlamaParse JSON, run `PrimeCut().ingest()` on it to get a `.poma` archive, then `records_from_archive()` + your own embedder to index content-free documents (`file_id`, `chunkset_index`, `embedding`). Retrieve with a normal `knn` search and pass the response to `assemble()`.

### Why not just concatenate LlamaParse's markdown and bulk-index it into Elasticsearch?

Concatenation discards LlamaParse's page numbers and heading levels before a document is ever written, and splitter overlap duplicates spans in the `dense_vector` index, crowding `top_k` with near-duplicates. Elasticsearch then ranks whatever context-free fragments it was given.

### What Elasticsearch fields should LlamaParse chunksets carry?

`file_id` as a keyword field and `chunkset_index` as an integer, plus `embedding` as a `dense_vector` field — the full content-free contract. Chunkset content, including LlamaParse's page numbers, lives on the volume instead of `_source`.

### Do LlamaParse's images survive the trip to Elasticsearch?

Not as bytes — LlamaParse keeps images server-side, so a saved result JSON has only `![](name)` references. POMA neutralizes these dead refs and counts each in `content_metadata` — visible, quantified loss, never silent junk in an indexed document.

### Can I change the embedding model after mapping a dense_vector field?

Not on the same field — `dims` are fixed at mapping time. Switching embedders after LlamaParse chunksets are already indexed means a new index with the new dims and a full reindex; there's no in-place resize.

## Related recipes

Same parser, different store: [LlamaParse → Pinecone](/pipelines/llamaparse-to-pinecone) · [LlamaParse → pgvector](/pipelines/llamaparse-to-pgvector) · [LlamaParse → Weaviate](/pipelines/llamaparse-to-weaviate)

Foundations: [The Optimal Chunker for LlamaParse](/optimal-chunker-llamaparse) · [Elasticsearch Chunking Strategy for RAG](/optimal-chunks-elasticsearch) · [All pipeline recipes](/pipelines/) · [RAG architecture guide](/guides/rag-architecture/)