Source: http://www.poma-ai.com/docs/pipelines/unstructured-to-elasticsearch

# The Missing Link Between Unstructured.io and Optimal Retrieval in Elasticsearch

<ByAuthor />

**The short answer:** Unstructured.io gives you a typed, ordered element list; Elasticsearch gives you mature `dense_vector` k-NN fused with two decades of BM25. Wired together with `chunk_by_title` and an unfiltered text field, they still leak the hierarchy Unstructured recovered — and Unstructured's own running-header elements pollute Elasticsearch's lexical scoring if you don't drop them first. The missing link is POMA: `PrimeCut().ingest()` consumes the raw element list, drops `Header`/`Footer`/`PageNumber` noise, emits hierarchy-preserving [chunksets](/learn/chunking/chunksets) into a portable `.poma` archive, and `poma.vektoria` indexes only `(vector, file_id, chunkset_index)` while content lives on a volume. Retrieval runs Elasticsearch's own `knn` query, then `assemble()` reassembles prompt-ready cheatsheets.

## What each end of the pipeline actually provides

**Unstructured.io** (`partition_pdf` or the hosted API) returns a flat, ordered list of typed elements — `Title`, `NarrativeText`, `ListItem`, `Table`, `Image`, plus `Header`/`Footer`/`PageNumber` — each with `element_id`, `text`, and `metadata` (`page_number`, `text_as_html` for tables, optionally `image_base64`). Element order is document order; nothing records heading depth. Details: [The Optimal Chunker for Unstructured.io](/optimal-chunker-unstructured).

**Elasticsearch** provides `dense_vector` fields for approximate k-NN, native BM25 on every analyzed text field, and a single request that combines `knn` with `bool`/`match`, fused server-side via `rank`. None of that decides what a document should represent — it ranks whatever you indexed. Details: [Elasticsearch Chunking Strategy for RAG](/optimal-chunks-elasticsearch).

## The naive wiring, and where it breaks

The common recipe — run `chunk_by_title` over the raw element list without filtering by type, embed each fragment, index it with the fragment text in an analyzed field — breaks in a pair-specific way:

- **`chunk_by_title` doesn't filter `Header`/`Footer`/`PageNumber` elements out.** They ride along into whatever fragment happens to contain them, and once indexed as analyzed text, a running header repeated on every page becomes a high-document-frequency term that skews Elasticsearch's corpus-wide BM25 idf weighting away from the substantive content a query is actually looking for.
- **A `Table` element's `text_as_html` can split mid-fragment.** Two Elasticsearch documents end up sharing a `file_id` but no lineage field, so a `bool.filter` scoped to one document can return half a table with no way to know the other half exists.
- **Overlap, if still applied, duplicates spans into both the HNSW graph and the inverted index** — larger index, slower merges, `top_k` crowded with near-duplicates.

## The pipeline, end to end

```bash
pip install unstructured elasticsearch
```

```python
from unstructured.partition.pdf import partition_pdf
from unstructured.staging.base import elements_to_json
from poma import PrimeCut
from poma.vektoria import assemble, records_from_archive, Volume
from poma.embeddings import get_embedder
from elasticsearch import Elasticsearch

# 1. Your existing Unstructured call — unchanged.
elements = partition_pdf(
    filename="contract.pdf",
    strategy="hi_res",
    infer_table_structure=True,            # populates metadata.text_as_html
    extract_image_block_types=["Image"],
    extract_image_block_to_payload=True,   # inline base64 → POMA can describe figures
)
elements_to_json(elements, filename="contract.unstructured.json")

# 2. The missing link — raw element list in, a portable .poma archive out.
client = PrimeCut()  # reads POMA_API_KEY
client.ingest("contract.unstructured.json", download_dir="archives", filename="contract.poma")

embedder = get_embedder("local:BAAI/bge-small-en-v1.5")
vol = Volume("s3://your-bucket/poma")
es = Elasticsearch("https://localhost:9200")

es.indices.create(index="poma", mappings={"properties": {
    "embedding": {"type": "dense_vector", "dims": embedder.dims, "index": True, "similarity": "cosine"},
    "file_id": {"type": "keyword"},
    "chunkset_index": {"type": "integer"},
}}, ignore=400)

# 3. Content-free ingest — the index holds only routing metadata; content lives on the volume.
records = records_from_archive("archives/contract.poma")  # -> list[Record] (id, text, payload)
for r in records:
    vol.write_doc(r.payload["file_id"], {"file_id": r.payload["file_id"],
                                          "chunks": r.payload["chunks"], "text": r.text})
    es.index(index="poma", id=r.id, document={
        "embedding": embedder.embed([r.text])[0],
        "file_id": r.payload["file_id"],
        "chunkset_index": r.payload["chunkset_index"],
    })

# 4. Retrieval — Elasticsearch's normal knn query, then assemble.
qv = embedder.embed(["early termination conditions"])[0]
res = es.search(index="poma", knn={"field": "embedding", "query_vector": qv,
                                    "k": 10, "num_candidates": 100})
context = assemble(res, volume=vol)  # -> [{"file_id", "content"}, ...]
```

Because `Header`, `Footer`, and `PageNumber` elements are dropped before chunking (not after indexing), the fragmentation-driven BM25 pollution described above never reaches Elasticsearch in the first place — POMA removes the noise upstream of the mapping decision. `_source` returns `file_id`/`chunkset_index` by default, so `assemble()` needs no extra request flag.

## Metadata mapping: POMA fields → Elasticsearch primitives

| POMA field (Unstructured-sourced) | Elasticsearch primitive | Notes |
| --- | --- | --- |
| chunkset id (`chunkset_uuid`) | document `_id` | stable across re-indexing |
| `to_embed` (tables spliced as `text_as_html`, headers/footers dropped) | `dense_vector` field | embedded before index, never stored as analyzed text |
| `file_id` | `keyword` field | returned in `_source` by default for `assemble()` |
| `chunkset_index` | `integer` field | returned in `_source` by default for `assemble()` |
| `page` (from `metadata.page_number`), `depth`, `chunk_index`, full chunkset text | volume document | content-free — never indexed, fetched by `assemble()` |

## Frequently asked questions

### How do I get Unstructured.io elements into Elasticsearch for RAG?

Save the element list with `elements_to_json`, ingest it with `PrimeCut`, and keep the `.poma` archive. Create a `dense_vector` mapping with `file_id`/`chunkset_index`, index documents from `records_from_archive`, write chunkset content to a volume, then retrieve with `es.search(knn=...)` plus `assemble(res, volume=vol)` — no extra metadata flag needed.

### Why does chunk_by_title pollute Elasticsearch's BM25 ranking with header and footer noise?

`chunk_by_title` doesn't filter element types, so repeated `Header`/`Footer`/`PageNumber` text rides into analyzed fields and inflates corpus-wide document frequency, diluting the idf weight of substantive query terms.

### What Elasticsearch mapping fields should Unstructured chunks carry?

`embedding` as `dense_vector` (dims matching your embedder, `similarity: cosine`), `file_id` as `keyword`, `chunkset_index` as `integer`. Page, depth, chunk index, and full text stay off the document and live on a volume.

### Do Unstructured's inline images survive into Elasticsearch retrieval?

Yes, if `extract_image_block_to_payload=true` was set — POMA describes the figure and the description becomes ordinary embedded, volume-stored text. Disk-based `image_path` images are neutralized and counted, never silently dropped.

### Does Elasticsearch's knn plus bool hybrid query need extra flags for Unstructured metadata?

No — `_source` returns `file_id`/`chunkset_index` by default, unlike Pinecone or Milvus which need an explicit ask. The content-free pattern keeps `_source` small regardless.

## Related recipes

Same parser, different store: [Unstructured.io → OpenSearch](/pipelines/unstructured-to-opensearch) · [Unstructured.io → Turbopuffer](/pipelines/unstructured-to-turbopuffer) · [Unstructured.io → Qdrant](/pipelines/unstructured-to-qdrant)

Foundations: [The Optimal Chunker for Unstructured.io](/optimal-chunker-unstructured) · [Elasticsearch Chunking Strategy for RAG](/optimal-chunks-elasticsearch) · [All pipeline recipes](/pipelines/) · [RAG chunking guide](/guides/rag-chunking/)