Unstructured.io to Elasticsearch: Chunking for RAG
The short answer: Unstructured.io gives you a typed, ordered element list; Elasticsearch gives you mature dense_vector k-NN fused with two decades of BM25. Wired together with chunk_by_title and an unfiltered text field, they still leak the hierarchy Unstructured recovered — and Unstructured's own running-header elements pollute Elasticsearch's lexical scoring if you don't drop them first. The missing link is POMA: PrimeCut().ingest() consumes the raw element list, drops Header/Footer/PageNumber noise, emits hierarchy-preserving chunksets into a portable .poma archive, and poma.vektoria indexes only (vector, file_id, chunkset_index) while content lives on a volume. Retrieval runs Elasticsearch's own knn query, then assemble() reassembles prompt-ready cheatsheets.
What each end of the pipeline actually provides
Unstructured.io (partition_pdf or the hosted API) returns a flat, ordered list of typed elements — Title, NarrativeText, ListItem, Table, Image, plus Header/Footer/PageNumber — each with element_id, text, and metadata (page_number, text_as_html for tables, optionally image_base64). Element order is document order; nothing records heading depth. Details: The Optimal Chunker for Unstructured.io.
Elasticsearch provides dense_vector fields for approximate k-NN, native BM25 on every analyzed text field, and a single request that combines knn with bool/match, fused server-side via rank. None of that decides what a document should represent — it ranks whatever you indexed. Details: Elasticsearch Chunking Strategy for RAG.
The naive wiring, and where it breaks
The common recipe — run chunk_by_title over the raw element list without filtering by type, embed each fragment, index it with the fragment text in an analyzed field — breaks in a pair-specific way:
chunk_by_titledoesn't filterHeader/Footer/PageNumberelements out. They ride along into whatever fragment happens to contain them, and once indexed as analyzed text, a running header repeated on every page becomes a high-document-frequency term that skews Elasticsearch's corpus-wide BM25 idf weighting away from the substantive content a query is actually looking for.- A
Tableelement'stext_as_htmlcan split mid-fragment. Two Elasticsearch documents end up sharing afile_idbut no lineage field, so abool.filterscoped to one document can return half a table with no way to know the other half exists. - Overlap, if still applied, duplicates spans into both the HNSW graph and the inverted index — larger index, slower merges,
top_kcrowded with near-duplicates.
The pipeline, end to end
pip install unstructured elasticsearchfrom unstructured.partition.pdf import partition_pdf
from unstructured.staging.base import elements_to_json
from poma import PrimeCut
from poma.vektoria import assemble, records_from_archive, Volume
from poma.embeddings import get_embedder
from elasticsearch import Elasticsearch
# 1. Your existing Unstructured call — unchanged.
elements = partition_pdf(
filename="contract.pdf",
strategy="hi_res",
infer_table_structure=True, # populates metadata.text_as_html
extract_image_block_types=["Image"],
extract_image_block_to_payload=True, # inline base64 → POMA can describe figures
)
elements_to_json(elements, filename="contract.unstructured.json")
# 2. The missing link — raw element list in, a portable .poma archive out.
client = PrimeCut() # reads POMA_API_KEY
client.ingest("contract.unstructured.json", download_dir="archives", filename="contract.poma")
embedder = get_embedder("local:BAAI/bge-small-en-v1.5")
vol = Volume("s3://your-bucket/poma")
es = Elasticsearch("http://localhost:9200") # local dev cluster (xpack.security.enabled=false); a secured cluster needs https + basic_auth/api_key + ca_certs
es.indices.create(index="poma", mappings={"properties": {
"embedding": {"type": "dense_vector", "dims": embedder.dims, "index": True, "similarity": "cosine"},
"file_id": {"type": "keyword"},
"chunkset_index": {"type": "integer"},
}}, ignore=400)
# 3. Content-free ingest — the index holds only routing metadata; content lives on the volume.
records = records_from_archive("archives/contract.poma") # -> list[Record] (id, text, payload)
# assemble() reads the FULL document (chunks + chunksets) from the volume — write it once,
# not one {"chunks": <indices>, "text"} stub per record (which also overwrote itself each turn).
from poma.utils import unpack_poma_archive
_data = unpack_poma_archive(poma_archive_path="archives/contract.poma")
file_id = records[0].payload["file_id"]
vol.write_doc(file_id, {"file_id": file_id, "chunks": _data["chunks"], "chunksets": _data["chunksets"]})
for r in records:
es.index(index="poma", id=r.id, document={
"embedding": embedder.embed([r.text])[0],
"file_id": r.payload["file_id"],
"chunkset_index": r.payload["chunkset_index"],
})
# 4. Retrieval — Elasticsearch's normal knn query, then assemble.
qv = embedder.embed(["early termination conditions"])[0]
es.indices.refresh(index="poma") # make the just-indexed docs searchable before querying
res = es.search(index="poma", knn={"field": "embedding", "query_vector": qv,
"k": 10, "num_candidates": 100})
context = assemble(res.body, volume=vol) # elasticsearch-py 8+ wraps the response (ObjectApiResponse) — pass the dict body # -> [{"file_id", "content"}, ...]Because Header, Footer, and PageNumber elements are dropped before chunking (not after indexing), the fragmentation-driven BM25 pollution described above never reaches Elasticsearch in the first place — POMA removes the noise upstream of the mapping decision. _source returns file_id/chunkset_index by default, so assemble() needs no extra request flag.
Metadata mapping: POMA fields → Elasticsearch primitives
| POMA field (Unstructured-sourced) | Elasticsearch primitive | Notes |
|---|---|---|
chunkset id (chunkset_uuid) | document _id | stable across re-indexing |
to_embed (tables spliced as text_as_html, headers/footers dropped) | dense_vector field | embedded before index, never stored as analyzed text |
file_id | keyword field | returned in _source by default for assemble() |
chunkset_index | integer field | returned in _source by default for assemble() |
page (from metadata.page_number), depth, chunk_index, full chunkset text | volume document | content-free — never indexed, fetched by assemble() |
Frequently asked questions
How do I get Unstructured.io elements into Elasticsearch for RAG?
Save the element list with elements_to_json, ingest it with PrimeCut, and keep the .poma archive. Create a dense_vector mapping with file_id/chunkset_index, index documents from records_from_archive, write chunkset content to a volume, then retrieve with es.search(knn=...) plus assemble(res, volume=vol) — no extra metadata flag needed.
Why does chunk_by_title pollute Elasticsearch's BM25 ranking with header and footer noise?
chunk_by_title doesn't filter element types, so repeated Header/Footer/PageNumber text rides into analyzed fields and inflates corpus-wide document frequency, diluting the idf weight of substantive query terms.
What Elasticsearch mapping fields should Unstructured chunks carry?
embedding as dense_vector (dims matching your embedder, similarity: cosine), file_id as keyword, chunkset_index as integer. Page, depth, chunk index, and full text stay off the document and live on a volume.
Do Unstructured's inline images survive into Elasticsearch retrieval?
Yes, if extract_image_block_to_payload=true was set — POMA describes the figure and the description becomes ordinary embedded, volume-stored text. Disk-based image_path images are neutralized and counted, never silently dropped.
Does Elasticsearch's knn plus bool hybrid query need extra flags for Unstructured metadata?
No — _source returns file_id/chunkset_index by default, unlike Pinecone or Milvus which need an explicit ask. The content-free pattern keeps _source small regardless.
Related recipes
Same parser, different store: Unstructured.io → OpenSearch · Unstructured.io → Turbopuffer · Unstructured.io → Qdrant
Foundations: The Optimal Chunker for Unstructured.io · Elasticsearch Chunking Strategy for RAG · All pipeline recipes · RAG chunking guide