Skip to content

Unstructured.io to Elasticsearch: Chunking for RAG ​

The short answer: Unstructured.io gives you a typed, ordered element list; Elasticsearch gives you mature dense_vector k-NN fused with two decades of BM25. Wired together with chunk_by_title and an unfiltered text field, they still leak the hierarchy Unstructured recovered — and Unstructured's own running-header elements pollute Elasticsearch's lexical scoring if you don't drop them first. The missing link is POMA: PrimeCut().ingest() consumes the raw element list, drops Header/Footer/PageNumber noise, emits hierarchy-preserving chunksets into a portable .poma archive, and poma.vektoria indexes only (vector, file_id, chunkset_index) while content lives on a volume. Retrieval runs Elasticsearch's own knn query, then assemble() reassembles prompt-ready cheatsheets.

What each end of the pipeline actually provides ​

Unstructured.io (partition_pdf or the hosted API) returns a flat, ordered list of typed elements — Title, NarrativeText, ListItem, Table, Image, plus Header/Footer/PageNumber — each with element_id, text, and metadata (page_number, text_as_html for tables, optionally image_base64). Element order is document order; nothing records heading depth. Details: The Optimal Chunker for Unstructured.io.

Elasticsearch provides dense_vector fields for approximate k-NN, native BM25 on every analyzed text field, and a single request that combines knn with bool/match, fused server-side via rank. None of that decides what a document should represent — it ranks whatever you indexed. Details: Elasticsearch Chunking Strategy for RAG.

The naive wiring, and where it breaks ​

The common recipe — run chunk_by_title over the raw element list without filtering by type, embed each fragment, index it with the fragment text in an analyzed field — breaks in a pair-specific way:

  • chunk_by_title doesn't filter Header/Footer/PageNumber elements out. They ride along into whatever fragment happens to contain them, and once indexed as analyzed text, a running header repeated on every page becomes a high-document-frequency term that skews Elasticsearch's corpus-wide BM25 idf weighting away from the substantive content a query is actually looking for.
  • A Table element's text_as_html can split mid-fragment. Two Elasticsearch documents end up sharing a file_id but no lineage field, so a bool.filter scoped to one document can return half a table with no way to know the other half exists.
  • Overlap, if still applied, duplicates spans into both the HNSW graph and the inverted index — larger index, slower merges, top_k crowded with near-duplicates.

The pipeline, end to end ​

bash
pip install unstructured elasticsearch
python
from unstructured.partition.pdf import partition_pdf
from unstructured.staging.base import elements_to_json
from poma import PrimeCut
from poma.vektoria import assemble, records_from_archive, Volume
from poma.embeddings import get_embedder
from elasticsearch import Elasticsearch

# 1. Your existing Unstructured call — unchanged.
elements = partition_pdf(
    filename="contract.pdf",
    strategy="hi_res",
    infer_table_structure=True,            # populates metadata.text_as_html
    extract_image_block_types=["Image"],
    extract_image_block_to_payload=True,   # inline base64 → POMA can describe figures
)
elements_to_json(elements, filename="contract.unstructured.json")

# 2. The missing link — raw element list in, a portable .poma archive out.
client = PrimeCut()  # reads POMA_API_KEY
client.ingest("contract.unstructured.json", download_dir="archives", filename="contract.poma")

embedder = get_embedder("local:BAAI/bge-small-en-v1.5")
vol = Volume("s3://your-bucket/poma")
es = Elasticsearch("http://localhost:9200")  # local dev cluster (xpack.security.enabled=false); a secured cluster needs https + basic_auth/api_key + ca_certs

es.indices.create(index="poma", mappings={"properties": {
    "embedding": {"type": "dense_vector", "dims": embedder.dims, "index": True, "similarity": "cosine"},
    "file_id": {"type": "keyword"},
    "chunkset_index": {"type": "integer"},
}}, ignore=400)

# 3. Content-free ingest — the index holds only routing metadata; content lives on the volume.
records = records_from_archive("archives/contract.poma")  # -> list[Record] (id, text, payload)
# assemble() reads the FULL document (chunks + chunksets) from the volume — write it once,
# not one {"chunks": <indices>, "text"} stub per record (which also overwrote itself each turn).
from poma.utils import unpack_poma_archive
_data = unpack_poma_archive(poma_archive_path="archives/contract.poma")
file_id = records[0].payload["file_id"]
vol.write_doc(file_id, {"file_id": file_id, "chunks": _data["chunks"], "chunksets": _data["chunksets"]})
for r in records:
    es.index(index="poma", id=r.id, document={
        "embedding": embedder.embed([r.text])[0],
        "file_id": r.payload["file_id"],
        "chunkset_index": r.payload["chunkset_index"],
    })

# 4. Retrieval — Elasticsearch's normal knn query, then assemble.
qv = embedder.embed(["early termination conditions"])[0]
es.indices.refresh(index="poma")  # make the just-indexed docs searchable before querying
res = es.search(index="poma", knn={"field": "embedding", "query_vector": qv,
                                    "k": 10, "num_candidates": 100})
context = assemble(res.body, volume=vol)  # elasticsearch-py 8+ wraps the response (ObjectApiResponse) — pass the dict body  # -> [{"file_id", "content"}, ...]

Because Header, Footer, and PageNumber elements are dropped before chunking (not after indexing), the fragmentation-driven BM25 pollution described above never reaches Elasticsearch in the first place — POMA removes the noise upstream of the mapping decision. _source returns file_id/chunkset_index by default, so assemble() needs no extra request flag.

Metadata mapping: POMA fields → Elasticsearch primitives ​

POMA field (Unstructured-sourced)Elasticsearch primitiveNotes
chunkset id (chunkset_uuid)document _idstable across re-indexing
to_embed (tables spliced as text_as_html, headers/footers dropped)dense_vector fieldembedded before index, never stored as analyzed text
file_idkeyword fieldreturned in _source by default for assemble()
chunkset_indexinteger fieldreturned in _source by default for assemble()
page (from metadata.page_number), depth, chunk_index, full chunkset textvolume documentcontent-free — never indexed, fetched by assemble()

Frequently asked questions ​

How do I get Unstructured.io elements into Elasticsearch for RAG? ​

Save the element list with elements_to_json, ingest it with PrimeCut, and keep the .poma archive. Create a dense_vector mapping with file_id/chunkset_index, index documents from records_from_archive, write chunkset content to a volume, then retrieve with es.search(knn=...) plus assemble(res, volume=vol) — no extra metadata flag needed.

chunk_by_title doesn't filter element types, so repeated Header/Footer/PageNumber text rides into analyzed fields and inflates corpus-wide document frequency, diluting the idf weight of substantive query terms.

What Elasticsearch mapping fields should Unstructured chunks carry? ​

embedding as dense_vector (dims matching your embedder, similarity: cosine), file_id as keyword, chunkset_index as integer. Page, depth, chunk index, and full text stay off the document and live on a volume.

Do Unstructured's inline images survive into Elasticsearch retrieval? ​

Yes, if extract_image_block_to_payload=true was set — POMA describes the figure and the description becomes ordinary embedded, volume-stored text. Disk-based image_path images are neutralized and counted, never silently dropped.

Does Elasticsearch's knn plus bool hybrid query need extra flags for Unstructured metadata? ​

No — _source returns file_id/chunkset_index by default, unlike Pinecone or Milvus which need an explicit ask. The content-free pattern keeps _source small regardless.

Same parser, different store: Unstructured.io → OpenSearch · Unstructured.io → Turbopuffer · Unstructured.io → Qdrant

Foundations: The Optimal Chunker for Unstructured.io · Elasticsearch Chunking Strategy for RAG · All pipeline recipes · RAG chunking guide