Unstructured.io to Weaviate: Chunking for RAG
The short answer: Unstructured.io gives you excellent typed elements; Weaviate gives you excellent built-in hybrid search — BM25F and vector similarity fused by a single alpha. Wired together naively — one object per element, or chunk_by_title fragments as objects — hybrid hits land on bare fragments with no section context, and Weaviate's cross-references have nothing to point at, because a flat element list contains no hierarchy. The missing link is POMA: PrimeCut().ingest() consumes the raw elements_to_json output (auto-detected), emits hierarchy-preserving chunksets, and each chunkset becomes one Weaviate object whose text already carries its full ancestor path — exactly what BM25F and the vector index should be scoring.
What each end of the pipeline actually provides
Unstructured.io (partition_pdf or the hosted API) returns a flat, ordered list of typed elements — Title, NarrativeText, ListItem, Table, Image — each with a stable element_id and metadata such as page_number and text_as_html for tables. What it doesn't provide: nesting. Nothing records which Title sits under which, and the built-in by_title and basic chunkers emit flat fragments. Details: The Optimal Chunker for Unstructured.io.
Weaviate provides collections with named vectors, hybrid search that fuses BM25F keyword scoring with vector similarity via alpha, filterable properties, and cross-references between collections. What it doesn't provide: any opinion about what an object should contain. Hybrid search ranks the objects you gave it — including context-free fragments, if that's what you inserted. Details: The Optimal Chunks for the Best Retrieval in Weaviate.
The naive wiring, and where it breaks
The common recipe — one Weaviate object per element (or per chunk_by_title fragment), properties copied from element.metadata — breaks in a pair-specific way:
- Element granularity defeats BM25F. Keyword scoring is Weaviate's differentiator, but a keyword hit on a bare
ListItemorNarrativeTextobject returns that fragment alone. The clause number matched; the section it governs is in some other object, unlinked. - Cross-references have nothing to point at. Weaviate can model parent-child relations between collections — but Unstructured's element list is flat, so there is no parent to reference. The schema feature that could carry lineage goes unused because the lineage was never extracted.
alphatuning can't fix granularity. Sliding between keyword and vector weighting changes which context-free fragment ranks first, not whether the winning object explains itself to the LLM.
The fix is upstream of the schema: build the hierarchy first, then insert units that carry it.
The pipeline, end to end
pip install unstructured weaviate-client sentence-transformers pomaimport os
import weaviate
from weaviate.classes.config import Configure, DataType, Property
from weaviate.classes.query import Filter
from unstructured.partition.pdf import partition_pdf
from unstructured.staging.base import elements_to_json
from sentence_transformers import SentenceTransformer
from poma import PrimeCut
from poma.utils import unpack_poma_archive
# 1. Your existing Unstructured call — unchanged.
elements = partition_pdf(
filename="contract.pdf",
strategy="hi_res",
infer_table_structure=True, # populates metadata.text_as_html
extract_image_block_types=["Image"],
extract_image_block_to_payload=True, # inline base64 → POMA can describe figures
)
elements_to_json(elements, filename="contract.unstructured.json")
# 2. The missing link — raw element list in, chunks + chunksets out.
poma = PrimeCut() # reads POMA_API_KEY
result = poma.ingest( # Unstructured shape auto-detected
"contract.unstructured.json",
download_dir="store",
filename="contract.poma", # keep the archive — page numbers live in it
)
# Depth comes off the SDK objects; page is on the archive's chunk records only.
depth_of = {c.chunk_index: c.depth for c in result.chunks}
page_of = {
c["chunk_index"]: c.get("page")
for c in unpack_poma_archive(poma_archive_path="store/contract.poma")["chunks"]
}
# 3. Weaviate — one object per chunkset, vectors supplied client-side.
client = weaviate.connect_to_weaviate_cloud(
cluster_url=os.environ["WEAVIATE_URL"],
auth_credentials=weaviate.classes.init.Auth.api_key(os.environ["WEAVIATE_API_KEY"]),
)
chunksets = client.collections.create(
"Chunkset",
vectorizer_config=Configure.Vectorizer.none(),
properties=[
Property(name="content", data_type=DataType.TEXT),
Property(name="file_id", data_type=DataType.TEXT),
Property(name="chunkset_index", data_type=DataType.INT),
Property(name="page", data_type=DataType.INT),
Property(name="depth", data_type=DataType.INT),
],
)
model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
with chunksets.batch.dynamic() as batch:
for cs in result.chunksets:
leaf = cs.chunks[-1] # document order — the last chunk index is the leaf
batch.add_object(
properties={
"content": cs.to_embed,
"file_id": cs.file_id,
"chunkset_index": cs.chunkset_index,
"page": page_of[leaf],
"depth": depth_of[leaf],
},
vector=model.encode(cs.to_embed).tolist(),
)
# 4. Hybrid retrieval: BM25F + vector, blended by alpha.
query = "What are the early termination conditions?"
hits = chunksets.query.hybrid(
query=query,
vector=model.encode(query).tolist(),
alpha=0.5,
limit=5,
filters=Filter.by_property("file_id").equal(result.chunksets[0].file_id),
)
for obj in hits.objects:
print(obj.properties["content"])
client.close()POMA validates the payload shape up front (a corrupted or mislabeled upload 422s immediately), splices text_as_html for tables, describes inline images, drops Header/Footer/PageNumber elements as page furniture, and rebuilds the cross-page heading tree before chunking. To force or suppress detection, set external_ocr_source to "unstructured" or "none". The final assembly step — deduplicating and merging retrieved chunksets into one prompt-ready cheatsheet — runs client-side on the hybrid results.
On our reference legal-document benchmark, this pipeline answers the same query with 337 tokens of retrieved context instead of 1,542 for a recursive character splitter, with zero information loss — methodology in Document Ingestion & Chunking for RAG. Because chunksets are overlap-free, BM25F term statistics aren't skewed by the same boundary text appearing in multiple objects.
Metadata mapping: POMA fields → Weaviate primitives
| POMA chunk field | Weaviate primitive | What it enables |
|---|---|---|
to_embed | object vector + content TEXT property (BM25F-indexed) | one object serves both halves of hybrid search |
file_id | TEXT property, filterable | scope queries to one document |
page of the chunkset's leaf chunk (from the archive's chunk records, filled from metadata.page_number) | INT property, filterable | page-cited answers, page-range filters |
depth of the leaf chunk | INT property, filterable | filter/re-rank by hierarchy level |
chunkset_index | INT property | stable ordering at assembly time |
| chunk ↔ chunkset lineage | cross-reference between collections (optional) | explicit tree traversal when you need it |
Frequently asked questions
How do I get Unstructured.io elements into Weaviate for RAG?
Save the element list with elements_to_json, run PrimeCut().ingest() on it (auto-detected, hierarchy rebuilt), then insert each chunkset as one object with file_id/chunkset_index/page/depth properties and its to_embed vector — page and depth taken from the chunkset's leaf chunk, with page read out of the .poma archive's chunk records. Query with query.hybrid(...), then assemble cheatsheets.
Should each Unstructured element become its own Weaviate object?
No — that's the wrong granularity. A BM25F hit on a bare ListItem arrives without section context, and no parent object exists to cross-reference because the element list is flat. Insert chunksets: self-explanatory root-to-leaf units with the ancestor path inside.
What alpha should Weaviate hybrid search use for Unstructured-parsed documents?
alpha blends BM25F (0) with vector similarity (1). Parsed documents carry exact tokens that dense embeddings blur, so start mid-range and tune on your query mix. No alpha restores section context that was never stored in the object.
Can Weaviate cross-references model Unstructured's document hierarchy?
Only after the hierarchy exists — the flat element list gives nothing to reference. Once POMA rebuilds the tree, chunk-to-chunkset cross-references are possible, but usually unnecessary: each chunkset already carries its full root-to-leaf path in its own text.
What Weaviate properties should Unstructured-derived chunks carry?
file_id (TEXT), chunkset_index (INT), plus the leaf chunk's page (INT) and depth (INT), the BM25F-indexed content property, and the vector from to_embed. depth is on the SDK's chunk objects; page is on the archive's chunk records, so read it back with unpack_poma_archive. Filter on the scalars to scope hybrid queries to a document or page range.
Related recipes
Same parser, different store: Unstructured → Qdrant · Unstructured → Pinecone · Unstructured → Chroma
Same store, different parser: Mistral OCR → Weaviate · LlamaParse → Weaviate · Marker → Weaviate
Foundations: The Optimal Chunker for Unstructured.io · The Optimal Chunks for Weaviate · All pipeline recipes · RAG chunking guide