Source: http://www.poma-ai.com/docs/pipelines/unstructured-to-weaviate

# The Missing Link Between Unstructured.io and Optimal Retrieval in Weaviate

<ByAuthor />

**The short answer:** Unstructured.io gives you excellent typed elements; Weaviate gives you excellent built-in hybrid search — BM25F and vector similarity fused by a single `alpha`. Wired together naively — one object per element, or `chunk_by_title` fragments as objects — hybrid hits land on bare fragments with no section context, and Weaviate's cross-references have nothing to point at, because a flat element list contains no hierarchy. The missing link is POMA: `PrimeCut().ingest()` consumes the raw `elements_to_json` output (auto-detected), emits hierarchy-preserving [chunksets](/learn/chunking/chunksets), and each chunkset becomes one Weaviate object whose text already carries its full ancestor path — exactly what BM25F and the vector index should be scoring.

## What each end of the pipeline actually provides

**Unstructured.io** (`partition_pdf` or the hosted API) returns a flat, ordered list of typed elements — `Title`, `NarrativeText`, `ListItem`, `Table`, `Image` — each with a stable `element_id` and metadata such as `page_number` and `text_as_html` for tables. What it doesn't provide: nesting. Nothing records which `Title` sits under which, and the built-in `by_title` and `basic` chunkers emit flat fragments. Details: [The Optimal Chunker for Unstructured.io](/optimal-chunker-unstructured).

**Weaviate** provides collections with named vectors, hybrid search that fuses BM25F keyword scoring with vector similarity via `alpha`, filterable properties, and cross-references between collections. What it doesn't provide: any opinion about what an object should contain. Hybrid search ranks the objects you gave it — including context-free fragments, if that's what you inserted. Details: [The Optimal Chunks for the Best Retrieval in Weaviate](/optimal-chunks-weaviate).

## The naive wiring, and where it breaks

The common recipe — one Weaviate object per element (or per `chunk_by_title` fragment), properties copied from `element.metadata` — breaks in a pair-specific way:

- **Element granularity defeats BM25F.** Keyword scoring is Weaviate's differentiator, but a keyword hit on a bare `ListItem` or `NarrativeText` object returns that fragment alone. The clause number matched; the section it governs is in some other object, unlinked.
- **Cross-references have nothing to point at.** Weaviate can model parent-child relations between collections — but Unstructured's element list is flat, so there is no parent to reference. The schema feature that could carry lineage goes unused because the lineage was never extracted.
- **`alpha` tuning can't fix granularity.** Sliding between keyword and vector weighting changes which context-free fragment ranks first, not whether the winning object explains itself to the LLM.

The fix is upstream of the schema: build the hierarchy first, then insert units that carry it.

## The pipeline, end to end

```bash
pip install unstructured weaviate-client sentence-transformers poma
```

```python
import os
import weaviate
from weaviate.classes.config import Configure, DataType, Property
from weaviate.classes.query import Filter
from unstructured.partition.pdf import partition_pdf
from unstructured.staging.base import elements_to_json
from sentence_transformers import SentenceTransformer
from poma import PrimeCut

# 1. Your existing Unstructured call — unchanged.
elements = partition_pdf(
    filename="contract.pdf",
    strategy="hi_res",
    infer_table_structure=True,            # populates metadata.text_as_html
    extract_image_block_types=["Image"],
    extract_image_block_to_payload=True,   # inline base64 → POMA can describe figures
)
elements_to_json(elements, filename="contract.unstructured.json")

# 2. The missing link — raw element list in, chunks + chunksets out.
poma = PrimeCut()  # reads POMA_API_KEY
result = poma.ingest("contract.unstructured.json")  # Unstructured shape auto-detected

# 3. Weaviate — one object per chunkset, vectors supplied client-side.
client = weaviate.connect_to_weaviate_cloud(
    cluster_url=os.environ["WEAVIATE_URL"],
    auth_credentials=weaviate.classes.init.Auth.api_key(os.environ["WEAVIATE_API_KEY"]),
)
chunksets = client.collections.create(
    "Chunkset",
    vectorizer_config=Configure.Vectorizer.none(),
    properties=[
        Property(name="content", data_type=DataType.TEXT),
        Property(name="file_id", data_type=DataType.TEXT),
        Property(name="page", data_type=DataType.INT),
        Property(name="depth", data_type=DataType.INT),
    ],
)

model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
with chunksets.batch.dynamic() as batch:
    for cs in result.chunksets:
        batch.add_object(
            properties={
                "content": cs.to_embed,
                "file_id": cs.file_id,
                "page": cs.page,
                "depth": cs.depth,
            },
            vector=model.encode(cs.to_embed).tolist(),
        )

# 4. Hybrid retrieval: BM25F + vector, blended by alpha.
query = "What are the early termination conditions?"
hits = chunksets.query.hybrid(
    query=query,
    vector=model.encode(query).tolist(),
    alpha=0.5,
    limit=5,
    filters=Filter.by_property("file_id").equal(result.chunksets[0].file_id),
)
for obj in hits.objects:
    print(obj.properties["content"])
client.close()
```

POMA validates the payload shape up front (a corrupted or mislabeled upload 422s immediately), splices `text_as_html` for tables, describes inline images, drops `Header`/`Footer`/`PageNumber` elements as page furniture, and rebuilds the cross-page heading tree before chunking. To force or suppress detection, set `external_ocr_source` to `"unstructured"` or `"none"`. The final assembly step — deduplicating and merging retrieved chunksets into one prompt-ready cheatsheet — runs client-side on the hybrid results.

On our reference legal-document benchmark, this pipeline answers the same query with **337 tokens** of retrieved context instead of **1,542** for a recursive character splitter, with zero information loss — methodology in [Document Ingestion & Chunking for RAG](/document-ingestion-chunking-rag). Because chunksets are overlap-free, BM25F term statistics aren't skewed by the same boundary text appearing in multiple objects.

## Metadata mapping: POMA fields → Weaviate primitives

| POMA chunk field | Weaviate primitive | What it enables |
| --- | --- | --- |
| `to_embed` | object vector + `content` TEXT property (BM25F-indexed) | one object serves both halves of hybrid search |
| `file_id` | TEXT property, filterable | scope queries to one document |
| `page` (from `metadata.page_number`) | INT property, filterable | page-cited answers, page-range filters |
| `depth` | INT property, filterable | filter/re-rank by hierarchy level |
| `chunk_index` | INT property | stable ordering at assembly time |
| chunk ↔ chunkset lineage | cross-reference between collections (optional) | explicit tree traversal when you need it |

## Frequently asked questions

### How do I get Unstructured.io elements into Weaviate for RAG?

Save the element list with `elements_to_json`, run `PrimeCut().ingest()` on it (auto-detected, hierarchy rebuilt), then insert each chunkset as one object with `file_id`/`page`/`depth` properties and its `to_embed` vector. Query with `query.hybrid(...)`, then assemble cheatsheets.

### Should each Unstructured element become its own Weaviate object?

No — that's the wrong granularity. A BM25F hit on a bare `ListItem` arrives without section context, and no parent object exists to cross-reference because the element list is flat. Insert chunksets: self-explanatory root-to-leaf units with the ancestor path inside.

### What alpha should Weaviate hybrid search use for Unstructured-parsed documents?

`alpha` blends BM25F (0) with vector similarity (1). Parsed documents carry exact tokens that dense embeddings blur, so start mid-range and tune on your query mix. No `alpha` restores section context that was never stored in the object.

### Can Weaviate cross-references model Unstructured's document hierarchy?

Only after the hierarchy exists — the flat element list gives nothing to reference. Once POMA rebuilds the tree, chunk-to-chunkset cross-references are possible, but usually unnecessary: each chunkset already carries its full root-to-leaf path in its own text.

### What Weaviate properties should Unstructured-derived chunks carry?

`file_id` (TEXT), `page` (INT), `depth` (INT), `chunk_index` (INT), plus the BM25F-indexed `content` property and the vector from `to_embed`. Filter on the scalars to scope hybrid queries to a document or page range.

## Related recipes

Same parser, different store: [Unstructured → Qdrant](/pipelines/unstructured-to-qdrant) · [Unstructured → Pinecone](/pipelines/unstructured-to-pinecone) · [Unstructured → Chroma](/pipelines/unstructured-to-chroma)

Same store, different parser: [Mistral OCR → Weaviate](/pipelines/mistral-ocr-to-weaviate) · [LlamaParse → Weaviate](/pipelines/llamaparse-to-weaviate) · [Marker → Weaviate](/pipelines/marker-to-weaviate)

Foundations: [The Optimal Chunker for Unstructured.io](/optimal-chunker-unstructured) · [The Optimal Chunks for Weaviate](/optimal-chunks-weaviate) · [All pipeline recipes](/pipelines/) · [RAG chunking guide](/guides/rag-chunking/)