Source: http://www.poma-ai.com/docs/optimal-chunker-unstructured

# The Optimal Chunker for Unstructured.io

<ByAuthor />

**The short answer:** the optimal chunker for Unstructured.io is one that consumes the raw element list *as-is*, uses the typed elements Unstructured recovered — Title, NarrativeText, Table, Image — to rebuild a real heading hierarchy, and emits retrieval units that carry their own context. POMA's bring-your-own-OCR connector does exactly this: you upload the unmodified `elements_to_json` output, and PrimeCut returns hierarchy-preserving **chunks** and **chunksets** — no re-parsing, no `chunk_by_title` fragments, no hand-rolled splitter.

This page explains what Unstructured gives you, where it stops, why its built-in chunking strategies fall short of retrieval-grade units, and how to go from `partition_pdf` to embedded chunksets in a few lines.

## What Unstructured.io gives you — and where it stops

Unstructured (the open-source `unstructured` library and the hosted Unstructured API) is one of the most widely used document partitioners in RAG stacks: you feed it a PDF, DOCX, HTML page, or email, and it returns a **flat, ordered list of typed elements**:

```json
[
  {
    "type": "Title",
    "text": "Financial highlights",
    "element_id": "8f4a2c…",
    "metadata": { "page_number": 3, "filename": "report.pdf" }
  },
  {
    "type": "Table",
    "text": "Revenue 2025 …",
    "element_id": "b91e07…",
    "metadata": { "page_number": 3, "text_as_html": "<table>…</table>" }
  }
]
```

That element typing is genuinely valuable: `Title` vs. `NarrativeText` vs. `ListItem` vs. `Table` is exactly the distinction a naive PDF-to-text pass loses, `text_as_html` preserves table cell structure, `page_number` survives in metadata, and `element_id` gives every element a stable identity. Element order is document order — Unstructured has already done the reading-order work.

What Unstructured deliberately does **not** do:

- **Hierarchy.** The element list is flat. Two `Title` elements sit side by side in the stream whether one is a chapter and the other a sub-subsection; nothing records that `### Early termination` lives inside `## Termination clauses` inside `# Master Services Agreement`. Typed elements ≠ a document tree.
- **Retrieval-ready chunking.** The built-in strategies (`by_title`, `basic`) group consecutive elements into fragments bounded by character limits. They are packing algorithms, not structure builders — see the next section.
- **Image semantics.** Depending on your call parameters, an `Image` element carries inline bytes (`image_base64`), a path on local disk (`image_path`), or just its position. Something downstream has to make figures searchable — or at least make their loss visible.

Unstructured's job ends at "here is what the document contains, in order, with types." The gap between *typed elements* and *good retrieval* is where RAG pipelines quietly fail.

## Why `by_title` and `basic` chunking emit flat fragments

Unstructured ships two chunking strategies. `basic` fills chunks to a character limit, splitting on element boundaries where possible. `by_title` is smarter: it starts a new chunk at each `Title` element, so a chunk never straddles a section boundary. Both, however, produce **flat fragments**:

| What Unstructured recovered | What `by_title` / `basic` chunking does with it |
| --- | --- |
| Typed elements (`Title`, `NarrativeText`, …) | Flattened — types gate chunk boundaries, then disappear into concatenated text |
| Nesting between headings | Never existed — a chunk knows its nearest `Title`, not the chapters above it |
| Long sections | Split again by `max_characters` — the tail fragments lose even the section title |
| `text_as_html` tables | Table becomes an isolated chunk or is truncated to fit the character budget |
| `Image` elements | No description step — figure content never becomes searchable text |
| `Header` / `Footer` / `PageNumber` | Concatenated into chunks as noise unless you filter them yourself |

The result is the classic fragment problem: at retrieval time the LLM reads a paragraph that begins mid-section, with no ancestor context, and answers as if the paragraph were the whole document. That failure mode — not partitioning accuracy — is where most Unstructured-based RAG pipelines lose their quality.

For the full taxonomy of these failure modes, see the [RAG chunking strategies guide](/rag-chunking-strategies-text-splitters).

## The optimal chunker: a real tree from the element stream

POMA's **bring-your-own-OCR** (BYOCR) connector was built for teams who already run their own parsing step. The connector consumes the *unmodified* Unstructured output — a bare element list or an `{"elements": […]}` envelope — and runs only POMA's downstream stages on it:

1. **Shape validation, up front.** The payload is fingerprinted (elements carrying `type` + `element_id`). A corrupted or mislabeled upload fails immediately with a clear 422 — never a silently degraded index.
2. **Page grouping and type mapping.** Elements are grouped by `metadata.page_number` and each type is mapped to its markdown equivalent — titles become headings, list items become lists — preserving the document order Unstructured established.
3. **Table splicing.** Each `Table` element's `text_as_html` is used instead of its flattened text, so rows and columns survive into the chunk layer as HTML and embeddings never see half a table.
4. **Image handling for both Unstructured cases.** Inline `image_base64` (set `extract_image_block_to_payload=true` when partitioning) is decoded and described — figures become searchable text. Images extracted to disk via `image_path` live outside the payload, so their refs are neutralized and **counted in `content_metadata`** — content loss is surfaced, never silent.
5. **Noise removal.** `Header`, `Footer`, and `PageNumber` elements are dropped — Unstructured already isolated the page furniture, and POMA honours that typing, so your index isn't 4% running headers.
6. **Heading enrichment and hierarchy.** Where the element stream under-marks headings, POMA's heading-injection pass restores them; then the cross-page heading tree is rebuilt and every sentence is emitted as a chunk with its `depth`, `page`, and a `to_embed` normalization — grouped into [chunksets](/learn/chunking/chunksets): root-to-leaf paths that keep each retrieved passage self-explanatory.

Because you already ran the parsing, **POMA charges only for the downstream structure and chunking value** — the OCR front-end is skipped entirely.

The result, measured on our reference legal-document benchmark: the same query answered with **337 tokens** of retrieved context instead of **1,542** for a recursive character splitter, with zero information loss. Methodology: [Document Ingestion & Chunking for RAG](/document-ingestion-chunking-rag).

## How to chunk Unstructured.io output with POMA

Run your partitioning exactly as you do today, save the element list as JSON, and hand it to PrimeCut:

```python
from unstructured.partition.pdf import partition_pdf
from unstructured.staging.base import elements_to_json
from poma import PrimeCut

# 1. Your existing Unstructured call — unchanged.
elements = partition_pdf(
    filename="contract.pdf",
    strategy="hi_res",
    infer_table_structure=True,            # populates metadata.text_as_html
    extract_image_block_types=["Image"],
    extract_image_block_to_payload=True,   # inline base64 → POMA can describe figures
)
elements_to_json(elements, filename="contract.unstructured.json")

# 2. Chunk the raw element list — POMA auto-detects the Unstructured shape.
client = PrimeCut()  # reads POMA_API_KEY
result = client.ingest("contract.unstructured.json")

print(f"chunks: {len(result.chunks)}")
print(f"chunksets: {len(result.chunksets)}")
print(result.chunksets[0].to_embed)  # embedding-ready, hierarchy included
```

The same applies to the hosted Unstructured API: save the response's element list to a JSON file and ingest it identically.

Two things worth knowing:

- **Auto-detection.** A JSON upload whose shape structurally fingerprints as an Unstructured result (elements carrying `type` + `element_id`) is routed through the BYOCR seam automatically. To force the declaration — or to opt a look-alike JSON *out* of detection — set the `external_ocr_source` parameter to `"unstructured"` or `"none"` respectively on the ingest request.
- **Dedicated endpoint.** For explicit routing, the API exposes a `chunk_external_ocr_result` endpoint that accepts `external_ocr_source: "unstructured"`. See the [API reference](https://api.poma-ai.com/v3/docs).

From `result.chunks` / `result.chunksets` onward, everything is your stack: your vector store, your embedding model, your retrieval strategy. See the [Qdrant](/sdk/integrations/qdrant), [LangChain](/sdk/integrations/langchain), and [LlamaIndex](/sdk/integrations/llamaindex) integrations.

## Chunking options for Unstructured.io output, compared

| Approach | Structure used | Ancestor hierarchy | Tables | Images | Page numbers kept |
| --- | --- | --- | --- | --- | --- |
| Flatten + character splitter | None | ✗ | Cut mid-row | Dropped | ✗ |
| `basic` chunking | Element boundaries | ✗ | Isolated or truncated | Not described | In metadata |
| `by_title` chunking | Nearest `Title` only | ✗ — flat fragments | Isolated or truncated | Not described | In metadata |
| **POMA BYOCR (PrimeCut)** | **Full heading tree, rebuilt across pages** | **✓ chunksets** | **`text_as_html` spliced, never cut** | **Described or visibly counted** | **✓ per chunk** |

## Frequently asked questions

### How do I chunk Unstructured.io elements for RAG?

Don't join the element texts into one string and run a character splitter, and don't stop at `chunk_by_title` — both flatten the typed structure Unstructured recovered. Save the element list with `elements_to_json` and feed the raw JSON to a structure-aware chunker. POMA's BYOCR connector accepts the unmodified element list, rebuilds a real heading hierarchy from the `Title` elements, and emits chunks plus chunksets ready for embedding.

### Is there an alternative to `chunk_by_title` in Unstructured?

Yes. `chunk_by_title` groups elements under the nearest `Title` into flat fragments: each chunk knows its immediate section but nothing about the chapters above it, and long sections get split again by character limits, losing even that link. POMA replaces the chunking step entirely — it builds a nested heading tree across pages and returns chunksets in which every retrieval unit carries its full root-to-leaf heading path.

### Can I use Unstructured.io output with POMA without re-parsing?

Yes — that's the point of the bring-your-own-OCR path. POMA skips its OCR front-end and runs only the downstream structure, chunking, and retrieval-preparation stages on your result. Upload the saved element-list JSON (auto-detected) or declare `external_ocr_source: "unstructured"`.

### What does Unstructured.io return, exactly?

A flat, ordered list of typed elements — `Title`, `NarrativeText`, `ListItem`, `Table`, `Image`, `Header`, `Footer`, `PageNumber`, and others — each with a `type`, its `text`, a stable `element_id`, and `metadata` such as `page_number`, `text_as_html` for tables, and optionally `image_base64`. Element order is document order, but the list itself is flat: hierarchy has to be rebuilt downstream.

### What happens to tables and images from Unstructured during chunking?

POMA splices each `Table` element's `text_as_html`, so tables survive as HTML and are never cut mid-row. Images with inline bytes (`extract_image_block_to_payload=true`) are decoded and described; images on disk via `image_path` are outside the payload, so their refs are neutralized and counted in `content_metadata` — visible loss, never silent. `Header`, `Footer`, and `PageNumber` elements are dropped as page furniture.

### Do Unstructured's typed elements give me document hierarchy?

No — typed elements are not hierarchy. Knowing an element is a `Title` tells you a heading exists, not which level it sits at or which sections it contains; the list stays flat. That's fine: partitioning and structuring are different jobs. A hierarchy-aware chunker infers heading levels from the element stream, nests sections across pages, and keeps every chunk linked to its full ancestor path.

## Take it straight to your vector database

End-to-end recipes for Unstructured.io + POMA + your store:

- [Unstructured to Qdrant](/pipelines/unstructured-to-qdrant)
- [Unstructured to Pinecone](/pipelines/unstructured-to-pinecone)
- [Unstructured to Weaviate](/pipelines/unstructured-to-weaviate)
- [Unstructured to pgvector](/pipelines/unstructured-to-pgvector)
- [Unstructured to Milvus](/pipelines/unstructured-to-milvus)
- [Unstructured to Chroma](/pipelines/unstructured-to-chroma)

## The rest of the series

The optimal chunker, for every OCR and parsing tool you already run:

- [Mistral OCR](/optimal-chunker-mistral-ocr)
- [LlamaParse](/optimal-chunker-llamaparse)
- [Azure Document Intelligence](/optimal-chunker-azure-document-intelligence)
- [Docling](/optimal-chunker-docling)
- [Marker](/optimal-chunker-marker)
- [AWS Textract](/optimal-chunker-textract)

Or start with the fundamentals: [RAG chunking guide](/guides/rag-chunking/) · [Document ingestion guide](/guides/document-ingestion/) · [POMA chunksets](/learn/chunking/chunksets).