Skip to content

PaddleOCR-VL to Weaviate: Chunking for RAG ​

The short answer: PaddleOCR-VL gives you self-hosted, layout-aware OCR behind your own HTTP endpoint; Weaviate gives you built-in hybrid search — BM25F plus vectors, blended by alpha — over filterable properties. Wired together naively — flatten, split, embed — they still produce mediocre RAG, and a pipeline that only handles one of PaddleOCR-VL's two accepted shapes can silently drop whole pages from both halves of the hybrid at once. The missing link is POMA: PrimeCut().ingest() consumes the raw layout-parsing JSON (auto-detected, either shape), emits hierarchy-preserving chunksets, and each becomes one Weaviate object with file_id/page/depth properties and your embedding as its vector.

What each end of the pipeline actually provides ​

PaddleOCR-VL runs self-hosted — PP-DocLayoutV3 for layout detection, PaddleOCR-VL served via vLLM for reading, behind a /layout-parsing HTTP API you operate. It returns either the PaddleX serving envelope (result.layoutParsingResults[], with a markdown object plus a prunedResult block list) or a raw save_to_json() page dict (parsing_res_list: flat blocks with block_label, block_content, block_id, block_order). Both are the tool's own fingerprints, and a batch job can legitimately emit either one per document. Details: The Optimal Chunker for PaddleOCR-VL.

Weaviate provides collections with named vectors, built-in hybrid search (BM25F + vector, blended by alpha), filterable properties, cross-references between collections, and generative/reranker modules — open source or Weaviate Cloud. What it doesn't provide: any opinion about what an object should contain. BM25F scores whatever text you stored; the vector index ranks whatever you embedded — nothing, if a page never made it into a collection object at all. Details: The Optimal Chunks for the Best Retrieval in Weaviate.

The naive wiring, and where it breaks ​

The common recipe — call /layout-parsing, assume every response has markdown.text, split, embed, data.insert(...) — breaks in a pair-specific way:

  • Handling only one accepted shape drops pages, not just quality. Code written against the serving envelope's markdown object throws or silently skips any page that instead arrives as a bare save_to_json() block dict — a real possibility across a self-hosted batch pipeline processing many documents over time. That page is absent from both BM25F and vector search, not degraded in either.
  • page_index vanishes at the join, so the one property Weaviate needs for page filters and page-cited answers holds nothing truthful.
  • Overlap and block-order mistakes pollute both hybrid halves at once. Splitter overlap duplicates boundary spans in both BM25F and vector rankings; sorting blocks by block_id instead of block_order scrambles the exact tokens BM25F depends on.

Chunksets fix this at the unit level: both PaddleOCR-VL shapes route through the same hierarchy-preserving assembly, no page silently dropped, no overlap, exact tokens intact. On a reference legal document, assembled cheatsheets delivered 337 tokens of retrieved context versus 1,542 from a recursive character splitter, with zero information loss (methodology).

The pipeline, end to end ​

bash
pip install requests poma weaviate-client sentence-transformers
python
import json
import os
import requests
import weaviate
from weaviate.classes.config import Configure, DataType, Property
from weaviate.classes.init import Auth
from weaviate.classes.query import Filter
from poma import PrimeCut
from poma.utils import unpack_poma_archive
from sentence_transformers import SentenceTransformer

# 1. Your existing self-hosted call — unchanged. `/layout-parsing` is the
#    PaddleX-compatible endpoint your layout-server + PaddleOCR-VL stack exposes.
resp = requests.post(
    "http://layout-server:40111/layout-parsing",
    json={"file": "<base64-encoded PDF>", "fileType": 0},
)
resp.raise_for_status()
with open("contract.paddleocr-vl.json", "w") as f:
    json.dump(resp.json(), f)

# 2. The missing link — raw result JSON in, chunks + chunksets out.
poma = PrimeCut()  # reads POMA_API_KEY
# keep the .poma archive: page numbers live on its chunk records, not on the SDK objects
result = poma.ingest(
    "contract.paddleocr-vl.json",  # PaddleOCR-VL shape auto-detected
    download_dir="store",
    filename="contract.poma",
)

# Chunk lookups: depth from the SDK objects, page from the archive's chunk records.
depth_of = {c.chunk_index: c.depth for c in result.chunks}
page_of = {
    c["chunk_index"]: c.get("page")
    for c in unpack_poma_archive(poma_archive_path="store/contract.poma")["chunks"]
}

# 3. One collection of chunksets, hierarchy as filterable properties.
client = weaviate.connect_to_weaviate_cloud(
    cluster_url=os.environ["WEAVIATE_URL"],
    auth_credentials=Auth.api_key(os.environ["WEAVIATE_API_KEY"]),
)
client.collections.create(
    "Chunkset",
    vectorizer_config=Configure.Vectorizer.none(),  # bring your own vectors
    properties=[
        Property(name="content", data_type=DataType.TEXT),
        Property(name="file_id", data_type=DataType.TEXT),
        Property(name="page", data_type=DataType.INT),
        Property(name="depth", data_type=DataType.INT),
        Property(name="chunkset_index", data_type=DataType.INT),
    ],
)
chunksets = client.collections.get("Chunkset")

model = SentenceTransformer("all-MiniLM-L6-v2")
file_id = result.chunks[0].file_id
for cs in result.chunksets:
    leaf = cs.chunks[-1]  # document order; the last chunk index is the leaf
    chunksets.data.insert(
        properties={"content": cs.to_embed, "file_id": file_id,
                    "page": page_of[leaf], "depth": depth_of[leaf],
                    "chunkset_index": cs.chunkset_index},
        vector=model.encode(cs.to_embed).tolist(),
    )

# 4. Hybrid query: BM25F on content + your vector, blended by alpha.
query = "What are the early termination conditions?"
res = chunksets.query.hybrid(
    query=query,
    vector=model.encode(query).tolist(),
    alpha=0.5,
    filters=Filter.by_property("file_id").equal(file_id),
    limit=3,
)
cheatsheet = "\n\n".join(o.properties["content"] for o in res.objects)
client.close()

POMA validates the payload shape up front (a corrupted or mislabeled upload 422s immediately), prefers markdown.text when present and falls back to parsing_res_list sorted by block_order when it isn't — for every page, not just the majority shape — drops running furniture, and rebuilds the cross-page heading tree before chunking. To force or suppress detection, set external_ocr_source to "paddleocr_vl" or "none".

Metadata mapping: POMA fields → Weaviate primitives ​

POMA chunk fieldWeaviate primitiveWhat it enables
to_embedobject vector (bring-your-own)paraphrase retrieval
to_embed textcontent TEXT property, BM25F-indexedexact-term half of hybrid search
file_idTEXT property, Filter.by_propertyscope queries to one document
page (on the archive's chunk records, from PaddleOCR-VL page_index, 1-based; take the chunkset's leaf chunk)INT propertypage filters, page-cited answers
depth (of the chunkset's leaf chunk)INT propertyfilter/re-rank by hierarchy level
chunkset_indexINT propertystable ordering, independent of block_id
chunkset lineageoptional cross-reference to a Chunk collectionchunk-level provenance without a second store

Frequently asked questions ​

How do I get self-hosted PaddleOCR-VL results into Weaviate for RAG? ​

Save the raw /layout-parsing JSON, run PrimeCut().ingest() on it (auto-detected, hierarchy rebuilt), create a collection with file_id/page/depth properties, insert one object per chunkset with your own vector, and retrieve with query.hybrid. page comes from the archive's chunk records (the chunkset's leaf chunk) — the SDK objects expose depth but not page.

What happens if a naive pipeline only handles PaddleOCR-VL's markdown shape? ​

Pages that instead arrive as a raw save_to_json() block dict get skipped or crash the pipeline — they never become Weaviate objects, so they're missing from BM25F and vector search alike, not just degraded in one.

What properties should a Weaviate collection have for PaddleOCR-VL content? ​

content (TEXT, BM25F), file_id (TEXT), page (INT, from page_index, read off the archive's chunk records), depth (INT), chunkset_index (INT). POMA emits all of these regardless of which accepted shape a given page arrived in.

Can I self-host both PaddleOCR-VL and Weaviate for a fully on-premises pipeline? ​

Yes — both run as open source you operate yourself. Only the layout-parsing result JSON needs to reach POMA's chunking API; documents, embeddings, and the index itself can stay on-premises.

What hybrid alpha should I use in Weaviate for PaddleOCR-VL content? ​

Start at alpha=0.5. PP-DocLayoutV3 preserves exact tokens worth matching on BM25F — but only if blocks were assembled in block_order, not scrambled by sorting on block_id.

Same parser, different store: PaddleOCR-VL → Qdrant · PaddleOCR-VL → Pinecone · PaddleOCR-VL → Chroma

Same store, different parser: Mistral OCR → Weaviate · LlamaParse → Weaviate · Azure Document Intelligence → Weaviate

Foundations: The Optimal Chunker for PaddleOCR-VL · The Optimal Chunks for Weaviate · All pipeline recipes · RAG chunking guide