PaddleOCR-VL to Weaviate: Chunking for RAG
The short answer: PaddleOCR-VL gives you self-hosted, layout-aware OCR behind your own HTTP endpoint; Weaviate gives you built-in hybrid search — BM25F plus vectors, blended by alpha — over filterable properties. Wired together naively — flatten, split, embed — they still produce mediocre RAG, and a pipeline that only handles one of PaddleOCR-VL's two accepted shapes can silently drop whole pages from both halves of the hybrid at once. The missing link is POMA: PrimeCut().ingest() consumes the raw layout-parsing JSON (auto-detected, either shape), emits hierarchy-preserving chunksets, and each becomes one Weaviate object with file_id/page/depth properties and your embedding as its vector.
What each end of the pipeline actually provides
PaddleOCR-VL runs self-hosted — PP-DocLayoutV3 for layout detection, PaddleOCR-VL served via vLLM for reading, behind a /layout-parsing HTTP API you operate. It returns either the PaddleX serving envelope (result.layoutParsingResults[], with a markdown object plus a prunedResult block list) or a raw save_to_json() page dict (parsing_res_list: flat blocks with block_label, block_content, block_id, block_order). Both are the tool's own fingerprints, and a batch job can legitimately emit either one per document. Details: The Optimal Chunker for PaddleOCR-VL.
Weaviate provides collections with named vectors, built-in hybrid search (BM25F + vector, blended by alpha), filterable properties, cross-references between collections, and generative/reranker modules — open source or Weaviate Cloud. What it doesn't provide: any opinion about what an object should contain. BM25F scores whatever text you stored; the vector index ranks whatever you embedded — nothing, if a page never made it into a collection object at all. Details: The Optimal Chunks for the Best Retrieval in Weaviate.
The naive wiring, and where it breaks
The common recipe — call /layout-parsing, assume every response has markdown.text, split, embed, data.insert(...) — breaks in a pair-specific way:
- Handling only one accepted shape drops pages, not just quality. Code written against the serving envelope's
markdownobject throws or silently skips any page that instead arrives as a baresave_to_json()block dict — a real possibility across a self-hosted batch pipeline processing many documents over time. That page is absent from both BM25F and vector search, not degraded in either. page_indexvanishes at the join, so the one property Weaviate needs for page filters and page-cited answers holds nothing truthful.- Overlap and block-order mistakes pollute both hybrid halves at once. Splitter overlap duplicates boundary spans in both BM25F and vector rankings; sorting blocks by
block_idinstead ofblock_orderscrambles the exact tokens BM25F depends on.
Chunksets fix this at the unit level: both PaddleOCR-VL shapes route through the same hierarchy-preserving assembly, no page silently dropped, no overlap, exact tokens intact. On a reference legal document, assembled cheatsheets delivered 337 tokens of retrieved context versus 1,542 from a recursive character splitter, with zero information loss (methodology).
The pipeline, end to end
pip install requests poma weaviate-client sentence-transformersimport json
import os
import requests
import weaviate
from weaviate.classes.config import Configure, DataType, Property
from weaviate.classes.init import Auth
from weaviate.classes.query import Filter
from poma import PrimeCut
from poma.utils import unpack_poma_archive
from sentence_transformers import SentenceTransformer
# 1. Your existing self-hosted call — unchanged. `/layout-parsing` is the
# PaddleX-compatible endpoint your layout-server + PaddleOCR-VL stack exposes.
resp = requests.post(
"http://layout-server:40111/layout-parsing",
json={"file": "<base64-encoded PDF>", "fileType": 0},
)
resp.raise_for_status()
with open("contract.paddleocr-vl.json", "w") as f:
json.dump(resp.json(), f)
# 2. The missing link — raw result JSON in, chunks + chunksets out.
poma = PrimeCut() # reads POMA_API_KEY
# keep the .poma archive: page numbers live on its chunk records, not on the SDK objects
result = poma.ingest(
"contract.paddleocr-vl.json", # PaddleOCR-VL shape auto-detected
download_dir="store",
filename="contract.poma",
)
# Chunk lookups: depth from the SDK objects, page from the archive's chunk records.
depth_of = {c.chunk_index: c.depth for c in result.chunks}
page_of = {
c["chunk_index"]: c.get("page")
for c in unpack_poma_archive(poma_archive_path="store/contract.poma")["chunks"]
}
# 3. One collection of chunksets, hierarchy as filterable properties.
client = weaviate.connect_to_weaviate_cloud(
cluster_url=os.environ["WEAVIATE_URL"],
auth_credentials=Auth.api_key(os.environ["WEAVIATE_API_KEY"]),
)
client.collections.create(
"Chunkset",
vectorizer_config=Configure.Vectorizer.none(), # bring your own vectors
properties=[
Property(name="content", data_type=DataType.TEXT),
Property(name="file_id", data_type=DataType.TEXT),
Property(name="page", data_type=DataType.INT),
Property(name="depth", data_type=DataType.INT),
Property(name="chunkset_index", data_type=DataType.INT),
],
)
chunksets = client.collections.get("Chunkset")
model = SentenceTransformer("all-MiniLM-L6-v2")
file_id = result.chunks[0].file_id
for cs in result.chunksets:
leaf = cs.chunks[-1] # document order; the last chunk index is the leaf
chunksets.data.insert(
properties={"content": cs.to_embed, "file_id": file_id,
"page": page_of[leaf], "depth": depth_of[leaf],
"chunkset_index": cs.chunkset_index},
vector=model.encode(cs.to_embed).tolist(),
)
# 4. Hybrid query: BM25F on content + your vector, blended by alpha.
query = "What are the early termination conditions?"
res = chunksets.query.hybrid(
query=query,
vector=model.encode(query).tolist(),
alpha=0.5,
filters=Filter.by_property("file_id").equal(file_id),
limit=3,
)
cheatsheet = "\n\n".join(o.properties["content"] for o in res.objects)
client.close()POMA validates the payload shape up front (a corrupted or mislabeled upload 422s immediately), prefers markdown.text when present and falls back to parsing_res_list sorted by block_order when it isn't — for every page, not just the majority shape — drops running furniture, and rebuilds the cross-page heading tree before chunking. To force or suppress detection, set external_ocr_source to "paddleocr_vl" or "none".
Metadata mapping: POMA fields → Weaviate primitives
| POMA chunk field | Weaviate primitive | What it enables |
|---|---|---|
to_embed | object vector (bring-your-own) | paraphrase retrieval |
to_embed text | content TEXT property, BM25F-indexed | exact-term half of hybrid search |
file_id | TEXT property, Filter.by_property | scope queries to one document |
page (on the archive's chunk records, from PaddleOCR-VL page_index, 1-based; take the chunkset's leaf chunk) | INT property | page filters, page-cited answers |
depth (of the chunkset's leaf chunk) | INT property | filter/re-rank by hierarchy level |
chunkset_index | INT property | stable ordering, independent of block_id |
| chunkset lineage | optional cross-reference to a Chunk collection | chunk-level provenance without a second store |
Frequently asked questions
How do I get self-hosted PaddleOCR-VL results into Weaviate for RAG?
Save the raw /layout-parsing JSON, run PrimeCut().ingest() on it (auto-detected, hierarchy rebuilt), create a collection with file_id/page/depth properties, insert one object per chunkset with your own vector, and retrieve with query.hybrid. page comes from the archive's chunk records (the chunkset's leaf chunk) — the SDK objects expose depth but not page.
What happens if a naive pipeline only handles PaddleOCR-VL's markdown shape?
Pages that instead arrive as a raw save_to_json() block dict get skipped or crash the pipeline — they never become Weaviate objects, so they're missing from BM25F and vector search alike, not just degraded in one.
What properties should a Weaviate collection have for PaddleOCR-VL content?
content (TEXT, BM25F), file_id (TEXT), page (INT, from page_index, read off the archive's chunk records), depth (INT), chunkset_index (INT). POMA emits all of these regardless of which accepted shape a given page arrived in.
Can I self-host both PaddleOCR-VL and Weaviate for a fully on-premises pipeline?
Yes — both run as open source you operate yourself. Only the layout-parsing result JSON needs to reach POMA's chunking API; documents, embeddings, and the index itself can stay on-premises.
What hybrid alpha should I use in Weaviate for PaddleOCR-VL content?
Start at alpha=0.5. PP-DocLayoutV3 preserves exact tokens worth matching on BM25F — but only if blocks were assembled in block_order, not scrambled by sorting on block_id.
Related recipes
Same parser, different store: PaddleOCR-VL → Qdrant · PaddleOCR-VL → Pinecone · PaddleOCR-VL → Chroma
Same store, different parser: Mistral OCR → Weaviate · LlamaParse → Weaviate · Azure Document Intelligence → Weaviate
Foundations: The Optimal Chunker for PaddleOCR-VL · The Optimal Chunks for Weaviate · All pipeline recipes · RAG chunking guide