Skip to content

Docling to pgvector: Chunking for RAG ​

The short answer: Docling gives you a typed document tree with explicit heading levels; pgvector gives you ANN search inside the relational engine you already run. Wired together naively — flatten to markdown, split, embed into one flat table — you throw away exactly the part each side is best at. The missing link is POMA: PrimeCut().ingest() consumes the saved DoclingDocument JSON (auto-detected), emits hierarchy-preserving chunksets, and those land in Postgres as what they naturally are — two joined tables, so one SQL statement runs the vector search and reconstructs the full document lineage.

What each end of the pipeline actually provides ​

Docling (IBM's open-source converter, also served via docling-serve) returns the DoclingDocument: a typed texts/tables/pictures/groups tree in which section_header items carry an explicit numeric level, tables are cell grids rather than pipe art, and repeating page furniture is pre-isolated in the furniture group with page_header/page_footer labels. What it doesn't provide: retrieval units. Its HybridChunker packs the tree into token windows — length control, not context. Details: The Optimal Chunker for Docling.

pgvector provides the vector column type, HNSW and IVFFlat indexes, distance operators that drop into ORDER BY, tsvector full-text search in the same engine, and transactions — all inside the Postgres you already operate. What it doesn't provide: any opinion about rows. It orders whatever you inserted by whatever distance you ask for. Details: The Optimal Chunks for the Best Retrieval in pgvector.

The naive wiring, and where it breaks ​

The common recipe — export_to_markdown() → RecursiveCharacterTextSplitter → embed → one flat documents(text, embedding) table — breaks in a pair-specific way: you are running a relational database and storing zero relations.

  • Docling's explicit section_header levels are parent–child data — precisely what Postgres was built to store and JOIN. Flattening reduces them to # glyphs before the first INSERT, so there is no depth column to filter on and no lineage to reconstruct. The one store in this series that could hold the hierarchy as rows gets handed a string.
  • Page provenance dies at flattening, so the schema has no page column, and page-cited answers or indexed page-range predicates are off the table.
  • Splitter overlap embeds every boundary span twice — a fatter chunksets table, a slower HNSW build, and LIMIT 5 results that repeat each other.
  • Multi-step ingest scripts without a transaction leave a half-ingested corpus after a crash — in the one store that offers atomicity for free.

The pipeline, end to end ​

bash
pip install docling poma "psycopg[binary]" pgvector

The schema stores POMA's output as what it is — a hierarchy:

sql
CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE chunks (
    id          bigserial PRIMARY KEY,
    file_id     text NOT NULL,
    chunk_index int  NOT NULL,
    depth       int  NOT NULL,   -- from Docling's explicit section_header levels
    page        int,
    content     text NOT NULL,   -- tables arrive as HTML, never cut mid-row
    UNIQUE (file_id, chunk_index)
);

CREATE TABLE chunksets (
    id        bigserial PRIMARY KEY,
    file_id   text NOT NULL,
    to_embed  text NOT NULL,
    embedding vector(384) NOT NULL,
    tsv       tsvector GENERATED ALWAYS AS (to_tsvector('english', to_embed)) STORED
);

CREATE TABLE chunkset_members (
    chunkset_id bigint NOT NULL REFERENCES chunksets(id) ON DELETE CASCADE,
    chunk_id    bigint NOT NULL REFERENCES chunks(id)    ON DELETE CASCADE,
    position    int    NOT NULL, -- 0 = document root … last = the leaf
    PRIMARY KEY (chunkset_id, chunk_id)
);

CREATE INDEX ON chunksets USING hnsw (embedding vector_cosine_ops);
CREATE INDEX ON chunksets USING gin (tsv);

Ingest is one transaction — the document and every chunk land atomically, or not at all:

python
import json
import os

import psycopg
from docling.document_converter import DocumentConverter
from pgvector.psycopg import register_vector
from poma import PrimeCut
from poma.utils import unpack_poma_archive

# 1. Docling — your existing conversion, unchanged.
converter = DocumentConverter()
doc = converter.convert("contract.pdf").document
with open("contract.docling.json", "w") as f:
    json.dump(doc.export_to_dict(), f)

# 2. The missing link — DoclingDocument tree in, chunks + chunksets out.
poma = PrimeCut()  # reads POMA_API_KEY
result = poma.ingest(
    "contract.docling.json",  # Docling shape auto-detected
    download_dir="store",
    filename="contract.poma",  # keep the archive: its chunk records carry `page`
)
raw = unpack_poma_archive(poma_archive_path="store/contract.poma")

# 3. Postgres — hierarchy as rows, atomically.
with psycopg.connect(os.environ["DATABASE_URL"]) as conn:
    register_vector(conn)
    with conn.transaction():
        chunk_ids = {}
        for c in raw["chunks"]:
            row = conn.execute(
                "INSERT INTO chunks (file_id, chunk_index, depth, page, content) "
                "VALUES (%s, %s, %s, %s, %s) RETURNING id",
                (c["file_id"], c["chunk_index"], c["depth"], c.get("page"), c["content"]),
            ).fetchone()
            chunk_ids[c["chunk_index"]] = row[0]

        for cs in result.chunksets:
            row = conn.execute(
                "INSERT INTO chunksets (file_id, to_embed, embedding) "
                "VALUES (%s, %s, %s) RETURNING id",
                (cs.file_id, cs.to_embed, embed(cs.to_embed)),  # your embedding model
            ).fetchone()
            for position, chunk_index in enumerate(cs.chunks):
                conn.execute(
                    "INSERT INTO chunkset_members (chunkset_id, chunk_id, position) "
                    "VALUES (%s, %s, %s)",
                    (row[0], chunk_ids[chunk_index], position),
                )

Retrieval is where the pairing pays off — ANN search and full lineage reconstruction in a single statement:

sql
WITH hits AS (
    SELECT id, embedding <=> %(query_vec)s AS distance
    FROM chunksets
    WHERE file_id = %(file_id)s          -- ordinary indexed predicate
    ORDER BY embedding <=> %(query_vec)s
    LIMIT 5
)
SELECT h.distance, m.position, c.depth, c.page, c.content
FROM hits h
JOIN chunkset_members m ON m.chunkset_id = h.id
JOIN chunks c           ON c.id = m.chunk_id
ORDER BY h.distance, m.position;

POMA validates the payload up front (a corrupted or mislabeled upload 422s immediately, never a silently degraded index), honors the furniture Docling already isolated, keeps tables as HTML, and accepts docling-serve md_content envelopes via a markdown passthrough. To force or suppress detection, set external_ocr_source to "docling" or "none". The returned rows merge into a deduplicated, prompt-ready cheatsheet — on the reference legal-document benchmark, 337 tokens of context versus 1,542 for a recursive-splitter baseline, with zero information loss (methodology).

Metadata mapping: POMA fields → pgvector primitives ​

POMA chunk fieldPostgres primitiveWhat it enables
to_embedvector(384) column (HNSW) + generated tsvector (GIN)dense ANN + keyword hybrid in one engine
file_idindexed text column on both tablesper-document scoping; the join key of the schema
pageint columnpage-cited answers, page-range predicates
depth (from Docling section_header level)int columnWHERE depth <= 2, structural re-ranking
chunk_indexint column, UNIQUE (file_id, chunk_index)stable ordering at assembly time
chunkset lineagechunkset_members join tableroot-to-leaf lineage in the same query as ANN

Frequently asked questions ​

How do I get Docling output into pgvector for RAG? ​

Save export_to_dict() as JSON, run PrimeCut().ingest() on it (auto-detected via schema_name), insert chunks and chunksets into two joined tables in one transaction, embed to_embed into a vector column with HNSW. Retrieval is one SQL statement: an ANN CTE joined back to chunks for full lineage.

What Postgres schema should I use for Docling chunks? ​

Three tables: chunks (file_id, chunk_index, depth, page, content), chunksets (to_embed, vector, generated tsvector), and chunkset_members linking them. Docling's explicit section_header levels become the depth integer — hierarchy as indexed columns, not flattened text.

Can I query Docling's heading levels with SQL in pgvector? ​

Yes. POMA writes each section_header item's explicit level to every chunk as depth. In an integer column, heading levels are ordinary SQL: WHERE depth <= 2 scopes to top-level sections, and a JOIN returns every ancestor of a matched chunkset alongside the vector search.

How do I combine vector search and keyword search for Docling documents in Postgres? ​

A generated tsvector column over to_embed, indexed with GIN, next to the HNSW index. Fuse the two ranked lists with reciprocal rank fusion in plain SQL. Exact tokens Docling preserved faithfully come from tsvector; paraphrases come from the embedding.

Do I need Docling's HybridChunker if I store chunks in pgvector? ​

No — token windows underuse Postgres. They control length but leave rows without machine-readable lineage. Chunksets keep hierarchy as relations: lineage is a JOIN, depth is a WHERE clause, and every retrieved unit is self-explanatory.

Same parser, different store: Docling → Qdrant · Docling → Milvus · Docling → Chroma

Same store, different parser: Mistral OCR → pgvector · Unstructured.io → pgvector · Textract → pgvector

Foundations: The Optimal Chunker for Docling · The Optimal Chunks for pgvector · All pipeline recipes · RAG chunking guide