Skip to content

Marker to pgvector: Chunking for RAG ​

The short answer: Marker turns PDFs into clean structure on your own hardware; pgvector adds ANN search to the Postgres you already operate. Wired together naively — flatten the markdown, split, embed, INSERT into one flat table — the pairing wastes both: page identity is gone, and Postgres's defining strength, the JOIN, has nothing to join. The missing link is POMA: PrimeCut().ingest() consumes the saved Marker JSON (the Document block tree auto-routes), emits hierarchy-preserving chunksets, and those land as two joined tables — so one SQL statement runs the ANN search and retrieves each hit's full root-to-leaf lineage.

What each end of the pipeline actually provides ​

Marker (by Datalab / VikParuchuri) is the fast open-source PDF→markdown favourite for local pipelines. With the JSON renderer (--output_format json) it returns a Document block tree: a root block_type: "Document" with Page children, per-block HTML content, tables as full <table> elements with row and column spans, and a side-channel images dict of {name: base64}. What it doesn't provide: cross-page hierarchy or retrieval units — its job ends at "here is markdown." Details: The Optimal Chunker for Marker.

pgvector adds a vector column type, HNSW and IVFFlat indexes, and cosine/L2/inner-product operators to Postgres — alongside everything Postgres already does: transactions, B-tree indexes, tsvector full-text search, and JOINs. What it doesn't provide: any opinion about what a row should contain. It finds nearest neighbors of whatever you embedded — including context-free fragments. Details: The Optimal Chunks for the Best Retrieval in pgvector.

The naive wiring, and where it breaks ​

The common recipe — rendered.markdown → RecursiveCharacterTextSplitter → embed → INSERT INTO rag_fragments (text, embedding) — breaks in a pair-specific way:

  • The page column is NULL on every row. Marker's markdown output is one flat string with no page boundaries, so the schema's most useful filter column can never be populated. Page-cited answers are off the table before the first query.
  • One flat table means nothing to JOIN. Splitter fragments have no identity — no file_id/chunk_index key, no membership in anything larger — so the relational engine you chose pgvector for is reduced to a key-value store with a vector column.
  • Overlap bloats the HNSW graph. Splitter overlap embeds every boundary span twice; in pgvector that means a larger index and top-k results where rows 2 and 3 are near-duplicates of row 1.
  • Non-transactional trickle ingest. A script that embeds and inserts row by row, then dies mid-document, leaves a half-indexed document that silently answers queries with partial context.
  • Dangling ![](name) refs from Marker's side-channel images dict end up embedded and tsvector-indexed as noise tokens.

The schema: chunks and chunksets as joined tables ​

POMA's output is naturally relational — chunks are rows, chunksets are groups of rows — which is why pgvector is the one store where the model maps to the engine's native primitive:

sql
CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE chunks (
    file_id     TEXT    NOT NULL,
    chunk_index INTEGER NOT NULL,
    page        INTEGER,
    depth       INTEGER,
    content     TEXT    NOT NULL,
    PRIMARY KEY (file_id, chunk_index)
);

CREATE TABLE chunksets (
    id            BIGSERIAL PRIMARY KEY,
    file_id       TEXT      NOT NULL,
    chunk_indices INTEGER[] NOT NULL,
    to_embed      TEXT      NOT NULL,
    embedding     vector(384) NOT NULL,
    tsv           tsvector GENERATED ALWAYS AS (to_tsvector('english', to_embed)) STORED
);

CREATE INDEX ON chunksets USING hnsw (embedding vector_cosine_ops);
CREATE INDEX ON chunksets USING gin (tsv);
CREATE INDEX ON chunks (file_id, page);

The pipeline, end to end ​

bash
pip install marker-pdf poma "psycopg[binary]" pgvector sentence-transformers
marker_single contract.pdf --output_format json --output_dir out/
python
import os
import psycopg
from pgvector.psycopg import register_vector
from sentence_transformers import SentenceTransformer
from poma import PrimeCut
from poma.utils import unpack_poma_archive

# 1. The missing link — raw Marker JSON in, chunks + chunksets out.
poma = PrimeCut()  # reads POMA_API_KEY
result = poma.ingest(
    "out/contract/contract.json",  # Document block tree auto-detected
    download_dir="store",
    filename="contract.poma",  # keep the archive: its chunk records carry `page`
)
raw = unpack_poma_archive(poma_archive_path="store/contract.poma")

# 2. Embed each chunkset's normalized text.
model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
embeddings = model.encode([cs.to_embed for cs in result.chunksets])

# 3. Transactional ingest — the document lands atomically or not at all.
with psycopg.connect(os.environ["DATABASE_URL"]) as conn:
    register_vector(conn)
    with conn.transaction():
        for c in raw["chunks"]:
            conn.execute(
                "INSERT INTO chunks (file_id, chunk_index, page, depth, content)"
                " VALUES (%s, %s, %s, %s, %s)",
                (c["file_id"], c["chunk_index"], c.get("page"), c["depth"], c["content"]),
            )
        for cs, emb in zip(result.chunksets, embeddings):
            conn.execute(
                "INSERT INTO chunksets (file_id, chunk_indices, to_embed, embedding)"
                " VALUES (%s, %s, %s, %s)",
                (cs.file_id, cs.chunks, cs.to_embed, emb),  # cs.chunks is a list of chunk indices
            )

Retrieval is where the JOIN pays off — ANN search, full-text rank, and lineage in one statement:

python
query = "What are the early termination conditions?"
qvec = model.encode(query)

rows = conn.execute(
    """
    WITH hits AS (
        SELECT id, file_id, chunk_indices,
               1 - (embedding <=> %(v)s) AS dense_score,
               ts_rank(tsv, plainto_tsquery('english', %(q)s)) AS text_score
        FROM chunksets
        ORDER BY embedding <=> %(v)s
        LIMIT 5
    )
    SELECT h.dense_score + h.text_score AS score,
           c.chunk_index, c.page, c.depth, c.content
    FROM hits h
    JOIN chunks c
      ON c.file_id = h.file_id
     AND c.chunk_index = ANY (h.chunk_indices)
    ORDER BY score DESC, c.chunk_index
    """,
    {"v": qvec, "q": query},
).fetchall()

Deduplicate and merge the retrieved rows into a cheatsheet — one prompt-ready context block — before handing them to the LLM. POMA validates the upload shape up front (a corrupted or mislabeled payload 422s immediately), splices Marker's side-channel image bytes into described text, strips running headers and footers, and rebuilds the cross-page heading tree before chunking. Bare markdown shapes ({markdown, images, metadata}) are not auto-routed — declare external_ocr_source="marker", or "none" to opt out of detection.

The payoff, measured on our reference legal-document benchmark: 337 tokens of retrieved context instead of 1,542 for a recursive character splitter, zero information loss. Methodology: Document Ingestion & Chunking for RAG.

Metadata mapping: POMA fields → Postgres primitives ​

POMA chunk fieldPostgres primitiveWhat it enables
to_embedvector(384) + HNSW index; generated tsvector + GIN indexdense + full-text hybrid in one statement
file_idTEXT, part of the primary keyper-document scoping, the lineage JOIN key
page (from Marker's Page blocks)INTEGER, B-tree indexedpage-cited answers, page-range WHERE
depthINTEGERfilter or re-rank by hierarchy level
chunk_indexINTEGER, part of the primary keystable ordering at assembly time
chunkset lineagechunk_indices INTEGER[] + JOINfull root-to-leaf context in the ANN query

Frequently asked questions ​

How do I get Marker output into pgvector for RAG? ​

Run Marker with the JSON renderer, hand the saved result to PrimeCut().ingest() (auto-detected, hierarchy rebuilt), embed each chunkset's to_embed, and INSERT chunks and chunksets into two joined tables in one transaction. Retrieval is one SQL statement: ANN over chunkset embeddings, JOINed to chunks for lineage.

What Postgres schema should I use for Marker chunks with pgvector? ​

Two joined tables: chunks (file_id, chunk_index, page, depth, content) and chunksets with chunk_indices, to_embed, an HNSW-indexed vector column, and a GIN-indexed generated tsvector. One JOIN retrieves a chunkset's full root-to-leaf lineage alongside the ANN hit.

Can I combine pgvector vector search with Postgres full-text search on Marker output? ​

Yes, in one statement — rank by cosine distance and ts_rank together. Parsed documents are full of exact tokens (clause numbers, part codes, defined terms) that dense embeddings blur; the hybrid catches both, with no infrastructure beyond Postgres.

Why are page numbers missing when I load Marker output into Postgres? ​

Marker's default markdown output has no page boundaries, so the page column is NULL on every row. Use --output_format json: the Document block tree keeps per-page blocks, and POMA carries page onto every chunk.

Do I need new infrastructure to run RAG on Marker output? ​

No. Marker runs on your hardware; Postgres you already operate gets CREATE EXTENSION vector and two tables; ingest is a normal transaction. The only new piece is the chunking layer, and POMA's BYOCR connector fills it from the saved result — no re-parsing.

Same parser, different store: Marker → Qdrant · Marker → Milvus · Marker → Chroma

Same store, different parser: Mistral OCR → pgvector · Docling → pgvector · Unstructured → pgvector

Foundations: The Optimal Chunker for Marker · The Optimal Chunks for pgvector · All pipeline recipes · RAG chunking guide