Source: http://www.poma-ai.com/docs/pipelines/marker-to-pgvector

# The Missing Link Between Marker and Optimal Retrieval in pgvector

<ByAuthor />

**The short answer:** Marker turns PDFs into clean structure on your own hardware; pgvector adds ANN search to the Postgres you already operate. Wired together naively — flatten the markdown, split, embed, INSERT into one flat table — the pairing wastes both: page identity is gone, and Postgres's defining strength, the JOIN, has nothing to join. The missing link is POMA: `PrimeCut().ingest()` consumes the saved Marker JSON (the Document block tree auto-routes), emits hierarchy-preserving [chunksets](/learn/chunking/chunksets), and those land as two joined tables — so one SQL statement runs the ANN search *and* retrieves each hit's full root-to-leaf lineage.

## What each end of the pipeline actually provides

**Marker** (by Datalab / VikParuchuri) is the fast open-source PDF→markdown favourite for local pipelines. With the JSON renderer (`--output_format json`) it returns a Document block tree: a root `block_type: "Document"` with Page children, per-block HTML content, tables as full `<table>` elements with row and column spans, and a side-channel `images` dict of `{name: base64}`. What it doesn't provide: cross-page hierarchy or retrieval units — its job ends at "here is markdown." Details: [The Optimal Chunker for Marker](/optimal-chunker-marker).

**pgvector** adds a `vector` column type, HNSW and IVFFlat indexes, and cosine/L2/inner-product operators to Postgres — alongside everything Postgres already does: transactions, B-tree indexes, `tsvector` full-text search, and JOINs. What it doesn't provide: any opinion about what a row should contain. It finds nearest neighbors of whatever you embedded — including context-free fragments. Details: [The Optimal Chunks for the Best Retrieval in pgvector](/optimal-chunks-pgvector).

## The naive wiring, and where it breaks

The common recipe — `rendered.markdown` → `RecursiveCharacterTextSplitter` → embed → `INSERT INTO rag_fragments (text, embedding)` — breaks in a pair-specific way:

- **The `page` column is NULL on every row.** Marker's markdown output is one flat string with no page boundaries, so the schema's most useful filter column can never be populated. Page-cited answers are off the table before the first query.
- **One flat table means nothing to JOIN.** Splitter fragments have no identity — no `file_id`/`chunk_index` key, no membership in anything larger — so the relational engine you chose pgvector for is reduced to a key-value store with a vector column.
- **Overlap bloats the HNSW graph.** Splitter overlap embeds every boundary span twice; in pgvector that means a larger index and top-k results where rows 2 and 3 are near-duplicates of row 1.
- **Non-transactional trickle ingest.** A script that embeds and inserts row by row, then dies mid-document, leaves a half-indexed document that silently answers queries with partial context.
- **Dangling `![](name)` refs** from Marker's side-channel images dict end up embedded and `tsvector`-indexed as noise tokens.

## The schema: chunks and chunksets as joined tables

POMA's output is naturally relational — chunks are rows, chunksets are groups of rows — which is why pgvector is the one store where the model maps to the engine's native primitive:

```sql
CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE chunks (
    file_id     TEXT    NOT NULL,
    chunk_index INTEGER NOT NULL,
    page        INTEGER,
    depth       INTEGER,
    content     TEXT    NOT NULL,
    PRIMARY KEY (file_id, chunk_index)
);

CREATE TABLE chunksets (
    id            BIGSERIAL PRIMARY KEY,
    file_id       TEXT      NOT NULL,
    chunk_indices INTEGER[] NOT NULL,
    to_embed      TEXT      NOT NULL,
    embedding     vector(384) NOT NULL,
    tsv           tsvector GENERATED ALWAYS AS (to_tsvector('english', to_embed)) STORED
);

CREATE INDEX ON chunksets USING hnsw (embedding vector_cosine_ops);
CREATE INDEX ON chunksets USING gin (tsv);
CREATE INDEX ON chunks (file_id, page);
```

## The pipeline, end to end

```bash
pip install marker-pdf poma "psycopg[binary]" pgvector sentence-transformers
marker_single contract.pdf --output_format json --output_dir out/
```

```python
import os
import psycopg
from pgvector.psycopg import register_vector
from sentence_transformers import SentenceTransformer
from poma import PrimeCut

# 1. The missing link — raw Marker JSON in, chunks + chunksets out.
poma = PrimeCut()  # reads POMA_API_KEY
result = poma.ingest("out/contract/contract.json")  # Document block tree auto-detected

# 2. Embed each chunkset's normalized text.
model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
embeddings = model.encode([cs.to_embed for cs in result.chunksets])

# 3. Transactional ingest — the document lands atomically or not at all.
with psycopg.connect(os.environ["DATABASE_URL"]) as conn:
    register_vector(conn)
    with conn.transaction():
        for chunk in result.chunks:
            conn.execute(
                "INSERT INTO chunks (file_id, chunk_index, page, depth, content)"
                " VALUES (%s, %s, %s, %s, %s)",
                (chunk.file_id, chunk.chunk_index, chunk.page, chunk.depth, chunk.content),
            )
        for cs, emb in zip(result.chunksets, embeddings):
            conn.execute(
                "INSERT INTO chunksets (file_id, chunk_indices, to_embed, embedding)"
                " VALUES (%s, %s, %s, %s)",
                (cs.file_id, [c.chunk_index for c in cs.chunks], cs.to_embed, emb),
            )
```

Retrieval is where the JOIN pays off — ANN search, full-text rank, and lineage in one statement:

```python
query = "What are the early termination conditions?"
qvec = model.encode(query)

rows = conn.execute(
    """
    WITH hits AS (
        SELECT id, file_id, chunk_indices,
               1 - (embedding <=> %(v)s) AS dense_score,
               ts_rank(tsv, plainto_tsquery('english', %(q)s)) AS text_score
        FROM chunksets
        ORDER BY embedding <=> %(v)s
        LIMIT 5
    )
    SELECT h.dense_score + h.text_score AS score,
           c.chunk_index, c.page, c.depth, c.content
    FROM hits h
    JOIN chunks c
      ON c.file_id = h.file_id
     AND c.chunk_index = ANY (h.chunk_indices)
    ORDER BY score DESC, c.chunk_index
    """,
    {"v": qvec, "q": query},
).fetchall()
```

Deduplicate and merge the retrieved rows into a cheatsheet — one prompt-ready context block — before handing them to the LLM. POMA validates the upload shape up front (a corrupted or mislabeled payload 422s immediately), splices Marker's side-channel image bytes into described text, strips running headers and footers, and rebuilds the cross-page heading tree before chunking. Bare markdown shapes (`{markdown, images, metadata}`) are not auto-routed — declare `external_ocr_source="marker"`, or `"none"` to opt out of detection.

The payoff, measured on our reference legal-document benchmark: **337 tokens** of retrieved context instead of **1,542** for a recursive character splitter, zero information loss. Methodology: [Document Ingestion & Chunking for RAG](/document-ingestion-chunking-rag).

## Metadata mapping: POMA fields → Postgres primitives

| POMA chunk field | Postgres primitive | What it enables |
| --- | --- | --- |
| `to_embed` | `vector(384)` + HNSW index; generated `tsvector` + GIN index | dense + full-text hybrid in one statement |
| `file_id` | `TEXT`, part of the primary key | per-document scoping, the lineage JOIN key |
| `page` (from Marker's Page blocks) | `INTEGER`, B-tree indexed | page-cited answers, page-range `WHERE` |
| `depth` | `INTEGER` | filter or re-rank by hierarchy level |
| `chunk_index` | `INTEGER`, part of the primary key | stable ordering at assembly time |
| chunkset lineage | `chunk_indices INTEGER[]` + `JOIN` | full root-to-leaf context in the ANN query |

## Frequently asked questions

### How do I get Marker output into pgvector for RAG?

Run Marker with the JSON renderer, hand the saved result to `PrimeCut().ingest()` (auto-detected, hierarchy rebuilt), embed each chunkset's `to_embed`, and INSERT chunks and chunksets into two joined tables in one transaction. Retrieval is one SQL statement: ANN over chunkset embeddings, JOINed to `chunks` for lineage.

### What Postgres schema should I use for Marker chunks with pgvector?

Two joined tables: `chunks (file_id, chunk_index, page, depth, content)` and `chunksets` with `chunk_indices`, `to_embed`, an HNSW-indexed `vector` column, and a GIN-indexed generated `tsvector`. One JOIN retrieves a chunkset's full root-to-leaf lineage alongside the ANN hit.

### Can I combine pgvector vector search with Postgres full-text search on Marker output?

Yes, in one statement — rank by cosine distance and `ts_rank` together. Parsed documents are full of exact tokens (clause numbers, part codes, defined terms) that dense embeddings blur; the hybrid catches both, with no infrastructure beyond Postgres.

### Why are page numbers missing when I load Marker output into Postgres?

Marker's default markdown output has no page boundaries, so the `page` column is NULL on every row. Use `--output_format json`: the Document block tree keeps per-page blocks, and POMA carries `page` onto every chunk.

### Do I need new infrastructure to run RAG on Marker output?

No. Marker runs on your hardware; Postgres you already operate gets `CREATE EXTENSION vector` and two tables; ingest is a normal transaction. The only new piece is the chunking layer, and POMA's BYOCR connector fills it from the saved result — no re-parsing.

## Related recipes

Same parser, different store: [Marker → Qdrant](/pipelines/marker-to-qdrant) · [Marker → Milvus](/pipelines/marker-to-milvus) · [Marker → Chroma](/pipelines/marker-to-chroma)

Same store, different parser: [Mistral OCR → pgvector](/pipelines/mistral-ocr-to-pgvector) · [Docling → pgvector](/pipelines/docling-to-pgvector) · [Unstructured → pgvector](/pipelines/unstructured-to-pgvector)

Foundations: [The Optimal Chunker for Marker](/optimal-chunker-marker) · [The Optimal Chunks for pgvector](/optimal-chunks-pgvector) · [All pipeline recipes](/pipelines/) · [RAG chunking guide](/guides/rag-chunking/)