pgvector Chunking Strategy for RAG: The Optimal Chunks for Retrieval
The short answer: the optimal chunks for pgvector are chunksets — self-explanatory units that carry their full heading lineage — stored as two joined tables: chunks for every typed unit and chunksets for the embedded retrieval units, with file_id, page, and depth as plain indexed columns. Because pgvector lives inside Postgres, one SQL statement can run the ANN search and JOIN back the full document lineage, something standalone vector stores need a second round trip for. POMA's PrimeCut emits exactly this relational shape from any document.
This page covers what pgvector does well, what it cannot do for you, and the schema plus query design that gets the most out of it.
pgvector puts vector search inside the database you already run
For teams already on Postgres, pgvector is the lowest-friction path to RAG — no new infrastructure, no second system of record:
- The
vectorcolumn type — embeddings live next to your relational data, governed by the same backups, roles, and replication. - HNSW and IVFFlat indexes — approximate nearest-neighbor search at scale, chosen per table to trade build cost against recall.
- Distance operators —
<=>(cosine distance),<->(L2),<#>(inner product) drop straight intoORDER BY. tsvectorin the same engine — Postgres full-text search gives you the keyword half of hybrid retrieval without another service.- Transactions — a document and all of its chunks land atomically, or not at all. No half-ingested corpus after a crashed pipeline.
None of these primitives, however, know anything about your documents. Postgres orders rows by whatever distance you ask for, over whatever vectors you inserted. If what you embedded is a context-free fragment — a paragraph cut loose from the section that gives it meaning — pgvector will faithfully return context-free fragments, in perfectly correct distance order. The ceiling on retrieval quality is set before the first INSERT.
What "optimal chunks" means for pgvector, concretely
Four properties separate a Postgres RAG schema that answers questions from one that returns trivia:
- Every retrieved row is self-explanatory. A chunkset is a root-to-leaf path through the document —
Master Services Agreement → Termination clauses → Early termination → "Either party may…". Retrieved alone, it still tells the LLM (and the human reading citations) exactly where it sits. A bare 512-token window does not. See POMA chunksets. - The hierarchy is relational, not stringly. This is pgvector's unique advantage:
chunksandchunksetsare naturally two tables joined by a membership relation. Where other stores flatten lineage into a metadata blob, Postgres keeps it as rows — so aJOINreconstructs the full ancestry in the same query as the ANN search, andWHERE file_id = …ordepth <= 2are ordinary indexed predicates. - No overlap, no near-duplicates. Overlap embeds every overlapped span twice: a fatter table, a slower HNSW build, and top-k results that repeat each other. Chunksets carry context structurally, so overlap is simply unnecessary and your
LIMIT 5returns five different candidates. - Hybrid with
tsvector. Legal, financial, and technical corpora are full of exact tokens (§ numbers, SKUs, defined terms) that dense embeddings smear. A generatedtsvectorcolumn with a GIN index gives you the keyword lane in the same engine — fuse the two ranked lists with reciprocal rank fusion in plain SQL.
The schema: chunks and chunksets as two joined tables
PrimeCut turns any document — PDF, DOCX, HTML, or a bring-your-own-OCR result JSON — into typed chunks and chunksets. In Postgres, that output maps onto a schema you would design anyway:
CREATE EXTENSION IF NOT EXISTS vector;
CREATE TABLE chunks (
id bigserial PRIMARY KEY,
file_id text NOT NULL,
chunk_index int NOT NULL,
depth int NOT NULL, -- hierarchy level: 0 = title, deeper = finer
page int,
content text NOT NULL, -- tables arrive as HTML, never cut mid-row
UNIQUE (file_id, chunk_index)
);
CREATE TABLE chunksets (
id bigserial PRIMARY KEY,
file_id text NOT NULL,
to_embed text NOT NULL, -- normalized text of the root-to-leaf unit
embedding vector(384) NOT NULL,
tsv tsvector GENERATED ALWAYS AS (to_tsvector('english', to_embed)) STORED
);
CREATE TABLE chunkset_members (
chunkset_id bigint NOT NULL REFERENCES chunksets(id) ON DELETE CASCADE,
chunk_id bigint NOT NULL REFERENCES chunks(id) ON DELETE CASCADE,
position int NOT NULL, -- 0 = document root … last = the leaf
PRIMARY KEY (chunkset_id, chunk_id)
);
CREATE INDEX ON chunksets USING hnsw (embedding vector_cosine_ops);
CREATE INDEX ON chunksets USING gin (tsv);
CREATE INDEX ON chunks (file_id, chunk_index);Ingest is one transaction — the document and every chunk land atomically:
import psycopg
from poma import PrimeCut
from poma.utils import unpack_poma_archive
client = PrimeCut() # reads POMA_API_KEY
result = client.ingest("contract.pdf", download_dir="store", filename="contract.poma")
raw = unpack_poma_archive(poma_archive_path="store/contract.poma") # chunk records carry `page`
with psycopg.connect() as conn, conn.transaction():
chunk_ids = {}
for c in raw["chunks"]:
row = conn.execute(
"INSERT INTO chunks (file_id, chunk_index, depth, page, content) "
"VALUES (%s, %s, %s, %s, %s) RETURNING id",
(c["file_id"], c["chunk_index"], c["depth"], c.get("page"), c["content"]),
).fetchone()
chunk_ids[c["chunk_index"]] = row[0]
for cs in result.chunksets:
row = conn.execute(
"INSERT INTO chunksets (file_id, to_embed, embedding) "
"VALUES (%s, %s, %s) RETURNING id",
(cs.file_id, cs.to_embed, embed(cs.to_embed)), # your embedding model
).fetchone()
for position, chunk_index in enumerate(cs.chunks):
conn.execute(
"INSERT INTO chunkset_members (chunkset_id, chunk_id, position) "
"VALUES (%s, %s, %s)",
(row[0], chunk_ids[chunk_index], position),
)And retrieval is where the relational design pays off — ANN search and full lineage reconstruction in a single statement:
WITH hits AS (
SELECT id, file_id, embedding <=> %(query_vec)s AS distance
FROM chunksets
WHERE file_id = %(file_id)s -- ordinary indexed predicate
ORDER BY embedding <=> %(query_vec)s
LIMIT 5
)
SELECT h.id AS chunkset_id,
h.distance,
m.position,
c.depth,
c.page,
c.content
FROM hits h
JOIN chunkset_members m ON m.chunkset_id = h.id
JOIN chunks c ON c.id = m.chunk_id
ORDER BY h.distance, m.position;Each hit comes back as ordered rows from root to leaf: every ancestor heading, its page, its depth. Feed those rows to POMA's cheatsheet assembly — retrieved chunksets share ancestors, and merging deduplicates the lineage into one coherent, prompt-ready block. On one reference legal document that meant 337 tokens of context versus 1,542 for a recursive-splitter baseline, with nothing lost. One document is an illustration, not a benchmark; the ingestion guide has the broader numbers.
Chunk shapes in pgvector, compared
| Property | Fixed 512-token chunks + overlap | Semantic chunks | POMA chunksets |
|---|---|---|---|
| Row is self-explanatory when retrieved alone | ✗ | Sometimes | ✓ (by construction) |
| Near-duplicate vectors in the HNSW/IVFFlat index | Many (overlap) | Few | None |
Hierarchy queryable with SQL (JOIN, WHERE depth…) | ✗ (flat text) | ✗ | ✓ (two joined tables) |
| Lineage retrieved in the same query as ANN search | ✗ | ✗ | ✓ (one JOIN) |
| Tables survive embedding | ✗ (cut mid-row) | Fragile | ✓ (HTML, never cut) |
Hybrid with tsvector | DIY over fragments | DIY | ✓ (generated column on to_embed) |
| Post-retrieval assembly | Concatenate raw hits | Concatenate raw hits | Cheatsheets (deduped lineage) |
Frequently asked questions
What is the optimal chunk size for pgvector?
There is no universal token count — the optimal unit is a chunk that is self-explanatory without its neighbors. Chunksets achieve that at any size, which is why they beat fixed 512-token windows: every row returned by an ANN query can be handed to the LLM alone and still be understood. If you must fix a size, start at 512 tokens — but the size knob cannot fix missing context.
What Postgres schema should I use for RAG chunks?
Two joined tables: chunks (file_id, chunk_index, depth, page, content) and chunksets (to_embed, vector column), linked by a membership table. The document's real hierarchy becomes relational, so a single JOIN reconstructs a retrieved chunkset's full root-to-leaf lineage in the same query that runs the vector search.
Should I use HNSW or IVFFlat in pgvector for RAG?
For most RAG workloads, HNSW: better recall at a given query latency, and it builds incrementally on an empty table. IVFFlat builds faster and uses less memory, but must be created after data exists and its recall depends on lists/probes tuning. Neither can compensate for context-free chunks.
How do I combine pgvector with full-text search for hybrid RAG?
Add a generated tsvector column over the chunkset's to_embed text, index it with GIN, and run to_tsquery matching alongside the vector query. Fuse the two ranked lists — reciprocal rank fusion is expressible in plain SQL with two CTEs and a FULL OUTER JOIN.
How does chunk overlap affect a pgvector index?
Every overlapped span is embedded twice: a fatter table, a slower HNSW build, and near-duplicate hits crowding out diverse results. Chunks that carry their hierarchy don't need overlap.
Can I retrieve a chunk's document hierarchy in the same SQL query as the vector search?
Yes — put the ANN search in a CTE (ORDER BY embedding <=> query_vector LIMIT k), then JOIN to the membership and chunks tables. One round trip returns each hit with every ancestor heading, page, and depth — the raw material for cheatsheet assembly.
First-party ingest: vektoria's push()
pgvector is one of only two databases (with Chroma) that has a real first-party vektoria ingest connector — one call handles chunking-to-vector in a content-free shape:
from poma.vektoria import push, get_connector
from poma.embeddings import get_embedder
stats = push(
"contract.poma",
volume="s3://your-bucket/poma",
connector=get_connector("pgvector", dsn=os.environ["POSTGRES_DSN"], table="poma_chunksets"),
embedder=get_embedder("local:BAAI/bge-small-en-v1.5"),
)push() writes content to the volume first, then embeds and upserts content-free vectors — (id, vector, {file_id, chunkset_index}), in a single poma_chunksets table with the vector extension — via the connector. This is a simpler alternative to the hand-rolled two-table schema above when you don't need the relational-lineage JOIN; use whichever fits your retrieval pattern.
Feed pgvector from the parser you already run
Pair this page with your ingestion side — the same chunksets, straight from your OCR/parsing tool's raw output:
- AWS Textract → pgvector
- Azure Document Intelligence → pgvector
- Docling → pgvector
- LlamaParse → pgvector
- Marker → pgvector
- Mistral OCR → pgvector
- PaddleOCR-VL → pgvector
- Unstructured.io → pgvector
Running a different store? The same chunk design applies: Azure AI Search · Chroma · Elasticsearch · FAISS · LanceDB · Milvus · MongoDB Atlas · OpenSearch · Pinecone · Qdrant · Redis · Turbopuffer · Vespa · Weaviate — or browse all pipeline recipes.
Fundamentals: RAG chunking guide · RAG architecture guide · POMA chunksets.