Source: http://www.poma-ai.com/docs/pipelines/marker-to-vespa

# The Missing Link Between Marker and Optimal Retrieval in Vespa

<ByAuthor />

**The short answer:** Marker gives you fast, local PDF parsing with a real Document block tree — per-page blocks, explicit types, and full HTML tables with spans intact. Vespa gives you one system that natively fuses vector search, BM25, and arbitrary ranking in a single query, with vectors as ordinary tensor fields alongside scalar attributes. Wired together naively, Marker's raw block HTML lands in a Vespa field with no tokenizer awareness of tags or dangling image refs, polluting whatever signal you build on it. The missing link is POMA: `PrimeCut().ingest()` turns the saved Marker JSON into hierarchy-preserving [chunksets](/learn/chunking/chunksets), and `poma.vektoria` feeds them into Vespa as content-free documents — `file_id`, `chunkset_index`, and an `embedding` tensor — while the actual chunkset text lives on a volume. Retrieval runs a YQL query, then `assemble()` reconstructs prompt-ready cheatsheets.

## What each end of the pipeline actually provides

**Marker** (by Datalab / VikParuchuri) runs on your own hardware and turns PDFs into clean structured output. With the JSON renderer (`--output_format json`) you get a Document block tree — Page children, per-block HTML, and full `<table>` elements with row and column spans intact. What it doesn't provide: cross-page hierarchy, retrieval units, or inline images — figure bytes live in a side-channel `images` dict keyed by name, with only `![](name)` references left in the text. Details: [The Optimal Chunker for Marker](/optimal-chunker-marker).

**Vespa** stores vectors as ordinary `tensor` fields inside the same document type as scalar attributes — no separate vector-store object. Its distinctive strengths: native BM25 on any string field, multi-phase rank profiles that fuse ANN and lexical scores in one pass, and tiered execution modes (`indexed` for large shared corpora, `streaming` for many small per-tenant partitions). What it doesn't provide: any interpretation of what a document field should hold, or HTML-aware tokenization — a string field indexes whatever text you fed it, tags and dangling references included. Details: [Vespa Chunking Strategy for RAG](/optimal-chunks-vespa).

## The naive wiring, and where it breaks

The common recipe — flatten Marker's markdown, split, embed, `app.feed_iterable()` a schema whose fields hold the raw text — breaks in ways specific to this pair:

- **Every figure vanishes before Vespa sees a field.** Marker's images live in the side-channel `images` dict; the markdown carries only `![](name)` references. Fed into any Vespa field as-is, those references become just another string token matching nothing — Vespa's tokenizer has no notion that a reference is supposed to point at bytes.
- **Raw block HTML pollutes text fields.** The JSON renderer's tables arrive as full `<table>` elements with colspans — genuinely useful structure Marker recovered. If that markup is dumped straight into a `string` field without conversion, Vespa indexes literal tag text (`<td colspan="3">`) as ordinary tokens; a query for the values inside the table ranks on tag noise rather than cell content.
- **Vespa's schema is fixed at deploy time.** Tensor dimensions and field types are declared in the `ApplicationPackage`, not inferred per document — feeding a raw, unshaped block stream means guessing field types before you've seen a well-formed document, and redeploying the schema when the shape changes.
- **Page boundaries never make it into a filterable field**, because Marker's markdown output alone drops them — only the JSON renderer's Page blocks carry them, and only if the ingest path reads them.

## The pipeline, end to end

```bash
pip install pyvespa

# Your existing Marker run — unchanged. The JSON renderer keeps the
# Document block tree that POMA auto-detects, tables and pages included.
marker_single contract.pdf --output_format json --output_dir out/
```

```python
from poma import PrimeCut
from poma.vektoria import assemble, records_from_archive, Volume
from poma.embeddings import get_embedder
from vespa.package import ApplicationPackage, Field, Schema, Document, HNSW, RankProfile
from vespa.deployment import VespaDocker

# 1. Chunk the raw Marker JSON and keep the .poma archive — the volume's source of truth.
client = PrimeCut()  # reads POMA_API_KEY
client.ingest("out/contract/contract.json", download_dir="archives", filename="contract.poma")

embedder = get_embedder("local:BAAI/bge-small-en-v1.5")
vol = Volume("s3://your-bucket/poma")

package = ApplicationPackage(
    name="contracts",
    schema=[Schema(
        name="chunkset",
        document=Document(fields=[
            Field(name="file_id", type="string", indexing=["attribute"]),
            Field(name="chunkset_index", type="int", indexing=["attribute"]),
            Field(
                name="embedding", type="tensor<float>(x[384])",
                indexing=["attribute", "index"],
                ann=HNSW(distance_metric="angular"),
            ),
        ]),
        rank_profiles=[RankProfile(
            name="ann",
            inputs=[("query(q)", "tensor<float>(x[384])")],
            first_phase="closeness(field, embedding)",
        )],
    )],
)
app = VespaDocker().deploy(application_package=package)

# 2. Content-free feed — the document holds only routing attributes; content lives on the volume.
records = records_from_archive("archives/contract.poma")  # -> list[Record] (id, text, payload)
docs = []
for r in records:
    vol.write_doc(r.payload["file_id"], {"file_id": r.payload["file_id"],
                                          "chunks": r.payload["chunks"], "text": r.text})
    docs.append({"id": r.id, "fields": {
        "file_id": r.payload["file_id"],
        "chunkset_index": r.payload["chunkset_index"],
        "embedding": embedder.embed([r.text])[0],
    }})
app.feed_iterable(docs, schema="chunkset", namespace="contracts")

# 3. Retrieval — select the routing fields explicitly in the YQL, then assemble.
qv = embedder.embed(["What are the early termination conditions?"])[0]
resp = app.query(
    yql="select file_id, chunkset_index from sources * where "
        "{targetHits:100}nearestNeighbor(embedding, q) limit 10",
    ranking="ann",
    body={"input.query(q)": qv},
)
context = assemble(resp, volume=vol)  # -> [{"file_id", "content"}, ...]
```

`records_from_archive` reads the `.poma` archive PrimeCut just wrote and returns one `Record` per chunkset: `.id` (deterministic `chunkset_uuid(file_id, chunkset_index)`), `.text` (the `to_embed` string used to build the embedding), and `.payload` (`file_id`, `chunkset_index`, `chunks`). The same `id` scheme means re-ingesting a document updates the same Vespa documents rather than duplicating them. On the reference legal-document benchmark this discipline answers with **337 tokens** of retrieved context instead of **1,542** for a recursive-splitter baseline — [methodology here](/document-ingestion-chunking-rag).

## Metadata mapping: POMA fields → Vespa primitives

| POMA field | Where it lives | Vespa role |
| --- | --- | --- |
| `to_embed` | embedded, written as the `embedding` field | `tensor<float>(x[N])` with an HNSW `ann` index |
| `file_id` | document attribute field | filterable via YQL predicates; must be `select`ed explicitly for `assemble()` |
| `chunkset_index` | document attribute field + volume key | forms the deterministic `id`; also selected explicitly in the YQL |
| `page`, `depth`, `chunks` (from Marker's Page blocks and block tree) | volume only (`record.payload`) | reconstructed into cheatsheet content, never stored as a Vespa document field |
| chunkset lineage | volume document | deduplicated and merged into one cheatsheet by `assemble()` |

## Frequently asked questions

### How do I get Marker output into Vespa for RAG?

Run Marker with the JSON renderer (`marker_single --output_format json`), hand the saved result to `PrimeCut().ingest()`, and keep the `.poma` archive. Turn each chunkset into a `Record` with `records_from_archive`, embed `record.text` into the tensor field, and `app.feed_iterable()` a content-free document (`file_id`, `chunkset_index`, `embedding`) per record. Retrieve with a YQL `nearestNeighbor` query and pass the response to `assemble()`.

### Why not just split Marker's markdown and feed it into a Vespa schema directly?

Marker's markdown drops page boundaries and leaves image bytes in a side-channel dict, and its JSON renderer's tables arrive as full HTML with no tokenizer awareness in Vespa. Feeding that raw block HTML into a string field indexes tag text as tokens instead of table values, and dangling `![](name)` refs become unmatched noise. The missing link converts blocks to clean text and keeps the fed document content-free.

### How should Marker chunksets be modeled in a Vespa document type?

One document type with `file_id` and `chunkset_index` as attribute fields and a `tensor<float>(x[N])` field with an HNSW index for the embedding. The document holds only the vector and two routing attributes; the chunkset's text and hierarchy live on a volume.

### Do Marker's tables and images survive the trip to Vespa?

Yes, upstream of Vespa. PrimeCut splices Marker's side-channel `images` dict back into its references and describes each figure, and keeps full HTML tables intact through the chunk layer. Vespa itself never stores images or tables under this pattern — only the embedding and two routing attributes — and the reconstructed content comes back from the volume via `assemble()`.

### Does Vespa retrieval need special handling for POMA's assemble() to work?

Yes — the YQL query must explicitly `select file_id, chunkset_index` alongside the `nearestNeighbor` match, or the response carries only scores and `assemble()` has nothing to resolve against the volume.

## Related recipes

Same parser, different store: [Marker → Qdrant](/pipelines/marker-to-qdrant) · [Marker → Weaviate](/pipelines/marker-to-weaviate) · [Marker → pgvector](/pipelines/marker-to-pgvector)

Foundations: [The Optimal Chunker for Marker](/optimal-chunker-marker) · [Vespa Chunking Strategy for RAG](/optimal-chunks-vespa) · [All pipeline recipes](/pipelines/) · [RAG chunking guide](/guides/rag-chunking/)