Source: http://www.poma-ai.com/docs/pipelines/textract-to-qdrant

# The Missing Link Between AWS Textract and Optimal Retrieval in Qdrant

<ByAuthor />

**The short answer:** AWS Textract gives you an excellent block graph — layout regions in multi-column-aware reading order, structured table grids, form checkboxes; Qdrant gives you excellent hybrid vector search. Wired together naively — flatten `Blocks[]` to LINE text, split, embed — they still produce mediocre RAG, because the reading order is scrambled and the hierarchy discarded before Qdrant ever sees a vector. The missing link is POMA: `PrimeCut().ingest()` consumes the raw `AnalyzeDocument` response (auto-detected), reads the LAYOUT spine properly, emits hierarchy-preserving [chunksets](/learn/chunking/chunksets), and `PomaQdrant.upsert_poma_points()` writes them as hybrid dense+sparse points with indexed hierarchy payloads. Retrieval comes back as assembled, prompt-ready cheatsheets.

## What each end of the pipeline actually provides

**AWS Textract** (`AnalyzeDocument` with `FeatureTypes=["LAYOUT", "TABLES"]`) returns a flat `Blocks[]` array: `LAYOUT_*` blocks in multi-column-aware reading order, `LAYOUT_TITLE` and `LAYOUT_SECTION_HEADER` roles marking structure, structured `TABLE`/`CELL`/`MERGED_CELL` grids, and `SELECTION_ELEMENT` checkbox state. What it doesn't provide: markdown, prose, cross-page hierarchy (a section header on page 41 has no machine-readable link to the title on page 3), or any Id link between a `LAYOUT_TABLE` region and its `TABLE` block. Details: [The Optimal Chunker for AWS Textract](/optimal-chunker-textract).

**Qdrant** provides named dense + sparse vectors on one point, payload indexes for filtered ANN, and HNSW with quantization options. What it doesn't provide: any opinion about what a point should contain. It retrieves nearest neighbors of whatever you embedded — including interleaved two-column garble, if that's what you gave it. Details: [The Optimal Chunks for the Best Retrieval in Qdrant](/optimal-chunks-qdrant).

## The naive wiring, and where it breaks

The common recipe — iterate `LINE` blocks, join with newlines, run a character splitter, embed, `client.upsert(...)` — breaks in a pair-specific way:

- **Reading order is scrambled before the first vector exists.** `LINE` blocks carry geometry, not sequence. Position-sorting a two-column report interleaves the columns — line 1 of column A, line 1 of column B — and the embedding model dutifully encodes the garble. Cosine similarity still "works", so Qdrant indexes and retrieves scrambled prose without a single error message; the failure surfaces only when a human reads the retrieved context. The LAYOUT spine that would have prevented it was in the response the whole time.
- **The hierarchy Textract marked is melted away.** `LAYOUT_TITLE` and `LAYOUT_SECTION_HEADER` roles become undifferentiated prose, so retrieved fragments arrive without lineage and the LLM answers out of context.
- **Overlap inflates the collection.** Splitter overlap embeds every boundary span twice — a measurably larger HNSW graph, and top-k results where hits 2 and 3 are near-duplicates of hit 1.
- **Page numbers vanish at the join**, so no Qdrant payload can cite a page.

POMA inverts all four: it follows the `LAYOUT_*` sequence as-is, promotes layout roles to headings and rebuilds the cross-page tree, emits overlap-free chunksets, and keeps `page` on every chunk. A Textract payload *without* LAYOUT blocks fails transparently with a 422 telling you to re-run with `FeatureTypes` including `"LAYOUT"` — refusing to ingest beats silently indexing scrambled text.

## The pipeline, end to end

```bash
pip install boto3 'poma[qdrant]'
```

```python
import json
import os

import boto3
from poma import PrimeCut
from poma.integrations.qdrant import PomaQdrant

# 1. Textract — your existing call; LAYOUT is required, TABLES recommended.
textract = boto3.client("textract")
with open("contract.png", "rb") as f:
    response = textract.analyze_document(
        Document={"Bytes": f.read()},
        FeatureTypes=["LAYOUT", "TABLES"],
    )
# Multi-page PDFs: use start_document_analysis with the same FeatureTypes,
# collect the paged results, and save the combined Blocks the same way.
with open("contract.textract.json", "w") as f:
    json.dump(response, f)

# 2. The missing link — raw Blocks[] in, chunks + chunksets out.
poma = PrimeCut()  # reads POMA_API_KEY
result = poma.ingest("contract.textract.json")  # Textract shape auto-detected

# 3. Qdrant — hybrid points with hierarchy payloads, one call.
qdrant = PomaQdrant(
    url=os.environ["QDRANT_URL"],
    api_key=os.environ["QDRANT_API_KEY"],
    cloud_inference=True,
    collection_name="contracts",
    dense_model="sentence-transformers/all-MiniLM-L6-v2",
    sparse_model="Qdrant/bm25",
    dense_size=384,
    auto_create_collection=True,
)
qdrant.upsert_poma_points(result)

# 4. Retrieve prompt-ready context.
cheatsheets = qdrant.get_cheatsheets(
    query="Which termination clauses require written notice?",
    limit=3,
)
print(cheatsheets[0]["content"])
```

Between steps 1 and 3, POMA validates the payload shape up front (a corrupted or mislabeled upload 422s immediately), splices structured `TABLE` blocks into their `LAYOUT_TABLE` regions by bounding-box geometry (≥ 0.5 containment — Textract provides no Id link), renders cell grids as HTML with `rowspan`/`colspan` from `MERGED_CELL` blocks, drops header/footer/page-number furniture, keeps `SELECTION_ELEMENT` checkboxes as ☒ / ☐ marks, and counts each byte-less `LAYOUT_FIGURE` as offloaded content in `content_metadata` — visible loss, never silent. To force or suppress detection, set `external_ocr_source` to `"textract"` or `"none"`.

Measured on our reference legal-document benchmark, this pipeline answers the same query with **337 tokens** of retrieved context instead of **1,542** for a recursive character splitter, with zero information loss. Methodology: [Document Ingestion & Chunking for RAG](/document-ingestion-chunking-rag).

## Metadata mapping: POMA fields → Qdrant primitives

| POMA chunk field | Qdrant primitive | What it enables |
| --- | --- | --- |
| `to_embed` | dense vector + BM25 sparse vector (named vectors, one point) | paraphrase + exact-term hybrid retrieval |
| `file_id` | payload field, **payload-indexed** | scope queries to one document |
| `page` (from Textract `PAGE` blocks) | payload field, **payload-indexed** | page-cited answers, page-range filters |
| `depth` | payload field | filter/re-rank by hierarchy level |
| `chunk_index` | payload field | stable ordering at assembly time |
| chunkset lineage | payload (`chunk_details`) | cheatsheet assembly without a second store |

## Frequently asked questions

### How do I get AWS Textract results into Qdrant for RAG?

Call `AnalyzeDocument` with `FeatureTypes` including `LAYOUT` (add `TABLES` for cell grids), save the raw JSON, and run `PrimeCut().ingest()` on it — auto-detected, LAYOUT reading order followed, hierarchy rebuilt. Then `PomaQdrant.upsert_poma_points(result)` writes hybrid points with `file_id`/`page`/`depth` payloads, and `get_cheatsheets(query=...)` retrieves assembled context.

### Why does my Textract-to-Qdrant pipeline return scrambled text for multi-column PDFs?

Because the text was scrambled before Qdrant saw a vector: flattening `Blocks[]` to position-sorted LINE text interleaves the columns, and embeddings encode the garble without complaint. The fix is upstream — follow the `LAYOUT_*` sequence, Textract's own multi-column-aware reading order. POMA does this, and 422s on LAYOUT-less payloads instead of indexing scrambled prose.

### What Qdrant payload fields should Textract chunks carry?

`file_id`, `page`, `depth`, `chunk_index` plus content; payload-index `file_id` and `page`. `PomaQdrant` writes these by default.

### Do Textract tables survive the trip to Qdrant?

Yes — request `TABLES` alongside `LAYOUT`. POMA splices `TABLE` blocks into `LAYOUT_TABLE` regions by geometry (no Id link exists in Textract) and renders the grid as HTML with `rowspan`/`colspan` from `MERGED_CELL` blocks, so a table is embedded whole inside its chunkset. Unclaimed `TABLE` blocks are appended, never dropped.

### Should the Qdrant collection use hybrid search for Textract content?

Yes. Textract's home turf — scanned contracts, filings, forms — is dense with exact tokens (clause numbers, field labels, case IDs) that dense embeddings blur. `PomaQdrant` writes BM25 sparse vectors by default and fuses both at query time.

## Related recipes

Same parser, different store: [Textract → Pinecone](/pipelines/textract-to-pinecone) · [Textract → Weaviate](/pipelines/textract-to-weaviate) · [Textract → pgvector](/pipelines/textract-to-pgvector)

Same store, different parser: [Mistral OCR → Qdrant](/pipelines/mistral-ocr-to-qdrant) · [Azure Document Intelligence → Qdrant](/pipelines/azure-document-intelligence-to-qdrant) · [Unstructured → Qdrant](/pipelines/unstructured-to-qdrant)

Foundations: [The Optimal Chunker for AWS Textract](/optimal-chunker-textract) · [The Optimal Chunks for Qdrant](/optimal-chunks-qdrant) · [All pipeline recipes](/pipelines/) · [RAG chunking guide](/guides/rag-chunking/)