Source: http://www.poma-ai.com/docs/grill/concepts/ingestion

# Grill Ingestion

Grill's ingestion entry point is **`POST /grill/ingest`**. It runs the same PrimeCut pipeline — parse, chunk, embed — but **persists the result inside your project's namespace** instead of returning a `.poma` archive. Once a job finishes the document is immediately searchable through `/grill/search`; you never download or re-upload anything.

## Why a separate endpoint

`POST /grill/ingest` and `POST /primeCut/ingest` look almost identical on the wire. They differ in what happens **after** the chunks are produced:

| Step | `/primeCut/ingest` | `/grill/ingest` |
| --- | --- | --- |
| Parse + chunk | ✅ same pipeline | ✅ same pipeline |
| Embed | Optional, depends on plan | ✅ always |
| Persist to project namespace (vectors + storage) | ❌ | ✅ |
| Make doc available to `/grill/search` | ❌ | ✅ |
| `.poma` archive download via `/jobs/{job_id}/download` | ✅ | ❌ |

If you want chunks to take home, use PrimeCut. If you want the document **searchable** through `/grill/search`, use Grill.

> One project = one product. A project created with `product:"primecut"` cannot call `/grill/ingest`, and vice versa. See [Create a Grill project](/grill/getting-started/projects).

## The job lifecycle

Ingestion is asynchronous. The `job_id` you get back moves through the standard POMA lifecycle:

```text
pending  ──▶ processing  ──▶ done       ◀── searchable from this point
                          └─▶ failed
```

At `done`, three things are true atomically: the document appears in `GET /grill/docs`, `GET /grill/docs/{docId}` returns its `DocInfo`, and `/grill/search` can retrieve from it. The SDK's `ingest()` **waits for this for you**; with raw HTTP you poll `GET /jobs/{job_id}/status` or stream `GET /status/v1/jobs/{job_id}`. There's no `.poma` to download — Grill keeps the artifacts server-side.

`docId` is derived from the filename (sanitised) plus a project salt; read the canonical value from `DocInfo.doc_id` and use it for `doc_filter` and `/grill/docs/{docId}`.

## Ingest with the Python SDK

The recommended path. The `Grill` client handles the octet-stream framing, polling, and status streaming for you, and reads the project API key from `POMA_GRILL_API_KEY`.

```python
# pip install poma
from poma import Grill

g = Grill()
result = g.ingest("manual.pdf")        # submit + wait, returns when done
print(result.job_id, result.status, result.usage)
```

**Batch** — submit everything first, collect later so the waits overlap:

```python
g = Grill()
job_ids = [g.submit(p) for p in ["a.pdf", "b.pdf", "c.pdf"]]
results = [g.collect(jid) for jid in job_ids]
```

**Async** — run the waits concurrently with `AsyncGrill`:

```python
import asyncio
from poma import AsyncGrill

async def ingest_all(paths: list[str]) -> None:
    async with AsyncGrill() as g:
        results = await asyncio.gather(*(g.ingest(p) for p in paths))
        for r in results:
            print(r.job_id, r.status)

asyncio.run(ingest_all(["a.pdf", "b.pdf", "c.pdf"]))
```

**From a URL** instead of a local file — pass `remote_url` and the server fetches it:

```python
g.ingest(remote_url="https://example.com/report.pdf")
```

Full signatures: [`Grill` reference](/sdk/reference/grill), [`AsyncGrill` reference](/sdk/reference/async-grill).

## Labels & metadata for filtering

Tag a document at ingest so queries can filter on it later. With the SDK these are keyword arguments on `ingest()` / `submit()`:

```python
g.ingest(
    "annual-report-2025.pdf",
    labels=["year:2025", "source:investor-relations"],   # categorical tags
    meta_int_1=1735689600,                                # e.g. a Unix timestamp
    meta_int_2=3,                                          # e.g. a revision number
)
```

Each maps to an ingest header and a search-time filter:

| SDK argument | Ingest header | Query-time filter | Use for |
| --- | --- | --- | --- |
| `labels=[…]` | `X-Labels` | `labels_any` / `labels_all` (SDK) · `meta_tags_any` / `meta_tags_all` (HTTP) | Categorical tags — `"year:2025"`, `"source:treasury"`. Max 64, ≤ 128 chars each. Stored **HMAC'd / opaque**, so matching is **equality-only**. |
| `meta_int_1=` / `meta_int_2=` | `X-Meta-Int-1` / `X-Meta-Int-2` | `meta_int_1_gte` / `_lte`, `meta_int_2_gte` / `_lte` | Range-queryable integers — a timestamp, a revision number. Stored plaintext. |

Pick labels by sensitivity: `X-Labels` values are HMAC'd before they reach the vector DB, so the plaintext is never stored — safe for sensitive tags, at the cost of equality-only matching. The HTTP API also accepts **`X-Unencrypted-Strings`** for plaintext, wildcard-filterable metadata (see the [API reference](/grill/reference/api)) — never put secrets or PII there.

See [Retrieval](/grill/concepts/retrieval) for how to apply these filters at search time.

## Direct API usage

If you're not on Python, `POST /grill/ingest` takes **raw file bytes** as `application/octet-stream`, with the filename in `Content-Disposition`:

```bash
curl -sS -X POST "$GRILL/ingest" \
  -H "authorization: Bearer $GRILL_KEY" \
  -H "content-type: application/octet-stream" \
  -H 'content-disposition: attachment; filename="manual.pdf"' \
  --data-binary @manual.pdf
```

- Multipart (`multipart/form-data`) is **not** accepted → `403`. Use octet-stream.
- Fetch from a URL instead of uploading bytes by setting `X-Remote-URL` (then `Content-Disposition` and the body are optional).
- Labels & metadata ride as headers: `X-Labels`, `X-Meta-Int-1` / `X-Meta-Int-2`, `X-Unencrypted-Strings`, plus `X-Base-URL` and `X-Completion` (completion webhook).

Full request/response shapes and every header are in the [API reference](/grill/reference/api).

## Bulk ingest from a bucket

Have a whole corpus in S3 or GCS? `POST /grill/ingestBucket` ingests the bucket (or a prefix) as one server-side task — no per-file uploads, no request-size limit, up to 500 MB per file and 500 GB per task, one active task per account. See the [API reference](/grill/reference/api#post-grill-ingestbucket) for the request shape, task lifecycle, and all limits.

## Supported file types

Grill inherits the full PrimeCut format set:

- **Documents:** `pdf`, `doc`, `docx`, `dotx`, `rtf`, `txt`, `md`, `html`, `htm`, `xml`
- **Data & structured text:** `json`, `yaml`, `toml`, `ini`, `env`, `cir`
- **Presentations:** `ppt`, `pptx`, `pptm`, `pps`, `ppsx`, `pot`, `potx`, `key`
- **Spreadsheets:** `xls`, `xlsx`, `xlsm`, `xlsb`, `xltx`, `csv`, `tsv`, `numbers`, `ods`, `odc`
- **Images:** `png`, `jpg`, `jpeg`, `gif`, `bmp`, `tif`, `tiff`, `svg`, `webp`, `ico`, `heic`, `heif`, `psd`
- **Audio:** `mp3`, `wav`, `m4a`, `aac`, `ogg`, `flac`, `opus`
- **Video:** `mp4`, `mov`, `webm`, `mkv`, `avi`, `m4v`, `mpeg`, `mpg`
- **Other:** `epub`, `mobi`, `djvu`, `dwg`, `dxf`, `dwf`, `dwfx`, `vsd`, `vsdx`, `ai`, `eps`, `ps`, `prn`, `xps`, `oxps`, `pub`, `mdi`, `pages`, `odp`, `odf`, `odt`

**Audio & video** are handled by a native media front-end — video is keyframe-sampled and transcribed (speech-to-text) with per-keyframe vision, audio runs the same understanding pass without frames — so both become searchable text like any other document.

**Ingesting from a URL:** set `X-Remote-URL` to have Grill fetch the bytes instead of uploading them (the `Content-Disposition` and body become optional). Public video-platform URLs (e.g. YouTube) are supported as a source too.

## Re-ingesting the same file

Re-uploading a file with the **same effective `docId`** replaces the existing document — old vectors and storage are discarded. There's no append mode; an updated PDF fully supersedes the previous version. For version history, ingest each version under a distinct filename so the `docId` differs. To remove a doc cleanly first, use `DELETE /grill/docs/{docId}` ([Document management](/grill/concepts/document-management)).

## Errors you will see

| Status | When | What to do |
| --- | --- | --- |
| `400` | No `X-Remote-URL` and a missing/invalid `Content-Disposition`, unsupported MIME, or empty body | Fix the headers; check the file is non-empty (or supply `X-Remote-URL`). |
| `401` | Missing or invalid Bearer token | Use a project API key — see [Authentication](/grill/getting-started/authentication). |
| `403` | Caller's project is `primecut`, or the request is multipart | Create a Grill project; switch to octet-stream. |
| `413` | Upload body exceeds the **50 MB** request limit | Host the file and pass `remote_url` / `X-Remote-URL` instead of uploading the bytes. |
| `500` | Server-side parse failure | Retry once; if it persists, contact support with the `job_id`. |

## Next

- [Retrieval](/grill/concepts/retrieval) — once the doc is in, how search behaves.
- [Document management](/grill/concepts/document-management) — list, inspect, delete.
- [API reference](/grill/reference/api) — full `/grill/ingest` request/response.