# Ingest API

> Private write contract on :8082: POST /v1/ingest, POST /v1/ingest/transcript, body cap, JSON error envelope, and 413 reject_oversize.

- Repository: papercomputeco/tapes
- GitHub: https://github.com/papercomputeco/tapes
- Human docs: https://grok-wiki.com/public/docs/papercomputeco-tapes-020596f21750
- Complete Markdown: https://grok-wiki.com/public/docs/papercomputeco-tapes-020596f21750/llms-full.txt

## Source Files

- `ingest/openapi.go`
- `ingest/ingest.go`
- `ingest/config.go`
- `ingest/responses.go`
- `ingest/CONTRACT`
- `e2e/hurls/ingest/ingest.hurl`
- `docs/apis.md`

---

---
title: "Ingest API"
description: "Private write contract on :8082: POST /v1/ingest, POST /v1/ingest/transcript, body cap, JSON error envelope, and 413 reject_oversize."
---

The ingest server is a private Fiber write surface on `ingest.listen` (`:8082` by default). Capture adapters — `tapes-extproc`, `tapesctl`, and `paperd` — POST completed LLM turns and harness transcripts here. The process appends to immutable `raw_turns` and queues derivation. It does not serve the read API, and it does not write embeddings (those belong to `tapes serve embed-worker`).

<Warning>
This surface is **not internet-facing**. It trusts identity in the request envelope and optional gateway headers. Exposing it through an edge gateway would let any caller write turns under an arbitrary session. Network policy, TLS, and who may reach `:8082` are deployment responsibilities.
</Warning>

## Listen and run

`tapes serve` starts ingest next to the proxy (`:8080`) and read API (`:8081`). Sidecar / gateway capture runs the process alone:

```bash
tapes serve ingest --postgres "$TAPES_STORAGE_POSTGRES_DSN"
```

| Key / flag | Default | Role |
| --- | --- | --- |
| `ingest.listen` | `:8082` | Listen address |
| `--listen` / `-l` | same | Standalone `tapes serve ingest` flag (`FlagIngestListenStandalone`) |
| `--ingest-listen` / `-i` | same | All-in-one `tapes serve` flag |
| `--postgres` | `storage.postgres_dsn` | Required for the raw-turn layer |
| `--project` | git repo name | Tag written onto worker jobs |

Embedding flags on `tapes serve ingest` are accepted for deploy compatibility and have **no effect**. Point capture clients at ingest, not the read API:

```bash
tapesctl start claude --tapes-url http://localhost:8082
```

## Published contract

The write contract is a **separate** OpenAPI document from the read API. Both servers compile their own route table and serve it at `GET /openapi`. There is no checked-in spec file.

| Surface | Port | Title | Seal file |
| --- | --- | --- | --- |
| Read API | `:8081` | public query/export | `api/CONTRACT` |
| Ingest API | `:8082` | `Tapes Ingest API` `1.0` | `ingest/CONTRACT` |

`ingest/CONTRACT` stores `CompiledDoc.Fingerprint()` — SHA-256 of the prose-stripped compiled JSON (the same bytes as `tapes dev openapi ingest --docs-root ''`). Route, schema, required-field, and status-code changes fail `ingest/openapi_seal_test.go` until the seal is bumped. Editing a Go doc comment is not a contract event.

A running binary reflects Go types into the document, so shapes and required fields are exact. Per-field prose is present only when the binary was compiled with a source tree (`tapes dev openapi ingest`). `GET /openapi` and `GET /metrics` are **not** listed in the document: the former is circular, the latter is Prometheus scrape-by-convention.

## Endpoints

| Method | Path | In published contract | Purpose |
| --- | --- | --- | --- |
| `GET` | `/ping` | yes | Process liveness. `200` does **not** mean a write would succeed. |
| `POST` | `/v1/ingest` | yes | Append one completed LLM turn. |
| `POST` | `/v1/ingest/transcript` | yes | Append one harness transcript (or Codex spawn/re-entry anchor). |
| `GET` | `/openapi` | no | Compiled ingest contract (JSON). |
| `GET` | `/metrics` | no | Prometheus registry for this process. |

There is no `POST /v1/ingest/batch` on this server.

```mermaid
sequenceDiagram
  participant Adapter as Capture adapter
  participant Ingest as ingest :8082
  participant Raw as raw_turns
  participant Pool as worker.Pool

  Adapter->>Ingest: POST /v1/ingest
  alt Content-Length over MaxIngestBodyBytes
    Ingest-->>Adapter: 413 reject_oversize
  else envelope decoded
    Ingest->>Ingest: reduce raw-only if needed
    Ingest->>Raw: PutRawTurn before provider parse
    alt processTurn admits
      Ingest->>Pool: Admit
      Ingest-->>Adapter: 202 accepted
    else unknown provider or invalid reduction
      Ingest-->>Adapter: 422 (raw row already stored)
    else queue or storage
      Ingest-->>Adapter: 502
    end
  end
```

:::endpoint GET /ping Liveness probe
**Operation id:** `ingestPing`

Does not touch PostgreSQL. A 200 means the Fiber process is serving.

<ResponseExample>
```json
{ "status": "ok" }
```
</ResponseExample>
:::

:::endpoint POST /v1/ingest Ingest one captured turn
**Operation id:** `ingestTurn`

Appends one completed LLM turn to `raw_turns` (`source: wire`) and enqueues the reduced turn for derivation.

**Idempotency.** When `meta.request_id` is set, `PutRawTurn` dedupes on `(org, request_id)`. An empty `request_id` disables raw-layer dedup for that row.

**Persist-before-parse.** The verbatim envelope is written to the raw layer *before* provider request parsing. A turn that later returns `422` is still captured; a parser fix re-derives it instead of requiring a re-capture. A raw-layer persist failure on this path is logged and not returned as HTTP — the handler still runs `processTurn`, which surfaces a real storage outage as `502`.

**Reduction.** The adapter may send:

| Payload | What ingest stores |
| --- | --- |
| Reduced `response` only | Historical shape. Stored as-is. |
| `response` + `raw_response` | Both. Ingest does **not** re-reduce; the live adapter may have seen framing the stored bytes no longer show. |
| `raw_response` only (empty reduction) | Server-side reduce with the shared `pkg/capture` reducers so two capture paths produce identical rows. Reduce failure is **not** an ingest failure: the bytes still land. Re-derive does **not** re-reduce those bytes today. |

Supported `provider` values: `openai`, `anthropic`, `ollama`. Anything else is `422`.

<ParamField body="provider" type="string" required>
Provider type: `openai`, `anthropic`, or `ollama`.
</ParamField>
<ParamField body="agent_name" type="string">
Optional agent tag (same role as the `X-Tapes-Agent-Name` capture header).
</ParamField>
<ParamField body="request" type="object" required>
Verbatim provider request body. Stored as raw JSON; never re-marshaled through a parsed struct.
</ParamField>
<ParamField body="response" type="object">
Already-reduced `llm.ChatResponse`. Required for the derive enqueue unless a raw-only reduce succeeded first. Validation: `message.role` set, `message.content` non-empty, every block has `type`.
</ParamField>
<ParamField body="raw_response" type="string">
Upstream response bytes, JSON base64. Stored under `raw_response_encoding` without decompressing.
</ParamField>
<ParamField body="raw_response_encoding" type="string">
Content-Encoding of `raw_response` (`identity`, `gzip`, …). Empty means identity.
</ParamField>
<ParamField body="raw_response_withheld" type="boolean">
Producer captured verbatim bytes and omitted them (usually to stay under the body cap). Marks `raw_response_dropped` when no bytes arrived. Ignored if bytes are also present.
</ParamField>
<ParamField body="meta" type="object">
Capture metadata. `request_id` is the canonical attempt id (dedup key). `upstream_request_id` is provider-issued. `thread_id` is the harness sub-thread (`""` for main). `ts_request` / `captured_at` are RFC 3339 capture clocks used when reducing raw-only. Unknown keys survive in the stored JSON.
</ParamField>
<ParamField body="session" type="object">
Optional session envelope. Absent → `harness_id="unknown"` and a synthetic `harness_session_id` from the turn Merkle-root prefix (16 hex chars).
</ParamField>

<RequestExample>
```json
{
  "provider": "openai",
  "agent_name": "e2e-test",
  "request": {
    "model": "gpt-4",
    "messages": [{ "role": "user", "content": "What is content addressing?" }]
  },
  "response": {
    "model": "gpt-4",
    "message": {
      "role": "assistant",
      "content": [{ "type": "text", "text": "Content addressing stores data by hash, not location." }]
    },
    "done": true,
    "stop_reason": "stop",
    "usage": { "prompt_tokens": 12, "completion_tokens": 25, "total_tokens": 37 }
  },
  "meta": { "request_id": "req-001" }
}
```
</RequestExample>

<ResponseExample>
```json
{ "status": "accepted" }
```
</ResponseExample>

`202` means captured and queued, not that sessions/traces/spans exist yet.

| Status | When |
| --- | --- |
| `202` | Turn reached the raw layer and the worker pool admitted it. |
| `400` | JSON decode failed, or `session` failed `IngestEnvelope.Validate`. |
| `413` | Body over `MaxIngestBodyBytes` (never parsed). |
| `422` | Well-formed envelope that cannot be processed: unknown provider, unparseable `request`, invalid reduced `response`. Raw row already stored. |
| `502` | Worker queue full, retained-byte budget exceeded, or other downstream failure. |
:::

:::endpoint POST /v1/ingest/transcript Ingest one harness transcript
**Operation id:** `ingestTranscript`

Appends one transcript file — main session, one subagent, or a Codex `sub_agent_activity` anchor — to `raw_turns` (`source: transcript`). No node-path / derive work happens here. Requires a driver that implements `storage.RawTurnStore` (Postgres). The in-memory driver returns `501`.

**Idempotency.** Dedup key is a content hash of `records`:

```text
transcript:{harness_session_id}:{agent_id|main}:{sha256(records)[:8]}
```

Re-uploading unchanged bytes returns `202` with `deduped: true`. A grown file appends a new version. The deriver reads the latest version per `(session, agent, lifecycle kind)`, so an `interacted` row does not supersede a spawn anchor.

<ParamField body="session" type="object" required>
Must include `harness_session_id`. Codex spawn anchors key this to the **root** session, never the child thread.
</ParamField>
<ParamField body="agent_id" type="string">
Empty for the main transcript; subagent / child thread id otherwise.
</ParamField>
<ParamField body="agent_type" type="string">
Harness `meta.json` agent type.
</ParamField>
<ParamField body="description" type="string">
Harness `meta.json` description (often a path).
</ParamField>
<ParamField body="tool_use_id" type="string">
Task / `spawn_agent` call that forked this agent — the causal edge the deriver joins.
</ParamField>
<ParamField body="kind" type="string">
Empty / omitted = spawn evidence. `"interacted"` = Codex re-entry (`send_message` / `followup_task`); stored for later rendering and ignored by derivation.
</ParamField>
<ParamField body="records" type="array" required>
Transcript JSONL as a JSON array, verbatim. A non-array is `400`.
</ParamField>

<ResponseField name="status" type="string">
Always `accepted` on `202`.
</ResponseField>
<ResponseField name="deduped" type="boolean">
`true` when this exact content version was already stored. That is success — do not retry.
</ResponseField>
<ResponseField name="records" type="integer">
Count of array elements in `records`.
</ResponseField>
<ResponseField name="agent_id" type="string">
Echo of the payload; empty for the main file.
</ResponseField>

<ResponseExample>
```json
{ "status": "accepted", "deduped": false, "records": 1, "agent_id": "" }
```
</ResponseExample>

| Status | When |
| --- | --- |
| `202` | Stored, or already present (`deduped: true`). |
| `400` | Malformed body, invalid session, missing `harness_session_id`, or `records` not a JSON array. |
| `413` | Body over `MaxIngestBodyBytes`. |
| `422` | Content Postgres JSONB refuses (`storage.ErrInvalidContent`). Retrying the same bytes will not succeed. |
| `500` | Server-side marshal of meta / session JSON failed. |
| `501` | Driver does not host the raw-turn layer. |
| `502` | Other `PutRawTurn` failures (outage). The compiled contract currently lists `500` for persist failure; adapters should treat both `500` and `502` as storage faults. |
:::

## Session envelope

`session` is `sessions.IngestEnvelope`. This deployment is **single-tenant**: ingest clears `org_id` on every write (payload and `x-paper-auth-org-id` alike) so a client cannot store rows the nil-scoped read side will never surface.

| Field | Constraint |
| --- | --- |
| `org_id` | If non-empty, must be a UUID at the HTTP boundary — then ingest blanks it. |
| `auth_subject` | Overridden by `x-paper-auth-subject` when that header is present. |
| `harness_id` | Empty normalizes to `unknown`. |
| `harness_session_id` | Required on the transcript path. On the turn path, missing / `unknown` harness synthesizes an id from the Merkle-root prefix. |
| `parent_harness_session_id` | Omit if absent. An explicit empty string is `400`. Parent and child share a harness; ingest placeholder-inserts the parent if its first turn has not landed. |
| `harness_metadata` | Must be a JSON object (Postgres `\|\|` merge). Arrays/scalars are `400`. |

Trusted headers (same names `extproc/headers` reads):

| Header | Role |
| --- | --- |
| `x-paper-auth-org-id` | Ignored for storage in single-tenant mode. |
| `x-paper-auth-subject` | Overrides `session.auth_subject` when set. The gateway must strip inbound client values. |

## Body cap and 413 `reject_oversize`

Two different ceilings apply. Confusing them produces the wrong operator action.

```text
POST body          MaxIngestBodyBytes   (~46.67 MiB)  →  413, turn never stored
raw_response bytes MaxRawResponseBytes  (8 MiB)       →  drop + mark, turn still accepted
```

`MaxIngestBodyBytes` is derived, not a free-standing magic number:

| Constant | Value | Role |
| --- | --- | --- |
| `MaxDecodedRequestBytes` | `32 MiB` | Anthropic Messages decoded-request ceiling. |
| `MaxRawResponseBytes` | `8 MiB` | Verbatim response stored on one row. |
| reserve | `4 MiB` | Reduced `response`, `meta`, JSON scaffolding. |
| `MaxIngestBodyBytes` | `32 MiB + 8 MiB × 4/3 + 4 MiB` = **48 933 546 bytes** | Fiber `BodyLimit`. |

The `× 4/3` term is base64 expansion of `raw_response` on the wire. Fiber's 4 MiB default is **not** the real limit — if it were, a legal raw-bearing envelope would 413 before the 8 MiB drop path could run.

Over-limit POSTs to `/v1/ingest` or `/v1/ingest/transcript`:

- HTTP `413`
- JSON body `{"error":"request body exceeds the ingest size limit"}` — same `llm.ErrorResponse` envelope as every other rejection
- Fiber rejects on declared `Content-Length` and does not parse the body
- One Prometheus increment: `tapes_ingest_writes_total{provider="unknown",status="reject_oversize"}`
- One warn log with `content_length`, `limit`, and `path`
- The accepted-size histogram is **not** updated (`bodyBytes=0`)

A `413` on any other path keeps Fiber's default (plain-text) handler and does **not** increment `reject_oversize`.

An 8 MiB+ `raw_response` that still fits the POST is stored without the bytes, with `raw_response_dropped=true`. Reduction already ran, so the row keeps request, reduced response, and session attribution. `raw_response_withheld: true` with no bytes sets the same marker (producer dropped to fit the transport).

## Error envelope

Every documented failure uses:

```json
{ "error": "<message>" }
```

That is `llm.ErrorResponse`. Capture adapters can parse all failures uniformly — including `413`.

Turn-path sentinels map to status:

| Sentinel | HTTP | Typical cause |
| --- | --- | --- |
| `ErrEnvelope` | `400` | Decode / session validation |
| `ErrUnprocessable` | `422` | Unknown provider, parse, invalid reduction |
| `ErrDownstream` | `502` | Worker queue full, storage |
| `ErrWorkerByteBudget` | `502` | Worker retained-byte budget (wraps `ErrDownstream`) |

`GET /openapi` compile failure is a process defect and uses `{ "error": "openapi_compile_failed", "message": "..." }` — that path is not part of the adapter write contract.

## Metrics

Scraped at `GET /metrics` from this process's private registry.

| Metric | Labels | Notes |
| --- | --- | --- |
| `tapes_ingest_writes_total` | `provider`, `status` | Empty provider becomes `unknown`. Transcript writes use `provider="transcript"`. |
| `tapes_ingest_dag_write_seconds` | `provider` | Enqueue latency. |
| `tapes_ingest_worker_queue_depth` | — | Ingest-side snapshot only. |
| `tapes_ingest_body_bytes` | `provider` | Accepted envelopes only. |
| `tapes_ingest_rawonly_stamp_total` | `provider`, `field`, `source` | Whether raw-only reduce restored `duration` / `created_at` from `elapsed_seconds`, `captured_at`, `ts_request`, or `fallback`. |

Write `status` values: `accepted`, `reject_envelope`, `reject_parse`, `unknown_provider`, `queue_full`, `queue_byte_budget`, `downstream_error`, `internal_error`, `reject_oversize`.

## Operator notes

<Steps>
<Step title="Confirm ingest, not the read API">
`GET http://127.0.0.1:8082/ping` → `{"status":"ok"}`. `GET /openapi` title is `Tapes Ingest API`. Pointing `tapesctl start` at `:8081` is the wrong contract.
</Step>
<Step title="Require Postgres for transcripts">
`POST /v1/ingest/transcript` without a `RawTurnStore` is `501`. Wire ingest still needs Postgres in production so persist-before-parse has a home.
</Step>
<Step title="Treat 202 as queued">
Sessions, traces, and spans appear after the derive worker projects `raw_turns`. Inspect them on `:8081`.
</Step>
<Step title="Do not retry a 413 by resending the same body">
Split or withhold `raw_response` (`raw_response_withheld: true`) so the reduced turn still lands. Retrying an unchanged oversize POST will 413 forever.
</Step>
<Step title="Do not retry transcript 202 with deduped=true">
That exact content version is already stored.
</Step>
</Steps>

<AccordionGroup>
<Accordion title="413 vs missing raw bytes">
`413` means the HTTP POST never entered the handler. A stored row with `raw_response` empty and `raw_response_dropped` true means the turn arrived and verbatim bytes were withheld or capped at 8 MiB. Those are opposite fidelity facts.
</Accordion>
<Accordion title="422 after a successful capture">
The raw envelope is already on disk. Fix the parser / reducer and re-derive. Do not re-POST unless `request_id` is empty (no dedup) or you intend a new attempt id.
</Accordion>
<Accordion title="502 queue_byte_budget vs queue_full">
Both are `502`. The metric label tells which worker ceiling saturated: retained bytes vs slot count.
</Accordion>
<Accordion title="Changing the contract">
Move a route, field, or status code only with a seal bump in `ingest/CONTRACT`. Out-of-repo adapters (`tapesctl`, `tapes-extproc`, `paperd`) are not built by this CI and will discover an unannounced change in production.
</Accordion>
</AccordionGroup>

## Related pages

<CardGroup>
<Card title="Read API vs ingest" href="/read-vs-ingest">
Two sealed contracts, two ports, and why `:8082` stays private.
</Card>
<Card title="Read API" href="/read-api">
The public `:8081` contract this write surface does not implement.
</Card>
<Card title="Capture and derive" href="/capture-and-derive">
What happens to a raw turn after `202`.
</Card>
<Card title="Gateway capture" href="/gateway-capture">
`tapes-extproc` POSTs completed turns at this envelope.
</Card>
<Card title="Prove the capture ratchet" href="/capture-ratchet">
Raw-only reduce and offline equivalence against stored reductions.
</Card>
<Card title="Split the stack" href="/split-the-stack">
Run ingest as its own process and failure domain.
</Card>
<Card title="Troubleshooting" href="/troubleshooting">
Wrong-port capture and ingest `413`.
</Card>
</CardGroup>