# Read API vs ingest

> Two sealed contracts, two ports, and the trust boundary: :8081 is the public read surface and :8082 is a private in-cluster write path.

- Repository: papercomputeco/tapes
- GitHub: https://github.com/papercomputeco/tapes
- Human docs: https://grok-wiki.com/public/docs/papercomputeco-tapes-020596f21750
- Complete Markdown: https://grok-wiki.com/public/docs/papercomputeco-tapes-020596f21750/llms-full.txt

## Source Files

- `ingest/openapi.go`
- `api/openapi_routes.go`
- `api/CONTRACT`
- `ingest/CONTRACT`
- `docs/apis.md`
- `cmd/tapes/serve/stack.go`

---

---
title: "Read API vs ingest"
description: "Two sealed contracts, two ports, and the trust boundary: :8081 is the public read surface and :8082 is a private in-cluster write path."
---

`tapes serve` starts two HTTP servers with two compiled OpenAPI documents. The read API (`Tapes API`, default `:8081`) is the client and operator query surface. The ingest API (`Tapes Ingest API`, default `:8082`) is the private write path every capture adapter POSTs completed turns and harness transcripts through. They share a PostgreSQL driver and nothing else: different Fiber apps, different listen addresses, different trust models.

A third listener, the capture proxy on `:8080`, is not either contract. Provider clients send LLM traffic there. Inspection, search, export, seed, skills, and MCP go to `:8081`. Capture writes go to `:8082`.

```mermaid
flowchart LR
  subgraph clients [Callers]
    readCli["tapesctl sessions / search / export / seed"]
    capCli["tapesctl start / capture / sync"]
    adapters["tapes-extproc, paperd, backfill"]
    llmClient["Provider clients"]
  end

  subgraph proxyPort ["proxy :8080"]
    proxyPaths["Provider-compatible paths"]
  end

  subgraph readPort ["read API :8081"]
    readOpenapi["GET /openapi"]
    readRoutes["sessions, traces, search, skills, admin, MCP, cassettes"]
  end

  subgraph ingestPort ["ingest API :8082"]
    ingestOpenapi["GET /openapi"]
    ingestWrite["POST /v1/ingest"]
    ingestTx["POST /v1/ingest/transcript"]
  end

  subgraph pg [PostgreSQL]
    rawTurns["raw_turns"]
    derived["sessions / traces / spans"]
  end

  readCli --> readPort
  capCli --> ingestWrite
  adapters --> ingestWrite
  adapters --> ingestTx
  llmClient --> proxyPaths
  proxyPaths --> ingestWrite
  ingestWrite --> rawTurns
  ingestTx --> rawTurns
  rawTurns --> derived
  derived --> readRoutes
```

## Two ports, two jobs

| Listener | Config key | Default | Process | Job |
| --- | --- | --- | --- | --- |
| Proxy | `proxy.listen` | `:8080` | `tapes serve` / `tapes serve proxy` | Provider-compatible capture |
| Read API | `api.listen` | `:8081` | `tapes serve` / `tapes serve api` | Query, export, search, skills, MCP, cassette proxy |
| Ingest | `ingest.listen` | `:8082` | `tapes serve` / `tapes serve ingest` | Append `raw_turns` |

`tapesctl` keeps one server URL (`tapes-url` in `~/.tapes/config.toml`). Point that at the read API. Override it on capture commands.

```bash
tapesctl config set tapes-url http://localhost:8081

tapesctl sessions list
tapesctl search "retry logic"
tapesctl seed

tapesctl start claude --tapes-url http://localhost:8082
tapesctl capture codex-app --tapes-url http://localhost:8082
tapesctl sync --tapes-url http://localhost:8082
```

Precedence on the client is `--tapes-url`, then `TAPES_URL`, then the configured value. With none of the three, commands that need a server fail instead of guessing a host.

<Warning>
A capture pointed at `:8081` reports success and stores nothing. `start`, `capture`, and `sync` must address ingest.
</Warning>

Standalone sidecar (gateway / Envoy capture, no local proxy):

```bash
tapes serve ingest --postgres "$TAPES_STORAGE_POSTGRES_DSN"
```

`--listen` / `-l` on that subcommand binds `ingest.listen`. Embedding flags are accepted for compatibility and have no effect: the embed worker is the only embedding writer.

## Trust boundary

The two documents are separate because they are not the same kind of surface.

| | Read API | Ingest API |
| --- | --- | --- |
| OpenAPI title | `Tapes API` | `Tapes Ingest API` |
| Intended callers | Clients through an edge gateway | In-cluster capture adapters only |
| Published stance | Query and operator surface | **Not internet-facing** |
| Persistence | Reads the derived model; delete/patch sessions; admin derive/repair | Appends immutable `raw_turns` (and transcript rows) |
| Org on the request | Not a client-chosen tenant. Reads scope to the single-tenant sentinel | Envelope `session.org_id` is cleared before store. Same sentinel as reads |

The ingest contract is explicit: these endpoints trust identity that arrives with the write. Publishing them as one document with the read API would imply a surface that is reachable and safe to call from outside.

Current tenancy is single-tenant. The write path forces `session.org_id` to empty and stores the same sentinel the read handlers use (`00000000-0000-0000-0000-000000000000`). A client cannot assert an org in the envelope and land rows the read side will never surface. The `org_id` field stays on the wire until the columns go.

`x-paper-auth-subject` can still stamp `auth_subject` on the transcript path. On the read side, `auth_subject` is a filter, not an identity grant: it narrows sessions and `/v1/stats` inside this tenant.

The API process does not implement a hosting topology. `GET /metrics` on both servers is outside any auth group (Prometheus scrape). Network policy, TLS, gateway grants, and who may reach `:8082` are deployment responsibilities. Do not infer a production boundary from local listen defaults or from the generated OpenAPI.

<Warning>
Do not put ingest behind the same edge gateway as the read API. The published ingest description exists so that mistake is obvious: a JWT holder who can POST turns can write capture data.
</Warning>

## What each contract publishes

Neither server checks in an OpenAPI file. Each compiles its document from the same `oasfiber` registrations that mount the routes, and serves it at `GET /openapi`.

### Read API (`:8081`)

`GET /openapi` on a running server is the **aggregate**: core routes plus each admitted cassette's operations rewritten under `/v1/cassettes/{name}`. The `api/CONTRACT` seal is the **core** document only (`tapes dev openapi api --docs-root ''`). Cassette churn does not move the seal.

| Area | Routes |
| --- | --- |
| Health | `GET /ping` — JSON string `pong` |
| Contract / viewer | `GET /openapi`, `GET /swagger` (HTML; not in the contract) |
| Metrics | `GET /metrics` (Prometheus; not in the contract) |
| Browser UI | `GET /` only with `--api-web-ui` / `api.web_ui` (HTML; not in the contract) |
| Sessions | `GET /v1/sessions`, `GET/PATCH/DELETE /v1/sessions/{id}`, traces, `raw_turns`, export |
| Traces / spans | `GET /v1/traces`, `GET /v1/traces/{trace_id}`, `GET /v1/traces/{trace_id}/spans/{span_id}` |
| Search / stats | `GET /v1/search/spans`, `GET /v1/stats` |
| Skills | `/v1/skills` and session-skill listing |
| MCP | `GET`/`POST`/`DELETE /v1/mcp` |
| Operator | `POST /v1/admin/seed/demo`, `POST /v1/admin/derive/run`, `POST /v1/admin/raw-turns/attribution-repair` |
| Cassettes | `GET /v1/cassettes`, per-cassette `openapi.json`, reverse-proxy `/v1/cassettes/{name}` and `/*` |

There is no `/v1/search`, `/v1/sessions/summary`, or hash-based session route. Session and trace/span paths use UUID ids. Semantic search exists only at `/v1/search/spans`.

The read API compresses responses. It is not the ingest write surface.

### Ingest API (`:8082`)

Routes are fixed at process start. `GET /openapi` is compiled once and cached. Capture adapters (`tapes-extproc`, `tapesctl`, `paperd`, `tapes backfill`) all write this envelope.

| Method | Path | Role |
| --- | --- | --- |
| `GET` | `/ping` | Liveness. `{ "status": "ok" }`. Does not check the database |
| `GET` | `/openapi` | This surface's contract (not in the document; circular) |
| `GET` | `/metrics` | Prometheus (not in the document) |
| `POST` | `/v1/ingest` | One completed LLM turn |
| `POST` | `/v1/ingest/transcript` | One harness transcript file or spawn-anchor row |

:::endpoint POST /v1/ingest Ingest one captured turn
Appends one completed turn to `raw_turns` **before** provider parsing. A turn that later 422s is still captured; a parser fix re-derives it.

Idempotent when the adapter supplies `meta.request_id`: a retry of the same turn dedupes at the raw layer.

The body may carry a reduced `response`, verbatim `raw_response` (base64), or both. Raw-only is reduced server-side with the shared `pkg/capture` reducers so two capture paths produce the same row for the same traffic.

**202** `{ "status": "accepted" }` — captured and queued. Derivation is asynchronous.

**400** malformed envelope or invalid session block.

**413** body exceeds `MaxIngestBodyBytes`.

**422** well-formed but unprocessable (unknown provider, and similar).

**502** a downstream dependency failed.
:::

:::endpoint POST /v1/ingest/transcript Ingest one harness transcript
Stores one main-session transcript, one subagent transcript, or a Codex spawn-anchor row. No node-path processing happens here; the deriver reconciles against the wire capture.

Dedup key includes a content hash of `records`. Re-uploading unchanged content returns **202** with `deduped: true`. A grown file appends a new version. The deriver reads the latest version per session, agent, and lifecycle kind.

**400** malformed body, invalid session, or `records` not a JSON array.

**413** over the same body limit.

**500** persist failed.

**501** driver does not host the raw-turn layer.
:::

## Body limit and error envelope

Every non-2xx on ingest uses `llm.ErrorResponse`:

```json
{ "error": "request body exceeds the ingest size limit" }
```

Fiber `BodyLimit` is `MaxIngestBodyBytes`, derived rather than a lone literal:

`MaxDecodedRequestBytes` (32 MiB) + `MaxRawResponseBytes` base64 expansion (`8 MiB * 4/3`) + 4 MiB reserve ≈ **46.67 MiB** (`48933546` bytes).

A POST over the limit is rejected **before parse**. The handler records one `tapes_ingest_writes_total{provider="unknown",status="reject_oversize"}` sample and a warn log with `content_length`, `limit`, and `path`. Only `POST /v1/ingest` and `POST /v1/ingest/transcript` take that path; a handler-returned 413 is not counted as oversize.

Separately, a `raw_response` larger than 8 MiB is dropped on persist and marked withheld/dropped. The turn can still land. That is not the HTTP 413.

The capture proxy and `tapes-extproc` share the same byte budget so they do not POST a body ingest will refuse.

## Sealed contracts

No generated OpenAPI document is checked in. What is checked in is a fingerprint:

| File | What it seals | How to recompute |
| --- | --- | --- |
| `api/CONTRACT` | Core read document, **prose stripped** | `tapes dev openapi api --docs-root ''` |
| `ingest/CONTRACT` | Ingest document, **prose stripped** | `tapes dev openapi ingest --docs-root ''` |

`api/openapi_seal_test.go` and `ingest/openapi_seal_test.go` recompile and compare. Moving a path, parameter, schema, or status code fails the test until the new `sha256:` value is written in the same change. Editing a Go doc comment is not a contract event. Text declared inline on a route registration is published surface and does move the seal.

A running binary has no source tree, so `GET /openapi` has route and operation prose but not per-field prose. From a checkout:

```bash
tapes dev openapi                 # read API, YAML, field prose from .
tapes dev openapi ingest
tapes dev openapi --format json
tapes dev openapi --docs-root ''  # shapes only — this is the sealed document
make contracts                    # writes both into ./build/contracts
```

The ingest seal exists because adapters live in this repo and outside it. An unannounced write-shape change is one those adapters discover in production.

## Data flow across the boundary

1. A capture adapter POSTs a turn or transcript to `:8082`.
2. Ingest persists the raw envelope first, then queues derivation.
3. The derive worker (in-process under `tapes serve`, or `tapes serve derive-worker`) projects `raw_turns` into sessions, traces, and spans. Re-derive is idempotent and prunes rows the current deriver no longer emits.
4. Clients read that projection on `:8081`. `DELETE /v1/sessions/{id}` removes the derived subtree and leaves `raw_turns` intact.
5. `tapesctl seed` and `POST /v1/admin/seed/demo` are **read-API** operator calls. They replay bundled corpora through the ingest write path, then derive.

Local `tapes serve` also runs an in-process embed worker (disable with `--embed-spans=false`). Embedding failures disable search; they do not take down ingest or the read API.

## Verify which surface you hit

<Steps>
<Step title="Confirm the stack">
```bash
tapes serve
```
Expect log lines for `starting proxy`, `starting api server`, and `starting ingest server` with the three listen addresses.
</Step>
<Step title="Read API">
```bash
curl -sS http://localhost:8081/ping
curl -sS http://localhost:8081/openapi | jq '.info.title, (.paths | keys | length)'
```
Expect a JSON `pong` and title `Tapes API`. With cassettes mounted, path count is larger than the core seal document.
</Step>
<Step title="Ingest API">
```bash
curl -sS http://localhost:8082/ping
curl -sS http://localhost:8082/openapi | jq '.info.title, [.paths | keys[]]'
```
Expect `{"status":"ok"}`, title `Tapes Ingest API`, and documented paths `/ping`, `/v1/ingest`, `/v1/ingest/transcript` only.
</Step>
</Steps>

## Failure modes

| Symptom | Cause | What to do |
| --- | --- | --- |
| Capture “succeeds”, no sessions | `tapes-url` is `:8081` | Pass `--tapes-url http://localhost:8082` on `start` / `capture` / `sync` |
| List/search/export fail after capture | Client still pointed at ingest | `tapesctl config set tapes-url http://localhost:8081` |
| `413` + `{"error":"request body exceeds the ingest size limit"}` | POST > `MaxIngestBodyBytes` | Check `tapes_ingest_writes_total{status="reject_oversize"}`; shrink the envelope or withhold raw bytes (`raw_response_withheld`) |
| `GET /ping` 200 on ingest, writes 502 | Process up, database/driver not | Ping is not a write probe |
| Read `/openapi` grew after a cassette admit | Aggregate document includes cassette ops | Expected. Core seal is unchanged |
| Seal test fails | Route, schema, or inline description moved | Recompile with `--docs-root ''` and bump the matching `CONTRACT` in the same PR |

## Next

<CardGroup>
<Card title="Read API" href="/read-api">
Core GET /openapi on :8081: sessions, traces, search, skills, admin, MCP, cassette merge, and CONTRACT rules.
</Card>
<Card title="Ingest API" href="/ingest-api">
POST /v1/ingest and /v1/ingest/transcript, body cap, JSON errors, and 413 reject_oversize.
</Card>
<Card title="Capture an agent" href="/capture-an-agent">
Point Claude, Codex, pi, or a generic client at capture; always write ingest on :8082.
</Card>
<Card title="Split the stack" href="/split-the-stack">
Run proxy, api, ingest, derive-worker, and embed-worker as separate processes.
</Card>
<Card title="Capture and derive" href="/capture-and-derive">
Append-only raw_turns and the idempotent projection into sessions, traces, and spans.
</Card>
<Card title="Troubleshooting" href="/troubleshooting">
Wrong-port capture, ingest 413, and attribution-repair 200 vs 202.
</Card>
</CardGroup>