# Capture and derive

> Append-only raw_turns, the lossy reduction stored beside raw_response, and the idempotent deriver that projects sessions, traces, and spans.

- Repository: papercomputeco/tapes
- GitHub: https://github.com/papercomputeco/tapes
- Human docs: https://grok-wiki.com/public/docs/papercomputeco-tapes-020596f21750
- Complete Markdown: https://grok-wiki.com/public/docs/papercomputeco-tapes-020596f21750/llms-full.txt

## Source Files

- `docs/architecture.md`
- `pkg/derive/deriver.go`
- `pkg/derive/worker/worker.go`
- `ingest/ingest.go`
- `proxy/proxy.go`
- `migrations/1781049035_raw_turns.up.sql`
- `pkg/capture/reducer.go`

---

---
title: "Capture and derive"
description: "Append-only raw_turns, the lossy reduction stored beside raw_response, and the idempotent deriver that projects sessions, traces, and spans."
---

Capture writes completed LLM turns into PostgreSQL `raw_turns` and never rewrites them. A derive worker then re-projects each dirty harness session into the read model of sessions, traces, and spans. `tapes serve` runs the proxy (`:8080`), private ingest (`:8082`), read API (`:8081`), and an in-process derive worker together; production splits `tapes serve ingest` and `tapes serve derive-worker` so capture and projection fail independently.

<Note>
`tapes` owns the database and this pipeline. `tapesctl` is the client: it points agents at capture and reads the derived surface. Do not mix `:8081` (read) with `:8082` (ingest).
</Note>

## Pipeline

```mermaid
flowchart TB
  subgraph Capture["Capture"]
    Agent["agent / tapesctl / gateway"]
    Proxy["proxy :8080"]
    Extproc["tapes-extproc"]
    Agent --> Proxy
    Agent --> Extproc
  end

  subgraph Write["Private write"]
    Ingest["ingest :8082<br/>POST /v1/ingest<br/>POST /v1/ingest/transcript"]
    Pool["proxy worker pool"]
    Proxy --> Pool
    Extproc --> Ingest
  end

  subgraph Store["PostgreSQL"]
    Raw["raw_turns<br/>request + response + raw_response"]
    Queue["derive_queue"]
    Sess["sessions row"]
    Proj["traces / spans / links"]
    Pool --> Raw
    Ingest --> Raw
    Pool --> Sess
    Ingest --> Sess
    Raw --> Queue
  end

  subgraph Derive["Derive worker"]
    Worker["poll / debounce / lock"]
    Deriver["Deriver.AddTurn + Finish"]
    Reconcile["ReconcileTranscripts"]
    Emit["EmitSpans"]
    Worker --> Deriver --> Reconcile --> Emit --> Proj
    Queue --> Worker
  end

  subgraph Read["Read API :8081"]
    API["/v1/sessions / traces / spans / raw_turns"]
    Proj --> API
    Raw --> API
  end
```

| Surface | Default | Role |
| --- | --- | --- |
| `proxy.listen` | `:8080` | Transparent provider proxy. Forwards upstream, then appends a wire `raw_turns` row through the in-process worker pool. |
| `ingest.listen` | `:8082` | Trusted write contract. `POST /v1/ingest` and `POST /v1/ingest/transcript`. |
| `api.listen` | `:8081` | Derived read surface. Does not accept capture. |
| Derive worker | in-process on `tapes serve`; else `tapes serve derive-worker` | Polls `derive_queue`, re-derives one session at a time. |

Embedding is a separate worker (`tapes serve embed-worker`). It is not a derive step; a down embed backend must not stall projection.

## Capture paths

Two writers land the same append-only table.

| Path | How it writes | What it stores |
| --- | --- | --- |
| Local proxy (`tapes serve` / `tapes serve proxy`) | After the upstream response, `proxy/worker` calls `PutRawTurn` on the Postgres driver. No HTTP hop to ingest. | `source=wire`, verbatim `raw_request`, reduced `response`. The local persist path does not attach `raw_response`. |
| Gateway / `tapesctl` / extproc | `POST /v1/ingest` on `:8082` | Same wire row, plus optional `raw_response` bytes and `raw_response_encoding`. |
| Harness transcript | `POST /v1/ingest/transcript` | `source=transcript`. `records` land in `raw_request`. The deriver uses these rows as the causal/fork skeleton, not as extra LLM calls. |

Supported wire providers are `anthropic`, `openai`, and `ollama`. Capture adapters reduce streaming (SSE / NDJSON) or one-shot JSON into `llm.ChatResponse` with the shared `pkg/capture` reducers. Server-side reducers exist today for Anthropic and OpenAI Responses; Ollama has no server-side reducer, so it cannot move to raw-only ingest until one exists.

A POST that is not a chat/completion body is still forwarded by the proxy; only completed chat turns enqueue capture.

## Append-only `raw_turns`

`raw_turns` is the immutable capture log. The deriver is a pure function of these rows: a classifier or span-shape change is a re-derive, not a re-capture.

| Column | Meaning |
| --- | --- |
| `source` | `wire` or `transcript`. |
| `provider`, `agent_name` | Wire identity. |
| `harness_id`, `harness_session_id` | Session key the worker derives as one unit. |
| `request_id` | Dedup key when non-empty. Retried POSTs of the same attempt are a no-op (`ON CONFLICT` on `(org_id, request_id)`). Empty `request_id` is plain append. |
| `raw_request` | Verbatim request JSON (or transcript `records`). Stored as JSONB without re-marshaling through parsed structs, so unknown fields survive. |
| `response` | Reduced `llm.ChatResponse`. This is what the deriver reads on a healthy turn. |
| `raw_response` | Upstream response bytes, stored under the original `Content-Encoding`. BYTEA, never JSON-scrubbed. |
| `raw_response_encoding` | `identity`, `gzip`, or empty (identity). |
| `raw_response_dropped` | `true` when verbatim bytes existed and were not stored. Distinguishes “never captured” from “had bytes, chose not to keep them”. |
| `meta` | Adapter metadata (`request_id`, `thread_id`, `content_type`, `ts_request`, `captured_at`, `elapsed_seconds`, …). |
| `session_envelope` | Session-tracking envelope, stored verbatim. |
| `received_at` | Ingest receive time. Chronology fallback when capture-side stamps are missing. |

JSON payloads are sanitized only for sequences Postgres JSONB cannot store. `raw_response` is not sanitized.

<Warning>
This deployment is single-tenant. Ingest clears `session.org_id` on the write path so a client cannot store rows the read side will never surface.
</Warning>

### Dedup and dirty marks

`PutRawTurn` appends the row and, when `harness_session_id` is set, upserts `derive_queue` in the same transaction. A retried POST that hits the unique index still marks the session dirty: a redundant mark only costs one idempotent derive.

Transcript rows use a content-addressed `request_id`:

```text
transcript:{harness_session_id}:{agentKey}:{sha256(records)[:8]}
```

`agentKey` is `agent_id` or `main`. Re-uploading unchanged content is a no-op (`deduped: true`). A grown file is a new row. The deriver keeps the latest version per `(session, agent, lifecycle kind)` so an `interacted` re-entry never supersedes a spawn anchor.

### Limits

| Limit | Value | On overflow |
| --- | --- | --- |
| Ingest HTTP body (`MaxIngestBodyBytes`) | `32 MiB` request + base64 of `8 MiB` raw response + `4 MiB` reserve ≈ `46.67 MiB` | `413` JSON `{"error":"..."}`, metric `tapes_ingest_writes_total{status="reject_oversize"}`. Body is not parsed. |
| Stored `raw_response` (`MaxRawResponseBytes`) | `8 MiB` | Bytes dropped, row still written, `raw_response_dropped=true`. Reduced `response` is kept. |
| Producer withheld bytes (`raw_response_withheld`) | flag on the envelope | Same dropped marker when no bytes arrived. If bytes are present, bytes win. |
| Proxy request buffer | same `MaxIngestBodyBytes` | `413`, connection closed on declared oversize. |

A raw-layer persist failure on `POST /v1/ingest` is logged and does not fail the HTTP write: the handler still runs session ingest. Treat a missing raw row after `202` as a storage outage to inspect, not as a client retry of a successful envelope.

## Lossy reduction beside `raw_response`

Reduction turns upstream bytes into one canonical `llm.ChatResponse`. It is lossy by design: fields the reducer does not model are gone from `response`. Two adapters that reduce the same traffic differently would produce different rows. The raw column exists so the reduction is auditable, not authoritative.

An ingest envelope may send:

| Payload | What ingest stores |
| --- | --- |
| Reduction only (`off`) | `response` only. Historical adapter shape. |
| Reduction + bytes (`dual`) | Both. Ingest keeps the adapter’s reduction; bytes sit beside it for `tapes raw equivalence`. |
| Bytes only (`raw`) | Ingest reduces with `ReduceStoredRawTurn` **before** the raw write, so the row carries both halves. |

`reduceRawOnly` is a no-op when a reduction is already present. An adapter that consumed the live stream may have seen framing the stored bytes no longer show; re-reducing that turn would lose information.

A failed raw-only reduction does **not** reject ingest. The bytes still land. Recovery is on the derive read path: `GetRawTurn` selects `raw_response` only when the stored reduction has no role or content blocks, and `recoverReduction` re-runs the Anthropic / OpenAI reducer before `Deriver.AddTurn`. A later reducer fix therefore recovers those turns on the next derive. Recovery is best-effort: one unreducible turn is logged and skipped; it does not fail the session.

`created_at` and `usage.total_duration_ns` are stamped at reduce time. Under `raw`, ingest restores them from `meta.captured_at` / `meta.ts_request` and `meta.elapsed_seconds`. A window can be byte-equivalent and still lose duration if those meta fields were empty.

See [Prove the capture ratchet](/capture-ratchet) for `off` / `dual` / `raw` and `tapes raw equivalence`.

## Ingest HTTP

:::endpoint POST /v1/ingest Append one completed wire turn
Appends `source=wire`. Persists the raw envelope **before** provider parse, so a `422` still leaves a row a later parser can re-derive.

**Success:** `202` `{"status":"accepted"}`. Capture is acknowledged; the projection is asynchronous.

**Errors:** `400` invalid envelope / session (`ErrEnvelope`); `422` unknown provider or unparseable request / empty reduction (`ErrUnprocessable`); `413` oversize; `502` worker saturation or storage (`ErrDownstream`). Same JSON envelope: `{"error":"..."}`.
:::

:::endpoint POST /v1/ingest/transcript Append one harness transcript or spawn-anchor row
Requires Postgres. Requires `session.harness_session_id`.

`agent_id` + `tool_use_id` carry the subagent fork edge. `kind` empty/`started` is spawn evidence; `kind=interacted` (`send_message`, `followup_task`) is stored and counted, then ignored by derivation.

**Success:** `202` with `status`, `deduped`, `records`, `agent_id`.

**Errors:** `400` envelope; `422` unstorable JSONB content; `501` if the raw-turn layer is unavailable; `502` storage.
:::

Successful ingest also UPSERTs the `sessions` row when a session envelope is present. The span writer skips a harness key with no sessions row: raw turns without a resolved session do not appear on the read API until that row exists. A bare proxied call with no envelope gets a synthetic `harness_session_id` from the in-memory merkle root prefix so turns still group.

## Derive queue and worker

`derive_queue` is keyed by `(org_id, harness_id, harness_session_id)`, not by the sessions UUID. Transcript ingest can write a raw row before a sessions row exists; dirty state is queue state.

| Column | Role |
| --- | --- |
| `dirtied_at` | Last mark. Debounce waits for this to go quiet. |
| `first_dirtied_at` | Survives re-marks. Bounds lag for a session that never settles. |

Worker defaults (`pkg/derive/worker`):

| Setting | Flag / key | Default | `tapes serve` in-process |
| --- | --- | --- | --- |
| Poll | `--poll-interval` / `derive_worker.poll_interval` | `5s` | `5s` |
| Debounce | `--debounce` / `derive_worker.debounce` | `20s` | `2s` |
| Max lag | `--max-derive-lag` / `derive_worker.max_derive_lag` | `45s` | `45s` |
| Sweep | `--sweep-interval` / `derive_worker.sweep_interval` | `1h` (+ once at startup) | `1h` |
| Sweep window | `--sweep-window` / `derive_worker.sweep_window` | `24h` (negative = all history) | `24h` |
| Concurrency | — | one session at a time | same |
| Drain | — | `30s` after SIGTERM/SIGINT | same |

A session derives when it has settled (`dirtied_at` older than debounce) **or** when `first_dirtied_at` is older than max lag. Continuously streaming sessions re-mark on every capture and never settle; the lag bound is what projects them.

Work is at-least-once:

1. Take a per-session Postgres advisory lock (`TryDeriveSessionLock`). Another replica skips (`locked`).
2. Re-read the queue row under the lock.
3. `RederiveSession`.
4. `ClearDeriveDirty` only if `dirtied_at` is unchanged. A turn that landed mid-derive leaves the row queued.

A derive error stays on that session and does not stall the page. Poll failures back off exponentially up to `30s`. `--wait-for-db` retries startup; otherwise an unreachable DSN fails fast. `--metrics-listen` serves `/metrics`, `/healthz`, `/readyz` (the ready probe runs `DeriveQueueStats`, the same query the loop needs).

Run the standalone worker with its own memory budget. A full derive once OOM-killed a 256Mi API pod; do not embed this loop in the API process. The worker applies a cgroup-derived soft `GOMEMLIMIT` unless one is already set.

```bash
tapes serve derive-worker --postgres "$TAPES_STORAGE_POSTGRES_DSN"
```

## Deriver

`RederiveSession` is the unit of work: one harness session, streamed in capture order.

1. Index raw rows. Wire rows sort by `CapturedAt`. Transcript rows keep the latest version per `(agent, kind)`.
2. `GetRawTurn` applies the latest attribution correction overlay. It does not mutate `raw_turns`.
3. Recover an empty reduction from `raw_response` when needed.
4. `Deriver.AddTurn` for each wire row: parse request, decode `response`, build the in-memory merkle chain, classify the call, dedup nodes by content hash.
5. `Finish` runs attach passes (verdicts, web summaries, plan-name-gen).
6. `ReconcileTranscripts` joins transcript spawn anchors onto wire chains.
7. `EmitSpans` writes traces, spans, and links. Session rollups (title, model usage, tasks, kind counts, status) fold in the same transaction.
8. Upsert the projection and prune rows the new set no longer contains, scoped to that session.

Re-running unchanged raw input upserts the same deterministic IDs and prunes zero rows. IDs are functions of wire identity (`request_id`, `tool_use_id`, `thread_id`), not of wall-clock derive time.

The persisted `nodes` table is gone. Merkle content addressing (`merkle.ProjectContent`) exists only in memory for identity and dedup. Memory tracks unique content, not the sum of every turn’s re-sent history.

A turn whose reduced response has no `role` or content blocks is `raw_only`: counted, not failed, and produces no chain. Parse failures are sampled (cap 20) on `RederiveReport.parse_failures`.

`CapturedAt` precedence for span start: `captured_at` rewound by `elapsed_seconds` when both are valid; else `ts_request`; else `captured_at` alone; else `received_at`. Elapsed values above seven days are treated as corrupt.

Call kinds are an open catalog (`main`, `offshoot:permission-check:stage1|stage2`, `offshoot:title-gen`, `offshoot:compaction`, …). Unknown kinds stay `unknown` rather than being silently bucketed. Span kinds and subagent rejoin live on [Sessions, traces, and spans](/sessions-traces-spans).

<AccordionGroup>
<Accordion title="RederiveReport fields">
<ResponseField name="raw_turns" type="int">Rows fed to this pass.</ResponseField>
<ResponseField name="parsed_turns" type="int">Rows that produced a chain.</ResponseField>
<ResponseField name="raw_only_turns" type="int">Rows skipped because the reduction has no role or content.</ResponseField>
<ResponseField name="parse_failures" type="string[]">Sampled parse errors (`raw_turn id=… request_id=…`).</ResponseField>
<ResponseField name="call_kinds" type="object">Counts by classified call kind, including `unknown`.</ResponseField>
<ResponseField name="reconcile" type="object">Transcript join stats, including `codex_threads_unanchored` and `codex_interacted_rows`.</ResponseField>
</Accordion>
</AccordionGroup>

## Manual re-derive

:::endpoint POST /v1/admin/derive/run Rebuild every persisted session from raw_turns
Same pass as `tapes dev rederive`. Enumerates sessions from the read model (so an emptied repair source is still pruned), then `RederiveSessionLocked` one session at a time.

Use after a deriver or classifier change. For a full historical sweep on the worker, set `--sweep-window` negative.
:::

`POST /v1/admin/raw-turns/attribution-repair` records an append-only correction and synchronously re-derives the previous and effective sessions. `200` finished; `202` means the correction committed but projection rebuild is pending — the worker converges; do not retry the repair.

`tapesctl seed` (or `POST /v1/admin/seed/demo`) replays bundled corpora through the normal raw write + derive path. Overwrite is rejected; seeding is idempotent against the raw layer.

## Verify

<Steps>
<Step title="Capture one turn">
Point an agent at ingest (`tapesctl start … --tapes-url http://localhost:8082`) or send traffic through `:8080`. Confirm `202` from ingest, or a growing `raw_turns` count in `tapes status`.
</Step>
<Step title="Wait for derive">
Local `tapes serve` uses a 2s debounce. Standalone workers wait 20s unless lag hits 45s. Then list sessions on the read API:

```bash
tapesctl sessions list --tapes-url http://localhost:8081
tapesctl sessions raw-turns <session-id>
tapesctl sessions traces <session-id>
```
</Step>
<Step title="Inspect the split">
`GET /v1/sessions/{id}/raw_turns` is the capture log. Trace and span routes are the projection. Session IDs on the read API are UUIDs, not merkle hashes, and not the harness session id `tapesctl start` prints.
</Step>
</Steps>

## Failure modes

| Symptom | Likely cause | What to do |
| --- | --- | --- |
| Session never appears | Client talked to `:8081` instead of `:8082`, or no `sessions` row for the harness key | Capture against ingest; check `tapes status` and ingest logs. |
| Raw row exists, no traces | Worker not running, debounce/lag not elapsed, or derive error on that session | `tapes serve derive-worker`; check worker logs / `derive_queue`; `POST /v1/admin/derive/run`. |
| `413` on ingest | Envelope over `MaxIngestBodyBytes` | Shrink the payload or drop `raw_response` and set `raw_response_withheld`. |
| `raw_response_dropped` | Bytes over 8 MiB, or producer withheld | Reduced `response` should still project; verbatim recovery is gone for that turn. |
| Empty projection after raw-only ingest | No reducer for the provider (Ollama), or reduce failed and bytes missing | Confirm `provider` and that `raw_response` is present; fix reducer and re-derive. |
| Subagent not nested under the spawn tool | Missing transcript / spawn-anchor upload | Upload `POST /v1/ingest/transcript`. Unanchored Codex threads parent to the trace root; `codex_threads_unanchored` increments. |
| `202` on attribution-repair | Correction committed; sync derive did not finish | Do not retry. Wait for the worker. |
| Worker OOM / API OOM | Derive sharing the API pod | Run `tapes serve derive-worker` with its own memory limit. |

## Next

<CardGroup>
<Card title="Sessions, traces, and spans" href="/sessions-traces-spans">
Deterministic IDs, span kinds, and how transcript anchors rejoin subagent threads.
</Card>
<Card title="Read API vs ingest" href="/read-vs-ingest">
Why `:8081` and `:8082` are separate sealed contracts.
</Card>
<Card title="Ingest API" href="/ingest-api">
`POST /v1/ingest` fields, body cap, and the JSON error envelope.
</Card>
<Card title="Prove the capture ratchet" href="/capture-ratchet">
`off` / `dual` / `raw` and `tapes raw equivalence`.
</Card>
<Card title="Split the stack" href="/split-the-stack">
Run proxy, ingest, derive-worker, and embed-worker as separate processes.
</Card>
<Card title="Gateway capture" href="/gateway-capture">
`tapes-extproc` Envoy adapter and `RawResponseMode`.
</Card>
</CardGroup>