# Prove the capture ratchet

> off, dual, and raw reducer modes, plus tapes raw equivalence classes, excluded fields, and the read-only comparison against stored reductions.

- Repository: papercomputeco/tapes
- GitHub: https://github.com/papercomputeco/tapes
- Human docs: https://grok-wiki.com/public/docs/papercomputeco-tapes-020596f21750
- Complete Markdown: https://grok-wiki.com/public/docs/papercomputeco-tapes-020596f21750/llms-full.txt

## Source Files

- `cmd/tapes/raw/equivalence.go`
- `pkg/rawequiv/rawequiv.go`
- `pkg/rawequiv/report.go`
- `ingest/rawreduce.go`
- `extproc/rawlane.go`
- `docs/cli.md`

---

---
title: "Prove the capture ratchet"
description: "off, dual, and raw reducer modes, plus tapes raw equivalence classes, excluded fields, and the read-only comparison against stored reductions."
---

`tapes raw equivalence` is a read-only operator scan over `raw_turns` that answers one question: if this environment flipped the capture adapter from `dual` to `raw`, would ingest's server-side reducer reproduce the reduction already stored in `response`? The command decodes stored `raw_response` bytes and runs `ingest.ReduceStoredRawTurn` — the same function ingest uses on a raw-only POST — then compares that result structurally against the adapter's stored reduction. It never migrates the schema, never writes a row, and never prints prompt or completion text.

The ratchet itself is configured on **tapes-extproc** (`TAPES_RAW_RESPONSE_MODE`), not on `tapes serve proxy`. Default is `off`. `raw` is the intended end state and is gated on a clean equivalence window; the code records that no environment is flipped to `raw` today.

## The ratchet

Two capture paths that each reduce independently can disagree. The shared `pkg/capture` reducers cannot prevent that while a second reducer still runs in the adapter. The end state is one reducer, server-side, for every wire path that can be reduced.

```mermaid
stateDiagram-v2
    [*] --> off
    off --> dual: ship reduction plus bytes
    dual --> dual: prove with tapes raw equivalence
    dual --> raw: zero blocking classes and stamp coverage is acceptable
    raw --> raw: ingest reduces only
```

| Mode | Adapter envelope | Who reduces | Stored fidelity |
| --- | --- | --- | --- |
| `off` | Reduction only. No verbatim bytes leave the process. | Adapter | `reduced` |
| `dual` | Reduction **and** verbatim upstream bytes. | Adapter (ingest keeps that reduction) | `raw` |
| `raw` | Verbatim bytes only. | Ingest, via `ReduceStoredRawTurn` | `raw` |

<ParamField body="TAPES_RAW_RESPONSE_MODE" type="off \| dual \| raw">
Environment variable on tapes-extproc. Empty or omitted is `off`. An unrecognized value is an error at parse time; `ConfigFromEnv` logs it and falls back to `off` so a typo does not stop serving traffic. Case-insensitive.
</ParamField>

`dual` is the proving rung: ingest still persists the adapter's live reduction (the adapter consumed the stream and may have seen framing the stored bytes no longer show), and stores the bytes beside it. Operator-visible derived content does not change. The bytes make "would `raw` have produced this row?" answerable offline.

## Envelope decisions on tapes-extproc

`decideRawLane` resolves mode plus per-turn size and decode facts.

- Modes that send bytes (`dual`, `raw`) attach `raw_response` only when the envelope still fits ingest's body limit after a 1 MiB reserve for the reduced response, meta, session, and JSON scaffolding. Base64 expansion of the `[]byte` field is what is measured.
- The adapter does **not** pre-drop at ingest's 8 MiB storage cap. Bytes between that cap and the transport budget are attached so ingest performs the drop-and-mark (`raw_response_dropped`, fidelity `degraded`). Dropping first would make the row look like a producer that never captured bytes.
- If the body would exceed `ingest.MaxIngestBodyBytes`, the adapter withholds bytes (`raw_response_withheld`) and keeps the reduction. Losing the whole turn at the 413 boundary is worse than losing only the verbatim lane.
- Under `raw`, if the receiver cannot decode the bytes, the turn degrades to dual rather than shipping unreducible archive (`encoding_not_decodable`). On the current dispatch path the body was already decoded with ingest's decoder, so this branch is a safety interlock for a future caller.
- Truncated/salvaged streams are not forced back to dual. Ingest salvages on the same rule as extproc (partial output plus `io.ErrUnexpectedEOF`) and reduces the result.
- A post-marshal backstop can still strip bytes (`oversize_stripped`) if the reserve was optimistic.

<Warning>
Do not switch an environment to `raw` while its ingest image predates the current decoder (zstd and stacked encodings). Under `dual` a decode mismatch is harmless because the reduction still ships. Under `raw` those bytes store as unreducible archive.
</Warning>

## Run the proof

The command lives on the **tapes** server binary, not on `tapesctl`. It needs a PostgreSQL DSN (`--postgres` or `storage.postgres_dsn`) and nothing else.

<ParamField body="--postgres" type="string" required>
PostgreSQL connection string. Bound to `storage.postgres_dsn`.
</ParamField>

<ParamField body="--since" type="duration | RFC 3339" default="24h">
Lower bound on `received_at`. A Go duration (`24h`, `168h`) is measured back from now (negative durations are flipped). An RFC 3339 instant pins a fixed window for re-runs after a reducer change.
</ParamField>

<ParamField body="--limit" type="int" default="1000">
Maximum wire turns, newest first (`received_at DESC, id DESC`). `<= 0` falls back to 1000.
</ParamField>

<ParamField body="--session" type="string">
Restrict to one `harness_session_id`.
</ParamField>

<ParamField body="--json" type="bool" default="false">
Emit the report as indented JSON instead of text.
</ParamField>

<ParamField body="--max-diffs" type="int" default="10">
Maximum structural differences recorded per divergent turn (`rawequiv.DefaultMaxDiffs`).
</ParamField>

<ParamField body="--max-report" type="int" default="20">
Maximum blocking turns listed in detail. Counts stay exact; extra blocking rows set `divergences_truncated`.
</ParamField>

<Steps>
<Step title="Collect dual traffic">
Set `TAPES_RAW_RESPONSE_MODE=dual` on tapes-extproc and capture a representative mix of providers, models, streaming vs one-shot, and encodings. Rows captured under `off` have no bytes and cannot prove anything.
</Step>
<Step title="Scan the window">
<CodeGroup>
```bash title="In-cluster against the tenant database"
kubectl exec -n <tenant-ns> deploy/tapes-api -- \
  tapes raw equivalence --since 24h --limit 5000
```

```bash title="Locally against a forwarded database"
tapes raw equivalence \
  --postgres "postgres://user:pass@127.0.0.1:15432/tapes" \
  --since 24h --limit 5000 --json
```
</CodeGroup>
</Step>
<Step title="Read the verdict, then stamp coverage">
A clean verdict with `equivalent > 0` and zero blocking classes is necessary. It is not sufficient: check whether `usage.total_duration_ns` would restore from `elapsed_seconds` or fall back to empty. Then decide whether to set `TAPES_RAW_RESPONSE_MODE=raw`.
</Step>
</Steps>

### What the scan reads

Only `source = 'wire'` rows. Transcript uploads (`POST /v1/ingest/transcript`) have no upstream bytes and are unaffected by capture mode.

The SELECT is local to the command (not a sqlc query). It does not join attribution corrections: those rewrite session membership, not bytes, reduction, or capture meta.

The pool is opened with `default_transaction_read_only=on` and `MaxConns = 1`. It does **not** go through `postgres.NewDriver` / `postgres.Open`, which run golang-migrate. Pointing the tool at a live tenant database therefore cannot mutate schema as a side effect. Every statement is a SELECT. Rows are streamed one at a time because a `raw_response` can be up to 8 MiB.

## Equivalence classes

Every examined row lands in exactly one class. The classes partition the window.

| Class | Blocks flip | Meaning |
| --- | --- | --- |
| `equivalent` | no | Re-reduction matches the stored reduction after tolerated fields are pruned. This is the count that supports a ratchet step. |
| `divergent` | **yes** | Both reductions exist and differ outside the two tolerated fields. |
| `undecodable` | **yes** | Stored bytes cannot be decoded under `raw_response_encoding`. Under `raw` the row would carry no reduction. |
| `unreducible` | **yes** | Bytes decoded but the reducer rejected them (missing `meta.content_type` on an Anthropic SSE body is the sharp case). Under `raw` the row would carry no reduction. |
| `no_reducer` | **yes** | No server-side reducer is registered for `provider`. `ollama` is in this bucket today. |
| `skipped_no_raw` | no | No verbatim bytes and none reported lost — producer was not sending raw when the turn was captured. |
| `skipped_dropped` | no | Bytes existed and were not kept (`raw_response_dropped`: producer withheld at the transport limit, or ingest dropped over the 8 MiB cap). Count these; a raw-only deployment cannot reconstruct them. |
| `skipped_no_reduction` | no | Bytes present but no adapter reduction (already raw-only, or stored JSON is not a `ChatResponse`). No second opinion to compare. |

Classification order is deliberate: `raw_response_dropped` is checked before absent bytes so a limit that bit is not reported as "producer never sent raw".

`undecodable`, `unreducible`, and `no_reducer` block alongside `divergent` because their consequence under `raw` is worse: a divergent row still has a reduction; these three store none. Ingest's `reduceRawOnly` does not fail the POST when reduction fails — the bytes land — but recovering the turn is possible in principle and not in practice. The deriver reads the reduced `response` column and does not select `raw_response`. A later reducer fix does not revive those rows on a plain re-derive.

A window with zero blocking rows and zero `equivalent` rows prints `VERDICT: no comparable turns in this window. Nothing was proven.` Dual traffic is a prerequisite, not an optional extra.

## What "equivalent" means

Comparison is structural, not byte-level. Postgres `jsonb` normalizes key order, whitespace, duplicate keys, and number formatting; the recomputed side is a fresh Go marshal. Both sides are decoded to a generic JSON tree so the storage round-trip is not reported as a divergence. Numbers compare as `float64` (so `1e2` equals `100`).

Exactly two fields are removed from both trees before the walk:

| Path | Why it is tolerated |
| --- | --- |
| `created_at` | Reducers stamp `CreatedAt` with `time.Now()` at reduction time (Anthropic). Two reductions of identical bytes taken at different instants differ by construction. OpenAI Responses copies the upstream `created_at`; the exclusion is still uniform. |
| `usage.total_duration_ns` | Proxy-measured wall clock. Neither Anthropic nor OpenAI put call duration in the body. A reduction of stored bytes cannot recover it from the payload. Under `raw`, ingest re-stamps from `meta.elapsed_seconds` (float64 seconds), which is a different rounding than the adapter's nanoseconds. |

After those paths are pruned, an emptied `usage` object is dropped on both sides so the adapter's duration-only `Usage` allocation does not surface as object-vs-absent.

Everything else is compared strictly: model, message role, every content block, stop reason, remaining usage counters, extra diagnostics, and any echoed raw body. A third difference is a `divergent` row, not a new tolerance.

### Diff shape

A difference is a JSON path plus a **rendering**, never the raw value.

| `kind` | Meaning |
| --- | --- |
| `missing_in_recomputed` | Stored reduction has a field the server-side path would lose. |
| `missing_in_stored` | Server-side reduction has a field the adapter did not store. |
| `type_mismatch` | Same path, different JSON types. |
| `value_mismatch` | Same path and type, different value. |
| `length_mismatch` | Arrays of different length; overlapping elements are also walked. |

String values print verbatim only on an allowlist of provider-controlled vocabulary (`model`, `stop_reason`, `done`, `message.role`, `message.content[].type`, `message.content[].name`, `message.content[].tool_use_id`, and a small `extra.*` set). Any other string is `string(len=N,sha256=abcd…)`. Numbers, bools, array lengths, and object key lists print as shape.

A truncated stored body that was salvaged is flagged on the outcome. A divergence on that row is expected: the adapter saw the whole stream live.

## Stamp coverage is a different question

Both excluded fields are ones `raw` restores from capture `meta`, not from the bytes. The report therefore tallies, over rows that were actually re-reduced, which source would have supplied each stamp.

Duration (`usage.total_duration_ns`):

| Source | Field | Behavior |
| --- | --- | --- |
| `elapsed_seconds` | `meta.elapsed_seconds` | Stamped when the value is finite, `> 0`, and `<= 7d`. Overwrites any provider-internal duration so every turn's duration means wall clock at the capture point. |
| `fallback` | none | Duration left empty. Derived span `duration_ns` would be NULL. |

CreatedAt (`created_at`), most precise first:

| Source | Field | Behavior |
| --- | --- | --- |
| `captured_at` | `meta.captured_at` | Completion instant (RFC 3339 Nano, same parser `derive.CapturedAt` uses). No released producer sends this yet. |
| `ts_request` | `meta.ts_request` | Request instant, plus `elapsed_seconds` when usable. Backfill rows already carry `ts_request`. |
| `fallback` | reducer default | OpenAI Responses already has the upstream timestamp; Anthropic has reduction-time `time.Now()` (ingest time under `raw`). |

A malformed present timestamp is logged and counted; it never rejects the turn.

<Warning>
A window can be perfectly `equivalent` and still lose data on the flip. If `usage.total_duration_ns` shows `fallback=N`, those N turns have no usable `meta.elapsed_seconds`. Check stamp coverage before changing `TAPES_RAW_RESPONSE_MODE`.
</Warning>

## Server-side reduction

`ReduceStoredRawTurn` is a pure function of the stored row: decode under `raw_response_encoding` (`capture.DecodeContentEncoding`; empty encoding is identity), reduce with the provider's `pkg/capture` reducer, then restore the two capture-side stamps. No database, no clock beyond what the reducer itself stamps, no metrics. Ingest emits metrics from the returned `StampSource` values; the CLI reports the same facts.

Registered reducers:

| Provider | Reducer |
| --- | --- |
| `anthropic` | `capture.NewAnthropicReducer()` |
| `openai` | `capture.NewOpenAIResponsesReducer()` |

There is no global registry. A provider with no entry never gets a server-side reduction. **Ollama traffic cannot move to `raw` until a reducer exists**; those rows class `no_reducer` and block the window.

`reduceRawOnly` on ingest is a no-op unless the payload has verbatim bytes **and** `ReducedResponseAbsent` is true. An adapter that already reduced keeps its result. That predicate is shared with the prover so the flip is measured on the same population ingest would rewrite.

## Limits that produce skipped_dropped

| Limit | Owner | Outcome |
| --- | --- | --- |
| `MaxRawResponseBytes` = 8 MiB | ingest persist | Bytes dropped, `raw_response_dropped = true`, fidelity `degraded`. Reduction, request, and session attribution still stored. |
| `MaxIngestBodyBytes` = 32 MiB request + 8 MiB×4/3 raw + 4 MiB reserve | ingest HTTP + extproc attach check | Whole body rejected (413) if it still arrives oversized. Extproc withholds bytes first so the turn is not lost. |
| Envelope reserve 1 MiB | extproc | Slack for reduction + meta + session; Dispatch re-checks the marshalled payload. |

Fidelity rollup on derived traces is worst-tier: `degraded` < `reduced` < `raw`. `skipped_dropped` does not fail the equivalence process, but a large count is a reason not to treat the window as representative of a raw-only deployment.

## Report and exit status

Text output always reprints the tolerated fields so a pasted report carries its own definition of equivalence.

```text
raw-response equivalence over 412 wire turn(s)
  window: since=24h limit=5000 session=-

  equivalent             398
  skipped_dropped          9
  skipped_no_raw           5

capture-side stamps mode=raw could restore (398 re-reduced turn(s)):
  usage.total_duration_ns  elapsed_seconds=390 fallback=8
  created_at               ts_request=398

equivalence excludes, by design:
  created_at
  usage.total_duration_ns

VERDICT: 398 turn(s) re-reduce identically. This window supports the ratchet.
```

If any class is blocking:

- stdout still holds the full report (including up to `--max-report` blocking rows with path + shape diffs, never body content).
- process exits non-zero with `raw-response equivalence failed: N of M turn(s) would not survive mode=raw`.

`--json` marshals the same `Report`: `window`, `counts`, `total`, `divergences`, `divergences_truncated`, `stamp_coverage`, `tolerated`.

Use it as a CI gate after a reducer change and before flipping `TAPES_RAW_RESPONSE_MODE` from `dual` to `raw`.

## Failure modes

| Symptom | What to check |
| --- | --- |
| `Nothing was proven` | Window is `off` traffic, transcript-only, or `--session` filtered away every dual row. Widen `--since` / `--limit` or confirm the adapter is on `dual`. |
| `undecodable` | Encoding the current ingest build cannot decode (`br` is not supported). Or bytes corrupt for the recorded encoding. Do not flip `raw` until decode matches. |
| `unreducible` | Often empty `meta.content_type` on a streaming Anthropic body: the reducer does not sniff SSE vs one-shot. Under `dual` the live reduction hides this. |
| `no_reducer` | `ollama` (or any unregistered provider) in the window. Keep those adapters on `off`/`dual`, or exclude them from the environment you intend to flip. |
| `divergent` | Real reducer disagreement. `missing_in_recomputed` is the direction that loses a field on the flip. |
| Equivalent but empty durations after a hypothetical flip | `fallback` on `usage.total_duration_ns`. Producers must send `meta.elapsed_seconds`. |
| `equivalence requires a postgres DSN` | Pass `--postgres` or set `storage.postgres_dsn`. |
| Command appears to write | It does not. If you see migrations, you invoked a different `tapes` entry point. |

## Next

<CardGroup>
<Card title="Gateway capture" href="/gateway-capture">
tapes-extproc env, gRPC recv limits, ingest POST of completed turns, and RawResponseMode.
</Card>
<Card title="CLI reference" href="/cli-reference">
tapes raw equivalence among the server commands. tapesctl is not this binary.
</Card>
<Card title="Ingest API" href="/ingest-api">
POST /v1/ingest, body cap, and 413 reject_oversize — the transport side of withheld bytes.
</Card>
<Card title="Capture and derive" href="/capture-and-derive">
Append-only raw_turns, the lossy reduction beside raw_response, and the deriver that reads response not raw_response.
</Card>
<Card title="Troubleshooting" href="/troubleshooting">
Wrong-port capture, ingest 413, and other operator failures adjacent to this gate.
</Card>
</CardGroup>