# Troubleshooting > Wrong-port capture, missing OpenAI embed keys, ingest 413, attribution-repair 200 vs 202, --wipe data loss, and GOEXPERIMENT=jsonv2 build failures. - Repository: papercomputeco/tapes - GitHub: https://github.com/papercomputeco/tapes - Human docs: https://grok-wiki.com/public/docs/papercomputeco-tapes-020596f21750 - Complete Markdown: https://grok-wiki.com/public/docs/papercomputeco-tapes-020596f21750/llms-full.txt ## Source Files - `docs/introduction.md` - `CONTRIBUTING.md` - `ingest/errorhandler.go` - `pkg/embeddings/openai/openai.go` - `api/admin_handlers.go` - `cmd/tapes/local/local.go` - `docs/apis.md` --- --- title: "Troubleshooting" description: "Wrong-port capture, missing OpenAI embed keys, ingest 413, attribution-repair 200 vs 202, --wipe data loss, and GOEXPERIMENT=jsonv2 build failures." --- Most operator-visible failures in tapes come from mixing the three listen surfaces, starting the embedder without a key, exceeding the ingest body cap, misreading an attribution-repair status, wiping the local Postgres volume, or building without `GOEXPERIMENT=jsonv2`. | Symptom | Likely cause | First check | | --- | --- | --- | | `tapesctl start` / `capture` / `sync` reports success, `tapesctl sessions list` is empty | Capture pointed at `:8081` instead of `:8082` | `tapesctl config get tapes-url` and the `--tapes-url` on the capture command | | `tapes serve` exits with `creating embedder: OPENAI_API_KEY is required for openai embeddings` | `embedding.provider = openai` and no key in the environment or `credentials.toml` | `tapes auth --list` and `echo $OPENAI_API_KEY` | | `GET /v1/search/spans` returns `503` | Search not configured, or no embed pass has created the projection | `tapes serve` (or `tapes serve embed-worker`) and `tapes status` | | `POST /v1/ingest` or `/v1/ingest/transcript` returns `413` | Body over `MaxIngestBodyBytes` (~46.67 MiB) | Ingest logs `ingest body over limit` and `tapes_ingest_writes_total{status="reject_oversize"}` | | Attribution repair returns `202` | Correction committed; synchronous re-derive did not finish | `projections_pending` — do **not** retry the repair | | Sessions vanish after `tapes local down` | `--wipe` deleted the Postgres data directory | Confirm the command you ran; `tapes local down` without `--wipe` keeps data | | `go build` / merkle canonicalize fails in a source checkout | `encoding/json/v2` and `jsontext` need `GOEXPERIMENT=jsonv2` | `make build-local` or `nix develop` | Default local ports: proxy `:8080`, read API `:8081`, private ingest `:8082`. `tapes` owns the database; `tapesctl` is the client. ## Wrong-port capture `tapesctl` stores one URL (`tapes-url`). Read commands need the read API. Capture commands (`start`, `capture`, `sync`) need the private ingest API. Pointing capture at `:8081` can report success and store nothing. | Port | Surface | Who should call it | | --- | --- | --- | | `:8080` | Provider-compatible proxy | Agent / LLM client (`ANTHROPIC_BASE_URL`, OpenAI base URL, Ollama `/api/chat`) | | `:8081` | Read API | `tapesctl sessions`, `search`, `export`, `seed`, `skill`; `tapes status --api-target` | | `:8082` | Private ingest | `tapesctl start`, `capture`, `sync`; gateway / `tapes-extproc` `POST /v1/ingest` | A capture pointed at the read port is not a hard error. The client talks HTTP to the wrong contract and the session never lands in `raw_turns`. Configure the URL most commands want, then override ingest on capture: ```bash tapesctl config set tapes-url http://localhost:8081 tapesctl start claude --tapes-url http://localhost:8082 tapesctl start codex --tapes-url http://localhost:8082 tapesctl sync --tapes-url http://localhost:8082 ``` Precedence on the client is `--tapes-url`, then `TAPES_URL`, then `~/.tapes/config.toml`. With none of the three, a command that needs a server fails rather than guessing a host. The client always reads `~/.tapes/config.toml`; a project-local `.tapes/` configures the **server** only. ```bash tapes local status curl -sS http://localhost:8081/ping curl -sS http://localhost:8082/ping tapes status ``` ```bash tapesctl config get tapes-url tapesctl config path ``` Read commands should hit `:8081`. Capture commands must pass `--tapes-url http://localhost:8082` unless that is the stored URL. ```bash tapesctl start claude --tapes-url http://localhost:8082 tapesctl sessions list --tapes-url http://localhost:8081 ``` `start` prints the harness session id. That is **not** the UUID `tapesctl sessions get` takes. Find the derived session in the list. Fixed-port capture still writes ingest on `:8082`. Traffic to `:8080` is the provider proxy, not the Tapes HTTP contract. ## Missing OpenAI embed keys `tapes auth` stores keys for **server-side** provider calls (span embedding and skill generation). Capture is transparent: the agent brings its own credentials. Switching embeddings to OpenAI does not capture with that key. `ollama` (default) or `openai`. Environment fallback used by `openai.NewEmbedder` when the config API key is empty. When `embedding.provider` is `openai`, `tapes serve` / `tapes serve api` / `tapes serve embed-worker` build the embedder at process start. A missing key fails boot: ```text creating embedder: OPENAI_API_KEY is required for openai embeddings ``` Key resolution: 1. `credentials.APIKeyForProvider("openai", configDir)` loads `.tapes/credentials.toml` **only if** `OPENAI_API_KEY` is unset. 2. If the env var is set, the stored file is ignored and `NewEmbedder` reads `OPENAI_API_KEY`. 3. If both are empty, construction fails with the error above. ```bash tapes auth openai tapes config set embedding.provider openai tapes serve ``` ```bash export OPENAI_API_KEY=sk-... tapes config set embedding.provider openai tapes serve ``` `tapes auth openai` writes `credentials.toml` under the resolved `.tapes/` directory (never `config.toml`). Supported providers: `openai`, `anthropic`. `tapes auth --list` shows stored providers; `tapes auth --remove openai` deletes the OpenAI entry. Switching provider remaps inherited local defaults: OpenAI uses `https://api.openai.com`, `text-embedding-3-large`, and `1024` dimensions unless you set those keys explicitly. Ollama stays `http://localhost:11434`, `embeddinggemma`, `768`. The embed worker's model and dimensions must match the pgvector table. `tapes auth` warns that `sk-proj-...` keys may lack required API scopes. Prefer a service-account key (`sk-svcacct-...`) when OpenAI rejects the call. ### Search 503 after a successful serve Boot can succeed with an embedder while search still returns `503`: | Error | Meaning | | --- | --- | | `span search is not configured: embedder and span embedding store are required` | No embedder / span store on the API (split `tapes serve api` without vector config, or `--embed-spans=false` plus no store). | | `span embeddings not initialized: run the embed pass (tapes serve embed-worker or tapes dev embed-spans)` | Writer has not created the projection yet. | ```bash curl -sS 'http://localhost:8081/v1/search/spans?query=retry' # 503 until an embed pass has run tapes serve embed-worker --postgres "$TAPES_STORAGE_POSTGRES_DSN" curl http://localhost:11434/api/tags # default Ollama embeddings ``` Empty search results are not an error. A configured but uninitialized surface is `503`. If native Ollama is installed but stopped, `tapes local up` does not start it. Start `ollama serve` (or the Ollama app) and `ollama pull embeddinggemma:latest`. ## Ingest 413 (`reject_oversize`) :::endpoint POST /v1/ingest Request bodies are capped at `MaxIngestBodyBytes`. An oversize POST is rejected before parse. **Limit.** `MaxDecodedRequestBytes` (32 MiB) + `MaxRawResponseBytes` (8 MiB) encoded at 4/3 + 4 MiB reserve ≈ **46.67 MiB** (`48933546` bytes). The same Fiber `BodyLimit` applies to `POST /v1/ingest/transcript`. **Response.** HTTP `413` with the standard ingest JSON envelope: ```json {"error":"request body exceeds the ingest size limit"} ``` **Metrics / logs.** One sample of `tapes_ingest_writes_total{provider="unknown",status="reject_oversize"}` and one `ingest body over limit` warn with `content_length`, `limit`, and `path`. Provider is `unknown` because the body is never parsed. The accepted-size histogram is not updated (`bodyBytes` is 0). **Scope.** Only POST `/v1/ingest` and `/v1/ingest/transcript` take this JSON handler. A handler-returned 413 that is not `fiber.ErrRequestEntityTooLarge` keeps Fiber's default response and is not counted as `reject_oversize`. ::: A 413 means the turn did **not** land. Shrink the payload (gateway adapters already drop or truncate `raw_response` to stay under the envelope). Do not retry the same oversize body. A turn that is accepted (`202` `{"status":"accepted"}`) is on the append-only `raw_turns` log. Derivation is asynchronous. That ingest `202` is unrelated to attribution-repair `202`. The capture proxy uses the same `MaxIngestBodyBytes` as its own request `BodyLimit`. Independently, ingest drops a stored `raw_response` larger than 8 MiB and still persists the reduced turn — that path is not a 413. ## Attribution-repair 200 vs 202 :::endpoint POST /v1/admin/raw-turns/attribution-repair Operator route on the **read** API (`:8081`). Records an append-only correction for exactly one raw turn without modifying `raw_turns`, then synchronously re-derives the previous and effective sessions. Select the row with exactly one of `raw_turn_id` or `paper_proxy_request_id`. Required: `harness_id`, `harness_session_id`, `reason`. `parent_harness_session_id` must be omitted rather than empty, and must not equal `harness_session_id`. `org` is not accepted from the body (single-tenant sentinel). ::: ```bash curl -sS -X POST http://localhost:8081/v1/admin/raw-turns/attribution-repair \ -H 'Content-Type: application/json' \ -d '{ "paper_proxy_request_id": "proxy-1", "harness_id": "codex", "harness_session_id": "child", "thread_id": "thread", "reason": "hook evidence" }' ``` | Status | Meaning | Retry? | | --- | --- | --- | | `200` | Repair applied. Projection rebuild finished. | No | | `202` | Correction **committed and effective**. Synchronous rebuild did not finish. `projections_pending` names stale sessions. The derive worker converges them. | **No** — retrying records another correction | | `400` | Invalid selector or replacement attribution | Fix the body | | `404` | Raw turn not found | Check the selector | | `409` | Selector is ambiguous | Narrow to one row | | `500` | Unexpected storage failure | Investigate logs | | `501` | Driver does not implement repair (e.g. in-memory) | Use Postgres | Correction row was written. Attribution before the overlay. Replacement attribution now in force. Sessions whose rebuild is outstanding. Present on `202`. Already marked dirty in the correction transaction. Best-effort delete of an emptied source-session row failed. Cosmetic: the leftover row anchors no effective turns. Independent of the status code. **Nothing retries it.** Treat `202` as success of the repair, not as a failed write. A `500` would invite a redundant retry; `202` exists so operators do not do that. Wait for the derive worker (`tapes serve` or `tapes serve derive-worker`). `POST /v1/admin/derive/run` rebuilds projections from `raw_turns` if you need a manual pass. Seeding via `POST /v1/admin/seed/demo` is idempotent; `"overwrite": true` is rejected. ## `--wipe` data loss On `tapes local down` only. Deletes the local Postgres data directory after removing containers. Default `false`. ```bash tapes local down # stop/remove tapes-local-postgres and tapes-local-ollama; keep data tapes local down --wipe # also os.RemoveAll(/postgres) ``` The data directory is `/postgres` (project `.tapes/`, `--config-dir`, or `~/.tapes/`). That volume holds captured `raw_turns` and the derived sessions/traces/spans. `--wipe` is permanent for that local database. `tapes local down` without `--wipe` is the normal stop. `tapes local up` after a wipe starts empty Postgres; re-seed with `tapesctl seed --tapes-url http://localhost:8081`. `tapes local` requires Docker. Default images: Postgres `public.ecr.aws/g4e5l3z3/papercomputeco/postgres:17.7-pgduckdb-1.1.1` (user/password/db `tapes`, port `5432`) and `ollama/ollama:latest` (port `11434`) unless native Ollama is already running. ## `GOEXPERIMENT=jsonv2` build failures Go module version is **1.26.1**. Merkle identity (`pkg/merkle`) imports `encoding/json/v2` and `encoding/json/jsontext` and RFC 8785-canonicalizes hash input so content hashes are stable across runs. As of Go 1.26.x that path requires `GOEXPERIMENT=jsonv2`. ```bash makefile make build-local # CGO_ENABLED=0 GOEXPERIMENT=jsonv2 go build -o ./build/ ./cli/tapes ``` ```bash nix nix develop # flake exports GOEXPERIMENT=jsonv2 make build-local ``` ```bash wrong go build ./cli/tapes # fails: json v2 / jsontext are experiment-gated ``` Use `make` for development. `make test`, `make check`, and `make format` run through Dagger and need Docker. Do not point DB-backed tests at an arbitrary stock Postgres; missing `pgvector` / `pg_duckdb` looks like application failure. `make install` copies via `install(1)` (new inode) so an in-place overwrite does not invalidate a running Mach-O signature on macOS. ## Diagnostic commands ```bash tapes version tapes local status tapes status --api-target http://localhost:8081 tapes config list tapes auth --list curl -sS http://localhost:8081/ping curl -sS http://localhost:8082/ping curl -sS http://localhost:8081/metrics curl -sS http://localhost:8082/metrics # look for reject_oversize tapesctl sessions list --tapes-url http://localhost:8081 ``` `tapes status` probes the **read** API (`client.api_target`, default `http://localhost:8081`). It does not prove ingest is reachable. ## Related pages Point Claude, Codex, pi, or a generic client at ingest `:8082`. Two contracts, two ports, and the trust boundary. `POST /v1/ingest`, body cap, JSON error envelope, `reject_oversize`. `ollama` vs `openai`, `tapes auth`, embed worker. `tapes local`, `serve`, `status`, `auth`, `config`. Nix / Make, `GOEXPERIMENT=jsonv2`, `make test`.