# Split the stack

> Run proxy, api, ingest, derive-worker, and embed-worker as separate processes, including flags, DSNs, and independent failure domains.

- Repository: papercomputeco/tapes
- GitHub: https://github.com/papercomputeco/tapes
- Human docs: https://grok-wiki.com/public/docs/papercomputeco-tapes-020596f21750
- Complete Markdown: https://grok-wiki.com/public/docs/papercomputeco-tapes-020596f21750/llms-full.txt

## Source Files

- `cmd/tapes/serve/stack.go`
- `cmd/tapes/serve/serve.go`
- `cmd/tapes/serve/api/api.go`
- `cmd/tapes/serve/proxy/proxy.go`
- `cmd/tapes/serve/ingest/ingest.go`
- `cmd/tapes/serve/deriveworker/deriveworker.go`
- `cmd/tapes/serve/embedworker/embedworker.go`

---

---
title: "Split the stack"
description: "Run proxy, api, ingest, derive-worker, and embed-worker as separate processes, including flags, DSNs, and independent failure domains."
---

`tapes serve` starts the capture proxy, read API, private ingest API, in-process derive worker, and (by default) in-process embed worker in one process. The same binary also exposes `tapes serve proxy`, `tapes serve api`, `tapes serve ingest`, `tapes serve derive-worker`, and `tapes serve embed-worker` so each surface can run as its own process, with its own listen address, memory budget, and crash domain.

`tapes` is the server and owns PostgreSQL. Capture clients (`tapesctl start`, `tapesctl capture`) and the Envoy adapter (`tapes-extproc`) are not serve subcommands.

<Warning>
A fatal error from any goroutine in `tapes serve` exits the whole process. Split the workers when derivation must keep projecting after a search-backend outage, or when a large-session derive must not share the API pod's memory.
</Warning>

## Process map

```mermaid
flowchart TB
  subgraph clients["Clients"]
    agent["Agent / provider SDK"]
    ctlRead["tapesctl read commands"]
    ctlCap["tapesctl start / capture"]
    gw["tapes-extproc or gateway"]
  end

  subgraph serveProcs["tapes serve processes"]
    proxy["proxy :8080"]
    ingest["ingest :8082"]
    api["api :8081"]
    derive["derive-worker"]
    embed["embed-worker"]
  end

  pg[("PostgreSQL\nstorage.postgres_dsn\nraw_turns + derived + pgvector")]
  embProv["Embedding provider\nembedding.provider / .target"]

  agent -->|"provider paths"| proxy
  ctlCap -->|"POST /v1/ingest*"| ingest
  gw -->|"POST /v1/ingest"| ingest
  ctlRead -->|"GET /v1/*"| api

  proxy --> pg
  ingest --> pg
  derive --> pg
  embed --> pg
  api --> pg
  embed --> embProv
  api -->|"search query embed"| embProv
```

| Process | Command | Default listen | Writes | Reads |
| --- | --- | --- | --- | --- |
| Combined stack | `tapes serve` | `:8080`, `:8081`, `:8082` | proxy + ingest + in-process workers | API |
| Proxy | `tapes serve proxy` | `proxy.listen` (`:8080`) | `raw_turns` via the proxy | — |
| Ingest | `tapes serve ingest` | `ingest.listen` (`:8082`) | `raw_turns` via HTTP | — |
| API | `tapes serve api` | `api.listen` (`:8081`) | admin derive / skills / cassettes | derived model + search |
| Derive worker | `tapes serve derive-worker` | none (optional `--metrics-listen`) | sessions / traces / spans | dirty queue + `raw_turns` |
| Embed worker | `tapes serve embed-worker` | none (optional `--metrics-listen`) | span vectors | derived LLM spans |

Cassette processes are operator-owned and are not started by `tapes serve`. Admit them with `--cassettes` on `tapes serve` or `tapes serve api`.

## Combined vs split

| Behavior | `tapes serve` | Split processes |
| --- | --- | --- |
| Derive debounce | `2s` (local capture loop) | `20s` (`derive_worker.debounce`) |
| Embed interval | `10s` when `--embed-spans` is on | `1m` (`embed_worker.interval`) |
| Embed on by default | yes (`--embed-spans`, disable with `=false`) | only if you run `embed-worker` |
| Embed setup failure | logs a warning and leaves search off; stack stays up | `embed-worker` exits; API stays up |
| Creating the embedder | fails the whole stack | fails `api` / `embed-worker` only |
| First service crash | process exits | other processes keep running |
| Worker HTTP probes | not served | `--metrics-listen` serves `/metrics`, `/healthz`, `/readyz`, `/ping` |

Standalone `derive-worker` is the production form. The package comment forbids hosting that loop inside the API container: a full derive once OOM-killed a 256Mi API pod. Extra derive or embed replicas are safe — derive takes a per-session Postgres advisory lock; embed is keyed by span identity and a content hash.

## Shared DSN

Every process that persists opens `storage.postgres_dsn` (`--postgres` / `TAPES_STORAGE_POSTGRES_DSN`). `vector_store.target` (`--vector-store-target`) defaults to that same DSN. `derive-worker` and `embed-worker` refuse to start when the DSN is empty.

<ParamField body="storage.postgres_dsn" type="string" required>
Capture and derived PostgreSQL URL. Example: `postgres://tapes:tapes@localhost:5432/tapes?sslmode=disable`.
</ParamField>

<ParamField body="vector_store.target" type="string">
pgvector connection. Unset copies `storage.postgres_dsn`.
</ParamField>

<ParamField body="proxy.project" type="string">
Project tag on captured sessions. Unset uses the Git repo name.
</ParamField>

Workers use a 4-connection pool and a 10s connect timeout. Unreachable Postgres fails startup unless `--wait-for-db` is set; with that flag the worker retries with exponential backoff (cap 30s) until the DSN answers or the process is cancelled.

All five processes must see the same database. Split listen addresses, not the store.

## Flags

Precedence is flag, then `TAPES_…`, then `.tapes/config.toml`, then built-in defaults. Dots become underscores: `storage.postgres_dsn` → `TAPES_STORAGE_POSTGRES_DSN`.

Parent `tapes serve` uses `--proxy-listen` / `-p`, `--api-listen` / `-a`, `--ingest-listen` / `-i`. Each standalone HTTP command uses `--listen` / `-l` bound to the same viper key.

### Parent `tapes serve`

| Flag | Config key | Default / notes |
| --- | --- | --- |
| `--proxy-listen` / `-p` | `proxy.listen` | `:8080` |
| `--api-listen` / `-a` | `api.listen` | `:8081` |
| `--ingest-listen` / `-i` | `ingest.listen` | `:8082` |
| `--api-web-ui` | `api.web_ui` | `false` |
| `--upstream` / `-u` | `proxy.upstream` | `http://localhost:11434` |
| `--provider` | `proxy.provider` | `ollama` (`anthropic`, `openai`, `ollama`) |
| `--postgres` | `storage.postgres_dsn` | unset |
| `--project` | `proxy.project` | Git repo name |
| `--vector-store-target` | `vector_store.target` | primary DSN |
| `--embedding-provider` | `embedding.provider` | `ollama` |
| `--embedding-target` | `embedding.target` | `http://localhost:11434` |
| `--embedding-model` | `embedding.model` | `embeddinggemma` |
| `--embedding-dimensions` | `embedding.dimensions` | `768` |
| `--cassettes` | `cassettes` | OpenAPI URLs, CSV or repeated |
| `--cassette-refresh` | (flag only) | `30s` |
| `--embed-spans` | (flag only) | `true` |

### Standalone listen and write path

`tapes serve proxy` and `tapes serve ingest` still accept `--embedding-*` and `--vector-store-target` so old deployments boot. Those flags do nothing: the embed worker is the only writer of vectors.

<Note>
You do not need both write paths. Use `proxy` when clients talk a provider protocol on `:8080`. Use `ingest` when `tapesctl` or a gateway POSTs completed turns to `:8082`. Running both against the same DSN is supported and is what `tapes serve` does.
</Note>

`tapes serve api` also takes `--web-ui` (not `--api-web-ui`), `--skill-model` / `skill.model`, `--cassettes`, and `--cassette-refresh`. The API builds a search embedder when a vector-store target is set (including the default copy of the Postgres DSN). It does not create the embedding schema; until a writer has run `EnsureSchema`, `GET /v1/search/spans` answers `503`.

### `tapes serve derive-worker`

<ParamField body="--postgres" type="string" required>
`storage.postgres_dsn`
</ParamField>

<ParamField body="--project" type="string">
`proxy.project`
</ParamField>

<ParamField body="--poll-interval" type="duration">
`derive_worker.poll_interval`. Default `5s`.
</ParamField>

<ParamField body="--debounce" type="duration">
`derive_worker.debounce`. Default `20s`. A session's dirty mark must stay quiet this long before derive, unless `--max-derive-lag` fires first.
</ParamField>

<ParamField body="--max-derive-lag" type="duration">
`derive_worker.max_derive_lag`. Default `45s`. Derives a still-streaming session so live views do not wait for a quiet gap.
</ParamField>

<ParamField body="--sweep-interval" type="duration">
`derive_worker.sweep_interval`. Default `1h`, plus one sweep at startup.
</ParamField>

<ParamField body="--sweep-window" type="duration">
`derive_worker.sweep_window`. Default `24h`. Negative sweeps all history (full re-derive after a deriver change).
</ParamField>

<ParamField body="--metrics-listen" type="string">
`derive_worker.metrics_listen`. Empty disables the listener.
</ParamField>

<ParamField body="--wait-for-db" type="boolean">
`derive_worker.wait_for_db`. Default fail-fast.
</ParamField>

The worker derives one session at a time. Poll failures back off exponentially, capped at 30s. SIGINT/SIGTERM drains the in-flight derive for up to 30s; a second signal kills immediately. If `GOMEMLIMIT` is unset, the process applies a cgroup-derived soft heap limit so a large-session re-parse does not OOM the container.

`POST /v1/admin/derive/run` on the API remains an on-demand escape hatch. It is not a substitute for the worker loop.

### `tapes serve embed-worker`

<ParamField body="--interval" type="duration">
`embed_worker.interval`. Default `1m`. One pass also runs at startup.
</ParamField>

<ParamField body="--batch-size" type="int">
`embed_worker.batch_size`. Default `100` when `0`.
</ParamField>

<ParamField body="--max-text-bytes" type="int">
`embed_worker.max_text_bytes`. Default `1 MiB` when `0`; negative disables. Oversized spans are recorded as `too_large`, not chunked.
</ParamField>

<ParamField body="--org" type="string">
`embed_worker.org`. Optional org UUID filter; default all orgs.
</ParamField>

<ParamField body="--metrics-listen" type="string">
`embed_worker.metrics_listen`
</ParamField>

<ParamField body="--wait-for-db" type="boolean">
`embed_worker.wait_for_db`
</ParamField>

`--embedding-provider`, `--embedding-target`, `--embedding-model`, and `--embedding-dimensions` must be a matching pair. `EnsureSchema` runs at startup and fails the process if an existing vector table disagrees. API keys come from `tapes auth` / `credentials.toml` via `APIKeyForProvider`, not from `config.toml`.

A per-span provider error is counted and retried on the next pass. Infrastructure failures back off up to 5m so a down backend is not hammered every interval. Drain and second-signal behavior match the derive worker.

## Run a split stack

<Steps>
<Step title="Share one DSN">
Point every process at the same store. `tapes local up` writes `storage.postgres_dsn` into the active `.tapes/config.toml`; in an orchestrator, inject `TAPES_STORAGE_POSTGRES_DSN` (or `--postgres`) on each container.
</Step>

<Step title="Start the write path you need">

<CodeGroup>
```bash title="Gateway / tapesctl capture"
tapes serve ingest \
  --listen :8082 \
  --postgres "$TAPES_STORAGE_POSTGRES_DSN"
```

```bash title="Fixed-port provider proxy"
tapes serve proxy \
  --listen :8080 \
  --provider anthropic \
  --upstream https://api.anthropic.com \
  --postgres "$TAPES_STORAGE_POSTGRES_DSN"
```
</CodeGroup>

Treat ingest as a private in-cluster write surface. Do not expose `:8082` as a public app port.
</Step>

<Step title="Start derivation and embedding as separate workloads">

```bash
tapes serve derive-worker \
  --postgres "$TAPES_STORAGE_POSTGRES_DSN" \
  --metrics-listen :9091 \
  --wait-for-db

tapes serve embed-worker \
  --postgres "$TAPES_STORAGE_POSTGRES_DSN" \
  --metrics-listen :9092 \
  --wait-for-db
```

Scale by adding replicas. Do not put either loop back inside the API process.
</Step>

<Step title="Start the read API">

```bash
tapes serve api \
  --listen :8081 \
  --postgres "$TAPES_STORAGE_POSTGRES_DSN" \
  --cassettes http://hello-world:9999/openapi
```

Point `tapesctl` read commands at `:8081` and capture commands at `:8082`.
</Step>

<Step title="Verify each failure domain">

```bash
curl -sS http://127.0.0.1:8081/ping
curl -sS http://127.0.0.1:8082/ping
curl -sS http://127.0.0.1:9091/healthz   # derive liveness; never depends on Postgres
curl -sS http://127.0.0.1:9091/readyz    # 503 until the dirty queue is pollable
curl -sS http://127.0.0.1:9092/readyz    # 503 until embed-worker can ping Postgres
tapes status --api-target http://localhost:8081
```

After a capture, wait at least the derive debounce (`20s` standalone, `2s` in `tapes serve`) before expecting sessions on `:8081`. Search stays empty until embed-worker has completed a pass.
</Step>
</Steps>

Root `docker-compose.yaml` runs the combined `serve` and publishes `:8080` and `:8081` only. The hello-world cassette compose runs `serve api` alone; add ingest and derive-worker as extra containers on the same network and DSN when you also want capture.

<RequestExample>
```bash title="Minimal three-process read path"
export TAPES_STORAGE_POSTGRES_DSN='postgres://tapes:tapes@localhost:5432/tapes?sslmode=disable'

tapes serve ingest --listen :8082 &
tapes serve derive-worker --wait-for-db &
tapes serve api --listen :8081 &

tapesctl config set tapes-url http://localhost:8081
tapesctl start claude --tapes-url http://localhost:8082
```
</RequestExample>

## Independent failure domains

| If this process dies | What continues | What stops |
| --- | --- | --- |
| `proxy` | ingest, derive, embed, API | new `:8080` captures |
| `ingest` | proxy, derive, embed, API | `tapesctl` / gateway POSTs to `:8082` |
| `api` | both write paths and both workers | reads, search, MCP, cassette proxy, admin derive |
| `derive-worker` | append-only `raw_turns`, embed of already-derived spans, API | new sessions / traces / spans go stale until the worker (or `POST /v1/admin/derive/run`) catches up |
| `embed-worker` | capture and derivation | search vectors go stale; configured-but-uninitialized search stays `503` |
| embedding backend | derive, ingest, proxy | embed-worker retries with backoff; query-time search embed on the API may fail |

`/healthz` on a worker is process liveness and must not depend on Postgres — a store outage must not get the pod killed. `/readyz` is the store (and, for derive, a pollable dirty queue). The metrics listener starts before the database connect so `/healthz` answers while `--wait-for-db` retries.

Embedding is never a step of the derive loop. A slow or down Ollama/OpenAI backend cannot stall projection of `raw_turns` into sessions, traces, and spans.

## What is not a serve process

- **`tapesctl`** — client. Capture talks to ingest; list/search/export talk to the API.
- **`tapes-extproc`** — separate image (`Dockerfile.extproc`). It POSTs completed turns to ingest and must not depend on the API, the database, or either worker being healthy.
- **Cassette binaries** — operator lifecycle. `tapes serve api --cassettes` only fetches their OpenAPI documents.

## Troubleshooting

| Symptom | Check |
| --- | --- |
| Combined stack exits when one listener dies | Expected. Split the failing service. |
| Sessions missing after capture | Derive worker down, or still inside debounce / max-lag. Confirm dirty-queue work with `--metrics-listen` or `POST /v1/admin/derive/run`. |
| Search `503` | Embed worker never ran `EnsureSchema`, or model/dimensions disagree with the vector table. |
| Search empty with `200` | Embed worker not running, or backend down. Derivation can still be current. |
| Worker exits immediately | Empty `--postgres`, or Postgres unreachable without `--wait-for-db`. |
| API boot fails on embeddings | Embedder constructor error. Fix `embedding.*` / `tapes auth`, or stop requiring search on that process. |
| Capture on the wrong port | `:8080` is the provider proxy; `:8082` is ingest. Mixing them drops the session. |
| Deprecated embedding flags on proxy/ingest | Logged as retired. Configure `embed-worker` instead. |

## Related pages

<CardGroup>
<Card title="Read API vs ingest" href="/read-vs-ingest">
Two ports and the trust boundary between `:8081` and `:8082`.
</Card>
<Card title="Capture and derive" href="/capture-and-derive">
Append-only `raw_turns` and the idempotent projection the derive worker runs.
</Card>
<Card title="Configure embeddings" href="/configure-embeddings">
Provider, model, dimensions, `tapes auth`, and disabling the embed worker.
</Card>
<Card title="Gateway capture" href="/gateway-capture">
`tapes-extproc` POSTs completed turns to ingest without sharing the serve process set.
</Card>
<Card title="CLI reference" href="/cli-reference">
Parent `tapes serve` flags and the rest of the server command surface.
</Card>
<Card title="Configuration reference" href="/configuration-reference">
Flag, `TAPES_`, and `config.toml` precedence for the keys on this page.
</Card>
</CardGroup>