# Sessions, traces, and spans

> Deterministic derived IDs, session rollups, trace-as-turn, span kinds, and how subagent threads rejoin through transcript anchors.

- Repository: papercomputeco/tapes
- GitHub: https://github.com/papercomputeco/tapes
- Human docs: https://grok-wiki.com/public/docs/papercomputeco-tapes-020596f21750
- Complete Markdown: https://grok-wiki.com/public/docs/papercomputeco-tapes-020596f21750/llms-full.txt

## Source Files

- `pkg/derive/spans.go`
- `pkg/derive/reconcile.go`
- `pkg/derive/fold.go`
- `api/sessions_handlers.go`
- `migrations/1781230000_span_model.up.sql`
- `fixtures/thread/README.md`
- `docs/architecture.md`

---

---
title: "Sessions, traces, and spans"
description: "Deterministic derived IDs, session rollups, trace-as-turn, span kinds, and how subagent threads rejoin through transcript anchors."
---

The deriver projects the append-only `raw_turns` log into a prune-stable read model: one **session** per harness run, one **trace** per user-visible turn, and **spans** for the work inside that turn. Ingest writes session identity only. `EmitSpans` (after `Finish` and `ReconcileTranscripts`) mints traces, spans, links, and session rollups as a pure function of the raw layer. Re-derive upserts the same rows in place and deletes projection rows that the new pass no longer emits.

<Info>
Session primary keys are ingest-minted UUIDs, stable on the natural key `(org_id, harness_id, harness_session_id)`. Trace and span IDs are deterministic text minted from wire identity (`request_id`, `tool_use_id`, `thread_id`). They are not UUIDs and not content hashes.
</Info>

```mermaid
flowchart TB
  subgraph ingest ["Ingest :8082"]
    raw["raw_turns append-only"]
    sess["sessions identity UPSERT"]
  end
  subgraph derive ["derive-worker"]
    rec["ReconcileTranscripts"]
    emit["EmitSpans"]
    fold["session rollups"]
  end
  subgraph store ["Projection 2026-06-15"]
    turns["span_turns_20260615"]
    spans["spans_20260615"]
    links["span_links_20260615"]
  end
  subgraph read ["Read API :8081"]
    list["GET /v1/sessions"]
    traces["GET /v1/sessions/{id}/traces"]
    one["GET /v1/traces/{trace_id}"]
  end
  raw --> rec
  sess --> rec
  rec --> emit
  emit --> fold
  emit --> turns
  emit --> spans
  emit --> links
  fold --> sess
  sess --> list
  turns --> traces
  spans --> traces
  links --> traces
  turns --> one
  spans --> one
  links --> one
```

## The derived tree

| Layer | Meaning | Identity |
| --- | --- | --- |
| Session | One harness or agent run | UUID `sessions.id`; natural key `(org_id, harness_id, harness_session_id)` |
| Trace | One user-visible turn | `trc_` + `callIdentity` |
| Span | One unit of work inside that turn | Prefix + wire id (see below) |
| Span link | Causality the parent tree cannot express | `(from_trace, from_span, to_trace, to_span, from_io, to_io)` |

Containment is `parent_span_id`. Links are a separate graph (`emits`, `feeds`, `rejoin`, `verdict`, `compaction-seam`). Subagent work does **not** open a second session: every thread-labeled span is emitted into the single host trace that owns the spawn.

```text
trace  trc_<request_id>                 one user turn
└── agent  agent_main_<request_id>      name = main
    ├── event  evt_<hash[:16]>          injected context (optional)
    ├── llm    llm_<request_id>         call_kind = main, thread_id = ""
    │   └── tool  <tool_use_id>         emits → tool input
    │       ├── llm  llm_<shadow_id>    offshoot:permission-check:*  (verdict)
    │       └── agent  agent_<thread>   name = subagent  (rejoin → tool)
    │           ├── llm  llm_<child>    call_kind = main, thread_id set
    │           └── tool  <child_use>   nested spawn (Codex / Claude)
    └── llm  llm_<compaction>           offshoot:compaction → next trace
```

## Session identity

Ingest UPSERTs `sessions` on `(org_id, harness_id, harness_session_id)` and mints `id` app-side (no Postgres default). A retry of the same natural key returns the same UUID. `org_id` is unconstrained in this repo; there is no `orgs` table.

<ParamField body="harness_id" type="string" required>
Harness namespace. Empty or `"unknown"` forces a synthetic `harness_session_id`.
</ParamField>
<ParamField body="harness_session_id" type="string" required>
Harness-native session id. Empty (or missing envelope) synthesizes the first 16 hex chars of the conversation-root Merkle hash. The root is stable across turns because later calls re-send history.
</ParamField>
<ParamField body="parent_session_id" type="uuid">
Harness fork parent, resolved inside the **child's** `harness_id`. Placeholder-inserted if the parent turn has not landed. This is not the subagent-thread join.
</ParamField>
<ParamField body="auth_subject" type="string">
Gateway-stamped JWT subject captured at ingest. List filter only; it grants nothing.
</ParamField>

`DELETE /v1/sessions/{id}` removes the session and derived traces/spans (child sessions cascade). `raw_turns` is left intact.

## Deterministic derived IDs

`callIdentity` is the suffix for traces and LLM/agent-main spans:

| Condition | Suffix |
| --- | --- |
| Non-empty wire `request_id` | `request_id` unchanged |
| Empty `request_id` (legal; disables capture dedup) | `{response_node_hash[:16]}_{raw_turn_id}` |

Empty `request_id` folds in store-assigned `raw_turn_id` so two distinct calls cannot collide and overwrite on upsert. Rows **with** a `request_id` keep the historical byte-identical id.

| Kind | Span ID |
| --- | --- |
| Main agent | `agent_main_` + `callIdentity` |
| Subagent | `agent_` + `thread_id` |
| LLM | `llm_` + `callIdentity` |
| Tool | provider `tool_use_id` |
| Event | `evt_` + node hash `[:16]` |
| Trace | `trc_` + `callIdentity` of the opening call |

Primary keys: `(org_id, trace_id)` on turns; `(org_id, trace_id, span_id)` on spans. Tool keys are scoped `harness_id|harness_session_id|tool_use_id` so two sessions cannot collide on a provider-assigned id. Subagents share the parent `SessionKey`; `thread_id` is not part of that key.

`writeSpanSet` upserts the emit set, then prunes turns/spans/links for covered sessions that are no longer in the keep-set. Unchanged raw rewrites in place and prune removes zero.

The served projection generation is `schema: "2026-06-15"` (`span_turns_20260615`, `spans_20260615`, `span_links_20260615`).

## Session rollups

`GET /v1/sessions` splits capture identity from deriver-owned `rollup`. Rollup fields are empty/`unknown`/zero until the session's first derive.

<ResponseField name="id" type="string">
Ingest-minted session UUID. This is the id `tapesctl sessions get` takes, not the harness session id `tapesctl start` prints.
</ResponseField>
<ResponseField name="display_title" type="string">
Server-resolved label, never empty: `display_name` → folded `rollup.title` → preview (skipped if it looks like JSON) → harness `name` → 12-char `harness_session_id` slice → session id.
</ResponseField>
<ResponseField name="display_name" type="string">
User rename via `PATCH /v1/sessions/{id}`. Writes `sessions.display_name` only; ingest never touches it. Null or empty after trim clears it. Max 200 characters after trim. Absent field is 400.
</ResponseField>
<ResponseField name="name" type="string">
Harness slug, or folded title as fallback when no name was captured. Ingest re-sends this every turn. Render `display_title`, not `name`.
</ResponseField>
<ResponseField name="live" type="boolean">
`ended_at` is null and `last_seen_at` is within 5 minutes. Not gated on `rollup.status` — an interactive session folds `completed` after every `end_turn` while still open.
</ResponseField>
<ResponseField name="rollup" type="object">
`status`, `title`, `preview`, `turn_count`, dominant `model`, cost-ordered `model_usage`, `kind_counts`, `tasks`, and `usage` (`input_tokens`, `output_tokens`, `cost_usd`).
</ResponseField>

### Title

`offshoot:title-gen` responses fold `{"title": "…"}` onto `sessions.derived_title` (latest wins, truncated at 255 UTF-8 bytes). Title-gen is a fold, not a tree fact.

### Status

`FoldSessionStatus` runs at emit time from tool spans plus the last `call_kind=main` LLM span with `thread_id == ""`. Ingest no longer writes status; it stays `unknown` until first derive.

| Status | When |
| --- | --- |
| `failed` | Unrecovered terminal error (`length` / `max_tokens` / `content_filter` / `*error*` stop, or a non-assistant leaf with a tool error); or `tool_error_count * 2 > tool_result_count` |
| `completed` | Git commit/push anywhere (after rule 1) **or** assistant leaf with `stop` / `end_turn` / `end-turn` / `eos` / `tool_use` / `tool_use_response` |
| `abandoned` | Non-assistant leaf, no terminal error |
| `unknown` | No terminal spine span, or unrecognized stop reason |

A mid-session tool error that the model then answers is recovered, not failed. Git activity outranks a high error rate.

### Model usage, tasks, kind counts

- **`model_usage`**: every LLM span, subagent models included, priced at derive time, sorted cost-desc then model name. Share is spend, not call count.
- **`model`**: dominant conversation-spine model (cost lead).
- **`tasks`**: `TaskCreate` / `TaskUpdate` replay in `StartedAt` then `seq` order (not lexicographic `trace_id`).
- **`kind_counts`**: per-`call_kind` span tallies.

A covered session always writes these folds, including empty `[]` / `{}`. Omitting a cleared fold would leave stale JSONB after a re-derive.

## Traces as turns

A new trace opens when a conversation-spine call (`KindMain`, `thread_id == ""`) carries a **fresh genuine prompt**: a first-captured user node with no `tool_result` blocks, after the last fresh assistant node. Re-sent history shares earlier content hashes and does not reopen a trace. `/exit` resume and `/model` switch re-hash recent turns; `lastFreshAssistantIdx` gates prompt, delta input, and event emission so the resume trace keeps only the new turn.

<ResponseField name="user_prompt" type="string">
Always present. Empty means a synthetic opener, not a missing field.
</ResponseField>
<ResponseField name="response_preview" type="string">
Text of the last spine `call_kind=main` LLM span with empty `thread_id`, truncated to 280 runes. Subagent and shadow calls never supply the turn answer.
</ResponseField>
<ResponseField name="synthetic" type="string">
`post-compaction` (continuation after a compaction LLM) or `shadow-opener` (shadow call before any spine call). Absent for genuine prompts.
</ResponseField>
<ResponseField name="source" type="string">
Promoted from `raw_turns.source`: `wire` or `transcript`. Today every served trace is `wire`. Transcript rows reconcile fork edges; they do not form a trace by themselves.
</ResponseField>
<ResponseField name="usage" type="object">
Tokens and `cost_usd` over **all** LLM spans, shadow included. Cache read/creation tokens are on the total only.
</ResponseField>
<ResponseField name="main_usage" type="object">
`call_kind=main` tokens across every thread (main agent + subagents). Shadow spend is `usage − main_usage`.
</ResponseField>
<ResponseField name="tool_calls" type="integer">
Tool-span count, folded at derive so `/v1/stats` does not scan `spans`.
</ResponseField>

Readers sort spans by `seq`, not `started_at`. Parallel tool batches share one timestamp; `seq` freezes walk order (block order).

## Span kinds and call kinds

Kinds (`CHECK (kind IN ('agent', 'step', 'llm', 'tool', 'event'))`):

| `kind` | Name / role |
| --- | --- |
| `agent` | `main` root, or `subagent` for a thread |
| `llm` | One captured API call. `name` is the model (fallback `"llm"`) |
| `tool` | One `tool_use` / `server_tool_use`. `name` is the tool; Codex `exec` / `exec_command` / `shell` with a command input renders as `Bash` |
| `event` | Injected context (`injected:*`) |
| `step` | Reserved, unused |

`call_kind` is the §2g taxonomy on LLM and event spans (`ClassifyCall`):

| `call_kind` | Role |
| --- | --- |
| `main` | Conversation spine |
| `offshoot:permission-check:stage1` / `stage2` | Security monitor |
| `offshoot:title-gen` | Session title fold |
| `offshoot:plan-name-gen` | Plan name |
| `offshoot:suggestion` | Typeahead (`[SUGGESTION MODE` prefix) |
| `offshoot:web-summary` | Fetched-page / search summary |
| `offshoot:probe` | `max_tokens=1`, no tools |
| `offshoot:compaction` | Context summary (main thread only) |
| `injected:mcp-instructions`, `injected:skills-list`, `injected:mode-banner`, `injected:claude-md`, `injected:system-insert` | Injected / mid-spine system inserts |
| `unknown` | No cataloged tell — surfaced, never silently bucketed |

A subagent call classified as compaction is treated as `main` (`#27`). Compaction is a main-thread session event.

Payloads are **delta-only**: LLM `input` is content first captured on that call (no re-sent history, no `tool_result`); tool results live only on the tool span `output`. First result wins. `raw_turn_id` is set on LLM spans (0 on assembled tool/agent spans). Permission-check spans carry a typed `verdict` extracted at derive time.

## Links

| `kind` | Edge |
| --- | --- |
| `emits` | LLM output → tool input |
| `feeds` | Tool output → later LLM input |
| `rejoin` | Subagent `agent` output → spawning tool output |
| `verdict` | Shadow LLM output → judged tool |
| `compaction-seam` | Compaction LLM → next trace's first LLM (cross-trace) |

Same-trace links sit on the turn. Cross-trace links sit on `SpanSet.Links` and on the composite `GET /v1/sessions/{id}/traces` top-level `links` array.

## Subagent threads

`meta.thread_id` is harness-native. Capture observes the harness's own provider headers; it does not strip them. The language-neutral contract is `fixtures/thread/` (sealed `DIGEST`); the header spellings live in `tapes-harnesses` `envelope::thread_id`.

| Harness | Rule |
| --- | --- |
| Claude Code | Non-empty `x-claude-code-agent-id` maps verbatim. Absent = root (`""`) |
| Codex | Both `thread-id` and `session-id` required. Equal pair = root (`""`). Divergent pair = `thread-id`. Lone member = no thread id |
| Precedence | Claude list is tried before the Codex pair |

Root-guard: a non-empty thread id on a root turn misroutes `terminalMainSpan` / `responsePreview` (they require `thread_id == ""`) and degrades derived status.

### Placement

`threadCall` creates one `agent` span named `subagent` (`agent_<thread_id>`), parents it to the spawning tool span, nests the thread's LLM and tool spans under it, and adds a `rejoin` link. The whole thread conversation collapses under the **first** spawn — `followup_task` / `send_message` do not open another agent span.

Codex children share the **root** `harness_session_id`. Only the root appears in the sessions list; child usage folds into the root.

Nested threads resolve to a fixed point: a grandchild waits until its launcher has emitted the spawn tool span. Missing or ambiguous anchors parent the agent span to the **trace root** (never a guessed tool).

### Transcript anchors

The spawn join is not on the LLM wire. It arrives as transcript-source rows on `POST /v1/ingest/transcript`.

**Claude.** Per-agent transcript `meta.json` names `agent_id` + spawning `tool_use_id` (`Task`). Reconcile prefers identity (`chain.thread_id == file.agent_id`); content-block overlap is the fallback for pre-thread-id captures. The fork `ParentToolUseID` is stamped down the whole main chain.

**Codex.** Parent rollouts carry `sub_agent_activity` joining `spawn_agent` `call_id` to child `thread_id`. The uploader ships one transcript row per spawned child, keyed to the **root** session id:

- `agent_id` = child thread id
- `tool_use_id` = `spawn_agent` call id (also the tool span id)
- `records` = exactly one `kind:"started"` rollout line
- `agent_type` / `description` become `subagent_type` / `description` on the spawn tool input (console-facing; wire `spawn_agent` args stay `{task_name, fork_turns, message}`)

Codex anchors **per call**, not per chain root: `fork_turns:"all"` grafts onto the parent spine; `fork_turns:"none"` siblings share a deduped root. A node stamp would miss or collide.

Degrade ladder:

1. Exact `agent_id == thread_id`
2. Unique `agent_path` ↔ `spawn_agent` `task_name` (ambiguous reuse refuses to guess)
3. Unanchored → trace-root parent; counted in `ReconcileStats.codex_threads_unanchored`

`kind:"interacted"` rows (`followup_task`, `send_message`) are inert: excluded from every join, counted only in `codex_interacted_rows`.

Independent readers of the same header contract: `extproc/headers`, `proxy/header`, `pkg/backfill` `threadIDFromHeaders`. A rule change must land in `fixtures/thread/cases/` and `DIGEST`.

## Read API

Session ids on path/query must be UUIDs (400 otherwise). List default `limit` 50, max 200.

| Method | Path | Returns |
| --- | --- | --- |
| `GET` | `/v1/sessions` | Paged identity + `rollup`. Default sort `last_active` desc |
| `GET` | `/v1/sessions/{id}` | Session only — no conversation |
| `GET` | `/v1/sessions/{id}/traces` | Composite: session, traces with spans, session-scoped links. `?payload=full\|preview` |
| `GET` | `/v1/traces?session_id=` | Turn headers, no span payloads |
| `GET` | `/v1/traces/{trace_id}` | One turn + spans + touching links |
| `GET` | `/v1/traces/{trace_id}/spans/{span_id}` | Full payload drill-in |
| `GET` | `/v1/sessions/{id}/raw_turns` | Wire-log headers (`source`, sizes, `request_id`) |
| `PATCH` | `/v1/sessions/{id}` | `display_name` only |
| `DELETE` | `/v1/sessions/{id}` | 204; derived subtree gone, `raw_turns` kept |

`payload=preview` truncates strings at 512 runes and sets `payload` so the console can drill in. Anything other than `preview` is full.

Harness filter: `harness_session_id` alone matches across harnesses (at most one row per harness). `harness_id` alone is 400. Combined with `cursor` / `sort` / `direction` / `since` / `until` is 400. `limit` is ignored on this path.

Client commands live in `tapesctl` (`sessions list|get|traces|raw-turns`). The id they take is `sessions.id`.

Semantic search is span-only over embedded **main-conversation LLM** spans (`GET /v1/search/spans`). It does not search session objects.

## Failure modes

| Symptom | Cause |
| --- | --- |
| Session listed, `rollup.status` is `unknown`, empty usage | Identity UPSERT succeeded; derive has not folded yet |
| Subagent appears as a sibling of `main`, not under a tool | Missing or ambiguous spawn anchor (`codex_threads_unanchored`) |
| Session status wrong / answer preview from a child | Root Codex turn kept `thread-id` because the equal-pair guard failed |
| Duplicate resume traces / replayed reminders | Resume re-hash without the last-fresh-assistant gate (fixed in emit) |
| Two LLM calls collapsed into one span | Empty `request_id` on both rows **and** identical 16-char hash prefix — should not happen after the `raw_turn_id` suffix |
| Stale tasks / kind_counts after a rebuild | Fixed by writing empty folds for every covered session |
| `501` on session/trace routes | Driver is not Postgres (`sessionsReader` / `spanModelReader`) |

## Next

<CardGroup>
  <Card title="Capture and derive" href="/capture-and-derive">
    Append-only raw_turns, reduction beside raw_response, and the idempotent deriver.
  </Card>
  <Card title="Inspect and export" href="/inspect-and-export">
    List sessions, browse traces and raw_turns, optional API web UI, JSONL export.
  </Card>
  <Card title="Read API" href="/read-api">
    Compiled GET /openapi for :8081, including sessions, traces, and spans.
  </Card>
  <Card title="Search spans" href="/search-spans">
    Semantic search over embedded main-conversation LLM spans.
  </Card>
  <Card title="Ingest API" href="/ingest-api">
    POST /v1/ingest and POST /v1/ingest/transcript, including spawn-anchor rows.
  </Card>
</CardGroup>