# Troubleshooting

> No-provenance, blocked candidate pool, 409 export-not-ready, SHA-256 mismatch, Harbor infrastructure failures, and leftover Docker resources.

- Repository: mupt-ai/self-bench
- GitHub: https://github.com/mupt-ai/self-bench
- Human docs: https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc
- Complete Markdown: https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc/llms-full.txt

## Source Files

- `src/harbor-results.ts`
- `src/run-wait.ts`
- `src/cli.ts`
- `src/api.ts`
- `src/temporal/workflow.ts`
- `tests/harbor-results.test.ts`
- `docs/operations.md`

---

---
title: "Troubleshooting"
description: "No-provenance, blocked candidate pool, 409 export-not-ready, SHA-256 mismatch, Harbor infrastructure failures, and leftover Docker resources."
---

`self-bench run`, `self-bench download`, `GET /v1/runs/:runId/export`, Harbor nop/oracle gates, and the Docker sandbox executor fail with a small set of stable strings and statuses. Treat `blocked`, `failed`, and `cancelled` as terminal unsuccessful run phases. `complete` is the only phase that attaches an `export` artifact. Candidate-level `infrastructure_failed` does not fail the workflow and is not replaced.

<Info>
`--output` implies `--wait` (2s poll). Closing the waiting CLI does not cancel Temporal. A stopped worker or Compose stack pauses work until the worker polls again.
</Info>

## Symptom index

| You see | Surface | Meaning |
| --- | --- | --- |
| `no sanitized local-session or GitHub pull-request provenance was found` | `self-bench run` (before `POST /v1/runs`) | Local session scan plus `gh pr list` both produced zero retained messages |
| `SelfBench run blocked` | `--wait` / `--output` | Discovery could not fill the authoring budgets |
| `candidate pool exhausted; missing easy=…, medium=…, hard=…` | Temporal `ApplicationFailure` type `CandidatePoolExhausted` | Same condition; counts are missing *authoring* slots, not accepted-task shortfalls |
| `{ "error": "run export is not ready" }` HTTP 409 | `GET /v1/runs/:runId/export`, `self-bench download` | Status has no `export` field yet (or the run never reached `complete`) |
| `downloaded export failed its SHA-256 integrity check` | `self-bench download` / `--output` | Streamed bytes ≠ `x-content-sha256`; dest file is deleted |
| `SelfBench API returned an export without integrity metadata` | download | Missing `x-content-sha256` or empty body |
| `Harbor … infrastructure failure` / `HarborInfrastructureFailure` | worker validation, `self-bench-validate` | Environment/auth/runtime, not a task assertion failure |
| `infrastructure_failed` on a task | `GET /v1/runs/:runId` | Harbor infra exhausted activity retries; candidate is not replaced; export still runs |
| leftover `selfbench-*` containers or volumes | host Docker | Crash or kill before `docker rm` / `volume rm` |
| `all predefined address pools have been fully subnetted` | Harbor trial `exception_info` | Docker network pool exhaustion (leftover networks) |

## Inspect the run first

```bash
export SELFBENCH_API_URL="${SELFBENCH_API_URL:-http://127.0.0.1:8080}"
self-bench list
self-bench status RUN_ID
```

<ParamField body="phase" type="string">
`queued` · `discovering` · `authoring` · `exporting` · `complete` · `blocked` · `failed` · `cancelled`
</ParamField>

<ParamField body="requestedByDifficulty" type="object">
Authoring budgets submitted as `easy` / `medium` / `hard`. Rejected candidates are not refilled.
</ParamField>

<ParamField body="discovery" type="object">
`wave`, `totalShards` (8), `completedShards`, `failedShards`, `candidates`.
</ParamField>

<ParamField body="tasks[].status" type="string">
`authoring` · `auditing` · `validating` · `reviewing` · `repairing` · `infrastructure_failed` · `rejected` · `accepted`
</ParamField>

<ParamField body="export" type="object">
Present only after `buildExport`. Fields: `uri`, `sha256` (64 hex), `sizeBytes`, `contentType`.
</ParamField>

`self-bench list` reports Temporal execution names (`RUNNING`, `COMPLETED`, `FAILED`, `CANCELED`), not the SelfBench `phase` field. Use `status` for phase, discovery, and per-candidate reasons.

```mermaid
stateDiagram-v2
  [*] --> discovering
  discovering --> blocked: no unique PRs or 300-candidate cap
  discovering --> authoring: budgets filled
  authoring --> exporting: all selected candidates finished
  exporting --> complete: export artifact written
  authoring --> failed: non-Harbor activity error
  discovering --> cancelled: Temporal cancel
  authoring --> cancelled: Temporal cancel
  blocked --> [*]: GET export 409
  failed --> [*]: GET export 409
  cancelled --> [*]: GET export 409
  complete --> [*]: GET export 200 + x-content-sha256
```

A candidate marked `infrastructure_failed` stays in `authoring`→`exporting`→`complete`. That is not `blocked`.

## No provenance

`self-bench run` collects two corpora on the operator machine, concatenates them, and refuses to start if the merge is empty:

```text
no sanitized local-session or GitHub pull-request provenance was found
```

This throw happens **before** `POST /v1/provenance` and `POST /v1/runs`. A `gh` failure is a different error (`gh exited N: …`).

### What is collected

| Source | Location / command | Kept when |
| --- | --- | --- |
| Pi | `$HOME/.pi/agent/sessions/**/*.jsonl` | File content or path contains a worktree path (or slash→dash encoding) |
| Claude Code | `$HOME/.claude/projects/**/*.jsonl` | Same path association |
| Codex | `$HOME/.codex/sessions` and `$HOME/.codex/archived_sessions` `**/*.jsonl` | Same path association |
| GitHub | `gh pr list --state merged --limit 500` | Non-draft, non-bot author, non-empty title, ≥20 changed lines, ≥1 changed file |

Only user-role session text is kept. Messages that redact to empty or start with injected prefixes (`# AGENTS.md instructions`, `<environment_context>`, `<skill name=`, `## Memory`, …) are dropped. GitHub bodies longer than 12 000 characters are discarded; the title is still kept. Bot authors are `is_bot === true` or a login ending in `[bot]`.

The CLI talks to `gh` on `PATH` for this scan. The worker separately needs `GH_TOKEN` (or `gh auth token` fallback inside activities) to clone during discovery/authoring.

<Steps>
<Step title="Confirm origin and worktrees">
`--repo` must be a Git checkout whose `origin` is `https://github.com/…` or `git@github.com:…`. Other remotes throw `unsupported origin URL`. Run `git -C REPO worktree list --porcelain` and confirm session files mention that absolute path (Claude project dirs often use the slash-encoded form).
</Step>
<Step title="Confirm gh can list merged PRs">
```bash
gh pr list --repo OWNER/NAME --state merged --limit 20 \
  --json number,title,author,isDraft,additions,deletions,changedFiles
```
PRs under 20 changed lines, drafts, and bots never enter the corpus. Repositories with only bot or tiny PRs and no local sessions fail this gate.
</Step>
<Step title="Point HOME at the machine that has the sessions">
Collection uses `process.env.HOME`. A worker container, CI user, or `sudo` with a different home will scan empty roots even if sessions exist on the laptop that authored the PRs.
</Step>
</Steps>

<Warning>
An empty accepted export is not this error. Discovery can fill budgets from GitHub titles alone; later audit/validation/review can reject every candidate and still reach `complete` with `acceptedCount: 0`.
</Warning>

## Blocked candidate pool

Discovery loops until every tier budget is filled, a wave adds **zero new `sourcePr` values**, or the discovered set hits **300**. Those last two terminals set `phase` to `blocked` and throw a non-retryable `ApplicationFailure`:

```text
candidate pool exhausted; missing easy=N, medium=N, hard=N
```

`--wait` / `--output` then throw `SelfBench run blocked`. The status JSON often has **no** `error` field on this path (the workflow does not copy the ApplicationFailure message onto `status` when the phase is already `blocked`). Read `requestedByDifficulty`, `discovered`, and `discovery.failedShards` instead.

### Why discovery stops short

- Eight shards per wave. Each shard is asked for `min(8, ceil(missing / 8) + 3)` candidates per non-zero tier.
- Retryable shard failures increment `failedShards` and contribute `[]`. Non-retryable shard failures (`InvalidDiscoveryShard`, cancel) fail the run instead of blocking.
- Unique-PR filter: a wave that only rediscovers already seen PRs is treated as empty → `blocked`.
- Hard/medium/easy slots are independent. A repo that only has easy-sized PRs blocks if you requested `medium` or `hard`.
- GitHub coarse filter is 20 lines / 1 file. Medium and hard still require larger implementation diffs later; discovery may simply not find enough merged PRs.

<Tip>
Tier flags are **authoring budgets**, not accepted-task targets. If discovery fills the budget and later stages reject every candidate, the run goes to `complete` and still builds an export. Lower `--easy-count` / `--medium-count` / `--hard-count` when the repo cannot supply that many distinct merged PRs.
</Tip>

Fix: add real merged human PRs (or local sessions that map to them), lower the budgets, check `GH_TOKEN` / `gh` inside the **worker** (clone/fetch, not the CLI scan), and confirm `discovery.failedShards` is not 8/8. Failed generation runs need a **new** `--run-id`; Temporal `workflowId` is the run ID.

## 409 export is not ready

:::endpoint GET /v1/runs/:runId/export Download the completed archive
Requires `Authorization: Bearer $SELFBENCH_API_TOKEN` when the token is configured.

**200** streams `status.export` with:

| Header | Value |
| --- | --- |
| `content-type` | `status.export.contentType` (gzip) |
| `content-length` | `status.export.sizeBytes` |
| `content-disposition` | `attachment; filename="selfbench-RUN_ID.tar.gz"` |
| `x-content-sha256` | `status.export.sha256` |

**409**

```json
{ "error": "run export is not ready" }
```

when the queried status has no `export` object. `self-bench download` rethrows that string.
:::

`export` is written only after `buildExport` and `phase: "complete"`. `queued`, `discovering`, `authoring`, `exporting`, `blocked`, `failed`, and `cancelled` all 409.

<RequestExample>
```bash
self-bench status RUN_ID
self-bench download RUN_ID ./self-bench-evals.tar.gz
```
</RequestExample>

<ResponseExample>
```json
{ "error": "run export is not ready" }
```
</ResponseExample>

<Check>
Wait until `phase` is `complete` and `export.sha256` is a 64-hex digest. `--output` already waits; a bare `download` does not.
</Check>

If `phase` is `complete` but `export` is missing, the status query fell back to Temporal describe after a failed `status` query. Retry `status`. There is no delete route; a failed run cannot be overwritten in place.

## SHA-256 mismatch

Download is create-only (`open(dest, "wx")`) and hashes the stream against `x-content-sha256`.

| Failure | CLI / store message | Dest file |
| --- | --- | --- |
| Header missing or no body | `SelfBench API returned an export without integrity metadata` | removed if created |
| Stream digest ≠ header | `downloaded export failed its SHA-256 integrity check` | removed |
| Dest already exists | Node `EEXIST` from `wx` | left untouched |
| Artifact store size or digest mismatch | `artifact integrity check failed: <uri>` | n/a (API 500 on export) |
| Idempotent put collision | `artifact already exists with different contents: …` | n/a |

The API also verifies `sizeBytes` and `sha256` when opening the artifact (local file or GCS object metadata + streamed digest). A 200 with a bad body is a transport/corruption problem; a 500 with `artifact integrity check failed` is a store problem.

`manifest.json` inside the archive also lists per-task SHA-256 values. That inner digest is not a signature: it lives in the same tarball.

```bash
# After a successful download
python3 - <<'PY'
import hashlib, sys
p = sys.argv[1]
print(hashlib.sha256(open(p,"rb").read()).hexdigest())
PY ./self-bench-evals.tar.gz
# must match the x-content-sha256 you saw / status.export.sha256
```

Retry the download to a **new** path. Do not reuse a partial file.

## Harbor infrastructure failures

The worker treats a Harbor trial as infrastructure when `exception_info.exception_type` is `AuthError`, **or** the message matches:

| Pattern | Typical cause |
| --- | --- |
| `unknown flag: --project-name` | Harbor / Compose CLI mismatch |
| `cannot connect to the docker daemon` | Socket not mounted or daemon down |
| `error during connect` / `connection refused` | Docker or Modal endpoint unreachable |
| `modal` + `unavailable` / `timed out` / `timeout` | Modal outage or auth/profile miss |
| `image build for im-[a-z0-9]+ failed` | Modal image build |
| `all predefined address pools have been fully subnetted` | Docker network pool exhausted |

A task Dockerfile / `uv sync` failure is **not** infrastructure. That is a candidate reject (or a `self-bench-validate` `accepted: false`).

### Worker behavior

`harbor run` non-zero exit **or** a classified trial exception becomes `ApplicationFailure` type `HarborInfrastructureFailure`. Validate/review/author activities retry up to **4** times (5s → 2 min backoff). After retries:

- the candidate is recorded `infrastructure_failed` with the inner message
- it is **not** replaced
- siblings continue
- `buildExport` still runs → `complete` with whatever was accepted

`self-bench-validate` does **not** retry that way. It records `status: "error"` and:

```text
Harbor nop infrastructure failure: …
Harbor oracle infrastructure failure: …
```

or `Harbor {nop|oracle} exited N: …` (last 1000 chars of stderr).

<Tabs>
<Tab title="Docker Harbor / sandboxes">
Worker startup runs `docker info` and `docker compose version` when `SELFBENCH_EXECUTION_BACKEND` or `SELFBENCH_HARBOR_ENVIRONMENT` is `docker`. The Compose worker mounts `/var/run/docker.sock` and joins `DOCKER_GID`.

```bash
docker info
docker compose version
self-bench up --backend docker
```

Address-pool errors: remove leftover Harbor/Compose networks (see leftover resources), then rerun the gate.
</Tab>
<Tab title="Modal Harbor / sandboxes">
`AuthError` / `Token missing` is always infrastructure. Empty `MODAL_TOKEN_ID` / `MODAL_TOKEN_SECRET` are deleted at worker start so they cannot override a mounted `~/.modal.toml`. Prefer `self-bench up --backend modal` (optional `--modal-config`).

Image-build `im-…` failures: check Modal build logs for that image id; they are retried then recorded `infrastructure_failed`.
</Tab>
</Tabs>

### Incomplete Harbor jobs (`self-bench-eval`)

A job whose `result.json` has `finished_at: null` is treated as incomplete. The matrix runner renames it to `JOB.incomplete-<epoch>` and starts a new job. A finished job with the same name is reused and not rerun.

```text
Harbor produced no result (exit N): …
expected one Harbor trial result in …, found N
```

Multiple trial directories plus a single `trial_results[]` entry use the aggregate trial (stale local trial dirs). Zero trials and a finished job throw.

## Leftover Docker resources

```text
Compose project "selfbench"          Disposable sandboxes
├── api, worker, temporal, postgres  ├── container selfbench-{runId20}-{stage16}-{8hex}
├── volume selfbench_artifacts       └── volume     same name
└── volume selfbench_temporal-postgres
```

The Docker executor always `docker rm --force` and `docker volume rm --force` after a sandbox. A host crash, `kill -9`, or daemon restart can leave both. Harbor Compose networks from crashed trials can exhaust default address pools.

`self-bench down` is `docker compose --file compose.yaml down` **without** `-v`. Named volumes keep Temporal history and generated artifacts. That is persistence, not a leak.

<Steps>
<Step title="List leftover sandboxes, not the stack volumes">
```bash
docker ps -a --filter name=selfbench-
docker volume ls --filter name=selfbench-
```
Keep `selfbench_artifacts` and `selfbench_temporal-postgres` unless you intend to wipe history.
</Step>
<Step title="Remove crashed sandbox containers and volumes">
```bash
docker ps -aq --filter name='selfbench-sb-' | xargs -r docker rm -f
docker volume ls -q --filter name='selfbench-sb-' | xargs -r docker volume rm
```
Sandbox names start with `selfbench-` plus the run-id prefix (`sb-…` for generated IDs). Do not `volume rm` the Compose volumes above.
</Step>
<Step title="If Harbor reports address pools exhausted">
```bash
docker network ls
docker network prune   # only unused networks; confirm nothing else needs them
```
Then rerun `self-bench-validate` or the matrix job. Harbor eval/validate already pass `--delete`.
</Step>
<Step title="Restart the worker after Docker comes back">
```bash
self-bench up --backend docker   # or modal
```
A worker that failed `docker info` at boot will not poll until it is recreated.
</Step>
</Steps>

<Warning>
`docker compose down -v` deletes `selfbench_artifacts` and `selfbench_temporal-postgres`. SelfBench has no remote delete API; that is the only local wipe. Exports already downloaded are unaffected.
</Warning>

## Adjacent failures

<AccordionGroup>
<Accordion title="Auth and credentials">
| Message / signal | Check |
| --- | --- |
| HTTP 401 `{ "error": "unauthorized" }` | `SELFBENCH_API_TOKEN` on CLI must match the API. Compose defaults the API to `selfbench-local` if unset. |
| `SELFBENCH_API_TOKEN is required when the API listens beyond loopback` | Non-`127.0.0.1` / `::1` / `localhost` bind without a token. |
| `AuthError: Token missing…` | Harbor/Modal client auth; classified as infrastructure. |
| `Pi auth does not contain an openai-codex subscription credential` | Fallback path when `OPENAI_API_KEY` is unset. |
| `… does not contain a ChatGPT subscription token set` | Codex `auth.json` must be `auth_mode: chatgpt` with `tokens`. |
</Accordion>

<Accordion title="CLI and API input errors">
| Message | Constraint |
| --- | --- |
| `the total candidate count must be between 1 and 100` | Sum of the three tier flags |
| `--backend must be "docker" or "modal"` | `self-bench up` |
| `--modal-config requires --backend modal` | flag pairing |
| `unsupported origin URL` | origin is not GitHub HTTPS/SSH |
| `SelfBench was built without commit metadata; set SELFBENCH_BUILD_COMMIT…` | packaged CLI without git metadata |
| `request body exceeds 104857600 bytes` | `POST /v1/provenance` cap (100 MiB) |
| `request body exceeds 10485760 bytes` | other POST bodies (10 MiB) |
| Zod / `SyntaxError` | HTTP 400 |
| Duplicate `--run-id` | Temporal `workflow.start` same `workflowId` → HTTP 500 |
</Accordion>

<Accordion title="Worker paused or Temporal failed">
If the worker or Compose stack stops, activities stop heartbeating; the workflow stays `RUNNING` / `discovering` or `authoring` until the worker returns. Discovery shards: 45 min start-to-close, 8 min inactivity. Authoring: 7 h start-to-close, 8 min inactivity. Review inactivity: 5 min. `InactivityTimeoutError` looks like `{command} produced no output for {ms}ms`.

When the status query fails, the API maps Temporal `FAILED` → `phase: "failed"` and may set `error` to `Temporal workflow failed`.
</Accordion>

<Accordion title="Evaluation companions">
`self-bench-eval` requires exactly one of `--export` / `--tasks`, at least one Harbor task inside, and `--model` in `gpt-5.6-sol`, `gpt-5.6-terra`, `gpt-5.6-luna`. `--environment` must be `docker` or `modal`. `found no Harbor tasks in …` means the extract step produced no `task.toml` directories.
</Accordion>
</AccordionGroup>

## Next

<CardGroup>
<Card title="Provenance sources" href="/provenance-sources">
Session roots, GitHub PR filters, sanitization, and the empty-corpus failure.
</Card>
<Card title="Run lifecycle" href="/run-lifecycle">
Phases, eight-shard discovery, budgets vs accepted tasks, blocked / failed / cancelled.
</Card>
<Card title="Manage runs" href="/manage-runs">
`status`, `download`, create-only writes, and integrity headers.
</Card>
<Card title="Evaluate with Harbor" href="/evaluate-with-harbor">
Extract an export, `self-bench-eval` resume, incomplete job archives.
</Card>
<Card title="Start the local stack" href="/start-local-stack">
`self-bench up` / `down`, named volumes, worker stop behavior.
</Card>
<Card title="Authentication" href="/authentication">
API Bearer token, `GH_TOKEN`, Modal profile vs token secrets.
</Card>
</CardGroup>