# Run lifecycle

> Temporal phases, per-candidate statuses, eight-shard discovery, candidate budgets versus accepted tasks, and blocked, failed, or cancelled terminals.

- Repository: mupt-ai/self-bench
- GitHub: https://github.com/mupt-ai/self-bench
- Human docs: https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc
- Complete Markdown: https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc/llms-full.txt

## Source Files

- `src/temporal/workflow.ts`
- `src/contracts.ts`
- `src/run-wait.ts`
- `src/cli.ts`
- `tests/workflow.test.ts`
- `tests/run-wait.test.ts`

---

---
title: "Run lifecycle"
description: "Temporal phases, per-candidate statuses, eight-shard discovery, candidate budgets versus accepted tasks, and blocked, failed, or cancelled terminals."
---

`self-bench run` starts Temporal workflow `selfBenchRunWorkflow` with `workflowId` equal to the run ID. The worker executes `executeRun`, installs the `status` query, and advances a single `phase` from `queued` through `discovering`, `authoring`, and `exporting`. The run finishes at `complete` with an export artifact, or at `blocked`, `failed`, or `cancelled`. Per-candidate `authoring` / `auditing` / `validating` / `reviewing` / `repairing` values are task statuses inside the authoring phase, not extra run phases.

<Info>
`--easy-count`, `--medium-count`, and `--hard-count` are authoring budgets. Discovery fills those slots, then each selected candidate is processed once. Rejected or infrastructure-failed candidates are not replaced. The export contains only accepted tasks, including when that set is empty.
</Info>

## Start a run

`POST /v1/runs` starts the workflow on `SELFBENCH_TASK_QUEUE` (default `selfbench-dev`) with `workflowExecutionTimeout` of 14 days. The API returns `202` and `{ "runId": "..." }` immediately. Discovery, authoring, Harbor validation, review, audit, and export run in the worker.

```bash
self-bench run \
  --repo /absolute/path/to/repository \
  --easy-count 5 \
  --medium-count 10 \
  --hard-count 5 \
  --model gpt-5.6-sol \
  --output ./self-bench-tasks.tar.gz
```

<ParamField body="runId" type="string" required>
3–63 characters matching `^[a-z0-9][a-z0-9-]{2,62}$`. CLI default is `sb-{YYYYMMDDHHmmss}-{8 hex}`. Temporal uses this as `workflowId`; a failed or cancelled run cannot be restarted under the same ID.
</ParamField>

<ParamField body="candidateCounts.easy|medium|hard" type="integer" required>
Nonnegative integers. Sum must be 1–100. Each field is also capped at 100.
</ParamField>

The CLI uploads provenance first, then submits the run. `--output` implies `--wait`. Without `--wait` or `--output`, the process exits after the `202`.

## Ownership

```mermaid
flowchart LR
  subgraph client [CLI]
    runCmd["self-bench run / status / cancel / download"]
  end
  subgraph api [API]
    postRuns["POST /v1/runs"]
    getStatus["GET /v1/runs/:runId"]
    postCancel["POST /v1/runs/:runId/cancel"]
    getExport["GET /v1/runs/:runId/export"]
  end
  subgraph temporal [Temporal]
    wf["selfBenchRunWorkflow<br/>workflowId = runId"]
    q["query status"]
  end
  subgraph worker [Worker]
    disc["discoverCandidateShard x8"]
    author["author / audit / validate / review / repair"]
    exp["buildExport"]
  end
  runCmd --> postRuns
  runCmd --> getStatus
  runCmd --> postCancel
  runCmd --> getExport
  postRuns --> wf
  getStatus --> q
  q --> wf
  postCancel --> wf
  wf --> disc
  wf --> author
  wf --> exp
  getExport --> wf
```

The API never mounts the Docker socket or model credentials. The worker polls `SELFBENCH_TASK_QUEUE` with `SELFBENCH_ACTIVITY_CONCURRENCY` (default `1` for Docker, `20` for Modal). `Promise.all` over selected candidates is therefore bounded by that concurrency.

## Run phases

```mermaid
stateDiagram-v2
  [*] --> queued
  queued --> discovering
  discovering --> authoring: each tier budget filled
  discovering --> blocked: no unique PRs or 300-candidate cap
  authoring --> exporting: every selected candidate finished
  exporting --> complete: export artifact stored
  discovering --> cancelled: Temporal cancel
  authoring --> cancelled: Temporal cancel
  discovering --> failed: non-retryable or uncaught error
  authoring --> failed: uncaught error
  exporting --> failed: export activity failed
  complete --> [*]
  blocked --> [*]
  failed --> [*]
  cancelled --> [*]
```

| `phase` | Meaning | Terminal |
| --- | --- | --- |
| `queued` | Initial query value before the first discovery wave. Also the API fallback when Temporal is `RUNNING` but the `status` query is not readable. | No |
| `discovering` | One or more eight-shard waves until every requested tier budget is filled. | No |
| `authoring` | Selected candidates run author → audit → validate → review, with at most one validation repair and one coupling repair each. | No |
| `exporting` | `buildExport` writes `manifest.json` plus accepted Harbor task archives. | No |
| `complete` | Export artifact is on the status payload. Download is allowed. | Yes, success |
| `blocked` | Discovery could not fill every requested tier. Thrown as non-retryable `CandidatePoolExhausted`. | Yes, wait fails |
| `failed` | Uncaught activity or Temporal close (`FAILED`, `TERMINATED`, `TIMED_OUT`). `error` is set when the workflow catch path ran. | Yes, wait fails |
| `cancelled` | Temporal cancellation observed (`CancelledFailure` or Temporal `CANCELED`). | Yes, wait fails |

`waitForRun` treats only `complete` as success. `blocked`, `failed`, and `cancelled` throw `SelfBench run <phase>` and, when `error` is a string, append `: <error>`. Default poll interval is 2000 ms. Phase-change lines go to stderr as JSON `{ runId, phase, accepted, rejected }`.

<Warning>
`GET /v1/runs` lists Temporal execution names (`RUNNING`, `COMPLETED`, `CANCELED`, `FAILED`). `GET /v1/runs/:runId` returns the workflow `phase`. Do not treat list `status` as `RunPhase`.
</Warning>

## Status payload

:::endpoint GET /v1/runs/:runId Live workflow status
The handler reads Temporal `describe()` plus query `status`. While the execution is `RUNNING`, or when the queried `phase` is already terminal, the query payload is returned as-is. Otherwise the API maps Temporal `COMPLETED` → `complete`, `CANCELED` → `cancelled`, `RUNNING` → `queued`, and every other close state → `failed`.
:::

<ResponseField name="runId" type="string">
Workflow ID.
</ResponseField>

<ResponseField name="phase" type="RunPhase">
One of the eight values above.
</ResponseField>

<ResponseField name="requested" type="integer">
`easy + medium + hard` from the submission.
</ResponseField>

<ResponseField name="requestedByDifficulty" type="object">
`{ easy, medium, hard }` authoring budgets.
</ResponseField>

<ResponseField name="discovered" type="integer">
Unique candidates materialized across discovery waves, capped at 300.
</ResponseField>

<ResponseField name="accepted" type="integer">
Tasks whose status is `accepted`. `infrastructure_failed` is not counted here.
</ResponseField>

<ResponseField name="rejected" type="integer">
Tasks whose status is `rejected`. `infrastructure_failed` is not counted here.
</ResponseField>

<ResponseField name="tasks" type="TaskProgress[]">
One row per selected candidate once authoring starts.
</ResponseField>

<ResponseField name="discovery" type="DiscoveryProgress">
Present after the first wave progress update: `wave`, `totalShards` (always 8), `completedShards`, `failedShards`, `candidates`.
</ResponseField>

<ResponseField name="export" type="ArtifactRef">
Set only in `complete`: `uri`, `sha256`, `sizeBytes`, `contentType`.
</ResponseField>

<ResponseField name="error" type="string">
Set on the `failed` catch path. `blocked` and `cancelled` do not populate this field on the query payload.
</ResponseField>

<ResponseExample>
```json
{
  "runId": "sb-20260814153000-a1b2c3d4",
  "phase": "authoring",
  "requested": 3,
  "requestedByDifficulty": { "easy": 1, "medium": 1, "hard": 1 },
  "discovered": 4,
  "accepted": 1,
  "rejected": 1,
  "discovery": {
    "wave": 1,
    "totalShards": 8,
    "completedShards": 8,
    "failedShards": 0,
    "candidates": 4
  },
  "tasks": [
    {
      "taskId": "easy-1-task",
      "candidateId": "w0s0-easy-1",
      "difficulty": "easy",
      "status": "accepted"
    },
    {
      "taskId": "medium-1",
      "candidateId": "medium-1",
      "difficulty": "medium",
      "status": "rejected",
      "reason": "not reproducible"
    },
    {
      "taskId": "hard-1-task",
      "candidateId": "w0s0-hard-1",
      "difficulty": "hard",
      "status": "validating"
    }
  ]
}
```
</ResponseExample>

## Candidate budgets versus accepted tasks

Discovery expands until `selectCandidates` can satisfy every requested tier, then authoring consumes **exactly** those selected rows.

| Count | What it measures |
| --- | --- |
| `requested` / `requestedByDifficulty` | Authoring slots to fill. |
| `discovered` | Unique PRs kept from shards (may exceed the budget because of overfetch). |
| Authored set | First-fit selection by interleaved shard rank, per tier, up to the budget. Extra discovered PRs are not authored. |
| `accepted` | Tasks that passed audit, Harbor nop/oracle, and coupling review. |
| Export `acceptedCount` | Same as accepted; rejected and `infrastructure_failed` rows are omitted. |

A mixed budget `{ easy: 1, medium: 1, hard: 1 }` with four discovered PRs authors three candidates. If the medium candidate is rejected, the extra easy PR is **not** substituted. The run still reaches `exporting` / `complete` with the remaining accepted tasks.

```text
requested  ── discovery waves ──► selected (exact budget)
                                      │
                                      ▼
                               author once each
                                      │
                    ┌─────────────────┼─────────────────┐
                    ▼                 ▼                 ▼
               accepted           rejected      infrastructure_failed
                    │
                    ▼
              buildExport(accepted only)
```

## Eight-shard discovery

Each wave starts eight `discoverCandidateShard` activities in parallel.

Provenance JSONL is partitioned by message index: `index % 8 === shardIndex`. An empty shard is valid. The activity prompt asks for at most the per-shard targets and forbids already-seen `sourcePr` values.

Per-tier shard target:

```text
0, if missing[tier] == 0
min(8, ceil(missing[tier] / 8) + 3), otherwise
```

Example: budget `{ easy: 1, medium: 1, hard: 0 }` requests `{ easy: 4, medium: 4, hard: 0 }` on wave 0. After easy is filled, wave 1 requests `{ easy: 0, medium: 4, hard: 0 }`.

After the eight activities return, the workflow **interleaves** shard lists (index 0 from every shard, then index 1, …) and drops duplicate `sourcePr` values. Later waves pass every already-kept PR in `excludedSourcePrs`.

| Constant | Value | Effect |
| --- | --- | --- |
| Shard count | 8 | Fixed `totalShards`. |
| Overfetch | +3 | Inflates per-shard targets so a later reject does not require a new wave for ranking — it still does not replace a rejected authored candidate. |
| Max per tier per shard | 8 | Hard cap on a shard plan. |
| Max discovered | 300 | Hitting the cap while a tier is still missing sets `blocked`. |
| Wave index | 0-based | Artifact prefix `runs/<runId>/discovery/wave-<n>/shard-<i>`. |
| Candidate ID prefix | `w{wave}s{shardIndex}-` | Applied when the plan is materialized. |

A validated shard writes `plan.json` and reuses that checkpoint on later attempts. Retryable shard failures increment `failedShards` and contribute an empty list. A non-retryable shard failure (`InvalidDiscoveryShard`, or a wrapped `ApplicationFailure` with `nonRetryable`) fails the whole run.

A wave that adds zero unique PRs while any budget remains sets `phase` to `blocked` and throws:

```text
candidate pool exhausted; missing easy=N, medium=N, hard=N
```

with type `CandidatePoolExhausted`.

## Per-candidate statuses

Selected candidates enter `tasks[]` with `taskId` initially equal to `candidateId`. After a successful author, `taskId` becomes the authored Harbor task ID. Duplicate authored task IDs are rejected (`authoring repeated task ID …`).

```mermaid
stateDiagram-v2
  [*] --> authoring
  authoring --> rejected: author outcome rejected or duplicate taskId
  authoring --> auditing: authored
  auditing --> rejected: static audit failed
  auditing --> validating: audit accepted
  validating --> repairing: Harbor nop/oracle failed
  validating --> reviewing: validation accepted
  repairing --> rejected: repair rejected or exhausted
  repairing --> auditing: repaired bundle
  reviewing --> repairing: coupling review failed
  reviewing --> accepted: review accepted
  authoring --> infrastructure_failed: HarborInfrastructureFailure
  validating --> infrastructure_failed: HarborInfrastructureFailure
```

| `tasks[].status` | When it is set |
| --- | --- |
| `authoring` | Candidate selected; `authorCandidate` running or checkpointed. |
| `auditing` | Static audit of the authored (or repaired) definition. |
| `validating` | Harbor nop, oracle, and determinism rerun. |
| `reviewing` | Independent coupling review. |
| `repairing` | One validation-repair activity, or one coupling-repair activity. |
| `accepted` | All gates passed. Included in the export. |
| `rejected` | Gate failed, repair declined, or repair attempts exhausted. Sibling candidates continue. |
| `infrastructure_failed` | `HarborInfrastructureFailure` (Docker/Modal/Harbor transport). Sibling candidates continue. Not replaced. |

Validation repair is a single activity attempt (`retry.maximumAttempts: 1`, 2-hour start-to-close). Exhaustion records `validation repair failed after its single activity attempt` and rejects **only that candidate**. Coupling repair uses the shared task-activity retry policy (4 attempts); exhaustion records `test repair failed after activity retries` and likewise rejects only that candidate.

A Temporal cancel during discovery or during either repair sets the **run** to `cancelled`.

## Authoring through export

Once every tier budget is filled, `phase` becomes `authoring` and the selected set is processed concurrently (worker concurrency still applies).

Each candidate:

1. `authorCandidate` — sandbox author, or reuse `definition.json` + `harbor-task.tar.gz` checkpoints.
2. `auditTask` — static definition audit.
3. `validateTask` — Harbor nop / oracle / determinism.
4. On validation failure: `repairValidationTask` once, then audit + validate again.
5. `reviewTask` — coupling review.
6. On review failure: `repairTask` once, then audit + validate + review again.
7. Mark `accepted` or `rejected`.

Then `phase` becomes `exporting`. `buildExport` always runs, even when `acceptedTaskIds` is empty. The archive is stored at `runs/<runId>/export/attempt-<n>/selfbench-<runId>.tar.gz` and attached to status as `export`.

:::endpoint GET /v1/runs/:runId/export Download the completed archive
Returns `409` `{ "error": "run export is not ready" }` until `export` is present. A ready response includes `x-content-sha256`, `content-type`, `content-length`, and `content-disposition: attachment; filename="selfbench-<runId>.tar.gz"`.
:::

`self-bench download` writes create-only (`wx`), verifies the streamed SHA-256 against that header, and deletes the file on mismatch.

## Observe, wait, and cancel

```bash
self-bench status RUN_ID
self-bench list
self-bench cancel RUN_ID
self-bench download RUN_ID ./self-bench-tasks.tar.gz
```

<Steps>
<Step title="Poll phase">
`self-bench status RUN_ID` prints the query payload. Use `phase`, `discovery`, and `tasks[].reason` while the run is live.
</Step>
<Step title="Wait for a terminal">
`self-bench run --wait` or `--output` polls every 2s and prints a stderr line only when `phase` changes. It returns the final status on `complete` and throws on `blocked`, `failed`, or `cancelled`.
</Step>
<Step title="Request cancel">
`POST /v1/runs/:runId/cancel` calls Temporal `cancel()` and returns `202` `{ "runId", "cancellationRequested": true }`. Activities use `WAIT_CANCELLATION_COMPLETED`. The run is not cancelled until the workflow observes the signal and sets `phase` to `cancelled`.
</Step>
<Step title="Download only after complete">
`self-bench download` against a live, blocked, failed, or cancelled run surfaces the API `409` as `run export is not ready`.
</Step>
</Steps>

## Timeouts and retries

| Surface | Start-to-close | Heartbeat | Attempts | Notes |
| --- | --- | --- | --- | --- |
| Workflow | 14 days | — | 1 execution | `workflowId` = run ID. |
| `discoverCandidateShard` | 1 hour | 10 minutes | 3 | Sandbox deadline 45 minutes; 8-minute inactivity. Retryable exhaustion becomes a failed shard, not a failed run. |
| Author / validate / review / audit / coupling repair / export | 7 hours | 10 minutes | 4 | Author sandbox 2 hours; review inactivity 5 minutes; author/discovery inactivity 8 minutes. |
| `repairValidationTask` | 2 hours | 10 minutes | 1 | Candidate rejected after that single attempt. |

Backoff for retried activities: 5 seconds initial, coefficient 2, max 2 minutes (discovery max 1 minute).

## Terminals

| Terminal | Typical cause | Export | Operator action |
| --- | --- | --- | --- |
| `complete` | Authoring finished; `buildExport` stored an archive. Accepted count may be 0. | Yes | `self-bench download` or `--output`. Inspect `tasks` for rejects. |
| `blocked` | A discovery wave added no unique PRs, or `discovered` reached 300 with a tier still missing. | No | Widen provenance, lower a tier budget, submit a **new** `runId`. Read `discovery` and the Temporal failure message `candidate pool exhausted; missing easy=…`. |
| `failed` | Non-retryable discovery input, uncaught activity error, or Temporal `FAILED` / `TERMINATED` / `TIMED_OUT`. | No | Read `error` (or the Temporal close reason if the query fell back). Fix credentials/backend and submit a new `runId`. |
| `cancelled` | `self-bench cancel` / `POST …/cancel` observed by the workflow. | No | Submit a new `runId` if you still want an export. |

<Note>
`HarborInfrastructureFailure` does **not** fail the run. The candidate is marked `infrastructure_failed` with the Harbor/Modal/Docker message, siblings continue, and the run can still `complete`.
</Note>

Duplicate `POST /v1/runs` with the same `runId` is a Temporal already-started error (API `500`), not a resume. There is no delete route for runs or artifacts.

## Errors to expect

| Signal | Meaning |
| --- | --- |
| `SelfBench run blocked` | Wait saw `phase: "blocked"`. Detail is present only if `error` is a string. |
| `SelfBench run failed: …` | Wait saw `failed` plus `error`. |
| `SelfBench run cancelled` | Wait saw `cancelled`. |
| `run export is not ready` | Download or `GET …/export` before `complete`. |
| `candidate pool exhausted; missing easy=…` | Workflow `CandidatePoolExhausted`. |
| `validation repair failed after its single activity attempt` | That candidate only. |
| `test repair failed after activity retries` | That candidate only. |
| `SelfBench status response is missing its phase` | Status JSON is not a run payload. |
| `downloaded export failed its SHA-256 integrity check` | Stream did not match `x-content-sha256`; output file removed. |

## Related pages

<CardGroup>
<Card title="Build a benchmark" href="/build-a-benchmark">
Submit `--repo`, tier budgets, `--run-id`, and `--output`.
</Card>
<Card title="Manage runs" href="/manage-runs">
`status`, `list`, `cancel`, and `download` against `SELFBENCH_API_URL`.
</Card>
<Card title="Validation and repair" href="/validation-and-repair">
Audit, Harbor nop/oracle, one validation repair, one coupling repair, then reject.
</Card>
<Card title="Difficulty gates" href="/difficulty-gates">
What discovery and authoring require for easy, medium, and hard.
</Card>
<Card title="Troubleshooting" href="/troubleshooting">
Empty provenance, blocked pools, `409` export-not-ready, and Harbor infrastructure failures.
</Card>
<Card title="Export format" href="/export-format">
`manifest.json` and per-task Harbor archives produced in `exporting`.
</Card>
</CardGroup>