# Overview

> What self-bench generates, who should run it, the CLI/API/worker surfaces, and the first docs routes after install.

- Repository: mupt-ai/self-bench
- GitHub: https://github.com/mupt-ai/self-bench
- Human docs: https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc
- Complete Markdown: https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc/llms-full.txt

## Source Files

- `README.md`
- `package.json`
- `src/cli.ts`
- `src/api.ts`
- `src/contracts.ts`
- `docs/task-construction.md`

---

---
title: "Overview"
description: "What self-bench generates, who should run it, the CLI/API/worker surfaces, and the first docs routes after install."
---

`self-bench` is the published npm package that turns completed work in a GitHub-origin Git checkout into a private Harbor evaluation archive. The `self-bench` CLI collects sanitized human requests and starts a Temporal `selfBenchRunWorkflow`; the HTTP API and worker author, audit, validate, and review candidates; a completed run downloads as a SHA-256-verified `.tar.gz` of accepted Harbor tasks.

<Info>
The recommended local path is Compose for Postgres, Temporal, the API, and the worker, with Modal as the sandbox and Harbor environment. `self-bench up --backend docker` keeps those sandboxes on the host instead.
</Info>

## What a run generates

A successful run exports only **accepted** tasks. Easy, medium, and hard `--*-count` values are **candidate authoring budgets** (total 1–100), not guaranteed accepted-task counts. Rejected candidates are not replaced.

Each accepted task is a native Harbor task compiled to `task.toml` schema `1.4`:

- a history-free base snapshot at the commit before the completed change
- a standalone `instruction.md` that preserves a retained human request
- hidden held-out tests (`tests/`) and a gold reference patch (`solution/`)
- agent and verifier Dockerfiles plus repository-native setup and `{tests}` commands

The outer archive is unencrypted and includes source snapshots, hidden tests, and reference solutions. Treat it as private benchmark material.

:::files
selfbench-<runId>.tar.gz
├── manifest.json          # schemaVersion 1, repo pin, build identity, task SHA-256s
└── tasks/
    └── <taskId>.tar.gz
        └── harbor-task/
            ├── task.toml
            ├── instruction.md
            ├── environment/   # agent image + repo.tar.gz
            ├── tests/         # test.patch, test.sh, verifier Dockerfile
            └── solution/      # gold.patch, solve.sh (oracle only)
:::

`manifest.json` records `runId`, `candidateCounts`, the pinned `repository.url` / `repository.commit`, `version` (`selfbenchCommit`, `executionBackend`, `sandboxImage`, `schema: 1`), `acceptedCount`, and per-task archive digests. The archive excludes Git history, local session/provenance records, and author or reviewer transcripts.

Evaluated agents receive the base snapshot, the instruction, and the declared environment. They do not receive held-out tests or the gold patch. Harbor mounts `solution/` only for the explicit oracle agent.

## Who should run it

Run self-bench when you control a Git checkout whose `origin` is a GitHub HTTPS or `git@github.com:` URL, the history contains completed work, and you want a **private** Harbor suite from that work—not a public dataset.

| Role | Uses | Needs |
| --- | --- | --- |
| Benchmark operator | `self-bench up` / `run` / `download`, then Harbor or `self-bench-eval` | Docker Compose, `gh` read access, `OPENAI_API_KEY` (or the documented Pi/Codex subscription fallbacks), and usually a Modal token |
| API client or reviewer | `SELFBENCH_API_URL` plus Bearer token; browser review UI at `/` | A running API and a completed export |
| Cloud operator | Split `node dist/api-main.js` and `node dist/temporal/worker-main.js` | Matching image digest and `SELFBENCH_TASK_QUEUE`, Temporal Cloud TLS + API key, GCS artifacts, Modal tokens |
| Maintainer | `bun install`, `bun run validate`, `bun run cli` | Bun 1.3.14+ for the documented global install and repo scripts; published bins require Node 22+ |

Do **not** point self-bench at untrusted repositories. Sandboxes check out the selected commit and run repository-native setup with a short-lived model credential. The worker is not a malware-analysis service.

Authoring submissions hard-code `authoring.provider` to `openai-codex` and `reasoningEffort` to `high`. The CLI default `--model` is `gpt-5.6-sol`. `self-bench-eval` defaults to the Codex matrix `gpt-5.6-sol`, `gpt-5.6-terra`, and `gpt-5.6-luna` at high reasoning with Harbor agent version `0.146.1`. Bring your own keys, Modal account, GitHub token, and Temporal deployment; the authoring and matrix model IDs are those literals, not a pluggable provider catalog.

<Warning>
Sanitization strips common credential forms and injected session harness text. It is not a general secret scanner. Exports remain sensitive after download.
</Warning>

## Surfaces

```mermaid
flowchart TB
  subgraph operator ["Operator machine"]
    Repo["Git checkout HEAD pin + origin"]
    Sessions["Pi / Claude Code / Codex session roots"]
    CLI["self-bench CLI"]
    ReviewFile["Local export gzip"]
  end

  subgraph control ["Control plane"]
    API["api-main.ts HTTP API"]
    UI["Review UI GET / and /assets/"]
    Temporal["Temporal selfBenchRunWorkflow"]
  end

  subgraph data ["Artifacts"]
    Store["createArtifactStore local dir or GCS"]
  end

  subgraph workerPlane ["Worker"]
    Worker["worker-main.ts activities"]
    Docker["Docker selfbench-sandbox:local"]
    Modal["Modal app selfbench"]
  end

  subgraph after ["After export"]
    Eval["self-bench-eval / harbor run"]
    HarborTask["Expanded harbor-task/"]
  end

  Repo --> CLI
  Sessions --> CLI
  CLI -->|"POST /v1/provenance then POST /v1/runs"| API
  API --> Temporal
  Temporal --> Worker
  Worker --> Store
  API --> Store
  Worker --> Docker
  Worker --> Modal
  API --> UI
  ReviewFile --> UI
  Store -->|"GET /v1/runs/:id/export"| CLI
  CLI --> HarborTask
  HarborTask --> Eval
```

The API never receives the Docker socket or model credentials. The worker owns `GH_TOKEN`, `OPENAI_API_KEY` or subscription JSON, Modal profile or `MODAL_TOKEN_*`, and Harbor execution.

### `self-bench` CLI

Published bin: `dist/cli.js`. Talks to `SELFBENCH_API_URL` (default `http://127.0.0.1:8080`) and sends `Authorization: Bearer $SELFBENCH_API_TOKEN` when that variable is set.

| Command | Behavior |
| --- | --- |
| `up` | `docker compose up -d --build`; `--backend docker` (default) or `modal`; sets `SELFBENCH_EXECUTION_BACKEND` and `SELFBENCH_HARBOR_ENVIRONMENT` together; Modal mounts `~/.modal.toml` unless `--modal-config` is set; Docker backend builds `selfbench-sandbox:local` first |
| `down` | `docker compose down`; named volumes `selfbench_temporal-postgres` and `selfbench_artifacts` remain |
| `run` | Resolves GitHub `origin` + `HEAD`, collects provenance, uploads JSONL, starts the workflow |
| `list` / `status` / `cancel` | GET `/v1/runs`, GET `/v1/runs/:runId`, POST `/v1/runs/:runId/cancel` |
| `download` | GET export; requires `x-content-sha256`; writes with create-only `wx`; deletes the file on hash mismatch |

`run` does only repository metadata and sanitized provenance upload locally. Discovery, authoring, validation, review, and audit run in the worker. Closing a waiting CLI does **not** cancel the workflow. `--output` implies `--wait`, polls until `complete`, and downloads the verified archive. Phases `blocked`, `failed`, and `cancelled` fail the wait.

```bash
self-bench --help
self-bench up --backend modal
export SELFBENCH_API_URL=http://127.0.0.1:8080
self-bench run --repo /absolute/path/to/repo --easy-count 10 --medium-count 10 --hard-count 10 --output ./self-bench-evals.tar.gz
```

`--run-id` must match `^[a-z0-9][a-z0-9-]{2,62}$`. The generated default is `sb-<UTC compact timestamp>-<8 hex chars>`. Uncommitted work is ignored; the run pins current `HEAD`.

### HTTP API

Process: `node dist/api-main.js` (`bun run dev:api` in a checkout). Listens on `SELFBENCH_API_HOST`:`SELFBENCH_API_PORT` (defaults `127.0.0.1:8080`). Binding beyond loopback without `SELFBENCH_API_TOKEN` fails at startup. Compose binds `0.0.0.0` and defaults the token to `selfbench-local` if unset.

| Method | Path | Auth | Result |
| --- | --- | --- | --- |
| `GET` | `/healthz` | none | `{ ok: true }` |
| `GET` | `/`, `/assets/*` | none | Review UI assets |
| `POST` | `/v1/provenance?runId=` | Bearer when token set | Store JSONL (max 100 MiB); `201` artifact ref |
| `POST` | `/v1/runs` | Bearer when token set | Start workflow; `202 { runId }`; 14-day execution timeout |
| `GET` | `/v1/runs` | Bearer when token set | Workflow list |
| `GET` | `/v1/runs/:runId` | Bearer when token set | `RunStatus` (phase, counts, per-task status, optional `export`) |
| `POST` | `/v1/runs/:runId/cancel` | Bearer when token set | `202 { cancellationRequested: true }` |
| `GET` | `/v1/runs/:runId/export` | Bearer when token set | gzip stream; `409` if not ready; `x-content-sha256` |

There is no remote delete route. Remove local volume data or GCS prefixes with operator tooling.

### Temporal worker

Process: `node dist/temporal/worker-main.js` (`bun run dev:worker`). Polls `SELFBENCH_TEMPORAL_NAMESPACE` / `SELFBENCH_TASK_QUEUE` (defaults `default` / `selfbench-dev`). Default activity concurrency is `1` for Docker and `20` for Modal.

Workflow phases: `queued` → `discovering` → `authoring` → `exporting` → `complete`. Terminals: `blocked` (candidate pool exhausted), `failed`, `cancelled`. Per-candidate statuses include `authoring`, `auditing`, `validating`, `reviewing`, `repairing`, `accepted`, `rejected`, and `infrastructure_failed`.

Discovery fans out across **eight** shards. If the worker or Compose stack stops, Temporal history and artifacts stay on named volumes; work resumes when a worker polls the same task queue. A stopped worker does not cancel the run.

### Companion commands

These bins operate on extracted Harbor tasks. They do not start Temporal runs.

| Bin | Job |
| --- | --- |
| `self-bench-eval` | Codex matrix (`--export` or `--tasks`) into `--jobs`; default environment `modal`, concurrency `3`; writes `summary.json`; reuses completed Harbor jobs |
| `self-bench-validate` | Harbor `nop` then `oracle` gates; writes `validation-summary.json` |
| `self-bench-reaudit` | Independent test-to-gold coupling review |
| `self-bench-repair` | Constrained repair of coupled tests; `--output` must be outside `--tasks` |
| `self-bench-agent-smoke` | Install and instantiate pinned Harbor agent adapters |

### Review UI

The API serves the Vite-built review app at `/`. The UI lists runs, opens a completed export or a local `.tar.gz`, inspects accepted tasks, drops selected tasks, and downloads `selfbench-<runId>-cleaned.tar.gz`. API list/status/export calls send the session-only Bearer token from the password field.

## Acceptance bar

An accepted task must:

1. Keep a real human request from local Pi (`~/.pi/agent/sessions`), Claude Code (`~/.claude/projects`), or Codex (`~/.codex/sessions` and `~/.codex/archived_sessions`) sessions that mention the checkout or its worktrees, or from a non-bot merged GitHub pull request (up to 500 listed).
2. Start from the pre-change base commit of that same pull request.
3. Meet the assigned difficulty size and fail-to-pass / pass-to-pass minima.
4. Pass static audit, Harbor nop, Harbor oracle, and a determinism rerun.
5. Pass an independent coupling review. The workflow allows one validation-repair attempt and one coupling-repair retry, then rejects.

If the CLI finds no sanitized local-session or GitHub pull-request messages, `run` fails before submit: `no sanitized local-session or GitHub pull-request provenance was found`. If discovery cannot fill the requested tier budgets, the workflow becomes `blocked` with `candidate pool exhausted; missing easy=…`.

| Difficulty | Implementation core | Fail-to-pass | Pass-to-pass |
| --- | --- | --- | --- |
| `easy` | ≥ 20 changed lines across ≥ 1 implementation path | ≥ 1 | no minimum |
| `medium` | ≥ 50 changed lines across ≥ 2 implementation paths | ≥ 1 | ≥ 1 |
| `hard` | ≥ 100 changed lines across ≥ 3 implementation paths | ≥ 1 | ≥ 2 |

## First path after install

<Steps>
<Step title="Install and authenticate">
Install the global CLI, create a Modal token if you will use `--backend modal`, and authenticate `gh` with repository read access. Set `OPENAI_API_KEY`, `GH_TOKEN`, and a random `SELFBENCH_API_TOKEN` in the same shell that starts Compose so the worker and API receive them.
</Step>
<Step title="Start the stack">

```bash
self-bench up --backend modal
export SELFBENCH_API_URL=http://127.0.0.1:8080
```

Verify: Compose is up, `GET http://127.0.0.1:8080/healthz` returns `{ ok: true }`, and `self-bench --help` prints the command list.
</Step>
<Step title="Submit a tiered run">

```bash
self-bench run \
  --repo /absolute/path/to/your/repository \
  --easy-count 10 \
  --medium-count 10 \
  --hard-count 10 \
  --output ./self-bench-evals.tar.gz
```

`--output` waits through Temporal phases and writes a SHA-256-checked archive. Closing this process leaves the workflow running.
</Step>
<Step title="Extract and evaluate">
Extract `manifest.json` and each `tasks/*.tar.gz` into a Harbor task directory (strip the outer `harbor-task/` prefix as in the quickstart). Run one Codex trial or `self-bench-eval --export ./self-bench-evals.tar.gz --jobs ./harbor-jobs`.
</Step>
</Steps>

<Check>
A completed download prints `{ "runId": "…", "output": "…" }`. A premature export GET is `409` with `run export is not ready`. A hash mismatch deletes the partial file.
</Check>

## Runtime defaults

| Knob | Default |
| --- | --- |
| CLI API base | `http://127.0.0.1:8080` |
| `up --backend` | `docker` |
| Authoring model | `gpt-5.6-sol` |
| Artifact store | local `.selfbench/artifacts` (Compose: `/var/lib/selfbench/artifacts`) |
| Harbor environment | same as execution backend |
| Modal image | `node:22-bookworm` |
| Worker image Harbor pin | `harbor[modal]==0.20.1.dev202608040148` |
| Provenance POST limit | 100 MiB |
| Other JSON bodies | 10 MiB |

## Next

<CardGroup>
<Card title="Installation" href="/installation">
Bun, Docker, Modal, GitHub, required environment variables, and the first `self-bench --help` check.
</Card>
<Card title="Quickstart" href="/quickstart">
Modal-backed local stack, a tiered `--output` run, extract accepted tasks, and one Codex trial.
</Card>
<Card title="Start the local stack" href="/start-local-stack">
`self-bench up` / `down`, Compose services, backends, named volumes, and a stopped worker.
</Card>
<Card title="Build a benchmark" href="/build-a-benchmark">
`self-bench run`: origin pin, 1–100 tier budgets, provenance upload, wait, and SHA-256 download.
</Card>
<Card title="Evaluate with Harbor" href="/evaluate-with-harbor">
Extract an export, run one task or the Codex matrix, resume jobs, and smoke adapters.
</Card>
<Card title="CLI reference" href="/cli-reference">
Commands, flags, run-id rules, wait and download semantics, and command-level errors.
</Card>
<Card title="HTTP API" href="/http-api">
Routes, Bearer auth, submission schema, provenance limits, export headers, and review assets.
</Card>
<Card title="Configuration reference" href="/configuration-reference">
`SELFBENCH_*` keys, loopback token rule, artifact backends, and Temporal fields.
</Card>
</CardGroup>