# CLI reference

> self-bench commands, flags, defaults, run-id rules, wait and download semantics, and command-level errors.

- Repository: mupt-ai/self-bench
- GitHub: https://github.com/mupt-ai/self-bench
- Human docs: https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc
- Complete Markdown: https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc/llms-full.txt

## Source Files

- `src/cli.ts`
- `tests/cli.test.ts`
- `package.json`
- `src/run-wait.ts`
- `README.md`

---

---
title: "CLI reference"
description: "self-bench commands, flags, defaults, run-id rules, wait and download semantics, and command-level errors."
---

`self-bench` is the npm `self-bench` binary (`dist/cli.js`). It starts and stops the packaged Compose stack, submits a tiered Harbor-benchmark run, and talks to `SELFBENCH_API_URL` for list, status, cancel, and SHA-256-verified export download. Discovery, authoring, validation, review, and audit do not run in this process.

The package also ships separate companion binaries (`self-bench-eval`, `self-bench-validate`, `self-bench-reaudit`, `self-bench-repair`, `self-bench-agent-smoke`). Those are not subcommands of `self-bench`.

<Note>
Requires Node.js 22 or newer. From a checkout, `bun run cli -- --help` runs the same entrypoint.
</Note>

## Command inventory

| Command | Purpose | Default exit path |
| --- | --- | --- |
| `self-bench up` | Build (Docker backend only) the local sandbox image, then `docker compose up -d --build` | Prints the stack URL |
| `self-bench down` | `docker compose down` on the packaged `compose.yaml` | Named volumes stay |
| `self-bench run` | Resolve repo HEAD + origin, upload provenance, `POST /v1/runs` | Prints the 202 payload unless `--wait` / `--output` continue |
| `self-bench status RUN_ID` | `GET /v1/runs/:runId` | Pretty-printed JSON |
| `self-bench cancel RUN_ID` | `POST /v1/runs/:runId/cancel` | Pretty-printed JSON |
| `self-bench list` | `GET /v1/runs` | Pretty-printed JSON array |
| `self-bench download RUN_ID OUTPUT.tar.gz` | `GET /v1/runs/:runId/export` with create-only write + SHA-256 check | Prints `{ runId, output }` |
| `self-bench help`, `--help`, `-h`, or no args | Print usage | Exit after help |

There is no `version` command and no per-command `--help`. `up` and `run` parse flags with `parseArgs({ strict: true })`, so unknown options fail. `down` takes no flags and ignores extra argv.

<RequestExample>
```bash
self-bench --help
```
</RequestExample>

<ResponseExample>
```text
SelfBench creates durable tiered Harbor evaluations.

Usage:
  self-bench up [--backend docker|modal] [--modal-config PATH]
  self-bench down
  self-bench run --repo PATH [--easy-count N] [--medium-count N] [--hard-count N]
                  [--model MODEL] [--run-id ID] [--wait] [--output OUTPUT.tar.gz]
  self-bench status RUN_ID
  self-bench cancel RUN_ID
  self-bench download RUN_ID OUTPUT.tar.gz
  self-bench list
```
</ResponseExample>

## Invocation environment

Every HTTP command uses:

| Variable | Default | Role |
| --- | --- | --- |
| `SELFBENCH_API_URL` | `http://127.0.0.1:8080` | API origin for run, status, cancel, list, download |
| `SELFBENCH_API_TOKEN` | unset | If set, sent as `Authorization: Bearer …` |
| `SELFBENCH_BUILD_COMMIT` | unset | Full 40-hex SHA stamped into `up` and `run` when the binary has no baked commit |

<Warning>
The packaged Compose file defaults the API token to `selfbench-local` when `SELFBENCH_API_TOKEN` is unset in the host environment. The CLI only sends a Bearer header when `SELFBENCH_API_TOKEN` is a non-empty string. Export the same token the stack is using or authenticated routes return `unauthorized`.
</Warning>

`run` also shells out to `git` (origin URL + `HEAD`) and `gh` (`pr list --state merged --limit 500`). `up` / `down` require `docker` with Compose.

## `self-bench up`

Starts Postgres, Temporal, the API, and the worker from the package's `compose.yaml` (Compose project name `selfbench`). The chosen backend is written into both sandbox execution and Harbor validation.

<ParamField body="--backend" type="docker | modal" default="docker">
Sets `SELFBENCH_EXECUTION_BACKEND` and `SELFBENCH_HARBOR_ENVIRONMENT` to the same value.
</ParamField>

<ParamField body="--modal-config" type="path">
Absolute or relative path to a Modal TOML profile. Allowed only with `--backend modal`. Default when omitted: `~/.modal.toml`. The resolved path is exported as `SELFBENCH_MODAL_CONFIG_PATH` and mounted onto the worker as `/root/.modal.toml`.
</ParamField>

Behavior by backend:

| `--backend` | Extra local work | Worker execution |
| --- | --- | --- |
| `docker` | `docker build -f Dockerfile.sandbox -t selfbench-sandbox:local` | Host Docker sandboxes; Compose default concurrency is 1 |
| `modal` | No sandbox image build | Modal sandboxes; Compose default concurrency is 20 |

Both backends then run `docker compose --file <package>/compose.yaml up -d --build` with `SELFBENCH_BUILD_COMMIT` resolved (see [Build commit](#build-commit)). Success prints:

```text
SelfBench is running with the <backend> backend at http://127.0.0.1:8080
```

Compose publishes the API on `127.0.0.1:8080` and Temporal on `127.0.0.1:7233`. Named volumes `selfbench_temporal-postgres` and `selfbench_artifacts` persist Temporal history and generated artifacts.

## `self-bench down`

Runs `docker compose --file <package>/compose.yaml down`. Containers stop; named volumes are not removed. Extra arguments after `down` are ignored (`--volumes` is not forwarded).

Stopping the waiting CLI does not stop the worker. If the worker or stack is down, an already-submitted Temporal workflow pauses until a worker with the same task queue comes back.

## `self-bench run`

Local work is repository metadata plus sanitized provenance upload. The API starts a Temporal workflow (`selfBenchRunWorkflow`) whose `workflowId` is the run ID. Workflow execution timeout on the API side is 14 days.

<ParamField body="--repo" type="path" required>
Git checkout. Resolved with `path.resolve` (relative paths are cwd-relative). Short flag: `-r`.
</ParamField>

<ParamField body="--easy-count" type="nonnegative integer" default="0">
Easy-tier authoring budget.
</ParamField>

<ParamField body="--medium-count" type="nonnegative integer" default="0">
Medium-tier authoring budget.
</ParamField>

<ParamField body="--hard-count" type="nonnegative integer" default="0">
Hard-tier authoring budget.
</ParamField>

<ParamField body="--model" type="string" default="gpt-5.6-sol">
Sent as `authoringModel`. The CLI does not enumerate allowed models. The API wraps it as `authoring.provider = openai-codex` and `reasoningEffort = high`.
</ParamField>

<ParamField body="--run-id" type="string">
Caller-chosen run ID. If omitted, the CLI generates one. Not validated locally; the API regex applies (see [Run IDs](#run-ids)).
</ParamField>

<ParamField body="--wait" type="boolean" default="false">
After submit, poll until a terminal phase. Phase changes go to stderr.
</ParamField>

<ParamField body="--output" type="path">
Implies wait. After `phase === "complete"`, download the export to this path. Short flag: `-o`.
</ParamField>

### Local repository pin

`run` executes:

```bash
git -C <repo> remote get-url origin
git -C <repo> rev-parse HEAD
```

`origin` must be `git@github.com:owner/repo` (normalized to `https://github.com/owner/repo`) or any URL that already starts with `https://`. Other remotes fail with `unsupported origin URL`. Uncommitted work is never included; only the `HEAD` SHA is sent.

GitHub provenance then requires a `github.com` owner/repo URL. A non-GitHub `https://` origin passes the CLI normalizer and fails later in GitHub URL parsing.

### Candidate budgets

Each `--*-count` must be a nonnegative integer. The **sum** must be between 1 and 100 inclusive. These values are authoring budgets, not accepted-task targets. Rejected candidates are not replaced. Omitting all three counts leaves the total at 0 and fails.

### Provenance upload

The CLI concatenates:

1. Sanitized user messages from Pi (`~/.pi/agent/sessions`), Claude Code (`~/.claude/projects`), and Codex (`~/.codex/sessions`, `~/.codex/archived_sessions`) that mention the checkout or its worktrees.
2. Titles and bodies from up to 500 recent merged, non-draft, non-bot GitHub pull requests that meet the coarse size filter (at least 20 changed lines and 1 changed file).

Home directory is `HOME` if set, otherwise `os.homedir()`. If the merged list is empty, `run` throws and does not call the API:

```text
no sanitized local-session or GitHub pull-request provenance was found
```

Otherwise it `POST`s newline-delimited JSON to `/v1/provenance?runId=<id>` with `Content-Type: application/x-ndjson`. The API stores that blob under `runs/<runId>/input/provenance.jsonl` (100 MiB body limit) and returns an artifact reference (`uri`, `sha256`, `sizeBytes`, `contentType`).

### Submit payload

`POST /v1/runs` with:

```json
{
  "runId": "sb-20260814153045-a1b2c3d4",
  "repository": { "url": "https://github.com/example/project.git", "commit": "<40-hex HEAD>" },
  "provenance": { "uri": "…", "sha256": "…", "sizeBytes": 1, "contentType": "application/x-ndjson" },
  "candidateCounts": { "easy": 10, "medium": 10, "hard": 10 },
  "authoringModel": "gpt-5.6-sol",
  "selfbenchCommit": "<40-hex>"
}
```

A successful API response is HTTP 202 `{ "runId": "…" }`. The CLI always prints that object plus counts:

```json
{
  "runId": "sb-20260814153045-a1b2c3d4",
  "provenanceMessages": 42,
  "localProvenanceMessages": 30,
  "githubPullRequestMessages": 12
}
```

Without `--wait` and without `--output`, `run` returns after that print. Closing the process does not cancel the workflow.

<RequestExample>
```bash
export SELFBENCH_API_URL=http://127.0.0.1:8080
export SELFBENCH_API_TOKEN=…

self-bench run \
  --repo /absolute/path/to/repository \
  --easy-count 10 \
  --medium-count 10 \
  --hard-count 10 \
  --model gpt-5.6-sol \
  --output ./self-bench-evals.tar.gz
```
</RequestExample>

## Run IDs

The API accepts only:

```text
^[a-z0-9][a-z0-9-]{2,62}$
```

That is 3–63 characters, starting with a lowercase letter or digit, then lowercase letters, digits, or hyphens. Uppercase, underscores, and leading hyphens are rejected.

The CLI default is:

```text
sb-<YYYYMMDDHHMMSS>-<8 hex chars>
```

built from `Date.toISOString()` with non-digits stripped (first 14 digits) and the first 8 characters of `crypto.randomUUID()`. That form always matches the regex.

The same ID is:

- the provenance query parameter
- the `POST /v1/runs` `runId`
- the Temporal `workflowId`

Reusing an ID against an existing workflow fails at `POST /v1/runs` (Temporal “already started”); the CLI surfaces the API `error` string. Provenance `POST` happens first, so a failed start can still leave `runs/<runId>/input/provenance.jsonl` stored.

`status`, `cancel`, and `download` interpolate the ID into `/v1/runs/:runId`. The API route itself only matches the same regex; an invalid ID typically returns `not found` rather than a validation message.

## Wait semantics

`--output` implies `--wait`. Wait has no timeout flag and no attempt limit. It loops until the status `phase` is terminal.

| Setting | Interval | Default |
| --- | --- | --- |
| Poll interval | `GET /v1/runs/:runId` | 2000 ms |
| Phase events | stderr, only when `phase` changes | `{ runId, phase, accepted, rejected }` |
| Success | `phase === "complete"` | With `--output`, download; otherwise pretty-print the last status on stdout |
| Failure | `blocked`, `failed`, `cancelled` | Throw `SelfBench run <phase>` or `SelfBench run <phase>: <error>` when `error` is a string |

A status object without a string `phase` fails with `SelfBench status response is missing its phase`.

Worker phases the poller can observe include `queued`, `discovering`, `authoring`, `exporting`, `complete`, `blocked`, `failed`, and `cancelled`. Per-task statuses (`auditing`, `validating`, `reviewing`, `repairing`, and so on) live on the status JSON `tasks` array; the wait line on stderr only reprints `accepted` / `rejected` counts.

```mermaid
sequenceDiagram
  participant CLI as self-bench run
  participant API as SELFBENCH_API_URL
  participant WF as Temporal workflow

  CLI->>CLI: git origin + HEAD
  CLI->>CLI: local sessions + gh pr list
  CLI->>API: POST /v1/provenance?runId=
  API-->>CLI: artifact ref
  CLI->>API: POST /v1/runs
  API->>WF: start workflowId = runId
  API-->>CLI: 202 runId
  loop every 2s until terminal phase
    CLI->>API: GET /v1/runs/:runId
    API-->>CLI: phase, accepted, rejected
    Note over CLI: stderr only on phase change
  end
  alt phase complete and --output set
    CLI->>API: GET /v1/runs/:runId/export
    API-->>CLI: gzip + x-content-sha256
    CLI->>CLI: wx create, hash stream, unlink on mismatch
  else phase blocked, failed, or cancelled
    CLI-->>CLI: throw SelfBench run …
  end
```

## Download semantics

`self-bench download RUN_ID OUTPUT.tar.gz` and `run --output` share one implementation.

1. `GET {SELFBENCH_API_URL}/v1/runs/{runId}/export` with the optional Bearer token.
2. Non-OK responses must be JSON. The CLI throws `value.error` or `SelfBench API returned <status>`. Before the workflow has an export, the API returns **409** `{ "error": "run export is not ready" }`.
3. A 200 response must include `x-content-sha256` and a body. Missing either throws `SelfBench API returned an export without integrity metadata`.
4. The destination is `path.resolve(OUTPUT)` and opened with flag `wx` (create exclusively). An existing file fails with the filesystem `EEXIST` error; the CLI does not overwrite.
5. Bytes are hashed while written. If the hex digest does not match `x-content-sha256`, the file is deleted and the CLI throws `downloaded export failed its SHA-256 integrity check`.
6. Any other write/hash failure also deletes the partial file.
7. Success prints:

```json
{
  "runId": "sb-20260814153045-a1b2c3d4",
  "output": "/absolute/path/to/self-bench-evals.tar.gz"
}
```

The API also sends `content-disposition: attachment; filename="selfbench-<runId>.tar.gz"`. The CLI ignores that name and uses the path you passed.

## `status`, `cancel`, and `list`

These commands are JSON passthroughs. They do not wait and do not write files.

| Command | Method and path | Typical body |
| --- | --- | --- |
| `status RUN_ID` | `GET /v1/runs/:runId` | `runId`, `phase`, `requested`, `requestedByDifficulty`, `discovered`, `accepted`, `rejected`, `tasks`, optional `discovery`, `export`, `error` |
| `cancel RUN_ID` | `POST /v1/runs/:runId/cancel` | `{ "runId", "cancellationRequested": true }` (HTTP 202) |
| `list` | `GET /v1/runs` | Array of `{ runId, status, startedAt, closedAt? }` for `WorkflowType = 'selfBenchRunWorkflow'` |

`cancel` requests Temporal cancellation. The wait helper treats `cancelled` as a failed terminal. Failed generation must be resubmitted with a new run ID.

## Build commit

`up` and `run` resolve a 40-hex SelfBench commit in this order:

1. `SELFBENCH_BUILD_COMMIT` if set
2. Baked `buildCommit` from package metadata, if it is 40 hex and not all zeros
3. `git -C <package root> rev-parse HEAD`

If none succeed, the CLI throws:

```text
SelfBench was built without commit metadata; set SELFBENCH_BUILD_COMMIT to a full commit SHA
```

The API may override the submitted SHA with its own configured `buildCommit`.

## Command-level errors

Failures throw `Error`. There is no top-level catch, so Node prints the message and stack and exits nonzero.

| Message / pattern | When |
| --- | --- |
| `unknown command: <name>` | First argv is not a known command or help flag |
| `--backend must be "docker" or "modal"` | `up --backend` is anything else |
| `--modal-config requires --backend modal` | `--modal-config` with Docker backend |
| `--repo is required` | `run` without `--repo` / `-r` |
| `--easy-count must be a nonnegative integer` (and medium/hard) | Non-integer or negative tier count |
| `the total candidate count must be between 1 and 100` | Sum is 0 or greater than 100 |
| `unsupported origin URL: …` | `origin` is neither `git@github.com:…` nor `https://…` |
| `no sanitized local-session or GitHub pull-request provenance was found` | Empty merged corpus |
| `SelfBench was built without commit metadata; …` | No usable SHA for `up`/`run` |
| `run ID is required` / `output path is required` | Missing positional args on status, cancel, or download |
| `SelfBench API returned <status>` | Non-OK JSON body without `error` |
| `unauthorized` | API token configured and header missing or wrong |
| `not found` | Unknown route or run ID that does not match the API path regex |
| `run export is not ready` | Download before `export` exists (HTTP 409) |
| `SelfBench API returned an export without integrity metadata` | Missing `x-content-sha256` or body |
| `downloaded export failed its SHA-256 integrity check` | Stream hash mismatch; output file removed |
| `SelfBench run blocked: …` / `failed` / `cancelled` | Wait hit a non-complete terminal phase |
| `SelfBench status response is missing its phase` | Poll JSON has no string `phase` |
| `Unknown option '…'` | Strict `parseArgs` on `up` or `run` |
| `git exited <n>: …` / `gh exited <n>: …` / `docker exited <n>: …` | Subprocess nonzero (stderr or stdout attached) |

Filesystem `EEXIST` from `open(..., "wx")` means the `--output` / download path already exists.

## Companion binaries

These are separate `bin` entries, not `self-bench` subcommands:

| Binary | Role |
| --- | --- |
| `self-bench-eval` | Harbor Codex matrix over an export or expanded tasks directory |
| `self-bench-validate` | Harbor nop and oracle gates |
| `self-bench-reaudit` | Coupling re-audit |
| `self-bench-repair` | Coupled-test repair |
| `self-bench-agent-smoke` | Harbor adapter install/instantiate smoke |

## Related pages

<CardGroup>
  <Card title="Start the local stack" href="/start-local-stack">
    Compose services, `--backend`, `--modal-config`, named volumes, and worker-stop behavior.
  </Card>
  <Card title="Build a benchmark" href="/build-a-benchmark">
    `self-bench run` workflow: GitHub origin, tier budgets, HEAD pin, and `--output`.
  </Card>
  <Card title="Manage runs" href="/manage-runs">
    list, status, cancel, and download against `SELFBENCH_API_URL`.
  </Card>
  <Card title="Companion commands" href="/companion-commands">
    Flags and outputs for eval, validate, reaudit, repair, and agent-smoke.
  </Card>
  <Card title="HTTP API" href="/http-api">
    Routes, Bearer auth, status codes, and export headers the CLI consumes.
  </Card>
  <Card title="Configuration reference" href="/configuration-reference">
    `SELFBENCH_*` keys, loopback token rule, and Temporal fields.
  </Card>
  <Card title="Troubleshooting" href="/troubleshooting">
    Empty provenance, blocked pool, 409 export-not-ready, and SHA-256 mismatch.
  </Card>
</CardGroup>