# Quickstart

> Shortest path: start the Modal-backed local stack, submit a tiered run with --output, extract accepted Harbor tasks, and verify one Codex trial.

- Repository: mupt-ai/self-bench
- GitHub: https://github.com/mupt-ai/self-bench
- Human docs: https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc
- Complete Markdown: https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc/llms-full.txt

## Source Files

- `README.md`
- `src/cli.ts`
- `compose.yaml`
- `docs/evaluations.md`
- `src/evaluate.ts`
- `tests/cli.test.ts`

---

---
title: "Quickstart"
description: "Shortest path: start the Modal-backed local stack, submit a tiered run with --output, extract accepted Harbor tasks, and verify one Codex trial."
---

`self-bench up --backend modal` starts the Compose project `selfbench` on loopback (Postgres, Temporal, API, worker) and sets both `SELFBENCH_EXECUTION_BACKEND` and `SELFBENCH_HARBOR_ENVIRONMENT` to `modal`. `self-bench run --output` uploads sanitized provenance, starts Temporal workflow `selfBenchRunWorkflow`, polls until phase `complete`, and writes a create-only SHA-256-verified export. Extract `tasks/*.tar.gz` and run one Harbor Codex trial with `--env modal`.

<Note>
Install, Modal login, `gh auth login`, and the first `self-bench --help` check live on [Installation](/installation). This page assumes `self-bench` is on `PATH`.
</Note>

## Preconditions

| Requirement | Role in this path |
| --- | --- |
| Docker with Compose | `self-bench up` / `down` run `docker compose --file <package>/compose.yaml` |
| Modal CLI + account | `modal token new` writes `~/.modal.toml` (override with `--modal-config`) |
| `gh` with repo read access | Local `gh pr list` provenance; worker clones with `GH_TOKEN` |
| `OPENAI_API_KEY` | Worker authoring/review/repair; Harbor Codex trial on the host |
| Git checkout with GitHub `origin` | CLI pins `HEAD`; uncommitted files are ignored |
| Completed work in history | Local Pi / Claude Code / Codex sessions and/or merged PRs |

Export these **before** `up`. Compose interpolates them into the API and worker at start time:

```bash
export OPENAI_API_KEY=...
export GH_TOKEN="$(gh auth token)"
export SELFBENCH_API_TOKEN="$(openssl rand -hex 24)"
```

<Warning>
The API image defaults `SELFBENCH_API_TOKEN` to `selfbench-local` when the host variable is unset. The CLI sends `Authorization: Bearer` only when `SELFBENCH_API_TOKEN` is set in the **client** environment. Set the same token on the host before `up` and keep it exported for `run`, `status`, `list`, `cancel`, and `download`.
</Warning>

<ParamField body="SELFBENCH_API_TOKEN" type="string" required>
Shared Bearer secret. Compose default if omitted: `selfbench-local`. Required on the API when it is not bound to loopback.
</ParamField>

<ParamField body="SELFBENCH_API_URL" type="string">
CLI base URL. Default `http://127.0.0.1:8080`. Set after `up` so later commands hit this stack.
</ParamField>

<ParamField body="OPENAI_API_KEY" type="string" required>
Preferred model credential. Takes precedence over `SELFBENCH_PI_AUTH_JSON` / `SELFBENCH_CODEX_AUTH_JSON` subscription fallbacks.
</ParamField>

<ParamField body="GH_TOKEN" type="string" required>
Worker GitHub credential (`https://x-access-token:…@github.com`). The CLI also shells out to `gh` for merged-PR provenance.
</ParamField>

## Start the Modal-backed stack

<Steps>
<Step title="Authenticate Modal">
```bash
modal token new
```
Default profile path is `~/.modal.toml`. Empty `MODAL_TOKEN_ID` / `MODAL_TOKEN_SECRET` values are stripped at worker startup so they cannot override a valid mounted profile.
</Step>

<Step title="Bring the stack up">
```bash
self-bench up --backend modal
export SELFBENCH_API_URL=http://127.0.0.1:8080
```

`--backend` must be `docker` or `modal`. Omitting it selects **docker** (builds `selfbench-sandbox:local` from `Dockerfile.sandbox` first). Modal does not build that image.

`--modal-config PATH` is allowed only with `--backend modal` and becomes `SELFBENCH_MODAL_CONFIG_PATH` (Compose bind-mounts it at `/root/.modal.toml:ro`).
</Step>

<Step title="Confirm the API">
Expected stdout from `up`:

```text
SelfBench is running with the modal backend at http://127.0.0.1:8080
```

```bash
curl -sS http://127.0.0.1:8080/healthz
```

```json
{ "ok": true }
```

`GET /healthz` is unauthenticated. Every `/v1/*` route requires the Bearer token when the API token is configured.
</Step>
</Steps>

Compose services and host binds:

| Service | Image / command | Host bind |
| --- | --- | --- |
| `postgres` | `postgres:17.6-bookworm` (`POSTGRES_DB=temporal`) | none |
| `temporal` | `temporalio/auto-setup:1.29.6` | `127.0.0.1:7233` |
| `api` | `selfbench:local` → `node dist/api-main.js` | `127.0.0.1:8080` |
| `worker` | same image → `node dist/temporal/worker-main.js` | none |

Named volumes `selfbench_temporal-postgres` and `selfbench_artifacts` keep Temporal history and generated artifacts across `self-bench down`. The API never receives the Docker socket or model credentials. Modal worker activity concurrency defaults to **20**.

```mermaid
flowchart LR
  subgraph operator ["Operator host"]
    CLI["self-bench CLI"]
    Checkout["Git checkout + session files"]
    HarborCLI["harbor / self-bench-eval"]
  end
  subgraph compose ["Compose project selfbench"]
    API["api :8080"]
    Worker["worker"]
    Temporal["temporal :7233"]
    PG[("postgres")]
    Art[("artifacts volume")]
  end
  subgraph modalCloud ["Modal"]
    Sandboxes["author / validate / review sandboxes"]
    EvalEnv["Harbor --env modal"]
  end
  Checkout --> CLI
  CLI -->|"POST /v1/provenance, POST /v1/runs"| API
  API --> Temporal
  Worker --> Temporal
  API --> Art
  Worker --> Art
  Worker --> Sandboxes
  HarborCLI --> EvalEnv
```

## Submit a tiered run with `--output`

`self-bench run` does **only** local git metadata and provenance upload. Discovery (8 shards), authoring, static audit, Harbor nop/oracle, determinism, review, one validation-repair, one coupling-repair, and export run in the worker.

<Steps>
<Step title="Choose a checkout and budgets">
The checkout must have `origin` as `https://github.com/…` or `git@github.com:…` (normalized to HTTPS). Other remotes fail with `unsupported origin URL`.

Tier flags are **candidate authoring budgets**, not accepted-task targets. Rejected candidates are not replaced. Combined total must be **1–100**. Each omitted count defaults to `0`.
</Step>

<Step title="Submit and wait for the archive">
```bash
self-bench run \
  --repo /absolute/path/to/your/repository \
  --easy-count 10 \
  --medium-count 10 \
  --hard-count 10 \
  --output ./self-bench-evals.tar.gz
```
</Step>
</Steps>

<ParamField body="--repo" type="path" required>
Git working tree. CLI runs `git remote get-url origin` and `git rev-parse HEAD`.
</ParamField>

<ParamField body="--easy-count / --medium-count / --hard-count" type="integer">
Nonnegative integers. Sum in `[1, 100]`. Eligibility later requires about 20 / 50 / 100 implementation lines across 1 / 2 / 3 paths.
</ParamField>

<ParamField body="--model" type="string">
Authoring model. Default `gpt-5.6-sol`. API stores `authoring.provider = openai-codex` and `reasoningEffort = high`.
</ParamField>

<ParamField body="--run-id" type="string">
Workflow id. Pattern `^[a-z0-9][a-z0-9-]{2,62}$` (3–63 chars). Default `sb-<14 datetime digits>-<8 uuid chars>`. Reusing an id against a live Temporal workflow fails at start.
</ParamField>

<ParamField body="--output" type="path">
Implies `--wait`. Polls `GET /v1/runs/:runId` every 2s, logs phase changes to **stderr**, then `GET /v1/runs/:runId/export`. Opens the destination with `wx` (must not already exist). Verifies `x-content-sha256`; mismatch deletes the file.
</ParamField>

<ParamField body="--wait" type="boolean">
Wait without downloading. `--output` already waits. Terminal phases `blocked`, `failed`, and `cancelled` throw `SelfBench run <phase>[: <error>]`.
</ParamField>

Local provenance is the union of:

- Sanitized user messages from `~/.pi/agent/sessions`, `~/.claude/projects`, `~/.codex/sessions`, and `~/.codex/archived_sessions` whose contents or paths mention the checkout or its worktrees
- Up to **500** merged, non-draft, non-bot GitHub PRs with a title, at least **20** changed lines, and at least **1** changed file

Zero messages abort **before** any API write:

```text
no sanitized local-session or GitHub pull-request provenance was found
```

<RequestExample>
```bash
self-bench run --repo /abs/path/to/repo --easy-count 1 --output ./self-bench-evals.tar.gz
```
</RequestExample>

<ResponseExample>
```json
{
  "runId": "sb-20260814153022-a1b2c3d4",
  "provenanceMessages": 42,
  "localProvenanceMessages": 17,
  "githubPullRequestMessages": 25
}
```

```json
{"runId":"sb-20260814153022-a1b2c3d4","phase":"discovering","accepted":0,"rejected":0}
{"runId":"sb-20260814153022-a1b2c3d4","phase":"authoring","accepted":0,"rejected":0}
{"runId":"sb-20260814153022-a1b2c3d4","phase":"exporting","accepted":1,"rejected":2}
{"runId":"sb-20260814153022-a1b2c3d4","phase":"complete","accepted":1,"rejected":2}
```

```json
{
  "runId": "sb-20260814153022-a1b2c3d4",
  "output": "/abs/path/to/self-bench-evals.tar.gz"
}
```
</ResponseExample>

Phases: `queued` → `discovering` → `authoring` → `exporting` → `complete`. Closing the waiter does **not** cancel the workflow (`workflowExecutionTimeout` is 14 days). Stopping the worker or Compose stack pauses activities until the worker is back; Temporal state stays in `selfbench_temporal-postgres`.

A run can reach `complete` with `accepted: 0` if every budgeted candidate is rejected. Discovery that cannot fill a requested tier ends `blocked` with `candidate pool exhausted; missing easy=…, medium=…, hard=…`.

## Extract accepted Harbor tasks

The download is `application/gzip` with `content-disposition: attachment; filename="selfbench-<runId>.tar.gz"`. Exports include base snapshots, held-out tests, and gold patches. They are unencrypted; treat them as private.

:::files
self-bench-evals.tar.gz
├── manifest.json
└── tasks/
    └── <taskId>.tar.gz
        └── harbor-task/
            ├── task.toml          # Harbor schema 1.4
            ├── instruction.md
            ├── environment/       # agent image + repo.tar.gz
            ├── tests/             # held-out patch + verifier
            └── solution/          # gold.patch + solve.sh (oracle only)
:::

`manifest.json` fields: `schemaVersion` (`1`), `runId`, `candidateCounts`, `repository` (`url`, `commit`), `version` (`selfbenchCommit`, `executionBackend`, `sandboxImage`, `schema`), `acceptedCount`, and `tasks[]` (`taskId`, per-archive `sha256`).

```bash
mkdir -p ./self-bench-export ./self-bench-tasks
tar -xzf ./self-bench-evals.tar.gz -C ./self-bench-export
jq '{runId, acceptedCount, tasks: [.tasks[].taskId]}' ./self-bench-export/manifest.json

TASK_ID="$(jq -r '.tasks[0].taskId' ./self-bench-export/manifest.json)"
mkdir -p "./self-bench-tasks/$TASK_ID"
tar -xzf "./self-bench-export/tasks/${TASK_ID}.tar.gz" -C "./self-bench-tasks/$TASK_ID"
test -f "./self-bench-tasks/${TASK_ID}/harbor-task/task.toml"
```

<Info>
`self-bench-eval --export FILE` materializes each `harbor-task/` under `<jobs>/tasks/<taskId>/` (so `task.toml` is at the task-directory root). Manual extract keeps the `harbor-task/` prefix. Point Harbor at the directory that contains `task.toml`.
</Info>

The evaluated agent receives the base snapshot and `instruction.md` only. Harbor mounts `solution/` solely for the oracle agent.

## Verify one Codex trial

Install the same Harbor pin the worker image uses (`HARBOR_VERSION=0.20.1.dev202608040148`):

```bash
uv tool install --python 3.12 'harbor[modal]==0.20.1.dev202608040148'
export OPENAI_API_KEY=...
```

<Tabs>
<Tab title="harbor (one task)">

```bash
harbor run \
  --path "./self-bench-tasks/${TASK_ID}/harbor-task" \
  --agent codex \
  --model gpt-5.6-sol \
  --ak version=0.146.1 \
  --ak reasoning_effort=high \
  --env modal \
  --jobs-dir ./harbor-jobs \
  --yes
```

`--env modal` must match the stack you started. A docker-backed stack would use `--env docker` instead.

</Tab>
<Tab title="self-bench-eval (one model)">

`self-bench-eval` wraps the same Codex adapter (`version=0.146.1`, `reasoning_effort=high`), writes `<jobs>/summary.json`, and reuses a completed Harbor job named `<taskId>-<model>`.

Omit `--model` and it runs the full matrix: `gpt-5.6-sol`, `gpt-5.6-terra`, `gpt-5.6-luna`. Pass `--model` once for a single trial per task. Default `--environment` is `modal`; default `--concurrency` is `3`.

```bash
self-bench-eval \
  --tasks "./self-bench-tasks/${TASK_ID}" \
  --jobs ./harbor-jobs \
  --environment modal \
  --model gpt-5.6-sol
```

```json
{
  "trials": 1,
  "passed": 1,
  "failed": 0
}
```

A trial `passed` when Harbor produced rewards, there is no exception, and either `rewards.reward >= 1` or every numeric reward is `>= 1`. Empty rewards fail.

Without `OPENAI_API_KEY`, pass `--auth /path/to/codex-auth.json` (`auth_mode: chatgpt` plus `tokens`).

</Tab>
</Tabs>

<Check>
A successful trial writes a finished Harbor job under `./harbor-jobs` with `result.json` and no `exception_info`. Infrastructure strings (Docker daemon, Modal timeout, image-build failure, Harbor `AuthError`) are not task verdicts.
</Check>

## If the waiter is gone

Closing `self-bench run --output` leaves the workflow running.

```bash
self-bench list
self-bench status RUN_ID
self-bench download RUN_ID ./self-bench-evals.tar.gz
self-bench cancel RUN_ID
```

| Command | HTTP |
| --- | --- |
| `list` | `GET /v1/runs` |
| `status RUN_ID` | `GET /v1/runs/:runId` |
| `cancel RUN_ID` | `POST /v1/runs/:runId/cancel` → `{ cancellationRequested: true }` |
| `download RUN_ID PATH` | `GET /v1/runs/:runId/export` |

`download` before `complete` returns **409** `{ "error": "run export is not ready" }`.

```bash
self-bench down
```

`down` is `docker compose down` and does **not** delete `selfbench_temporal-postgres` or `selfbench_artifacts`.

The API also serves the review UI at `http://127.0.0.1:8080/` for a completed run.

## Failures on this path

| Symptom | Cause |
| --- | --- |
| `unknown command` / missing `self-bench` | Package not on `PATH`; see [Installation](/installation) |
| `--backend must be "docker" or "modal"` | Typo or omitted value treated as the docker default |
| `--modal-config requires --backend modal` | Config path passed with docker backend |
| `401 unauthorized` | CLI token missing or not the token Compose baked in at `up` |
| Empty-corpus error | No session files mentioning the checkout and no qualifying merged PRs |
| `unsupported origin URL` | `origin` is not GitHub HTTPS or `git@github.com:` |
| `the total candidate count must be between 1 and 100` | All counts `0`, or sum `> 100` |
| `SelfBench run blocked: candidate pool exhausted…` | Discovery could not fill a requested tier |
| `SelfBench run failed` / `cancelled` | Workflow error or `cancel`; `--output` does not download |
| `run export is not ready` | `download` while phase is not `complete` |
| `downloaded export failed its SHA-256 integrity check` | Streamed bytes ≠ `x-content-sha256`; dest file removed |
| `EEXIST` on `--output` | Destination already exists (`wx`) |
| Harbor `AuthError` / Modal timeout | Host `OPENAI_API_KEY` or Modal profile missing for the **eval** machine |
| `found no Harbor tasks` | Export `acceptedCount` is `0`, or `--tasks` dir has no `task.toml` |

## Next

<CardGroup>
<Card title="Start the local stack" href="/start-local-stack">
Compose services, `--backend docker|modal`, `--modal-config`, named volumes, and worker-stop behavior.
</Card>
<Card title="Build a benchmark" href="/build-a-benchmark">
`--repo`, 1–100 tier budgets, `--model`, `--run-id`, HEAD pin, provenance upload, `--output` wait.
</Card>
<Card title="Evaluate with Harbor" href="/evaluate-with-harbor">
Full extract, one task vs Codex matrix, `self-bench-eval` resume, adapter smoke.
</Card>
<Card title="Manage runs" href="/manage-runs">
`list`, `status`, `cancel`, `download`, phase polling, create-only writes.
</Card>
<Card title="Troubleshooting" href="/troubleshooting">
No-provenance, blocked pool, 409, SHA-256 mismatch, Harbor infrastructure failures.
</Card>
</CardGroup>