# Installation
> Prerequisites, global install, Modal and GitHub auth, required environment variables, and the first successful CLI help check.
- Repository: mupt-ai/self-bench
- GitHub: https://github.com/mupt-ai/self-bench
- Human docs: https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc
- Complete Markdown: https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc/llms-full.txt
## Source Files
- `README.md`
- `package.json`
- `compose.yaml`
- `Dockerfile`
- `Dockerfile.sandbox`
- `scripts/verify-package.ts`
---
---
title: "Installation"
description: "Prerequisites, global install, Modal and GitHub auth, required environment variables, and the first successful CLI help check."
---
The published `self-bench` package (currently `0.3.3`) is a Node `>=22` CLI. A global install puts six `#!/usr/bin/env node` binaries on `PATH` and ships `compose.yaml`, `Dockerfile`, `Dockerfile.sandbox`, and `src/skills/selfbench/SKILL.md` next to `dist/cli.js`, so later `self-bench up` can build the Compose stack from the package root rather than a git checkout.
`self-bench --help` (also `-h`, `help`, or no arguments) is the install success check. It does not contact the API. `self-bench up`, `self-bench run`, Harbor on the host, and Temporal Cloud are later steps.
## Prerequisites
| Requirement | Used for | Notes |
| --- | --- | --- |
| [Bun](https://bun.sh/) `1.3.14` or newer | Documented global install, CI, and `npm` publish | CI and publish pin Bun `1.3.14` |
| Node.js `>=22` | Runtime for every packaged binary | `package.json` `engines.node`; worker/API image is `node:22-bookworm` |
| Docker with Compose | `self-bench up` | Worker image embeds Docker CLI `29.4.0`; worker also runs `docker info` and `docker compose version` when the execution or Harbor backend is `docker` |
| [Modal](https://modal.com/) CLI, account, and token | Recommended Modal sandboxes | `pip install modal`, then `modal token new` |
| [`gh`](https://cli.github.com/) with repo read access | Host provenance and worker GitHub reads | Write access is not required |
| `OPENAI_API_KEY` | Worker discovery, authoring, review, repair | README requires access to `gpt-5.6-sol`, `gpt-5.6-terra`, and `gpt-5.6-luna` |
| A Git checkout whose `origin` is GitHub | `self-bench run` | HTTPS or `git@github.com:` SSH; not required to print CLI help |
Host Harbor and `uv` are not install prerequisites. The API/worker image already installs `harbor[modal]==0.20.1.dev202608040148` via `uv`. Install Harbor on the host only when you evaluate an export later.
## Install the CLI
```bash
bun add --global self-bench
```
The package is published public to `https://registry.npmjs.org/` as `self-bench`. The binaries are Node scripts; Node `>=22` is required.
```bash
npm install --global self-bench
```
Tagged releases publish with `npm publish --access public` after `bun run validate`. The tag name must be `v` plus `package.json` `version`.
A successful global install exposes:
| Binary | Entry | Role |
| --- | --- | --- |
| `self-bench` | `dist/cli.js` | Stack, submit, list, status, cancel, download |
| `self-bench-eval` | `dist/eval-main.js` | Harbor Codex matrix |
| `self-bench-validate` | `dist/validate-main.js` | Offline Harbor validation |
| `self-bench-reaudit` | `dist/reaudit-main.js` | Offline review re-audit |
| `self-bench-repair` | `dist/repair-main.js` | Offline repair |
| `self-bench-agent-smoke` | `dist/agent-smoke-main.js` | Harbor adapter smoke |
`self-bench up` resolves the package root as the parent of `dist/cli.js` and requires these published files to exist there:
:::files
self-bench/ # node_modules/self-bench or Bun global package
├── dist/
│ └── cli.js # self-bench
├── compose.yaml
├── Dockerfile
├── Dockerfile.sandbox
└── src/skills/selfbench/SKILL.md
:::
Package verification packs the tarball, installs it, runs `self-bench --help`, and fails unless stdout contains `self-bench up` and those four assets are present.
## Authenticate Modal and GitHub
```bash
pip install modal
modal token new
```
`self-bench up --backend modal` mounts `~/.modal.toml` into the worker at `/root/.modal.toml`. Override the host path with `--modal-config /absolute/path/to/.modal.toml`. `--modal-config` without `--backend modal` is an error.
A secret manager may set `MODAL_TOKEN_ID` and `MODAL_TOKEN_SECRET` instead of a profile file. Empty values for those two keys are deleted at worker startup so they cannot override a valid mounted profile.
```bash
gh auth login
export GH_TOKEN="$(gh auth token)"
```
The host CLI uses `gh pr list --state merged --limit 500` when you later run `self-bench run`. The Compose worker does not inherit the host `gh` login; it needs `GH_TOKEN` in the environment. Worker sandboxes that receive a token write `https://x-access-token:…@github.com` git credentials. If `GH_TOKEN` is unset, the worker process may fall back to `gh auth token` on its own `PATH`, which fails inside the container.
`OPENAI_API_KEY` is the supported model credential. Existing deployments may set both `SELFBENCH_PI_AUTH_JSON` and `SELFBENCH_CODEX_AUTH_JSON` instead. When `OPENAI_API_KEY` is set (after trim), it wins for both Pi and Codex sandboxes.
## Required environment variables
Export these in the same shell you will use for `self-bench up` and later CLI calls. Compose interpolates host values when the stack starts; changing them afterward does not update a running container.
```bash
export OPENAI_API_KEY=...
export GH_TOKEN="$(gh auth token)"
export SELFBENCH_API_TOKEN="$(openssl rand -hex 24)"
```
After the stack is up, point the CLI at the published API port (optional on the default):
```bash
export SELFBENCH_API_URL=http://127.0.0.1:8080
```
Worker and evaluation sandboxes. Trimmed; empty is treated as unset.
Passed into the Compose `worker` service. Host `self-bench run` still needs `gh` on `PATH`.
Protects `/v1/*` with `Authorization: Bearer`. Compose binds the API at `SELFBENCH_API_HOST=0.0.0.0`, so a non-empty token is required (loopback hosts `127.0.0.1`, `::1`, and `localhost` are the only hosts that may start without one).
CLI client base URL. Default `http://127.0.0.1:8080`. The CLI sends a Bearer header only when `SELFBENCH_API_TOKEN` is set in its own environment.
| Variable | Default if unset | Who reads it |
| --- | --- | --- |
| `SELFBENCH_API_TOKEN` | Compose substitutes `selfbench-local` | API process and CLI |
| `SELFBENCH_API_URL` | `http://127.0.0.1:8080` | CLI only |
| `SELFBENCH_API_HOST` / `SELFBENCH_API_PORT` | `127.0.0.1` / `8080` in process config; Compose overrides host to `0.0.0.0` | API |
| `SELFBENCH_EXECUTION_BACKEND` | `docker` | Worker; `self-bench up --backend` sets this |
| `SELFBENCH_HARBOR_ENVIRONMENT` | same as execution backend | Worker; `up` sets this to the same backend |
| `SELFBENCH_MODAL_CONFIG_PATH` | `/dev/null` in Compose | Worker volume mount; `up --backend modal` sets `~/.modal.toml` |
| `MODAL_TOKEN_ID` / `MODAL_TOKEN_SECRET` | empty | Worker; empty strings are stripped |
| `SELFBENCH_PI_AUTH_JSON` / `SELFBENCH_CODEX_AUTH_JSON` | unset | Worker fallback if no API key |
| `SELFBENCH_BUILD_COMMIT` | baked at build, else `git rev-parse HEAD` | `up` / `run` |
If you skip `SELFBENCH_API_TOKEN`, Compose still starts the API with `selfbench-local`, but the CLI sends no `Authorization` header. `/healthz` stays open; `/v1/*` returns `401` `{"error":"unauthorized"}`. Export the same token before `up` and in every later CLI shell. Review UI `GET /` and `GET /assets/*` are also unauthenticated.
The API never receives `OPENAI_API_KEY`, `GH_TOKEN`, Modal tokens, or the Docker socket. Those stay on the worker.
```mermaid
flowchart LR
subgraph host [Host after install]
CLI["self-bench"]
Profile["~/.modal.toml"]
Gh["gh + GH_TOKEN"]
Key["OPENAI_API_KEY"]
Tok["SELFBENCH_API_TOKEN"]
end
subgraph later [Started later by self-bench up]
API["api :8080"]
Worker["worker"]
Temporal["temporal :7233"]
PG["postgres"]
end
CLI -->|"Bearer + SELFBENCH_API_URL"| API
Tok --> API
Key --> Worker
Gh --> Worker
Profile -->|"mount or MODAL_TOKEN_*"| Worker
API --- Temporal
Worker --- Temporal
Temporal --- PG
```
## First CLI help check
```bash
self-bench --help
```
`self-bench`, `self-bench -h`, and `self-bench help` print the same text. No API, Docker, Modal, or GitHub credentials are required.
Stdout must include the `self-bench up` usage line. That is the same predicate `scripts/verify-package.ts` uses after a pack-and-install.
```text
SelfBench creates durable tiered Harbor evaluations.
Usage:
self-bench up [--backend docker|modal] [--modal-config PATH]
self-bench down
self-bench run --repo PATH [--easy-count N] [--medium-count N] [--hard-count N]
[--model MODEL] [--run-id ID] [--wait] [--output OUTPUT.tar.gz]
self-bench status RUN_ID
self-bench cancel RUN_ID
self-bench download RUN_ID OUTPUT.tar.gz
self-bench list
The up command starts the local stack and configures both sandbox execution and Harbor validation for
one backend. Modal uses ~/.modal.toml unless --modal-config overrides it.
The tier counts are candidate authoring budgets, not accepted-task targets. Rejected candidates are not
replaced, and the export contains only accepted tasks. The run command performs only repository metadata
and sanitized provenance upload locally; discovery, authoring, validation, review, and audit run remotely.
--output implies --wait, blocks until completion, and downloads the SHA-256-verified export.
```
An unknown first argument throws `unknown command: …`. That is a failed install or a typo, not a credentials problem.
Optional companion check (not required for install):
```bash
self-bench-eval --help
```
## Source checkout (maintainers)
From a clone, do not use the published tarball:
```bash
git clone https://github.com/mupt-ai/self-bench.git
cd self-bench
bun install --frozen-lockfile
bun run cli -- --help
```
`bun run validate` runs Biome, typecheck, tests, production build, and `bun run verify:package`.
## Install-time failures
| Symptom | Cause | Fix |
| --- | --- | --- |
| `self-bench: command not found` | Global bin directory not on `PATH` | Re-open the shell or use the package manager's global bin path |
| Help omits `self-bench up` | Incomplete or stale package | Reinstall `self-bench@0.3.3` or newer; do not copy `dist/cli.js` alone |
| `SELFBENCH_API_TOKEN is required when the API listens beyond loopback` | API host is not `127.0.0.1` / `::1` / `localhost` and the token is missing | Set a non-empty `SELFBENCH_API_TOKEN` (Compose always binds `0.0.0.0`) |
| `401 unauthorized` after a later `list`/`run` | CLI token missing or different from the value interpolated at `up` | Export the same token used when Compose started |
| `--modal-config requires --backend modal` | Flag used with Docker backend | Pass `--backend modal` or drop `--modal-config` |
| Modal SDK ignores `~/.modal.toml` | Empty `MODAL_TOKEN_ID` / `MODAL_TOKEN_SECRET` in the worker env | Leave them unset, or rely on worker startup stripping blanks |
| `SelfBench was built without commit metadata` | `dist/build-metadata.js` is all-zero and `git rev-parse` cannot run | Set `SELFBENCH_BUILD_COMMIT` to a 40-character SHA |
Do not treat a successful help print as a running stack. `self-bench up --backend modal` is the next operator command.
## Next
Start the Modal-backed local stack, submit a tiered run with `--output`, and verify one Codex trial.
`self-bench up` / `down`, Compose services, `--backend`, `--modal-config`, and named volumes.
`OPENAI_API_KEY` precedence, Pi/Codex JSON fallbacks, `GH_TOKEN`, API Bearer, and Modal profile versus token secrets.
Every `SELFBENCH_*` key, defaults, loopback token rule, and Temporal fields.
Commands, flags, defaults, run-id rules, and command-level errors.
Clone, `bun install`, `bun run validate`, source CLI, and npm tag publish.