# self-bench Documentation > Operator and integrator reference for the self-bench CLI, HTTP API, Temporal worker, and Harbor export pipeline that turns completed repository work into private coding-agent evaluations. ## Context Links - [Agent index](https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc/llms.txt) - [Human interactive docs](https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc) - [GitHub repository](https://github.com/mupt-ai/self-bench) ## Repository Metadata - Repository: mupt-ai/self-bench - Generated: 2026-08-15T02:16:57.473Z - Updated: 2026-08-15T02:36:00.741Z - Runtime: Grok CLI - Format: Documentation - Pages: 25 ## Page Index - 01. [Overview](https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc/pages/01-overview.md) - What self-bench generates, who should run it, the CLI/API/worker surfaces, and the first docs routes after install. - 02. [Installation](https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc/pages/02-installation.md) - Prerequisites, global install, Modal and GitHub auth, required environment variables, and the first successful CLI help check. - 03. [Quickstart](https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc/pages/03-quickstart.md) - Shortest path: start the Modal-backed local stack, submit a tiered run with --output, extract accepted Harbor tasks, and verify one Codex trial. - 04. [Provenance sources](https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc/pages/04-provenance-sources.md) - Where self-bench collects human requests: Pi, Claude Code, and Codex session roots, merged GitHub pull requests, sanitization rules, and the empty-corpus failure. - 05. [Difficulty gates](https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc/pages/05-difficulty-gates.md) - easy, medium, and hard eligibility: implementation file and line minima, fail-to-pass and pass-to-pass counts, and static audit blockers. - 06. [Run lifecycle](https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc/pages/06-run-lifecycle.md) - Temporal phases, per-candidate statuses, eight-shard discovery, candidate budgets versus accepted tasks, and blocked, failed, or cancelled terminals. - 07. [Validation and repair](https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc/pages/07-validation-and-repair.md) - Static audit, Harbor nop and oracle gates, determinism rerun, one validation-repair attempt, one coupling-repair retry, then reject. - 08. [Agent-visible boundary](https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc/pages/08-agent-visible-boundary.md) - What a Harbor trial mounts, what stays hidden, gold-only contract artifacts, coupling verdicts, and the held-out-test-only repair constraint. - 09. [Start the local stack](https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc/pages/09-start-the-local-stack.md) - self-bench up and down: Compose services, --backend docker or modal, --modal-config, named volumes, and what happens when the worker stops. - 10. [Build a benchmark](https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc/pages/10-build-a-benchmark.md) - self-bench run: --repo GitHub origin, 1-100 tier budgets, --model, --run-id, HEAD pin, provenance upload, and --output wait plus SHA-256 download. - 11. [Manage runs](https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc/pages/11-manage-runs.md) - list, status, cancel, and download against SELFBENCH_API_URL, including phase polling, create-only writes, and integrity headers. - 12. [Evaluate with Harbor](https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc/pages/12-evaluate-with-harbor.md) - Extract an export, run one task or the Codex matrix via self-bench-eval, resume completed jobs, and smoke Harbor agent adapters. - 13. [Review exports](https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc/pages/13-review-exports.md) - Open the API-hosted review UI or a local archive, inspect accepted tasks, drop tasks, and download a cleaned tar.gz. - 14. [Deploy Temporal Cloud](https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc/pages/14-deploy-temporal-cloud.md) - Split API and worker processes, GCS artifacts, Temporal TLS and API key, Modal tokens, and matching image digest plus task queue. - 15. [CLI reference](https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc/pages/15-cli-reference.md) - self-bench commands, flags, defaults, run-id rules, wait and download semantics, and command-level errors. - 16. [Companion commands](https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc/pages/16-companion-commands.md) - Flags, defaults, and outputs for self-bench-eval, self-bench-validate, self-bench-reaudit, self-bench-repair, and self-bench-agent-smoke. - 17. [HTTP API](https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc/pages/17-http-api.md) - Routes, Bearer auth, status codes, run submission schema, provenance upload limits, export headers, and review-asset serving. - 18. [Configuration reference](https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc/pages/18-configuration-reference.md) - SELFBENCH_* environment keys, defaults, validation, loopback token rule, artifact backends, and Temporal connection fields. - 19. [Task definition](https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc/pages/19-task-definition.md) - definition.json fields, {tests} placeholder rules, toolchains, timeouts, resources, submit_task output files, and schemaVersion 1 constraints. - 20. [Harbor task compiler](https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc/pages/20-harbor-task-compiler.md) - How authored patches become a Harbor task: task.toml schema 1.4, agent and verifier Dockerfiles, toolchain layers, and dependency-setup.patch. - 21. [Export format](https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc/pages/21-export-format.md) - Outer tar.gz layout, manifest.json fields, per-task harbor-task contents, SHA-256 checks, and what the archive excludes. - 22. [Execution backends](https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc/pages/22-execution-backends.md) - Docker versus Modal sandboxes: images, default concurrency, resource limits, inactivity timeouts, and Harbor environment pairing. - 23. [Authentication](https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc/pages/23-authentication.md) - OPENAI_API_KEY precedence, Pi and Codex subscription JSON fallbacks, GH_TOKEN, API Bearer token, and Modal profile versus token secrets. - 24. [Troubleshooting](https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc/pages/24-troubleshooting.md) - No-provenance, blocked candidate pool, 409 export-not-ready, SHA-256 mismatch, Harbor infrastructure failures, and leftover Docker resources. - 25. [Contributing](https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc/pages/25-contributing.md) - Clone, bun install, bun run validate, source CLI, API/worker/review dev servers, CI, package verification, and npm tag publish. ## Source File Index - `.github/workflows/ci.yml` - `.github/workflows/publish-npm.yml` - `biome.json` - `compose.yaml` - `Dockerfile` - `Dockerfile.sandbox` - `docs/evaluations.md` - `docs/operations.md` - `docs/task-construction.md` - `package.json` - `README.md` - `review/src/App.tsx` - `review/src/export-loader.test.ts` - `review/src/export-loader.ts` - `review/src/types.ts` - `scripts/verify-package.ts` - `src/agent-smoke-main.ts` - `src/agent-smoke.ts` - `src/api-main.ts` - `src/api.ts` - `src/artifacts.ts` - `src/audit.ts` - `src/cli.ts` - `src/codex-review.ts` - `src/config.ts` - `src/contracts.ts` - `src/coupling.ts` - `src/docker-executor.ts` - `src/eval-main.ts` - `src/evaluate.ts` - `src/extensions/authoring.ts` - `src/extensions/discovery.ts` - `src/github.ts` - `src/harbor-results.ts` - `src/harbor-task.ts` - `src/modal-auth.ts` - `src/modal-executor.ts` - `src/provenance.ts` - `src/reaudit-main.ts` - `src/repair-main.ts` - `src/repair.ts` - `src/run-wait.ts` - `src/sandbox.ts` - `src/skills/selfbench/SKILL.md` - `src/subscription-auth.ts` - `src/temporal/activities.ts` - `src/temporal/connection.ts` - `src/temporal/worker-main.ts` - `src/temporal/workflow.ts` - `src/validate-main.ts` - `src/validation-repair.ts` - `tests/audit.test.ts` - `tests/cli.test.ts` - `tests/contracts.test.ts` - `tests/coupling.test.ts` - `tests/evaluate.test.ts` - `tests/harbor-results.test.ts` - `tests/harbor-task.test.ts` - `tests/provenance.test.ts` - `tests/run-wait.test.ts` - `tests/subscription-auth.test.ts` - `tests/workflow.test.ts` - `tsconfig.build.json` --- ## 01. Overview > What self-bench generates, who should run it, the CLI/API/worker surfaces, and the first docs routes after install. - Page Markdown: https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc/pages/01-overview.md - Generated: 2026-08-15T02:04:08.496Z ### Source Files - `README.md` - `package.json` - `src/cli.ts` - `src/api.ts` - `src/contracts.ts` - `docs/task-construction.md` --- title: "Overview" description: "What self-bench generates, who should run it, the CLI/API/worker surfaces, and the first docs routes after install." --- `self-bench` is the published npm package that turns completed work in a GitHub-origin Git checkout into a private Harbor evaluation archive. The `self-bench` CLI collects sanitized human requests and starts a Temporal `selfBenchRunWorkflow`; the HTTP API and worker author, audit, validate, and review candidates; a completed run downloads as a SHA-256-verified `.tar.gz` of accepted Harbor tasks. The recommended local path is Compose for Postgres, Temporal, the API, and the worker, with Modal as the sandbox and Harbor environment. `self-bench up --backend docker` keeps those sandboxes on the host instead. ## What a run generates A successful run exports only **accepted** tasks. Easy, medium, and hard `--*-count` values are **candidate authoring budgets** (total 1–100), not guaranteed accepted-task counts. Rejected candidates are not replaced. Each accepted task is a native Harbor task compiled to `task.toml` schema `1.4`: - a history-free base snapshot at the commit before the completed change - a standalone `instruction.md` that preserves a retained human request - hidden held-out tests (`tests/`) and a gold reference patch (`solution/`) - agent and verifier Dockerfiles plus repository-native setup and `{tests}` commands The outer archive is unencrypted and includes source snapshots, hidden tests, and reference solutions. Treat it as private benchmark material. :::files selfbench-.tar.gz ├── manifest.json # schemaVersion 1, repo pin, build identity, task SHA-256s └── tasks/ └── .tar.gz └── harbor-task/ ├── task.toml ├── instruction.md ├── environment/ # agent image + repo.tar.gz ├── tests/ # test.patch, test.sh, verifier Dockerfile └── solution/ # gold.patch, solve.sh (oracle only) ::: `manifest.json` records `runId`, `candidateCounts`, the pinned `repository.url` / `repository.commit`, `version` (`selfbenchCommit`, `executionBackend`, `sandboxImage`, `schema: 1`), `acceptedCount`, and per-task archive digests. The archive excludes Git history, local session/provenance records, and author or reviewer transcripts. Evaluated agents receive the base snapshot, the instruction, and the declared environment. They do not receive held-out tests or the gold patch. Harbor mounts `solution/` only for the explicit oracle agent. ## Who should run it Run self-bench when you control a Git checkout whose `origin` is a GitHub HTTPS or `git@github.com:` URL, the history contains completed work, and you want a **private** Harbor suite from that work—not a public dataset. | Role | Uses | Needs | | --- | --- | --- | | Benchmark operator | `self-bench up` / `run` / `download`, then Harbor or `self-bench-eval` | Docker Compose, `gh` read access, `OPENAI_API_KEY` (or the documented Pi/Codex subscription fallbacks), and usually a Modal token | | API client or reviewer | `SELFBENCH_API_URL` plus Bearer token; browser review UI at `/` | A running API and a completed export | | Cloud operator | Split `node dist/api-main.js` and `node dist/temporal/worker-main.js` | Matching image digest and `SELFBENCH_TASK_QUEUE`, Temporal Cloud TLS + API key, GCS artifacts, Modal tokens | | Maintainer | `bun install`, `bun run validate`, `bun run cli` | Bun 1.3.14+ for the documented global install and repo scripts; published bins require Node 22+ | Do **not** point self-bench at untrusted repositories. Sandboxes check out the selected commit and run repository-native setup with a short-lived model credential. The worker is not a malware-analysis service. Authoring submissions hard-code `authoring.provider` to `openai-codex` and `reasoningEffort` to `high`. The CLI default `--model` is `gpt-5.6-sol`. `self-bench-eval` defaults to the Codex matrix `gpt-5.6-sol`, `gpt-5.6-terra`, and `gpt-5.6-luna` at high reasoning with Harbor agent version `0.146.1`. Bring your own keys, Modal account, GitHub token, and Temporal deployment; the authoring and matrix model IDs are those literals, not a pluggable provider catalog. Sanitization strips common credential forms and injected session harness text. It is not a general secret scanner. Exports remain sensitive after download. ## Surfaces ```mermaid flowchart TB subgraph operator ["Operator machine"] Repo["Git checkout HEAD pin + origin"] Sessions["Pi / Claude Code / Codex session roots"] CLI["self-bench CLI"] ReviewFile["Local export gzip"] end subgraph control ["Control plane"] API["api-main.ts HTTP API"] UI["Review UI GET / and /assets/"] Temporal["Temporal selfBenchRunWorkflow"] end subgraph data ["Artifacts"] Store["createArtifactStore local dir or GCS"] end subgraph workerPlane ["Worker"] Worker["worker-main.ts activities"] Docker["Docker selfbench-sandbox:local"] Modal["Modal app selfbench"] end subgraph after ["After export"] Eval["self-bench-eval / harbor run"] HarborTask["Expanded harbor-task/"] end Repo --> CLI Sessions --> CLI CLI -->|"POST /v1/provenance then POST /v1/runs"| API API --> Temporal Temporal --> Worker Worker --> Store API --> Store Worker --> Docker Worker --> Modal API --> UI ReviewFile --> UI Store -->|"GET /v1/runs/:id/export"| CLI CLI --> HarborTask HarborTask --> Eval ``` The API never receives the Docker socket or model credentials. The worker owns `GH_TOKEN`, `OPENAI_API_KEY` or subscription JSON, Modal profile or `MODAL_TOKEN_*`, and Harbor execution. ### `self-bench` CLI Published bin: `dist/cli.js`. Talks to `SELFBENCH_API_URL` (default `http://127.0.0.1:8080`) and sends `Authorization: Bearer $SELFBENCH_API_TOKEN` when that variable is set. | Command | Behavior | | --- | --- | | `up` | `docker compose up -d --build`; `--backend docker` (default) or `modal`; sets `SELFBENCH_EXECUTION_BACKEND` and `SELFBENCH_HARBOR_ENVIRONMENT` together; Modal mounts `~/.modal.toml` unless `--modal-config` is set; Docker backend builds `selfbench-sandbox:local` first | | `down` | `docker compose down`; named volumes `selfbench_temporal-postgres` and `selfbench_artifacts` remain | | `run` | Resolves GitHub `origin` + `HEAD`, collects provenance, uploads JSONL, starts the workflow | | `list` / `status` / `cancel` | GET `/v1/runs`, GET `/v1/runs/:runId`, POST `/v1/runs/:runId/cancel` | | `download` | GET export; requires `x-content-sha256`; writes with create-only `wx`; deletes the file on hash mismatch | `run` does only repository metadata and sanitized provenance upload locally. Discovery, authoring, validation, review, and audit run in the worker. Closing a waiting CLI does **not** cancel the workflow. `--output` implies `--wait`, polls until `complete`, and downloads the verified archive. Phases `blocked`, `failed`, and `cancelled` fail the wait. ```bash self-bench --help self-bench up --backend modal export SELFBENCH_API_URL=http://127.0.0.1:8080 self-bench run --repo /absolute/path/to/repo --easy-count 10 --medium-count 10 --hard-count 10 --output ./self-bench-evals.tar.gz ``` `--run-id` must match `^[a-z0-9][a-z0-9-]{2,62}$`. The generated default is `sb--<8 hex chars>`. Uncommitted work is ignored; the run pins current `HEAD`. ### HTTP API Process: `node dist/api-main.js` (`bun run dev:api` in a checkout). Listens on `SELFBENCH_API_HOST`:`SELFBENCH_API_PORT` (defaults `127.0.0.1:8080`). Binding beyond loopback without `SELFBENCH_API_TOKEN` fails at startup. Compose binds `0.0.0.0` and defaults the token to `selfbench-local` if unset. | Method | Path | Auth | Result | | --- | --- | --- | --- | | `GET` | `/healthz` | none | `{ ok: true }` | | `GET` | `/`, `/assets/*` | none | Review UI assets | | `POST` | `/v1/provenance?runId=` | Bearer when token set | Store JSONL (max 100 MiB); `201` artifact ref | | `POST` | `/v1/runs` | Bearer when token set | Start workflow; `202 { runId }`; 14-day execution timeout | | `GET` | `/v1/runs` | Bearer when token set | Workflow list | | `GET` | `/v1/runs/:runId` | Bearer when token set | `RunStatus` (phase, counts, per-task status, optional `export`) | | `POST` | `/v1/runs/:runId/cancel` | Bearer when token set | `202 { cancellationRequested: true }` | | `GET` | `/v1/runs/:runId/export` | Bearer when token set | gzip stream; `409` if not ready; `x-content-sha256` | There is no remote delete route. Remove local volume data or GCS prefixes with operator tooling. ### Temporal worker Process: `node dist/temporal/worker-main.js` (`bun run dev:worker`). Polls `SELFBENCH_TEMPORAL_NAMESPACE` / `SELFBENCH_TASK_QUEUE` (defaults `default` / `selfbench-dev`). Default activity concurrency is `1` for Docker and `20` for Modal. Workflow phases: `queued` → `discovering` → `authoring` → `exporting` → `complete`. Terminals: `blocked` (candidate pool exhausted), `failed`, `cancelled`. Per-candidate statuses include `authoring`, `auditing`, `validating`, `reviewing`, `repairing`, `accepted`, `rejected`, and `infrastructure_failed`. Discovery fans out across **eight** shards. If the worker or Compose stack stops, Temporal history and artifacts stay on named volumes; work resumes when a worker polls the same task queue. A stopped worker does not cancel the run. ### Companion commands These bins operate on extracted Harbor tasks. They do not start Temporal runs. | Bin | Job | | --- | --- | | `self-bench-eval` | Codex matrix (`--export` or `--tasks`) into `--jobs`; default environment `modal`, concurrency `3`; writes `summary.json`; reuses completed Harbor jobs | | `self-bench-validate` | Harbor `nop` then `oracle` gates; writes `validation-summary.json` | | `self-bench-reaudit` | Independent test-to-gold coupling review | | `self-bench-repair` | Constrained repair of coupled tests; `--output` must be outside `--tasks` | | `self-bench-agent-smoke` | Install and instantiate pinned Harbor agent adapters | ### Review UI The API serves the Vite-built review app at `/`. The UI lists runs, opens a completed export or a local `.tar.gz`, inspects accepted tasks, drops selected tasks, and downloads `selfbench--cleaned.tar.gz`. API list/status/export calls send the session-only Bearer token from the password field. ## Acceptance bar An accepted task must: 1. Keep a real human request from local Pi (`~/.pi/agent/sessions`), Claude Code (`~/.claude/projects`), or Codex (`~/.codex/sessions` and `~/.codex/archived_sessions`) sessions that mention the checkout or its worktrees, or from a non-bot merged GitHub pull request (up to 500 listed). 2. Start from the pre-change base commit of that same pull request. 3. Meet the assigned difficulty size and fail-to-pass / pass-to-pass minima. 4. Pass static audit, Harbor nop, Harbor oracle, and a determinism rerun. 5. Pass an independent coupling review. The workflow allows one validation-repair attempt and one coupling-repair retry, then rejects. If the CLI finds no sanitized local-session or GitHub pull-request messages, `run` fails before submit: `no sanitized local-session or GitHub pull-request provenance was found`. If discovery cannot fill the requested tier budgets, the workflow becomes `blocked` with `candidate pool exhausted; missing easy=…`. | Difficulty | Implementation core | Fail-to-pass | Pass-to-pass | | --- | --- | --- | --- | | `easy` | ≥ 20 changed lines across ≥ 1 implementation path | ≥ 1 | no minimum | | `medium` | ≥ 50 changed lines across ≥ 2 implementation paths | ≥ 1 | ≥ 1 | | `hard` | ≥ 100 changed lines across ≥ 3 implementation paths | ≥ 1 | ≥ 2 | ## First path after install Install the global CLI, create a Modal token if you will use `--backend modal`, and authenticate `gh` with repository read access. Set `OPENAI_API_KEY`, `GH_TOKEN`, and a random `SELFBENCH_API_TOKEN` in the same shell that starts Compose so the worker and API receive them. ```bash self-bench up --backend modal export SELFBENCH_API_URL=http://127.0.0.1:8080 ``` Verify: Compose is up, `GET http://127.0.0.1:8080/healthz` returns `{ ok: true }`, and `self-bench --help` prints the command list. ```bash self-bench run \ --repo /absolute/path/to/your/repository \ --easy-count 10 \ --medium-count 10 \ --hard-count 10 \ --output ./self-bench-evals.tar.gz ``` `--output` waits through Temporal phases and writes a SHA-256-checked archive. Closing this process leaves the workflow running. Extract `manifest.json` and each `tasks/*.tar.gz` into a Harbor task directory (strip the outer `harbor-task/` prefix as in the quickstart). Run one Codex trial or `self-bench-eval --export ./self-bench-evals.tar.gz --jobs ./harbor-jobs`. A completed download prints `{ "runId": "…", "output": "…" }`. A premature export GET is `409` with `run export is not ready`. A hash mismatch deletes the partial file. ## Runtime defaults | Knob | Default | | --- | --- | | CLI API base | `http://127.0.0.1:8080` | | `up --backend` | `docker` | | Authoring model | `gpt-5.6-sol` | | Artifact store | local `.selfbench/artifacts` (Compose: `/var/lib/selfbench/artifacts`) | | Harbor environment | same as execution backend | | Modal image | `node:22-bookworm` | | Worker image Harbor pin | `harbor[modal]==0.20.1.dev202608040148` | | Provenance POST limit | 100 MiB | | Other JSON bodies | 10 MiB | ## Next Bun, Docker, Modal, GitHub, required environment variables, and the first `self-bench --help` check. Modal-backed local stack, a tiered `--output` run, extract accepted tasks, and one Codex trial. `self-bench up` / `down`, Compose services, backends, named volumes, and a stopped worker. `self-bench run`: origin pin, 1–100 tier budgets, provenance upload, wait, and SHA-256 download. Extract an export, run one task or the Codex matrix, resume jobs, and smoke adapters. Commands, flags, run-id rules, wait and download semantics, and command-level errors. Routes, Bearer auth, submission schema, provenance limits, export headers, and review assets. `SELFBENCH_*` keys, loopback token rule, artifact backends, and Temporal fields. --- ## 02. Installation > Prerequisites, global install, Modal and GitHub auth, required environment variables, and the first successful CLI help check. - Page Markdown: https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc/pages/02-installation.md - Generated: 2026-08-15T02:05:15.617Z ### Source Files - `README.md` - `package.json` - `compose.yaml` - `Dockerfile` - `Dockerfile.sandbox` - `scripts/verify-package.ts` --- title: "Installation" description: "Prerequisites, global install, Modal and GitHub auth, required environment variables, and the first successful CLI help check." --- The published `self-bench` package (currently `0.3.3`) is a Node `>=22` CLI. A global install puts six `#!/usr/bin/env node` binaries on `PATH` and ships `compose.yaml`, `Dockerfile`, `Dockerfile.sandbox`, and `src/skills/selfbench/SKILL.md` next to `dist/cli.js`, so later `self-bench up` can build the Compose stack from the package root rather than a git checkout. `self-bench --help` (also `-h`, `help`, or no arguments) is the install success check. It does not contact the API. `self-bench up`, `self-bench run`, Harbor on the host, and Temporal Cloud are later steps. ## Prerequisites | Requirement | Used for | Notes | | --- | --- | --- | | [Bun](https://bun.sh/) `1.3.14` or newer | Documented global install, CI, and `npm` publish | CI and publish pin Bun `1.3.14` | | Node.js `>=22` | Runtime for every packaged binary | `package.json` `engines.node`; worker/API image is `node:22-bookworm` | | Docker with Compose | `self-bench up` | Worker image embeds Docker CLI `29.4.0`; worker also runs `docker info` and `docker compose version` when the execution or Harbor backend is `docker` | | [Modal](https://modal.com/) CLI, account, and token | Recommended Modal sandboxes | `pip install modal`, then `modal token new` | | [`gh`](https://cli.github.com/) with repo read access | Host provenance and worker GitHub reads | Write access is not required | | `OPENAI_API_KEY` | Worker discovery, authoring, review, repair | README requires access to `gpt-5.6-sol`, `gpt-5.6-terra`, and `gpt-5.6-luna` | | A Git checkout whose `origin` is GitHub | `self-bench run` | HTTPS or `git@github.com:` SSH; not required to print CLI help | Host Harbor and `uv` are not install prerequisites. The API/worker image already installs `harbor[modal]==0.20.1.dev202608040148` via `uv`. Install Harbor on the host only when you evaluate an export later. ## Install the CLI ```bash bun add --global self-bench ``` The package is published public to `https://registry.npmjs.org/` as `self-bench`. The binaries are Node scripts; Node `>=22` is required. ```bash npm install --global self-bench ``` Tagged releases publish with `npm publish --access public` after `bun run validate`. The tag name must be `v` plus `package.json` `version`. A successful global install exposes: | Binary | Entry | Role | | --- | --- | --- | | `self-bench` | `dist/cli.js` | Stack, submit, list, status, cancel, download | | `self-bench-eval` | `dist/eval-main.js` | Harbor Codex matrix | | `self-bench-validate` | `dist/validate-main.js` | Offline Harbor validation | | `self-bench-reaudit` | `dist/reaudit-main.js` | Offline review re-audit | | `self-bench-repair` | `dist/repair-main.js` | Offline repair | | `self-bench-agent-smoke` | `dist/agent-smoke-main.js` | Harbor adapter smoke | `self-bench up` resolves the package root as the parent of `dist/cli.js` and requires these published files to exist there: :::files self-bench/ # node_modules/self-bench or Bun global package ├── dist/ │ └── cli.js # self-bench ├── compose.yaml ├── Dockerfile ├── Dockerfile.sandbox └── src/skills/selfbench/SKILL.md ::: Package verification packs the tarball, installs it, runs `self-bench --help`, and fails unless stdout contains `self-bench up` and those four assets are present. ## Authenticate Modal and GitHub ```bash pip install modal modal token new ``` `self-bench up --backend modal` mounts `~/.modal.toml` into the worker at `/root/.modal.toml`. Override the host path with `--modal-config /absolute/path/to/.modal.toml`. `--modal-config` without `--backend modal` is an error. A secret manager may set `MODAL_TOKEN_ID` and `MODAL_TOKEN_SECRET` instead of a profile file. Empty values for those two keys are deleted at worker startup so they cannot override a valid mounted profile. ```bash gh auth login export GH_TOKEN="$(gh auth token)" ``` The host CLI uses `gh pr list --state merged --limit 500` when you later run `self-bench run`. The Compose worker does not inherit the host `gh` login; it needs `GH_TOKEN` in the environment. Worker sandboxes that receive a token write `https://x-access-token:…@github.com` git credentials. If `GH_TOKEN` is unset, the worker process may fall back to `gh auth token` on its own `PATH`, which fails inside the container. `OPENAI_API_KEY` is the supported model credential. Existing deployments may set both `SELFBENCH_PI_AUTH_JSON` and `SELFBENCH_CODEX_AUTH_JSON` instead. When `OPENAI_API_KEY` is set (after trim), it wins for both Pi and Codex sandboxes. ## Required environment variables Export these in the same shell you will use for `self-bench up` and later CLI calls. Compose interpolates host values when the stack starts; changing them afterward does not update a running container. ```bash export OPENAI_API_KEY=... export GH_TOKEN="$(gh auth token)" export SELFBENCH_API_TOKEN="$(openssl rand -hex 24)" ``` After the stack is up, point the CLI at the published API port (optional on the default): ```bash export SELFBENCH_API_URL=http://127.0.0.1:8080 ``` Worker and evaluation sandboxes. Trimmed; empty is treated as unset. Passed into the Compose `worker` service. Host `self-bench run` still needs `gh` on `PATH`. Protects `/v1/*` with `Authorization: Bearer`. Compose binds the API at `SELFBENCH_API_HOST=0.0.0.0`, so a non-empty token is required (loopback hosts `127.0.0.1`, `::1`, and `localhost` are the only hosts that may start without one). CLI client base URL. Default `http://127.0.0.1:8080`. The CLI sends a Bearer header only when `SELFBENCH_API_TOKEN` is set in its own environment. | Variable | Default if unset | Who reads it | | --- | --- | --- | | `SELFBENCH_API_TOKEN` | Compose substitutes `selfbench-local` | API process and CLI | | `SELFBENCH_API_URL` | `http://127.0.0.1:8080` | CLI only | | `SELFBENCH_API_HOST` / `SELFBENCH_API_PORT` | `127.0.0.1` / `8080` in process config; Compose overrides host to `0.0.0.0` | API | | `SELFBENCH_EXECUTION_BACKEND` | `docker` | Worker; `self-bench up --backend` sets this | | `SELFBENCH_HARBOR_ENVIRONMENT` | same as execution backend | Worker; `up` sets this to the same backend | | `SELFBENCH_MODAL_CONFIG_PATH` | `/dev/null` in Compose | Worker volume mount; `up --backend modal` sets `~/.modal.toml` | | `MODAL_TOKEN_ID` / `MODAL_TOKEN_SECRET` | empty | Worker; empty strings are stripped | | `SELFBENCH_PI_AUTH_JSON` / `SELFBENCH_CODEX_AUTH_JSON` | unset | Worker fallback if no API key | | `SELFBENCH_BUILD_COMMIT` | baked at build, else `git rev-parse HEAD` | `up` / `run` | If you skip `SELFBENCH_API_TOKEN`, Compose still starts the API with `selfbench-local`, but the CLI sends no `Authorization` header. `/healthz` stays open; `/v1/*` returns `401` `{"error":"unauthorized"}`. Export the same token before `up` and in every later CLI shell. Review UI `GET /` and `GET /assets/*` are also unauthenticated. The API never receives `OPENAI_API_KEY`, `GH_TOKEN`, Modal tokens, or the Docker socket. Those stay on the worker. ```mermaid flowchart LR subgraph host [Host after install] CLI["self-bench"] Profile["~/.modal.toml"] Gh["gh + GH_TOKEN"] Key["OPENAI_API_KEY"] Tok["SELFBENCH_API_TOKEN"] end subgraph later [Started later by self-bench up] API["api :8080"] Worker["worker"] Temporal["temporal :7233"] PG["postgres"] end CLI -->|"Bearer + SELFBENCH_API_URL"| API Tok --> API Key --> Worker Gh --> Worker Profile -->|"mount or MODAL_TOKEN_*"| Worker API --- Temporal Worker --- Temporal Temporal --- PG ``` ## First CLI help check ```bash self-bench --help ``` `self-bench`, `self-bench -h`, and `self-bench help` print the same text. No API, Docker, Modal, or GitHub credentials are required. Stdout must include the `self-bench up` usage line. That is the same predicate `scripts/verify-package.ts` uses after a pack-and-install. ```text SelfBench creates durable tiered Harbor evaluations. Usage: self-bench up [--backend docker|modal] [--modal-config PATH] self-bench down self-bench run --repo PATH [--easy-count N] [--medium-count N] [--hard-count N] [--model MODEL] [--run-id ID] [--wait] [--output OUTPUT.tar.gz] self-bench status RUN_ID self-bench cancel RUN_ID self-bench download RUN_ID OUTPUT.tar.gz self-bench list The up command starts the local stack and configures both sandbox execution and Harbor validation for one backend. Modal uses ~/.modal.toml unless --modal-config overrides it. The tier counts are candidate authoring budgets, not accepted-task targets. Rejected candidates are not replaced, and the export contains only accepted tasks. The run command performs only repository metadata and sanitized provenance upload locally; discovery, authoring, validation, review, and audit run remotely. --output implies --wait, blocks until completion, and downloads the SHA-256-verified export. ``` An unknown first argument throws `unknown command: …`. That is a failed install or a typo, not a credentials problem. Optional companion check (not required for install): ```bash self-bench-eval --help ``` ## Source checkout (maintainers) From a clone, do not use the published tarball: ```bash git clone https://github.com/mupt-ai/self-bench.git cd self-bench bun install --frozen-lockfile bun run cli -- --help ``` `bun run validate` runs Biome, typecheck, tests, production build, and `bun run verify:package`. ## Install-time failures | Symptom | Cause | Fix | | --- | --- | --- | | `self-bench: command not found` | Global bin directory not on `PATH` | Re-open the shell or use the package manager's global bin path | | Help omits `self-bench up` | Incomplete or stale package | Reinstall `self-bench@0.3.3` or newer; do not copy `dist/cli.js` alone | | `SELFBENCH_API_TOKEN is required when the API listens beyond loopback` | API host is not `127.0.0.1` / `::1` / `localhost` and the token is missing | Set a non-empty `SELFBENCH_API_TOKEN` (Compose always binds `0.0.0.0`) | | `401 unauthorized` after a later `list`/`run` | CLI token missing or different from the value interpolated at `up` | Export the same token used when Compose started | | `--modal-config requires --backend modal` | Flag used with Docker backend | Pass `--backend modal` or drop `--modal-config` | | Modal SDK ignores `~/.modal.toml` | Empty `MODAL_TOKEN_ID` / `MODAL_TOKEN_SECRET` in the worker env | Leave them unset, or rely on worker startup stripping blanks | | `SelfBench was built without commit metadata` | `dist/build-metadata.js` is all-zero and `git rev-parse` cannot run | Set `SELFBENCH_BUILD_COMMIT` to a 40-character SHA | Do not treat a successful help print as a running stack. `self-bench up --backend modal` is the next operator command. ## Next Start the Modal-backed local stack, submit a tiered run with `--output`, and verify one Codex trial. `self-bench up` / `down`, Compose services, `--backend`, `--modal-config`, and named volumes. `OPENAI_API_KEY` precedence, Pi/Codex JSON fallbacks, `GH_TOKEN`, API Bearer, and Modal profile versus token secrets. Every `SELFBENCH_*` key, defaults, loopback token rule, and Temporal fields. Commands, flags, defaults, run-id rules, and command-level errors. Clone, `bun install`, `bun run validate`, source CLI, and npm tag publish. --- ## 03. Quickstart > Shortest path: start the Modal-backed local stack, submit a tiered run with --output, extract accepted Harbor tasks, and verify one Codex trial. - Page Markdown: https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc/pages/03-quickstart.md - Generated: 2026-08-15T02:05:33.948Z ### Source Files - `README.md` - `src/cli.ts` - `compose.yaml` - `docs/evaluations.md` - `src/evaluate.ts` - `tests/cli.test.ts` --- title: "Quickstart" description: "Shortest path: start the Modal-backed local stack, submit a tiered run with --output, extract accepted Harbor tasks, and verify one Codex trial." --- `self-bench up --backend modal` starts the Compose project `selfbench` on loopback (Postgres, Temporal, API, worker) and sets both `SELFBENCH_EXECUTION_BACKEND` and `SELFBENCH_HARBOR_ENVIRONMENT` to `modal`. `self-bench run --output` uploads sanitized provenance, starts Temporal workflow `selfBenchRunWorkflow`, polls until phase `complete`, and writes a create-only SHA-256-verified export. Extract `tasks/*.tar.gz` and run one Harbor Codex trial with `--env modal`. Install, Modal login, `gh auth login`, and the first `self-bench --help` check live on [Installation](/installation). This page assumes `self-bench` is on `PATH`. ## Preconditions | Requirement | Role in this path | | --- | --- | | Docker with Compose | `self-bench up` / `down` run `docker compose --file /compose.yaml` | | Modal CLI + account | `modal token new` writes `~/.modal.toml` (override with `--modal-config`) | | `gh` with repo read access | Local `gh pr list` provenance; worker clones with `GH_TOKEN` | | `OPENAI_API_KEY` | Worker authoring/review/repair; Harbor Codex trial on the host | | Git checkout with GitHub `origin` | CLI pins `HEAD`; uncommitted files are ignored | | Completed work in history | Local Pi / Claude Code / Codex sessions and/or merged PRs | Export these **before** `up`. Compose interpolates them into the API and worker at start time: ```bash export OPENAI_API_KEY=... export GH_TOKEN="$(gh auth token)" export SELFBENCH_API_TOKEN="$(openssl rand -hex 24)" ``` The API image defaults `SELFBENCH_API_TOKEN` to `selfbench-local` when the host variable is unset. The CLI sends `Authorization: Bearer` only when `SELFBENCH_API_TOKEN` is set in the **client** environment. Set the same token on the host before `up` and keep it exported for `run`, `status`, `list`, `cancel`, and `download`. Shared Bearer secret. Compose default if omitted: `selfbench-local`. Required on the API when it is not bound to loopback. CLI base URL. Default `http://127.0.0.1:8080`. Set after `up` so later commands hit this stack. Preferred model credential. Takes precedence over `SELFBENCH_PI_AUTH_JSON` / `SELFBENCH_CODEX_AUTH_JSON` subscription fallbacks. Worker GitHub credential (`https://x-access-token:…@github.com`). The CLI also shells out to `gh` for merged-PR provenance. ## Start the Modal-backed stack ```bash modal token new ``` Default profile path is `~/.modal.toml`. Empty `MODAL_TOKEN_ID` / `MODAL_TOKEN_SECRET` values are stripped at worker startup so they cannot override a valid mounted profile. ```bash self-bench up --backend modal export SELFBENCH_API_URL=http://127.0.0.1:8080 ``` `--backend` must be `docker` or `modal`. Omitting it selects **docker** (builds `selfbench-sandbox:local` from `Dockerfile.sandbox` first). Modal does not build that image. `--modal-config PATH` is allowed only with `--backend modal` and becomes `SELFBENCH_MODAL_CONFIG_PATH` (Compose bind-mounts it at `/root/.modal.toml:ro`). Expected stdout from `up`: ```text SelfBench is running with the modal backend at http://127.0.0.1:8080 ``` ```bash curl -sS http://127.0.0.1:8080/healthz ``` ```json { "ok": true } ``` `GET /healthz` is unauthenticated. Every `/v1/*` route requires the Bearer token when the API token is configured. Compose services and host binds: | Service | Image / command | Host bind | | --- | --- | --- | | `postgres` | `postgres:17.6-bookworm` (`POSTGRES_DB=temporal`) | none | | `temporal` | `temporalio/auto-setup:1.29.6` | `127.0.0.1:7233` | | `api` | `selfbench:local` → `node dist/api-main.js` | `127.0.0.1:8080` | | `worker` | same image → `node dist/temporal/worker-main.js` | none | Named volumes `selfbench_temporal-postgres` and `selfbench_artifacts` keep Temporal history and generated artifacts across `self-bench down`. The API never receives the Docker socket or model credentials. Modal worker activity concurrency defaults to **20**. ```mermaid flowchart LR subgraph operator ["Operator host"] CLI["self-bench CLI"] Checkout["Git checkout + session files"] HarborCLI["harbor / self-bench-eval"] end subgraph compose ["Compose project selfbench"] API["api :8080"] Worker["worker"] Temporal["temporal :7233"] PG[("postgres")] Art[("artifacts volume")] end subgraph modalCloud ["Modal"] Sandboxes["author / validate / review sandboxes"] EvalEnv["Harbor --env modal"] end Checkout --> CLI CLI -->|"POST /v1/provenance, POST /v1/runs"| API API --> Temporal Worker --> Temporal API --> Art Worker --> Art Worker --> Sandboxes HarborCLI --> EvalEnv ``` ## Submit a tiered run with `--output` `self-bench run` does **only** local git metadata and provenance upload. Discovery (8 shards), authoring, static audit, Harbor nop/oracle, determinism, review, one validation-repair, one coupling-repair, and export run in the worker. The checkout must have `origin` as `https://github.com/…` or `git@github.com:…` (normalized to HTTPS). Other remotes fail with `unsupported origin URL`. Tier flags are **candidate authoring budgets**, not accepted-task targets. Rejected candidates are not replaced. Combined total must be **1–100**. Each omitted count defaults to `0`. ```bash self-bench run \ --repo /absolute/path/to/your/repository \ --easy-count 10 \ --medium-count 10 \ --hard-count 10 \ --output ./self-bench-evals.tar.gz ``` Git working tree. CLI runs `git remote get-url origin` and `git rev-parse HEAD`. Nonnegative integers. Sum in `[1, 100]`. Eligibility later requires about 20 / 50 / 100 implementation lines across 1 / 2 / 3 paths. Authoring model. Default `gpt-5.6-sol`. API stores `authoring.provider = openai-codex` and `reasoningEffort = high`. Workflow id. Pattern `^[a-z0-9][a-z0-9-]{2,62}$` (3–63 chars). Default `sb-<14 datetime digits>-<8 uuid chars>`. Reusing an id against a live Temporal workflow fails at start. Implies `--wait`. Polls `GET /v1/runs/:runId` every 2s, logs phase changes to **stderr**, then `GET /v1/runs/:runId/export`. Opens the destination with `wx` (must not already exist). Verifies `x-content-sha256`; mismatch deletes the file. Wait without downloading. `--output` already waits. Terminal phases `blocked`, `failed`, and `cancelled` throw `SelfBench run [: ]`. Local provenance is the union of: - Sanitized user messages from `~/.pi/agent/sessions`, `~/.claude/projects`, `~/.codex/sessions`, and `~/.codex/archived_sessions` whose contents or paths mention the checkout or its worktrees - Up to **500** merged, non-draft, non-bot GitHub PRs with a title, at least **20** changed lines, and at least **1** changed file Zero messages abort **before** any API write: ```text no sanitized local-session or GitHub pull-request provenance was found ``` ```bash self-bench run --repo /abs/path/to/repo --easy-count 1 --output ./self-bench-evals.tar.gz ``` ```json { "runId": "sb-20260814153022-a1b2c3d4", "provenanceMessages": 42, "localProvenanceMessages": 17, "githubPullRequestMessages": 25 } ``` ```json {"runId":"sb-20260814153022-a1b2c3d4","phase":"discovering","accepted":0,"rejected":0} {"runId":"sb-20260814153022-a1b2c3d4","phase":"authoring","accepted":0,"rejected":0} {"runId":"sb-20260814153022-a1b2c3d4","phase":"exporting","accepted":1,"rejected":2} {"runId":"sb-20260814153022-a1b2c3d4","phase":"complete","accepted":1,"rejected":2} ``` ```json { "runId": "sb-20260814153022-a1b2c3d4", "output": "/abs/path/to/self-bench-evals.tar.gz" } ``` Phases: `queued` → `discovering` → `authoring` → `exporting` → `complete`. Closing the waiter does **not** cancel the workflow (`workflowExecutionTimeout` is 14 days). Stopping the worker or Compose stack pauses activities until the worker is back; Temporal state stays in `selfbench_temporal-postgres`. A run can reach `complete` with `accepted: 0` if every budgeted candidate is rejected. Discovery that cannot fill a requested tier ends `blocked` with `candidate pool exhausted; missing easy=…, medium=…, hard=…`. ## Extract accepted Harbor tasks The download is `application/gzip` with `content-disposition: attachment; filename="selfbench-.tar.gz"`. Exports include base snapshots, held-out tests, and gold patches. They are unencrypted; treat them as private. :::files self-bench-evals.tar.gz ├── manifest.json └── tasks/ └── .tar.gz └── harbor-task/ ├── task.toml # Harbor schema 1.4 ├── instruction.md ├── environment/ # agent image + repo.tar.gz ├── tests/ # held-out patch + verifier └── solution/ # gold.patch + solve.sh (oracle only) ::: `manifest.json` fields: `schemaVersion` (`1`), `runId`, `candidateCounts`, `repository` (`url`, `commit`), `version` (`selfbenchCommit`, `executionBackend`, `sandboxImage`, `schema`), `acceptedCount`, and `tasks[]` (`taskId`, per-archive `sha256`). ```bash mkdir -p ./self-bench-export ./self-bench-tasks tar -xzf ./self-bench-evals.tar.gz -C ./self-bench-export jq '{runId, acceptedCount, tasks: [.tasks[].taskId]}' ./self-bench-export/manifest.json TASK_ID="$(jq -r '.tasks[0].taskId' ./self-bench-export/manifest.json)" mkdir -p "./self-bench-tasks/$TASK_ID" tar -xzf "./self-bench-export/tasks/${TASK_ID}.tar.gz" -C "./self-bench-tasks/$TASK_ID" test -f "./self-bench-tasks/${TASK_ID}/harbor-task/task.toml" ``` `self-bench-eval --export FILE` materializes each `harbor-task/` under `/tasks//` (so `task.toml` is at the task-directory root). Manual extract keeps the `harbor-task/` prefix. Point Harbor at the directory that contains `task.toml`. The evaluated agent receives the base snapshot and `instruction.md` only. Harbor mounts `solution/` solely for the oracle agent. ## Verify one Codex trial Install the same Harbor pin the worker image uses (`HARBOR_VERSION=0.20.1.dev202608040148`): ```bash uv tool install --python 3.12 'harbor[modal]==0.20.1.dev202608040148' export OPENAI_API_KEY=... ``` ```bash harbor run \ --path "./self-bench-tasks/${TASK_ID}/harbor-task" \ --agent codex \ --model gpt-5.6-sol \ --ak version=0.146.1 \ --ak reasoning_effort=high \ --env modal \ --jobs-dir ./harbor-jobs \ --yes ``` `--env modal` must match the stack you started. A docker-backed stack would use `--env docker` instead. `self-bench-eval` wraps the same Codex adapter (`version=0.146.1`, `reasoning_effort=high`), writes `/summary.json`, and reuses a completed Harbor job named `-`. Omit `--model` and it runs the full matrix: `gpt-5.6-sol`, `gpt-5.6-terra`, `gpt-5.6-luna`. Pass `--model` once for a single trial per task. Default `--environment` is `modal`; default `--concurrency` is `3`. ```bash self-bench-eval \ --tasks "./self-bench-tasks/${TASK_ID}" \ --jobs ./harbor-jobs \ --environment modal \ --model gpt-5.6-sol ``` ```json { "trials": 1, "passed": 1, "failed": 0 } ``` A trial `passed` when Harbor produced rewards, there is no exception, and either `rewards.reward >= 1` or every numeric reward is `>= 1`. Empty rewards fail. Without `OPENAI_API_KEY`, pass `--auth /path/to/codex-auth.json` (`auth_mode: chatgpt` plus `tokens`). A successful trial writes a finished Harbor job under `./harbor-jobs` with `result.json` and no `exception_info`. Infrastructure strings (Docker daemon, Modal timeout, image-build failure, Harbor `AuthError`) are not task verdicts. ## If the waiter is gone Closing `self-bench run --output` leaves the workflow running. ```bash self-bench list self-bench status RUN_ID self-bench download RUN_ID ./self-bench-evals.tar.gz self-bench cancel RUN_ID ``` | Command | HTTP | | --- | --- | | `list` | `GET /v1/runs` | | `status RUN_ID` | `GET /v1/runs/:runId` | | `cancel RUN_ID` | `POST /v1/runs/:runId/cancel` → `{ cancellationRequested: true }` | | `download RUN_ID PATH` | `GET /v1/runs/:runId/export` | `download` before `complete` returns **409** `{ "error": "run export is not ready" }`. ```bash self-bench down ``` `down` is `docker compose down` and does **not** delete `selfbench_temporal-postgres` or `selfbench_artifacts`. The API also serves the review UI at `http://127.0.0.1:8080/` for a completed run. ## Failures on this path | Symptom | Cause | | --- | --- | | `unknown command` / missing `self-bench` | Package not on `PATH`; see [Installation](/installation) | | `--backend must be "docker" or "modal"` | Typo or omitted value treated as the docker default | | `--modal-config requires --backend modal` | Config path passed with docker backend | | `401 unauthorized` | CLI token missing or not the token Compose baked in at `up` | | Empty-corpus error | No session files mentioning the checkout and no qualifying merged PRs | | `unsupported origin URL` | `origin` is not GitHub HTTPS or `git@github.com:` | | `the total candidate count must be between 1 and 100` | All counts `0`, or sum `> 100` | | `SelfBench run blocked: candidate pool exhausted…` | Discovery could not fill a requested tier | | `SelfBench run failed` / `cancelled` | Workflow error or `cancel`; `--output` does not download | | `run export is not ready` | `download` while phase is not `complete` | | `downloaded export failed its SHA-256 integrity check` | Streamed bytes ≠ `x-content-sha256`; dest file removed | | `EEXIST` on `--output` | Destination already exists (`wx`) | | Harbor `AuthError` / Modal timeout | Host `OPENAI_API_KEY` or Modal profile missing for the **eval** machine | | `found no Harbor tasks` | Export `acceptedCount` is `0`, or `--tasks` dir has no `task.toml` | ## Next Compose services, `--backend docker|modal`, `--modal-config`, named volumes, and worker-stop behavior. `--repo`, 1–100 tier budgets, `--model`, `--run-id`, HEAD pin, provenance upload, `--output` wait. Full extract, one task vs Codex matrix, `self-bench-eval` resume, adapter smoke. `list`, `status`, `cancel`, `download`, phase polling, create-only writes. No-provenance, blocked pool, 409, SHA-256 mismatch, Harbor infrastructure failures. --- ## 04. Provenance sources > Where self-bench collects human requests: Pi, Claude Code, and Codex session roots, merged GitHub pull requests, sanitization rules, and the empty-corpus failure. - Page Markdown: https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc/pages/04-provenance-sources.md - Generated: 2026-08-15T02:05:10.760Z ### Source Files - `src/provenance.ts` - `src/github.ts` - `src/cli.ts` - `src/extensions/discovery.ts` - `tests/provenance.test.ts` - `docs/task-construction.md` --- title: "Provenance sources" description: "Where self-bench collects human requests: Pi, Claude Code, and Codex session roots, merged GitHub pull requests, sanitization rules, and the empty-corpus failure." --- `self-bench run` collects the human-request corpus on the operator machine, sanitizes it, and uploads NDJSON to `POST /v1/provenance?runId=...` before the Temporal workflow starts. Discovery later ranks candidates only from that uploaded corpus. A model-authored prompt may restate a retained request; it may not invent behavior from the implementation or tests. The CLI concatenates two collectors, then fails closed if both return nothing: 1. Local user turns from Pi, Claude Code, and Codex session trees under `$HOME` that mention the `--repo` checkout or one of its Git worktrees. 2. Exact titles and optional bodies from up to 500 merged, non-bot GitHub pull requests that clear a coarse size floor. ```mermaid flowchart LR subgraph Operator["Operator machine"] HOME["$HOME session roots
.pi / .claude / .codex"] GH["gh pr list --state merged --limit 500"] CLI["self-bench run"] HOME --> CLI GH --> CLI CLI -->|0 messages| FAIL["Error: no sanitized local-session
or GitHub pull-request provenance was found"] end subgraph API["API artifact store"] UP["POST /v1/provenance"] STORE["runs/{runId}/input/provenance.jsonl"] UP --> STORE end subgraph Worker["Discovery sandbox"] SHARD["8 shards: index % 8"] FILE["/work/provenance.jsonl"] BIND["Match sourceType + sessionId + messageIndex"] SHARD --> FILE --> BIND end CLI -->|"application/x-ndjson ≤ 100 MiB"| UP STORE --> SHARD ``` ## Collection during `self-bench run` `self-bench run --repo PATH` pins `origin` and `HEAD`, then collects provenance locally. Uncommitted files are not part of the pin. `origin` must be `https://github.com/owner/repo` or `git@github.com:owner/repo` (optional `.git`). SSH remotes are rewritten to HTTPS before GitHub collection. Git checkout whose `origin` is a GitHub URL. Worktree paths from `git worktree list --porcelain` become the local-session association needles. Session scan root. The CLI uses `process.env.HOME`, then `os.homedir()`. Tests and CI that set `HOME` only see sessions under that directory. Local collection and repository resolution run in parallel. GitHub collection runs next, using the resolved origin. The uploaded corpus is local messages followed by GitHub messages. ```bash self-bench run \ --repo /absolute/path/to/repository \ --easy-count 5 \ --medium-count 10 \ --hard-count 5 \ --output ./self-bench-tasks.tar.gz ``` ```json { "runId": "sb-20260814153000-a1b2c3d4", "provenanceMessages": 42, "localProvenanceMessages": 18, "githubPullRequestMessages": 24 } ``` `provenanceMessages` is the uploaded count after sanitization. It is not an accepted-task count. Discovery, authoring, validation, review, and audit run remotely after this upload. The CLI is the supported collector. A direct `POST /v1/provenance` stores whatever NDJSON you send; the API does not re-run session scans or the empty-corpus check. ## Local session roots `collectRepositoryProvenance` walks these directories under `$HOME`, recursively, and reads only `*.jsonl` files. Missing roots are skipped. | `sourceType` | Root under `$HOME` | Session id when present in the file | | --- | --- | --- | | `pi` | `.pi/agent/sessions` | record `type=session` field `id` | | `claude-code` | `.claude/projects` | record field `sessionId` | | `codex` | `.codex/sessions` | `session_meta.payload.id` | | `codex` | `.codex/archived_sessions` | same Codex parser | If a file has no embedded session id, the CLI uses the filename (`basename`). A file is kept only when its path **or** raw contents include a worktree path from the `--repo` checkout, or that path with `/` replaced by `-`. The dash encoding matches Claude Code project directories such as `-Users-you-src-project`. A Codex or Pi transcript that merely mentions the absolute worktree path in a user turn is also kept. Each matching file is parsed as JSON, a `{ "messages": [...] }` wrapper, or JSONL. The collector then keeps **user** turns only: | Format | How user text is read | | --- | --- | | `codex` | `event_msg` payloads with `user_message`. If none exist, `response_item` messages with `role=user`. | | `claude-code` | records with `type` `user` or `assistant` and a nested `message`. Records that set `sourceToolAssistantUUID` are dropped (tool-echoed user turns). | | `pi` | records with `type=message` and a nested `message`. | | `generic` | top-level `role`/`content`, else nested `message`. | The CLI tags files by the root they came from (`pi`, `claude-code`, or `codex`). `generic` is accepted later by `submit_discovery`; the CLI session walk does not emit it. `messageIndex` counts only retained user turns in that session (0-based). Empty strings after trim and redaction are dropped. ## Merged GitHub pull requests `collectGitHubPullRequestProvenance` runs local `gh`: ```bash gh pr list \ --repo owner/repo \ --state merged \ --limit 500 \ --json number,title,body,url,author,isDraft,additions,deletions,changedFiles ``` `owner/repo` is the lowercased GitHub repository from `origin`. `gh` must be on `PATH` and authenticated with read access. A non-zero `gh` exit is a hard CLI error (`gh exited N: ...`), not the empty-corpus message. A listed PR becomes one corpus record when **all** of the following hold: | Check | Rule | | --- | --- | | Draft | `isDraft` is not `true` | | Author | `author.login` is a string, `author.is_bot` is not `true`, and `login` does not end with `[bot]` (case-insensitive) | | Identity | positive `number`, non-empty `url`, non-empty trimmed `title` | | Size floor | `additions + deletions >= 20` and `changedFiles >= 1` | | URL bind | `url` is `https://github.com/owner/repo/pull/{number}` for the same repository | This size floor is the easy-tier line minimum only. Discovery still assigns `easy` / `medium` / `hard` from the implementation core. Record content: - Body present and `length <= 12000` after trim → `{title}\n\n{body}` - Missing body, or body longer than 12 000 characters → **title only** (the body is omitted, not truncated) ```json { "sourceType": "github-pull-request", "sessionId": "github:example/project#42", "messageIndex": 0, "content": "Add the public routing API\n\nExpose routing through the documented client.", "sourcePr": 42, "sourceUrl": "https://github.com/example/project/pull/42" } ``` HTTPS and SSH origins both bind to the same lowercased `owner/repo`. A PR URL from another repository, or a number that does not match the URL, is rejected. ## Record shape Every uploaded line is one JSON object: `pi` | `claude-code` | `codex` | `generic` | `github-pull-request` Local session id, filename fallback, or `github:{owner/repo}#{number}`. Non-negative index of the retained user turn. GitHub records always use `0`. Sanitized request text. Discovery copies this onto the candidate; it does not keep the discovery agent's `request` string. GitHub records only. Positive pull-request number. GitHub records only. Canonical `https://github.com/owner/repo/pull/{n}` URL. Local and GitHub records are not cross-deduplicated. The same wording can appear once as a session turn and again as a PR title/body. ## Sanitization User text is passed through `redactSecrets`, then dropped if it looks like injected harness context. Dropped when the trimmed string **starts with** any of: | Prefix | Typical source | | --- | --- | | `# AGENTS.md instructions` | injected agent instructions | | `# Review Guidelines` | review boilerplate | | `` | host environment dump | | `` | permission preamble | | `` | collaboration preamble | | `` | skill injection | | `` | app injection | | `` | plugin injection | | ` Redaction is a fixed regex list. Session files and PR bodies can still contain secrets that do not match these patterns. Treat the uploaded corpus as sensitive run input, not as scrubbed public data. ## Upload :::endpoint POST /v1/provenance Store the sanitized corpus for a run. **Query** | Name | Type | Constraint | | --- | --- | --- | | `runId` | string | `^[a-z0-9][a-z0-9-]{2,62}$` | **Request** - `Content-Type: application/x-ndjson` - Body: one JSON object per line - Limit: **100 MiB** (`100 * 1024 * 1024`). Larger bodies fail with `request body exceeds … bytes` (HTTP 500). Other API JSON routes use a 10 MiB default. **Auth** `Authorization: Bearer $SELFBENCH_API_TOKEN` when the API token is configured. Unauthenticated if the token is unset. **Storage key** `runs/{runId}/input/provenance.jsonl` **Success:** `201` with an artifact ref (`uri`, `sha256`, `sizeBytes`, `contentType`). `self-bench run` posts that ref on `POST /v1/runs`. ::: Invalid `runId` is HTTP 400. The API does not parse or validate line schema at upload time. ## How discovery uses the corpus The workflow splits the uploaded lines across **eight** shards (`index % 8`). Each shard sandbox receives `/work/provenance.jsonl` and must call `submit_discovery` exactly once. An empty candidate list is valid for a thin shard. Discovery rules encoded in the shard prompt and enforced after the tool call: - Prefer local `pi`, `claude-code`, and `codex` turns when they clearly describe the same change as a merged PR on `SOURCE_REPO_URL`. - `github-pull-request` is valid fallback provenance and may be used **only** for its own `sourcePr` and `sourceUrl`. - Select a request only by an exact `sourceType` + `sessionId` + `messageIndex` present in that shard. Invented ids fail with `candidate … references unknown provenance`. - Pairing a GitHub record with a different PR fails `assertProvenanceMatchesPullRequest`. - Repository names or PR URLs inside message text are context only. `sourceUrl` / `sourcePr` must belong to the run's origin. - Candidate `request` is replaced with the corpus `content` before authoring. Authoring mounts the staged record at `/work/provenance.json`: ```json { "source": { "sourceType": "codex", "sessionId": "session-1", "messageIndex": 0, "content": "…" }, "messages": [{ "role": "user", "content": "…" }] } ``` The authoring skill may use that file only to preserve the original intent. The Harbor export rebuilds `harbor-task/` from the accepted definition and patches; it does **not** include `provenance.jsonl`, session files, or author/reviewer transcripts. ## Empty corpus and related failures ### CLI: no retained messages If local collection and GitHub collection both return zero messages, `self-bench run` throws **before** upload: ```text no sanitized local-session or GitHub pull-request provenance was found ``` Typical causes: | Symptom | What to check | | --- | --- | | No local hits | Sessions live under a different `$HOME`. Files are not `*.jsonl`. Transcript never mentions the worktree path (and the file path is not the dash-encoded worktree). | | Only injected turns | Remaining user text starts with an injected prefix and was dropped. | | No GitHub hits | No merged PRs in the `gh` window, only drafts/bots, titles empty, or every PR is under 20 changed lines / 0 files. | | `gh` not usable | Missing binary, auth, or repo read access. You see `gh exited …`, not the empty-corpus string. | | Unsupported `origin` | Non-GitHub remote fails in `resolveRepository` before GitHub collection. | Run `git -C "$REPO" remote get-url origin` and `git -C "$REPO" worktree list --porcelain`. The origin must be GitHub. Every `worktree ` path is a local-session needle. Look for `*.jsonl` under `$HOME/.pi/agent/sessions`, `$HOME/.claude/projects`, `$HOME/.codex/sessions`, and `$HOME/.codex/archived_sessions`. Confirm the worktree path or its dash-encoded form appears in the file path or contents. Run the same `gh pr list --state merged --limit 500 --json …` command the CLI uses. Drafts, `*[bot]` authors, and PRs with fewer than 20 changed lines never enter the corpus. ### After upload: corpus present but no candidates A non-empty upload can still leave discovery with nothing it may pair to a merged PR on the origin. When a wave returns no new unique `sourcePr` values, the workflow enters `blocked` with: ```text candidate pool exhausted; missing easy=N, medium=N, hard=N ``` That is a later terminal than the CLI empty-corpus error. Shard failures that are not cancellation or non-retryable activity errors contribute zero candidates and can produce the same blocked pool. ## Related pages `self-bench run` flags, HEAD pin, provenance upload, and `--output` wait. Eight-shard discovery, candidate budgets, and the blocked pool terminal. Implementation-line and test-count minima applied after a request is paired. `POST /v1/provenance` among the other authenticated run routes. Local `gh` / `GH_TOKEN` used for GitHub reads after collection. No-provenance, blocked candidate pool, and related operator failures. --- ## 05. Difficulty gates > easy, medium, and hard eligibility: implementation file and line minima, fail-to-pass and pass-to-pass counts, and static audit blockers. - Page Markdown: https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc/pages/05-difficulty-gates.md - Generated: 2026-08-15T02:03:44.028Z ### Source Files - `src/audit.ts` - `src/contracts.ts` - `src/skills/selfbench/SKILL.md` - `docs/task-construction.md` - `tests/audit.test.ts` --- title: "Difficulty gates" description: "easy, medium, and hard eligibility: implementation file and line minima, fail-to-pass and pass-to-pass counts, and static audit blockers." --- `auditTaskDefinition` is the mechanical eligibility check for every authored Harbor task. After authoring, and again after either repair, Temporal `auditTask` reads `definition.json`, `solution/gold.patch`, and `tests/test.patch`. A clean report moves the candidate to Harbor nop/oracle validation. Any blocker rejects the candidate with `status: "auditing"` then `status: "rejected"` and `reason` set to the joined blocker list. Tiers `easy`, `medium`, and `hard` are size-and-structure eligibility rules, not measured model-success rates. `--easy-count`, `--medium-count`, and `--hard-count` are authoring budgets. Rejected candidates are not replaced. The export contains only tasks that pass static audit, Harbor validation, and independent review. ## Tier thresholds Thresholds are hardcoded. Discovery assigns the highest honest tier; authoring and repair must keep that assigned `difficulty`. | `difficulty` | Gold implementation files | Gold changed lines | `failToPass` | `passToPass` | | --- | ---: | ---: | ---: | ---: | | `easy` | ≥ 1 | ≥ 20 | ≥ 1 | no minimum | | `medium` | ≥ 2 | ≥ 50 | ≥ 1 | ≥ 1 | | `hard` | ≥ 3 | ≥ 100 | ≥ 1 | ≥ 2 | Every accepted task also needs a held-out test patch with no path overlap against gold, a `testCommand` that expands `{tests}` exactly once as an unquoted argument list, deterministic repository-native setup, and a passing nop/oracle split. Generated or vendored suitability is a review judgment, not a second size classifier. Git LFS, submodules, and service-heavy suites have no special path. Static audit does **not** count `failToPass.length`. The ≥ 1 fail-to-pass rule is `taskDefinitionSchema` (`failToPass.min(1)`) plus Harbor nop. An empty `failToPass` never reaches `auditTaskDefinition` because definition parse fails first. ## What the size gate counts The file and line minima apply only to **gold**. Held-out tests have a separate “changes at least one file” check. | Metric | Source | How it is counted | | --- | --- | --- | | `implementationFiles` | `gold.patch` | One entry per `diff --git a/… b/` dest path | | `implementationChangedLines` | `gold.patch` | Every `+` or `-` line except `+++` / `---` headers | | `testFiles` | `test.patch` | Same dest-path rule as gold | Hunk headers, `diff --git` lines, and `---` / `+++` file headers do not count as changed lines. Additions and deletions both count. The authoring skill requires gold to contain only the non-test implementation core; if the author leaves tests, generated files, or cleanup in gold, those paths inflate the mechanical totals. ```text diff --git a/src/a.ts b/src/a.ts ← dest path counts as 1 file --- a/src/a.ts ← ignored +++ b/src/a.ts ← ignored @@ -1,2 +1,3 @@ ← ignored -old ← 1 changed line +new ← 1 changed line +extra ← 1 changed line ``` GitHub provenance uses a coarser floor **before** discovery: merged non-bot PRs need `additions + deletions >= 20` and `changedFiles >= 1`. That filter is the easy floor on GitHub’s PR totals, not the gold-only audit metrics, and it is not raised for medium or hard. Local Pi, Claude Code, and Codex messages have no line filter; discovery must still reject a tiny implementation core. ## Fail-to-pass and pass-to-pass | Field | Meaning | Enforced by | | --- | --- | --- | | `failToPass` | Selectors that fail on the base snapshot plus held-out tests, and pass after gold | Schema ≥ 1; Harbor nop/oracle | | `passToPass` | Selectors that already pass on the base and must stay green | Audit length vs tier; Harbor nop/oracle | | `testPaths` | Held-out files the verifier protects | Schema ≥ 1; not a size gate | | `{tests}` | Placeholder substituted with the selected selectors | Schema: contains `{tests}`; audit: exactly once, unquoted, no hardcoded selector paths | For Harbor, empty `passToPass` (legal on `easy`) becomes a no-op `true` in the verifier script. Medium and hard reject that empty list at static audit: ```text medium mode requires at least 1 pass-to-pass regression tests hard mode requires at least 2 pass-to-pass regression tests ``` `testCommand` examples: | Command | Schema | Static audit | | --- | --- | --- | | `bun test {tests}` | pass | pass | | `bun test "{tests}"` | pass | reject — quoted placeholder | | `bun test {tests} {tests}` | pass | reject — placeholder not exactly once | | `bun test {tests} && bun test tests/new.ts` | pass | reject if `tests/new.ts` is in `failToPass` or `passToPass` | ## Where the gates fire ```mermaid stateDiagram-v2 [*] --> authoring authoring --> rejected: no task or difficulty mismatch authoring --> auditing: submit_task auditing --> rejected: blockers auditing --> validating: accepted validating --> reviewing: nop and oracle pass validating --> repairing: one validation repair reviewing --> accepted reviewing --> repairing: one coupling repair repairing --> auditing: same difficulty ``` | Stage | What is locked | Failure | | --- | --- | --- | | Discovery | One `difficulty` per PR: highest honest tier | Shard over the per-tier cap is a hard error; empty pool after waves → run `blocked` | | Authoring | `definition.difficulty` must equal `candidate.difficulty` | `authored task does not match its assigned PR, base commit, and difficulty` | | `auditTask` | Gold files/lines, `passToPass` length, disjoint patches, `{tests}` shape | `reason` = `blockers.join("; ")` | | Validation repair | Cannot change `difficulty`, identity, prompt, gold, or base | Immutable-field error, then reject | | Coupling repair | Held-out tests only; then full re-audit | Same audit blockers apply to the repaired bundle | Discovery prompt rule: assign **exactly one** tier and choose the **highest** the separable implementation core meets. A 100-line, 3-file change is hard even if the run still has empty easy slots. Authoring must reject the candidate instead of retiering or weakening thresholds. `submit_task` writes `definition.json`, `gold.patch`, and `test.patch` once. ## Static audit blockers `auditTask` stores the JSON report at: ```text runs//audits//.json ``` `true` only when `blockers` is empty. Human-readable reject reasons. Joined with `"; "` onto `TaskProgress.reason`. Gold dest-path count. Gold `+`/`-` line count excluding `+++`/`---`. Held-out dest-path count. `0` is always a blocker. | Condition | `blockers` entry | | --- | --- | | Same dest path in gold and tests | `gold and held-out test patches overlap: ` | | Too few gold files | ` mode requires at least implementation files; found ` | | Too few gold lines | ` mode requires at least changed implementation lines; found ` | | Empty test patch | `held-out test patch changes no files` | | Too few `passToPass` | ` mode requires at least pass-to-pass regression tests` | | Selector path appears outside `{tests}` | `test command must not hard-code fail-to-pass or pass-to-pass paths outside "{tests}"` | | `{tests}` missing or repeated | `test command must contain "{tests}" exactly once` | | `"{tests}"` or `'{tests}'` | `test command must expand "{tests}" as an unquoted shell argument list` | ```json { "accepted": false, "blockers": [ "medium mode requires at least 50 changed implementation lines; found 49" ], "metrics": { "implementationFiles": 2, "implementationChangedLines": 49, "testFiles": 1 } } ``` `self-bench-validate` reruns Harbor nop/oracle only. `self-bench-reaudit` reruns test-to-gold coupling review. Neither command reruns `auditTaskDefinition`. ## Requested budgets CLI authoring budget for `easy`. Combined total across tiers must be 1–100. CLI authoring budget for `medium`. CLI authoring budget for `hard`. `runRequestSchema.candidateCounts` is the same shape: each tier `0–100`, sum `1–100`. HTTP `POST /v1/runs` accepts `candidateCounts` and Temporal status exposes `requestedByDifficulty`. ```bash self-bench run \ --repo /absolute/path/to/repository \ --easy-count 5 \ --medium-count 10 \ --hard-count 5 \ --output ./self-bench-tasks.tar.gz ``` Discovery expands until each requested tier is filled or the pool is exhausted (`MAX_DISCOVERED_CANDIDATES` is 300). Exhaustion is terminal: ```text candidate pool exhausted; missing easy=2, medium=0, hard=1 ``` The run phase becomes `blocked`. That is a coverage failure, not a size-audit failure. ## Diagnose a rejected candidate `self-bench status ` (or `GET /v1/runs/`) lists `tasks[]` with `difficulty`, `status`, and `reason`. Audit rejects look like the blocker strings above. Difficulty-mismatch rejects say the authored task does not match the assigned PR, base commit, and difficulty. Open the audit JSON for that `taskId`. `metrics.implementationFiles` and `metrics.implementationChangedLines` must meet the assigned `difficulty`, not a lower one. Do not retier the definition. Confirm gold and tests share no dest paths, `test.patch` is non-empty, `{tests}` appears once and unquoted, and selected selectors are not repeated in the rest of `testCommand`. A rejected hard candidate does not free a hard budget for another PR. If discovery never found enough honest candidates, the whole run blocks instead of exporting a partial set. ## Related pages Discovery shards, candidate budgets versus accepted tasks, and `blocked` / `rejected` terminals. Harbor nop/oracle, determinism, one validation repair, one coupling repair, then reject. `definition.json` fields, `{tests}` rules, and `schemaVersion` 1 constraints. Session roots, merged GitHub PRs, and the GitHub 20-line / 1-file ingest floor. Empty provenance and blocked candidate-pool failures. --- ## 06. Run lifecycle > Temporal phases, per-candidate statuses, eight-shard discovery, candidate budgets versus accepted tasks, and blocked, failed, or cancelled terminals. - Page Markdown: https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc/pages/06-run-lifecycle.md - Generated: 2026-08-15T02:04:36.849Z ### Source Files - `src/temporal/workflow.ts` - `src/contracts.ts` - `src/run-wait.ts` - `src/cli.ts` - `tests/workflow.test.ts` - `tests/run-wait.test.ts` --- title: "Run lifecycle" description: "Temporal phases, per-candidate statuses, eight-shard discovery, candidate budgets versus accepted tasks, and blocked, failed, or cancelled terminals." --- `self-bench run` starts Temporal workflow `selfBenchRunWorkflow` with `workflowId` equal to the run ID. The worker executes `executeRun`, installs the `status` query, and advances a single `phase` from `queued` through `discovering`, `authoring`, and `exporting`. The run finishes at `complete` with an export artifact, or at `blocked`, `failed`, or `cancelled`. Per-candidate `authoring` / `auditing` / `validating` / `reviewing` / `repairing` values are task statuses inside the authoring phase, not extra run phases. `--easy-count`, `--medium-count`, and `--hard-count` are authoring budgets. Discovery fills those slots, then each selected candidate is processed once. Rejected or infrastructure-failed candidates are not replaced. The export contains only accepted tasks, including when that set is empty. ## Start a run `POST /v1/runs` starts the workflow on `SELFBENCH_TASK_QUEUE` (default `selfbench-dev`) with `workflowExecutionTimeout` of 14 days. The API returns `202` and `{ "runId": "..." }` immediately. Discovery, authoring, Harbor validation, review, audit, and export run in the worker. ```bash self-bench run \ --repo /absolute/path/to/repository \ --easy-count 5 \ --medium-count 10 \ --hard-count 5 \ --model gpt-5.6-sol \ --output ./self-bench-tasks.tar.gz ``` 3–63 characters matching `^[a-z0-9][a-z0-9-]{2,62}$`. CLI default is `sb-{YYYYMMDDHHmmss}-{8 hex}`. Temporal uses this as `workflowId`; a failed or cancelled run cannot be restarted under the same ID. Nonnegative integers. Sum must be 1–100. Each field is also capped at 100. The CLI uploads provenance first, then submits the run. `--output` implies `--wait`. Without `--wait` or `--output`, the process exits after the `202`. ## Ownership ```mermaid flowchart LR subgraph client [CLI] runCmd["self-bench run / status / cancel / download"] end subgraph api [API] postRuns["POST /v1/runs"] getStatus["GET /v1/runs/:runId"] postCancel["POST /v1/runs/:runId/cancel"] getExport["GET /v1/runs/:runId/export"] end subgraph temporal [Temporal] wf["selfBenchRunWorkflow
workflowId = runId"] q["query status"] end subgraph worker [Worker] disc["discoverCandidateShard x8"] author["author / audit / validate / review / repair"] exp["buildExport"] end runCmd --> postRuns runCmd --> getStatus runCmd --> postCancel runCmd --> getExport postRuns --> wf getStatus --> q q --> wf postCancel --> wf wf --> disc wf --> author wf --> exp getExport --> wf ``` The API never mounts the Docker socket or model credentials. The worker polls `SELFBENCH_TASK_QUEUE` with `SELFBENCH_ACTIVITY_CONCURRENCY` (default `1` for Docker, `20` for Modal). `Promise.all` over selected candidates is therefore bounded by that concurrency. ## Run phases ```mermaid stateDiagram-v2 [*] --> queued queued --> discovering discovering --> authoring: each tier budget filled discovering --> blocked: no unique PRs or 300-candidate cap authoring --> exporting: every selected candidate finished exporting --> complete: export artifact stored discovering --> cancelled: Temporal cancel authoring --> cancelled: Temporal cancel discovering --> failed: non-retryable or uncaught error authoring --> failed: uncaught error exporting --> failed: export activity failed complete --> [*] blocked --> [*] failed --> [*] cancelled --> [*] ``` | `phase` | Meaning | Terminal | | --- | --- | --- | | `queued` | Initial query value before the first discovery wave. Also the API fallback when Temporal is `RUNNING` but the `status` query is not readable. | No | | `discovering` | One or more eight-shard waves until every requested tier budget is filled. | No | | `authoring` | Selected candidates run author → audit → validate → review, with at most one validation repair and one coupling repair each. | No | | `exporting` | `buildExport` writes `manifest.json` plus accepted Harbor task archives. | No | | `complete` | Export artifact is on the status payload. Download is allowed. | Yes, success | | `blocked` | Discovery could not fill every requested tier. Thrown as non-retryable `CandidatePoolExhausted`. | Yes, wait fails | | `failed` | Uncaught activity or Temporal close (`FAILED`, `TERMINATED`, `TIMED_OUT`). `error` is set when the workflow catch path ran. | Yes, wait fails | | `cancelled` | Temporal cancellation observed (`CancelledFailure` or Temporal `CANCELED`). | Yes, wait fails | `waitForRun` treats only `complete` as success. `blocked`, `failed`, and `cancelled` throw `SelfBench run ` and, when `error` is a string, append `: `. Default poll interval is 2000 ms. Phase-change lines go to stderr as JSON `{ runId, phase, accepted, rejected }`. `GET /v1/runs` lists Temporal execution names (`RUNNING`, `COMPLETED`, `CANCELED`, `FAILED`). `GET /v1/runs/:runId` returns the workflow `phase`. Do not treat list `status` as `RunPhase`. ## Status payload :::endpoint GET /v1/runs/:runId Live workflow status The handler reads Temporal `describe()` plus query `status`. While the execution is `RUNNING`, or when the queried `phase` is already terminal, the query payload is returned as-is. Otherwise the API maps Temporal `COMPLETED` → `complete`, `CANCELED` → `cancelled`, `RUNNING` → `queued`, and every other close state → `failed`. ::: Workflow ID. One of the eight values above. `easy + medium + hard` from the submission. `{ easy, medium, hard }` authoring budgets. Unique candidates materialized across discovery waves, capped at 300. Tasks whose status is `accepted`. `infrastructure_failed` is not counted here. Tasks whose status is `rejected`. `infrastructure_failed` is not counted here. One row per selected candidate once authoring starts. Present after the first wave progress update: `wave`, `totalShards` (always 8), `completedShards`, `failedShards`, `candidates`. Set only in `complete`: `uri`, `sha256`, `sizeBytes`, `contentType`. Set on the `failed` catch path. `blocked` and `cancelled` do not populate this field on the query payload. ```json { "runId": "sb-20260814153000-a1b2c3d4", "phase": "authoring", "requested": 3, "requestedByDifficulty": { "easy": 1, "medium": 1, "hard": 1 }, "discovered": 4, "accepted": 1, "rejected": 1, "discovery": { "wave": 1, "totalShards": 8, "completedShards": 8, "failedShards": 0, "candidates": 4 }, "tasks": [ { "taskId": "easy-1-task", "candidateId": "w0s0-easy-1", "difficulty": "easy", "status": "accepted" }, { "taskId": "medium-1", "candidateId": "medium-1", "difficulty": "medium", "status": "rejected", "reason": "not reproducible" }, { "taskId": "hard-1-task", "candidateId": "w0s0-hard-1", "difficulty": "hard", "status": "validating" } ] } ``` ## Candidate budgets versus accepted tasks Discovery expands until `selectCandidates` can satisfy every requested tier, then authoring consumes **exactly** those selected rows. | Count | What it measures | | --- | --- | | `requested` / `requestedByDifficulty` | Authoring slots to fill. | | `discovered` | Unique PRs kept from shards (may exceed the budget because of overfetch). | | Authored set | First-fit selection by interleaved shard rank, per tier, up to the budget. Extra discovered PRs are not authored. | | `accepted` | Tasks that passed audit, Harbor nop/oracle, and coupling review. | | Export `acceptedCount` | Same as accepted; rejected and `infrastructure_failed` rows are omitted. | A mixed budget `{ easy: 1, medium: 1, hard: 1 }` with four discovered PRs authors three candidates. If the medium candidate is rejected, the extra easy PR is **not** substituted. The run still reaches `exporting` / `complete` with the remaining accepted tasks. ```text requested ── discovery waves ──► selected (exact budget) │ ▼ author once each │ ┌─────────────────┼─────────────────┐ ▼ ▼ ▼ accepted rejected infrastructure_failed │ ▼ buildExport(accepted only) ``` ## Eight-shard discovery Each wave starts eight `discoverCandidateShard` activities in parallel. Provenance JSONL is partitioned by message index: `index % 8 === shardIndex`. An empty shard is valid. The activity prompt asks for at most the per-shard targets and forbids already-seen `sourcePr` values. Per-tier shard target: ```text 0, if missing[tier] == 0 min(8, ceil(missing[tier] / 8) + 3), otherwise ``` Example: budget `{ easy: 1, medium: 1, hard: 0 }` requests `{ easy: 4, medium: 4, hard: 0 }` on wave 0. After easy is filled, wave 1 requests `{ easy: 0, medium: 4, hard: 0 }`. After the eight activities return, the workflow **interleaves** shard lists (index 0 from every shard, then index 1, …) and drops duplicate `sourcePr` values. Later waves pass every already-kept PR in `excludedSourcePrs`. | Constant | Value | Effect | | --- | --- | --- | | Shard count | 8 | Fixed `totalShards`. | | Overfetch | +3 | Inflates per-shard targets so a later reject does not require a new wave for ranking — it still does not replace a rejected authored candidate. | | Max per tier per shard | 8 | Hard cap on a shard plan. | | Max discovered | 300 | Hitting the cap while a tier is still missing sets `blocked`. | | Wave index | 0-based | Artifact prefix `runs//discovery/wave-/shard-`. | | Candidate ID prefix | `w{wave}s{shardIndex}-` | Applied when the plan is materialized. | A validated shard writes `plan.json` and reuses that checkpoint on later attempts. Retryable shard failures increment `failedShards` and contribute an empty list. A non-retryable shard failure (`InvalidDiscoveryShard`, or a wrapped `ApplicationFailure` with `nonRetryable`) fails the whole run. A wave that adds zero unique PRs while any budget remains sets `phase` to `blocked` and throws: ```text candidate pool exhausted; missing easy=N, medium=N, hard=N ``` with type `CandidatePoolExhausted`. ## Per-candidate statuses Selected candidates enter `tasks[]` with `taskId` initially equal to `candidateId`. After a successful author, `taskId` becomes the authored Harbor task ID. Duplicate authored task IDs are rejected (`authoring repeated task ID …`). ```mermaid stateDiagram-v2 [*] --> authoring authoring --> rejected: author outcome rejected or duplicate taskId authoring --> auditing: authored auditing --> rejected: static audit failed auditing --> validating: audit accepted validating --> repairing: Harbor nop/oracle failed validating --> reviewing: validation accepted repairing --> rejected: repair rejected or exhausted repairing --> auditing: repaired bundle reviewing --> repairing: coupling review failed reviewing --> accepted: review accepted authoring --> infrastructure_failed: HarborInfrastructureFailure validating --> infrastructure_failed: HarborInfrastructureFailure ``` | `tasks[].status` | When it is set | | --- | --- | | `authoring` | Candidate selected; `authorCandidate` running or checkpointed. | | `auditing` | Static audit of the authored (or repaired) definition. | | `validating` | Harbor nop, oracle, and determinism rerun. | | `reviewing` | Independent coupling review. | | `repairing` | One validation-repair activity, or one coupling-repair activity. | | `accepted` | All gates passed. Included in the export. | | `rejected` | Gate failed, repair declined, or repair attempts exhausted. Sibling candidates continue. | | `infrastructure_failed` | `HarborInfrastructureFailure` (Docker/Modal/Harbor transport). Sibling candidates continue. Not replaced. | Validation repair is a single activity attempt (`retry.maximumAttempts: 1`, 2-hour start-to-close). Exhaustion records `validation repair failed after its single activity attempt` and rejects **only that candidate**. Coupling repair uses the shared task-activity retry policy (4 attempts); exhaustion records `test repair failed after activity retries` and likewise rejects only that candidate. A Temporal cancel during discovery or during either repair sets the **run** to `cancelled`. ## Authoring through export Once every tier budget is filled, `phase` becomes `authoring` and the selected set is processed concurrently (worker concurrency still applies). Each candidate: 1. `authorCandidate` — sandbox author, or reuse `definition.json` + `harbor-task.tar.gz` checkpoints. 2. `auditTask` — static definition audit. 3. `validateTask` — Harbor nop / oracle / determinism. 4. On validation failure: `repairValidationTask` once, then audit + validate again. 5. `reviewTask` — coupling review. 6. On review failure: `repairTask` once, then audit + validate + review again. 7. Mark `accepted` or `rejected`. Then `phase` becomes `exporting`. `buildExport` always runs, even when `acceptedTaskIds` is empty. The archive is stored at `runs//export/attempt-/selfbench-.tar.gz` and attached to status as `export`. :::endpoint GET /v1/runs/:runId/export Download the completed archive Returns `409` `{ "error": "run export is not ready" }` until `export` is present. A ready response includes `x-content-sha256`, `content-type`, `content-length`, and `content-disposition: attachment; filename="selfbench-.tar.gz"`. ::: `self-bench download` writes create-only (`wx`), verifies the streamed SHA-256 against that header, and deletes the file on mismatch. ## Observe, wait, and cancel ```bash self-bench status RUN_ID self-bench list self-bench cancel RUN_ID self-bench download RUN_ID ./self-bench-tasks.tar.gz ``` `self-bench status RUN_ID` prints the query payload. Use `phase`, `discovery`, and `tasks[].reason` while the run is live. `self-bench run --wait` or `--output` polls every 2s and prints a stderr line only when `phase` changes. It returns the final status on `complete` and throws on `blocked`, `failed`, or `cancelled`. `POST /v1/runs/:runId/cancel` calls Temporal `cancel()` and returns `202` `{ "runId", "cancellationRequested": true }`. Activities use `WAIT_CANCELLATION_COMPLETED`. The run is not cancelled until the workflow observes the signal and sets `phase` to `cancelled`. `self-bench download` against a live, blocked, failed, or cancelled run surfaces the API `409` as `run export is not ready`. ## Timeouts and retries | Surface | Start-to-close | Heartbeat | Attempts | Notes | | --- | --- | --- | --- | --- | | Workflow | 14 days | — | 1 execution | `workflowId` = run ID. | | `discoverCandidateShard` | 1 hour | 10 minutes | 3 | Sandbox deadline 45 minutes; 8-minute inactivity. Retryable exhaustion becomes a failed shard, not a failed run. | | Author / validate / review / audit / coupling repair / export | 7 hours | 10 minutes | 4 | Author sandbox 2 hours; review inactivity 5 minutes; author/discovery inactivity 8 minutes. | | `repairValidationTask` | 2 hours | 10 minutes | 1 | Candidate rejected after that single attempt. | Backoff for retried activities: 5 seconds initial, coefficient 2, max 2 minutes (discovery max 1 minute). ## Terminals | Terminal | Typical cause | Export | Operator action | | --- | --- | --- | --- | | `complete` | Authoring finished; `buildExport` stored an archive. Accepted count may be 0. | Yes | `self-bench download` or `--output`. Inspect `tasks` for rejects. | | `blocked` | A discovery wave added no unique PRs, or `discovered` reached 300 with a tier still missing. | No | Widen provenance, lower a tier budget, submit a **new** `runId`. Read `discovery` and the Temporal failure message `candidate pool exhausted; missing easy=…`. | | `failed` | Non-retryable discovery input, uncaught activity error, or Temporal `FAILED` / `TERMINATED` / `TIMED_OUT`. | No | Read `error` (or the Temporal close reason if the query fell back). Fix credentials/backend and submit a new `runId`. | | `cancelled` | `self-bench cancel` / `POST …/cancel` observed by the workflow. | No | Submit a new `runId` if you still want an export. | `HarborInfrastructureFailure` does **not** fail the run. The candidate is marked `infrastructure_failed` with the Harbor/Modal/Docker message, siblings continue, and the run can still `complete`. Duplicate `POST /v1/runs` with the same `runId` is a Temporal already-started error (API `500`), not a resume. There is no delete route for runs or artifacts. ## Errors to expect | Signal | Meaning | | --- | --- | | `SelfBench run blocked` | Wait saw `phase: "blocked"`. Detail is present only if `error` is a string. | | `SelfBench run failed: …` | Wait saw `failed` plus `error`. | | `SelfBench run cancelled` | Wait saw `cancelled`. | | `run export is not ready` | Download or `GET …/export` before `complete`. | | `candidate pool exhausted; missing easy=…` | Workflow `CandidatePoolExhausted`. | | `validation repair failed after its single activity attempt` | That candidate only. | | `test repair failed after activity retries` | That candidate only. | | `SelfBench status response is missing its phase` | Status JSON is not a run payload. | | `downloaded export failed its SHA-256 integrity check` | Stream did not match `x-content-sha256`; output file removed. | ## Related pages Submit `--repo`, tier budgets, `--run-id`, and `--output`. `status`, `list`, `cancel`, and `download` against `SELFBENCH_API_URL`. Audit, Harbor nop/oracle, one validation repair, one coupling repair, then reject. What discovery and authoring require for easy, medium, and hard. Empty provenance, blocked pools, `409` export-not-ready, and Harbor infrastructure failures. `manifest.json` and per-task Harbor archives produced in `exporting`. --- ## 07. Validation and repair > Static audit, Harbor nop and oracle gates, determinism rerun, one validation-repair attempt, one coupling-repair retry, then reject. - Page Markdown: https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc/pages/07-validation-and-repair.md - Generated: 2026-08-15T02:04:38.219Z ### Source Files - `src/temporal/workflow.ts` - `src/temporal/activities.ts` - `src/audit.ts` - `src/validation-repair.ts` - `src/repair.ts` - `docs/task-construction.md` --- title: "Validation and repair" description: "Static audit, Harbor nop and oracle gates, determinism rerun, one validation-repair attempt, one coupling-repair retry, then reject." --- After authoring, `selfBenchRunWorkflow` accepts a candidate only when static audit, Harbor `nop` and `oracle` gates, and coupling review all pass. A Harbor gate failure schedules **one** `repairValidationTask` activity. A coupled review schedules **one** `repairTask` cycle. The next failure at the same gate rejects that candidate and does not replace it. The run phase stays `authoring` while each candidate moves through per-task statuses `auditing` → `validating` → `reviewing` → `accepted`. Both repair kinds set status `repairing`. Harbor infrastructure errors set `infrastructure_failed` instead of entering repair. ## Candidate pipeline ```mermaid stateDiagram-v2 [*] --> auditing: authored auditing --> rejected: static blockers auditing --> validating: audit.accepted validating --> infrastructure_failed: HarborInfrastructureFailure validating --> validation_repair: first nop/oracle failure validating --> rejected: Harbor failure after a repair validating --> reviewing: nop and oracle passed validation_repair --> rejected: activity threw or AuthorOutcome rejected validation_repair --> auditing: harness rewritten reviewing --> coupling_repair: first coupled verdict reviewing --> rejected: coupled after test repair reviewing --> accepted: clean coupling_repair --> rejected: activity retries exhausted or AuthorOutcome rejected coupling_repair --> auditing: tests rewritten ``` A candidate can receive both repairs in order: failed Harbor gates first, then failed coupling review. After `repairTask`, a new Harbor failure does **not** schedule `repairValidationTask`. After either repair, static audit runs again before Harbor. | Gate | Activity | Temporal `startToCloseTimeout` | Temporal `retry.maximumAttempts` | Workflow retries | | --- | --- | --- | --- | --- | | Static audit | `auditTask` | 7 hours | 4 | After each successful repair | | Harbor `nop` + `oracle` | `validateTask` | 7 hours | 4 | Once after validation repair; once after coupling repair | | Validation repair | `repairValidationTask` | 2 hours | **1** | None | | Coupling review | `reviewTask` | 7 hours | 4 | Once after coupling repair | | Coupling repair | `repairTask` | 7 hours | 4 | One scheduled cycle only | `repairValidationTask` is the only activity pinned to a single attempt. `repairTask` may Temporal-retry the sandbox, but the workflow still schedules that cycle only once. Cancellation on either repair activity cancels the run. ## Static audit `auditTask` reads `definition.json`, `solution/gold.patch`, and `tests/test.patch`, then calls `auditTaskDefinition`. The report is stored at `runs//audits//.json`. Rejection reason is the blockers joined with `"; "`. | Check | Rejects when | | --- | --- | | Patch overlap | Any path appears in both gold and held-out test patches | | Implementation files | Gold file count below the tier minimum | | Implementation lines | Gold `+`/`-` lines (excluding `+++`/`---`) below the tier minimum | | Held-out tests | Test patch changes no files | | Pass-to-pass count | `definition.passToPass.length` below the tier minimum | | Hard-coded selectors | `testCommand` with `{tests}` removed still contains a `failToPass` or `passToPass` path | | `{tests}` count | `testCommand` does not contain `{tests}` exactly once | | `{tests}` quoting | `testCommand` matches `(["'])\{tests\}\1` | Tier minima used by audit: | `difficulty` | Gold files | Gold changed lines | `passToPass` length | | --- | --- | --- | --- | | `easy` | 1 | 20 | 0 | | `medium` | 2 | 50 | 1 | | `hard` | 3 | 100 | 2 | `failToPass` must contain at least one selector because `taskDefinitionSchema` requires `.min(1)`. Audit does not re-check that. A passing easy audit with an empty `passToPass` list is valid; the Harbor verifier still needs a `pass_to_pass` reward of `1` (see below). Returned metrics: `implementationFiles`, `implementationChangedLines`, `testFiles`. ## Harbor nop and oracle `validateTask` extracts the Harbor bundle and runs two sequential `harbor run` jobs against `SELFBENCH_HARBOR_ENVIRONMENT`, falling back to `SELFBENCH_EXECUTION_BACKEND` (`docker` by default): ```bash harbor run \ --path \ --agent nop|oracle \ --env docker|modal \ --job-name --<8 hex> \ --jobs-dir /jobs \ --delete --yes --quiet ``` Each job has a 3-hour command timeout. Rewards come from `trial.verifier_result.rewards`. Any `trial.exception_info` fails the gate. Matching infrastructure exceptions (`AuthError` or known Docker/Modal/build failures) throw `HarborInfrastructureFailure` instead of becoming a validation failure. | Agent | Must hold | Meaning | | --- | --- | --- | | `nop` | `fail_to_pass === 0` | Held-out fail-to-pass tests fail on the base snapshot | | `nop` | `pass_to_pass >= 1` | Regression command succeeds (or `true` when `passToPass` is empty) | | `nop` | `setup_completed >= 1` | Verifier reached the test stage | | `oracle` | `patch_applied >= 1` | Gold / agent patch applied | | `oracle` | `fail_to_pass >= 1` | Fail-to-pass tests pass after gold | | `oracle` | `pass_to_pass >= 1` | Regression command succeeds | | `oracle` | `deterministic >= 1` | Fail-to-pass command passed a second time | | `oracle` | `setup_completed >= 1` | Verifier reached the test stage | Both agents must pass. Failure reason format: ```text Harbor gates failed: nop= (); oracle= () nop verifier tail: ... oracle verifier tail: ... ``` Checks logged: `patch_applied`, `fail_to_pass`, `pass_to_pass`, `deterministic`, `setup_completed`, `fail_to_pass_exit_code`, `fail_to_pass_repeat_exit_code`, `pass_to_pass_exit_code`. Verifier tails are truncated to the last 8 KB. Missing numeric rewards count as `0`. The compiled `tests/test.sh` applies an empty `nop` agent patch as a no-op (`-s` requires a non-empty `/opt/selfbench/agent.patch`), reapplies `tests/test.patch`, then runs fail-to-pass and pass-to-pass. An empty `passToPass` list compiles to the command `true`, so `pass_to_pass` can still be `1` on easy tasks. Each verifier command may retry up to three times on registry/network log patterns (`ECONNRESET`, `ETIMEDOUT`, `registry.npmjs`, and similar) with 10s / 20s backoff. Assertion failures fail immediately. A Harbor process non-zero exit or a classified infrastructure exception does not enter validation repair. The candidate status becomes `infrastructure_failed`. Sibling candidates continue; the run can still export whatever was accepted. ## Determinism rerun Determinism is not a third Harbor job. During the **oracle** trial, `tests/test.sh` runs the fail-to-pass command twice: 1. First success sets `fail_to_pass=1` and `fail_to_pass_exit_code=0`. 2. Immediate second success sets `deterministic=1` and `fail_to_pass_repeat_exit_code=0`. 3. Second failure leaves `deterministic=0` and records `fail_to_pass_repeat_exit_code`. `validateTask` requires `deterministic >= 1` on oracle only. `nop` does not check that reward. Flaky fail-to-pass tests therefore fail oracle and, on the first failure, enter validation repair with those diagnostics. ## Validation repair On the first `validation.accepted === false`, status becomes `repairing` and the workflow calls `repairValidationTask` with the failed `ValidationResult`. ### Sandbox The activity mounts the current Harbor bundle plus the original definition and the validation reason, then runs `dist/sandbox-validation-repair.bundle.js`. | Input | Path | | --- | --- | | Task archive | `/work/task.tar.gz` | | Original definition | `/work/original-definition.json` | | Diagnostics | `/work/validation-diagnostics.txt` (the Harbor reason, including bounded verifier tails) | | Repairer | `/work/sandbox-validation-repair.js` | | Output | Path | | --- | --- | | New definition | `/work/repaired-definition.json` | | New held-out patch | `/work/repaired-test.patch` | | Report | `/work/repair-report.json` | Sandbox wall clock is 2 hours. Agent inactivity timeout is 8 minutes. Inside the sandbox, `pi --print` runs for up to 90 minutes with `--tools read,bash,grep,find,ls`, `--thinking high`, and `--no-session`. Provider is `openai` when `OPENAI_API_KEY` is set, otherwise `openai-codex` with `SELFBENCH_PI_AUTH_JSON`. Model is `SELFBENCH_REPAIR_MODEL`, set from `run.authoring.model`. The working tree is the base snapshot from `tests/repo.tar.gz` with the current `tests/test.patch` already applied. `/work/gold.patch` and `/work/test.sh` are present for inspection only. ### Allowed edits `assertValidationRepair` enforces: **Immutable `definition.json` fields:** `schemaVersion`, `difficulty`, `taskId`, `repo`, `baseCommit`, `workdir`, `sourcePr`, `sourceUrl`, `prompt`. **Mutable `definition.json` fields** (named in the repair prompt): `setupCommand`, `testCommand`, `failToPass`, `passToPass`, `testPaths`, `toolchains`, `timeouts`, `resources`. **File paths:** every changed path must already appear in the original held-out test patch, and that original patch must be test-only. Test-only means a path segment matching `tests?`, `__tests__`, `e2e`, `integration`, `fixtures?`, `selfbench`, `selfbench-tests`, `selfbench_tests`, or `.selfbench-tests`, or a filename matching `(^|\.)(test|spec)\.[^.]+$`. Application code, `instruction.md`, and `solution/gold.patch` stay unchanged. Selecting additional existing pass-to-pass identifiers without editing those files is allowed. Leaving both the definition and the test patch unchanged fails the sandbox (`validation repair left the task unchanged`). ### After a successful repair The activity writes the new `tests/test.patch`, calls `refreshHarborTask` (rewrites `tests/test.sh`, `task.toml`, both Dockerfiles, and `dependency-setup.patch`), and stores a checkpoint: ```text runs//validation-repairs/// ├── definition.json ├── harbor-task.tar.gz └── attempt-/ ├── sandbox.log └── report.json ``` If both checkpoint objects already exist, the activity returns them without launching another model. An incomplete pair (definition xor bundle) throws. The workflow then re-runs `auditTask` and `validateTask`. A second Harbor failure rejects with `validation.reason` or `"validation rejected repaired harness"`. There is no second validation-repair. If the activity throws (other than cancellation), the candidate is rejected with `"validation repair failed after its single activity attempt"`. ## Coupling review and test repair ### Review `reviewTask` scans held-out tests for contract artifacts, greps the base snapshot, and runs `dist/sandbox-review.bundle.js` (15-minute sandbox, 5-minute inactivity timeout). The reviewer model constant is `gpt-5.6-sol`. Auth is `OPENAI_API_KEY` or `SELFBENCH_PI_AUTH_JSON`. `resolveCouplingReview` marks the task `coupled` when any of these hold: - the model verdict is `coupled` - a gold-introduced artifact is missing from the review findings - any finding has disposition `gold_only` `accepted` is `resolution.verdict === "clean"`. Reports land at `runs//reviews///attempt-.json`. Gold-only contract artifacts, what the Harbor trial mounts, and the held-out-test-only constraint are specified on [Agent-visible boundary](/agent-visible-boundary). ### Test repair On the first coupled verdict, status becomes `repairing` and the workflow calls `repairTask` with the review artifact. The sandbox runs `codex exec` (`--ephemeral`, `--dangerously-bypass-approvals-and-sandbox`) for up to 90 minutes inside a 2-hour / 8-minute-inactivity sandbox. Auth is `OPENAI_API_KEY` or `SELFBENCH_CODEX_AUTH_JSON`. Model is again `run.authoring.model`. `assertRepairPaths` allows edits only to files already listed in the original `tests/test.patch`. Unlike validation repair, this step cannot change `definition.json`. The sandbox must produce a **different** non-empty Git test patch. Successful output replaces only the Harbor bundle. Checkpoint: ```text runs//repairs// ├── harbor-task.tar.gz └── attempt-/ ├── sandbox.log └── report.json ``` An existing `harbor-task.tar.gz` checkpoint is reused. Manifest metadata records `testPatchSha256` plus `repair.model` and `repair.originalTestPatchSha256`. The workflow then repeats **audit → validate → review**. Any failure rejects: | Second-gate failure | Default reason | | --- | --- | | Audit | `"audit rejected repaired task"` | | Harbor | `"validation rejected repaired task"` | | Review | `"review rejected repaired task"` | | `repairTask` throw after Temporal retries | `"test repair failed after activity retries"` | ## Per-candidate terminals | `TaskProgress.status` | When | | --- | --- | | `rejected` | Authoring, audit, a second Harbor/review failure, or a repair that returns `kind: "rejected"` / throws | | `infrastructure_failed` | `HarborInfrastructureFailure` from `validateTask` after Temporal retries | | `accepted` | Audit, both Harbor agents, and a `clean` review all passed | Rejected and infrastructure-failed candidates are not replaced from the unused discovery pool. The run still exports accepted tasks. `true` only when both Harbor agents pass and the trial has no `exception_info`. `passed`, `result` artifact (`nop.json`), optional `output` artifact (`nop-verifier.log`). `passed`, `result` artifact (`oracle.json`), optional `output` artifact (`oracle-verifier.log`). Present when either agent failed. Passed to the validation-repair prompt as `/work/validation-diagnostics.txt`. Validation artifacts: ```text runs//validation///attempt-/ ├── nop.json ├── oracle.json ├── nop-verifier.log # when verifier output exists └── oracle-verifier.log ``` ## Offline companion commands The Temporal loop is the only path that sequences audit → Harbor → one validation repair → one coupling repair. Companion binaries operate on **already expanded** Harbor task directories. Run Harbor `nop` then `oracle` only. No static audit, no determinism job beyond the oracle reward, no repair. Directory of expanded Harbor tasks (each must contain `task.toml`). Harbor jobs directory. Also receives `validation-summary.json`. Harbor `--env`. Default: `modal` (not the worker's `SELFBENCH_HARBOR_ENVIRONMENT` default). Parallel tasks. Default: `10`. Each Harbor invocation adds `--n-concurrent 1 --max-retries 1` and a 4-hour timeout. ```bash self-bench-validate --tasks ./expanded --jobs ./harbor-jobs --environment modal ``` ```json { "total": 3, "accepted": 2, "rejected": 1, "errors": 0 } ``` A task is accepted when the same reward predicates as `validateTask` hold. Process failures become `status: "error"` entries, not Harbor infrastructure task statuses. Sandboxed coupling review only. Writes a report with `provisional: true`. ```bash self-bench-reaudit --tasks ./expanded --output ./coupling-review.json [--task TASK_ID] [--concurrency 10] ``` Offline **coupling** repair for tasks whose review verdict is `coupled`. Not validation repair. `--output` must be outside `--tasks`. Uncoupled tasks are copied unchanged unless `--task` is set. ```bash self-bench-repair \ --tasks ./expanded \ --review ./coupling-review.json \ --output ./repaired \ [--task TASK_ID] \ [--concurrency 9] \ [--model gpt-5.6-sol] ``` Re-run `self-bench-validate` and `self-bench-reaudit` on `./repaired` yourself. The companion does not repeat audit or Harbor. Full flags live on [Companion commands](/companion-commands). ## Troubleshooting | Symptom | Likely cause | | --- | --- | | Audit: gold/test overlap | Author included implementation paths in `test.patch` | | Audit: `{tests}` quoted or duplicated | `testCommand` must expand `{tests}` once as an unquoted argv list | | `nop` `fail_to_pass !== 0` | New tests already pass on the base snapshot | | `oracle` `fail_to_pass === 0` | Gold does not satisfy the selected fail-to-pass tests, or setup never reached them | | `oracle` `deterministic === 0` | Fail-to-pass is flaky; check `fail_to_pass_repeat_exit_code` and `oracle-verifier.log` | | `pass_to_pass === 0` | Listed regressions failed; an empty list should still yield `1` via `true` | | `setup_completed === 0` / `patch_applied === 0` | Agent patch missing or `git apply` failed (including test-patch reapply) | | `infrastructure_failed` | Harbor CLI crash, Docker/Modal/auth/build exception — not a harness bug | | Validation repair: "left the task unchanged" | Model did not edit allowed tests or mutable definition fields | | Validation repair: "test-only patch paths" | Original `test.patch` already touched product files | | Test repair: "outside the held-out tests" | Model created a new file instead of editing existing test-patch paths | | `"validation rejected repaired harness"` | The single validation-repair attempt did not produce a passing nop/oracle split | | `"review rejected repaired task"` | The single coupling-repair cycle is still `coupled` | `self-bench-validate --environment` defaults to `modal`. A worker using Docker Harbor (`SELFBENCH_EXECUTION_BACKEND=docker` without `SELFBENCH_HARBOR_ENVIRONMENT`) will not match that default. ## Related pages Tier file/line minima and which blockers are static versus Harbor-proven. What a trial mounts, gold-only artifacts, and why repair may edit only held-out tests. How `test.sh` encodes nop, oracle, and the fail-to-pass rerun. Run phases versus per-candidate statuses, including `rejected` and `infrastructure_failed`. `self-bench-validate`, `self-bench-reaudit`, and `self-bench-repair` flags. Export, Harbor infrastructure, and leftover Docker resources. --- ## 08. Agent-visible boundary > What a Harbor trial mounts, what stays hidden, gold-only contract artifacts, coupling verdicts, and the held-out-test-only repair constraint. - Page Markdown: https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc/pages/08-agent-visible-boundary.md - Generated: 2026-08-15T02:04:41.799Z ### Source Files - `src/coupling.ts` - `src/codex-review.ts` - `src/skills/selfbench/SKILL.md` - `src/harbor-task.ts` - `docs/task-construction.md` - `tests/coupling.test.ts` --- title: "Agent-visible boundary" description: "What a Harbor trial mounts, what stays hidden, gold-only contract artifacts, coupling verdicts, and the held-out-test-only repair constraint." --- A compiled Harbor task (`schema_version = "1.4"`, `compiler_revision = 23`) splits the workspace into an agent image, a separate verifier image, and a `solution/` tree that Harbor mounts only for the `oracle` agent. A coding-agent trial (`harbor run --agent codex`, or any adapter other than `oracle`) receives a history-free `git archive` of `baseCommit` plus `instruction.md`. It does not receive `tests/test.patch`, `tests/test.sh`, or `solution/gold.patch`. The run export still contains snapshots, held-out tests, and the reference patch. Those files are operator-visible and unencrypted. They stay hidden from coding-agent trials, not from anyone who can read the archive. ## Trial mounts `compileHarborTask` writes this Harbor bundle. `environment/Dockerfile` never copies `gold.patch`. `tests/Dockerfile` copies only verifier inputs. `task.toml` sets `[verifier] environment_mode = "separate"`. :::files harbor-task/ ├── task.toml ├── instruction.md ├── .selfbench-manifest.json ├── environment/ │ ├── Dockerfile │ └── repo.tar.gz ├── tests/ │ ├── Dockerfile │ ├── repo.tar.gz │ ├── test.patch │ ├── test.sh │ └── dependency-setup.patch # only when gold changes a recognized lockfile └── solution/ ├── gold.patch └── solve.sh ::: ```mermaid flowchart TB subgraph bundle ["harbor-task/"] instruction["instruction.md = definition.prompt"] envDir["environment/repo.tar.gz + Dockerfile"] testsDir["tests/test.patch + test.sh + Dockerfile"] solutionDir["solution/gold.patch + solve.sh"] toml["task.toml"] end subgraph agentTrial ["Coding-agent trial --agent other than oracle"] agentUser["USER agent, /app"] agentSnap["history-free base + setupCommand"] agentInst["instruction.md"] agentNet["network allowlist: chatgpt.com, openai.com"] agentOut["/opt/selfbench/agent.patch"] end subgraph verifierEnv ["Verifier environment_mode = separate"] verRoot["user = root"] verTests["/tests/test.patch mode 600"] verDep["dependency-setup.patch then git reset"] verReward["/logs/verifier/reward.json"] end subgraph oracleOnly ["harbor run --agent oracle"] solve["solve.sh applies /solution/gold.patch"] end instruction --> agentInst envDir --> agentSnap envDir --> verRoot toml --> agentUser toml --> agentNet toml --> agentOut testsDir --> verTests testsDir --> verDep testsDir --> verReward solutionDir --> solve ``` | Surface | Who sees it | Role | | --- | --- | --- | | `instruction.md` | Coding agent, nop, oracle | Standalone engineer request. Compiler writes `definition.prompt` only. | | `environment/` image | Coding agent, nop | `repo.tar.gz` at `baseCommit`, fresh `git init`, one `base` commit, then `setupCommand`. | | `tests/` image | Verifier only | Applies held-out tests after collecting `agent.patch`. | | `solution/` | `oracle` only | `solve.sh` runs `git apply /solution/gold.patch`. Never mounted for coding-agent trials. | | `task.toml` `[metadata]` | Harbor / operators | `selfbench_task_id`, `difficulty`, `repo`, `base_commit`, `workdir`, `source_pr`, `compiler_revision`. Not copied into `/app` by `environment/Dockerfile`. | | `.selfbench-manifest.json` | Operators | SHA-256 of definition, test patch, and gold patch. | Validation uses Harbor's built-in agents, not a coding adapter: ```bash harbor run --path ./harbor-task --agent nop --env modal --yes harbor run --path ./harbor-task --agent oracle --env modal --yes ``` A later eval trial uses a coding adapter (`codex` in `self-bench-eval`) against the same bundle. That trial still uses the agent image and `instruction.md` only. ## Agent-visible workspace The agent image: 1. Extracts `environment/repo.tar.gz` into `/app` (no Git history). 2. Creates a new repo and commits `base`. 3. Runs `setupCommand` from `definition.workdir`. 4. Copies `/app/.git` to root-owned `/opt/selfbench/base.git` (`chmod 700`). 5. Switches to `USER agent`. `task.toml` then constrains the running agent: | Field | Value | | --- | --- | | `[agent].user` | `agent` | | `[agent].timeout_sec` | `timeouts.agentSeconds` | | `[agent].network_mode` | `allowlist` | | `[agent].allowed_hosts` | `chatgpt.com`, `*.chatgpt.com`, `openai.com`, `*.openai.com` | | `artifacts` | `/opt/selfbench/agent.patch` | Harbor collects that artifact with the pinned `base.git`: ```bash git --git-dir=/opt/selfbench/base.git --work-tree=/app add -A git --git-dir=/opt/selfbench/base.git --work-tree=/app diff --cached --binary HEAD > /opt/selfbench/agent.patch ``` Image build and verifier networks are `public`. Only the coding-agent runtime uses the OpenAI/ChatGPT allowlist. ## What stays hidden | Hidden from the coding agent | Where it lives | Why | | --- | --- | --- | | Held-out tests | `tests/test.patch` | Grader, not part of the request. | | Verifier script and selectors | `tests/test.sh` | Expands `{tests}` into `failToPass` / `passToPass`. | | Reference implementation | `solution/gold.patch` | Oracle only. | | Gold lockfile preinstall | `tests/dependency-setup.patch` | Verifier image only; source is reset to `HEAD` after setup. | | Provenance / session records | Authoring sandbox `/work/provenance.json` | Never compiled into the Harbor task. | | Author and reviewer transcripts | Temporal artifacts under `runs/.../reviews` and `.../repairs` | Excluded from the export. | | Completed commit and full Git history | Source checkout during authoring | Agent snapshot is `git archive` of `baseCommit`. | `instruction.md` must not name pull-request numbers or URLs, commit hashes, branch names, test names, test paths, test commands, gold-patch content, or held-out-test content. `definition.json` still stores `sourcePr`, `sourceUrl`, and `baseCommit` for operators; those fields are not copied into `instruction.md`. If gold changes a recognized lockfile (`package.json`, `bun.lock`, `uv.lock`, `go.mod`, `Cargo.lock`, and the other names in `dependencyManifestPatch`), the verifier applies that subset, reruns `setupCommand`, then `git reset --hard` and deletes the patch. That keeps oracle dependencies honest without putting gold source into the agent image. ## Held-out test protection at verify time `tests/test.sh` grades `agent.patch`, not the gold patch: 1. Apply `/opt/selfbench/agent.patch` with `--exclude` for every path in `testPaths` and every path in `test.patch`. 2. `git restore` and `git clean` those paths back to `HEAD`. 3. Apply `/tests/test.patch`, then `chown root` and `chmod a-w` on each held-out path. 4. Delete `/tests/test.patch`. 5. Run fail-to-pass, repeat fail-to-pass (determinism), then pass-to-pass as user `verifier`. Reward is 1 only when `patch_applied`, `fail_to_pass`, `pass_to_pass`, and `deterministic` are all 1. Validation interprets the same JSON as: | Gate | Required `reward.json` | | --- | --- | | `nop` | `fail_to_pass === 0`, `pass_to_pass >= 1`, `setup_completed >= 1` | | `oracle` | `patch_applied`, `fail_to_pass`, `pass_to_pass`, `deterministic`, and `setup_completed` all `>= 1` | An agent that overwrites a held-out test file cannot keep that change. The verifier restores the path and re-applies the hidden patch. ## Gold-only contract artifacts After nop/oracle pass, `reviewTask` scans added lines in `tests/test.patch` and keeps only artifacts that also appear in gold additions. `CouplingEvidence` version written into the review report. Deterministic classifier. Quoted `/path` tokens are `endpoint_path`. `type/subtype` tokens are `media_type`. Header-like tokens on a `headers` / `content-type` line are `header_name`. Quoted object keys, bare `key:`, and `.member` names of length 3–64 are `field_name`. Always `true` in emitted evidence. Candidates that never appear in gold additions are dropped. Verbatim `includes` for non-fields. `field_name` uses a word-boundary match so `id` does not hit `valid`. Exact string hit from `rg --fixed-strings` over the extracted `environment/repo.tar.gz` (`--hidden`, `--glob !.git/**`). Every artifact with `!presentInPrompt && !presentInBase`. Message shape: `held-out tests assert gold-only "" at :patch-line-N`. Fixture literals that never appear in gold additions are ignored. `expect(status).toBe(200)` and `"org_test"` do not become artifacts unless gold introduced them. ```text added test token | v present in gold additions? no --> ignore yes --> in prompt or base snapshot? yes --> not a blocker no --> couplingEvidence.blockers[] ``` The scanner does not invent helper names, SQL, or UI copy that are not quoted or dotted as above. The independent reviewer still has to judge those as `other`. ## Coupling verdicts `reviewCouplingWithCodex` forces one `submit_review` call on `gpt-5.6-sol` (`COUPLING_REVIEW_MODEL`). Auth is `OPENAI_API_KEY`, or `SELFBENCH_PI_AUTH_JSON` / Pi `openai-codex` access when no API key is set. Reviewer claim. `resolveCouplingReview` may override `clean` to `coupled`. `base_contract`, `prompt_contract`, `external_contract`, `gold_only`, or `not_contract`. Same four scanner categories plus `other`. Required. Must describe an alternative correct implementation and whether the tests would accept it. `resolveCouplingReview` fails closed: | Condition | Forced verdict | `reason` | | --- | --- | --- | | A blocker artifact has no finding with the exact same `artifact` string | `coupled` | `review did not resolve deterministic coupling evidence for: …` | | Any finding has `disposition: "gold_only"` | `coupled` | `review identified gold-only artifacts: …` if the reviewer still said `clean` | | Reviewer `verdict` is `coupled` | `coupled` | Reviewer `reason` | | All blockers resolved and no `gold_only` | `clean` | Reviewer `reason` | Allowed justifications for a blocker: | Disposition | When it is valid | | --- | --- | | `prompt_contract` | The authentic request text requires that exact artifact. | | `base_contract` | The extracted base snapshot already establishes it. A gold mention is not evidence. | | `external_contract` | The request names a standard protocol that fixes the token independently of gold. | | `not_contract` | Incidental test-language or framework syntax, not a product contract. | | `gold_only` | Always `coupled`. | `reviewTask` writes `runs//reviews///attempt-.json` with the resolved verdict, `couplingEvidence`, and `reviewer: "gpt-5.6-sol"`. `accepted` is true only when the resolved verdict is `clean`. Re-run the same gate on an extracted export: ```bash self-bench-reaudit --tasks ./expanded-tasks --output ./coupling-review.json ``` The report is `provisional: true` and counts `clean`, `coupled`, and `errors`. A test asserts `routing_strategy` and gold added that field. The prompt says only “use the resolved strategy”. `baseArtifacts` is empty, so the blocker is `field_name "routing_strategy"`. - Findings `[]` and `verdict: "clean"` → `coupled`, `missingArtifacts: ["routing_strategy"]`. - Finding `{ artifact: "routing_strategy", disposition: "gold_only" }` → `coupled`. - Finding `{ artifact: "routing_strategy", disposition: "external_contract" }` with `verdict: "clean"` → `clean` (only if the request actually names a protocol that fixes the field). ## Held-out-test-only repair A `coupled` review schedules **one** `repairTask`. After repair the candidate repeats `auditTask` → `validateTask` → `reviewTask`. A second coupled verdict rejects the candidate. There is no second coupling-repair cycle. `repairTask` may Temporal-retry the sandbox up to `maximumAttempts: 4`. Exhaustion rejects with `test repair failed after activity retries`. That is infrastructure retry, not a second design pass. ### Coupling repair The sandbox unpacks the Harbor task, applies the **current** `tests/test.patch` to the base snapshot, and runs Codex against `repairPrompt`. Allowed edits: paths already present in the original `test.patch` (`assertRepairPaths`). Forbidden: - Application / gold paths - New files outside that path set - `instruction.md` / `definition.prompt` - `solution/gold.patch` Hard failures: | Error | Cause | | --- | --- | | `repair changed files outside the held-out tests: ` | Working tree touched a path not in the original test patch. | | `repair produced no held-out test patch` | `git diff HEAD` is not a Git patch. | | `repair left the held-out test patch unchanged` | Bytes equal the input `test.patch`. | | `original held-out test patch changes no files` | Empty original patch. | The prompt forbids deleting assertions just to silence the report. If no uncoupled public seam exists inside the allowed files, the agent must leave the tests in place and explain the blocker — that path then fails the unchanged-patch check and the candidate is rejected. Default model: `SELFBENCH_REPAIR_MODEL` or `gpt-5.6-sol`. Auth: `OPENAI_API_KEY` or `SELFBENCH_CODEX_AUTH_JSON`. Offline companion: ```bash self-bench-repair \ --tasks ./expanded-tasks \ --review ./coupling-review.json \ --output ./repaired-tasks \ --model gpt-5.6-sol ``` Only tasks whose review `resolution.verdict` is `coupled` are repaired. Clean tasks are copied unchanged unless `--task` selects one ID. ### Validation repair (harness, still not gold) A failed nop/oracle gate is a different one-shot activity (`repairValidationTask`, `maximumAttempts: 1`). It may edit the same held-out test paths **and** only these definition fields: `setupCommand`, `testCommand`, `failToPass`, `passToPass`, `testPaths`, `toolchains`, `timeouts`, `resources`. It must not change `schemaVersion`, `difficulty`, `taskId`, `repo`, `baseCommit`, `workdir`, `sourcePr`, `sourceUrl`, or `prompt`. Paths must also look like tests (`tests/`, `__tests__`, `e2e`, `*.test.*`, and the other `isTestOnlyPath` patterns). Changing an immutable field throws `validation repair changed immutable definition field `. ## Authoring vs evaluation The authoring sandbox **does** see provenance, both commits, and the completed diff. That is how `goldPatch` and `testPatch` are split. The authored `prompt` is what later becomes `instruction.md`. Held-out tests must exercise an existing public API, command, persistence boundary, or extension seam. They must not import gold-only private helpers or pin exact internal SQL, query counts, schema or index names, object identity, telemetry shape, incidental error wording, endpoint/response presentation, or UI copy/order unless the authentic request makes that artifact public. A second correct implementation with different files and helper names must still pass. ## Troubleshooting | Symptom | Likely cause | What to check | | --- | --- | --- | | Review `accepted: false` with `gold-only` | Tests assert a token gold invented | Rewrite assertions onto a public seam, or reject the candidate. | | `review did not resolve deterministic coupling evidence` | Reviewer omitted a blocker `artifact` string | Re-audit; findings must match blocker tokens exactly. | | Oracle passes, coding agent still fails | Expected. Hidden tests are the grader. | Confirm `instruction.md` states the required behavior without naming tests. | | Agent overwrote a test file and still failed | `test.sh` restored and re-applied `test.patch` | Do not treat test files as a writable contract. | | `repair changed files outside the held-out tests` | Repair touched app code or a new file | Keep edits inside the original `test.patch` paths. | | `Harbor infrastructure failure` | Harbor/`nop`/`oracle` crashed before rewards | Retry is automatic for validation; not a coupling verdict. | ## Next Static audit, nop/oracle, determinism, one validation-repair attempt, one coupling-repair retry, then reject. How patches become `task.toml`, agent/verifier Dockerfiles, toolchain layers, and `dependency-setup.patch`. `definition.json` fields, `{tests}` rules, and what `submit_task` is allowed to emit. Extract an export and run coding-agent trials without mounting `solution/`. `self-bench-reaudit` and `self-bench-repair` flags, defaults, and report JSON. What the archive includes (tests and gold) and what it excludes (history, provenance, transcripts). --- ## 09. Start the local stack > self-bench up and down: Compose services, --backend docker or modal, --modal-config, named volumes, and what happens when the worker stops. - Page Markdown: https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc/pages/09-start-the-local-stack.md - Generated: 2026-08-15T02:08:03.067Z ### Source Files - `src/cli.ts` - `compose.yaml` - `Dockerfile` - `Dockerfile.sandbox` - `src/temporal/worker-main.ts` - `tests/cli.test.ts` --- title: "Start the local stack" description: "self-bench up and down: Compose services, --backend docker or modal, --modal-config, named volumes, and what happens when the worker stops." --- `self-bench up` starts the packaged Compose project `selfbench`. It resolves `compose.yaml` next to the installed CLI (parent of `dist/cli.js`), optionally builds `selfbench-sandbox:local`, then runs `docker compose --file up -d --build`. `self-bench down` runs `docker compose down` against the same file and does **not** pass `--volumes`, so Temporal history and generated artifacts stay on the named volumes. The command sets both `SELFBENCH_EXECUTION_BACKEND` and `SELFBENCH_HARBOR_ENVIRONMENT` to the same `--backend` value. Sandbox execution and Harbor `nop` / `oracle` gates therefore use one backend. Export `SELFBENCH_API_TOKEN` **before** `self-bench up`. Compose binds the API at `0.0.0.0:8080` inside the container and defaults the token to `selfbench-local` when the host env is unset. The CLI sends `Authorization: Bearer` only when `SELFBENCH_API_TOKEN` is set in the calling shell. A running API with a token and a CLI without one returns `401 unauthorized`. ## Prerequisites - Docker Engine with the Compose plugin - Node.js 22+ (or a global `self-bench` install that already ships `dist/cli.js`) - For `--backend modal`: a Modal profile at `~/.modal.toml`, or `MODAL_TOKEN_ID` / `MODAL_TOKEN_SECRET` - Worker credentials in the host environment (Compose interpolates them into the worker only): `OPENAI_API_KEY`, `GH_TOKEN` Install and auth steps live on [Installation](/installation). This page assumes those are already set. ```bash export OPENAI_API_KEY=... export GH_TOKEN="$(gh auth token)" export SELFBENCH_API_TOKEN="$(openssl rand -hex 24)" ``` ## Commands ```text self-bench up [--backend docker|modal] [--modal-config PATH] self-bench down ``` Selects sandbox execution **and** Harbor `--env`. Any other value fails with `--backend must be "docker" or "modal"`. Host path mounted read-only at `/root/.modal.toml` in the worker. Allowed only with `--backend modal`. Default when omitted: `~/.modal.toml` (`os.homedir()` + `/.modal.toml`). `down` takes no flags. ```bash self-bench up --backend modal ``` ```text SelfBench is running with the modal backend at http://127.0.0.1:8080 ``` On success, point later CLI commands at that listener (this is also the CLI default): ```bash export SELFBENCH_API_URL=http://127.0.0.1:8080 ``` ## Start the stack ```bash modal token new ``` ```bash self-bench up --backend modal ``` Does **not** build `Dockerfile.sandbox`. Sets `SELFBENCH_MODAL_CONFIG_PATH` to `~/.modal.toml` unless `--modal-config` overrides it. ```bash self-bench up --backend modal --modal-config /absolute/path/to/.modal.toml ``` ```bash curl -s http://127.0.0.1:8080/healthz ``` Expected body: ```json { "ok": true } ``` ```bash self-bench up --backend docker ``` Builds `Dockerfile.sandbox` as `selfbench-sandbox:local` **before** Compose. Compose then sets `SELFBENCH_DOCKER_IMAGE=selfbench-sandbox:local`. The worker container starts as `root`, mounts `/var/run/docker.sock`, and on boot runs `docker info` and `docker compose version` (30s timeout each) because execution or Harbor is `docker`. Failure exits the process; Compose `restart: on-failure` retries it. ```bash curl -s http://127.0.0.1:8080/healthz ``` `up` also injects `SELFBENCH_BUILD_COMMIT`: the current `SELFBENCH_BUILD_COMMIT` env if set, else the baked `buildCommit` from package metadata, else `git rev-parse HEAD` in the package root. A missing or all-zero SHA fails with `SelfBench was built without commit metadata; set SELFBENCH_BUILD_COMMIT to a full commit SHA`. Every `up` rebuilds `selfbench:local` from the packaged `Dockerfile` (`--build`). ## Compose project The packaged file is `compose.yaml` with top-level `name: selfbench`. That name is stable across global npm/bun install paths, so the same volumes are reused. :::files self-bench package (parent of dist/cli.js) ├── compose.yaml ├── Dockerfile → image selfbench:local (API + worker) ├── Dockerfile.sandbox → image selfbench-sandbox:local (docker backend only) └── dist/ ├── api-main.js → API default CMD └── temporal/worker-main.js ::: | Service | Image | Command | Host publish | Restart | | --- | --- | --- | --- | --- | | `postgres` | `postgres:17.6-bookworm` | image default | none | none | | `temporal` | `temporalio/auto-setup:1.29.6` | image default | `127.0.0.1:7233` | `on-failure` | | `api` | `selfbench:local` | `node dist/api-main.js` | `127.0.0.1:8080` | `on-failure` | | `worker` | `selfbench:local` | `node dist/temporal/worker-main.js` | none | `on-failure` | Postgres becomes healthy via `pg_isready -U temporal` (5s interval, 20 retries). Temporal waits for that. API and worker only `depends_on: temporal` with no health gate: if Temporal is not accepting connections yet, the Node process exits and Compose restarts it. There is no Temporal Web UI in this project. Postgres is not published on the host. Local DB user and password are both `temporal` (`POSTGRES_DB=temporal`). ```mermaid flowchart TB subgraph host [Host] CLI["self-bench CLI"] Sock["/var/run/docker.sock"] ModalFile["SELFBENCH_MODAL_CONFIG_PATH or ~/.modal.toml"] Creds["OPENAI_API_KEY GH_TOKEN MODAL_TOKEN_*"] end subgraph compose [Compose project selfbench] PG["postgres"] Temporal["temporal :7233"] API["api :8080"] Worker["worker"] end subgraph volumes [Named volumes] PGVol["selfbench_temporal-postgres"] ArtVol["selfbench_artifacts"] end subgraph exec [Execution] Sandbox["selfbench-sandbox:local"] ModalSB["Modal sandboxes"] end CLI -->|"compose up -d --build"| compose CLI -->|"docker build -f Dockerfile.sandbox"| Sandbox PG --> PGVol Temporal --> PG API --> Temporal API --> ArtVol Worker --> Temporal Worker --> ArtVol Worker --> Sock Worker --> ModalFile Creds --> Worker Worker --> Sandbox Worker --> ModalSB ``` ## What each service gets Compose shares an environment anchor (`selfbench-environment`) between API and worker, then adds credentials only on the worker. | Key | API | Worker | Source | | --- | --- | --- | --- | | `SELFBENCH_API_HOST` | `0.0.0.0` | same | Compose | | `SELFBENCH_API_TOKEN` | `${SELFBENCH_API_TOKEN:-selfbench-local}` | same | Host env | | `SELFBENCH_TEMPORAL_ADDRESS` | `temporal:7233` | same | Compose | | `SELFBENCH_ARTIFACT_BACKEND` | `local` | same | Compose | | `SELFBENCH_ARTIFACT_DIR` | `/var/lib/selfbench/artifacts` | same | Compose | | `SELFBENCH_EXECUTION_BACKEND` | from `--backend` | same | `self-bench up` | | `SELFBENCH_HARBOR_ENVIRONMENT` | from `--backend` | same | `self-bench up` | | `SELFBENCH_DOCKER_IMAGE` | `selfbench-sandbox:local` | same | Compose | | `SELFBENCH_BUILD_COMMIT` | from CLI | same | `self-bench up` | | `GH_TOKEN` | — | `${GH_TOKEN:-}` | Host env | | `OPENAI_API_KEY` | — | `${OPENAI_API_KEY:-}` | Host env | | `MODAL_TOKEN_ID` / `MODAL_TOKEN_SECRET` | — | interpolated | Host env | | `SELFBENCH_PI_AUTH_JSON` / `SELFBENCH_CODEX_AUTH_JSON` | — | interpolated | Host env | The API image default user is `node`. The worker service overrides `user: root` and `group_add: ${DOCKER_GID:-0}` so it can use the mounted Docker socket. The API never receives the socket or model / GitHub / Modal credentials. Inside the container the worker polls Temporal namespace `default` and task queue `selfbench-dev` (config defaults; Compose does not override them). Startup log: ```text SelfBench worker polling default/selfbench-dev with activity concurrency N ``` `N` is `1` for docker and `20` for modal unless `SELFBENCH_ACTIVITY_CONCURRENCY` is set (1–100). Unauthenticated surfaces on the published API: | Method | Path | Behavior | | --- | --- | --- | | `GET` | `/healthz` | `{ "ok": true }` | | `GET` | `/` and `/assets/*` | review UI static files | Every `/v1/*` route requires `Authorization: Bearer $SELFBENCH_API_TOKEN` when the token is configured (always true in this Compose file). ## Backends `self-bench up` writes both backend env vars to the **same** value. Harbor validation calls `harbor run --env `. | `--backend` | Sandbox image | Harbor `--env` | Activity concurrency | Extra `up` work | Worker Docker probe | | --- | --- | --- | --- | --- | --- | | `docker` | `selfbench-sandbox:local` | `docker` | 1 | `docker build -f Dockerfile.sandbox -t selfbench-sandbox:local` | `docker info` + `docker compose version` | | `modal` | Modal image default `node:22-bookworm` | `modal` | 20 | mount Modal profile | skipped | `--modal-config` with `--backend docker` fails: `--modal-config requires --backend modal`. When `--backend` is not `modal`, Compose leaves `SELFBENCH_MODAL_CONFIG_PATH` unset and the worker mount becomes `/dev/null:/root/.modal.toml:ro`. Empty `MODAL_TOKEN_ID` / `MODAL_TOKEN_SECRET` (unset or whitespace) are deleted at worker process start so they cannot override a valid mounted `~/.modal.toml`. Non-empty token env vars win over the profile. Resource limits, inactivity timeouts, and Modal app/image knobs are on [Execution backends](/execution-backends). ## Named volumes | Compose volume | Docker volume name | Mount | Holds | | --- | --- | --- | --- | | `temporal-postgres` | `selfbench_temporal-postgres` | `postgres:/var/lib/postgresql/data` | Temporal workflow history | | `artifacts` | `selfbench_artifacts` | API and worker `/var/lib/selfbench/artifacts` | Provenance, per-run artifacts, export tarballs | `self-bench down` is `docker compose --file down` with no `--volumes`. Containers and the default network go away; both volumes remain. There are no application database migrations. Back up those two volumes before replacing images if history or exports must survive an upgrade. Self-bench has no HTTP delete route. Wipe data with ordinary Docker volume tooling, for example `docker volume rm selfbench_artifacts selfbench_temporal-postgres` after `down`. ## What happens when the worker stops Closing a waiting `self-bench run --wait` / `--output` process does **not** cancel the Temporal workflow. The API already started it with `workflowId = runId` and `workflowExecutionTimeout: "14 days"`. ```mermaid stateDiagram-v2 [*] --> ContainersDown ContainersDown --> WorkerPolling: self-bench up WorkerPolling --> WorkflowPaused: worker exits or stack stops WorkflowPaused --> WorkerPolling: worker process returns WorkerPolling --> ContainersDown: self-bench down ``` | Event | Workflows | In-flight activities | Volumes | Next action | | --- | --- | --- | --- | --- | | Worker process crash | Persist in Temporal / Postgres | Heartbeats stop | Unchanged | Compose `restart: on-failure` starts the worker again | | `docker stop` on worker (not a non-zero exit) | Persist | Heartbeats stop | Unchanged | Stays down until `docker compose start` or another `self-bench up` | | Whole stack crash / `self-bench down` | Persist on `selfbench_temporal-postgres` | Activities abort | Persist | `self-bench up` remounts volumes; worker resumes `default/selfbench-dev` | | Host crash during a Docker sandbox | Persist | Sandbox may be left on the host | Persist | Inspect leftover `selfbench---` containers/volumes | Activity proxy timeouts (so a dead worker does not hang a run forever): | Activities | `startToCloseTimeout` | `heartbeatTimeout` | Retry | | --- | --- | --- | --- | | Author, validate, review, repair, audit, export | 7 hours | 10 minutes | 4 attempts, 5s → 2m backoff | | Discovery shards | 1 hour | 10 minutes | 3 attempts, 5s → 1m backoff | | Validation-repair | 2 hours | 10 minutes | 1 attempt | After ~10 minutes without a heartbeat, Temporal marks the activity failed and retries according to the table. The worker does not resume the interrupted OS process; it takes the next retry. Docker sandboxes are created as disposable containers and volumes named `selfbench---` and are `docker rm` / `docker volume rm` on normal completion. A crash can leave those host resources for an operator to remove. Modal sandboxes are disposable on the Modal side. `postgres` has **no** `restart:` policy. If that container dies and stays down, Temporal and then the API/worker fail until Postgres is running again. ## Stop the stack ```bash self-bench down ``` Use `self-bench list` / `self-bench status RUN_ID` after the next `up` to see runs that were in flight. Cancel only with `self-bench cancel RUN_ID`. ## Linux Docker group If the worker cannot use the mounted socket, set `DOCKER_GID` to the host `docker` group id before `up`: ```bash export DOCKER_GID="$(getent group docker | cut -d: -f3)" self-bench up --backend docker ``` Compose default is `0`. ## Failure modes The API container has a token (`selfbench-local` or whatever was exported at `up` time). Export the **same** `SELFBENCH_API_TOKEN` in the CLI shell. Changing the host token after `up` does not rotate the running API; run `self-bench down` and `up` again. `--backend docker` (or a compose override that leaves Harbor on docker) requires a working host Docker API from inside the worker. Check `/var/run/docker.sock` and `DOCKER_GID`. Blank `MODAL_TOKEN_ID` / `MODAL_TOKEN_SECRET` are stripped. Non-empty values override `/root/.modal.toml`. Confirm `--backend modal` and that `--modal-config` is an existing host file (Compose mounts it read-only). The CLI could not resolve a 40-hex SHA. Set `SELFBENCH_BUILD_COMMIT` to the self-bench commit you intend to record on runs. They connect to `temporal:7233` with no wait script. Watch `docker compose --file "$(node -p "require('path').join(require('path').dirname(require.resolve('self-bench/package.json')), 'compose.yaml')")" logs temporal` until auto-setup finishes, then confirm the API log `SelfBench API listening on http://0.0.0.0:8080`. Host-side `bun run dev:api` / `bun run dev:worker` are checkout-only processes. They are not the Compose stack and do not create these volumes. See [Contributing](/contributing). ## Next Submit a tiered run against this stack and download an export. `self-bench run` flags, HEAD pin, and `--output` wait semantics. `list`, `status`, `cancel`, and `download` against `SELFBENCH_API_URL`. Docker vs Modal images, concurrency, limits, and Harbor pairing. Split API and worker, GCS artifacts, matching image digest and task queue. 401s, leftover Docker resources, and export integrity failures. --- ## 10. Build a benchmark > self-bench run: --repo GitHub origin, 1-100 tier budgets, --model, --run-id, HEAD pin, provenance upload, and --output wait plus SHA-256 download. - Page Markdown: https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc/pages/10-build-a-benchmark.md - Generated: 2026-08-15T02:07:29.644Z ### Source Files - `src/cli.ts` - `src/provenance.ts` - `src/temporal/workflow.ts` - `src/extensions/authoring.ts` - `src/skills/selfbench/SKILL.md` - `docs/task-construction.md` --- title: "Build a benchmark" description: "self-bench run: --repo GitHub origin, 1-100 tier budgets, --model, --run-id, HEAD pin, provenance upload, and --output wait plus SHA-256 download." --- `self-bench run` pins a Git checkout (`origin` URL + `HEAD` SHA), uploads a sanitized NDJSON provenance corpus, and starts Temporal workflow `selfBenchRunWorkflow` via `POST /v1/runs`. `--easy-count`, `--medium-count`, and `--hard-count` are candidate authoring budgets whose total must be 1–100. `--output` implies wait: the CLI polls until `complete`, then writes a create-only SHA-256-verified export. The CLI performs only repository metadata and provenance upload locally. Discovery, authoring, validation, review, audit, and export run on the API/worker stack. Closing a waiting CLI does not cancel the workflow. Rejected candidates are not replaced. The export contains only accepted tasks and can finish with fewer tasks than the budget, including zero accepted tasks. ## Prerequisites - A running stack (`self-bench up`). See [Start the local stack](/start-local-stack). - `SELFBENCH_API_URL` if the API is not `http://127.0.0.1:8080`. - The same `SELFBENCH_API_TOKEN` the API expects. Local Compose defaults the API token to `selfbench-local` when the host does not set one. When the CLI variable is set, every request sends `Authorization: Bearer $SELFBENCH_API_TOKEN`. - A Git checkout whose `origin` can be resolved. SSH `git@github.com:owner/repo` is rewritten to `https://github.com/owner/repo`. Other remotes must already start with `https://`. GitHub pull-request provenance requires a `github.com` URL. - Authenticated `gh` so the CLI can list merged pull requests. - At least one sanitized local-session or GitHub pull-request message. An empty corpus fails before submit. - Worker credentials for remote work: `GH_TOKEN` to clone the pinned origin, plus `OPENAI_API_KEY` (or the subscription JSON fallbacks). See [Authentication](/authentication). The published CLI refuses to submit if it cannot resolve a 40-character SelfBench commit. Resolution order is `SELFBENCH_BUILD_COMMIT`, baked `buildCommit` from the package build, then `git rev-parse HEAD` of the self-bench source tree. ## Submit a run ```bash export SELFBENCH_API_URL=http://127.0.0.1:8080 export SELFBENCH_API_TOKEN="${SELFBENCH_API_TOKEN:-selfbench-local}" ``` Pass `--repo` as an absolute or relative path. Counts default to `0` and must sum to 1–100. ```bash self-bench run \ --repo /absolute/path/to/your/repository \ --easy-count 10 \ --medium-count 10 \ --hard-count 10 \ --model gpt-5.6-sol \ --output ./self-bench-evals.tar.gz ``` Omit `--output` and `--wait` to print the accept payload and return while Temporal continues. Reattach later with [Manage runs](/manage-runs). Stdout after a successful `POST /v1/runs` (`202`) looks like: ```json { "runId": "sb-20260814123456-a1b2c3d4", "provenanceMessages": 42, "localProvenanceMessages": 17, "githubPullRequestMessages": 25 } ``` `--wait` and `--output` poll `GET /v1/runs/:runId` every 2 seconds. Each **phase change** is written to stderr: ```json {"runId":"sb-20260814123456-a1b2c3d4","phase":"discovering","accepted":0,"rejected":0} ``` On `complete`, `--output` downloads the export. Success prints `{ "runId": "...", "output": "/abs/path/self-bench-evals.tar.gz" }`. Confirm `tar -tzf` shows `manifest.json` and `tasks/`. Path to the Git checkout. Short flag `-r`. Resolved with `path.resolve`. Easy-tier authoring budget. Default `0`. Nonnegative integer. Medium-tier authoring budget. Default `0`. Hard-tier authoring budget. Default `0`. Authoring model string. Default `gpt-5.6-sol`. Posted as `authoringModel`. The API stores `{ provider: "openai-codex", model, reasoningEffort: "high" }`. Sandboxes use that model as `AUTHOR_MODEL`. Temporal workflow ID. Default `sb-<14-digit UTC timestamp>-<8 UUID chars>`. The API accepts `^[a-z0-9][a-z0-9-]{2,62}$` (3–63 characters). The CLI does not validate locally; an invalid ID fails at provenance upload. Poll until `complete`. Default `false`. `--output` implies wait even when this flag is omitted. Destination path for the export `.tar.gz`. Short flag `-o`. Implies wait, then downloads after `complete`. ## Local work: pin, collect, upload ### HEAD pin The CLI records: | Field | Source | | --- | --- | | `repository.url` | `git remote get-url origin`, with `git@github.com:` rewritten to `https://github.com/` | | `repository.commit` | `git rev-parse HEAD` (full 40-character SHA) | Uncommitted files are not part of the pin. Workers clone `SOURCE_REPO_URL` and check out that SHA (`git fetch origin $SOURCE_COMMIT`). Each candidate still uses its own pull-request `baseCommit` and `completedCommit`. ### Provenance corpus The CLI concatenates, then rejects an empty result with `no sanitized local-session or GitHub pull-request provenance was found`: 1. Local user messages from `$HOME/.pi/agent/sessions`, `$HOME/.claude/projects`, `$HOME/.codex/sessions`, and `$HOME/.codex/archived_sessions` (`.jsonl` only), matched to the checkout or its `git worktree list` paths. 2. Up to 500 merged, non-draft, non-bot GitHub pull requests via `gh pr list`, kept when the title is non-empty, `changedFiles >= 1`, and `additions + deletions >= 20`. Bodies longer than 12,000 characters are dropped (title only). Secrets and injected harness prefixes are stripped. GitHub records keep `sourceType: "github-pull-request"` plus `sourcePr` / `sourceUrl`. Details: [Provenance sources](/provenance-sources). ### Upload then start ```http POST /v1/provenance?runId=sb-20260814123456-a1b2c3d4 Content-Type: application/x-ndjson Authorization: Bearer $SELFBENCH_API_TOKEN ``` ```http POST /v1/runs Content-Type: application/json Authorization: Bearer $SELFBENCH_API_TOKEN { "runId": "sb-20260814123456-a1b2c3d4", "repository": { "url": "https://github.com/example/project.git", "commit": "0123456789abcdef0123456789abcdef01234567" }, "provenance": { "uri": "file:///var/lib/selfbench/artifacts/runs/.../provenance.jsonl", "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "sizeBytes": 4096, "contentType": "application/x-ndjson" }, "candidateCounts": { "easy": 10, "medium": 10, "hard": 10 }, "authoringModel": "gpt-5.6-sol", "selfbenchCommit": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" } ``` Provenance bodies are capped at 100 MiB and stored at `runs//input/provenance.jsonl` (`201` + artifact ref). Run bodies use the 10 MiB default. The API wraps the submission into `RunRequest`, starts `selfBenchRunWorkflow` with `workflowId = runId`, `workflowExecutionTimeout` 14 days, and returns `202 { "runId" }`. `version.selfbenchCommit` is `SELFBENCH_BUILD_COMMIT` on the API when set, otherwise the CLI SHA. `version.executionBackend` and `sandboxImage` come from worker/API config, not CLI flags. Reusing a `runId` that already exists as a Temporal workflow ID fails start. ```mermaid sequenceDiagram participant CLI as self-bench run participant API as HTTP API participant Artifacts as Artifact store participant Temporal as Temporal participant Worker as Worker sandboxes CLI->>CLI: origin URL + HEAD SHA CLI->>CLI: local sessions + gh merged PRs CLI->>API: POST /v1/provenance?runId= API->>Artifacts: put runs/id/input/provenance.jsonl API-->>CLI: 201 ArtifactRef CLI->>API: POST /v1/runs API->>Temporal: start selfBenchRunWorkflow API-->>CLI: 202 runId loop every 2s until complete or failed phase CLI->>API: GET /v1/runs/:runId API->>Temporal: status query API-->>CLI: phase, accepted, rejected end CLI->>API: GET /v1/runs/:runId/export API->>Artifacts: openRead(export) API-->>CLI: gzip + x-content-sha256 ``` ## Budgets versus accepted tasks | Rule | Behavior | | --- | --- | | Total 1–100 | CLI rejects any other sum. API also requires each of `easy` / `medium` / `hard` in `0..100` and the same total bounds. | | Budget, not target | Discovery fills each requested tier, then authors exactly those selected candidates. | | No replacement | A rejected or `infrastructure_failed` candidate does not consume a reserve PR. | | Empty export is allowed | After authoring, the workflow always enters `exporting` and can `complete` with `acceptedTaskIds: []`. | Discovery uses eight shards and stops when every requested tier is filled, or enters `blocked` with `candidate pool exhausted; missing easy=…, medium=…, hard=…`. Lifecycle and per-candidate statuses: [Run lifecycle](/run-lifecycle). Eligibility thresholds: [Difficulty gates](/difficulty-gates). ## Wait and download `waitForRun` treats `complete` as success. `blocked`, `failed`, and `cancelled` throw `SelfBench run [: ]` and skip download. | Mode | After submit | | --- | --- | | neither flag | Print accept JSON and exit. Workflow keeps running. | | `--wait` only | Poll; print the final status JSON on stdout. | | `--output PATH` | Poll; on `complete`, `GET /v1/runs/:runId/export`. | Download rules: - `409` with `run export is not ready` if `status.export` is missing. - Response must include `x-content-sha256` and a body, or the CLI throws `SelfBench API returned an export without integrity metadata`. - Destination is opened with `wx` (create-only). An existing file fails. - Bytes are hashed while streaming. A mismatch throws `downloaded export failed its SHA-256 integrity check` and deletes the partial file. - The API also sends `content-disposition: attachment; filename="selfbench-.tar.gz"`. The CLI writes the path you passed, not that name. The `x-content-sha256` value is the artifact digest of the outer archive, not a signature. Archive layout: [Export format](/export-format). ## Errors | Condition | What you see | | --- | --- | | Missing `--repo` | `--repo is required` | | Total counts outside 1–100 | `the total candidate count must be between 1 and 100` | | Non-integer / negative count | `--easy-count must be a nonnegative integer` (same for medium/hard) | | Origin is not `git@github.com:` and not `https://` | `unsupported origin URL: …` | | Origin is HTTPS but not GitHub | Fails in GitHub provenance parse: `unsupported GitHub repository URL` | | Empty corpus | `no sanitized local-session or GitHub pull-request provenance was found` | | No SelfBench commit metadata | `SelfBench was built without commit metadata; set SELFBENCH_BUILD_COMMIT to a full commit SHA` | | Missing / wrong API token | `unauthorized` | | Invalid `runId` | `400` from Zod on `/v1/provenance` or `/v1/runs` | | Provenance > 100 MiB | `request body exceeds 104857600 bytes` | | Status JSON lacks `phase` | `SelfBench status response is missing its phase` | | `blocked` / `failed` / `cancelled` | `SelfBench run blocked: …` (or `failed` / `cancelled`) | | Export not ready | `run export is not ready` | | SHA-256 mismatch | `downloaded export failed its SHA-256 integrity check`; output file removed | A stopped worker pauses Temporal activities; it does not cancel the run. Restart the stack and poll or download the same `runId`. ## Next Phases, eight-shard discovery, and candidate versus accepted counts. `list`, `status`, `cancel`, and a later `download` against the same API. Extract the export and run `self-bench-eval` or Harbor. `manifest.json`, per-task `harbor-task` archives, and SHA-256 fields. Full flag set, wait/download semantics, and command-level errors. Empty provenance, blocked pools, 409 export-not-ready, and hash mismatches. --- ## 11. Manage runs > list, status, cancel, and download against SELFBENCH_API_URL, including phase polling, create-only writes, and integrity headers. - Page Markdown: https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc/pages/11-manage-runs.md - Generated: 2026-08-15T02:08:06.168Z ### Source Files - `src/cli.ts` - `src/api.ts` - `src/run-wait.ts` - `src/contracts.ts` - `tests/run-wait.test.ts` - `docs/operations.md` --- title: "Manage runs" description: "list, status, cancel, and download against SELFBENCH_API_URL, including phase polling, create-only writes, and integrity headers." --- `self-bench list`, `status`, `cancel`, and `download` are one-shot HTTP clients of the API at `SELFBENCH_API_URL` (default `http://127.0.0.1:8080`). They send `Authorization: Bearer $SELFBENCH_API_TOKEN` when that variable is set, print the JSON body, and never own Temporal execution. Closing the CLI — including a waiting `self-bench run --wait` or `--output` process — does not cancel the workflow. The local Compose API always has a token (`SELFBENCH_API_TOKEN` or the compose default `selfbench-local`). A CLI that omits the token receives `401` `{ "error": "unauthorized" }`. ## Prerequisites ```bash export SELFBENCH_API_URL=http://127.0.0.1:8080 export SELFBENCH_API_TOKEN="$SELFBENCH_API_TOKEN" ``` `SELFBENCH_API_URL` is read only by the CLI. The API process binds `SELFBENCH_API_HOST`:`SELFBENCH_API_PORT` (compose publishes `127.0.0.1:8080`). ```bash curl -sS "$SELFBENCH_API_URL/healthz" ``` Expected: `{ "ok": true }`. `/healthz` is unauthenticated. Every `/v1/runs` route requires the Bearer token when the API was started with `SELFBENCH_API_TOKEN`. CLI base URL. Default `http://127.0.0.1:8080`. Shared secret. When set, the CLI adds `Authorization: Bearer …`. The API compares it with a timing-safe equality check. A non-loopback API bind refuses to start without this token. ## Commands | Command | HTTP | CLI output | | --- | --- | --- | | `self-bench list` | `GET /v1/runs` | Temporal execution summaries (JSON array) | | `self-bench status RUN_ID` | `GET /v1/runs/:runId` | `RunStatus` JSON | | `self-bench cancel RUN_ID` | `POST /v1/runs/:runId/cancel` | `{ runId, cancellationRequested: true }` | | `self-bench download RUN_ID OUTPUT.tar.gz` | `GET /v1/runs/:runId/export` | `{ runId, output }` after a verified write | ```bash self-bench list self-bench status RUN_ID self-bench cancel RUN_ID self-bench download RUN_ID ./self-bench-evals.tar.gz ``` `RUN_ID` is the first positional argument. `download` requires a second path. These commands take no flags. `self-bench run --wait` and `self-bench run --output PATH` are the only surfaces that poll `GET /v1/runs/:runId`. `--output` implies `--wait`, then calls the same download path. ## Run IDs API run paths accept only `^[a-z0-9][a-z0-9-]{2,62}$` (3–63 characters, start with a letter or digit). A custom `--run-id` on submit must match that regex. Generated IDs look like `sb-20260814153022-a1b2c3d4`. An ID that fails the path regex never reaches Temporal: the API returns `404` `{ "error": "not found" }`. Generation workflows that fail must be resubmitted under a **new** run ID. There is no remote delete route. ## List runs `GET /v1/runs` walks Temporal visibility for `WorkflowType = 'selfBenchRunWorkflow'` and returns every matching execution. There is no pagination, filter, or limit. ```json [ { "runId": "sb-20260814153022-a1b2c3d4", "status": "RUNNING", "startedAt": "2026-08-14T15:30:22.000Z", "closedAt": null } ] ``` Temporal workflow ID. Same string as the submitted run ID. Temporal execution name (`RUNNING`, `COMPLETED`, `FAILED`, `CANCELED`, `TERMINATED`, `TIMED_OUT`, …). This is **not** the application `phase`. ISO-8601 start time. ISO-8601 close time when the execution has closed; omitted or null while running. `list.status` and `status.phase` are different vocabularies. Temporal `CANCELED` maps to application phase `cancelled`. Use `self-bench status` for accepted/rejected counts, per-candidate stages, and export readiness. ## Read status `GET /v1/runs/:runId` returns the workflow `status` query when the execution is `RUNNING` or the query already reports a terminal application phase. Otherwise the API overlays the Temporal execution onto `phase`. If the query handler is unavailable but `describe()` succeeds, the body shrinks to `{ runId, phase }`. `RUNNING` in that fallback is reported as `queued`. ```json { "runId": "sb-20260814153022-a1b2c3d4", "phase": "authoring", "requested": 20, "requestedByDifficulty": { "easy": 5, "medium": 10, "hard": 5 }, "discovered": 24, "accepted": 3, "rejected": 2, "tasks": [ { "taskId": "project-pr-123", "candidateId": "project-pr-123", "difficulty": "medium", "status": "validating" } ], "discovery": { "wave": 0, "totalShards": 8, "completedShards": 8, "failedShards": 0, "candidates": 24 } } ``` | Field | Meaning | | --- | --- | | `phase` | Application phase: `queued`, `discovering`, `authoring`, `exporting`, `complete`, `blocked`, `failed`, `cancelled` | | `requested` | Sum of the three tier budgets | | `requestedByDifficulty` | `{ easy, medium, hard }` budgets | | `discovered` | Unique candidates collected so far | | `accepted` / `rejected` | Counts derived from `tasks[].status` | | `tasks[]` | Per-candidate `authoring` → `auditing` → `validating` → `reviewing` → `repairing` → `accepted` / `rejected` / `infrastructure_failed` | | `discovery` | Current wave, eight shards, completed/failed shard counts, candidate count | | `export` | Present after a successful export: `{ uri, sha256, sizeBytes, contentType }` | | `error` | Set on `blocked` / `failed` (and on some Temporal overlays such as `Temporal workflow timed_out`) | A completed status includes `export` with `contentType` `application/gzip` and a 64-character lowercase hex `sha256`. That object is what `download` and the review UI consume. Phase meanings, shard refill, and terminal `blocked` / `failed` / `cancelled` behavior live on [Run lifecycle](/run-lifecycle). ## Phase polling `waitForRun` polls `GET /v1/runs/:runId` every **2000 ms**. It emits `onPhase` only when `phase` changes. `self-bench run` writes those events to **stderr** as one JSON object per change: ```json {"runId":"sb-20260814153022-a1b2c3d4","phase":"authoring","accepted":1,"rejected":0} ``` ```mermaid stateDiagram-v2 [*] --> Poll Poll --> EmitPhase: phase changed EmitPhase --> Decide Poll --> Decide: same phase Decide --> Complete: phase == complete Decide --> Fail: blocked or failed or cancelled Decide --> Sleep: otherwise Sleep --> Poll: 2000 ms ``` | Terminal `phase` | Waiter behavior | | --- | --- | | `complete` | Returns the status. `--output` then downloads. `--wait` without `--output` prints the status on stdout. | | `blocked`, `failed`, `cancelled` | Throws `SelfBench run ` plus `: ` when `error` is a string. No download. | | Missing `phase` | Throws `SelfBench status response is missing its phase`. | `self-bench status` does not poll. Re-run it, or use `run --wait` / `--output`. ## Cancel a run :::endpoint POST /v1/runs/:runId/cancel Request Temporal cancellation **Auth:** Bearer token when configured. **Success:** `202` ```json { "runId": "sb-20260814153022-a1b2c3d4", "cancellationRequested": true } ``` The handler calls Temporal `workflow.cancel()` and returns immediately. In-flight activities use `WAIT_CANCELLATION_COMPLETED`. The workflow sets `phase` to `cancelled` when it observes cancellation, then rethrows so Temporal records `CANCELED`. A stopped worker does not finish the cancel until it restarts. A waiting CLI is not cancelled by Ctrl-C; call this route (or `self-bench cancel`) explicitly. ::: `self-bench list` still shows Temporal `CANCELED` after the execution closes. Failed generation is not retryable under the same ID. ## Download an export :::endpoint GET /v1/runs/:runId/export Stream the completed archive **Auth:** Bearer token when configured. **Success headers:** | Header | Value | | --- | --- | | `content-type` | Artifact `contentType` (`application/gzip`) | | `content-length` | Artifact `sizeBytes` | | `content-disposition` | `attachment; filename="selfbench-.tar.gz"` | | `x-content-sha256` | Artifact SHA-256 (64 hex chars) | **Errors:** `409` `{ "error": "run export is not ready" }` when `status.export` is missing. `401` if the token is required and missing or wrong. The API streams `artifacts.openRead(export)`, which checks stored length and SHA-256 while reading. A store integrity failure becomes a mid-stream error after headers may already be sent. ::: ```bash CLI self-bench download sb-20260814153022-a1b2c3d4 ./self-bench-evals.tar.gz ``` ```bash curl curl -fL \ -H "Authorization: Bearer $SELFBENCH_API_TOKEN" \ -D - \ -o ./self-bench-evals.tar.gz \ "$SELFBENCH_API_URL/v1/runs/sb-20260814153022-a1b2c3d4/export" ``` Successful CLI stdout: ```json { "runId": "sb-20260814153022-a1b2c3d4", "output": "/absolute/path/to/self-bench-evals.tar.gz" } ``` The hosted review UI lists the same `/v1/runs` array and fetches `/export` for a completed run. Cleaning or dropping tasks is a review-UI concern, not a CLI download flag. ## Create-only writes and integrity ```mermaid sequenceDiagram participant CLI as self-bench download participant API as HTTP API participant WF as Temporal status query participant Store as Artifact store CLI->>API: GET /v1/runs/:runId/export API->>WF: query status alt export missing API-->>CLI: 409 run export is not ready else export present API->>Store: openRead(export) API-->>CLI: 200 + x-content-sha256 + body CLI->>CLI: open(output, wx) CLI->>CLI: stream SHA-256 alt header missing or hash mismatch or write error CLI->>CLI: rm output CLI-->>CLI: throw else digest matches header CLI-->>CLI: print runId and output end end ``` Rules the CLI enforces: 1. **Integrity header required.** Missing `x-content-sha256` or an empty body throws `SelfBench API returned an export without integrity metadata`. 2. **Create-only file.** The destination is opened with `wx`. An existing path fails (typically `EEXIST`); the CLI does not overwrite. 3. **Streamed SHA-256.** Bytes are hashed while written. The hex digest must equal `x-content-sha256`. 4. **Failed writes are deleted.** Any error after `open` — including `downloaded export failed its SHA-256 integrity check` — `rm`s the destination so a partial archive cannot be mistaken for a good export. Artifact `put` / `putFile` on the server also use create-only (`wx` / exclusive link). A key may be rewritten only when the existing bytes already match. Exports contain source snapshots, held-out tests, and reference solutions. They are unencrypted. Treat the output path as sensitive. ## HTTP mapping | Method | Path | Status | Body | | --- | --- | --- | --- | | `GET` | `/healthz` | 200 | `{ ok: true }` (no auth) | | `GET` | `/v1/runs` | 200 | Execution array | | `GET` | `/v1/runs/:runId` | 200 | `RunStatus` | | `POST` | `/v1/runs/:runId/cancel` | 202 | `{ runId, cancellationRequested }` | | `GET` | `/v1/runs/:runId/export` | 200 / 409 | gzip stream or `{ error }` | | `POST` | `/v1/runs` | 202 | `{ runId }` — submit only; see [Build a benchmark](/build-a-benchmark) | JSON error envelope: `{ "error": "" }` with `400` (Zod / `SyntaxError`), `401`, `404`, `409`, or `500`. The CLI throws `error` when present, otherwise `SelfBench API returned `. A Temporal describe/query failure for an unknown ID surfaces as `500` with the Temporal message. Duplicate `POST /v1/runs` for an existing workflow ID also fails at Temporal start (treat as a new-ID problem, not an in-place retry). Workflows are started with `workflowExecutionTimeout: "14 days"`. A timeout overlays `phase: "failed"` and, if the query has no `error`, `Temporal workflow timed_out`. ## Errors | Symptom | Cause | What to do | | --- | --- | --- | | `unauthorized` | Token configured on the API, missing or wrong on the CLI | Export the same `SELFBENCH_API_TOKEN` the API process has. Compose default is `selfbench-local`. | | `run ID is required` / `output path is required` | Missing positional args | `status`/`cancel` need `RUN_ID`; `download` needs `RUN_ID` and a path. | | `not found` | Path failed the run-ID regex, or no such route | Check the ID charset and length. | | `run export is not ready` | `phase` is not `complete`, or `export` is absent | Wait, or inspect `self-bench status`. Do not retry download on `blocked` / `failed` / `cancelled`. | | `downloaded export failed its SHA-256 integrity check` | Stream digest ≠ `x-content-sha256` | The partial file is removed. Retry once; persistent mismatch is a store/transport problem. | | `SelfBench API returned an export without integrity metadata` | Response lacked `x-content-sha256` or a body | The API always sets the header on a successful export; a proxy or mock is dropping it. | | `EEXIST` | Destination already exists | Choose a new path. Download will not overwrite. | | `SelfBench run blocked: candidate pool exhausted; missing easy=…` | Discovery could not fill the requested budgets | New run ID; see [Run lifecycle](/run-lifecycle) and [Troubleshooting](/troubleshooting). | | `SelfBench run failed: …` / `SelfBench run cancelled` | Waiter saw a terminal unsuccessful phase | Read `status.error`. Cancelled waits are expected after `self-bench cancel`. | | Work appears stuck after `list` shows `RUNNING` | Worker or Compose stack stopped | Restart the worker. History and artifacts stay on the named volumes. | | Need to drop a completed run | No `DELETE` route | Delete the local artifact volume or the GCS prefix with operator tooling. | ## Related pages Phases, shard discovery, budgets versus accepted tasks, and terminal blocked/failed/cancelled states. `self-bench run` submission, `--run-id`, `--wait`, and `--output`. Full route table, Bearer auth, provenance limits, and export headers. API-hosted review UI, drop tasks, and a cleaned tar.gz. Outer archive layout, `manifest.json`, and SHA-256 checks. 409 export-not-ready, SHA-256 mismatch, and leftover Docker resources. --- ## 12. Evaluate with Harbor > Extract an export, run one task or the Codex matrix via self-bench-eval, resume completed jobs, and smoke Harbor agent adapters. - Page Markdown: https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc/pages/12-evaluate-with-harbor.md - Generated: 2026-08-15T02:08:36.223Z ### Source Files - `src/eval-main.ts` - `src/evaluate.ts` - `src/agent-smoke-main.ts` - `src/agent-smoke.ts` - `docs/evaluations.md` - `tests/evaluate.test.ts` --- title: "Evaluate with Harbor" description: "Extract an export, run one task or the Codex matrix via self-bench-eval, resume completed jobs, and smoke Harbor agent adapters." --- `self-bench-eval` is the published companion binary that turns a completed self-bench export into Harbor Codex trials. It accepts either the outer `.tar.gz` or an already-expanded task tree, fans out one Harbor job per `(task, model)` pair, writes `summary.json` under `--jobs`, and skips Harbor when that job already has a finished result. A single-task `harbor run` path remains available without the matrix helper. `self-bench-agent-smoke` is a separate install-only check of Harbor's pinned agent adapters, not a scoring run. Generation-time Harbor **nop** / **oracle** gates belong to authoring, not this workflow. See [Validation and repair](/validation-and-repair). Exports contain repository snapshots, held-out tests, and reference solutions. They are sensitive and unencrypted. Keep them private and extract only archives from a trusted self-bench deployment. ## Prerequisites | Requirement | Role | | --- | --- | | `self-bench` package (`node` ≥ 22) | Provides `self-bench-eval` and `self-bench-agent-smoke` | | Harbor `harbor[modal]==0.20.1.dev202608040148` | Task runner. Put `harbor` on `PATH` or pass `--harbor` | | Python 3.12 + `uv` | Harbor install | | `OPENAI_API_KEY` **or** Codex ChatGPT `auth.json` | Codex trial auth. API key wins when set | | Modal account/token | Required for `--environment modal` (default) | | Docker daemon | Required for `--environment docker` | Install the pinned Harbor release: ```bash uv tool install --python 3.12 'harbor[modal]==0.20.1.dev202608040148' harbor --help ``` From a self-bench checkout after `bun run build`, the same binaries are `node dist/eval-main.js` and `node dist/agent-smoke-main.js`. `self-bench-eval` does not read `SELFBENCH_API_URL`, `SELFBENCH_HARBOR_ENVIRONMENT`, or `SELFBENCH_CODEX_AUTH_JSON`. Those keys belong to the API/worker and generation sandboxes. Evaluation talks to a local Harbor executable and uses `OPENAI_API_KEY` or a Codex `auth.json` file. ## Get an export The input is the SHA-256-verified outer archive from a completed run (`self-bench run --output` or `self-bench download`). Download is create-only: the destination path must not already exist. The API response must include `x-content-sha256`; a mismatch deletes the partial file. ```bash title="Wait and download during submit" self-bench run \ --repo /absolute/path/to/your/repository \ --easy-count 10 \ --medium-count 10 \ --hard-count 10 \ --output ./self-bench-evals.tar.gz ``` ```bash title="Download a finished run" self-bench download RUN_ID ./self-bench-evals.tar.gz ``` The outer archive is: :::files manifest.json tasks/ ├── TASK_ID.tar.gz └── … ::: Each inner archive is a compiled Harbor task under `harbor-task/` (`task.toml` schema 1.4, `instruction.md`, `environment/`, `tests/`, `solution/`). Full field layout is on [Export format](/export-format). ## Extract tasks Use one of these layouts. `self-bench-eval --export` does the inner extract itself into `/tasks/`. `--tasks` expects each immediate subdirectory to contain `task.toml`. Pass the outer archive. On first use, the runner unpacks `tasks/*.tar.gz` and copies each `harbor-task/` to `/tasks//`. If `/tasks/` already has one or more `task.toml` directories, that tree is reused and the export is **not** re-read. ```bash self-bench-eval --export ./self-bench-evals.tar.gz --jobs ./matrix-jobs ``` Matches the README layout: each task directory *is* the Harbor task ( `task.toml` at the top of the folder). ```bash mkdir -p ./self-bench-export ./self-bench-tasks tar -xzf ./self-bench-evals.tar.gz -C ./self-bench-export for archive in ./self-bench-export/tasks/*.tar.gz; do task_id="$(basename "$archive" .tar.gz)" mkdir -p "./self-bench-tasks/$task_id" tar -xzf "$archive" --strip-components=1 -C "./self-bench-tasks/$task_id" done ``` Then pass `--tasks ./self-bench-tasks`. ```bash mkdir -p ./export ./selected-task tar -xzf ./self-bench-evals.tar.gz -C ./export TASK_ID="$(jq -r '.tasks[0].taskId' ./export/manifest.json)" tar -xzf "./export/tasks/$TASK_ID.tar.gz" -C ./selected-task ``` Raw Harbor uses `--path ./selected-task/harbor-task`. The matrix helper uses `--tasks ./selected-task` because `harbor-task/task.toml` is the discovery signal. Discovery ignores files and directories that do not contain a readable `task.toml`. Empty inputs fail with `found no Harbor tasks in …`. Providing both `--export` and `--tasks` fails with `provide exactly one of exportPath or tasksPath`. ## What a trial mounts A Codex (or other coding-agent) trial receives the base snapshot, `instruction.md`, and the compiled agent environment. It does not receive the held-out test patch or the reference solution. Harbor mounts `solution/` only for the explicit `oracle` agent. Compiled `task.toml` also pins the agent network allowlist to `chatgpt.com`, `*.chatgpt.com`, `openai.com`, and `*.openai.com`, and collects `/opt/selfbench/agent.patch` as the trial artifact. Resource and timeout numbers come from the task definition, not from `self-bench-eval` flags. See [Agent-visible boundary](/agent-visible-boundary) and [Harbor task compiler](/harbor-task-compiler). ## Run one task This path calls Harbor directly. It does not write `summary.json` and does not implement self-bench's job-resume rules. ```bash uv tool install --python 3.12 'harbor[modal]==0.20.1.dev202608040148' export OPENAI_API_KEY=... ``` ```bash harbor run \ --path ./selected-task/harbor-task \ --agent codex \ --model gpt-5.6-sol \ --ak version=0.146.1 \ --ak reasoning_effort=high \ --env modal \ --jobs-dir ./harbor-jobs \ --yes ``` Codex is pinned to `@openai/codex@0.146.1` via `--ak version=0.146.1`. Reasoning is `high`. Harbor can take the expanded tree and all three matrix models in one invocation (no per-job resume): ```bash harbor run \ --path ./self-bench-tasks \ --agent codex \ --model gpt-5.6-luna \ --model gpt-5.6-terra \ --model gpt-5.6-sol \ --ak version=0.146.1 \ --ak reasoning_effort=high \ --env modal \ --jobs-dir ./harbor-jobs \ --n-concurrent 20 \ --yes ``` ## Run the Codex matrix `self-bench-eval` evaluates every discovered task with the fixed matrix `gpt-5.6-sol`, `gpt-5.6-terra`, and `gpt-5.6-luna` unless `--model` narrows the set. Work is the cartesian product `models × tasks`, executed with `--concurrency` (default `3`). Each Harbor subprocess uses `--n-concurrent 1`. ```bash export OPENAI_API_KEY=... self-bench-eval \ --export ./self-bench-evals.tar.gz \ --jobs ./matrix-jobs \ --harbor harbor \ --environment modal \ --concurrency 20 ``` Restrict models by repeating `--model`. Values must be members of the matrix; duplicates are dropped. ```bash self-bench-eval \ --tasks ./self-bench-tasks \ --jobs ./matrix-jobs \ --model gpt-5.6-sol \ --model gpt-5.6-luna ``` ### Flags Outer self-bench `.tar.gz`. Mutually exclusive with `--tasks`. Present in usage and parsing; omitted from the `--help` options list. Directory whose child folders each contain `task.toml`. Mutually exclusive with `--export`. Job root. Created if missing. Holds Harbor job directories, optional `tasks/` materialization, and `summary.json`. Harbor executable. Default: `harbor`. `modal` or `docker`. Default: `modal`. Any other value throws `--environment must be docker or modal`. Positive integer of in-flight trials. Default: `3`. Repeatable. Allowed values: `gpt-5.6-sol`, `gpt-5.6-terra`, `gpt-5.6-luna`. Omit to run all three. Codex ChatGPT `auth.json` used only when `OPENAI_API_KEY` is unset. Default path: `~/.codex/auth.json`. `--jobs` is required. `--concurrency` must be a positive integer. An empty model list (API use of `runMatrix`) throws `provide at least one evaluation model`. ### Authentication | Condition | Behavior | | --- | --- | | `OPENAI_API_KEY` set (non-empty after trim) | Harbor inherits the process environment. `summary.json` records `auth: "openai-api-key"`. `--auth` is ignored. | | `OPENAI_API_KEY` unset | Runner reads `--auth` or `~/.codex/auth.json`. File must be ChatGPT subscription JSON: `auth_mode` is `chatgpt` and `tokens` is an object. API-key-mode files are rejected. | | Subscription child env | `OPENAI_API_KEY` is deleted. `CODEX_FORCE_AUTH_JSON=1` and `CODEX_AUTH_JSON_PATH` point at the auth file. `summary.json` records `auth: "codex-subscription"`. | Auth is checked once before any trial. A missing or invalid subscription file aborts the process; it is not recorded as a per-trial exception. ### Harbor invocation Each missing or incomplete trial runs: ```bash harbor run \ --path \ --agent codex \ --model \ --ak version=0.146.1 \ --ak reasoning_effort=high \ --env \ --job-name \ --jobs-dir \ --n-concurrent 1 \ --max-retries 1 \ --delete \ --yes \ --quiet ``` Job names are `${taskId}-${model}`, lowercased, with characters outside `[a-z0-9_.-]` replaced by `-`. The subprocess budget is four hours (`timeoutMs: 14400000`); a timeout SIGTERMs Harbor, then SIGKILLs after five seconds. `--environment` selects Harbor's execution backend only. It does not change the compiled Dockerfiles. Pairing details for Docker versus Modal are on [Execution backends](/execution-backends). ### Pass / fail `summarizeResult` reads the first Harbor `trial_results[]` entry, or the trial object itself. | Outcome | Rule | | --- | --- | | `passed: true` | No exception **and** at least one numeric reward **and** (`rewards.reward >= 1` when that key exists, otherwise every numeric reward `>= 1`) | | `passed: false` with `exception` | `exception_info.message`, string `exception`, or a thrown Harbor/process error | | `passed: false` without exception | Rewards object empty, or a required reward `< 1` | | Diagnostic zeros | Allowed only when `rewards.reward` is present and `>= 1` (for example `fail_to_pass_exit_code: 0`) | If Harbor exits and `/` still has no finished trial, the summary is `Harbor produced no result (exit N)` plus the last 1,000 characters of stderr (or stdout). Spawn failures (Harbor missing from `PATH`) become the same `exception` field. ### Outputs Progress is printed on **stderr**: ```text [3/9] gpt-5.6-sol task-a: passed [4/9] gpt-5.6-terra task-a: failed [5/9] gpt-5.6-luna task-a: error ``` Status is `passed`, `failed`, or `error` (`exception` set). ```json title="stdout" { "trials": 9, "passed": 6, "failed": 3 } ``` `/summary.json` is rewritten at the end of every successful `runMatrix` call (`schemaVersion` 1): Always `1`. Always `codex`. Always `0.146.1`. `openai-api-key` or `codex-subscription`. Always `high`. Models that were scheduled. Discovered Harbor task directories. One object per scheduled pair: `taskId`, `model`, `jobName`, `passed`, `rewards`, optional `exception`. Harbor's own per-job tree is `//` (`result.json` plus a trial subdirectory with optional `verifier/test-stdout.txt` and `verifier/test-stderr.txt`). ## Resume completed jobs Re-run the same `--jobs` directory after a restart. Resume is per deterministic job name, not per export hash. ```mermaid stateDiagram-v2 [*] --> InspectJob InspectJob --> Reuse: finished trial result.json InspectJob --> Archive: missing, unfinished, or finished_at is null Archive --> HarborRun: rename to job.incomplete- HarborRun --> Summarize: trial result present HarborRun --> FailedTrial: no result or spawn error Reuse --> Summarize ``` | On disk | Action | | --- | --- | | One finished trial (or a single aggregate `trial_results` entry) | Skip Harbor. Summarize the stored trial. | | Job directory with `finished_at: null` and no usable trial | Treat as missing. `tryReadHarborJobResult` returns nothing. | | Incomplete or leftover directory | Rename to `.incomplete-` before starting Harbor. | | No directory | Start Harbor. | | `/tasks/` already has `task.toml` dirs | Skip `--export` extraction. | Multiple trial subdirectories plus a one-element aggregate `trial_results` fall back to the aggregate. Multiple trial directories without that aggregate throw and fail the process (not recorded as a soft trial exception). Raw `harbor run` with several `--model` flags does **not** use this resume logic. ## Smoke Harbor agent adapters `self-bench-agent-smoke` runs Harbor with `--install-only` for every adapter in the pinned Harbor roster (38 names, including `oracle`, `nop`, `codex`, and `pi`). It does not score tasks and does not pass `--model` or Codex `--ak` flags. ```bash self-bench-agent-smoke \ --task ./self-bench-tasks/TASK_ID \ --jobs ./adapter-jobs \ --environment modal \ --concurrency 4 ``` `--task` must be one Harbor task directory (the folder that contains `task.toml`). `--jobs` is required. Expanded Harbor task directory. Job root. Reports land in `/adapter-smoke/`. Harbor executable. Default: `harbor`. `modal` or `docker`. Default: `modal`. Positive integer of in-flight adapter checks. Default: `4`. Each adapter is invoked as: ```bash harbor run \ --path \ --agent \ --env \ --job-name install--<8-char-uuid> \ --jobs-dir \ --install-only \ --n-concurrent 1 \ --max-retries 0 \ --delete \ --yes \ --quiet ``` `installed` is `exitCode === 0`. Timeout is one hour. Harbor job names include a random suffix, so **resume is the JSON report**, not the Harbor job directory. | Path | Meaning | | --- | --- | | `/adapter-smoke/.json` | Per-adapter result. If the file exists, the adapter is not re-run. Written with exclusive create (`wx`). | | `/adapter-smoke/summary.json` | `schemaVersion: 1`, `mode: "install-only"`, `environment`, counts, and the full `results` array | ```json title="stdout" { "adapters": 38, "installed": 36, "failed": 2 } ``` Each result includes `agent`, `installed`, `exitCode`, `jobName`, and `outputTail` (last 2,000 characters of combined stdout/stderr). `oracle`, `nop`, `acp`, `terminus-2`, `claude-code`, `copilot-cli`, `aider`, `cline-cli`, `codex`, `cortex-code`, `cursor-cli`, `gemini-cli`, `antigravity-cli`, `antigravity-sdk`, `rovodev-cli`, `goose`, `grok-build`, `hermes`, `kimi-code`, `kimi-cli`, `langgraph`, `deerflow`, `mini-swe-agent`, `nemo-agent`, `swe-agent`, `opencode`, `mimo`, `openclaw`, `openhands`, `openhands-sdk`, `pi`, `qwen-coder`, `devin`, `trae-agent`, `computer-1`, `eve`, `dspy-rlm`, `vibe`. ## Jobs directory layout After a matrix run that materialized an export: :::files matrix-jobs/ ├── summary.json ├── tasks/ │ └── TASK_ID/ │ ├── task.toml │ ├── instruction.md │ ├── environment/ │ ├── tests/ │ └── solution/ ├── task-id-gpt-5.6-sol/ │ ├── result.json │ └── / │ ├── result.json │ └── verifier/ └── task-id-gpt-5.6-sol.incomplete-/ # only after an interrupted retry ::: Smoke reports are isolated under `adapter-smoke/` in the same `--jobs` root if you reuse it. Prefer a separate jobs directory so matrix `summary.json` and adapter reports do not share a tree by accident. ## Errors | Signal | Cause | | --- | --- | | `--jobs is required` | Missing jobs directory flag | | `provide exactly one of exportPath or tasksPath` | Both or neither of `--export` / `--tasks` | | `found no Harbor tasks in …` | No child directory with `task.toml`, or export `tasks/*.tar.gz` produced none | | `--model must be one of gpt-5.6-sol, gpt-5.6-terra, gpt-5.6-luna` | Unknown `--model` | | `--environment must be docker or modal` | Typo or unsupported backend | | `--concurrency must be a positive integer` | Zero, negative, or non-integer | | `… does not contain a ChatGPT subscription token set` | Subscription fallback file is missing `auth_mode: "chatgpt"` + `tokens` | | `Harbor produced no result (exit N)` | Harbor exited without a finished trial (including the 4-hour kill) | | `invalid Harbor result for task/model` | Trial payload is not an object | | `expected one Harbor trial result …` | Ambiguous on-disk job layout (process abort) | | Spawn `harbor` error in `exception` | Executable missing or not runnable | | `downloaded export failed its SHA-256 integrity check` | `self-bench download` / `--output` only — not `self-bench-eval` | Harbor `--delete` is always passed. Interrupted Docker networks or address-pool exhaustion are infrastructure failures; see [Troubleshooting](/troubleshooting). Drop rejected tasks in the review UI and download a cleaned archive before evaluating if the exported set is larger than you want to score. See [Review exports](/review-exports). ## Next Flag lists and stdout contracts for `self-bench-eval`, `self-bench-agent-smoke`, and the other companion binaries. Outer `tar.gz` layout, `manifest.json`, per-task `harbor-task` contents, and SHA-256 fields. What a coding-agent trial mounts, what stays in `tests/` and `solution/`, and oracle-only gold. Docker versus Modal sandboxes and how Harbor `--env` pairs with the compiled images. `OPENAI_API_KEY` precedence, Codex `auth.json` shape, and Modal tokens. 409 export-not-ready, SHA-256 mismatch, Harbor infrastructure errors, leftover Docker resources. --- ## 13. Review exports > Open the API-hosted review UI or a local archive, inspect accepted tasks, drop tasks, and download a cleaned tar.gz. - Page Markdown: https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc/pages/13-review-exports.md - Generated: 2026-08-15T02:09:06.992Z ### Source Files - `review/src/App.tsx` - `review/src/export-loader.ts` - `review/src/types.ts` - `review/src/export-loader.test.ts` - `src/api.ts` --- title: "Review exports" description: "Open the API-hosted review UI or a local archive, inspect accepted tasks, drop tasks, and download a cleaned tar.gz." --- The review UI is a React SPA built from `review/` and served by the self-bench API at `GET /`. It lists Temporal workflows, downloads `GET /v1/runs/:runId/export` when an export exists, or opens a local `.tar.gz` / `.tgz`, then lets you mark accepted Harbor tasks for removal and download `selfbench-{runId}-cleaned.tar.gz` in the browser. Drops are session state only: the API has no review-write or cleaned-export upload route. Exports include repository snapshots, held-out tests, and reference solutions. They are unencrypted. Keep the original and cleaned archives private. This page is the operator review surface. Independent model coupling review during generation is a different gate; it already finished before a task appears in the export. ## Prerequisites - A running API (`self-bench up`, Compose `api`, or `node dist/api-main.js`) if you want the hosted UI and run list. - `SELFBENCH_API_TOKEN` when the API is configured with a token. Compose sets `SELFBENCH_API_HOST=0.0.0.0` and defaults the token to `selfbench-local` unless you export another value. - A completed run (`phase` `complete` with an `export` artifact) or a previously downloaded export archive. - A browser. There is no `self-bench review` CLI command. `GET /` and `GET /assets/*` are served before Bearer auth. Listing runs and downloading an export still require `Authorization: Bearer …` when `SELFBENCH_API_TOKEN` is set. Paste that token into **API token (session only)**; the field is React state and is not persisted. ## Open the review UI ```bash self-bench up --backend modal export SELFBENCH_API_URL=http://127.0.0.1:8080 ``` Compose publishes the API on `127.0.0.1:8080`. The image serves the Vite build from `dist/review` next to `dist/api.js`. Browse to `http://127.0.0.1:8080/`. The page title is **SelfBench Review**. On load it calls `GET /v1/runs`. If list/export calls return `401`, paste the same value as `SELFBENCH_API_TOKEN` into the password field, then click **Refresh runs**. Use **Open gzip** on a file with extension `.gz` or `.tgz` (accept `application/gzip`). The UI does not call `/v1/runs/:runId/export` for that file. The run list still queries the API when the page can reach it. ```bash self-bench download RUN_ID ./self-bench-evals.tar.gz ``` Then open that file in the UI. CLI download verifies `x-content-sha256`; the file picker does not. ```bash bun run dev:api bun run dev:worker bun run dev:review ``` `review/vite.config.ts` proxies `/v1` to `http://127.0.0.1:8080`. Use this pair for source UI work: `tsx src/api-main.ts` resolves review assets relative to `src/`, and `src/review` is not the built SPA. Production and the Compose image serve `dist/review` from the compiled API. ## Two intake paths | Intake | How | Status header | Task list source | | --- | --- | --- | --- | | Completed run | Click a row. Parallel `GET /v1/runs/:runId` and `GET /v1/runs/:runId/export`. | API `phase` (for example `complete`) | Inner `tasks/*.tar.gz` from the export | | Local gzip | **Open gzip** | Synthetic phase `local export` | Same archive layout | Opening a run that has no `export` yet fails with **Export is not ready (409)** (`{"error":"run export is not ready"}`). The left list still renders Temporal `status.name` values such as `RUNNING` or `COMPLETED`; that is workflow status, not the SelfBench `phase` field. The UI does not render per-candidate rejection reasons from `GET /v1/runs/:runId`. Use `self-bench status RUN_ID` or that JSON status payload for `rejected` / `infrastructure_failed` rows. ```mermaid flowchart LR subgraph Browser["Review SPA"] App["App"] Loader["loadExport / buildCleanExport"] end subgraph API["self-bench API"] Assets["GET / and GET /assets/*"] List["GET /v1/runs"] Status["GET /v1/runs/:runId"] ExportRoute["GET /v1/runs/:runId/export"] end subgraph Store["Artifact store"] Original["selfbench-{runId}.tar.gz"] end LocalFile["Local .tar.gz / .tgz"] Cleaned["selfbench-{runId}-cleaned.tar.gz"] Assets --> App App -->|"Bearer when token set"| List App --> Status App --> ExportRoute ExportRoute --> Original Original --> Loader LocalFile --> Loader Loader --> Cleaned ``` ## Inspect an accepted task After an export loads, the summary shows `manifest.repository.url`, the run id, `phase`, `manifest.acceptedCount`, and the first 12 characters of `manifest.repository.commit`. The task list is every inner archive named `tasks/{taskId}.tar.gz` that `manifest.json` lists. Each row is **keep** or **marked for removal**. Selecting a row opens a panel that decodes text files after stripping the `harbor-task/` prefix: | Panel block | File | Fallback | | --- | --- | --- | | Header (`difficulty` · `repo`) | `definition.json` fields | `unknown` and empty repo | | Prompt | `definition.json` `prompt` | `instruction.md`, else `No prompt` | | Task definition | `definition.json` | `Missing definition.json` | | Gold patch | `solution/gold.patch` | `Missing patch` | | Held-out test patch | `tests/test.patch` | `Missing patch` | The authoring sandbox tars only `harbor-task/` after compiling `instruction.md`, `task.toml`, environment/verifier Dockerfiles, patches, and `.selfbench-manifest.json`. `definition.json` is stored as a separate authoring artifact and is often absent from the compiled task archive. When it is missing, read the prompt from `instruction.md` (the compiler writes `definition.prompt` there) and treat the definition block as absent rather than as a corrupt export. The loader also decodes `task.toml`, `.selfbench-manifest.json`, `tests/dependency-setup.patch`, and any other `*.json` path, but the panel does not render them. Environment snapshots such as `environment/repo.tar.gz` stay binary. Patches render with `@pierre/diffs` `FileDiff`: split view above 900px, unified below, light/dark from `prefers-color-scheme`. If the renderer throws, the raw patch text is shown. Copied from `manifest.json`. Generated exports use `1`. Copied from `manifest.json`. Becomes the cleaned download basename. Number of accepted tasks in the loaded archive. The cleaned download rewrites this to the kept count. Inner archive name without `.tar.gz`. SHA-256 of that inner `tasks/{taskId}.tar.gz`. Cleaning keeps the original digest for every kept task. ## Drop tasks and download a cleaned archive **Mark for removal** adds the `taskId` to an in-memory `Set`. **Keep task** removes it. Changing runs or opening another gzip clears the set. Nothing is written to Temporal or the artifact store. The primary button is **Download export copy** when the set is empty, or **Download cleaned export (N)** when N tasks remain. Both paths call `buildCleanExport` and trigger a browser download named `selfbench-{runId}-cleaned.tar.gz`. ```bash mkdir -p ./cleaned && tar -tzf ./selfbench-RUN_ID-cleaned.tar.gz jq '{runId, acceptedCount, tasks: [.tasks[].taskId]}' \ <(tar -xOf ./selfbench-RUN_ID-cleaned.tar.gz manifest.json) ``` Expect `manifest.json` plus one `tasks/{taskId}.tar.gz` per kept id. `acceptedCount` equals the kept length. `candidateCounts`, `repository`, and `version` are unchanged copies of the source manifest. What cleaning changes: | Field / entry | Behavior | | --- | --- | | `acceptedCount` | Set to the number of kept tasks (may be `0`) | | `manifest.tasks` | Filtered to kept ids; original `sha256` values retained | | `tasks/{taskId}.tar.gz` | Original gzip bytes copied; not recompressed | | Removed task archives | Omitted | | `candidateCounts` | Not rewritten | | Server artifact | Unchanged | The cleaned file is a new gzip (new tar mtimes). It does not carry `x-content-sha256`. `self-bench-eval --export` accepts it because the outer layout is the same as a generated export. Self-bench has no remote deletion route. Cleaning is a local republish step. To discard a whole run's artifacts, delete the local artifact volume or the GCS run prefix with operator tooling. ## API surface the UI uses :::endpoint GET / serve the review SPA Unauthenticated. Maps `/` to `dist/review/index.html` (`cache-control: no-cache`). Paths under `/assets/` map into the same directory with `public, max-age=31536000, immutable`. Paths that escape that root return `400` `invalid asset path`. Missing files return `404` `asset not found`. Content types: `.html` → `text/html; charset=utf-8`, `.css` → `text/css; charset=utf-8`, `.js` → `text/javascript; charset=utf-8`. ::: :::endpoint GET /v1/runs list workflows Requires Bearer auth when `SELFBENCH_API_TOKEN` is set. Returns Temporal executions with `WorkflowType = 'selfBenchRunWorkflow'`. ```json [ { "runId": "sb-20260814120000-abcd1234", "status": "COMPLETED", "startedAt": "2026-08-14T12:00:00.000Z", "closedAt": "2026-08-14T14:10:00.000Z" } ] ``` ::: :::endpoint GET /v1/runs/:runId read run status `:runId` must match `^[a-z0-9][a-z0-9-]{2,62}$`. Used with the export GET when you click a run. `queued` · `discovering` · `authoring` · `exporting` · `complete` · `blocked` · `failed` · `cancelled` Present only after `buildExport` stored the archive. Fields: `uri`, `sha256`, `sizeBytes`, `contentType`. ::: :::endpoint GET /v1/runs/:runId/export download the generated archive `409` when `export` is missing. On success: | Header | Value | | --- | --- | | `content-type` | Artifact `contentType` (`application/gzip`) | | `content-length` | `sizeBytes` | | `content-disposition` | `attachment; filename="selfbench-{runId}.tar.gz"` | | `x-content-sha256` | Artifact SHA-256 | The UI feeds the response blob into `loadExport`. The CLI `self-bench download` also consumes this route and refuses the file if the body hash does not match the header. ::: ## Loader constraints `loadExport` gunzips the outer archive, requires a top-level `manifest.json`, then for every `manifest.tasks[]` entry requires `tasks/{taskId}.tar.gz`, gunzips that archive, and indexes files after removing a leading `harbor-task/` prefix. Extra members not listed in the manifest are ignored. ```text manifest.json tasks/ ├── task-one.tar.gz └── task-two.tar.gz ``` ```json { "schemaVersion": 1, "runId": "sb-20260814120000-abcd1234", "candidateCounts": { "easy": 10, "medium": 10, "hard": 10 }, "repository": { "url": "https://github.com/example/repo", "commit": "0123456789abcdef0123456789abcdef01234567" }, "version": { "selfbenchCommit": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", "executionBackend": "modal", "sandboxImage": "node:22-bookworm", "schema": 1 }, "acceptedCount": 2, "tasks": [ { "taskId": "task-one", "sha256": "aa…64 hex…" }, { "taskId": "task-two", "sha256": "bb…64 hex…" } ] } ``` Hard failures (shown in the red banner): | Message | Cause | | --- | --- | | `Unable to list runs ({status})` | `GET /v1/runs` not OK (often `401`) | | `Unable to load run ({status})` | Status GET failed | | `Export is not ready ({status})` | Export GET failed; `409` means no artifact yet | | `export is missing manifest.json` | Not a self-bench outer archive | | `export is missing tasks/{taskId}.tar.gz` | Manifest/archive mismatch | ## After you publish a cleaned export Outer tar.gz layout, manifest fields, and what the archive excludes. Point `self-bench-eval --export` at the cleaned file or extract one `harbor-task`. ## Troubleshooting | Symptom | What to check | | --- | --- | | **No runs found** | API unreachable, wrong token, or no `selfBenchRunWorkflow` executions. Confirm `GET http://127.0.0.1:8080/healthz` returns `{"ok":true}`. | | **401 on refresh** | Paste `SELFBENCH_API_TOKEN`. Compose default is `selfbench-local` if you did not set one. | | **Export is not ready (409)** | Wait until `self-bench status RUN_ID` shows `phase: "complete"` and an `export` object. Blocked or failed runs never grow an export. | | **Missing definition.json** | Expected for compiler-produced Harbor tasks. Use **Prompt** from `instruction.md`. | | **Missing patch** | Inner archive lacks `solution/gold.patch` or `tests/test.patch`. | | **Pierre renderer falls back to text** | Invalid or unsupported diff; the raw patch is still shown. | | **Cleaned file rejected by a hash check** | Only the API original has `x-content-sha256`. Re-hash the cleaned bytes yourself if you need a digest. | | **Need the original, not a recopy** | `self-bench download RUN_ID OUTPUT.tar.gz` writes create-only and verifies SHA-256. | ## Related pages `list`, `status`, `cancel`, and SHA-256 `download` against `SELFBENCH_API_URL`. Bearer auth, status codes, export headers, and review-asset serving. `self-bench up` / `down`, Compose services, and the `127.0.0.1:8080` publish. `409` export-not-ready, SHA-256 mismatch, and leftover Docker resources. `bun run dev:review`, `bun test tests review/src`, and `build:review`. --- ## 14. Deploy Temporal Cloud > Split API and worker processes, GCS artifacts, Temporal TLS and API key, Modal tokens, and matching image digest plus task queue. - Page Markdown: https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc/pages/14-deploy-temporal-cloud.md - Generated: 2026-08-15T02:08:52.466Z ### Source Files - `docs/operations.md` - `Dockerfile` - `src/api-main.ts` - `src/temporal/worker-main.ts` - `src/temporal/connection.ts` - `src/config.ts` --- title: "Deploy Temporal Cloud" description: "Split API and worker processes, GCS artifacts, Temporal TLS and API key, Modal tokens, and matching image digest plus task queue." --- Cloud deployments run two processes from the same `Dockerfile` image: `node dist/api-main.js` as a request-oriented HTTP service, and `node dist/temporal/worker-main.js` as a long-running Temporal worker. Both connect to Temporal Cloud with TLS plus `SELFBENCH_TEMPORAL_API_KEY`, share one `SELFBENCH_TASK_QUEUE`, and store run artifacts in GCS. The worker owns Modal, GitHub, model, and Harbor credentials and must stay up for the 14-day workflow execution timeout. This repository defines the application boundary, not turnkey cloud infrastructure. Project, region, ingress, IAM, GCS, and Temporal provisioning stay deployment-specific. ```mermaid flowchart LR subgraph Clients CLI["self-bench CLI"] end subgraph ApiProcess["API process"] API["node dist/api-main.js"] end subgraph WorkerProcess["Worker process"] Worker["node dist/temporal/worker-main.js"] Harbor["harbor --env modal"] end subgraph TemporalCloud["Temporal Cloud"] NS["namespace + TLS + API key"] TQ["SELFBENCH_TASK_QUEUE"] end subgraph Gcs["GCS"] Bucket["gs://bucket/prefix"] end subgraph Modal["Modal"] Sandboxes["Modal sandboxes"] end CLI -->|"Bearer SELFBENCH_API_TOKEN"| API API -->|"start / query / cancel"| NS API -->|"put / get"| Bucket Worker -->|"poll"| TQ Worker -->|"put / get"| Bucket Worker --> Harbor Worker --> Sandboxes ``` Do not run the worker on a scale-to-zero request platform. `self-bench run` starts `selfBenchRunWorkflow` with `workflowExecutionTimeout: "14 days"`. Work pauses when the worker stops; Temporal history remains until a matching worker polls again. ## What you provision | Surface | Role | self-bench does not create | | --- | --- | --- | | Temporal Cloud namespace | Workflow history, visibility, task queue | Namespace, TLS, API key, gRPC address | | GCS bucket + prefix | Provenance, checkpoints, exports | Bucket, IAM, lifecycle rules | | API container | HTTP + Temporal client | Cloud Run (or equivalent) service, ingress | | Worker container | Temporal poller + Modal + Harbor | Always-on compute | | Modal account | Disposable sandboxes and Harbor `--env modal` | Tokens / workspace | | Secrets | API token, Temporal API key, Modal tokens, `GH_TOKEN`, `OPENAI_API_KEY` | Secret manager | The local analog of the process split is `compose.yaml`: same `selfbench:local` image, API default command, worker `command: ["node", "dist/temporal/worker-main.js"]`. Cloud replaces Compose Postgres/Temporal with Temporal Cloud and the named `artifacts` volume with GCS. ## Image and process split Build `Dockerfile`. The image is `node:22-bookworm` with Harbor `harbor[modal]==0.20.1.dev202608040148`, `gh`, `uv`, and a Docker CLI (used only if a process is configured for Docker). Default `USER` is `node`. Default `CMD` is the API. ```bash title="Build" docker build -t selfbench: . ``` ```bash title="API command" node dist/api-main.js # package.json: bun run start:api ``` ```bash title="Worker command" node dist/temporal/worker-main.js # package.json: bun run start:worker ``` | Process | Bind / poll | Credentials it should hold | | --- | --- | --- | | API | `SELFBENCH_API_HOST`:`SELFBENCH_API_PORT` (cloud: `0.0.0.0:8080`) | `SELFBENCH_API_TOKEN`, Temporal TLS + API key, GCS | | Worker | Temporal namespace / task queue | Same Temporal + GCS, plus Modal, `GH_TOKEN`, `OPENAI_API_KEY` | The API is a regular HTTP service suitable for Cloud Run. `src/api-main.ts` loads config, calls `startApi`, logs `SelfBench API listening on http://:`, and stops on `SIGINT` / `SIGTERM`. The worker strips empty `MODAL_TOKEN_ID` / `MODAL_TOKEN_SECRET` values, then fails startup with `docker info` and `docker compose version` if `SELFBENCH_EXECUTION_BACKEND` or `SELFBENCH_HARBOR_ENVIRONMENT` is `docker`. Cloud workers must set both to `modal` so they do not need a Docker socket. After connect, the worker logs: ```text SelfBench worker polling / with activity concurrency ``` Keep API and worker on the **same image digest**. The API starts workflows; the worker loads `workflow.js` and activity implementations from that image. Divergent digests can desynchronize workflow code, Harbor version, and default sandbox image. ## Temporal Cloud connection Both processes call `Connection.connect` / `NativeConnection.connect` with `address`, `tls`, and optional `apiKey`. There is no client-certificate or custom CA field in config. gRPC address of the Temporal frontend. Default `127.0.0.1:7233` is the local Compose listener, not Temporal Cloud. Temporal namespace. Default `default`. Must be the string `true` or `false`. Default `false`. Temporal Cloud requires `true`. Passed through only when set. Omit locally; required for Temporal Cloud API-key auth. Queue the API uses in `client.workflow.start` and the worker polls. Default `selfbench-dev`. Set the **same** address, namespace, TLS flag, API key, and task queue on both processes. Use a dedicated cloud queue name (not the `selfbench-dev` default) so a local `self-bench up` worker cannot steal cloud tasks. The API starts `selfBenchRunWorkflow` with `workflowId` equal to the run ID. `GET /v1/runs` lists executions with Temporal visibility query `WorkflowType = 'selfBenchRunWorkflow'`. ## GCS artifacts Set `SELFBENCH_ARTIFACT_BACKEND=gcs` on both processes. `loadConfig` then requires `SELFBENCH_GCS_BUCKET` or throws `SELFBENCH_GCS_BUCKET is required for GCS`. `GcsArtifactStore` constructs `@google-cloud/storage` `new Storage()` with no explicit credentials. The process identity must already be authorized (attached service account or Application Default Credentials). Object names are `/` after stripping leading and trailing slashes from the prefix. | Key | Default | Constraint | | --- | --- | --- | | `SELFBENCH_ARTIFACT_BACKEND` | `local` | Cloud: `gcs` | | `SELFBENCH_GCS_BUCKET` | unset | Required when backend is `gcs` | | `SELFBENCH_GCS_PREFIX` | `selfbench` | App-level confinement; also grant IAM only to this prefix | Writes use `ifGenerationMatch: 0` (create-if-absent). A second put of the same object is accepted only when the stored `sha256` metadata matches. Conflicting contents throw `artifact already exists with different contents: gs:///`. Reads reject URIs outside the configured bucket or prefix (`artifact is outside configured bucket`). Keys may not be empty, start with `/`, or contain `..`. :::files gs://// └── runs// ├── input/provenance.jsonl ├── discovery/wave-/shard-/... ├── validation///attempt-/... └── export.tar.gz ::: There is no HTTP delete route. Remove GCS prefixes with operator tooling. Exports contain source snapshots, held-out tests, and reference solutions. They are sensitive and unencrypted. Artifact refs carry `sizeBytes` and SHA-256; `get` / `openRead` verify both. ## API environment Bind beyond loopback only with a token. `SELFBENCH_API_HOST` values that skip the token check are `127.0.0.1`, `::1`, and `localhost`. `0.0.0.0` requires `SELFBENCH_API_TOKEN` or startup throws `SELFBENCH_API_TOKEN is required when the API listens beyond loopback`. ```text SELFBENCH_API_HOST=0.0.0.0 SELFBENCH_API_PORT=8080 SELFBENCH_API_TOKEN=... SELFBENCH_ARTIFACT_BACKEND=gcs SELFBENCH_GCS_BUCKET=... SELFBENCH_GCS_PREFIX=selfbench SELFBENCH_TEMPORAL_ADDRESS=... SELFBENCH_TEMPORAL_NAMESPACE=... SELFBENCH_TEMPORAL_API_KEY=... SELFBENCH_TEMPORAL_TLS=true SELFBENCH_TASK_QUEUE=selfbench-prod ``` `GET /healthz` is unauthenticated and returns `{ "ok": true }`. Every other `/v1/*` route requires `Authorization: Bearer ` when the token is configured (timing-safe compare). `GET /` and `/assets/*` serve the bundled review UI without that check. Optional: `SELFBENCH_BUILD_COMMIT` (40-char hex). When set, `POST /v1/runs` records it as `version.selfbenchCommit` instead of the CLI-submitted SHA. Do not put `OPENAI_API_KEY`, `GH_TOKEN`, or Modal tokens on the API process. ## Worker environment Share Temporal, task queue, and GCS with the API, then add execution and credentials: ```text SELFBENCH_ARTIFACT_BACKEND=gcs SELFBENCH_GCS_BUCKET=... SELFBENCH_GCS_PREFIX=selfbench SELFBENCH_TEMPORAL_ADDRESS=... SELFBENCH_TEMPORAL_NAMESPACE=... SELFBENCH_TEMPORAL_API_KEY=... SELFBENCH_TEMPORAL_TLS=true SELFBENCH_TASK_QUEUE=selfbench-prod SELFBENCH_EXECUTION_BACKEND=modal SELFBENCH_HARBOR_ENVIRONMENT=modal SELFBENCH_ACTIVITY_CONCURRENCY=20 SELFBENCH_MODAL_APP=selfbench MODAL_TOKEN_ID=... MODAL_TOKEN_SECRET=... GH_TOKEN=... OPENAI_API_KEY=... ``` | Key | Default | Notes | | --- | --- | --- | | `SELFBENCH_EXECUTION_BACKEND` | `docker` | Cloud: `modal` | | `SELFBENCH_HARBOR_ENVIRONMENT` | same as execution backend | Cloud: `modal` so Harbor runs `--env modal` | | `SELFBENCH_ACTIVITY_CONCURRENCY` | `20` when Modal, else `1` | Integer 1–100; `maxConcurrentActivityTaskExecutions` | | `SELFBENCH_MODAL_APP` | `selfbench` | `ModalClient.apps.fromName(..., { createIfMissing: true })` | | `SELFBENCH_MODAL_ENVIRONMENT` | unset | Optional Modal environment name | | `SELFBENCH_MODAL_IMAGE` | `node:22-bookworm` | Registry image the worker layers tools onto | | `OPENAI_API_KEY` | unset | Preferred model auth; wins over subscription JSON | | `SELFBENCH_PI_AUTH_JSON` / `SELFBENCH_CODEX_AUTH_JSON` | unset | Compatibility fallbacks only | | `GH_TOKEN` | unset | Preferred GitHub read token; else `gh auth token` | `removeEmptyModalCredentialOverrides()` deletes `MODAL_TOKEN_ID` and `MODAL_TOKEN_SECRET` when they are empty or whitespace so they cannot override a mounted `~/.modal.toml`. Cloud workers have no profile mount; set both tokens to non-empty secret values. Use separate least-privilege service accounts: API and worker need GCS object access under the configured prefix; only the worker needs Modal, GitHub, and model secrets. ## Matching digest and task queue | Must match on API and worker | Why | | --- | --- | | Image digest | Workflow + activity code, Harbor version, default Modal image | | `SELFBENCH_TASK_QUEUE` | API `workflow.start({ taskQueue })` vs worker `Worker.create({ taskQueue })` | | Temporal address, namespace, TLS, API key | Both sides of the same Cloud namespace | | GCS bucket + prefix | API writes provenance; worker reads it and writes exports | A queue mismatch leaves new runs in Temporal with no poller. An image mismatch can break deterministic workflow replay. A GCS prefix mismatch makes the worker fail to read `runs//input/provenance.jsonl`. ## Deploy Create a TLS-enabled Temporal Cloud namespace and an API key. Create a GCS bucket and a prefix (default `selfbench`). Attach distinct service accounts that can only read/write that prefix. Build `Dockerfile` and record the digest. Pass `SELFBENCH_BUILD_COMMIT=<40-char sha>` as a build-arg if you want that SHA baked into `dist/build-metadata.js` during `bun run build`. Deploy the digest with default `CMD` (`node dist/api-main.js`), port `8080`, and the API environment above. Confirm `GET /healthz` returns `{ "ok": true }` and the process log includes `SelfBench API listening on http://0.0.0.0:8080`. Deploy the **same digest** with command `node dist/temporal/worker-main.js` on always-on compute. Confirm the poll log shows the intended namespace, queue, and concurrency. Do not scale this service to zero. From an operator machine that can reach the API: ```bash export SELFBENCH_API_URL=https:// export SELFBENCH_API_TOKEN=... self-bench list ``` `SELFBENCH_API_URL` defaults to `http://127.0.0.1:8080`. The CLI sends `Authorization: Bearer $SELFBENCH_API_TOKEN` when that variable is set. ```bash curl -sS "$SELFBENCH_API_URL/healthz" self-bench list ``` ```json { "ok": true } ``` ```json [] ``` An empty `self-bench list` array is success: the API authenticated, connected to Temporal Cloud, and ran the visibility query. ## Verification signals | Check | Expected | | --- | --- | | `GET /healthz` | `200` `{ "ok": true }` | | API log | `SelfBench API listening on http://0.0.0.0:8080` | | Worker log | `SelfBench worker polling / with activity concurrency 20` | | `self-bench list` with token | JSON array (possibly empty) | | `self-bench list` without token | `unauthorized` when the API token is configured | | Submit a run | `202` `{ "runId": "..." }`; worker activity starts; GCS objects appear under the prefix | Closing the waiting CLI does not cancel the workflow. `self-bench cancel RUN_ID` requests Temporal cancellation through the API. ## Failure modes | Symptom | Likely cause | | --- | --- | | Process exits: `SELFBENCH_API_TOKEN is required when the API listens beyond loopback` | `SELFBENCH_API_HOST=0.0.0.0` without a token | | Process exits: `SELFBENCH_GCS_BUCKET is required for GCS` | Backend `gcs` and bucket unset | | Worker exits on `docker info` | `SELFBENCH_EXECUTION_BACKEND` or `SELFBENCH_HARBOR_ENVIRONMENT` is `docker` | | Runs stay queued / list shows `RUNNING` with no candidate progress | Queue mismatch, worker down, or worker still connecting | | `artifact is outside configured bucket` | API and worker disagree on bucket or prefix | | `artifact already exists with different contents` | Same object key, different SHA-256 | | `401` `{ "error": "unauthorized" }` | Missing or wrong Bearer token | | Modal auth falls through unexpectedly | Empty `MODAL_TOKEN_*` env vars were not replaced with real secrets | | Export `409` `run export is not ready` | Workflow not in a terminal success path yet | Sandboxes receive a source checkout plus a short-lived model credential. Use self-bench only with repositories you trust to execute. Modal sandboxes are disposable; the worker is not a malware-analysis service. ## Next Compose equivalent of the API/worker split, named volumes, and `--backend modal`. Full `SELFBENCH_*` key list, defaults, loopback token rule, and Temporal fields. `OPENAI_API_KEY` precedence, GitHub token, API Bearer, Modal profile versus tokens. Docker versus Modal concurrency, images, and Harbor environment pairing. `list`, `status`, `cancel`, and SHA-256 download against `SELFBENCH_API_URL`. Export-not-ready, integrity mismatches, and leftover execution resources. --- ## 15. CLI reference > self-bench commands, flags, defaults, run-id rules, wait and download semantics, and command-level errors. - Page Markdown: https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc/pages/15-cli-reference.md - Generated: 2026-08-15T02:09:38.124Z ### Source Files - `src/cli.ts` - `tests/cli.test.ts` - `package.json` - `src/run-wait.ts` - `README.md` --- title: "CLI reference" description: "self-bench commands, flags, defaults, run-id rules, wait and download semantics, and command-level errors." --- `self-bench` is the npm `self-bench` binary (`dist/cli.js`). It starts and stops the packaged Compose stack, submits a tiered Harbor-benchmark run, and talks to `SELFBENCH_API_URL` for list, status, cancel, and SHA-256-verified export download. Discovery, authoring, validation, review, and audit do not run in this process. The package also ships separate companion binaries (`self-bench-eval`, `self-bench-validate`, `self-bench-reaudit`, `self-bench-repair`, `self-bench-agent-smoke`). Those are not subcommands of `self-bench`. Requires Node.js 22 or newer. From a checkout, `bun run cli -- --help` runs the same entrypoint. ## Command inventory | Command | Purpose | Default exit path | | --- | --- | --- | | `self-bench up` | Build (Docker backend only) the local sandbox image, then `docker compose up -d --build` | Prints the stack URL | | `self-bench down` | `docker compose down` on the packaged `compose.yaml` | Named volumes stay | | `self-bench run` | Resolve repo HEAD + origin, upload provenance, `POST /v1/runs` | Prints the 202 payload unless `--wait` / `--output` continue | | `self-bench status RUN_ID` | `GET /v1/runs/:runId` | Pretty-printed JSON | | `self-bench cancel RUN_ID` | `POST /v1/runs/:runId/cancel` | Pretty-printed JSON | | `self-bench list` | `GET /v1/runs` | Pretty-printed JSON array | | `self-bench download RUN_ID OUTPUT.tar.gz` | `GET /v1/runs/:runId/export` with create-only write + SHA-256 check | Prints `{ runId, output }` | | `self-bench help`, `--help`, `-h`, or no args | Print usage | Exit after help | There is no `version` command and no per-command `--help`. `up` and `run` parse flags with `parseArgs({ strict: true })`, so unknown options fail. `down` takes no flags and ignores extra argv. ```bash self-bench --help ``` ```text SelfBench creates durable tiered Harbor evaluations. Usage: self-bench up [--backend docker|modal] [--modal-config PATH] self-bench down self-bench run --repo PATH [--easy-count N] [--medium-count N] [--hard-count N] [--model MODEL] [--run-id ID] [--wait] [--output OUTPUT.tar.gz] self-bench status RUN_ID self-bench cancel RUN_ID self-bench download RUN_ID OUTPUT.tar.gz self-bench list ``` ## Invocation environment Every HTTP command uses: | Variable | Default | Role | | --- | --- | --- | | `SELFBENCH_API_URL` | `http://127.0.0.1:8080` | API origin for run, status, cancel, list, download | | `SELFBENCH_API_TOKEN` | unset | If set, sent as `Authorization: Bearer …` | | `SELFBENCH_BUILD_COMMIT` | unset | Full 40-hex SHA stamped into `up` and `run` when the binary has no baked commit | The packaged Compose file defaults the API token to `selfbench-local` when `SELFBENCH_API_TOKEN` is unset in the host environment. The CLI only sends a Bearer header when `SELFBENCH_API_TOKEN` is a non-empty string. Export the same token the stack is using or authenticated routes return `unauthorized`. `run` also shells out to `git` (origin URL + `HEAD`) and `gh` (`pr list --state merged --limit 500`). `up` / `down` require `docker` with Compose. ## `self-bench up` Starts Postgres, Temporal, the API, and the worker from the package's `compose.yaml` (Compose project name `selfbench`). The chosen backend is written into both sandbox execution and Harbor validation. Sets `SELFBENCH_EXECUTION_BACKEND` and `SELFBENCH_HARBOR_ENVIRONMENT` to the same value. Absolute or relative path to a Modal TOML profile. Allowed only with `--backend modal`. Default when omitted: `~/.modal.toml`. The resolved path is exported as `SELFBENCH_MODAL_CONFIG_PATH` and mounted onto the worker as `/root/.modal.toml`. Behavior by backend: | `--backend` | Extra local work | Worker execution | | --- | --- | --- | | `docker` | `docker build -f Dockerfile.sandbox -t selfbench-sandbox:local` | Host Docker sandboxes; Compose default concurrency is 1 | | `modal` | No sandbox image build | Modal sandboxes; Compose default concurrency is 20 | Both backends then run `docker compose --file /compose.yaml up -d --build` with `SELFBENCH_BUILD_COMMIT` resolved (see [Build commit](#build-commit)). Success prints: ```text SelfBench is running with the backend at http://127.0.0.1:8080 ``` Compose publishes the API on `127.0.0.1:8080` and Temporal on `127.0.0.1:7233`. Named volumes `selfbench_temporal-postgres` and `selfbench_artifacts` persist Temporal history and generated artifacts. ## `self-bench down` Runs `docker compose --file /compose.yaml down`. Containers stop; named volumes are not removed. Extra arguments after `down` are ignored (`--volumes` is not forwarded). Stopping the waiting CLI does not stop the worker. If the worker or stack is down, an already-submitted Temporal workflow pauses until a worker with the same task queue comes back. ## `self-bench run` Local work is repository metadata plus sanitized provenance upload. The API starts a Temporal workflow (`selfBenchRunWorkflow`) whose `workflowId` is the run ID. Workflow execution timeout on the API side is 14 days. Git checkout. Resolved with `path.resolve` (relative paths are cwd-relative). Short flag: `-r`. Easy-tier authoring budget. Medium-tier authoring budget. Hard-tier authoring budget. Sent as `authoringModel`. The CLI does not enumerate allowed models. The API wraps it as `authoring.provider = openai-codex` and `reasoningEffort = high`. Caller-chosen run ID. If omitted, the CLI generates one. Not validated locally; the API regex applies (see [Run IDs](#run-ids)). After submit, poll until a terminal phase. Phase changes go to stderr. Implies wait. After `phase === "complete"`, download the export to this path. Short flag: `-o`. ### Local repository pin `run` executes: ```bash git -C remote get-url origin git -C rev-parse HEAD ``` `origin` must be `git@github.com:owner/repo` (normalized to `https://github.com/owner/repo`) or any URL that already starts with `https://`. Other remotes fail with `unsupported origin URL`. Uncommitted work is never included; only the `HEAD` SHA is sent. GitHub provenance then requires a `github.com` owner/repo URL. A non-GitHub `https://` origin passes the CLI normalizer and fails later in GitHub URL parsing. ### Candidate budgets Each `--*-count` must be a nonnegative integer. The **sum** must be between 1 and 100 inclusive. These values are authoring budgets, not accepted-task targets. Rejected candidates are not replaced. Omitting all three counts leaves the total at 0 and fails. ### Provenance upload The CLI concatenates: 1. Sanitized user messages from Pi (`~/.pi/agent/sessions`), Claude Code (`~/.claude/projects`), and Codex (`~/.codex/sessions`, `~/.codex/archived_sessions`) that mention the checkout or its worktrees. 2. Titles and bodies from up to 500 recent merged, non-draft, non-bot GitHub pull requests that meet the coarse size filter (at least 20 changed lines and 1 changed file). Home directory is `HOME` if set, otherwise `os.homedir()`. If the merged list is empty, `run` throws and does not call the API: ```text no sanitized local-session or GitHub pull-request provenance was found ``` Otherwise it `POST`s newline-delimited JSON to `/v1/provenance?runId=` with `Content-Type: application/x-ndjson`. The API stores that blob under `runs//input/provenance.jsonl` (100 MiB body limit) and returns an artifact reference (`uri`, `sha256`, `sizeBytes`, `contentType`). ### Submit payload `POST /v1/runs` with: ```json { "runId": "sb-20260814153045-a1b2c3d4", "repository": { "url": "https://github.com/example/project.git", "commit": "<40-hex HEAD>" }, "provenance": { "uri": "…", "sha256": "…", "sizeBytes": 1, "contentType": "application/x-ndjson" }, "candidateCounts": { "easy": 10, "medium": 10, "hard": 10 }, "authoringModel": "gpt-5.6-sol", "selfbenchCommit": "<40-hex>" } ``` A successful API response is HTTP 202 `{ "runId": "…" }`. The CLI always prints that object plus counts: ```json { "runId": "sb-20260814153045-a1b2c3d4", "provenanceMessages": 42, "localProvenanceMessages": 30, "githubPullRequestMessages": 12 } ``` Without `--wait` and without `--output`, `run` returns after that print. Closing the process does not cancel the workflow. ```bash export SELFBENCH_API_URL=http://127.0.0.1:8080 export SELFBENCH_API_TOKEN=… self-bench run \ --repo /absolute/path/to/repository \ --easy-count 10 \ --medium-count 10 \ --hard-count 10 \ --model gpt-5.6-sol \ --output ./self-bench-evals.tar.gz ``` ## Run IDs The API accepts only: ```text ^[a-z0-9][a-z0-9-]{2,62}$ ``` That is 3–63 characters, starting with a lowercase letter or digit, then lowercase letters, digits, or hyphens. Uppercase, underscores, and leading hyphens are rejected. The CLI default is: ```text sb--<8 hex chars> ``` built from `Date.toISOString()` with non-digits stripped (first 14 digits) and the first 8 characters of `crypto.randomUUID()`. That form always matches the regex. The same ID is: - the provenance query parameter - the `POST /v1/runs` `runId` - the Temporal `workflowId` Reusing an ID against an existing workflow fails at `POST /v1/runs` (Temporal “already started”); the CLI surfaces the API `error` string. Provenance `POST` happens first, so a failed start can still leave `runs//input/provenance.jsonl` stored. `status`, `cancel`, and `download` interpolate the ID into `/v1/runs/:runId`. The API route itself only matches the same regex; an invalid ID typically returns `not found` rather than a validation message. ## Wait semantics `--output` implies `--wait`. Wait has no timeout flag and no attempt limit. It loops until the status `phase` is terminal. | Setting | Interval | Default | | --- | --- | --- | | Poll interval | `GET /v1/runs/:runId` | 2000 ms | | Phase events | stderr, only when `phase` changes | `{ runId, phase, accepted, rejected }` | | Success | `phase === "complete"` | With `--output`, download; otherwise pretty-print the last status on stdout | | Failure | `blocked`, `failed`, `cancelled` | Throw `SelfBench run ` or `SelfBench run : ` when `error` is a string | A status object without a string `phase` fails with `SelfBench status response is missing its phase`. Worker phases the poller can observe include `queued`, `discovering`, `authoring`, `exporting`, `complete`, `blocked`, `failed`, and `cancelled`. Per-task statuses (`auditing`, `validating`, `reviewing`, `repairing`, and so on) live on the status JSON `tasks` array; the wait line on stderr only reprints `accepted` / `rejected` counts. ```mermaid sequenceDiagram participant CLI as self-bench run participant API as SELFBENCH_API_URL participant WF as Temporal workflow CLI->>CLI: git origin + HEAD CLI->>CLI: local sessions + gh pr list CLI->>API: POST /v1/provenance?runId= API-->>CLI: artifact ref CLI->>API: POST /v1/runs API->>WF: start workflowId = runId API-->>CLI: 202 runId loop every 2s until terminal phase CLI->>API: GET /v1/runs/:runId API-->>CLI: phase, accepted, rejected Note over CLI: stderr only on phase change end alt phase complete and --output set CLI->>API: GET /v1/runs/:runId/export API-->>CLI: gzip + x-content-sha256 CLI->>CLI: wx create, hash stream, unlink on mismatch else phase blocked, failed, or cancelled CLI-->>CLI: throw SelfBench run … end ``` ## Download semantics `self-bench download RUN_ID OUTPUT.tar.gz` and `run --output` share one implementation. 1. `GET {SELFBENCH_API_URL}/v1/runs/{runId}/export` with the optional Bearer token. 2. Non-OK responses must be JSON. The CLI throws `value.error` or `SelfBench API returned `. Before the workflow has an export, the API returns **409** `{ "error": "run export is not ready" }`. 3. A 200 response must include `x-content-sha256` and a body. Missing either throws `SelfBench API returned an export without integrity metadata`. 4. The destination is `path.resolve(OUTPUT)` and opened with flag `wx` (create exclusively). An existing file fails with the filesystem `EEXIST` error; the CLI does not overwrite. 5. Bytes are hashed while written. If the hex digest does not match `x-content-sha256`, the file is deleted and the CLI throws `downloaded export failed its SHA-256 integrity check`. 6. Any other write/hash failure also deletes the partial file. 7. Success prints: ```json { "runId": "sb-20260814153045-a1b2c3d4", "output": "/absolute/path/to/self-bench-evals.tar.gz" } ``` The API also sends `content-disposition: attachment; filename="selfbench-.tar.gz"`. The CLI ignores that name and uses the path you passed. ## `status`, `cancel`, and `list` These commands are JSON passthroughs. They do not wait and do not write files. | Command | Method and path | Typical body | | --- | --- | --- | | `status RUN_ID` | `GET /v1/runs/:runId` | `runId`, `phase`, `requested`, `requestedByDifficulty`, `discovered`, `accepted`, `rejected`, `tasks`, optional `discovery`, `export`, `error` | | `cancel RUN_ID` | `POST /v1/runs/:runId/cancel` | `{ "runId", "cancellationRequested": true }` (HTTP 202) | | `list` | `GET /v1/runs` | Array of `{ runId, status, startedAt, closedAt? }` for `WorkflowType = 'selfBenchRunWorkflow'` | `cancel` requests Temporal cancellation. The wait helper treats `cancelled` as a failed terminal. Failed generation must be resubmitted with a new run ID. ## Build commit `up` and `run` resolve a 40-hex SelfBench commit in this order: 1. `SELFBENCH_BUILD_COMMIT` if set 2. Baked `buildCommit` from package metadata, if it is 40 hex and not all zeros 3. `git -C rev-parse HEAD` If none succeed, the CLI throws: ```text SelfBench was built without commit metadata; set SELFBENCH_BUILD_COMMIT to a full commit SHA ``` The API may override the submitted SHA with its own configured `buildCommit`. ## Command-level errors Failures throw `Error`. There is no top-level catch, so Node prints the message and stack and exits nonzero. | Message / pattern | When | | --- | --- | | `unknown command: ` | First argv is not a known command or help flag | | `--backend must be "docker" or "modal"` | `up --backend` is anything else | | `--modal-config requires --backend modal` | `--modal-config` with Docker backend | | `--repo is required` | `run` without `--repo` / `-r` | | `--easy-count must be a nonnegative integer` (and medium/hard) | Non-integer or negative tier count | | `the total candidate count must be between 1 and 100` | Sum is 0 or greater than 100 | | `unsupported origin URL: …` | `origin` is neither `git@github.com:…` nor `https://…` | | `no sanitized local-session or GitHub pull-request provenance was found` | Empty merged corpus | | `SelfBench was built without commit metadata; …` | No usable SHA for `up`/`run` | | `run ID is required` / `output path is required` | Missing positional args on status, cancel, or download | | `SelfBench API returned ` | Non-OK JSON body without `error` | | `unauthorized` | API token configured and header missing or wrong | | `not found` | Unknown route or run ID that does not match the API path regex | | `run export is not ready` | Download before `export` exists (HTTP 409) | | `SelfBench API returned an export without integrity metadata` | Missing `x-content-sha256` or body | | `downloaded export failed its SHA-256 integrity check` | Stream hash mismatch; output file removed | | `SelfBench run blocked: …` / `failed` / `cancelled` | Wait hit a non-complete terminal phase | | `SelfBench status response is missing its phase` | Poll JSON has no string `phase` | | `Unknown option '…'` | Strict `parseArgs` on `up` or `run` | | `git exited : …` / `gh exited : …` / `docker exited : …` | Subprocess nonzero (stderr or stdout attached) | Filesystem `EEXIST` from `open(..., "wx")` means the `--output` / download path already exists. ## Companion binaries These are separate `bin` entries, not `self-bench` subcommands: | Binary | Role | | --- | --- | | `self-bench-eval` | Harbor Codex matrix over an export or expanded tasks directory | | `self-bench-validate` | Harbor nop and oracle gates | | `self-bench-reaudit` | Coupling re-audit | | `self-bench-repair` | Coupled-test repair | | `self-bench-agent-smoke` | Harbor adapter install/instantiate smoke | ## Related pages Compose services, `--backend`, `--modal-config`, named volumes, and worker-stop behavior. `self-bench run` workflow: GitHub origin, tier budgets, HEAD pin, and `--output`. list, status, cancel, and download against `SELFBENCH_API_URL`. Flags and outputs for eval, validate, reaudit, repair, and agent-smoke. Routes, Bearer auth, status codes, and export headers the CLI consumes. `SELFBENCH_*` keys, loopback token rule, and Temporal fields. Empty provenance, blocked pool, 409 export-not-ready, and SHA-256 mismatch. --- ## 16. Companion commands > Flags, defaults, and outputs for self-bench-eval, self-bench-validate, self-bench-reaudit, self-bench-repair, and self-bench-agent-smoke. - Page Markdown: https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc/pages/16-companion-commands.md - Generated: 2026-08-15T02:09:58.074Z ### Source Files - `src/eval-main.ts` - `src/validate-main.ts` - `src/reaudit-main.ts` - `src/repair-main.ts` - `src/agent-smoke-main.ts` - `package.json` --- title: "Companion commands" description: "Flags, defaults, and outputs for self-bench-eval, self-bench-validate, self-bench-reaudit, self-bench-repair, and self-bench-agent-smoke." --- The `self-bench` npm package installs five extra binaries next to `self-bench`: `self-bench-eval`, `self-bench-validate`, `self-bench-reaudit`, `self-bench-repair`, and `self-bench-agent-smoke`. They are not `self-bench` subcommands, they do not talk to `SELFBENCH_API_URL`, and they operate on already expanded Harbor tasks (or, for the matrix runner, a `.tar.gz` export). Each command parses flags with Node `parseArgs({ strict: true })`, writes progress to stderr, and prints JSON on stdout. | Binary | Job | Default Harbor env | Default concurrency | | --- | --- | --- | --- | | `self-bench-eval` | Codex matrix over an export or task directory | `modal` | `3` | | `self-bench-validate` | Harbor `nop` then `oracle` gates | `modal` | `10` | | `self-bench-reaudit` | Sandboxed Sol coupling review | n/a (`SELFBENCH_EXECUTION_BACKEND`) | `10` | | `self-bench-repair` | Sandboxed Codex repair of coupled tests | n/a (`SELFBENCH_EXECUTION_BACKEND`) | `9` | | `self-bench-agent-smoke` | `--install-only` check of every pinned Harbor adapter | `modal` | `4` | `self-bench-eval`, `self-bench-validate`, and `self-bench-agent-smoke` invoke the Harbor CLI. `self-bench-reaudit` and `self-bench-repair` start sandboxes through `loadConfig().execution` (`SELFBENCH_EXECUTION_BACKEND`, default `docker`) and require the published `dist/sandbox-review.bundle.js` or `dist/sandbox-repair.bundle.js` assets. ```mermaid flowchart LR subgraph Inputs Export["export.tar.gz"] Tasks["expanded task dirs"] Review["coupling report.json"] end subgraph HarborCLI["Harbor CLI"] Eval["self-bench-eval"] Validate["self-bench-validate"] Smoke["self-bench-agent-smoke"] end subgraph Sandbox["SELFBENCH_EXECUTION_BACKEND"] Reaudit["self-bench-reaudit"] Repair["self-bench-repair"] end Export --> Eval Tasks --> Eval Tasks --> Validate Tasks --> Smoke Tasks --> Reaudit Tasks --> Repair Reaudit --> Review Review --> Repair ``` ## Shared conventions - **Install surface.** `package.json` `bin` maps each name to `dist/*-main.js`. Runtime requirement is Node `>=22`. - **Help.** `-h` / `--help` prints usage and exits `0`. - **Unknown flags.** `strict: true` rejects them. - **Integers.** `--concurrency` (and any other numeric flag these commands parse) must be an integer `>= 1`. - **Progress vs result.** Human-readable lines go to stderr. The last stdout payload is JSON. - **Failure style.** Missing required flags and invalid combinations `throw`. Uncaught exceptions exit non-zero. Per-task sandbox or Harbor failures are usually folded into the JSON report instead of aborting the whole batch. - **Task discovery differs.** Eval and validate treat a directory as a task when it contains `task.toml`. Reaudit and repair treat a directory as a task when it contains `instruction.md`. - **Not the Temporal pipeline.** Generation, one validation-repair attempt, and one coupling-repair retry live in the worker. These binaries are offline operator tools for already-written Harbor tasks. ```bash self-bench-eval --help self-bench-validate --help self-bench-reaudit --help self-bench-repair --help self-bench-agent-smoke --help ``` ```bash node dist/eval-main.js --help node dist/validate-main.js --help node dist/reaudit-main.js --help node dist/repair-main.js --help node dist/agent-smoke-main.js --help ``` The checkout must already be built (`bun run build`) so `dist/` and the sandbox bundles exist. ## self-bench-eval Runs every selected Harbor task through Harbor's `codex` adapter for a fixed model matrix: `gpt-5.6-sol`, `gpt-5.6-terra`, and `gpt-5.6-luna`. Each trial pins Codex `0.146.1` and `reasoning_effort=high`. Provide exactly one of `--export` or `--tasks`. `--jobs` is required. ```bash self-bench-eval (--export FILE.tar.gz | --tasks DIRECTORY) --jobs DIRECTORY [options] ``` Self-bench export archive. Extracted once into `/tasks/` unless that directory already contains Harbor tasks. Directory of already expanded Harbor tasks (one subdirectory per task, each with `task.toml`). Harbor jobs directory. Created if missing. Also receives `summary.json` and, for `--export`, the materialized `tasks/` tree. Harbor executable. Default: `harbor` on `PATH`. Passed to Harbor as `--env`. Default: `modal`. Any other value throws `--environment must be docker or modal`. Concurrent trials (task × model pairs). Default: `3`. Restrict the matrix. Repeatable; duplicates are dropped. Invalid names throw `--model must be one of gpt-5.6-sol, gpt-5.6-terra, gpt-5.6-luna`. Codex ChatGPT `auth.json` used only when `OPENAI_API_KEY` is unset. Default fallback: `~/.codex/auth.json`. ### Task input `--export` and `--tasks` are mutually exclusive. Passing both, or neither, throws `provide exactly one of exportPath or tasksPath`. An empty export or directory throws `found no Harbor tasks in …`. `--export` expects the published archive shape: `tasks/.tar.gz`, each inner archive containing a `harbor-task/` directory. The runner copies each `harbor-task/` to `/tasks//`. If `/tasks/` already has one or more `task.toml` directories, extraction is skipped and those tasks are reused. `--tasks` accepts any count of tasks, including fewer than ten. ### Authentication `OPENAI_API_KEY` (trimmed, non-empty) wins. Harbor then runs with the process environment as-is, and `summary.json` records `"auth": "openai-api-key"`. Without that key, the runner reads `--auth` or `~/.codex/auth.json` and requires `auth_mode: "chatgpt"` plus a `tokens` object. It then: - deletes `OPENAI_API_KEY` from the Harbor child environment - sets `CODEX_FORCE_AUTH_JSON=1` - sets `CODEX_AUTH_JSON_PATH` to the resolved auth file `summary.json` records `"auth": "codex-subscription"`. ### Harbor trial Each trial job name is deterministic: ```text ${taskId}-${model} → lowercased, non [a-z0-9_.-] replaced with - ``` Harbor is invoked as: ```bash harbor run \ --path \ --agent codex \ --model \ --ak version=0.146.1 \ --ak reasoning_effort=high \ --env \ --job-name \ --jobs-dir \ --n-concurrent 1 \ --max-retries 1 \ --delete --yes --quiet ``` The child is allowed to fail; timeout is 4 hours. A completed Harbor result is reused and the trial is not rerun. An incomplete job directory is renamed to `.incomplete-` before a fresh run. If Harbor exits without a readable result, the trial is recorded as failed with `exception` starting `Harbor produced no result`. A trial **passes** when it has no exception, at least one numeric verifier reward, and either `rewards.reward >= 1` (other reward fields may be zero diagnostics) or, if `reward` is absent, every numeric reward is `>= 1`. Empty rewards fail. ### Outputs Stderr progress: ```text [3/9] gpt-5.6-sol task-id: passed ``` Status is `passed`, `failed`, or `error` (exception present). Stdout: ```json { "trials": 9, "passed": 6, "failed": 3 } ``` `/summary.json` is the durable record: Always `1`. Always `codex`. `0.146.1`. Which credential path was used. Always `high`. Models that were scheduled. Distinct task directories. Per-trial `taskId`, `model`, `jobName`, `passed`, `rewards`, optional `exception`. ```bash title="Export matrix" export OPENAI_API_KEY=... self-bench-eval \ --export ./self-bench-evals.tar.gz \ --jobs ./matrix-jobs \ --environment modal \ --concurrency 20 ``` ```bash title="Single model, expanded tasks, subscription auth" self-bench-eval \ --tasks ./self-bench-tasks \ --jobs ./matrix-jobs \ --model gpt-5.6-sol \ --auth ~/.codex/auth.json ``` `self-bench-eval` does not classify Harbor infrastructure exceptions separately. A Docker/Modal/auth failure becomes a failed trial with `exception` set, unlike `self-bench-validate`, which treats those as task-level errors. ## self-bench-validate Runs Harbor `nop` then `oracle` against every `task.toml` subdirectory. There is no `--harbor` flag: the executable is always `harbor`. There is no resume key — each gate uses a fresh job name. ```bash self-bench-validate --tasks DIRECTORY --jobs DIRECTORY [options] ``` Directory of expanded Harbor tasks. Subdirectories without `task.toml` are ignored. Harbor jobs directory (created recursively) and location of `validation-summary.json`. Passed as Harbor `--env`. Default: `modal`. Other values throw `--environment must be modal or docker`. Concurrent tasks (each task still runs `nop` then `oracle` sequentially). Default: `10`. Each gate: ```bash harbor run \ --path \ --agent nop|oracle \ --env \ --job-name --<8 hex chars> \ --jobs-dir \ --n-concurrent 1 \ --max-retries 1 \ --delete --yes --quiet ``` Timeout is 4 hours. A non-zero Harbor exit becomes a task `error` (`Harbor exited : …`, last 1000 characters of stderr). After a zero exit, the command reads the job result. AuthError or matching infrastructure text (`unknown flag: --project-name`, Docker daemon/connect failures, Modal timeout, image-build failure, Docker address-pool exhaustion) throws `Harbor infrastructure failure: …` and is recorded as `status: "error"`, not a clean reject. ### Gate thresholds | Agent | Pass when | | --- | --- | | `nop` | `fail_to_pass === 0` and `pass_to_pass >= 1` and `setup_completed >= 1` | | `oracle` | `patch_applied`, `fail_to_pass`, `pass_to_pass`, `deterministic`, and `setup_completed` are all `>= 1` | Missing numeric rewards are treated as `0` via `?? 0`, except nop `fail_to_pass`, which must be present and exactly `0`. A task is `accepted` only when both gates pass. ### Outputs Stderr: `validating ` then `: accepted|rejected` or `: error: …`. Stdout is the summary only: ```json { "total": 4, "accepted": 3, "rejected": 1, "errors": 0 } ``` `/validation-summary.json` adds `schemaVersion: 1`, `generatedAt`, `sourceTasks`, `environment`, and a `tasks` array. Each entry is either `{ taskId, status: "validated", accepted, nop, oracle }` with `passed`/`rewards` per gate, or `{ taskId, status: "error", accepted: false, error }`. ```bash title="Validate expanded tasks" self-bench-validate \ --tasks ./self-bench-tasks \ --jobs ./validation-jobs \ --environment modal \ --concurrency 10 ``` ## self-bench-reaudit Re-runs independent test-to-gold coupling review on expanded tasks. It is not a Harbor command. Each task is extracted from `environment/repo.tar.gz` and reviewed in a sandbox with `dist/sandbox-review.bundle.js` as `gpt-5.6-sol` (`COUPLING_REVIEW_MODEL`). ```bash self-bench-reaudit --tasks DIRECTORY --output REPORT.json [options] ``` Directory of expanded tasks. A child is included only if it has `instruction.md`. JSON report path. Parent directories are created. Concurrent sandbox reviews. Default: `10`. Limit the run to one task id (directory basename). Missing id throws `task not found: `. An empty selection throws `no tasks found`. Each review sandbox: - `runId`: `selfbench-reaudit-v2` - timeout: 15 minutes - inputs: `/work/sandbox-review.js`, `/work/review-input.md` (prompt, `tests/test.patch`, `solution/gold.patch`, plus deterministic coupling evidence from the base snapshot) - output: `/work/review.json` - command: `node /work/sandbox-review.js` Secrets: `OPENAI_API_KEY` when set; otherwise `SELFBENCH_PI_AUTH_JSON` from `SELFBENCH_PI_AUTH_JSON` or `~/.pi/agent/auth.json` (`openai-codex` oauth `access` + `refresh`). The inner reviewer also requires `SELFBENCH_REVIEW_OUTPUT=/work/review.json`. The written verdict is not the raw model JSON alone. `resolveCouplingReview` forces `coupled` when the model says `coupled`, when any gold-only finding has disposition `gold_only`, or when deterministic gold-only artifacts are missing from the review findings. The report is marked `provisional: true`. ### Outputs Stderr: `reviewing ` then `: clean|coupled` or `: error: …`. Stdout: ```json { "output": "/abs/path/REPORT.json", "total": 4, "clean": 2, "coupled": 1, "errors": 1 } ``` The report file: `1`. `gpt-5.6-sol`. ISO timestamp. Resolved `--tasks` path. Always `true` on this CLI. `reviewed` entries include `sandboxId`, `couplingEvidence`, `review`, and `resolution`; `error` entries include `error`. `total`, `clean`, `coupled`, `errors`. ## self-bench-repair Applies one Codex repair sandbox to every `coupled` task in a reaudit report and writes a new task tree. Repair may change only paths already present in the held-out `tests/test.patch`. It does not edit the instruction, base snapshot, or gold patch. ```bash self-bench-repair --tasks DIRECTORY --review REPORT.json --output DIRECTORY [options] ``` Source expanded tasks (`instruction.md` discovery). Coupling report from `self-bench-reaudit`. Must contain a `tasks` array where every entry has `taskId` and `resolution.verdict` of `clean` or `coupled`. Destination directory. Must not equal `--tasks` and must not be inside it (`--output must be outside --tasks`). **Deleted recursively** before the run. Concurrent repair sandboxes. Default: `9`. Codex model passed as `SELFBENCH_REPAIR_MODEL`. Default: `gpt-5.6-sol`. Repair only that coupled task. Throws `coupled task not found in review: ` if it is missing or not `coupled`. When set, clean tasks are **not** copied (`copiedClean` is `0`). Without `--task`, every non-coupled source task is copied into `--output` unchanged. Coupled tasks are packed as `task.tar.gz` and sent to a sandbox: - `runId`: `selfbench-repair-v1` - timeout: 2 hours - resources: 4 CPU, 8192 MiB - files: `/work/task.tar.gz`, `/work/review.json`, `/work/sandbox-repair.js` - outputs: `/work/repaired-task.tar.gz`, `/work/repair-report.json` - secrets: `OPENAI_API_KEY` or `SELFBENCH_CODEX_AUTH_JSON` (from `SELFBENCH_CODEX_AUTH_JSON`, `CODEX_AUTH_JSON_PATH`, or `~/.codex/auth.json`) Inside the sandbox, Codex runs `codex exec --model --dangerously-bypass-approvals-and-sandbox --ephemeral --ignore-user-config --json` with a 90-minute timeout. A no-op patch, a missing `diff --git` patch, or edits outside the original test paths fail that task. Successful repair overwrites `tests/test.patch`, records `testPatchSha256` and a `repair` block on `.selfbench-manifest.json`, and unpacks the new tree under `--output//`. ### Outputs The command prints the full report on stdout and writes the same object to `/repair-report.json`. `1`. ISO timestamp. Resolved `--tasks`. Resolved `--review`. Resolved `--output`. `--model` value. Unchanged tasks copied when `--task` is omitted. Sandboxes that returned a tree and report. Sandbox or extraction failures. Per coupled task: `status: "repaired"` plus the inner report (`taskId`, `model`, `changedPaths`, `originalTestPatchSha256`, `repairedTestPatchSha256`, `codexOutputTail`, `sandboxId`) or `status: "error"`. ```bash title="Reaudit then repair" self-bench-reaudit --tasks ./self-bench-tasks --output ./coupling-report.json self-bench-repair \ --tasks ./self-bench-tasks \ --review ./coupling-report.json \ --output ./repaired-tasks ``` `--output` is wiped at the start of every run. Point it at a dedicated directory. ## self-bench-agent-smoke Attempts Harbor `--install-only` for every adapter in `HARBOR_AGENT_ADAPTERS` (38 unique names, including `oracle`, `nop`, `codex`, and `pi`). It does not run a full agent trial. ```bash self-bench-agent-smoke --task DIRECTORY --jobs DIRECTORY [options] ``` One Harbor task directory (not a parent of many tasks). Harbor jobs directory. Per-adapter JSON is stored under `/adapter-smoke/`. Harbor executable. Default: `harbor`. Harbor `--env`. Default: `modal`. Concurrent adapter checks. Default: `4`. Each check: ```bash harbor run \ --path \ --agent \ --env \ --job-name install--<8 hex chars> \ --jobs-dir \ --install-only \ --n-concurrent 1 \ --max-retries 0 \ --delete --yes --quiet ``` Timeout is 1 hour. `installed` is `exitCode === 0`. Resume: if `/adapter-smoke/.json` already exists, that file is reused and Harbor is not invoked again. New reports are written with exclusive create (`wx`). After the batch, `/adapter-smoke/summary.json` is rewritten with `schemaVersion: 1`, `mode: "install-only"`, `environment`, counts, and the full `results` array (`agent`, `installed`, `exitCode`, `jobName`, `outputTail` last 2000 characters of combined stdout/stderr). Pinned adapters: `oracle`, `nop`, `acp`, `terminus-2`, `claude-code`, `copilot-cli`, `aider`, `cline-cli`, `codex`, `cortex-code`, `cursor-cli`, `gemini-cli`, `antigravity-cli`, `antigravity-sdk`, `rovodev-cli`, `goose`, `grok-build`, `hermes`, `kimi-code`, `kimi-cli`, `langgraph`, `deerflow`, `mini-swe-agent`, `nemo-agent`, `swe-agent`, `opencode`, `mimo`, `openclaw`, `openhands`, `openhands-sdk`, `pi`, `qwen-coder`, `devin`, `trae-agent`, `computer-1`, `eve`, `dspy-rlm`, `vibe`. Stdout: ```json { "adapters": 38, "installed": 36, "failed": 2 } ``` ```bash title="Smoke adapters on one extracted task" self-bench-agent-smoke \ --task ./selected-task/harbor-task \ --jobs ./adapter-jobs \ --environment modal \ --concurrency 4 ``` ## Errors | Condition | Command | Message / behavior | | --- | --- | --- | | Missing `--jobs` | eval, validate, smoke | `--jobs is required` | | Missing `--tasks` | validate, reaudit, repair | `--tasks is required` | | Missing `--output` | reaudit, repair | `--output is required` | | Missing `--review` | repair | `--review is required` | | Missing `--task` | smoke | `--task is required` | | Both or neither eval inputs | eval | `provide exactly one of exportPath or tasksPath` | | Empty task set | eval | `found no Harbor tasks in …` | | Empty task set | reaudit | `no tasks found` or `task not found: …` | | `--environment` not `docker`/`modal` | eval, validate, smoke | `--environment must be docker or modal` (validate wording: `modal or docker`) | | `--concurrency` `< 1` or non-integer | all five | ` must be a positive integer` | | Unknown `--model` | eval | `--model must be one of gpt-5.6-sol, gpt-5.6-terra, gpt-5.6-luna` | | Repair output inside source | repair | `--output must be outside --tasks` | | Invalid review JSON | repair | `invalid coupling review report` / `contains invalid task entries` | | `--task` not coupled | repair | `coupled task not found in review: …` | | Missing ChatGPT subscription file | eval (no API key) | `… does not contain a ChatGPT subscription token set` | | Missing Pi/Codex subscription | reaudit / repair (no API key) | Pi `openai-codex` credential required, or Codex ChatGPT token set required | | Harbor non-zero / infrastructure | validate | task `status: "error"` | | Harbor no result | eval | trial `exception`, counts as failed | | Missing sandbox bundle | reaudit / repair | `readFile` of `dist/sandbox-*.bundle.js` fails | :::files matrix-jobs/ summary.json tasks//task.toml -gpt-5.6-sol/ -gpt-5.6-sol.incomplete-/ # only after a retry validation-jobs/ validation-summary.json -nop-/ -oracle-/ adapter-jobs/ adapter-smoke/summary.json adapter-smoke/codex.json repaired-tasks/ repair-report.json /instruction.md ::: ## Environment that these commands honor | Variable | Used by | Role | | --- | --- | --- | | `OPENAI_API_KEY` | eval, reaudit, repair | Preferred model credential | | `CODEX_AUTH_JSON_PATH` | eval (via `--auth` default), repair | Codex ChatGPT `auth.json` path; default `~/.codex/auth.json` | | `SELFBENCH_CODEX_AUTH_JSON` | repair | Inline Codex subscription JSON | | `SELFBENCH_PI_AUTH_JSON` | reaudit | Inline Pi `openai-codex` credential | | `SELFBENCH_EXECUTION_BACKEND` | reaudit, repair | `docker` (default) or `modal` sandbox backend | | `SELFBENCH_DOCKER_IMAGE` | reaudit, repair | Default `selfbench-sandbox:local` | | `SELFBENCH_MODAL_APP` / `SELFBENCH_MODAL_IMAGE` / `SELFBENCH_MODAL_ENVIRONMENT` | reaudit, repair | Modal sandbox when execution backend is `modal` | Harbor `docker` vs `modal` for eval, validate, and smoke is the `--environment` flag, not `SELFBENCH_HARBOR_ENVIRONMENT`. Documented Harbor install for evaluations is `harbor[modal]==0.20.1.dev202608040148`. ## Next Extract an export, run one Codex trial or the matrix, resume jobs, and smoke adapters. `self-bench` itself: up, run, list, status, cancel, download. How the worker applies nop, oracle, determinism, and one constrained repair. API key precedence and Pi / Codex subscription JSON fallbacks. --- ## 17. HTTP API > Routes, Bearer auth, status codes, run submission schema, provenance upload limits, export headers, and review-asset serving. - Page Markdown: https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc/pages/17-http-api.md - Generated: 2026-08-15T02:12:55.120Z ### Source Files - `src/api.ts` - `src/api-main.ts` - `src/contracts.ts` - `docs/operations.md` - `review/src/App.tsx` --- title: "HTTP API" description: "Routes, Bearer auth, status codes, run submission schema, provenance upload limits, export headers, and review-asset serving." --- `startApi()` is the Node `http.createServer` process started by `src/api-main.ts` (`node dist/api-main.js` / `bun run start:api`). It binds `SELFBENCH_API_HOST`:`SELFBENCH_API_PORT`, opens a Temporal client, and creates the configured artifact store. It does not run Harbor sandboxes or hold `OPENAI_API_KEY` / `GH_TOKEN`. JSON responses are pretty-printed (`JSON.stringify(..., null, 2)` plus a trailing newline) with `content-type: application/json; charset=utf-8`. ## Process and bind | Item | Value | | --- | --- | | Entry | `node dist/api-main.js` (image `CMD`); dev `bun run dev:api` → `tsx src/api-main.ts` | | Default bind | `127.0.0.1:8080` | | Compose | Container `SELFBENCH_API_HOST=0.0.0.0`, published `127.0.0.1:8080:8080` | | Ready log | `SelfBench API listening on http://:` | | Shutdown | `SIGINT` / `SIGTERM` close the HTTP server and Temporal connection, then exit `0` | | Workflow start | `selfBenchRunWorkflow`, `workflowId` = `runId`, `taskQueue` = `SELFBENCH_TASK_QUEUE`, `workflowExecutionTimeout` = `14 days` | `loadConfig()` refuses to start when the bind host is not loopback and `SELFBENCH_API_TOKEN` is unset. Loopback hosts are `127.0.0.1`, `::1`, and `localhost`. Local Compose sets `SELFBENCH_API_TOKEN` to `${SELFBENCH_API_TOKEN:-selfbench-local}` because the API process listens on `0.0.0.0` inside the container. ```mermaid flowchart LR subgraph Clients CLI["self-bench CLI
SELFBENCH_API_URL"] Review["Review UI GET /"] end subgraph APIproc["API process"] HTTP["http.createServer"] end subgraph TemporalNS["Temporal"] WF["selfBenchRunWorkflow"] end subgraph Artifacts["Artifact store"] Local["local directory"] GCS["GCS bucket/prefix"] end Worker["worker process"] CLI -->|"Bearer + JSON/JSONL"| HTTP Review -->|"optional Bearer on /v1"| HTTP HTTP -->|"start / query / cancel / list"| WF HTTP -->|"put provenance, openRead export"| Artifacts Worker --> WF Worker --> Artifacts ``` ## Authentication When `SELFBENCH_API_TOKEN` is unset, `/v1` routes accept requests with no `Authorization` header. When the token is set, `/v1` routes require `Authorization: Bearer `. The server strips a case-insensitive `Bearer ` prefix and compares the remainder with `crypto.timingSafeEqual` after a length check. Missing, wrong-length, or mismatched tokens return `401` `{ "error": "unauthorized" }`. There is no `WWW-Authenticate` header. Unauthenticated even when a token is configured: | Method | Path | Purpose | | --- | --- | --- | | `GET` | `/healthz` | Liveness `{ "ok": true }` | | `GET` | `/` | Review SPA `index.html` | | `GET` | `/assets/*` | Hashed Vite review assets | `/v1` data routes stay behind the token. The review HTML can load without credentials; listing runs and downloading exports cannot. The CLI sends `Authorization` only when `SELFBENCH_API_TOKEN` is set in the client environment. Pointing `SELFBENCH_API_URL` at a token-protected API without exporting the same token yields `401`. ## Route inventory | Method | Path | Auth when token set | Success | Body limit | | --- | --- | --- | --- | --- | | `GET` | `/healthz` | No | `200` | — | | `GET` | `/` | No | `200` | — | | `GET` | `/assets/*` | No | `200` | — | | `POST` | `/v1/provenance?runId=` | Yes | `201` | 100 MiB | | `POST` | `/v1/runs` | Yes | `202` | 10 MiB | | `GET` | `/v1/runs` | Yes | `200` | — | | `GET` | `/v1/runs/:runId` | Yes | `200` | — | | `POST` | `/v1/runs/:runId/cancel` | Yes | `202` | — | | `GET` | `/v1/runs/:runId/export` | Yes | `200` | — | `:runId` must match `^[a-z0-9][a-z0-9-]{2,62}$` (3–63 characters, starts with a letter or digit, lowercase, digits, hyphens). Unknown paths return `404` `{ "error": "not found" }`. There is no `DELETE`, `PUT`, `PATCH`, pagination, or CORS handling. ## Run ID Same regex as the path segment and as `submission.runId`. Used as the Temporal `workflowId`. The CLI default is `sb-<14 datetime digits>-<8 uuid chars>`. Reuse of an existing `runId` on `POST /v1/runs` fails at Temporal start and surfaces as `500` with the client error message. Failed generation needs a new ID. ## POST /v1/provenance Stores the raw request body at artifact key `runs//input/provenance.jsonl` with content type `application/x-ndjson`. The API does not parse or validate JSONL. :::endpoint POST /v1/provenance Store sanitized provenance JSONL Target run ID. Invalid or missing values are Zod errors → `400`. **Limits** - Maximum body: `100 * 1024 * 1024` bytes (100 MiB). - Oversized body: `500` `{ "error": "request body exceeds 104857600 bytes" }` (not `413`). - Incoming `Content-Type` is ignored; the stored reference always uses `application/x-ndjson`. Identical bytes at the same key are idempotent (same SHA-256 and size). Different bytes at the same key fail the artifact store and return `500`. `file://` for the local store, or `gs:////runs//input/provenance.jsonl` for GCS. 64 lowercase hex digits of the stored bytes. Byte length of the stored body. Always `application/x-ndjson`. ::: ```bash curl -sS -X POST \ -H "Authorization: Bearer $SELFBENCH_API_TOKEN" \ -H "Content-Type: application/x-ndjson" \ --data-binary @provenance.jsonl \ "$SELFBENCH_API_URL/v1/provenance?runId=sb-20260814120000-deadbeef" ``` ```json { "uri": "file:///var/lib/selfbench/artifacts/runs/sb-20260814120000-deadbeef/input/provenance.jsonl", "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "sizeBytes": 2048, "contentType": "application/x-ndjson" } ``` ## POST /v1/runs Parses the JSON body with the HTTP `submissionSchema`, then builds a `runRequestSchema` value and starts Temporal. Clients do not send `authoring` or `version`; the API stamps those fields. :::endpoint POST /v1/runs Start a tiered candidate workflow `^[a-z0-9][a-z0-9-]{2,62}$`. `{ "url": "", "commit": "<40-hex SHA>" }`. `url` is `z.string().url()`. Artifact ref from `POST /v1/provenance`: `uri`, `sha256` (64 hex), `sizeBytes` (≥ 0), `contentType`. `easy`, `medium`, `hard`: integers `0`–`100` each. After stamping, `runRequestSchema` also requires the sum to be `1`–`100`. Non-empty. Default `gpt-5.6-sol`. Full 40-hex SHA. Overridden by `SELFBENCH_BUILD_COMMIT` on the API process when that env is set. **Stamped workflow input (not accepted from the client)** ```json { "authoring": { "provider": "openai-codex", "model": "", "reasoningEffort": "high" }, "version": { "selfbenchCommit": "", "executionBackend": "", "sandboxImage": "", "schema": 1 } } ``` Default JSON body limit is `10 * 1024 * 1024` bytes. Invalid JSON is `400`. Zod failures (including a zero or >100 candidate sum) are `400`. Success is `202` `{ "runId": "" }`. ::: ```json { "runId": "sb-20260814120000-deadbeef", "repository": { "url": "https://github.com/example/repo.git", "commit": "0123456789abcdef0123456789abcdef01234567" }, "provenance": { "uri": "file:///var/lib/selfbench/artifacts/runs/sb-20260814120000-deadbeef/input/provenance.jsonl", "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "sizeBytes": 2048, "contentType": "application/x-ndjson" }, "candidateCounts": { "easy": 5, "medium": 10, "hard": 5 }, "authoringModel": "gpt-5.6-sol", "selfbenchCommit": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" } ``` ```json { "runId": "sb-20260814120000-deadbeef" } ``` ```mermaid sequenceDiagram participant CLI as self-bench CLI participant API as API process participant Store as Artifact store participant T as Temporal CLI->>API: POST /v1/provenance?runId= API->>Store: put runs/{id}/input/provenance.jsonl Store-->>API: ArtifactRef API-->>CLI: 201 ArtifactRef CLI->>API: POST /v1/runs API->>T: start selfBenchRunWorkflow T-->>API: started API-->>CLI: 202 runId CLI->>API: GET /v1/runs/{id} API->>T: query status + describe T-->>API: RunStatus API-->>CLI: 200 RunStatus CLI->>API: GET /v1/runs/{id}/export API->>Store: openRead(export) API-->>CLI: 200 gzip + x-content-sha256 ``` ## GET /v1/runs Lists Temporal executions with visibility query `WorkflowType = 'selfBenchRunWorkflow'`. No pagination or sort is applied. Temporal `workflowId`. Temporal execution status name (`RUNNING`, `COMPLETED`, `CANCELED`, `FAILED`, …), not a `RunPhase`. `startTime` as ISO-8601. `closeTime` as ISO-8601 when the execution is closed. ## GET /v1/runs/:runId Queries Temporal `status` and `describe()`. While the execution is `RUNNING`, or when the query already reports a terminal phase (`complete`, `blocked`, `failed`, `cancelled`), the query payload is returned as-is. Otherwise the API overlays a phase derived from Temporal: | Temporal `status.name` | Overlaid `phase` | | --- | --- | | `COMPLETED` | `complete` | | `CANCELED` | `cancelled` | | `RUNNING` | `queued` | | anything else | `failed` | If the query fails but `describe()` succeeds, the body is only `{ "runId", "phase" }`. If `describe()` also fails (unknown ID), the outer handler returns `500`, not `404`. ### Status body Workflow / run ID. `queued` · `discovering` · `authoring` · `exporting` · `complete` · `blocked` · `failed` · `cancelled`. Sum of the three tier budgets. `{ easy, medium, hard }`. Unique candidates collected so far. Tasks with status `accepted`. Tasks with status `rejected`. Per-candidate progress. Each item has `taskId`, `candidateId`, `difficulty`, `status`, optional `reason`. Optional: `wave`, `totalShards`, `completedShards`, `failedShards`, `candidates`. Optional artifact ref once `buildExport` finishes. Present on a successful `complete` run. Optional failure detail. Overlay may set `error` to `Temporal workflow ` when Temporal is failed-like and the query has no `error`. Task `status` values: `authoring`, `auditing`, `validating`, `reviewing`, `repairing`, `infrastructure_failed`, `rejected`, `accepted`. `GET /v1/runs` `status` is Temporal’s name (`CANCELED`). `GET /v1/runs/:runId` `phase` uses SelfBench spelling (`cancelled`). ## POST /v1/runs/:runId/cancel Calls Temporal `cancel()` and returns immediately. ```json { "runId": "sb-20260814120000-deadbeef", "cancellationRequested": true } ``` `202` means the cancel was requested, not that `phase` is already `cancelled`. Poll `GET /v1/runs/:runId` for the terminal phase. ## GET /v1/runs/:runId/export Streams the artifact referenced by `status.export`. Missing or empty `export` returns `409`: ```json { "error": "run export is not ready" } ``` Success headers: | Header | Value | | --- | --- | | `content-type` | `status.export.contentType` (exports are stored as `application/gzip`) | | `content-length` | `status.export.sizeBytes` | | `content-disposition` | `attachment; filename="selfbench-.tar.gz"` | | `x-content-sha256` | `status.export.sha256` (64 hex) | `openRead` checks size before streaming and hashes the bytes. A mid-stream integrity failure destroys the socket after headers are sent. The CLI requires `x-content-sha256`, writes with create-only `wx`, and deletes the file if the digest mismatches. ## Review-asset serving Vite builds the review app to `dist/review`. The API maps: | Request | File | `cache-control` | | --- | --- | --- | | `GET /` | `dist/review/index.html` | `no-cache` | | `GET /assets/...` | `dist/review/assets/...` | `public, max-age=31536000, immutable` | Resolved paths must stay under the `review` root; escape attempts return `400` `{ "error": "invalid asset path" }`. Missing files return `404` `{ "error": "asset not found" }`. Content types: `.html` → `text/html; charset=utf-8`, `.css` → `text/css; charset=utf-8`, `.js` → `text/javascript; charset=utf-8`, otherwise `application/octet-stream`. Only `/` and `/assets/*` are served this way. `GET /index.html` is not a public alias. The SPA lists `GET /v1/runs`, then loads `GET /v1/runs/:runId` and `GET /v1/runs/:runId/export` in parallel. An optional password field supplies `Authorization: Bearer …` for this tab only. `bun run dev:review` proxies `/v1` to `http://127.0.0.1:8080`. ## Status codes | Code | When | | --- | --- | | `200` | Health, status, list, review asset, export stream | | `201` | Provenance stored | | `202` | Workflow started or cancel requested | | `400` | Zod validation, JSON `SyntaxError`, or invalid review asset path | | `401` | Token configured and Bearer missing/wrong | | `404` | Unknown route, or review asset missing | | `409` | Export requested before `status.export` exists | | `500` | Everything else, including oversized bodies, artifact conflicts, Temporal client errors, and unknown run IDs | Error bodies are `{ "error": "" }`. Zod and `SyntaxError` use `error.message`. After headers are sent (export stream), the handler destroys the response instead of writing JSON. ## Clients | Client | Base URL | Token | | --- | --- | --- | | `self-bench` | `SELFBENCH_API_URL` or `http://127.0.0.1:8080` | `SELFBENCH_API_TOKEN` if set | | Review UI | Same origin as the API (or Vite `/v1` proxy) | Session-only input | CLI mapping: `run` → provenance + create; `list` → `GET /v1/runs`; `status` → `GET /v1/runs/:id`; `cancel` → `POST .../cancel`; `download` → `GET .../export`. `--output` waits until `phase === "complete"` (poll every 2s) and then downloads. `blocked`, `failed`, and `cancelled` abort the wait. ## What this API does not do - Collect provenance itself (the client uploads JSONL). - Replace rejected candidates or accept a new `runId` on a failed run. - Delete artifacts or workflows. Remove local volume data or GCS prefixes with operator tooling. - Expose model, GitHub, or Modal credentials. Those belong on the worker. ## Failure modes | Symptom | Likely cause | | --- | --- | | Process exits at startup: `SELFBENCH_API_TOKEN is required when the API listens beyond loopback` | Bind host is not `127.0.0.1` / `::1` / `localhost` and no token is set | | `401 unauthorized` | Token configured on the server, missing or different on the client | | `400` on create | Bad `runId`, non-URL `repository.url`, short commit SHA, candidate sum `0` or `>100`, invalid provenance ref | | `409 run export is not ready` | Phase is not a completed export (`blocked` / `failed` / in progress) | | `500 request body exceeds … bytes` | Provenance `>100` MiB or JSON body `>10` MiB | | `500` on `GET /v1/runs/:id` | Temporal cannot describe that workflow ID | | SHA-256 mismatch on download | Stream truncated or corrupted; CLI deletes the partial file | ## Related pages CLI list, status, cancel, wait, and create-only download against this API. API Bearer token, CLI `SELFBENCH_API_TOKEN`, and worker credential split. `SELFBENCH_API_*`, loopback token rule, artifact backends, Temporal fields. Hosted review UI and local archive inspection. `tar.gz` layout and SHA-256 fields the download headers describe. `409` export-not-ready, SHA-256 mismatch, and related operator failures. --- ## 18. Configuration reference > SELFBENCH_* environment keys, defaults, validation, loopback token rule, artifact backends, and Temporal connection fields. - Page Markdown: https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc/pages/18-configuration-reference.md - Generated: 2026-08-15T02:12:39.049Z ### Source Files - `src/config.ts` - `compose.yaml` - `docs/operations.md` - `src/temporal/worker-main.ts` - `src/modal-auth.ts` --- title: "Configuration reference" description: "SELFBENCH_* environment keys, defaults, validation, loopback token rule, artifact backends, and Temporal connection fields." --- `loadConfig()` parses `process.env` once at process start for `node dist/api-main.js`, `node dist/temporal/worker-main.js`, `self-bench-repair`, and `self-bench-reaudit`. It validates `SELFBENCH_*` keys with Zod, applies defaults, enforces the loopback token rule, and returns a typed `SelfBenchConfig` with `apiHost` / `apiPort` / `apiToken`, `artifact` (`local` or `gcs`), `execution` (`docker` or `modal`), `harborEnvironment`, `activityConcurrency`, optional `buildCommit`, and `temporal`. Extra process-environment keys are ignored. The `self-bench` CLI does **not** call `loadConfig()`; it is an HTTP client that reads `SELFBENCH_API_URL` and `SELFBENCH_API_TOKEN`. ```mermaid flowchart TB subgraph cli ["self-bench CLI"] APIURL["SELFBENCH_API_URL"] CLITOKEN["SELFBENCH_API_TOKEN as Bearer"] UP["up writes backend + modal mount"] end subgraph consumers ["loadConfig() consumers"] API["api-main"] WRK["worker-main"] COMP["repair / reaudit"] end subgraph cfg ["SelfBenchConfig"] LISTEN["apiHost / apiPort / apiToken"] ART["artifact: local | gcs"] EX["execution: docker | modal"] HB["harborEnvironment"] TMP["temporal address / namespace / tls / apiKey / taskQueue"] end subgraph secrets ["Worker secrets outside the schema"] OA["OPENAI_API_KEY"] GH["GH_TOKEN"] MD["MODAL_TOKEN_ID / MODAL_TOKEN_SECRET"] SUB["SELFBENCH_PI_AUTH_JSON / SELFBENCH_CODEX_AUTH_JSON"] end CLI -->|HTTP| API API --> cfg WRK --> cfg COMP --> cfg WRK --> secrets UP --> WRK ``` ## Who reads which keys | Process | Entry | Parser | Role | | --- | --- | --- | --- | | API | `node dist/api-main.js` (image default `CMD`) | `loadConfig()` | Bind address, Bearer token, Temporal **client**, artifact store | | Worker | `node dist/temporal/worker-main.js` | `loadConfig()` after empty Modal-token cleanup | Temporal **worker**, sandboxes, Harbor `--env`, artifact store | | `self-bench-repair` / `self-bench-reaudit` | companion CLIs | `loadConfig()` | Same execution backend as the worker; no Temporal listen | | `self-bench` | `dist/cli.js` | raw `process.env` | `SELFBENCH_API_URL`, client `SELFBENCH_API_TOKEN`, `up` overlay | | `self-bench-eval` / `self-bench-validate` / `self-bench-agent-smoke` | companion CLIs | flags + auth env | Do **not** call `loadConfig()` | `bun run dev:api`, `bun run start:api`, `bun run dev:worker`, and `bun run start:worker` use the same parsers as the packaged Node entrypoints. ## `SELFBENCH_*` keys parsed by `loadConfig()` Unknown keys on `process.env` are stripped. Empty string is **not** treated as unset except for `SELFBENCH_BUILD_COMMIT` (preprocessed to missing) and the worker's Modal token cleanup (not part of this schema). ### API listen and token Bind address passed to `server.listen`. Compose sets `0.0.0.0` inside the container. TCP port. Coerced integer in `1..65535`. Compose publishes `127.0.0.1:8080:8080`. Non-empty Bearer secret. Optional only on loopback. Omitted from `SelfBenchConfig` when unset. Compose default is `selfbench-local` via `${SELFBENCH_API_TOKEN:-selfbench-local}`. Full 40-hex commit SHA. Empty string is treated as unset. Stored lowercased as `buildCommit`. When set, the API prefers it over the run body's `selfbenchCommit`. ### Artifact backend Selects `LocalArtifactStore` or `GcsArtifactStore`. Compose forces `local`. Root directory for `local`. Resolved to an absolute path. Compose uses `/var/lib/selfbench/artifacts` on the named `artifacts` volume. Required when the backend is `gcs`. Startup throws `SELFBENCH_GCS_BUCKET is required for GCS`. Object-name prefix. Leading and trailing slashes are stripped. Reads reject URIs outside `gs:////…`. ### Execution and Harbor Sandbox executor. `self-bench up --backend` writes this for Compose. Harbor `--env` for nop/oracle gates. Defaults to `SELFBENCH_EXECUTION_BACKEND`. `self-bench up` always sets both to the same value. Image for Docker sandboxes. `self-bench up --backend docker` builds `Dockerfile.sandbox` to this tag. Modal app name (`apps.fromName`, create-if-missing). Modal environment name for the app and ephemeral secrets. Registry image Modal extends with `gh`, Pi, and Codex. `maxConcurrentActivityTaskExecutions`, integer `1..100`. When unset: `1` for Docker, `20` for Modal. ### Temporal connection gRPC target. Compose sets `temporal:7233`. Published on the host as `127.0.0.1:7233`. Namespace for the Temporal client, worker, and workflow start. Shared task queue. API `workflow.start` and the worker poller must use the same value. String enum only (`true` / `false`). Transformed to a boolean. Missing env becomes `false`. Case-sensitive: `TRUE` fails parse. Passed to Temporal `Connection.connect` / `NativeConnection.connect` only when non-empty. ## Keys outside `loadConfig()` These names appear in Compose, the CLI, or workers but are not fields on `SelfBenchConfig`. | Key | Default | Used by | | --- | --- | --- | | `SELFBENCH_API_URL` | `http://127.0.0.1:8080` | CLI HTTP client (`list`, `status`, `run`, `cancel`, `download`) | | `SELFBENCH_MODAL_CONFIG_PATH` | `/dev/null` in Compose; `~/.modal.toml` from `self-bench up --backend modal` | Host bind-mount onto `/root/.modal.toml` (read-only) | | `DOCKER_GID` | `0` | Compose `group_add` so the worker can use the Docker socket | | `OPENAI_API_KEY` | unset | Worker sandboxes; preferred over subscription JSON | | `SELFBENCH_PI_AUTH_JSON` | else `~/.pi/agent/auth.json` | Pi / review / discovery / authoring fallback | | `SELFBENCH_CODEX_AUTH_JSON` | else `CODEX_AUTH_JSON_PATH` or `~/.codex/auth.json` | Codex / coupling-repair fallback | | `CODEX_AUTH_JSON_PATH` | `~/.codex/auth.json` | File fallback for Codex subscription auth | | `GH_TOKEN` | else `gh auth token` | Worker GitHub reads; Compose passes it only to the worker | | `MODAL_TOKEN_ID` / `MODAL_TOKEN_SECRET` | unset | Modal SDK; empty values are deleted at worker start | `self-bench-eval`, `self-bench-validate`, and `self-bench-agent-smoke` take `--environment docker|modal` on the command line (eval/validate default `modal`). They do not read `SELFBENCH_EXECUTION_BACKEND`. ## Loopback token rule After Zod parse, `loadConfig()` requires a token unless the host is one of these **exact** strings: - `127.0.0.1` - `::1` - `localhost` Any other host — including Compose's `0.0.0.0` — throws: ```text SELFBENCH_API_TOKEN is required when the API listens beyond loopback ``` An empty `SELFBENCH_API_TOKEN` fails Zod (`min(1)`) before this check. Compose's `${SELFBENCH_API_TOKEN:-selfbench-local}` substitutes the local default when the variable is unset or empty. When a token is present, every route except `GET /healthz`, `GET /`, and `GET /assets/*` requires `Authorization: Bearer ` compared with `timingSafeEqual`. When the token is omitted on loopback, `authorized()` returns true and no Bearer header is required. The CLI sends `Authorization` only when `SELFBENCH_API_TOKEN` is a non-empty string. The Compose API binds `0.0.0.0` inside the container, so the local stack always needs a token. Export the same value in the CLI shell. The published port is still `127.0.0.1:8080` on the host. ## Validation and derived defaults | Check | Failure | | --- | --- | | Enum / port / concurrency / SHA / TLS parse | Zod error; process exits at `loadConfig()` | | Host not loopback and no token | `SELFBENCH_API_TOKEN is required when the API listens beyond loopback` | | `SELFBENCH_ARTIFACT_BACKEND=gcs` without bucket | `SELFBENCH_GCS_BUCKET is required for GCS` | | `SELFBENCH_BUILD_COMMIT` set but not 40 hex | Zod regex failure (runtime) or `SELFBENCH_BUILD_COMMIT must be a full commit SHA` (image build script) | | CLI has no baked SHA and no git | `SelfBench was built without commit metadata; set SELFBENCH_BUILD_COMMIT to a full commit SHA` | | `self-bench up --backend` not `docker` or `modal` | `--backend must be "docker" or "modal"` | | `--modal-config` without Modal backend | `--modal-config requires --backend modal` | Derived after a successful parse: | Field | Rule | | --- | --- | | `activityConcurrency` | Env override, else `20` when execution is `modal`, else `1` | | `harborEnvironment` | Env override, else the execution backend | | `buildCommit` | Lowercased SHA, omitted when unset | | `execution` | `{ kind: "docker", image }` or `{ kind: "modal", app, image, environment? }` | | `artifact` | `{ kind: "local", directory }` or `{ kind: "gcs", bucket, prefix }` | | `temporal.apiKey` / `apiToken` | Omitted when the env value is empty | The worker then: 1. Calls `removeEmptyModalCredentialOverrides()` so blank `MODAL_TOKEN_ID` / `MODAL_TOKEN_SECRET` cannot hide a mounted `~/.modal.toml`. 2. If `execution.kind === "docker"` **or** `harborEnvironment === "docker"`, runs `docker info` and `docker compose version` (30s each) before connecting to Temporal. 3. Creates the worker with `maxConcurrentActivityTaskExecutions: config.activityConcurrency`. 4. Logs `SelfBench worker polling / with activity concurrency `. The API logs `SelfBench API listening on http://:` after `listen` succeeds. ## Artifact backends `createArtifactStore(config.artifact)` is shared by the API (provenance upload, export download) and the worker (run artifacts). Writes are create-only: local uses `wx` / hard-link, GCS uses `ifGenerationMatch: 0`. An identical SHA-256 + size is treated as idempotent; different contents throw `artifact already exists with different contents`. Keys must be non-empty, must not start with `/`, and must not contain a `..` path segment (`unsafe artifact key`). ### Local - Root is `resolve(SELFBENCH_ARTIFACT_DIR)`. - Object path is `root/key`; reads reject any resolved path outside that root (`artifact path escapes root`). - URI shape: `file://`. - Compose mounts the `artifacts` volume at `/var/lib/selfbench/artifacts` on **both** API and worker. ### GCS - Client is `new Storage()` — the process Application Default Credentials, not a `SELFBENCH_*` key. - Object name is `/` when the prefix is non-empty. - URI shape: `gs:///`. - `get` / `openRead` reject a different bucket or a path outside the configured prefix (`artifact is outside configured bucket`). - Custom metadata stores `sha256`; streamed reads also check size and digest (`artifact integrity check failed`). Exports are unencrypted. Confine IAM to the configured prefix independently of the application prefix check. ## Temporal connection fields `connectTemporalClient` (API) and `connectTemporalWorker` (worker) pass the same object: ```ts { address: config.temporal.address, tls: config.temporal.tls, ...(config.temporal.apiKey ? { apiKey: config.temporal.apiKey } : {}), } ``` There is no extra check that TLS is on when an API key is set. The API `Client` uses `namespace`. `workflow.start` uses `taskQueue` and a 14-day `workflowExecutionTimeout`. The worker uses the same `namespace` and `taskQueue`. API and worker must share image digest, `SELFBENCH_TASK_QUEUE`, Temporal address/namespace/TLS/API key, and artifact backend. A queue mismatch silently parks workflows with no poller. Compose Temporal is `temporalio/auto-setup:1.29.6` on Postgres 17 with development credentials `temporal` / `temporal`. State lives in the `selfbench_temporal-postgres` volume. Self-bench has no application database migrations. ## Compose overlay versus schema defaults `self-bench up` runs `docker compose --file compose.yaml up -d --build` after writing: | Variable | Docker backend | Modal backend | | --- | --- | --- | | `SELFBENCH_EXECUTION_BACKEND` | `docker` | `modal` | | `SELFBENCH_HARBOR_ENVIRONMENT` | `docker` | `modal` | | `SELFBENCH_BUILD_COMMIT` | resolved 40-hex SHA | same | | `SELFBENCH_MODAL_CONFIG_PATH` | unset (Compose falls back to `/dev/null`) | `--modal-config` or `~/.modal.toml` | Compose then overlays schema defaults: ```text SELFBENCH_API_HOST=0.0.0.0 SELFBENCH_API_TOKEN=${SELFBENCH_API_TOKEN:-selfbench-local} SELFBENCH_TEMPORAL_ADDRESS=temporal:7233 SELFBENCH_ARTIFACT_BACKEND=local SELFBENCH_ARTIFACT_DIR=/var/lib/selfbench/artifacts SELFBENCH_DOCKER_IMAGE=selfbench-sandbox:local ``` Worker-only Compose environment: `GH_TOKEN`, `OPENAI_API_KEY`, `MODAL_TOKEN_ID`, `MODAL_TOKEN_SECRET`, `SELFBENCH_PI_AUTH_JSON`, `SELFBENCH_CODEX_AUTH_JSON`. The worker runs as `root`, adds `DOCKER_GID`, and mounts `/var/run/docker.sock`. The API image user is `node` and never receives the Docker socket or model credentials. ## Example environments ```bash title="Schema defaults (no Compose)" # API: http://127.0.0.1:8080 — token optional # artifacts: .selfbench/artifacts # Temporal: 127.0.0.1:7233 / default / selfbench-dev / tls=false # execution: docker / selfbench-sandbox:local / concurrency 1 ``` ```bash title="Local Compose (self-bench up)" export OPENAI_API_KEY=... export GH_TOKEN="$(gh auth token)" export SELFBENCH_API_TOKEN="$(openssl rand -hex 24)" export SELFBENCH_API_URL=http://127.0.0.1:8080 self-bench up --backend modal ``` ```bash title="Split API + worker (Temporal Cloud + GCS + Modal)" # API process SELFBENCH_API_HOST=0.0.0.0 SELFBENCH_API_TOKEN=... SELFBENCH_ARTIFACT_BACKEND=gcs SELFBENCH_GCS_BUCKET=... SELFBENCH_GCS_PREFIX=selfbench SELFBENCH_TEMPORAL_ADDRESS=... SELFBENCH_TEMPORAL_NAMESPACE=... SELFBENCH_TEMPORAL_API_KEY=... SELFBENCH_TEMPORAL_TLS=true SELFBENCH_TASK_QUEUE=selfbench-prod # Worker process (same Temporal + GCS + queue + image digest) SELFBENCH_EXECUTION_BACKEND=modal SELFBENCH_HARBOR_ENVIRONMENT=modal MODAL_TOKEN_ID=... MODAL_TOKEN_SECRET=... GH_TOKEN=... OPENAI_API_KEY=... ``` The worker owns model, GitHub, Modal, and Harbor credentials. Keep API and worker service accounts separate. ## Sandbox-injected `SELFBENCH_*` keys The worker writes these into sandbox processes. They are not operator configuration. | Key | Purpose | | --- | --- | | `SELFBENCH_DISCOVERY_OUTPUT` | `/work/discovery.json` | | `SELFBENCH_TASK_OUTPUT` | `/work/tasks` | | `SELFBENCH_REVIEW_OUTPUT` | `/work/review.json` | | `SELFBENCH_REPAIR_MODEL` | Authoring / repair model name (companion default `gpt-5.6-sol`) | | `SELFBENCH_PI_AUTH_JSON` / `SELFBENCH_CODEX_AUTH_JSON` | Stage-specific subscription material when no `OPENAI_API_KEY` | Sandboxes also receive `OPENAI_API_KEY` and `GH_TOKEN` when those credentials exist on the worker. Bootstrap writes Pi auth to `$HOME/.pi/agent/auth.json` and a GitHub `x-access-token` credential helper, then deletes both on exit. - **API exits at startup on Compose without a token** — host is `0.0.0.0`. Set `SELFBENCH_API_TOKEN` or accept `selfbench-local`. - **CLI `401 unauthorized`** — CLI token missing or different from the API token. - **Workflows stay queued** — API and worker `SELFBENCH_TASK_QUEUE` or Temporal namespace/address differ. - **GCS startup throw** — backend `gcs` without `SELFBENCH_GCS_BUCKET`. - **Modal profile ignored** — empty `MODAL_TOKEN_*` in Compose; the worker now deletes those empties. Non-empty invalid tokens still override the mount. - **Worker refuses to start** — Docker backend or Docker Harbor without a working `docker` / `docker compose` on the worker. - **CLI cannot resolve commit** — published package without git and without `SELFBENCH_BUILD_COMMIT`. ## Next Prerequisites and the first environment exports before `self-bench up`. How `up` / `down` apply the Compose overlay, backends, and volumes. Split API and worker, GCS, TLS, API key, and matching task queue. `OPENAI_API_KEY` precedence, subscription JSON, `GH_TOKEN`, Bearer, Modal profile versus tokens. Docker versus Modal images, concurrency, and Harbor pairing. Which routes honor the Bearer token and how provenance/export use the artifact store. Export 409, SHA-256 mismatch, leftover Docker resources. --- ## 19. Task definition > definition.json fields, {tests} placeholder rules, toolchains, timeouts, resources, submit_task output files, and schemaVersion 1 constraints. - Page Markdown: https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc/pages/19-task-definition.md - Generated: 2026-08-15T02:12:15.390Z ### Source Files - `src/contracts.ts` - `src/extensions/authoring.ts` - `src/audit.ts` - `src/skills/selfbench/SKILL.md` - `src/harbor-task.ts` - `tests/contracts.test.ts` --- title: "Task definition" description: "definition.json fields, {tests} placeholder rules, toolchains, timeouts, resources, submit_task output files, and schemaVersion 1 constraints." --- An authored SelfBench task is one `schemaVersion` `1` `definition.json` plus two Git patches. The authoring tool `submit_task` writes those three files under `$SELFBENCH_TASK_OUTPUT//`. After the call, the sandbox compiles that directory into a Harbor bundle and the worker stores the JSON as `runs//authoring//definition.json`. The compiled `harbor-task/` tree does not include `definition.json`; the compiler copies `prompt` into `instruction.md` and maps timeouts, resources, and toolchains into `task.toml` and the Dockerfiles. ## Authored output `submit_task` requires `SELFBENCH_TASK_OUTPUT`. It creates `$SELFBENCH_TASK_OUTPUT//` (not recursively) and writes three files with exclusive create (`wx`): :::files $SELFBENCH_TASK_OUTPUT// ├── definition.json # pretty-printed JSON, trailing newline ├── test.patch # held-out tests; must start with "diff --git " └── gold.patch # non-test reference implementation; must start with "diff --git " ::: The authoring sandbox sets `SELFBENCH_TASK_OUTPUT=/work/tasks`, then copies `/work/tasks/*/definition.json` to `/work/definition.json` and packs `/work/harbor-task` as `/work/task.tar.gz`. Authoring must produce **exactly one** task directory. A second `submit_task` for the same `taskId` fails because the directory and files already exist. `schemaVersion` `1` task object. Extra keys are rejected at `submit_task` (`additionalProperties: false`). Binary-safe Git patch for held-out tests and minimal fixtures. Paths must stay inside the repository and must not overlap any path in `goldPatch`. Binary-safe Git patch for the implementation required by the prompt. Exclude tests, generated artifacts, formatting churn, and unrelated cleanup. ```json title="submit_task arguments" { "definition": { "schemaVersion": 1, "difficulty": "medium", "taskId": "project-pr-123", "repo": "example/project", "baseCommit": "0123456789abcdef0123456789abcdef01234567", "workdir": ".", "setupCommand": "bun install --frozen-lockfile", "testCommand": "bun test {tests}", "failToPass": ["tests/new-behavior.test.ts"], "passToPass": ["tests/existing-a.test.ts", "tests/existing-b.test.ts"], "testPaths": ["tests/new-behavior.test.ts"], "toolchains": ["node", "bun"], "sourcePr": 123, "sourceUrl": "https://github.com/example/project/pull/123", "prompt": "Implement the requested behavior...", "timeouts": { "setupSeconds": 900, "agentSeconds": 2400, "testsSeconds": 900 }, "resources": { "cpus": 4, "memoryMb": 8192, "storageMb": 20480 } }, "testPatch": "diff --git a/tests/new-behavior.test.ts b/tests/new-behavior.test.ts\n...", "goldPatch": "diff --git a/src/feature.ts b/src/feature.ts\n..." } ``` ```text title="submit_task result" Submitted project-pr-123. ``` Call `submit_task` exactly once. Do not write an alternate task format. ## schemaVersion 1 `schemaVersion` is the literal integer `1`. That is the only accepted task-definition version. Persistence and compile parse the file with Zod `taskDefinitionSchema`. `submit_task` uses a matching TypeBox object that also locks `schemaVersion` to `1` and rejects unknown keys. The worker then requires the authored `sourcePr`, `baseCommit`, and `difficulty` to match the assigned candidate. A mismatch rejects the candidate with `authored task does not match its assigned PR, base commit, and difficulty`. Other `schemaVersion: 1` documents in the repo (export `manifest.json`, coupling reports, eval summaries) are not this object. ## Field reference | Field | Type | Constraints | | --- | --- | --- | | `schemaVersion` | integer | Literal `1` | | `difficulty` | string | `easy`, `medium`, or `hard` | | `taskId` | string | `^[A-Za-z0-9][A-Za-z0-9._-]*$` | | `repo` | string | Non-empty. Free-form label (owner/name or clone URL) | | `baseCommit` | string | Full 40-character hex SHA | | `workdir` | string | Non-empty repository-relative path. `.` is valid | | `setupCommand` | string | Non-empty. Runs from `/app/` during image build | | `testCommand` | string | Must contain the literal `{tests}` placeholder | | `failToPass` | string[] | At least one selector | | `passToPass` | string[] | May be empty (easy has no minimum) | | `testPaths` | string[] | At least one repository-relative held-out path | | `toolchains` | enum[] | At least one of `uv`, `bun`, `go`, `node`, `python`, `rust` | | `sourcePr` | integer | ≥ 1 | | `sourceUrl` | string | Must be a URL when the definition is persisted | | `prompt` | string | Non-empty standalone instruction | | `timeouts.setupSeconds` | integer | ≥ 1 | | `timeouts.agentSeconds` | integer | ≥ 1 | | `timeouts.testsSeconds` | integer | ≥ 1 | | `resources.cpus` | number | `> 0` (not required to be an integer) | | `resources.memoryMb` | integer | ≥ 1 | | `resources.storageMb` | integer | ≥ 1 | Directory name under `SELFBENCH_TASK_OUTPUT` and the Harbor task name `selfbench/`. Pinned history-free snapshot. The compiler runs `git cat-file -e ^{commit}` and `git archive` on that commit. Working directory relative to the extracted snapshot. The compiler resolves `workdir` and `posix.join(workdir, testPath)` against `/repo` and throws `task path escapes repository` if either leaves the tree. Use `.` or a subdirectory. Do not use `..` or an absolute path such as `/app`. Frozen, repository-native install and environment prep. Put dependency install, native builds, fixture generation, and browser/runtime install here — not in `testCommand`. Selectors that must fail on the base snapshot plus the held-out tests, and pass after the gold implementation. Each value is substituted into `{tests}` as its own shell-quoted argument. Values may be paths or runner-native names. Regression selectors that already pass at the base. An empty list compiles to the verifier command `true`. Held-out paths relative to `workdir`. Combined with every path in `test.patch` to mark files the agent patch must not overwrite. Copied verbatim (trimmed, plus a trailing newline) to Harbor `instruction.md`. Must not mention the PR, commits, patches, test names, or the hidden implementation. ## `{tests}` placeholder `testCommand` is a single repository-native command template. At compile time every `{tests}` occurrence is replaced with the selected selectors, each single-quoted: ```text bun test {tests} + failToPass ["tests/new-behavior.test.ts"] → bun test 'tests/new-behavior.test.ts' ``` Zod persistence only requires that the string **contain** `{tests}`. Static audit is stricter and is what accepts or rejects the candidate: | Rule | Blocker | | --- | --- | | `{tests}` appears exactly once | `test command must contain "{tests}" exactly once` | | The placeholder is not wrapped in matching quotes | `test command must expand "{tests}" as an unquoted shell argument list` | | No `failToPass` or `passToPass` selector appears elsewhere in the command | `test command must not hard-code fail-to-pass or pass-to-pass paths outside "{tests}"` | Do not write `bun test "{tests}"`, assign the placeholder to one scalar, chain two equivalent suites (`… {tests} && … {tests}`), or append a selected path beside the placeholder. Use one repository-native runner and bundler. If focused tests cannot be preflighted in the authoring sandbox, reject the candidate instead of guessing `testCommand`. ```bash title="Bun" bun test {tests} ``` ```bash title="npm" npm test -- {tests} ``` ```bash title="pytest" pytest {tests} ``` Pytest, Go, Rust, and custom runners use their own selector syntax. The compiler does not rewrite the runner; it only substitutes the list. ## Toolchains `toolchains` selects setup layers on `ubuntu:24.04`. The compiler always adds implied layers: | Declared | Also installed | Image layer | | --- | --- | --- | | `uv` | — | `uv` into `/usr/local/bin` | | `python` | `uv` | CPython 3.11, 3.12, 3.13 via `uv`; `python`/`python3` point at 3.12 | | `node` | — | Node.js 22.14.0, Corepack, `PLAYWRIGHT_BROWSERS_PATH=/opt/playwright` | | `bun` | `node` | `bun@1.3.14` via npm | | `go` | — | Go 1.25.0 | | `rust` | — | rustup minimal profile, toolchain 1.90.0 | Install order is `uv`, `python`, `node`, `bun`, `go`, `rust`. Choose only the toolchains the clean checkout needs. When `gold.patch` changes a recognized dependency manifest or lockfile, the **verifier** image applies a extracted `dependency-setup.patch`, re-runs `setupCommand`, then resets the tree to the base snapshot. The agent image never sees that layer. Manifest basenames include `package.json`, `package-lock.json`, `bun.lock` / `bun.lockb`, `pnpm-lock.yaml`, `yarn.lock`, `pyproject.toml`, `uv.lock`, `go.mod` / `go.sum`, and `Cargo.toml` / `Cargo.lock`. For Playwright tests, install browsers and OS deps in `setupCommand` (for example `pnpm exec playwright install --with-deps chromium`). The shared `PLAYWRIGHT_BROWSERS_PATH` is available to the verifier user. ## Timeouts and resources The schema has **no defaults**. The authoring prompt tells the model to use these values unless the repository needs more: | Field | Authoring default | | --- | --- | | `timeouts.setupSeconds` | `900` | | `timeouts.agentSeconds` | `2400` | | `timeouts.testsSeconds` | `900` | | `resources.cpus` | `4` | | `resources.memoryMb` | `8192` | | `resources.storageMb` | `20480` | The Harbor compiler maps those fields as: | Definition field | Harbor destination | | --- | --- | | `timeouts.agentSeconds` | `[agent] timeout_sec` | | `timeouts.setupSeconds + timeouts.testsSeconds` | `[verifier] timeout_sec` | | `min(timeouts.testsSeconds, 300)` | `[[verifier.collect]] timeout_sec` (agent-patch collection only) | | `timeouts.setupSeconds + 600` | `[environment]` and `[verifier.environment] build_timeout_sec` | | `resources.cpus` | both environment `cpus` | | `resources.memoryMb` | both environment `memory_mb` | | `resources.storageMb` | both environment `storage_mb` | These limits apply to the compiled Harbor agent and verifier. They are not the authoring-sandbox timeout (`AUTHORING_TIMEOUT_MS` is two hours). ## Static audit beyond the schema After compile, static audit reads `definition.json` together with both patches. Schema-valid JSON can still fail: | Check | Requirement | | --- | --- | | Patch overlap | No path appears in both `gold.patch` and `test.patch` | | Gold size | `easy`: ≥ 1 file / 20 changed lines; `medium`: ≥ 2 / 50; `hard`: ≥ 3 / 100 | | Held-out tests | `test.patch` changes at least one file | | `passToPass` count | `easy`: 0; `medium`: ≥ 1; `hard`: ≥ 2 | | `{tests}` | Exactly once, unquoted, no hardcoded selected paths | Changed-line counts are `+`/`-` hunk lines in the gold patch (`+++`/`---` headers excluded). `failToPass` already has a schema minimum of one. ## What validation repair may change One validation-repair attempt may edit `/work/definition.json` and existing held-out test files only. **Immutable:** `schemaVersion`, `difficulty`, `taskId`, `repo`, `baseCommit`, `workdir`, `sourcePr`, `sourceUrl`, `prompt`. Changing any of these throws `validation repair changed immutable definition field `. **Mutable:** `setupCommand`, `testCommand`, `failToPass`, `passToPass`, `testPaths`, `toolchains`, `timeouts`, `resources`. Coupling repair does not rewrite the definition. After a successful validation repair, `refreshHarborTask` regenerates `tests/test.sh`, `task.toml`, and the Dockerfiles from the new definition. ## Errors | Signal | Cause | | --- | --- | | `SELFBENCH_TASK_OUTPUT is not configured` | `submit_task` ran without the output root | | `submit_task received an incomplete task` | Missing `taskId`, `testPatch`, or `goldPatch` | | Exclusive-create / existing directory | Second submit for the same `taskId` | | `authoring must produce exactly one task; found N` | Zero or multiple directories under the output root | | Zod parse failure | Invalid `schemaVersion`, SHA, URL, empty arrays, or missing `{tests}` | | `test.patch is not a Git patch` / `gold.patch is not a Git patch` | Patch does not start with `diff --git ` | | `task path escapes repository` | `workdir` or a `testPaths` entry leaves the repo | | `test.patch changes no files` | Empty or unparseable held-out patch | | `authored task does not match its assigned PR, base commit, and difficulty` | Identity fields disagree with the candidate | | Audit blockers joined by `; ` | Size, overlap, or `{tests}` rules failed | ## Next How `definition.json` and the two patches become `task.toml`, Dockerfiles, and `dependency-setup.patch`. easy / medium / hard file and line minima, fail-to-pass and pass-to-pass counts. Static audit, Harbor nop/oracle, determinism, and the one validation-repair attempt. What a Harbor trial mounts, and which contract artifacts stay gold-only. Outer `tar.gz` layout. Compiled tasks ship as `harbor-task/`, not as raw `definition.json`. --- ## 20. Harbor task compiler > How authored patches become a Harbor task: task.toml schema 1.4, agent and verifier Dockerfiles, toolchain layers, and dependency-setup.patch. - Page Markdown: https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc/pages/20-harbor-task-compiler.md - Generated: 2026-08-15T02:13:13.196Z ### Source Files - `src/harbor-task.ts` - `tests/harbor-task.test.ts` - `src/contracts.ts` - `src/temporal/activities.ts` - `docs/task-construction.md` - `Dockerfile.sandbox` --- title: "Harbor task compiler" description: "How authored patches become a Harbor task: task.toml schema 1.4, agent and verifier Dockerfiles, toolchain layers, and dependency-setup.patch." --- `compileHarborTask` turns one authored directory (`definition.json`, `test.patch`, `gold.patch`) plus a Git checkout into a Harbor-native `harbor-task/` tree. It pins Harbor `schema_version` to `1.4` and SelfBench `compilerRevision` to `23`, writes `task.toml`, builds separate agent and verifier images from `ubuntu:24.04`, and extracts `dependency-setup.patch` only when the gold patch touches a recognized lockfile or package manifest. Authoring sandboxes invoke the compiler through `sandbox-author`: exactly one task subdirectory under `SELFBENCH_TASK_OUTPUT`, then `tar -czf /work/task.tar.gz -C /work harbor-task`. Temporal stores `definition.json` as a sibling artifact. The Harbor bundle does not include `definition.json`. `refreshHarborTask` rewrites scripts, Dockerfiles, and `task.toml` after validation repair, coupling repair, and export without re-archiving the repository. `Dockerfile.sandbox` is the authoring sandbox (Node 22, Pi, Codex, `gh`). Harbor trial images are generated by the compiler and start from `ubuntu:24.04`. Do not treat the sandbox image as the agent or verifier base. ## Compile inputs Directory that must contain `definition.json` (schemaVersion 1), `test.patch`, and `gold.patch`. Git checkout that contains `definition.baseCommit` (full 40-character SHA). The compiler runs `git cat-file -e ^{commit}` then `git archive --format=tar.gz`. Destination wiped with `rm -rf` then populated as a Harbor task root (later packed as the `harbor-task/` member of a task archive). `loadAuthoredTask` parses `definition.json` with `taskDefinitionSchema`, rejects `workdir` / `testPaths` that escape the repository, and requires both patches to start with `diff --git `. It then path-checks **test** hunks only: the test patch must change at least one file, and no `b/` path may be `.git` or escape the repository. | Input | Required shape | Compiler use | | --- | --- | --- | | `definition.json` | schemaVersion `1` | Timeouts, resources, toolchains, `{tests}` command, prompt, workdir, setup | | `test.patch` | Git patch | Copied to `tests/test.patch`; drives `test.sh` excludes and held-out protection | | `gold.patch` | Git patch | Copied to `solution/gold.patch`; scanned for dependency-manifest hunks | | `baseCommit` | Object in the checkout | History-free `repo.tar.gz` snapshot for both images | Gold-patch paths are not escape-checked. Static audit and authoring rules still require gold and test paths to be disjoint; the compiler does not re-check that split. ## Output layout :::files harbor-task/ ├── task.toml ├── instruction.md ├── .selfbench-manifest.json ├── environment/ │ ├── Dockerfile │ └── repo.tar.gz ├── tests/ │ ├── Dockerfile │ ├── repo.tar.gz │ ├── test.patch │ ├── test.sh │ └── dependency-setup.patch └── solution/ ├── gold.patch └── solve.sh ::: `dependency-setup.patch` is written only when `dependencyManifestPatch(gold.patch)` is non-empty. Both `solve.sh` and `test.sh` are `chmod 0755`. `instruction.md` is `definition.prompt` trimmed plus a trailing newline. Always `selfbench`. `1.4`, also written as `schema_version` in `task.toml`. Integer `23`. Also stored as `[metadata].compiler_revision`. Copied from `definition.taskId`. `refreshHarborTask` rejects a definition whose `taskId` does not match the manifest. `easy`, `medium`, or `hard`. SHA-256 of `JSON.stringify(definition)` (no pretty-print). SHA-256 of the bytes currently in `tests/test.patch`. SHA-256 of the bytes currently in `solution/gold.patch`. ## Compile versus refresh ```mermaid flowchart LR subgraph authored["Authored directory"] def["definition.json"] testp["test.patch"] goldp["gold.patch"] end subgraph checkout["Pinned checkout"] archive["git archive baseCommit"] end subgraph bundle["harbor-task/"] meta["task.toml + instruction.md + .selfbench-manifest.json"] subgraph agentEnv["environment/"] adf["Dockerfile USER agent"] asnap["repo.tar.gz"] end subgraph verEnv["tests/"] vdf["Dockerfile stays root"] vsnap["repo.tar.gz"] held["test.patch + test.sh"] dep["dependency-setup.patch?"] end subgraph goldOnly["solution/ oracle only"] gp["gold.patch"] ss["solve.sh"] end end def --> meta testp --> held goldp --> gp goldp --> dep archive --> asnap archive --> vsnap ``` | Function | Rebuilds snapshot | Rewrites `instruction.md` / `gold.patch` / `solve.sh` | Rewrites Dockerfiles, `task.toml`, `test.sh`, dependency layer | | --- | --- | --- | --- | | `compileHarborTask` | Yes (`git archive`) | Yes | Yes | | `refreshHarborTask` | No | No | Yes; deletes stale `tests/dependency-setup.patch` when gold no longer touches a manifest | Repair stages write a new `tests/test.patch` (and sometimes a new definition), then call `refreshHarborTask`. Export unpacks each accepted bundle, refreshes `harbor-task/`, and re-packs that directory as `tasks/.tar.gz`. A prompt change in a later definition does not update `instruction.md`. Only a full recompile rewrites the agent-visible instruction. ## task.toml schema 1.4 The compiler emits this contract. Harbor then builds `environment/` for the agent and, because `environment_mode = "separate"`, builds `tests/` as the verifier image. ```toml title="Compiled task.toml" schema_version = "1.4" artifacts = ["/opt/selfbench/agent.patch"] [task] name = "selfbench/example" version = "1.0.0" description = "Reproduce example from its authentic engineer request." keywords = ["software-engineering", "private-swe", "selfbench", "medium"] [metadata] selfbench_task_id = "example" difficulty = "medium" repo = "example/repo" base_commit = "0123456789abcdef0123456789abcdef01234567" workdir = "project" source_pr = 1 compiler_revision = 23 [agent] timeout_sec = 2400.0 user = "agent" network_mode = "allowlist" allowed_hosts = ["chatgpt.com", "*.chatgpt.com", "openai.com", "*.openai.com"] [verifier] timeout_sec = 1800.0 user = "root" environment_mode = "separate" network_mode = "public" [[verifier.collect]] service = "main" user = "root" timeout_sec = 300.0 command = "git --git-dir=/opt/selfbench/base.git --work-tree=/app add -A && git --git-dir=/opt/selfbench/base.git --work-tree=/app diff --cached --binary HEAD > /opt/selfbench/agent.patch" [environment] network_mode = "public" build_timeout_sec = 1500.0 cpus = 4 memory_mb = 8192 storage_mb = 20480 [verifier.environment] network_mode = "public" build_timeout_sec = 1500.0 cpus = 4 memory_mb = 8192 storage_mb = 20480 ``` `[task].version` is always `"1.0.0"` (Harbor task version, not the compiler revision). `timeout_sec` and `build_timeout_sec` are written as floats. | `task.toml` field | Source | | --- | --- | | `[agent].timeout_sec` | `timeouts.agentSeconds` | | `[verifier].timeout_sec` | `timeouts.setupSeconds + timeouts.testsSeconds` | | `[[verifier.collect]].timeout_sec` | `min(timeouts.testsSeconds, 300)` | | `[environment].build_timeout_sec` | `timeouts.setupSeconds + 600` | | `[verifier.environment].build_timeout_sec` | `timeouts.setupSeconds + 600` | | `cpus` / `memory_mb` / `storage_mb` | `resources.cpus` / `memoryMb` / `storageMb` on both environment tables | Agent network is hardcoded to the ChatGPT / OpenAI allowlist. Verifier and image builds use `network_mode = "public"` so `setupCommand` and test-time installs can reach package registries. Collect runs in the **agent** service (`service = "main"`) against `/opt/selfbench/base.git`, which exists only in the agent image. The resulting `/opt/selfbench/agent.patch` is the Harbor artifact the verifier consumes. `solution/` is for the Harbor `oracle` agent (`solve.sh` applies `gold.patch`); coding-agent trials do not mount it. ## Agent and verifier Dockerfiles Both images share `toolchainDockerfile` + `repositoryDockerfile`, then diverge. Shared repository layer: 1. `COPY repo.tar.gz` and extract to `/app`. 2. `git init`, identity `selfbench@local` / `selfbench`, commit message `base`. 3. `mkdir -p /opt/uv-cache` (`chmod 777`). 4. `cd /app/` and `bash -lc ''`. 5. If `python` is selected, `uv pip install --python 'setuptools>=70' wheel`. ```dockerfile title="environment/Dockerfile (agent)" # toolchain + repository layers, then: RUN useradd --create-home --shell /bin/bash agent RUN git -C /app reset --hard -q HEAD \ && git -C /app clean -fdq \ && mkdir -p /opt/selfbench \ && cp -a /app/.git /opt/selfbench/base.git \ && chown -R agent:agent /app /home/agent /opt/uv-cache \ && chown -R root:root /opt/selfbench \ && chmod 700 /opt/selfbench ENV UV_CACHE_DIR=/home/agent/.cache/uv \ UV_NO_BUILD_ISOLATION=1 USER agent WORKDIR /app ``` ```dockerfile title="tests/Dockerfile (verifier)" # toolchain + repository layers, optional gold dependency layer, then: RUN useradd --create-home --shell /bin/bash verifier \ && chown -R verifier:verifier /app /opt/uv-cache \ && mkdir -p /opt/selfbench \ && chmod 700 /opt/selfbench ENV UV_CACHE_DIR=/home/verifier/.cache/uv \ UV_NO_BUILD_ISOLATION=1 COPY test.patch test.sh /tests/ RUN chmod 700 /tests && chmod 600 /tests/test.patch && chmod +x /tests/test.sh WORKDIR /app ``` | Concern | Agent image | Verifier image | | --- | --- | --- | | Harbor user | `agent` | `root` (tests run via `runuser -u verifier`) | | Base git for collect | `/opt/selfbench/base.git` (`chmod 700`, root-owned) | Empty `/opt/selfbench` only | | Gold patch in Dockerfile | Never | Only the filtered `dependency-setup.patch` layer | | Held-out tests | Absent | `COPY test.patch test.sh /tests/` | | Build context | `environment/` (`Dockerfile` + `repo.tar.gz`) | `tests/` (`Dockerfile` + `repo.tar.gz` + tests + optional dep patch) | The agent layer resets `/app` after `setupCommand` so the trial starts from a clean tree while `/opt/uv-cache` keeps warmed packages. The verifier leaves `/app` at the post-setup state (or post gold-dep setup + reset; see below). ## Toolchain layers `definition.toolchains` is a non-empty list of `uv` | `python` | `node` | `bun` | `go` | `rust`. The compiler installs implied peers: `python` adds `uv`; `bun` adds `node`. Layers always emit in this order: `uv`, `python`, `node`, `bun`, `go`, `rust`. Base image env on every task: - `FROM ubuntu:24.04` - `DEBIAN_FRONTEND=noninteractive` - `UV_LINK_MODE=copy`, `UV_CACHE_DIR=/opt/uv-cache`, `UV_PYTHON_INSTALL_DIR=/usr/local/share/uv/python`, `UV_PYTHON_BIN_DIR=/usr/local/bin` - `RUSTUP_HOME=/usr/local/rustup`, `CARGO_HOME=/usr/local/cargo`, `COREPACK_HOME=/opt/corepack` - `PATH` prepends `/usr/local/go/bin` and `/usr/local/cargo/bin` - apt: `bash` `build-essential` `ca-certificates` `curl` `git` `jq` `passwd` `pkg-config` `procps` `unzip` `xz-utils` | Toolchain | Install | | --- | --- | | `uv` | `https://astral.sh/uv/install.sh` into `/usr/local/bin` (`UV_NO_MODIFY_PATH=1`) | | `python` | `uv python install 3.11 3.12 3.13`; `python` and `python3` symlink to 3.12 | | `node` | Node `v22.14.0` (amd64→`x64`, arm64→`arm64`); `corepack enable`; `PLAYWRIGHT_BROWSERS_PATH=/opt/playwright` | | `bun` | `npm install --global bun@1.3.14` (after Node) | | `go` | Go `1.25.0` tarball for `dpkg --print-architecture` | | `rust` | rustup minimal profile, default toolchain `1.90.0`, no PATH mutation | `setupCommand` must finish everything `testCommand` needs: frozen package-manager install, native builds, fixture generation, and browser/OS deps (for example `pnpm exec playwright install --with-deps chromium`). The compiler only exposes `PLAYWRIGHT_BROWSERS_PATH`; it does not install browsers. ## dependency-setup.patch When gold changes a recognized manifest, the compiler copies **only those hunks** into `tests/dependency-setup.patch` and injects a verifier build layer: ```dockerfile COPY dependency-setup.patch /tmp/selfbench-dependency-setup.patch RUN git -C /app apply --binary --whitespace=nowarn /tmp/selfbench-dependency-setup.patch \ && cd /app/ \ && bash -lc '' \ && git -C /app reset --hard -q HEAD \ && git -C /app clean -fdq \ && rm /tmp/selfbench-dependency-setup.patch ``` The image then has gold-state dependencies while `/app` source is back at the base commit. Oracle can apply `gold.patch` on a clean tree without the agent image ever seeing gold. Matching uses the `b/` basename: | Ecosystem | Basenames | | --- | --- | | JavaScript | `package.json`, `package-lock.json`, `npm-shrinkwrap.json`, `yarn.lock`, `pnpm-lock.yaml`, `bun.lock`, `bun.lockb`, `deno.lock` | | Python | `pyproject.toml`, `uv.lock`, `poetry.lock`, `Pipfile`, `Pipfile.lock`, `requirements*.txt` | | Go / Rust | `go.mod`, `go.sum`, `Cargo.toml`, `Cargo.lock` | `src/package-handler.ts` is not a manifest. `apps/api/pyproject.toml` is. Implementation files in the same gold patch are dropped from the extracted patch. ## Verifier script `tests/test.sh` is generated at compile/refresh time. It does **not** run `setupCommand` (that already ran at image build). Harbor collect must have written `/opt/selfbench/agent.patch` first. Missing file → `patch_applied=0`. Empty file (typical nop) → skip apply, leave `patch_applied=1`. Non-empty file → `git apply --binary --whitespace=nowarn` with `--exclude` for every held-out path (union of `workdir`+`testPaths` and `b/` paths in `test.patch`). `git restore` and `git clean` each protected path from `HEAD`, apply `/tests/test.patch`, then `chown root` and `chmod a-w,go+rX` those paths. Delete `/tests/test.patch` from the running container. `cd /app/`. Copy `/opt/uv-cache` to a verifier-owned temp dir. Substitute `{tests}` with shell-quoted `failToPass` or `passToPass` (empty `passToPass` becomes `true`). Each command runs as `verifier` with `UV_NO_BUILD_ISOLATION=1`. Fail-to-pass is executed twice; `deterministic=1` only if both succeed. Reward is `1` only when `patch_applied`, `fail_to_pass`, `pass_to_pass`, and `deterministic` are all `1`. The script always `exit 0`; Harbor reads `/logs/verifier/reward.json`. `run_verifier_command` retries up to 3 times with `10 * attempt` seconds of backoff when the log matches infrastructure noise (`ECONNRESET`, `ETIMEDOUT`, `ENOTFOUND`, `registry.npmjs`, `FetchError`, and similar). Assertion failures fail immediately. After each attempt it `pkill -KILL` processes owned by `verifier`. ```json title="/logs/verifier/reward.json" { "reward": 1, "patch_applied": 1, "fail_to_pass": 1, "pass_to_pass": 1, "deterministic": 1, "setup_completed": 1, "fail_to_pass_exit_code": 0, "fail_to_pass_repeat_exit_code": 0, "pass_to_pass_exit_code": 0 } ``` `setup_completed` tracks whether the agent patch stage succeeded; it is not a second setup command. Temporal nop/oracle gates read these numeric fields (nop expects `fail_to_pass=0` and `pass_to_pass>=1`; oracle expects all four success bits). Oracle solution script: ```bash #!/bin/bash set -euo pipefail git -C /app apply --binary --whitespace=nowarn /solution/gold.patch ``` ## Errors | Message | When | | --- | --- | | `test.patch is not a Git patch` / `gold.patch is not a Git patch` | File does not start with `diff --git ` | | `test.patch changes no files` | No `diff --git a/… b/…` hunks | | `task path escapes repository: …` | `workdir` or `workdir`+`testPaths` resolves outside the repo root | | `test patch path escapes repository: …` | A test `b/` path is `.git` or escapes the repo | | `bundle task does not match ` | `refreshHarborTask` manifest `taskId` ≠ definition `taskId` | | `git cat-file` non-zero | `baseCommit` is not a commit in `repositoryDirectory` | | Zod parse error | `definition.json` fails `taskDefinitionSchema` (including missing `{tests}`) | | `authoring must produce exactly one task; found N` | `sandbox-author` saw zero or multiple task directories | ## Troubleshooting | Symptom | Likely cause | | --- | --- | | Oracle fails apply / stale deps after a lockfile bump | Gold changed a manifest but the name is not in the basename list, so no `dependency-setup.patch` layer | | Agent image contains gold | It must not. `environment/Dockerfile` never copies `gold.patch`. If you see gold in the agent context, the archive is not compiler output | | Verifier cannot reach npm/PyPI | `setupCommand` or tests need public network; that is enabled on verifier/build, not on the agent allowlist | | `{tests}` still in `test.sh` | Compile/refresh did not run after the definition change | | `instruction.md` stale after repair | Refresh does not rewrite it; only `compileHarborTask` does | | Playwright browsers missing | Not installed by the toolchain layer; they belong in `setupCommand` | | Harbor infrastructure failure on build | Image build timeout is `setupSeconds + 600`; a long `setupCommand` can exceed it | | Reward all zeros, `patch_applied=0` | Collect did not produce `/opt/selfbench/agent.patch`, or apply failed (including protected-path excludes) | Harbor validation and eval invoke `harbor run --path `. The worker uses `--agent nop` then `--agent oracle`. Downstream Codex trials use the same compiled tree. ## Related pages `definition.json` fields, `{tests}` placeholder rules, and `submit_task` outputs the compiler consumes. What a trial mounts versus gold-only `solution/` and held-out tests. Outer `tar.gz`, `manifest.json`, and per-task `harbor-task/` archives. Nop/oracle gates that read `reward.json`, then `refreshHarborTask` after a repair. Run an extracted compiled task with `harbor run` or `self-bench-eval`. --- ## 21. Export format > Outer tar.gz layout, manifest.json fields, per-task harbor-task contents, SHA-256 checks, and what the archive excludes. - Page Markdown: https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc/pages/21-export-format.md - Generated: 2026-08-15T02:12:35.731Z ### Source Files - `src/temporal/activities.ts` - `src/harbor-task.ts` - `review/src/export-loader.ts` - `review/src/types.ts` - `src/artifacts.ts` - `docs/evaluations.md` - `docs/task-construction.md` --- title: "Export format" description: "Outer tar.gz layout, manifest.json fields, per-task harbor-task contents, SHA-256 checks, and what the archive excludes." --- `buildExport` writes one unencrypted gzip tar after Temporal reaches the `exporting` phase. The archive contains only tasks that finished `accepted`. The worker stores it as `runs//export/attempt-/selfbench-.tar.gz` with `contentType` `application/gzip`. `GET /v1/runs/:runId/export` streams that object; `self-bench download` and `self-bench run --output` write it locally after a streamed SHA-256 check. The archive includes history-free repository snapshots, held-out tests, and the gold patch. It is not encrypted. Treat it as private benchmark material from a trusted SelfBench deployment. ## Outer archive The outer tar is created with `tar -czf … -C manifest.json tasks`. The `tasks/` directory is always present, even when `acceptedCount` is `0`. :::files selfbench-.tar.gz ├── manifest.json └── tasks/ ├── .tar.gz └── .tar.gz ::: | Path | Role | | --- | --- | | `manifest.json` | UTF-8 JSON, pretty-printed, trailing newline. Schema `1`. | | `tasks/.tar.gz` | One native Harbor task. The filename stem is `taskId`. | Download names: | Surface | Filename | | --- | --- | | API `Content-Disposition` | `selfbench-.tar.gz` | | Review UI cleaned download | `selfbench--cleaned.tar.gz` | `taskId` matches `^[A-Za-z0-9][A-Za-z0-9._-]*$`. The review loader requires a `tasks/.tar.gz` entry for every `manifest.tasks[]` row and fails with `export is missing tasks/.tar.gz` when one is absent. ## manifest.json `schemaVersion` is the literal `1`. `acceptedCount` is `tasks.length` after the accepted-only filter. `candidateCounts` is the **requested** authoring budget from the run (`easy` / `medium` / `hard`), not the number of accepted tasks. Always `1`. Workflow id: `^[a-z0-9][a-z0-9-]{2,62}$`. `{ easy, medium, hard }` integers from the run request (each 0–100, sum 1–100). `{ url, commit }`. `commit` is the 40-character HEAD pin of the source checkout at submit time. Per-task snapshots use each task’s `baseCommit`, which can differ. Build identity copied from the run request. Number of packed task archives. `{ taskId, sha256 }[]`. `sha256` is the lowercase hex digest of that inner `tasks/.tar.gz` **after** `refreshHarborTask` and re-tar. `version` fields: | Field | Type | Meaning | | --- | --- | --- | | `selfbenchCommit` | 40-char SHA | SelfBench image / CLI commit used for the run | | `executionBackend` | `"docker"` \| `"modal"` | Sandbox backend that authored and validated | | `sandboxImage` | string | Worker sandbox image | | `schema` | `1` | Run-request schema | The run’s `authoring` block (`provider`, `model`, `reasoningEffort`) is **not** copied into `manifest.json`. ```json manifest.json { "schemaVersion": 1, "runId": "sb-20260814-example", "candidateCounts": { "easy": 10, "medium": 10, "hard": 10 }, "repository": { "url": "https://github.com/acme/payments", "commit": "0123456789abcdef0123456789abcdef01234567" }, "version": { "selfbenchCommit": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", "executionBackend": "modal", "sandboxImage": "selfbench-sandbox:local", "schema": 1 }, "acceptedCount": 2, "tasks": [ { "taskId": "payments-pr-1842", "sha256": "b3c1c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0" }, { "taskId": "payments-pr-1901", "sha256": "d4e2d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1" } ] } ``` ## Per-task Harbor archive Each inner archive is `tar -czf … -C harbor-task`. Extract with `--strip-components=1` if you want `task.toml` at the destination root. `self-bench-eval` copies `harbor-task/` to `/tasks//`. Before packing, `buildExport` loads the stored `definition.json` artifact and runs `refreshHarborTask` so `task.toml`, Dockerfiles, `tests/test.sh`, optional `tests/dependency-setup.patch`, and `.selfbench-manifest.json` match the accepted definition (including any validation-repair edits). :::files .tar.gz └── harbor-task/ ├── task.toml ├── instruction.md ├── .selfbench-manifest.json ├── environment/ │ ├── Dockerfile │ └── repo.tar.gz ├── tests/ │ ├── Dockerfile │ ├── repo.tar.gz │ ├── test.patch │ ├── test.sh │ └── dependency-setup.patch # only when gold changes a lockfile/manifest └── solution/ ├── gold.patch └── solve.sh ::: | Path | Packed? | Notes | | --- | --- | --- | | `task.toml` | always | Harbor `schema_version` `"1.4"`. Name `selfbench/`. | | `instruction.md` | always | Trimmed `definition.prompt` plus a trailing newline. | | `.selfbench-manifest.json` | always | Compiler sidecar; not a Harbor input. | | `environment/Dockerfile` | always | Agent image. Does not copy `gold.patch`. | | `environment/repo.tar.gz` | always | `git archive` of `definition.baseCommit`. | | `tests/Dockerfile` | always | Verifier image. Copies `test.patch` and `test.sh`. | | `tests/repo.tar.gz` | always | Same snapshot bytes as `environment/repo.tar.gz`. | | `tests/test.patch` | always | Held-out tests. Must start with `diff --git `. | | `tests/test.sh` | always | Substitutes `{tests}`; mode `0755`. | | `tests/dependency-setup.patch` | optional | Gold hunks for recognized lockfiles/manifests only. Removed on refresh when gold no longer changes them. | | `solution/gold.patch` | always | Reference implementation. Harbor mounts `solution/` only for the oracle agent. | | `solution/solve.sh` | always | `git apply` of `gold.patch`; mode `0755`. | | `definition.json` | **not packed** | Stored as a separate run artifact and used at refresh time. The review UI shows `Missing definition.json` unless a later packer added one. | `.selfbench-manifest.json` fields: | Field | Meaning | | --- | --- | | `generator` | `"selfbench"` | | `harborSchemaVersion` | `"1.4"` | | `compilerRevision` | Integer written by the compiler (currently `23`) | | `taskId` | Must match the accepted definition | | `difficulty` | `easy` \| `medium` \| `hard` | | `definitionSha256` | SHA-256 of `JSON.stringify(definition)` | | `testPatchSha256` | SHA-256 of `tests/test.patch` | | `goldPatchSha256` | SHA-256 of `solution/gold.patch` | `refreshHarborTask` refuses a bundle whose sidecar `taskId` does not match the definition. `repo.tar.gz` is a `git archive --format=tar.gz` of the task base commit. It has no `.git` directory. The agent and verifier Dockerfiles unpack it and create a fresh repo with a single `base` commit. ## SHA-256 checks Digests are lowercase hex SHA-256 from Node `crypto`. They detect accidental corruption. They are not signatures. ```mermaid flowchart TB subgraph Store["Artifact store"] Put["putFile computes sha256 and sizeBytes"] Get["get / openRead reject digest or size mismatch"] end subgraph HTTP["GET /v1/runs/:runId/export"] H1["content-type from ArtifactRef"] H2["content-length = sizeBytes"] H3["x-content-sha256 = sha256"] end subgraph Clients["Clients"] CLI["self-bench download / --output"] Review["Review UI loadExport"] Eval["self-bench-eval --export"] end subgraph Interior["Inside the tar"] Inner["manifest.tasks[].sha256 of each inner tar.gz"] end Put --> Get --> H1 Get --> H2 Get --> H3 H3 --> CLI HTTP --> Review CLI -->|"does not read"| Inner Review -->|"does not verify"| Inner Eval -->|"lists tasks/*.tar.gz; ignores manifest"| Inner ``` ### Outer archive (authoritative for download) The store records `{ uri, sha256, sizeBytes, contentType }` on `putFile`. Local and GCS reads recompute the digest and byte length and throw `artifact integrity check failed: ` on mismatch. When `status.export` is present, the API streams that object with: | Header | Value | | --- | --- | | `content-type` | `application/gzip` | | `content-length` | `status.export.sizeBytes` | | `content-disposition` | `attachment; filename="selfbench-.tar.gz"` | | `x-content-sha256` | `status.export.sha256` | `self-bench download RUN_ID OUTPUT.tar.gz` and `self-bench run --output OUTPUT.tar.gz`: 1. `GET /v1/runs//export` (Bearer `SELFBENCH_API_TOKEN` when set). 2. Require `x-content-sha256` and a response body. Missing header → `SelfBench API returned an export without integrity metadata`. 3. Create `OUTPUT` with exclusive `wx` (will not overwrite an existing file). 4. Hash the stream while writing. 5. On digest mismatch, delete `OUTPUT` and throw `downloaded export failed its SHA-256 integrity check`. 6. On success, print `{ "runId", "output" }`. `--output` implies `--wait`. Download starts only after phase `complete`. The review UI fetches the same route and gunzips the blob. It does not re-hash against `x-content-sha256`. Integrity for that path is the store check on `openRead`. ### Per-task digests (manifest only) Each `tasks[].sha256` is `sha256(readFile(tasks/.tar.gz))` after refresh. No current consumer (`loadExport`, `self-bench-eval`, Harbor) checks those values. A cleaned review archive copies the original inner bytes and keeps the original per-task digests; it only rewrites `acceptedCount` and the `tasks` array. Because `manifest.json` lives inside the same tar as the files it names, a rewritten archive can update both together. Extract only from a SelfBench API you trust, then keep the file private. ## What the archive excludes The export is accepted Harbor tasks plus the run pin. Everything else stays in the artifact store or never left the authoring sandbox. | Excluded | Where it actually lives | | --- | --- | | Rejected, blocked, or `infrastructure_failed` candidates | Run status `tasks[]`, not the tar | | Git history, remotes, worktrees, uncommitted files | Snapshots are `git archive` of `baseCommit` only | | Local Pi / Claude Code / Codex sessions | Sanitized JSONL at `runs//input/provenance.jsonl` | | Per-candidate provenance JSON | `runs//provenance/.json` | | Author, reviewer, and repair transcripts / sandbox logs | `runs//authoring/`, `reviews/`, `validation/`, `*-repairs/` | | Stored `definition.json` | `runs//authoring//definition.json` (and repair checkpoints) | | Static audit and coupling-review reports | `runs//audits/`, `runs//reviews/` | | Harbor nop / oracle job artifacts | Validation attempt prefixes | | Source transcripts inside gold or test patches | Authoring skill forbids them; not a separate export file | `candidateCounts` in the manifest can be larger than `acceptedCount`. Rejected candidates are not replaced and are not packed. ## Extract a task ```bash self-bench download "$RUN_ID" ./self-bench-evals.tar.gz # or: self-bench run --repo /path/to/repo --easy-count 1 --output ./self-bench-evals.tar.gz ``` Confirm the CLI printed `{ "runId", "output" }` and the file exists. ```bash mkdir -p ./export tar -xzf ./self-bench-evals.tar.gz -C ./export jq '.acceptedCount, .tasks[0].taskId' ./export/manifest.json ``` ```bash TASK_ID="$(jq -r '.tasks[0].taskId' ./export/manifest.json)" mkdir -p "./selected-task" tar -xzf "./export/tasks/${TASK_ID}.tar.gz" -C ./selected-task test -f ./selected-task/harbor-task/task.toml ``` Point Harbor at `./selected-task/harbor-task`. ```bash extract-all-for-harbor mkdir -p ./export ./self-bench-tasks tar -xzf ./self-bench-evals.tar.gz -C ./export for archive in ./export/tasks/*.tar.gz; do task_id="$(basename "$archive" .tar.gz)" mkdir -p "./self-bench-tasks/${task_id}" tar -xzf "$archive" --strip-components=1 -C "./self-bench-tasks/${task_id}" done ``` ```bash self-bench-eval self-bench-eval --export ./self-bench-evals.tar.gz --jobs ./matrix-jobs ``` `self-bench-eval --export` extracts every `tasks/*.tar.gz` that contains `harbor-task/task.toml`. It does not read `manifest.json` and does not compare per-task SHA-256. If `/tasks/` already has task directories, it reuses them and does not unpack again. The review loader is stricter: missing `manifest.json` throws `export is missing manifest.json`, and extra inner archives that are not listed in `manifest.tasks` are ignored. ## Cleaned review archives `buildCleanExport` writes a new outer gzip tar: 1. Drop rows whose `taskId` is marked for removal. 2. Set `acceptedCount` to the kept count. 3. Copy kept `tasks/.tar.gz` bytes unchanged. 4. Leave `runId`, `candidateCounts`, `repository`, and `version` as they were. It does not re-run `refreshHarborTask` and does not recompute inner SHA-256 values. ## Errors | Signal | Cause | | --- | --- | | `409` `{ "error": "run export is not ready" }` | Status has no `export` yet (`queued`, `discovering`, `authoring`, `exporting`, or a failed/blocked/cancelled terminal). | | `SelfBench API returned an export without integrity metadata` | Response lacked `x-content-sha256` or a body. | | `downloaded export failed its SHA-256 integrity check` | Stream digest ≠ header. Destination file is deleted. | | `EEXIST` from `open(…, "wx")` | Output path already exists. Choose a new path. | | `export is missing manifest.json` | Review UI / `loadExport` on a non-export gzip. | | `export is missing tasks/.tar.gz` | Manifest row with no matching inner archive. | | `artifact integrity check failed: ` | Stored object bytes no longer match the ArtifactRef. | | `bundle task does not match ` | Export refresh: sidecar `taskId` ≠ stored definition. | Compare a local file to the status object when you did not use the CLI: ```bash # status.export.sha256 from GET /v1/runs/:runId shasum -a 256 ./self-bench-evals.tar.gz ``` ## Related pages Extract an export and run one task or the Codex matrix. Inspect accepted tasks, drop rows, and download a cleaned tar.gz. How patches become `task.toml`, Dockerfiles, and `dependency-setup.patch`. `definition.json` fields used at export refresh time. Download, wait, and integrity headers on `SELFBENCH_API_URL`. `409` export-not-ready and SHA-256 mismatch. --- ## 22. Execution backends > Docker versus Modal sandboxes: images, default concurrency, resource limits, inactivity timeouts, and Harbor environment pairing. - Page Markdown: https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc/pages/22-execution-backends.md - Generated: 2026-08-15T02:13:01.560Z ### Source Files - `src/sandbox.ts` - `src/docker-executor.ts` - `src/modal-executor.ts` - `src/modal-auth.ts` - `src/config.ts` - `src/cli.ts` - `Dockerfile.sandbox` --- title: "Execution backends" description: "Docker versus Modal sandboxes: images, default concurrency, resource limits, inactivity timeouts, and Harbor environment pairing." --- `createSandboxExecutor` selects `DockerSandboxExecutor` or `ModalSandboxExecutor` from `SELFBENCH_EXECUTION_BACKEND` (`docker` by default). Those sandboxes run discovery, authoring, coupling review, and repair. Harbor nop and oracle gates use a second knob, `SELFBENCH_HARBOR_ENVIRONMENT`, passed as `harbor run --env`. `self-bench up --backend` writes both knobs to the same value and records `executionBackend` plus `sandboxImage` on each run. ## Two knobs, one local pairing | Knob | Controls | Local `self-bench up` | Config default | | --- | --- | --- | --- | | `SELFBENCH_EXECUTION_BACKEND` | Worker `SandboxExecutor` | `--backend` | `docker` | | `SELFBENCH_HARBOR_ENVIRONMENT` | Worker `harbor run --env` | same `--backend` | falls back to the execution backend | `self-bench up` always pairs them. A Temporal Cloud worker or a manually composed stack can set the two independently. If either is `docker`, worker startup runs `docker info` and `docker compose version` (30s each) before polling Temporal. Companion CLIs do **not** read those worker keys. `self-bench-eval`, `self-bench-validate`, and `self-bench-agent-smoke` default `--environment modal` on the machine that invokes Harbor. `self-bench-eval --environment` defaults to `modal` even when the generating worker used Docker. Pass `--environment docker` to match a Docker-validated export. ```mermaid flowchart TB subgraph cli [CLI and Compose] UP["self-bench up --backend"] EXEC["SELFBENCH_EXECUTION_BACKEND"] HARB["SELFBENCH_HARBOR_ENVIRONMENT"] UP --> EXEC UP --> HARB end subgraph worker [Worker process] LOAD["loadConfig"] FACTORY["createSandboxExecutor"] DX["DockerSandboxExecutor"] MX["ModalSandboxExecutor"] VAL["validateTask"] EXEC --> LOAD HARB --> LOAD LOAD --> FACTORY FACTORY -->|"kind docker"| DX FACTORY -->|"kind modal"| MX LOAD --> VAL end subgraph dockerHost [Host Docker daemon] SIMG["selfbench-sandbox:local"] CTR["selfbench-RUN-STAGE-suffix container + volume"] DX --> SIMG DX --> CTR end subgraph modalCloud [Modal] APP["apps.fromName selfbench"] MSB["sandboxes.create workdir /work"] MX --> APP MX --> MSB end subgraph harborRt [Harbor from the worker image] HCMD["harbor run --env docker|modal"] TOML["task.toml cpus / memory_mb / storage_mb"] VAL --> HCMD HCMD --> TOML end ``` ## Choose a backend Use a profile at `~/.modal.toml` (`modal token new`) or set `MODAL_TOKEN_ID` and `MODAL_TOKEN_SECRET`. Empty token env vars are deleted at worker startup so they cannot override a mounted profile. ```bash self-bench up --backend modal ``` Override the profile path when it is not `~/.modal.toml`: ```bash self-bench up --backend modal --modal-config /absolute/path/to/.modal.toml ``` `--modal-config` is rejected unless `--backend` is `modal`. The CLI prints `SelfBench is running with the modal backend at http://127.0.0.1:8080`. The worker log includes `activity concurrency 20` unless `SELFBENCH_ACTIVITY_CONCURRENCY` overrides it. ```bash self-bench up --backend docker ``` This builds `Dockerfile.sandbox` as `selfbench-sandbox:local`, then `docker compose up -d --build`. Compose pins `SELFBENCH_DOCKER_IMAGE` to `selfbench-sandbox:local`. The worker service runs as `root`, joins `DOCKER_GID` (default `0`), and mounts `/var/run/docker.sock`. The API service does not receive the socket or model credentials. The CLI prints `SelfBench is running with the docker backend at http://127.0.0.1:8080`. The worker log includes `activity concurrency 1` unless `SELFBENCH_ACTIVITY_CONCURRENCY` overrides it. `--backend` defaults to `docker`. The README quickstart uses `--backend modal` so sandboxes leave the host. ## Backend comparison | Property | Docker | Modal | | --- | --- | --- | | Executor | `DockerSandboxExecutor` | `ModalSandboxExecutor` | | Image | `SELFBENCH_DOCKER_IMAGE` (`selfbench-sandbox:local`) | `SELFBENCH_MODAL_IMAGE` (`node:22-bookworm`) plus inline install layers | | App / environment | host Docker daemon | `SELFBENCH_MODAL_APP` (`selfbench`), optional `SELFBENCH_MODAL_ENVIRONMENT` | | Default activity concurrency | `1` | `20` | | CPU / memory defaults | `--cpus 4`, `--memory 8192m` | `cpu: 4`, `memoryMiB: 8192` | | Storage limit | not applied by the executor | not applied by the executor | | Hard timeout | `SandboxRequest.timeoutMs` on `docker start --attach` | same value on `sandboxes.create({ timeoutMs })` | | Inactivity timeout | `runCommand` kills the attach process | executor timer calls `sandbox.terminate()` | | Extra idle cap | none | `idleTimeoutMs = min(timeoutMs, 10 minutes)` | | Workdir | `/work` named volume | `/work` | | Secrets | merged into container `--env` | `secrets.fromObject` on `exec` | | Missing required output after exit `0` | throws | omitted; caller diagnoses from stdout/stderr | | Cleanup | `docker rm --force` and `docker volume rm --force` | `sandbox.terminate()` | | `close()` | no-op | `ModalClient.close()` | `SELFBENCH_ACTIVITY_CONCURRENCY` is an integer from 1 to 100. It becomes Temporal `maxConcurrentActivityTaskExecutions`. Docker defaults to 1 because sandboxes share the host. ## Images ### Docker: `Dockerfile.sandbox` `self-bench up --backend docker` builds: ```bash docker build -f Dockerfile.sandbox -t selfbench-sandbox:local . ``` `up --backend modal` skips that build. The image is `node:22-bookworm` plus: | Install | Pin | | --- | --- | | apt | `bash` `ca-certificates` `curl` `git` `jq` `ripgrep` `unzip` `xz-utils` | | GitHub CLI | `gh` 2.89.0 (`amd64` and `arm64` SHA-256 checked) | | Pi | `@earendil-works/pi-coding-agent@0.84.0` | | Codex | `@openai/codex@0.146.1` | `WORKDIR` is `/work`. Compose does not interpolate `SELFBENCH_DOCKER_IMAGE`; the local stack always uses `selfbench-sandbox:local`. A standalone worker (Temporal Cloud) honors the env var / default. ### Modal: registry base plus dockerfile commands `ModalSandboxExecutor` calls `images.fromRegistry(SELFBENCH_MODAL_IMAGE)` and then the same tool installs as `Dockerfile.sandbox`, with two differences: - `gh` is the **amd64** 2.89.0 tarball only (SHA `d0422caade520530e76c1c558da47daebaa8e1203d6b7ff10ad7d6faba3490d8`). - `PATH` is forced to `/root/.local/bin` plus the usual Unix locations. The Modal app is created if missing. Each sandbox is tagged `run_id` and `stage`. ### Worker image (not the sandbox) `Dockerfile` builds the API/worker image (`selfbench:local` in Compose). It ships the Docker CLI 29.4.0 and `harbor[modal]==0.20.1.dev202608040148`. Harbor validation therefore runs **inside the worker container**, not inside `selfbench-sandbox:local`. ## Sandbox contract Every executor implements: ```ts run(request: SandboxRequest, options?: SandboxRunOptions): Promise close(): void ``` Workflow or companion run id. Docker names use the first 20 characters; Modal uses 24. Both append stage (16 chars) and an 8-character UUID suffix. Activity label. Docker names become `selfbench-{runId}-{stage}-{suffix}` (lowercase `[a-z0-9_.-]`). Modal names omit the `selfbench-` prefix and keep mixed case `[a-zA-Z0-9._-]`. Executed in `/work`. Authoring uses `bash -lc` around the Pi bootstrap; review/repair invoke bundled Node scripts. Written before exec. Docker rejects any path that is not under `/work/`. Copied after the process exits. Docker throws if the process exits 0 without a listed path. Non-secret env (repo URL, model name, output paths). Short-lived credentials only: `OPENAI_API_KEY` or a stage-specific `SELFBENCH_PI_AUTH_JSON` / `SELFBENCH_CODEX_AUTH_JSON`, plus `GH_TOKEN` when cloning. Hard wall clock for the sandbox process. No stdout/stderr for this long raises `InactivityTimeoutError` (`{scope} produced no output for {timeoutMs}ms`). Defaults to `4` when omitted. Defaults to `8192` when omitted. `SandboxRunOptions.signal` force-removes the Docker container or terminates the Modal sandbox. `onProgress` reports stdout/stderr byte counts for Temporal heartbeats. Stdout/stderr are capped at 8 MiB (`RollingOutput`). Older output is replaced with `[selfbench: earlier output truncated]`. ## Resource limits There are two resource surfaces. Do not treat them as one. ### Authoring-family sandboxes Discovery, authoring, review, and repair omit `cpu` / `memoryMiB` in the worker and therefore get **4 CPU and 8192 MiB**. `self-bench-repair` sets those same values explicitly. Neither executor sets a disk quota. ### Harbor task resources Authored `definition.json` `resources` compile into `task.toml`: ```toml [environment] cpus = 4 memory_mb = 8192 storage_mb = 20480 [verifier.environment] cpus = 4 memory_mb = 8192 storage_mb = 20480 ``` Authoring prompt defaults are 4 CPU, 8192 MB memory, 20480 MB storage, and timeouts 900 / 2400 / 900 seconds (setup / agent / tests). Those Harbor limits apply when Harbor builds agent and verifier environments, not when `DockerSandboxExecutor` creates the authoring container. `storageMb` is a Harbor `task.toml` field. Changing it does not add a Docker `--storage-opt` or a Modal disk request on the SelfBench sandbox. ## Timeouts | Stage | `stage` value | Hard timeout | Inactivity | Temporal start-to-close | | --- | --- | --- | --- | --- | | Discovery shard | `discover-{wave}-{shardIndex}` | 45 min | 8 min | 1 hour, 3 attempts | | Author | `author-{candidateId}` | 2 hours | 8 min | 7 hours, 4 attempts | | Harbor nop / oracle | Harbor CLI, not `SandboxExecutor` | 3 hours per `harbor run` | — | same 7-hour activity budget | | Validation repair | `validation-repair-{taskId}` | 2 hours | 8 min | 2 hours, 1 attempt | | Coupling review | `review-{taskId}` | 15 min | 5 min | 7 hours, 4 attempts | | Coupling repair | `repair-{taskId}` | 2 hours | 8 min | 7 hours, 4 attempts | | `self-bench-reaudit` | task id | 15 min | unset | n/a | | `self-bench-repair` | task id | 2 hours | unset | n/a | | Companion Harbor runs | `harbor run` | 4 hours | — | n/a | Modal also sets `idleTimeoutMs` to `min(timeoutMs, 10 minutes)` on every sandbox create. Review's 15-minute hard timeout is therefore paired with a 10-minute Modal idle cap. Discovery uses eight independently retryable shards. Heartbeat timeout for proxied activities is 10 minutes. Harbor infrastructure failures (`HarborInfrastructureFailure`, including non-zero `harbor` exits and Harbor infrastructure errors in the job result) mark the candidate `infrastructure_failed` after activity retries rather than failing the whole workflow. ## Harbor environment pairing Worker validation: ```bash harbor run \ --path TASK_DIR \ --agent nop|oracle \ --env "$SELFBENCH_HARBOR_ENVIRONMENT" \ --job-name ... \ --jobs-dir ... \ --delete --yes --quiet ``` ```bash title="Paired local stack" self-bench up --backend modal # execution=modal, harbor=modal self-bench up --backend docker # execution=docker, harbor=docker ``` ```bash title="Independent worker env" SELFBENCH_EXECUTION_BACKEND=modal SELFBENCH_HARBOR_ENVIRONMENT=modal # or mix, e.g. modal sandboxes + docker Harbor (requires Docker on the worker) ``` ```bash title="Companion Harbor (defaults to modal)" self-bench-validate --tasks ./tasks --jobs ./jobs --environment docker self-bench-eval --export ./export.tar.gz --jobs ./jobs --environment modal self-bench-agent-smoke --task ./task --jobs ./jobs --environment modal ``` Companion defaults: | Command | `--environment` default | `--concurrency` default | | --- | --- | --- | | `self-bench-validate` | `modal` | `10` | | `self-bench-eval` | `modal` | `3` | | `self-bench-agent-smoke` | `modal` | `4` | `self-bench-eval` and `self-bench-validate` pass `--n-concurrent 1 --max-retries 1` into each Harbor job. The worker gate does not pass those flags. Each submitted run stores: ```json { "version": { "executionBackend": "docker", "sandboxImage": "selfbench-sandbox:local", "schema": 1 } } ``` Modal runs record `sandboxImage` as `SELFBENCH_MODAL_IMAGE` (the registry base), not the layered Modal image id. ## Configuration Worker sandbox backend. Default `docker`. Harbor `--env`. Default: same as `SELFBENCH_EXECUTION_BACKEND`. Overrides the Docker `1` / Modal `20` default. Docker sandbox image. Default `selfbench-sandbox:local`. Compose hardcodes that tag. Modal registry base. Default `node:22-bookworm`. Not set in `compose.yaml`; the default applies unless the worker env adds it. Modal app name. Default `selfbench`. Optional Modal environment passed to `apps.fromName` and `secrets.fromObject`. Compose bind-mount source for `/root/.modal.toml`. `up --backend modal` sets this to `~/.modal.toml` or `--modal-config`. Otherwise Compose mounts `/dev/null`. Optional token pair. Blank values are stripped in `worker-main` before `loadConfig`. ```bash # Cloud worker (same image digest as the API) export SELFBENCH_EXECUTION_BACKEND=modal export SELFBENCH_HARBOR_ENVIRONMENT=modal export SELFBENCH_MODAL_APP=selfbench export MODAL_TOKEN_ID=... export MODAL_TOKEN_SECRET=... node dist/temporal/worker-main.js ``` ```text SelfBench worker polling default/selfbench-dev with activity concurrency 20 ``` ## Cleanup and leftover Docker resources On success or failure, `DockerSandboxExecutor` force-removes the container and its named volume, then deletes the host temp root (`selfbench-docker-*`). A host or daemon crash can leave `selfbench-*` containers and volumes. ```bash docker ps -a --filter name=selfbench- docker volume ls --filter name=selfbench- docker rm --force "$(docker ps -aq --filter name=selfbench-)" docker volume rm --force $(docker volume ls -q --filter name=selfbench-) ``` Modal sandboxes are terminated in a `finally` block. `self-bench down` stops Compose services and keeps the `selfbench_artifacts` and `selfbench_temporal-postgres` named volumes. Sandboxes receive a source checkout plus a short-lived model credential. Use SelfBench only with repositories you trust to execute. ## Errors | Signal | Meaning | | --- | --- | | `--backend must be "docker" or "modal"` | `self-bench up` flag is not one of those two values | | `--modal-config requires --backend modal` | profile override used with Docker | | `--environment must be docker or modal` | companion Harbor flag invalid | | `sandbox path must be beneath /work` | Docker file or output path rejected | | `sandbox {id} exited successfully without output {path}` | Docker exit 0, missing `outputPaths` entry | | `InactivityTimeoutError` | no process output for 8 min (agents) or 5 min (review) | | `Harbor … infrastructure failure` | Harbor CLI or job-level infrastructure error | | Worker fails `docker info` | execution or Harbor is `docker` but the daemon is unreachable | ## Next `self-bench up` / `down`, Compose services, `--modal-config`, and named volumes. Full `SELFBENCH_*` table, loopback token rule, and Temporal fields. `OPENAI_API_KEY` precedence, subscription JSON fallbacks, and Modal profile versus tokens. `self-bench-eval` / Harbor `--env` after an export. Modal worker, matching image digest, and task queue. Harbor infrastructure failures and leftover Docker resources. --- ## 23. Authentication > OPENAI_API_KEY precedence, Pi and Codex subscription JSON fallbacks, GH_TOKEN, API Bearer token, and Modal profile versus token secrets. - Page Markdown: https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc/pages/23-authentication.md - Generated: 2026-08-15T02:13:40.625Z ### Source Files - `src/subscription-auth.ts` - `src/codex-review.ts` - `src/evaluate.ts` - `src/config.ts` - `docs/operations.md` - `tests/subscription-auth.test.ts` --- title: "Authentication" description: "OPENAI_API_KEY precedence, Pi and Codex subscription JSON fallbacks, GH_TOKEN, API Bearer token, and Modal profile versus token secrets." --- Self-bench never hosts a model provider account. Operators supply credentials; the HTTP API holds only `SELFBENCH_API_TOKEN`, and the Temporal worker injects a **single selected model credential** (plus `GH_TOKEN` on clone stages) into each sandbox. `OPENAI_API_KEY` wins whenever it is non-empty after trim. ChatGPT subscription JSON is a compatibility fallback for existing Pi and Codex logins. Compose binds the API to `0.0.0.0` inside the container and defaults `SELFBENCH_API_TOKEN` to `selfbench-local`. Export the same token on the host before `self-bench run`, or every `/v1/*` call returns `401`. ## Credential map | Secret | Who reads it | Required when | Fallback | | --- | --- | --- | --- | | `OPENAI_API_KEY` | Worker, companion sandboxes, `self-bench-eval` | Recommended for discovery, authoring, review, repair, and Harbor Codex trials | Trimmed; whitespace-only is treated as unset | | `SELFBENCH_PI_AUTH_JSON` | Worker via `loadPiModelAuth()` | `OPENAI_API_KEY` unset and a Pi stage runs | `~/.pi/agent/auth.json` | | `SELFBENCH_CODEX_AUTH_JSON` | Worker via `loadCodexModelAuth()` | `OPENAI_API_KEY` unset and coupling repair runs | `CODEX_AUTH_JSON_PATH` or `~/.codex/auth.json` | | `GH_TOKEN` | CLI `gh` process; worker `githubToken()` | Private clones and in-sandbox `gh`; merged-PR collection on the host | `gh auth token` on the worker; host `gh auth login` for `self-bench run` | | `SELFBENCH_API_TOKEN` | API and CLI | API host is not loopback; always set in Compose | Unset is allowed only on `127.0.0.1`, `::1`, or `localhost` | | `MODAL_TOKEN_ID` / `MODAL_TOKEN_SECRET` | Worker Modal SDK | Cloud workers without a mounted profile | `~/.modal.toml` mounted at `/root/.modal.toml` | | `SELFBENCH_TEMPORAL_API_KEY` | API and worker Temporal clients | Temporal Cloud | Local Temporal has no API key | ## Ownership ```mermaid flowchart TB subgraph host [Operator host] CLI["self-bench CLI"] GH["gh auth login / GH_TOKEN"] Key["OPENAI_API_KEY"] PiFile["~/.pi/agent/auth.json"] CodexFile["~/.codex/auth.json"] ModalFile["~/.modal.toml"] ApiTok["SELFBENCH_API_TOKEN"] end subgraph compose [Compose / cloud processes] API["API: node dist/api-main.js"] Worker["Worker: node dist/temporal/worker-main.js"] end subgraph sandboxes [Per-stage sandbox secrets] PiStages["OPENAI_API_KEY or SELFBENCH_PI_AUTH_JSON"] CodexStage["OPENAI_API_KEY or SELFBENCH_CODEX_AUTH_JSON"] GitSecret["GH_TOKEN on discover and author only"] end CLI -->|"Bearer"| API GH --> CLI ApiTok --> API ApiTok --> CLI Key --> Worker PiFile --> Worker CodexFile --> Worker ModalFile --> Worker GH --> Worker Worker --> PiStages Worker --> CodexStage Worker --> GitSecret ``` Compose gives the API the shared Temporal and artifact settings only. `GH_TOKEN`, `OPENAI_API_KEY`, Modal tokens, and the two subscription JSON variables are attached to the **worker** service. The API never receives the Docker socket or model credentials. ## Model authentication `openAiApiKey()` reads `process.env.OPENAI_API_KEY`, trims it, and returns `undefined` when the result is empty. Both `loadPiModelAuth()` and `loadCodexModelAuth()` take that key first. ```mermaid flowchart LR env["OPENAI_API_KEY trimmed"] env -->|set| api["provider openai + apiKey"] env -->|unset| pi["Pi: openai-codex + stripped authJson"] env -->|unset| cd["Codex: stripped ChatGPT tokens"] ``` Subscription JSON is a compatibility path. Operations docs recommend `OPENAI_API_KEY` for discovery, authoring, coupling review, constrained repair, and model evaluation. Trimmed OpenAI API key. When set, Pi sandboxes use `--provider openai` and Codex review posts to `https://api.openai.com/v1/responses` with `Authorization: Bearer `. Inline Pi `auth.json`. Used only when `OPENAI_API_KEY` is unset. Inline Codex `auth.json`. Used only when `OPENAI_API_KEY` is unset. Filesystem path for Codex subscription JSON. Default `~/.codex/auth.json`. Also the default `--auth` path for `self-bench-eval`. ### Pi subscription JSON `loadPiSubscriptionAuth()` reads `SELFBENCH_PI_AUTH_JSON` or `~/.pi/agent/auth.json`. The file may contain other providers; only the `openai-codex` object is forwarded. Required shape: ```json { "openai-codex": { "type": "oauth", "access": "", "refresh": "" } } ``` Anything else throws `Pi auth does not contain an openai-codex subscription credential`. Tests assert that sibling keys such as `anthropic` never leave the host. In discovery and authoring sandboxes, the bootstrap script writes that JSON to `$HOME/.pi/agent/auth.json` (`0600`) and selects `--provider openai-codex` when `OPENAI_API_KEY` is absent. The file is deleted on `EXIT`. Coupling review (`sandbox-review`) also uses Pi auth. With only `authJson`, `reviewCouplingWithCodex()` posts to `https://chatgpt.com/backend-api/codex/responses`, sets `ChatGPT-Account-Id` from the JWT claim `https://api.openai.com/auth.chatgpt_account_id`, and sends `Originator: selfbench`. Missing `access` throws `Pi auth does not contain an OpenAI Codex subscription access token`. Missing account id throws `OpenAI Codex subscription token has no ChatGPT account ID`. Neither key nor `authJson` throws `OpenAI model authentication is required`. If both `apiKey` and `authJson` reach the reviewer, the API key wins and the public Responses endpoint is used. ### Codex subscription JSON `loadCodexSubscriptionAuth()` reads `SELFBENCH_CODEX_AUTH_JSON`, else `CODEX_AUTH_JSON_PATH`, else `~/.codex/auth.json`. `assertCodexSubscriptionAuth()` requires `auth_mode === "chatgpt"` and a `tokens` object. `auth_mode: "apikey"` is rejected even when `tokens` is present. Forwarded payload: ```json { "auth_mode": "chatgpt", "tokens": { }, "last_refresh": "" } ``` Coupling **repair** (`sandbox-repair`, `self-bench-repair`) is the Codex-auth consumer. It writes `SELFBENCH_CODEX_AUTH_JSON` to `~/.codex/auth.json` (`0600`), or `{ "OPENAI_API_KEY": "" }` when the API key is set. Missing both throws `OPENAI_API_KEY or SELFBENCH_CODEX_AUTH_JSON is required`. ## Stage-to-secret matrix | Stage | Loader | Secret injected | | --- | --- | --- | | Discovery shard | `loadPiModelAuth()` + `githubToken()` | `OPENAI_API_KEY` or `SELFBENCH_PI_AUTH_JSON`, plus `GH_TOKEN` when resolved | | Author candidate | same | same | | Validation repair | `loadPiModelAuth()` | `OPENAI_API_KEY` or `SELFBENCH_PI_AUTH_JSON` | | Coupling review / `self-bench-reaudit` | `loadPiModelAuth()` | `OPENAI_API_KEY` or `SELFBENCH_PI_AUTH_JSON` | | Coupling repair / `self-bench-repair` | `loadCodexModelAuth()` | `OPENAI_API_KEY` or `SELFBENCH_CODEX_AUTH_JSON` | | Harbor nop / oracle | none | No model secret; Harbor environment only | | `self-bench-eval` matrix | `openAiApiKey()` then Codex file | `OPENAI_API_KEY`, or `CODEX_FORCE_AUTH_JSON=1` + `CODEX_AUTH_JSON_PATH` | Docker sandboxes merge `secrets` into container env. Modal sandboxes wrap the same map in `client.secrets.fromObject(...)` and do not put those values in the non-secret `env` object. ## GitHub Two different surfaces use GitHub credentials. ### Host CLI: merged pull requests `self-bench run` calls `gh pr list --repo --state merged --limit 500` on the **operator host**. Self-bench does not pass a token into that process. Authenticate with `gh auth login`, or export `GH_TOKEN` so the `gh` CLI picks it up. Write access is not required. Empty local-session plus GitHub corpora fail with `no sanitized local-session or GitHub pull-request provenance was found`. ### Worker sandboxes: clone and in-sandbox `gh` `githubToken()` returns `process.env.GH_TOKEN` if that variable is set (not trimmed), otherwise `gh auth token` with `allowFailure: true`. A non-zero `gh` exit or empty stdout yields `undefined`; discovery and authoring then omit `GH_TOKEN`. When present, bootstrap writes `https://x-access-token:${GH_TOKEN}@github.com` to `$HOME/.git-credentials` (`0600`) and enables `credential.helper store` before `git clone` of `SOURCE_REPO_URL`. The credentials file is removed on `EXIT`. Public HTTPS clones work without a token. Compose interpolates `GH_TOKEN: ${GH_TOKEN:-}`. An empty host variable leaves the worker without a usable GitHub token. Export `GH_TOKEN="$(gh auth token)"` before `self-bench up`. Provenance sanitization redacts `Authorization: Bearer …`, `sk-…`, and `ghp_` / `github_pat_` forms. That is not a general secret scanner. ## API Bearer token `loadConfig()` requires `SELFBENCH_API_TOKEN` (min length 1) when `SELFBENCH_API_HOST` is not `127.0.0.1`, `::1`, or `localhost`. Compose sets `SELFBENCH_API_HOST=0.0.0.0`, so the token is mandatory for the local stack. Shared secret. The CLI sends `Authorization: Bearer ${SELFBENCH_API_TOKEN}` only when the variable is set. The API compares with `crypto.timingSafeEqual` after stripping a leading `Bearer ` (case-insensitive). Unequal lengths fail closed. CLI base URL. Default `http://127.0.0.1:8080`. Unauthenticated routes (even when a token is configured): | Method | Path | Purpose | | --- | --- | --- | | `GET` | `/healthz` | `{ "ok": true }` | | `GET` | `/` | Review UI `index.html` | | `GET` | `/assets/*` | Review UI static assets | Every other route, including `/v1/provenance`, `/v1/runs`, status, cancel, and export, returns `401` `{ "error": "unauthorized" }` when a token is configured and the header is missing or wrong. ```bash export SELFBENCH_API_TOKEN="$(openssl rand -hex 24)" export SELFBENCH_API_URL=http://127.0.0.1:8080 curl -sS -H "Authorization: Bearer $SELFBENCH_API_TOKEN" "$SELFBENCH_API_URL/v1/runs" ``` ```json { "error": "unauthorized" } ``` README generates a 24-byte hex token. Compose default is `selfbench-local` when the host variable is unset — the CLI will not send a header unless you export the same value. ## Modal profile versus token secrets `self-bench up --backend modal` sets `SELFBENCH_MODAL_CONFIG_PATH` to `~/.modal.toml` unless `--modal-config PATH` overrides it. `--modal-config` with `--backend docker` fails: `--modal-config requires --backend modal`. Compose mounts that path at `/root/.modal.toml:ro`. Docker-backend `up` leaves `SELFBENCH_MODAL_CONFIG_PATH` unset, so the mount source is `/dev/null`. Worker startup calls `removeEmptyModalCredentialOverrides()` **before** `loadConfig()`. Empty or whitespace `MODAL_TOKEN_ID` and `MODAL_TOKEN_SECRET` are deleted so they cannot mask a valid mounted profile. Non-empty values are kept for secret-manager / Temporal Cloud workers. ```bash title="Local Modal profile" modal token new self-bench up --backend modal # or: self-bench up --backend modal --modal-config /absolute/path/to/.modal.toml ``` ```text title="Cloud worker tokens" SELFBENCH_EXECUTION_BACKEND=modal SELFBENCH_HARBOR_ENVIRONMENT=modal MODAL_TOKEN_ID=... MODAL_TOKEN_SECRET=... ``` `ModalSandboxExecutor` constructs `new ModalClient()` with no explicit credentials; the Modal SDK uses the profile or the token pair from the process environment. ## Harbor evaluation `self-bench-eval` prefers `OPENAI_API_KEY`. If it is unset, it validates `--auth FILE` (default `~/.codex/auth.json`) with `assertCodexSubscriptionAuth()` before launching Harbor. Subscription trials delete `OPENAI_API_KEY` from the child env and set: - `CODEX_FORCE_AUTH_JSON=1` - `CODEX_AUTH_JSON_PATH=` `summary.json` records `"auth": "openai-api-key"` or `"auth": "codex-subscription"`. Compiled Harbor tasks allowlist `chatgpt.com`, `*.chatgpt.com`, `openai.com`, and `*.openai.com` on the agent network. ```bash export OPENAI_API_KEY=... self-bench-eval --export ./self-bench-evals.tar.gz --jobs ./matrix-jobs --environment modal # compatibility: # unset OPENAI_API_KEY # self-bench-eval --export ./self-bench-evals.tar.gz --jobs ./matrix-jobs --auth ~/.codex/auth.json ``` Harbor nop/oracle validation does not inject model credentials. `self-bench-agent-smoke` does not implement a separate auth loader. ## Local setup ```bash gh auth login modal token new ``` Read access to merged pull requests is enough. ```bash export OPENAI_API_KEY=... export GH_TOKEN="$(gh auth token)" export SELFBENCH_API_TOKEN="$(openssl rand -hex 24)" ``` Keep `SELFBENCH_API_TOKEN` in the same shell that later runs `self-bench run`. ```bash self-bench up --backend modal export SELFBENCH_API_URL=http://127.0.0.1:8080 curl -sS "$SELFBENCH_API_URL/healthz" curl -sS -H "Authorization: Bearer $SELFBENCH_API_TOKEN" "$SELFBENCH_API_URL/v1/runs" ``` Expect `{ "ok": true }` then a JSON array (possibly empty). A missing Bearer against Compose yields `unauthorized`. ## Cloud worker secrets A Temporal Cloud worker from the same image digest needs the model, GitHub, and Modal tokens. The API process on Cloud Run (or equivalent) needs `SELFBENCH_API_TOKEN`, GCS, and Temporal TLS + `SELFBENCH_TEMPORAL_API_KEY` — not `OPENAI_API_KEY` or `GH_TOKEN`. ```text # worker SELFBENCH_EXECUTION_BACKEND=modal SELFBENCH_HARBOR_ENVIRONMENT=modal MODAL_TOKEN_ID=... MODAL_TOKEN_SECRET=... GH_TOKEN=... OPENAI_API_KEY=... # optional compatibility: # SELFBENCH_PI_AUTH_JSON=... # SELFBENCH_CODEX_AUTH_JSON=... ``` Use separate least-privilege identities. Sandboxes execute a trusted repository checkout plus a short-lived model credential; this is not a malware-analysis boundary. ## Failures | Symptom | Cause | | --- | --- | | `SELFBENCH_API_TOKEN is required when the API listens beyond loopback` | `SELFBENCH_API_HOST` is not `127.0.0.1` / `::1` / `localhost` and the token is missing | | `401` `unauthorized` | CLI token unset or mismatched with Compose (`selfbench-local` vs generated hex) | | `Pi auth does not contain an openai-codex subscription credential` | Fallback file missing `openai-codex` oauth `access` + `refresh` | | `… does not contain a ChatGPT subscription token set` | Codex file is `auth_mode: "apikey"` or lacks `tokens` | | `OpenAI model authentication is required` | Review sandbox received neither key nor Pi `authJson` | | `OPENAI_API_KEY or SELFBENCH_PI_AUTH_JSON is required` | Review / validation-repair entrypoint | | `OPENAI_API_KEY or SELFBENCH_CODEX_AUTH_JSON is required` | Coupling-repair entrypoint | | Private `git clone` fails in discover/author | Worker `GH_TOKEN` empty; Compose passed through an unset host variable | | Modal SDK ignores mounted `~/.modal.toml` | Empty `MODAL_TOKEN_*` leaked into the worker (fixed by startup strip) | | Harbor Codex trial cannot authenticate | Matrix has no `OPENAI_API_KEY` and no valid ChatGPT `auth.json` | ## Next Global install, Modal and GitHub login, required exports, first `self-bench` help check. `SELFBENCH_*` keys, loopback token rule, artifact backends, Temporal fields. Bearer-protected routes, status codes, provenance limits, export integrity headers. `self-bench up` backends, `--modal-config`, named volumes. `self-bench-eval` API-key path and `--auth` ChatGPT fallback. Split API/worker secrets, Modal tokens, Temporal TLS and API key. --- ## 24. Troubleshooting > No-provenance, blocked candidate pool, 409 export-not-ready, SHA-256 mismatch, Harbor infrastructure failures, and leftover Docker resources. - Page Markdown: https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc/pages/24-troubleshooting.md - Generated: 2026-08-15T02:16:57.415Z ### Source Files - `src/harbor-results.ts` - `src/run-wait.ts` - `src/cli.ts` - `src/api.ts` - `src/temporal/workflow.ts` - `tests/harbor-results.test.ts` - `docs/operations.md` --- title: "Troubleshooting" description: "No-provenance, blocked candidate pool, 409 export-not-ready, SHA-256 mismatch, Harbor infrastructure failures, and leftover Docker resources." --- `self-bench run`, `self-bench download`, `GET /v1/runs/:runId/export`, Harbor nop/oracle gates, and the Docker sandbox executor fail with a small set of stable strings and statuses. Treat `blocked`, `failed`, and `cancelled` as terminal unsuccessful run phases. `complete` is the only phase that attaches an `export` artifact. Candidate-level `infrastructure_failed` does not fail the workflow and is not replaced. `--output` implies `--wait` (2s poll). Closing the waiting CLI does not cancel Temporal. A stopped worker or Compose stack pauses work until the worker polls again. ## Symptom index | You see | Surface | Meaning | | --- | --- | --- | | `no sanitized local-session or GitHub pull-request provenance was found` | `self-bench run` (before `POST /v1/runs`) | Local session scan plus `gh pr list` both produced zero retained messages | | `SelfBench run blocked` | `--wait` / `--output` | Discovery could not fill the authoring budgets | | `candidate pool exhausted; missing easy=…, medium=…, hard=…` | Temporal `ApplicationFailure` type `CandidatePoolExhausted` | Same condition; counts are missing *authoring* slots, not accepted-task shortfalls | | `{ "error": "run export is not ready" }` HTTP 409 | `GET /v1/runs/:runId/export`, `self-bench download` | Status has no `export` field yet (or the run never reached `complete`) | | `downloaded export failed its SHA-256 integrity check` | `self-bench download` / `--output` | Streamed bytes ≠ `x-content-sha256`; dest file is deleted | | `SelfBench API returned an export without integrity metadata` | download | Missing `x-content-sha256` or empty body | | `Harbor … infrastructure failure` / `HarborInfrastructureFailure` | worker validation, `self-bench-validate` | Environment/auth/runtime, not a task assertion failure | | `infrastructure_failed` on a task | `GET /v1/runs/:runId` | Harbor infra exhausted activity retries; candidate is not replaced; export still runs | | leftover `selfbench-*` containers or volumes | host Docker | Crash or kill before `docker rm` / `volume rm` | | `all predefined address pools have been fully subnetted` | Harbor trial `exception_info` | Docker network pool exhaustion (leftover networks) | ## Inspect the run first ```bash export SELFBENCH_API_URL="${SELFBENCH_API_URL:-http://127.0.0.1:8080}" self-bench list self-bench status RUN_ID ``` `queued` · `discovering` · `authoring` · `exporting` · `complete` · `blocked` · `failed` · `cancelled` Authoring budgets submitted as `easy` / `medium` / `hard`. Rejected candidates are not refilled. `wave`, `totalShards` (8), `completedShards`, `failedShards`, `candidates`. `authoring` · `auditing` · `validating` · `reviewing` · `repairing` · `infrastructure_failed` · `rejected` · `accepted` Present only after `buildExport`. Fields: `uri`, `sha256` (64 hex), `sizeBytes`, `contentType`. `self-bench list` reports Temporal execution names (`RUNNING`, `COMPLETED`, `FAILED`, `CANCELED`), not the SelfBench `phase` field. Use `status` for phase, discovery, and per-candidate reasons. ```mermaid stateDiagram-v2 [*] --> discovering discovering --> blocked: no unique PRs or 300-candidate cap discovering --> authoring: budgets filled authoring --> exporting: all selected candidates finished exporting --> complete: export artifact written authoring --> failed: non-Harbor activity error discovering --> cancelled: Temporal cancel authoring --> cancelled: Temporal cancel blocked --> [*]: GET export 409 failed --> [*]: GET export 409 cancelled --> [*]: GET export 409 complete --> [*]: GET export 200 + x-content-sha256 ``` A candidate marked `infrastructure_failed` stays in `authoring`→`exporting`→`complete`. That is not `blocked`. ## No provenance `self-bench run` collects two corpora on the operator machine, concatenates them, and refuses to start if the merge is empty: ```text no sanitized local-session or GitHub pull-request provenance was found ``` This throw happens **before** `POST /v1/provenance` and `POST /v1/runs`. A `gh` failure is a different error (`gh exited N: …`). ### What is collected | Source | Location / command | Kept when | | --- | --- | --- | | Pi | `$HOME/.pi/agent/sessions/**/*.jsonl` | File content or path contains a worktree path (or slash→dash encoding) | | Claude Code | `$HOME/.claude/projects/**/*.jsonl` | Same path association | | Codex | `$HOME/.codex/sessions` and `$HOME/.codex/archived_sessions` `**/*.jsonl` | Same path association | | GitHub | `gh pr list --state merged --limit 500` | Non-draft, non-bot author, non-empty title, ≥20 changed lines, ≥1 changed file | Only user-role session text is kept. Messages that redact to empty or start with injected prefixes (`# AGENTS.md instructions`, ``, ` `--repo` must be a Git checkout whose `origin` is `https://github.com/…` or `git@github.com:…`. Other remotes throw `unsupported origin URL`. Run `git -C REPO worktree list --porcelain` and confirm session files mention that absolute path (Claude project dirs often use the slash-encoded form). ```bash gh pr list --repo OWNER/NAME --state merged --limit 20 \ --json number,title,author,isDraft,additions,deletions,changedFiles ``` PRs under 20 changed lines, drafts, and bots never enter the corpus. Repositories with only bot or tiny PRs and no local sessions fail this gate. Collection uses `process.env.HOME`. A worker container, CI user, or `sudo` with a different home will scan empty roots even if sessions exist on the laptop that authored the PRs. An empty accepted export is not this error. Discovery can fill budgets from GitHub titles alone; later audit/validation/review can reject every candidate and still reach `complete` with `acceptedCount: 0`. ## Blocked candidate pool Discovery loops until every tier budget is filled, a wave adds **zero new `sourcePr` values**, or the discovered set hits **300**. Those last two terminals set `phase` to `blocked` and throw a non-retryable `ApplicationFailure`: ```text candidate pool exhausted; missing easy=N, medium=N, hard=N ``` `--wait` / `--output` then throw `SelfBench run blocked`. The status JSON often has **no** `error` field on this path (the workflow does not copy the ApplicationFailure message onto `status` when the phase is already `blocked`). Read `requestedByDifficulty`, `discovered`, and `discovery.failedShards` instead. ### Why discovery stops short - Eight shards per wave. Each shard is asked for `min(8, ceil(missing / 8) + 3)` candidates per non-zero tier. - Retryable shard failures increment `failedShards` and contribute `[]`. Non-retryable shard failures (`InvalidDiscoveryShard`, cancel) fail the run instead of blocking. - Unique-PR filter: a wave that only rediscovers already seen PRs is treated as empty → `blocked`. - Hard/medium/easy slots are independent. A repo that only has easy-sized PRs blocks if you requested `medium` or `hard`. - GitHub coarse filter is 20 lines / 1 file. Medium and hard still require larger implementation diffs later; discovery may simply not find enough merged PRs. Tier flags are **authoring budgets**, not accepted-task targets. If discovery fills the budget and later stages reject every candidate, the run goes to `complete` and still builds an export. Lower `--easy-count` / `--medium-count` / `--hard-count` when the repo cannot supply that many distinct merged PRs. Fix: add real merged human PRs (or local sessions that map to them), lower the budgets, check `GH_TOKEN` / `gh` inside the **worker** (clone/fetch, not the CLI scan), and confirm `discovery.failedShards` is not 8/8. Failed generation runs need a **new** `--run-id`; Temporal `workflowId` is the run ID. ## 409 export is not ready :::endpoint GET /v1/runs/:runId/export Download the completed archive Requires `Authorization: Bearer $SELFBENCH_API_TOKEN` when the token is configured. **200** streams `status.export` with: | Header | Value | | --- | --- | | `content-type` | `status.export.contentType` (gzip) | | `content-length` | `status.export.sizeBytes` | | `content-disposition` | `attachment; filename="selfbench-RUN_ID.tar.gz"` | | `x-content-sha256` | `status.export.sha256` | **409** ```json { "error": "run export is not ready" } ``` when the queried status has no `export` object. `self-bench download` rethrows that string. ::: `export` is written only after `buildExport` and `phase: "complete"`. `queued`, `discovering`, `authoring`, `exporting`, `blocked`, `failed`, and `cancelled` all 409. ```bash self-bench status RUN_ID self-bench download RUN_ID ./self-bench-evals.tar.gz ``` ```json { "error": "run export is not ready" } ``` Wait until `phase` is `complete` and `export.sha256` is a 64-hex digest. `--output` already waits; a bare `download` does not. If `phase` is `complete` but `export` is missing, the status query fell back to Temporal describe after a failed `status` query. Retry `status`. There is no delete route; a failed run cannot be overwritten in place. ## SHA-256 mismatch Download is create-only (`open(dest, "wx")`) and hashes the stream against `x-content-sha256`. | Failure | CLI / store message | Dest file | | --- | --- | --- | | Header missing or no body | `SelfBench API returned an export without integrity metadata` | removed if created | | Stream digest ≠ header | `downloaded export failed its SHA-256 integrity check` | removed | | Dest already exists | Node `EEXIST` from `wx` | left untouched | | Artifact store size or digest mismatch | `artifact integrity check failed: ` | n/a (API 500 on export) | | Idempotent put collision | `artifact already exists with different contents: …` | n/a | The API also verifies `sizeBytes` and `sha256` when opening the artifact (local file or GCS object metadata + streamed digest). A 200 with a bad body is a transport/corruption problem; a 500 with `artifact integrity check failed` is a store problem. `manifest.json` inside the archive also lists per-task SHA-256 values. That inner digest is not a signature: it lives in the same tarball. ```bash # After a successful download python3 - <<'PY' import hashlib, sys p = sys.argv[1] print(hashlib.sha256(open(p,"rb").read()).hexdigest()) PY ./self-bench-evals.tar.gz # must match the x-content-sha256 you saw / status.export.sha256 ``` Retry the download to a **new** path. Do not reuse a partial file. ## Harbor infrastructure failures The worker treats a Harbor trial as infrastructure when `exception_info.exception_type` is `AuthError`, **or** the message matches: | Pattern | Typical cause | | --- | --- | | `unknown flag: --project-name` | Harbor / Compose CLI mismatch | | `cannot connect to the docker daemon` | Socket not mounted or daemon down | | `error during connect` / `connection refused` | Docker or Modal endpoint unreachable | | `modal` + `unavailable` / `timed out` / `timeout` | Modal outage or auth/profile miss | | `image build for im-[a-z0-9]+ failed` | Modal image build | | `all predefined address pools have been fully subnetted` | Docker network pool exhausted | A task Dockerfile / `uv sync` failure is **not** infrastructure. That is a candidate reject (or a `self-bench-validate` `accepted: false`). ### Worker behavior `harbor run` non-zero exit **or** a classified trial exception becomes `ApplicationFailure` type `HarborInfrastructureFailure`. Validate/review/author activities retry up to **4** times (5s → 2 min backoff). After retries: - the candidate is recorded `infrastructure_failed` with the inner message - it is **not** replaced - siblings continue - `buildExport` still runs → `complete` with whatever was accepted `self-bench-validate` does **not** retry that way. It records `status: "error"` and: ```text Harbor nop infrastructure failure: … Harbor oracle infrastructure failure: … ``` or `Harbor {nop|oracle} exited N: …` (last 1000 chars of stderr). Worker startup runs `docker info` and `docker compose version` when `SELFBENCH_EXECUTION_BACKEND` or `SELFBENCH_HARBOR_ENVIRONMENT` is `docker`. The Compose worker mounts `/var/run/docker.sock` and joins `DOCKER_GID`. ```bash docker info docker compose version self-bench up --backend docker ``` Address-pool errors: remove leftover Harbor/Compose networks (see leftover resources), then rerun the gate. `AuthError` / `Token missing` is always infrastructure. Empty `MODAL_TOKEN_ID` / `MODAL_TOKEN_SECRET` are deleted at worker start so they cannot override a mounted `~/.modal.toml`. Prefer `self-bench up --backend modal` (optional `--modal-config`). Image-build `im-…` failures: check Modal build logs for that image id; they are retried then recorded `infrastructure_failed`. ### Incomplete Harbor jobs (`self-bench-eval`) A job whose `result.json` has `finished_at: null` is treated as incomplete. The matrix runner renames it to `JOB.incomplete-` and starts a new job. A finished job with the same name is reused and not rerun. ```text Harbor produced no result (exit N): … expected one Harbor trial result in …, found N ``` Multiple trial directories plus a single `trial_results[]` entry use the aggregate trial (stale local trial dirs). Zero trials and a finished job throw. ## Leftover Docker resources ```text Compose project "selfbench" Disposable sandboxes ├── api, worker, temporal, postgres ├── container selfbench-{runId20}-{stage16}-{8hex} ├── volume selfbench_artifacts └── volume same name └── volume selfbench_temporal-postgres ``` The Docker executor always `docker rm --force` and `docker volume rm --force` after a sandbox. A host crash, `kill -9`, or daemon restart can leave both. Harbor Compose networks from crashed trials can exhaust default address pools. `self-bench down` is `docker compose --file compose.yaml down` **without** `-v`. Named volumes keep Temporal history and generated artifacts. That is persistence, not a leak. ```bash docker ps -a --filter name=selfbench- docker volume ls --filter name=selfbench- ``` Keep `selfbench_artifacts` and `selfbench_temporal-postgres` unless you intend to wipe history. ```bash docker ps -aq --filter name='selfbench-sb-' | xargs -r docker rm -f docker volume ls -q --filter name='selfbench-sb-' | xargs -r docker volume rm ``` Sandbox names start with `selfbench-` plus the run-id prefix (`sb-…` for generated IDs). Do not `volume rm` the Compose volumes above. ```bash docker network ls docker network prune # only unused networks; confirm nothing else needs them ``` Then rerun `self-bench-validate` or the matrix job. Harbor eval/validate already pass `--delete`. ```bash self-bench up --backend docker # or modal ``` A worker that failed `docker info` at boot will not poll until it is recreated. `docker compose down -v` deletes `selfbench_artifacts` and `selfbench_temporal-postgres`. SelfBench has no remote delete API; that is the only local wipe. Exports already downloaded are unaffected. ## Adjacent failures | Message / signal | Check | | --- | --- | | HTTP 401 `{ "error": "unauthorized" }` | `SELFBENCH_API_TOKEN` on CLI must match the API. Compose defaults the API to `selfbench-local` if unset. | | `SELFBENCH_API_TOKEN is required when the API listens beyond loopback` | Non-`127.0.0.1` / `::1` / `localhost` bind without a token. | | `AuthError: Token missing…` | Harbor/Modal client auth; classified as infrastructure. | | `Pi auth does not contain an openai-codex subscription credential` | Fallback path when `OPENAI_API_KEY` is unset. | | `… does not contain a ChatGPT subscription token set` | Codex `auth.json` must be `auth_mode: chatgpt` with `tokens`. | | Message | Constraint | | --- | --- | | `the total candidate count must be between 1 and 100` | Sum of the three tier flags | | `--backend must be "docker" or "modal"` | `self-bench up` | | `--modal-config requires --backend modal` | flag pairing | | `unsupported origin URL` | origin is not GitHub HTTPS/SSH | | `SelfBench was built without commit metadata; set SELFBENCH_BUILD_COMMIT…` | packaged CLI without git metadata | | `request body exceeds 104857600 bytes` | `POST /v1/provenance` cap (100 MiB) | | `request body exceeds 10485760 bytes` | other POST bodies (10 MiB) | | Zod / `SyntaxError` | HTTP 400 | | Duplicate `--run-id` | Temporal `workflow.start` same `workflowId` → HTTP 500 | If the worker or Compose stack stops, activities stop heartbeating; the workflow stays `RUNNING` / `discovering` or `authoring` until the worker returns. Discovery shards: 45 min start-to-close, 8 min inactivity. Authoring: 7 h start-to-close, 8 min inactivity. Review inactivity: 5 min. `InactivityTimeoutError` looks like `{command} produced no output for {ms}ms`. When the status query fails, the API maps Temporal `FAILED` → `phase: "failed"` and may set `error` to `Temporal workflow failed`. `self-bench-eval` requires exactly one of `--export` / `--tasks`, at least one Harbor task inside, and `--model` in `gpt-5.6-sol`, `gpt-5.6-terra`, `gpt-5.6-luna`. `--environment` must be `docker` or `modal`. `found no Harbor tasks in …` means the extract step produced no `task.toml` directories. ## Next Session roots, GitHub PR filters, sanitization, and the empty-corpus failure. Phases, eight-shard discovery, budgets vs accepted tasks, blocked / failed / cancelled. `status`, `download`, create-only writes, and integrity headers. Extract an export, `self-bench-eval` resume, incomplete job archives. `self-bench up` / `down`, named volumes, worker stop behavior. API Bearer token, `GH_TOKEN`, Modal profile vs token secrets. --- ## 25. Contributing > Clone, bun install, bun run validate, source CLI, API/worker/review dev servers, CI, package verification, and npm tag publish. - Page Markdown: https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc/pages/25-contributing.md - Generated: 2026-08-15T02:16:51.923Z ### Source Files - `package.json` - `README.md` - `.github/workflows/ci.yml` - `.github/workflows/publish-npm.yml` - `scripts/verify-package.ts` - `biome.json` - `tsconfig.build.json` --- title: "Contributing" description: "Clone, bun install, bun run validate, source CLI, API/worker/review dev servers, CI, package verification, and npm tag publish." --- `self-bench` 0.3.3 is an MIT TypeScript package. Local work uses Bun against `bun.lock`; published bins are compiled Node entry points under `dist/`. `bun run validate` is the contributor, CI, and `prepublishOnly` gate: Biome plus two TypeScript projects, `bun test`, a full `dist/` build, then a pack-and-install check that the `self-bench` binary still prints `self-bench up`. CI and the Docker build image pin **Bun 1.3.14**. `package.json` `engines.node` is `>=22`. The publish workflow also installs **Node 24**. There is no separate `CONTRIBUTING.md`; the scripts and workflows below are the contract. ## Prerequisites | Tool | Where it is pinned | Needed for | | --- | --- | --- | | Bun 1.3.14 | `.github/workflows/ci.yml`, `Dockerfile` (`oven/bun:1.3.14-debian`) | install, tests, `bun run cli`, `bun pm pack` | | Node `>=22` | `package.json` `engines`, runtime image `node:22-bookworm` | `tsx` / `node dist/*` after compile | | Docker + Compose | `src/cli.ts` `up`/`down`, `src/temporal/worker-main.ts` | local Temporal, image builds, Docker sandbox worker | | Git | `scripts/write-build-metadata.ts`, CLI `rev-parse` | `SELFBENCH_BUILD_COMMIT` fallback | Model keys, `GH_TOKEN`, and Modal tokens are not required to lint, typecheck, test, or pack. They are required only if you start a worker that authors or validates tasks. See [Installation](/installation) and [Authentication](/authentication). ## Clone and install ```bash git clone https://github.com/mupt-ai/self-bench.git cd self-bench ``` ```bash bun install --frozen-lockfile ``` CI uses the same flag. A lockfile drift fails install instead of rewriting `bun.lock`. ```bash bun run validate ``` Success ends with `verify-package` printing `verified self-bench@` (currently `self-bench@0.3.3`) after a temporary `bun pm pack` install. ```bash bun run cli -- --help ``` The extra `--` is required so Bun does not consume `--help`. The process is `bun run src/cli.ts`. Expected usage includes `self-bench up`, `down`, `run`, `status`, `cancel`, `download`, and `list`. ## Script map | Script | Command | Role | | --- | --- | --- | | `validate` | `bun run check && bun run test && bun run build && bun run verify:package` | Single local/CI/publish gate | | `check` | `biome check . && tsc --noEmit -p tsconfig.json && bun run typecheck:review` | Lint/format + server/tests types + review types | | `test` | `bun test tests review/src` | `bun:test` under `tests/` and `review/src/` | | `format` | `biome format --write .` | Rewrite using Biome (spaces, width 100) | | `typecheck:review` | `tsc --noEmit -p review/tsconfig.json` | Review Vite/React project | | `cli` | `bun run src/cli.ts` | Source CLI (not the published `dist/cli.js`) | | `dev:api` | `tsx src/api-main.ts` | Host API against Temporal | | `dev:worker` | `tsx src/temporal/worker-main.ts` | Host Temporal worker | | `dev:review` | `vite --config review/vite.config.ts` | Review UI with `/v1` proxy | | `build` | `clean` + `build:server` + `build:review` | Populate `dist/` | | `start:api` | `node dist/api-main.js` | Compiled API (Compose/image default) | | `start:worker` | `node dist/temporal/worker-main.js` | Compiled worker | | `verify:package` | `bun scripts/verify-package.ts` | Pack, install, help + shipped-asset check | | `prepack` | `bun run build` | Runs before `bun pm pack` / `npm pack` | | `prepublishOnly` | `bun run validate` | Runs before `npm publish` | Biome (`biome.json`) lints and formats only `src/**/*.ts`, `tests/**/*.ts`, and `review/src/**/*.{ts,tsx}`. The server `tsconfig.json` typechecks `src/**/*.ts` and `tests/**/*.ts` with `strict`, `noUncheckedIndexedAccess`, and `exactOptionalPropertyTypes`. `tsconfig.build.json` emits `dist/` from `src/**/*.ts` and excludes `src/**/*.test.ts` and `src/review_dist`. ## Source CLI and companion bins Published `bin` entries all point at compiled files: | Bin | Source | Dist | | --- | --- | --- | | `self-bench` | `src/cli.ts` | `dist/cli.js` | | `self-bench-eval` | `src/eval-main.ts` | `dist/eval-main.js` | | `self-bench-validate` | `src/validate-main.ts` | `dist/validate-main.js` | | `self-bench-reaudit` | `src/reaudit-main.ts` | `dist/reaudit-main.js` | | `self-bench-repair` | `src/repair-main.ts` | `dist/repair-main.js` | | `self-bench-agent-smoke` | `src/agent-smoke-main.ts` | `dist/agent-smoke-main.js` | ```bash title="Source (checkout)" bun run cli -- --help bun src/eval-main.ts --help bun src/validate-main.ts --help bun src/reaudit-main.ts --help bun src/repair-main.ts --help bun src/agent-smoke-main.ts --help ``` ```bash title="Compiled (after bun run build)" node dist/cli.js --help node dist/eval-main.js --help ``` `src/cli.ts` resolves `compose.yaml` as `../compose.yaml` from the CLI file, so source `up`/`down` use the checkout root and a published install uses the package root. `up --backend docker` builds `Dockerfile.sandbox` as `selfbench-sandbox:local` before `docker compose --file compose.yaml up -d --build`. Help verification in `scripts/verify-package.ts` only asserts that installed `self-bench --help` contains `self-bench up`. ```bash bun run cli -- --help ``` ```text SelfBench creates durable tiered Harbor evaluations. Usage: self-bench up [--backend docker|modal] [--modal-config PATH] self-bench down self-bench run --repo PATH [--easy-count N] [--medium-count N] [--hard-count N] [--model MODEL] [--run-id ID] [--wait] [--output OUTPUT.tar.gz] self-bench status RUN_ID self-bench cancel RUN_ID self-bench download RUN_ID OUTPUT.tar.gz self-bench list ``` Default API target for the CLI is `SELFBENCH_API_URL` or `http://127.0.0.1:8080`. If `SELFBENCH_API_TOKEN` is set, the CLI sends `Authorization: Bearer …`. ## Host API, worker, and review `dev:*` runs processes on the host. Compose `api` / `worker` run `node dist/api-main.js` and `node dist/temporal/worker-main.js` from image `selfbench:local`. Do not run both pairs against the same port and task queue. ```mermaid flowchart LR subgraph host [Host checkout] CLI["bun run cli / src/cli.ts"] Review["dev:review Vite"] API["dev:api tsx src/api-main.ts"] Worker["dev:worker tsx src/temporal/worker-main.ts"] end subgraph compose [compose.yaml] PG["postgres:17.6-bookworm"] Temporal["temporalio/auto-setup:1.29.6 :7233"] CAPI["api node dist/api-main.js :8080"] CWorker["worker node dist/temporal/worker-main.js"] end Review -->|"proxy /v1"| API CLI -->|SELFBENCH_API_URL| API API --> Temporal Worker --> Temporal CAPI --> Temporal CWorker --> Temporal Temporal --> PG ``` `startApi` opens a Temporal client before `listen`. `dev:api` therefore fails if nothing is bound at `SELFBENCH_TEMPORAL_ADDRESS` (default `127.0.0.1:7233`). The worker polls `SELFBENCH_TEMPORAL_NAMESPACE` / `SELFBENCH_TASK_QUEUE` (defaults `default` / `selfbench-dev`). Compose does not override the queue name, so a host `dev:worker` and the Compose `worker` steal the same activities if both are up. Compose also publishes `127.0.0.1:8080`; a second API on the default port will not bind. ```bash docker compose up -d postgres temporal ``` Leave `api` and `worker` down so host processes own `:8080` and `selfbench-dev`. ```bash bun run dev:api ``` Logs `SelfBench API listening on http://127.0.0.1:8080` when `SELFBENCH_API_HOST` / `SELFBENCH_API_PORT` are unset. `GET /` and `GET /assets/*` are served from a `review/` directory next to the compiled API module (`dist/review` after `build:review`). `tsx src/api-main.ts` therefore looks at `src/review`, which is not the Vite app. Use `dev:review` for UI work, or `bun run start:api` after a build to serve `dist/review`. ```bash bun run dev:worker ``` On `SELFBENCH_EXECUTION_BACKEND=docker` or `SELFBENCH_HARBOR_ENVIRONMENT=docker`, startup runs `docker info` and `docker compose version` (30s timeout) before connecting. It logs `SelfBench worker polling / with activity concurrency `. Docker default concurrency is `1`; Modal default is `20`. The worker calls `removeEmptyModalCredentialOverrides()` first so empty `MODAL_TOKEN_*` values cannot mask a mounted profile. ```bash bun run dev:review ``` `review/vite.config.ts` does not set `server.port` (Vite default **5173**). It proxies `/v1` to `http://127.0.0.1:8080` and lists extra `server.allowedHosts` / `preview.allowedHosts`. The UI calls `/v1/runs` and `/v1/runs/:id/export` with an optional Bearer token field; it can also open a local export file without the API. Default `127.0.0.1`. Binding anything other than `127.0.0.1`, `::1`, or `localhost` requires `SELFBENCH_API_TOKEN`. Default `8080`. Compose publishes `127.0.0.1:8080:8080`. Optional on loopback. Compose defaults to `selfbench-local` when the host env is unset. `/healthz`, `/`, and `/assets/*` stay unauthenticated; `/v1/*` requires Bearer when the token is set. Default `127.0.0.1:7233` on the host. Compose services use `temporal:7233`. Default `selfbench-dev`. API workflow starts and the worker must match. Optional 40-char hex SHA. `scripts/write-build-metadata.ts` writes `dist/build-metadata.js`. If unset, the script uses `git rev-parse HEAD`, else forty zeros. The CLI refuses an all-zero built commit and re-reads `HEAD` from the package/checkout root. For the full Compose image path (`self-bench up` / `self-bench down`), use [Start the local stack](/start-local-stack). Host `start:api` / `start:worker` match the image `CMD` / worker `command`. ## Build outputs `bun run build` deletes `dist/`, then: 1. `tsc -p tsconfig.build.json` → `dist/**/*.js` plus declaration and source maps. 2. `bun build` of `src/sandbox-author.ts`, `src/sandbox-review.ts`, `src/sandbox-repair.ts`, `src/sandbox-validation-repair.ts` → `dist/sandbox-*.bundle.js` (`--target=node`). 3. `bun scripts/write-build-metadata.ts` → `dist/build-metadata.js`. 4. `vite build --config review/vite.config.ts` → `dist/review` (`emptyOutDir: true`). The production image (`Dockerfile`) runs `bun run build` in `oven/bun:1.3.14-debian`, then copies `dist/`, `node_modules`, `src/extensions`, `src/skills`, and `package.json` into `node:22-bookworm`. Default `CMD` is `node dist/api-main.js`. Tests, Markdown, and `.git` are excluded by `.dockerignore`. ## Package verification `bun run verify:package` (`scripts/verify-package.ts`): 1. `bun pm pack --destination ` (triggers `prepack` → `build`). 2. `bun init --yes` in a fresh directory and `bun add --no-save` the tarball. 3. Run `node_modules/.bin/self-bench --help` and require the substring `self-bench up`. 4. Read these installed paths (must exist): - `compose.yaml` - `Dockerfile` - `Dockerfile.sandbox` - `src/skills/selfbench/SKILL.md` 5. Assert installed `package.json` `name` and `version` match the workspace. `package.json` `files` is the publish allowlist: `dist`, `src`, `docs`, `compose.yaml`, `Dockerfile`, `Dockerfile.sandbox`, `bun.lock`, `scripts`, `biome.json`, `tsconfig.json`, `tsconfig.build.json`, `.dockerignore`. `publishConfig` is `access: public` on `https://registry.npmjs.org/`. `validate` already includes `verify:package`. Re-run `bun run verify:package` alone after changing `bin`, `files`, Compose, Dockerfiles, or the shipped skill. ## CI `.github/workflows/ci.yml`: | Field | Value | | --- | --- | | Name | `CI` | | Triggers | `push` to `main`, every `pull_request` | | Permissions | `contents: read` | | Concurrency | `ci-${{ github.workflow }}-${{ github.ref }}`, `cancel-in-progress: true` | | Runner | `ubuntu-latest` | | Steps | `actions/checkout@v4` → `oven-sh/setup-bun@v2` (`bun-version: 1.3.14`) → `bun install --frozen-lockfile` → `bun run validate` | A red CI check is a failed `validate`. There is no second job matrix and no release job in this workflow. ## npm tag publish `.github/workflows/publish-npm.yml`: | Field | Value | | --- | --- | | Name | `Publish npm package` | | Triggers | `workflow_dispatch`, `push` tags `v*` | | Permissions | `contents: read`, `id-token: write` | | Concurrency | `npm-publish`, `cancel-in-progress: false` | | Bun | 1.3.14 | | Node | 24, `registry-url: https://registry.npmjs.org`, `package-manager-cache: false` | Sequence: 1. `bun install --frozen-lockfile` 2. `bun run validate` (includes `prepublishOnly`-equivalent work; `npm publish` then runs `prepublishOnly` → `validate` again) 3. **Tag pushes only:** `node -p "require('./package.json').version"` must equal `${GITHUB_REF_NAME#v}` (tag `v0.3.3` ↔ version `0.3.3`) 4. `npm publish --access public` `workflow_dispatch` skips the tag/version equality check and publishes whatever `package.json` `version` is on the checked-out revision. The workflow does not bump the version, create a git tag, or pass `npm publish --provenance`. Ship a versioned release by setting `package.json` `version`, committing, and pushing a matching `v*` tag. Do not retag an already-published version; npm will reject the duplicate. ## Repository layout :::files self-bench/ src/ # API, CLI, Temporal worker, sandboxes, skills api-main.ts # bun run dev:api / node dist/api-main.js cli.ts # bun run cli / dist/cli.js temporal/worker-main.ts skills/selfbench/SKILL.md review/ # Vite + React review UI vite.config.ts # /v1 → http://127.0.0.1:8080 src/export-loader.ts # also tested via bun test review/src tests/ # bun:test suite (cli, workflow, audit, …) scripts/ verify-package.ts write-build-metadata.ts compose.yaml # postgres, temporal, api, worker Dockerfile # bun build stage + node:22 runtime Dockerfile.sandbox # selfbench-sandbox:local biome.json tsconfig.json # noEmit, src + tests tsconfig.build.json # emit dist/ from src .github/workflows/ci.yml .github/workflows/publish-npm.yml ::: ## Failure modes | Symptom | Likely cause | | --- | --- | | `bun install --frozen-lockfile` fails | `bun.lock` out of date; do not paper over it in CI | | `biome check .` fails | Format or recommended lint in `src/`, `tests/`, or `review/src/`; run `bun run format` only if the diff is intended | | `tsc --noEmit` fails | Server/tests `tsconfig.json` or `review/tsconfig.json` | | `dev:api` exits at startup | Temporal not on `127.0.0.1:7233`, or `:8080` already used by Compose `api` | | Worker exits after `docker info` | Docker backend/harbor without a working engine or Compose plugin | | Two workers idle/steal work | Host `dev:worker` and Compose `worker` share `selfbench-dev` | | `GET /` 404 on `dev:api` | Review assets live in `dist/review`, not `src/review`; use `dev:review` or `start:api` after build | | `verify-package` “did not print the expected CLI help” | `printHelp()` no longer contains `self-bench up`, or `bin.self-bench` is not `dist/cli.js` | | Missing `compose.yaml` / Dockerfiles / `SKILL.md` in the tarball | Path dropped from `files` or not present at pack time | | Publish “Verify tag matches package version” | Tag is not `v` + exact `package.json` `version` | | `SELFBENCH_BUILD_COMMIT must be a full commit SHA` | Env set to a short SHA or non-hex value | `src/` is published, so a broken source CLI can still ship even when `dist/` is what users run. Keep `validate` green before tagging. ## Related pages Bun, global install, Modal/GitHub auth, and the first `self-bench --help` from a published bin. `self-bench up` / `down`, Compose services, backends, and named volumes. `SELFBENCH_*` keys, loopback token rule, artifact backends, Temporal fields. Routes the review proxy and `dev:api` serve, including `/healthz` and Bearer `/v1`. API-hosted `dist/review` versus the Vite app and local archive inspection. Flags and errors for `up`, `run`, `status`, `cancel`, `download`, `list`. `self-bench-eval`, `validate`, `reaudit`, `repair`, and `agent-smoke`. Split `node dist/api-main.js` and `node dist/temporal/worker-main.js` with matching image digest and queue. ---