# Companion commands
> Flags, defaults, and outputs for self-bench-eval, self-bench-validate, self-bench-reaudit, self-bench-repair, and self-bench-agent-smoke.
- Repository: mupt-ai/self-bench
- GitHub: https://github.com/mupt-ai/self-bench
- Human docs: https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc
- Complete Markdown: https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc/llms-full.txt
## Source Files
- `src/eval-main.ts`
- `src/validate-main.ts`
- `src/reaudit-main.ts`
- `src/repair-main.ts`
- `src/agent-smoke-main.ts`
- `package.json`
---
---
title: "Companion commands"
description: "Flags, defaults, and outputs for self-bench-eval, self-bench-validate, self-bench-reaudit, self-bench-repair, and self-bench-agent-smoke."
---
The `self-bench` npm package installs five extra binaries next to `self-bench`: `self-bench-eval`, `self-bench-validate`, `self-bench-reaudit`, `self-bench-repair`, and `self-bench-agent-smoke`. They are not `self-bench` subcommands, they do not talk to `SELFBENCH_API_URL`, and they operate on already expanded Harbor tasks (or, for the matrix runner, a `.tar.gz` export). Each command parses flags with Node `parseArgs({ strict: true })`, writes progress to stderr, and prints JSON on stdout.
| Binary | Job | Default Harbor env | Default concurrency |
| --- | --- | --- | --- |
| `self-bench-eval` | Codex matrix over an export or task directory | `modal` | `3` |
| `self-bench-validate` | Harbor `nop` then `oracle` gates | `modal` | `10` |
| `self-bench-reaudit` | Sandboxed Sol coupling review | n/a (`SELFBENCH_EXECUTION_BACKEND`) | `10` |
| `self-bench-repair` | Sandboxed Codex repair of coupled tests | n/a (`SELFBENCH_EXECUTION_BACKEND`) | `9` |
| `self-bench-agent-smoke` | `--install-only` check of every pinned Harbor adapter | `modal` | `4` |
`self-bench-eval`, `self-bench-validate`, and `self-bench-agent-smoke` invoke the Harbor CLI. `self-bench-reaudit` and `self-bench-repair` start sandboxes through `loadConfig().execution` (`SELFBENCH_EXECUTION_BACKEND`, default `docker`) and require the published `dist/sandbox-review.bundle.js` or `dist/sandbox-repair.bundle.js` assets.
```mermaid
flowchart LR
subgraph Inputs
Export["export.tar.gz"]
Tasks["expanded task dirs"]
Review["coupling report.json"]
end
subgraph HarborCLI["Harbor CLI"]
Eval["self-bench-eval"]
Validate["self-bench-validate"]
Smoke["self-bench-agent-smoke"]
end
subgraph Sandbox["SELFBENCH_EXECUTION_BACKEND"]
Reaudit["self-bench-reaudit"]
Repair["self-bench-repair"]
end
Export --> Eval
Tasks --> Eval
Tasks --> Validate
Tasks --> Smoke
Tasks --> Reaudit
Tasks --> Repair
Reaudit --> Review
Review --> Repair
```
## Shared conventions
- **Install surface.** `package.json` `bin` maps each name to `dist/*-main.js`. Runtime requirement is Node `>=22`.
- **Help.** `-h` / `--help` prints usage and exits `0`.
- **Unknown flags.** `strict: true` rejects them.
- **Integers.** `--concurrency` (and any other numeric flag these commands parse) must be an integer `>= 1`.
- **Progress vs result.** Human-readable lines go to stderr. The last stdout payload is JSON.
- **Failure style.** Missing required flags and invalid combinations `throw`. Uncaught exceptions exit non-zero. Per-task sandbox or Harbor failures are usually folded into the JSON report instead of aborting the whole batch.
- **Task discovery differs.** Eval and validate treat a directory as a task when it contains `task.toml`. Reaudit and repair treat a directory as a task when it contains `instruction.md`.
- **Not the Temporal pipeline.** Generation, one validation-repair attempt, and one coupling-repair retry live in the worker. These binaries are offline operator tools for already-written Harbor tasks.
```bash
self-bench-eval --help
self-bench-validate --help
self-bench-reaudit --help
self-bench-repair --help
self-bench-agent-smoke --help
```
```bash
node dist/eval-main.js --help
node dist/validate-main.js --help
node dist/reaudit-main.js --help
node dist/repair-main.js --help
node dist/agent-smoke-main.js --help
```
The checkout must already be built (`bun run build`) so `dist/` and the sandbox bundles exist.
## self-bench-eval
Runs every selected Harbor task through Harbor's `codex` adapter for a fixed model matrix: `gpt-5.6-sol`, `gpt-5.6-terra`, and `gpt-5.6-luna`. Each trial pins Codex `0.146.1` and `reasoning_effort=high`. Provide exactly one of `--export` or `--tasks`. `--jobs` is required.
```bash
self-bench-eval (--export FILE.tar.gz | --tasks DIRECTORY) --jobs DIRECTORY [options]
```
Self-bench export archive. Extracted once into `/tasks/` unless that directory already contains Harbor tasks.
Directory of already expanded Harbor tasks (one subdirectory per task, each with `task.toml`).
Harbor jobs directory. Created if missing. Also receives `summary.json` and, for `--export`, the materialized `tasks/` tree.
Harbor executable. Default: `harbor` on `PATH`.
Passed to Harbor as `--env`. Default: `modal`. Any other value throws `--environment must be docker or modal`.
Concurrent trials (task × model pairs). Default: `3`.
Restrict the matrix. Repeatable; duplicates are dropped. Invalid names throw `--model must be one of gpt-5.6-sol, gpt-5.6-terra, gpt-5.6-luna`.
Codex ChatGPT `auth.json` used only when `OPENAI_API_KEY` is unset. Default fallback: `~/.codex/auth.json`.
### Task input
`--export` and `--tasks` are mutually exclusive. Passing both, or neither, throws `provide exactly one of exportPath or tasksPath`. An empty export or directory throws `found no Harbor tasks in …`.
`--export` expects the published archive shape: `tasks/.tar.gz`, each inner archive containing a `harbor-task/` directory. The runner copies each `harbor-task/` to `/tasks//`. If `/tasks/` already has one or more `task.toml` directories, extraction is skipped and those tasks are reused.
`--tasks` accepts any count of tasks, including fewer than ten.
### Authentication
`OPENAI_API_KEY` (trimmed, non-empty) wins. Harbor then runs with the process environment as-is, and `summary.json` records `"auth": "openai-api-key"`.
Without that key, the runner reads `--auth` or `~/.codex/auth.json` and requires `auth_mode: "chatgpt"` plus a `tokens` object. It then:
- deletes `OPENAI_API_KEY` from the Harbor child environment
- sets `CODEX_FORCE_AUTH_JSON=1`
- sets `CODEX_AUTH_JSON_PATH` to the resolved auth file
`summary.json` records `"auth": "codex-subscription"`.
### Harbor trial
Each trial job name is deterministic:
```text
${taskId}-${model} → lowercased, non [a-z0-9_.-] replaced with -
```
Harbor is invoked as:
```bash
harbor run \
--path \
--agent codex \
--model \
--ak version=0.146.1 \
--ak reasoning_effort=high \
--env \
--job-name \
--jobs-dir \
--n-concurrent 1 \
--max-retries 1 \
--delete --yes --quiet
```
The child is allowed to fail; timeout is 4 hours. A completed Harbor result is reused and the trial is not rerun. An incomplete job directory is renamed to `.incomplete-` before a fresh run. If Harbor exits without a readable result, the trial is recorded as failed with `exception` starting `Harbor produced no result`.
A trial **passes** when it has no exception, at least one numeric verifier reward, and either `rewards.reward >= 1` (other reward fields may be zero diagnostics) or, if `reward` is absent, every numeric reward is `>= 1`. Empty rewards fail.
### Outputs
Stderr progress:
```text
[3/9] gpt-5.6-sol task-id: passed
```
Status is `passed`, `failed`, or `error` (exception present).
Stdout:
```json
{
"trials": 9,
"passed": 6,
"failed": 3
}
```
`/summary.json` is the durable record:
Always `1`.
Always `codex`.
`0.146.1`.
Which credential path was used.
Always `high`.
Models that were scheduled.
Distinct task directories.
Per-trial `taskId`, `model`, `jobName`, `passed`, `rewards`, optional `exception`.
```bash title="Export matrix"
export OPENAI_API_KEY=...
self-bench-eval \
--export ./self-bench-evals.tar.gz \
--jobs ./matrix-jobs \
--environment modal \
--concurrency 20
```
```bash title="Single model, expanded tasks, subscription auth"
self-bench-eval \
--tasks ./self-bench-tasks \
--jobs ./matrix-jobs \
--model gpt-5.6-sol \
--auth ~/.codex/auth.json
```
`self-bench-eval` does not classify Harbor infrastructure exceptions separately. A Docker/Modal/auth failure becomes a failed trial with `exception` set, unlike `self-bench-validate`, which treats those as task-level errors.
## self-bench-validate
Runs Harbor `nop` then `oracle` against every `task.toml` subdirectory. There is no `--harbor` flag: the executable is always `harbor`. There is no resume key — each gate uses a fresh job name.
```bash
self-bench-validate --tasks DIRECTORY --jobs DIRECTORY [options]
```
Directory of expanded Harbor tasks. Subdirectories without `task.toml` are ignored.
Harbor jobs directory (created recursively) and location of `validation-summary.json`.
Passed as Harbor `--env`. Default: `modal`. Other values throw `--environment must be modal or docker`.
Concurrent tasks (each task still runs `nop` then `oracle` sequentially). Default: `10`.
Each gate:
```bash
harbor run \
--path \
--agent nop|oracle \
--env \
--job-name --<8 hex chars> \
--jobs-dir \
--n-concurrent 1 \
--max-retries 1 \
--delete --yes --quiet
```
Timeout is 4 hours. A non-zero Harbor exit becomes a task `error` (`Harbor exited : …`, last 1000 characters of stderr). After a zero exit, the command reads the job result. AuthError or matching infrastructure text (`unknown flag: --project-name`, Docker daemon/connect failures, Modal timeout, image-build failure, Docker address-pool exhaustion) throws `Harbor infrastructure failure: …` and is recorded as `status: "error"`, not a clean reject.
### Gate thresholds
| Agent | Pass when |
| --- | --- |
| `nop` | `fail_to_pass === 0` and `pass_to_pass >= 1` and `setup_completed >= 1` |
| `oracle` | `patch_applied`, `fail_to_pass`, `pass_to_pass`, `deterministic`, and `setup_completed` are all `>= 1` |
Missing numeric rewards are treated as `0` via `?? 0`, except nop `fail_to_pass`, which must be present and exactly `0`. A task is `accepted` only when both gates pass.
### Outputs
Stderr: `validating ` then `: accepted|rejected` or `: error: …`.
Stdout is the summary only:
```json
{
"total": 4,
"accepted": 3,
"rejected": 1,
"errors": 0
}
```
`/validation-summary.json` adds `schemaVersion: 1`, `generatedAt`, `sourceTasks`, `environment`, and a `tasks` array. Each entry is either `{ taskId, status: "validated", accepted, nop, oracle }` with `passed`/`rewards` per gate, or `{ taskId, status: "error", accepted: false, error }`.
```bash title="Validate expanded tasks"
self-bench-validate \
--tasks ./self-bench-tasks \
--jobs ./validation-jobs \
--environment modal \
--concurrency 10
```
## self-bench-reaudit
Re-runs independent test-to-gold coupling review on expanded tasks. It is not a Harbor command. Each task is extracted from `environment/repo.tar.gz` and reviewed in a sandbox with `dist/sandbox-review.bundle.js` as `gpt-5.6-sol` (`COUPLING_REVIEW_MODEL`).
```bash
self-bench-reaudit --tasks DIRECTORY --output REPORT.json [options]
```
Directory of expanded tasks. A child is included only if it has `instruction.md`.
JSON report path. Parent directories are created.
Concurrent sandbox reviews. Default: `10`.
Limit the run to one task id (directory basename). Missing id throws `task not found: `.
An empty selection throws `no tasks found`.
Each review sandbox:
- `runId`: `selfbench-reaudit-v2`
- timeout: 15 minutes
- inputs: `/work/sandbox-review.js`, `/work/review-input.md` (prompt, `tests/test.patch`, `solution/gold.patch`, plus deterministic coupling evidence from the base snapshot)
- output: `/work/review.json`
- command: `node /work/sandbox-review.js`
Secrets: `OPENAI_API_KEY` when set; otherwise `SELFBENCH_PI_AUTH_JSON` from `SELFBENCH_PI_AUTH_JSON` or `~/.pi/agent/auth.json` (`openai-codex` oauth `access` + `refresh`). The inner reviewer also requires `SELFBENCH_REVIEW_OUTPUT=/work/review.json`.
The written verdict is not the raw model JSON alone. `resolveCouplingReview` forces `coupled` when the model says `coupled`, when any gold-only finding has disposition `gold_only`, or when deterministic gold-only artifacts are missing from the review findings.
The report is marked `provisional: true`.
### Outputs
Stderr: `reviewing ` then `: clean|coupled` or `: error: …`.
Stdout:
```json
{
"output": "/abs/path/REPORT.json",
"total": 4,
"clean": 2,
"coupled": 1,
"errors": 1
}
```
The report file:
`1`.
`gpt-5.6-sol`.
ISO timestamp.
Resolved `--tasks` path.
Always `true` on this CLI.
`reviewed` entries include `sandboxId`, `couplingEvidence`, `review`, and `resolution`; `error` entries include `error`.
`total`, `clean`, `coupled`, `errors`.
## self-bench-repair
Applies one Codex repair sandbox to every `coupled` task in a reaudit report and writes a new task tree. Repair may change only paths already present in the held-out `tests/test.patch`. It does not edit the instruction, base snapshot, or gold patch.
```bash
self-bench-repair --tasks DIRECTORY --review REPORT.json --output DIRECTORY [options]
```
Source expanded tasks (`instruction.md` discovery).
Coupling report from `self-bench-reaudit`. Must contain a `tasks` array where every entry has `taskId` and `resolution.verdict` of `clean` or `coupled`.
Destination directory. Must not equal `--tasks` and must not be inside it (`--output must be outside --tasks`). **Deleted recursively** before the run.
Concurrent repair sandboxes. Default: `9`.
Codex model passed as `SELFBENCH_REPAIR_MODEL`. Default: `gpt-5.6-sol`.
Repair only that coupled task. Throws `coupled task not found in review: ` if it is missing or not `coupled`. When set, clean tasks are **not** copied (`copiedClean` is `0`).
Without `--task`, every non-coupled source task is copied into `--output` unchanged. Coupled tasks are packed as `task.tar.gz` and sent to a sandbox:
- `runId`: `selfbench-repair-v1`
- timeout: 2 hours
- resources: 4 CPU, 8192 MiB
- files: `/work/task.tar.gz`, `/work/review.json`, `/work/sandbox-repair.js`
- outputs: `/work/repaired-task.tar.gz`, `/work/repair-report.json`
- secrets: `OPENAI_API_KEY` or `SELFBENCH_CODEX_AUTH_JSON` (from `SELFBENCH_CODEX_AUTH_JSON`, `CODEX_AUTH_JSON_PATH`, or `~/.codex/auth.json`)
Inside the sandbox, Codex runs `codex exec --model --dangerously-bypass-approvals-and-sandbox --ephemeral --ignore-user-config --json` with a 90-minute timeout. A no-op patch, a missing `diff --git` patch, or edits outside the original test paths fail that task.
Successful repair overwrites `tests/test.patch`, records `testPatchSha256` and a `repair` block on `.selfbench-manifest.json`, and unpacks the new tree under `--output//`.
### Outputs
The command prints the full report on stdout and writes the same object to `