# Contributing

> Clone, bun install, bun run validate, source CLI, API/worker/review dev servers, CI, package verification, and npm tag publish.

- Repository: mupt-ai/self-bench
- GitHub: https://github.com/mupt-ai/self-bench
- Human docs: https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc
- Complete Markdown: https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc/llms-full.txt

## Source Files

- `package.json`
- `README.md`
- `.github/workflows/ci.yml`
- `.github/workflows/publish-npm.yml`
- `scripts/verify-package.ts`
- `biome.json`
- `tsconfig.build.json`

---

---
title: "Contributing"
description: "Clone, bun install, bun run validate, source CLI, API/worker/review dev servers, CI, package verification, and npm tag publish."
---

`self-bench` 0.3.3 is an MIT TypeScript package. Local work uses Bun against `bun.lock`; published bins are compiled Node entry points under `dist/`. `bun run validate` is the contributor, CI, and `prepublishOnly` gate: Biome plus two TypeScript projects, `bun test`, a full `dist/` build, then a pack-and-install check that the `self-bench` binary still prints `self-bench up`.

<Note>
CI and the Docker build image pin **Bun 1.3.14**. `package.json` `engines.node` is `>=22`. The publish workflow also installs **Node 24**. There is no separate `CONTRIBUTING.md`; the scripts and workflows below are the contract.
</Note>

## Prerequisites

| Tool | Where it is pinned | Needed for |
| --- | --- | --- |
| Bun 1.3.14 | `.github/workflows/ci.yml`, `Dockerfile` (`oven/bun:1.3.14-debian`) | install, tests, `bun run cli`, `bun pm pack` |
| Node `>=22` | `package.json` `engines`, runtime image `node:22-bookworm` | `tsx` / `node dist/*` after compile |
| Docker + Compose | `src/cli.ts` `up`/`down`, `src/temporal/worker-main.ts` | local Temporal, image builds, Docker sandbox worker |
| Git | `scripts/write-build-metadata.ts`, CLI `rev-parse` | `SELFBENCH_BUILD_COMMIT` fallback |

Model keys, `GH_TOKEN`, and Modal tokens are not required to lint, typecheck, test, or pack. They are required only if you start a worker that authors or validates tasks. See [Installation](/installation) and [Authentication](/authentication).

## Clone and install

<Steps>
<Step title="Clone the repository">

```bash
git clone https://github.com/mupt-ai/self-bench.git
cd self-bench
```

</Step>
<Step title="Install with the locked graph">

```bash
bun install --frozen-lockfile
```

CI uses the same flag. A lockfile drift fails install instead of rewriting `bun.lock`.

</Step>
<Step title="Run the full gate">

```bash
bun run validate
```

Success ends with `verify-package` printing `verified self-bench@<version>` (currently `self-bench@0.3.3`) after a temporary `bun pm pack` install.

</Step>
<Step title="Confirm the source CLI">

```bash
bun run cli -- --help
```

The extra `--` is required so Bun does not consume `--help`. The process is `bun run src/cli.ts`. Expected usage includes `self-bench up`, `down`, `run`, `status`, `cancel`, `download`, and `list`.

</Step>
</Steps>

## Script map

| Script | Command | Role |
| --- | --- | --- |
| `validate` | `bun run check && bun run test && bun run build && bun run verify:package` | Single local/CI/publish gate |
| `check` | `biome check . && tsc --noEmit -p tsconfig.json && bun run typecheck:review` | Lint/format + server/tests types + review types |
| `test` | `bun test tests review/src` | `bun:test` under `tests/` and `review/src/` |
| `format` | `biome format --write .` | Rewrite using Biome (spaces, width 100) |
| `typecheck:review` | `tsc --noEmit -p review/tsconfig.json` | Review Vite/React project |
| `cli` | `bun run src/cli.ts` | Source CLI (not the published `dist/cli.js`) |
| `dev:api` | `tsx src/api-main.ts` | Host API against Temporal |
| `dev:worker` | `tsx src/temporal/worker-main.ts` | Host Temporal worker |
| `dev:review` | `vite --config review/vite.config.ts` | Review UI with `/v1` proxy |
| `build` | `clean` + `build:server` + `build:review` | Populate `dist/` |
| `start:api` | `node dist/api-main.js` | Compiled API (Compose/image default) |
| `start:worker` | `node dist/temporal/worker-main.js` | Compiled worker |
| `verify:package` | `bun scripts/verify-package.ts` | Pack, install, help + shipped-asset check |
| `prepack` | `bun run build` | Runs before `bun pm pack` / `npm pack` |
| `prepublishOnly` | `bun run validate` | Runs before `npm publish` |

Biome (`biome.json`) lints and formats only `src/**/*.ts`, `tests/**/*.ts`, and `review/src/**/*.{ts,tsx}`. The server `tsconfig.json` typechecks `src/**/*.ts` and `tests/**/*.ts` with `strict`, `noUncheckedIndexedAccess`, and `exactOptionalPropertyTypes`. `tsconfig.build.json` emits `dist/` from `src/**/*.ts` and excludes `src/**/*.test.ts` and `src/review_dist`.

## Source CLI and companion bins

Published `bin` entries all point at compiled files:

| Bin | Source | Dist |
| --- | --- | --- |
| `self-bench` | `src/cli.ts` | `dist/cli.js` |
| `self-bench-eval` | `src/eval-main.ts` | `dist/eval-main.js` |
| `self-bench-validate` | `src/validate-main.ts` | `dist/validate-main.js` |
| `self-bench-reaudit` | `src/reaudit-main.ts` | `dist/reaudit-main.js` |
| `self-bench-repair` | `src/repair-main.ts` | `dist/repair-main.js` |
| `self-bench-agent-smoke` | `src/agent-smoke-main.ts` | `dist/agent-smoke-main.js` |

<CodeGroup>

```bash title="Source (checkout)"
bun run cli -- --help
bun src/eval-main.ts --help
bun src/validate-main.ts --help
bun src/reaudit-main.ts --help
bun src/repair-main.ts --help
bun src/agent-smoke-main.ts --help
```

```bash title="Compiled (after bun run build)"
node dist/cli.js --help
node dist/eval-main.js --help
```

</CodeGroup>

`src/cli.ts` resolves `compose.yaml` as `../compose.yaml` from the CLI file, so source `up`/`down` use the checkout root and a published install uses the package root. `up --backend docker` builds `Dockerfile.sandbox` as `selfbench-sandbox:local` before `docker compose --file compose.yaml up -d --build`. Help verification in `scripts/verify-package.ts` only asserts that installed `self-bench --help` contains `self-bench up`.

<RequestExample>

```bash
bun run cli -- --help
```

</RequestExample>

<ResponseExample>

```text
SelfBench creates durable tiered Harbor evaluations.

Usage:
  self-bench up [--backend docker|modal] [--modal-config PATH]
  self-bench down
  self-bench run --repo PATH [--easy-count N] [--medium-count N] [--hard-count N]
                  [--model MODEL] [--run-id ID] [--wait] [--output OUTPUT.tar.gz]
  self-bench status RUN_ID
  self-bench cancel RUN_ID
  self-bench download RUN_ID OUTPUT.tar.gz
  self-bench list
```

</ResponseExample>

Default API target for the CLI is `SELFBENCH_API_URL` or `http://127.0.0.1:8080`. If `SELFBENCH_API_TOKEN` is set, the CLI sends `Authorization: Bearer …`.

## Host API, worker, and review

`dev:*` runs processes on the host. Compose `api` / `worker` run `node dist/api-main.js` and `node dist/temporal/worker-main.js` from image `selfbench:local`. Do not run both pairs against the same port and task queue.

```mermaid
flowchart LR
  subgraph host [Host checkout]
    CLI["bun run cli / src/cli.ts"]
    Review["dev:review Vite"]
    API["dev:api tsx src/api-main.ts"]
    Worker["dev:worker tsx src/temporal/worker-main.ts"]
  end
  subgraph compose [compose.yaml]
    PG["postgres:17.6-bookworm"]
    Temporal["temporalio/auto-setup:1.29.6 :7233"]
    CAPI["api node dist/api-main.js :8080"]
    CWorker["worker node dist/temporal/worker-main.js"]
  end
  Review -->|"proxy /v1"| API
  CLI -->|SELFBENCH_API_URL| API
  API --> Temporal
  Worker --> Temporal
  CAPI --> Temporal
  CWorker --> Temporal
  Temporal --> PG
```

<Warning>
`startApi` opens a Temporal client before `listen`. `dev:api` therefore fails if nothing is bound at `SELFBENCH_TEMPORAL_ADDRESS` (default `127.0.0.1:7233`). The worker polls `SELFBENCH_TEMPORAL_NAMESPACE` / `SELFBENCH_TASK_QUEUE` (defaults `default` / `selfbench-dev`). Compose does not override the queue name, so a host `dev:worker` and the Compose `worker` steal the same activities if both are up. Compose also publishes `127.0.0.1:8080`; a second API on the default port will not bind.
</Warning>

<Steps>
<Step title="Start only Postgres and Temporal">

```bash
docker compose up -d postgres temporal
```

Leave `api` and `worker` down so host processes own `:8080` and `selfbench-dev`.

</Step>
<Step title="Run the TypeScript API">

```bash
bun run dev:api
```

Logs `SelfBench API listening on http://127.0.0.1:8080` when `SELFBENCH_API_HOST` / `SELFBENCH_API_PORT` are unset.

`GET /` and `GET /assets/*` are served from a `review/` directory next to the compiled API module (`dist/review` after `build:review`). `tsx src/api-main.ts` therefore looks at `src/review`, which is not the Vite app. Use `dev:review` for UI work, or `bun run start:api` after a build to serve `dist/review`.

</Step>
<Step title="Run the TypeScript worker">

```bash
bun run dev:worker
```

On `SELFBENCH_EXECUTION_BACKEND=docker` or `SELFBENCH_HARBOR_ENVIRONMENT=docker`, startup runs `docker info` and `docker compose version` (30s timeout) before connecting. It logs `SelfBench worker polling <namespace>/<taskQueue> with activity concurrency <n>`. Docker default concurrency is `1`; Modal default is `20`.

The worker calls `removeEmptyModalCredentialOverrides()` first so empty `MODAL_TOKEN_*` values cannot mask a mounted profile.

</Step>
<Step title="Run the review Vite app">

```bash
bun run dev:review
```

`review/vite.config.ts` does not set `server.port` (Vite default **5173**). It proxies `/v1` to `http://127.0.0.1:8080` and lists extra `server.allowedHosts` / `preview.allowedHosts`. The UI calls `/v1/runs` and `/v1/runs/:id/export` with an optional Bearer token field; it can also open a local export file without the API.

</Step>
</Steps>

<ParamField body="SELFBENCH_API_HOST" type="string">
Default `127.0.0.1`. Binding anything other than `127.0.0.1`, `::1`, or `localhost` requires `SELFBENCH_API_TOKEN`.
</ParamField>

<ParamField body="SELFBENCH_API_PORT" type="string">
Default `8080`. Compose publishes `127.0.0.1:8080:8080`.
</ParamField>

<ParamField body="SELFBENCH_API_TOKEN" type="string">
Optional on loopback. Compose defaults to `selfbench-local` when the host env is unset. `/healthz`, `/`, and `/assets/*` stay unauthenticated; `/v1/*` requires Bearer when the token is set.
</ParamField>

<ParamField body="SELFBENCH_TEMPORAL_ADDRESS" type="string">
Default `127.0.0.1:7233` on the host. Compose services use `temporal:7233`.
</ParamField>

<ParamField body="SELFBENCH_TASK_QUEUE" type="string">
Default `selfbench-dev`. API workflow starts and the worker must match.
</ParamField>

<ParamField body="SELFBENCH_BUILD_COMMIT" type="string">
Optional 40-char hex SHA. `scripts/write-build-metadata.ts` writes `dist/build-metadata.js`. If unset, the script uses `git rev-parse HEAD`, else forty zeros. The CLI refuses an all-zero built commit and re-reads `HEAD` from the package/checkout root.
</ParamField>

For the full Compose image path (`self-bench up` / `self-bench down`), use [Start the local stack](/start-local-stack). Host `start:api` / `start:worker` match the image `CMD` / worker `command`.

## Build outputs

`bun run build` deletes `dist/`, then:

1. `tsc -p tsconfig.build.json` → `dist/**/*.js` plus declaration and source maps.
2. `bun build` of `src/sandbox-author.ts`, `src/sandbox-review.ts`, `src/sandbox-repair.ts`, `src/sandbox-validation-repair.ts` → `dist/sandbox-*.bundle.js` (`--target=node`).
3. `bun scripts/write-build-metadata.ts` → `dist/build-metadata.js`.
4. `vite build --config review/vite.config.ts` → `dist/review` (`emptyOutDir: true`).

The production image (`Dockerfile`) runs `bun run build` in `oven/bun:1.3.14-debian`, then copies `dist/`, `node_modules`, `src/extensions`, `src/skills`, and `package.json` into `node:22-bookworm`. Default `CMD` is `node dist/api-main.js`. Tests, Markdown, and `.git` are excluded by `.dockerignore`.

## Package verification

`bun run verify:package` (`scripts/verify-package.ts`):

1. `bun pm pack --destination <tmp>` (triggers `prepack` → `build`).
2. `bun init --yes` in a fresh directory and `bun add --no-save` the tarball.
3. Run `node_modules/.bin/self-bench --help` and require the substring `self-bench up`.
4. Read these installed paths (must exist):
   - `compose.yaml`
   - `Dockerfile`
   - `Dockerfile.sandbox`
   - `src/skills/selfbench/SKILL.md`
5. Assert installed `package.json` `name` and `version` match the workspace.

`package.json` `files` is the publish allowlist: `dist`, `src`, `docs`, `compose.yaml`, `Dockerfile`, `Dockerfile.sandbox`, `bun.lock`, `scripts`, `biome.json`, `tsconfig.json`, `tsconfig.build.json`, `.dockerignore`. `publishConfig` is `access: public` on `https://registry.npmjs.org/`.

<Check>
`validate` already includes `verify:package`. Re-run `bun run verify:package` alone after changing `bin`, `files`, Compose, Dockerfiles, or the shipped skill.
</Check>

## CI

`.github/workflows/ci.yml`:

| Field | Value |
| --- | --- |
| Name | `CI` |
| Triggers | `push` to `main`, every `pull_request` |
| Permissions | `contents: read` |
| Concurrency | `ci-${{ github.workflow }}-${{ github.ref }}`, `cancel-in-progress: true` |
| Runner | `ubuntu-latest` |
| Steps | `actions/checkout@v4` → `oven-sh/setup-bun@v2` (`bun-version: 1.3.14`) → `bun install --frozen-lockfile` → `bun run validate` |

A red CI check is a failed `validate`. There is no second job matrix and no release job in this workflow.

## npm tag publish

`.github/workflows/publish-npm.yml`:

| Field | Value |
| --- | --- |
| Name | `Publish npm package` |
| Triggers | `workflow_dispatch`, `push` tags `v*` |
| Permissions | `contents: read`, `id-token: write` |
| Concurrency | `npm-publish`, `cancel-in-progress: false` |
| Bun | 1.3.14 |
| Node | 24, `registry-url: https://registry.npmjs.org`, `package-manager-cache: false` |

Sequence:

1. `bun install --frozen-lockfile`
2. `bun run validate` (includes `prepublishOnly`-equivalent work; `npm publish` then runs `prepublishOnly` → `validate` again)
3. **Tag pushes only:** `node -p "require('./package.json').version"` must equal `${GITHUB_REF_NAME#v}` (tag `v0.3.3` ↔ version `0.3.3`)
4. `npm publish --access public`

<Warning>
`workflow_dispatch` skips the tag/version equality check and publishes whatever `package.json` `version` is on the checked-out revision. The workflow does not bump the version, create a git tag, or pass `npm publish --provenance`.
</Warning>

Ship a versioned release by setting `package.json` `version`, committing, and pushing a matching `v*` tag. Do not retag an already-published version; npm will reject the duplicate.

## Repository layout

:::files
self-bench/
  src/                    # API, CLI, Temporal worker, sandboxes, skills
    api-main.ts           # bun run dev:api / node dist/api-main.js
    cli.ts                # bun run cli / dist/cli.js
    temporal/worker-main.ts
    skills/selfbench/SKILL.md
  review/                 # Vite + React review UI
    vite.config.ts        # /v1 → http://127.0.0.1:8080
    src/export-loader.ts  # also tested via bun test review/src
  tests/                  # bun:test suite (cli, workflow, audit, …)
  scripts/
    verify-package.ts
    write-build-metadata.ts
  compose.yaml            # postgres, temporal, api, worker
  Dockerfile              # bun build stage + node:22 runtime
  Dockerfile.sandbox      # selfbench-sandbox:local
  biome.json
  tsconfig.json           # noEmit, src + tests
  tsconfig.build.json     # emit dist/ from src
  .github/workflows/ci.yml
  .github/workflows/publish-npm.yml
:::

## Failure modes

| Symptom | Likely cause |
| --- | --- |
| `bun install --frozen-lockfile` fails | `bun.lock` out of date; do not paper over it in CI |
| `biome check .` fails | Format or recommended lint in `src/`, `tests/`, or `review/src/`; run `bun run format` only if the diff is intended |
| `tsc --noEmit` fails | Server/tests `tsconfig.json` or `review/tsconfig.json` |
| `dev:api` exits at startup | Temporal not on `127.0.0.1:7233`, or `:8080` already used by Compose `api` |
| Worker exits after `docker info` | Docker backend/harbor without a working engine or Compose plugin |
| Two workers idle/steal work | Host `dev:worker` and Compose `worker` share `selfbench-dev` |
| `GET /` 404 on `dev:api` | Review assets live in `dist/review`, not `src/review`; use `dev:review` or `start:api` after build |
| `verify-package` “did not print the expected CLI help” | `printHelp()` no longer contains `self-bench up`, or `bin.self-bench` is not `dist/cli.js` |
| Missing `compose.yaml` / Dockerfiles / `SKILL.md` in the tarball | Path dropped from `files` or not present at pack time |
| Publish “Verify tag matches package version” | Tag is not `v` + exact `package.json` `version` |
| `SELFBENCH_BUILD_COMMIT must be a full commit SHA` | Env set to a short SHA or non-hex value |

`src/` is published, so a broken source CLI can still ship even when `dist/` is what users run. Keep `validate` green before tagging.

## Related pages

<CardGroup>
<Card title="Installation" href="/installation">
Bun, global install, Modal/GitHub auth, and the first `self-bench --help` from a published bin.
</Card>
<Card title="Start the local stack" href="/start-local-stack">
`self-bench up` / `down`, Compose services, backends, and named volumes.
</Card>
<Card title="Configuration reference" href="/configuration-reference">
`SELFBENCH_*` keys, loopback token rule, artifact backends, Temporal fields.
</Card>
<Card title="HTTP API" href="/http-api">
Routes the review proxy and `dev:api` serve, including `/healthz` and Bearer `/v1`.
</Card>
<Card title="Review exports" href="/review-exports">
API-hosted `dist/review` versus the Vite app and local archive inspection.
</Card>
<Card title="CLI reference" href="/cli-reference">
Flags and errors for `up`, `run`, `status`, `cancel`, `download`, `list`.
</Card>
<Card title="Companion commands" href="/companion-commands">
`self-bench-eval`, `validate`, `reaudit`, `repair`, and `agent-smoke`.
</Card>
<Card title="Deploy Temporal Cloud" href="/deploy-temporal-cloud">
Split `node dist/api-main.js` and `node dist/temporal/worker-main.js` with matching image digest and queue.
</Card>
</CardGroup>