# Build a benchmark

> self-bench run: --repo GitHub origin, 1-100 tier budgets, --model, --run-id, HEAD pin, provenance upload, and --output wait plus SHA-256 download.

- Repository: mupt-ai/self-bench
- GitHub: https://github.com/mupt-ai/self-bench
- Human docs: https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc
- Complete Markdown: https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc/llms-full.txt

## Source Files

- `src/cli.ts`
- `src/provenance.ts`
- `src/temporal/workflow.ts`
- `src/extensions/authoring.ts`
- `src/skills/selfbench/SKILL.md`
- `docs/task-construction.md`

---

---
title: "Build a benchmark"
description: "self-bench run: --repo GitHub origin, 1-100 tier budgets, --model, --run-id, HEAD pin, provenance upload, and --output wait plus SHA-256 download."
---

`self-bench run` pins a Git checkout (`origin` URL + `HEAD` SHA), uploads a sanitized NDJSON provenance corpus, and starts Temporal workflow `selfBenchRunWorkflow` via `POST /v1/runs`. `--easy-count`, `--medium-count`, and `--hard-count` are candidate authoring budgets whose total must be 1–100. `--output` implies wait: the CLI polls until `complete`, then writes a create-only SHA-256-verified export.

The CLI performs only repository metadata and provenance upload locally. Discovery, authoring, validation, review, audit, and export run on the API/worker stack. Closing a waiting CLI does not cancel the workflow.

<Warning>
Rejected candidates are not replaced. The export contains only accepted tasks and can finish with fewer tasks than the budget, including zero accepted tasks.
</Warning>

## Prerequisites

- A running stack (`self-bench up`). See [Start the local stack](/start-local-stack).
- `SELFBENCH_API_URL` if the API is not `http://127.0.0.1:8080`.
- The same `SELFBENCH_API_TOKEN` the API expects. Local Compose defaults the API token to `selfbench-local` when the host does not set one. When the CLI variable is set, every request sends `Authorization: Bearer $SELFBENCH_API_TOKEN`.
- A Git checkout whose `origin` can be resolved. SSH `git@github.com:owner/repo` is rewritten to `https://github.com/owner/repo`. Other remotes must already start with `https://`. GitHub pull-request provenance requires a `github.com` URL.
- Authenticated `gh` so the CLI can list merged pull requests.
- At least one sanitized local-session or GitHub pull-request message. An empty corpus fails before submit.
- Worker credentials for remote work: `GH_TOKEN` to clone the pinned origin, plus `OPENAI_API_KEY` (or the subscription JSON fallbacks). See [Authentication](/authentication).

<Note>
The published CLI refuses to submit if it cannot resolve a 40-character SelfBench commit. Resolution order is `SELFBENCH_BUILD_COMMIT`, baked `buildCommit` from the package build, then `git rev-parse HEAD` of the self-bench source tree.
</Note>

## Submit a run

<Steps>
<Step title="Point the CLI at the API">

```bash
export SELFBENCH_API_URL=http://127.0.0.1:8080
export SELFBENCH_API_TOKEN="${SELFBENCH_API_TOKEN:-selfbench-local}"
```

</Step>
<Step title="Choose a checkout and tier budgets">

Pass `--repo` as an absolute or relative path. Counts default to `0` and must sum to 1–100.

</Step>
<Step title="Submit and wait for the export">

```bash
self-bench run \
  --repo /absolute/path/to/your/repository \
  --easy-count 10 \
  --medium-count 10 \
  --hard-count 10 \
  --model gpt-5.6-sol \
  --output ./self-bench-evals.tar.gz
```

Omit `--output` and `--wait` to print the accept payload and return while Temporal continues. Reattach later with [Manage runs](/manage-runs).

</Step>
<Step title="Confirm the accept payload">

Stdout after a successful `POST /v1/runs` (`202`) looks like:

```json
{
  "runId": "sb-20260814123456-a1b2c3d4",
  "provenanceMessages": 42,
  "localProvenanceMessages": 17,
  "githubPullRequestMessages": 25
}
```

</Step>
<Step title="Watch phase changes, then verify the archive">

`--wait` and `--output` poll `GET /v1/runs/:runId` every 2 seconds. Each **phase change** is written to stderr:

```json
{"runId":"sb-20260814123456-a1b2c3d4","phase":"discovering","accepted":0,"rejected":0}
```

On `complete`, `--output` downloads the export. Success prints `{ "runId": "...", "output": "/abs/path/self-bench-evals.tar.gz" }`. Confirm `tar -tzf` shows `manifest.json` and `tasks/`.
</Step>
</Steps>

<ParamField body="repo" type="string" required>
Path to the Git checkout. Short flag `-r`. Resolved with `path.resolve`.
</ParamField>

<ParamField body="easy-count" type="integer">
Easy-tier authoring budget. Default `0`. Nonnegative integer.
</ParamField>

<ParamField body="medium-count" type="integer">
Medium-tier authoring budget. Default `0`.
</ParamField>

<ParamField body="hard-count" type="integer">
Hard-tier authoring budget. Default `0`.
</ParamField>

<ParamField body="model" type="string">
Authoring model string. Default `gpt-5.6-sol`. Posted as `authoringModel`. The API stores `{ provider: "openai-codex", model, reasoningEffort: "high" }`. Sandboxes use that model as `AUTHOR_MODEL`.
</ParamField>

<ParamField body="run-id" type="string">
Temporal workflow ID. Default `sb-<14-digit UTC timestamp>-<8 UUID chars>`. The API accepts `^[a-z0-9][a-z0-9-]{2,62}$` (3–63 characters). The CLI does not validate locally; an invalid ID fails at provenance upload.
</ParamField>

<ParamField body="wait" type="boolean">
Poll until `complete`. Default `false`. `--output` implies wait even when this flag is omitted.
</ParamField>

<ParamField body="output" type="string">
Destination path for the export `.tar.gz`. Short flag `-o`. Implies wait, then downloads after `complete`.
</ParamField>

## Local work: pin, collect, upload

### HEAD pin

The CLI records:

| Field | Source |
| --- | --- |
| `repository.url` | `git remote get-url origin`, with `git@github.com:` rewritten to `https://github.com/` |
| `repository.commit` | `git rev-parse HEAD` (full 40-character SHA) |

Uncommitted files are not part of the pin. Workers clone `SOURCE_REPO_URL` and check out that SHA (`git fetch origin $SOURCE_COMMIT`). Each candidate still uses its own pull-request `baseCommit` and `completedCommit`.

### Provenance corpus

The CLI concatenates, then rejects an empty result with `no sanitized local-session or GitHub pull-request provenance was found`:

1. Local user messages from `$HOME/.pi/agent/sessions`, `$HOME/.claude/projects`, `$HOME/.codex/sessions`, and `$HOME/.codex/archived_sessions` (`.jsonl` only), matched to the checkout or its `git worktree list` paths.
2. Up to 500 merged, non-draft, non-bot GitHub pull requests via `gh pr list`, kept when the title is non-empty, `changedFiles >= 1`, and `additions + deletions >= 20`. Bodies longer than 12,000 characters are dropped (title only).

Secrets and injected harness prefixes are stripped. GitHub records keep `sourceType: "github-pull-request"` plus `sourcePr` / `sourceUrl`. Details: [Provenance sources](/provenance-sources).

### Upload then start

<RequestExample>
```http
POST /v1/provenance?runId=sb-20260814123456-a1b2c3d4
Content-Type: application/x-ndjson
Authorization: Bearer $SELFBENCH_API_TOKEN
```

```http
POST /v1/runs
Content-Type: application/json
Authorization: Bearer $SELFBENCH_API_TOKEN

{
  "runId": "sb-20260814123456-a1b2c3d4",
  "repository": {
    "url": "https://github.com/example/project.git",
    "commit": "0123456789abcdef0123456789abcdef01234567"
  },
  "provenance": {
    "uri": "file:///var/lib/selfbench/artifacts/runs/.../provenance.jsonl",
    "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "sizeBytes": 4096,
    "contentType": "application/x-ndjson"
  },
  "candidateCounts": { "easy": 10, "medium": 10, "hard": 10 },
  "authoringModel": "gpt-5.6-sol",
  "selfbenchCommit": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
}
```
</RequestExample>

Provenance bodies are capped at 100 MiB and stored at `runs/<runId>/input/provenance.jsonl` (`201` + artifact ref). Run bodies use the 10 MiB default.

The API wraps the submission into `RunRequest`, starts `selfBenchRunWorkflow` with `workflowId = runId`, `workflowExecutionTimeout` 14 days, and returns `202 { "runId" }`. `version.selfbenchCommit` is `SELFBENCH_BUILD_COMMIT` on the API when set, otherwise the CLI SHA. `version.executionBackend` and `sandboxImage` come from worker/API config, not CLI flags.

Reusing a `runId` that already exists as a Temporal workflow ID fails start.

```mermaid
sequenceDiagram
  participant CLI as self-bench run
  participant API as HTTP API
  participant Artifacts as Artifact store
  participant Temporal as Temporal
  participant Worker as Worker sandboxes

  CLI->>CLI: origin URL + HEAD SHA
  CLI->>CLI: local sessions + gh merged PRs
  CLI->>API: POST /v1/provenance?runId=
  API->>Artifacts: put runs/id/input/provenance.jsonl
  API-->>CLI: 201 ArtifactRef
  CLI->>API: POST /v1/runs
  API->>Temporal: start selfBenchRunWorkflow
  API-->>CLI: 202 runId
  loop every 2s until complete or failed phase
    CLI->>API: GET /v1/runs/:runId
    API->>Temporal: status query
    API-->>CLI: phase, accepted, rejected
  end
  CLI->>API: GET /v1/runs/:runId/export
  API->>Artifacts: openRead(export)
  API-->>CLI: gzip + x-content-sha256
```

## Budgets versus accepted tasks

| Rule | Behavior |
| --- | --- |
| Total 1–100 | CLI rejects any other sum. API also requires each of `easy` / `medium` / `hard` in `0..100` and the same total bounds. |
| Budget, not target | Discovery fills each requested tier, then authors exactly those selected candidates. |
| No replacement | A rejected or `infrastructure_failed` candidate does not consume a reserve PR. |
| Empty export is allowed | After authoring, the workflow always enters `exporting` and can `complete` with `acceptedTaskIds: []`. |

Discovery uses eight shards and stops when every requested tier is filled, or enters `blocked` with `candidate pool exhausted; missing easy=…, medium=…, hard=…`. Lifecycle and per-candidate statuses: [Run lifecycle](/run-lifecycle). Eligibility thresholds: [Difficulty gates](/difficulty-gates).

## Wait and download

`waitForRun` treats `complete` as success. `blocked`, `failed`, and `cancelled` throw `SelfBench run <phase>[: <error>]` and skip download.

| Mode | After submit |
| --- | --- |
| neither flag | Print accept JSON and exit. Workflow keeps running. |
| `--wait` only | Poll; print the final status JSON on stdout. |
| `--output PATH` | Poll; on `complete`, `GET /v1/runs/:runId/export`. |

Download rules:

- `409` with `run export is not ready` if `status.export` is missing.
- Response must include `x-content-sha256` and a body, or the CLI throws `SelfBench API returned an export without integrity metadata`.
- Destination is opened with `wx` (create-only). An existing file fails.
- Bytes are hashed while streaming. A mismatch throws `downloaded export failed its SHA-256 integrity check` and deletes the partial file.
- The API also sends `content-disposition: attachment; filename="selfbench-<runId>.tar.gz"`. The CLI writes the path you passed, not that name.

The `x-content-sha256` value is the artifact digest of the outer archive, not a signature. Archive layout: [Export format](/export-format).

## Errors

| Condition | What you see |
| --- | --- |
| Missing `--repo` | `--repo is required` |
| Total counts outside 1–100 | `the total candidate count must be between 1 and 100` |
| Non-integer / negative count | `--easy-count must be a nonnegative integer` (same for medium/hard) |
| Origin is not `git@github.com:` and not `https://` | `unsupported origin URL: …` |
| Origin is HTTPS but not GitHub | Fails in GitHub provenance parse: `unsupported GitHub repository URL` |
| Empty corpus | `no sanitized local-session or GitHub pull-request provenance was found` |
| No SelfBench commit metadata | `SelfBench was built without commit metadata; set SELFBENCH_BUILD_COMMIT to a full commit SHA` |
| Missing / wrong API token | `unauthorized` |
| Invalid `runId` | `400` from Zod on `/v1/provenance` or `/v1/runs` |
| Provenance > 100 MiB | `request body exceeds 104857600 bytes` |
| Status JSON lacks `phase` | `SelfBench status response is missing its phase` |
| `blocked` / `failed` / `cancelled` | `SelfBench run blocked: …` (or `failed` / `cancelled`) |
| Export not ready | `run export is not ready` |
| SHA-256 mismatch | `downloaded export failed its SHA-256 integrity check`; output file removed |

<Tip>
A stopped worker pauses Temporal activities; it does not cancel the run. Restart the stack and poll or download the same `runId`.
</Tip>

## Next

<CardGroup>
<Card title="Run lifecycle" href="/run-lifecycle">
Phases, eight-shard discovery, and candidate versus accepted counts.
</Card>
<Card title="Manage runs" href="/manage-runs">
`list`, `status`, `cancel`, and a later `download` against the same API.
</Card>
<Card title="Evaluate with Harbor" href="/evaluate-with-harbor">
Extract the export and run `self-bench-eval` or Harbor.
</Card>
<Card title="Export format" href="/export-format">
`manifest.json`, per-task `harbor-task` archives, and SHA-256 fields.
</Card>
<Card title="CLI reference" href="/cli-reference">
Full flag set, wait/download semantics, and command-level errors.
</Card>
<Card title="Troubleshooting" href="/troubleshooting">
Empty provenance, blocked pools, 409 export-not-ready, and hash mismatches.
</Card>
</CardGroup>