# Evaluate with Harbor

> Extract an export, run one task or the Codex matrix via self-bench-eval, resume completed jobs, and smoke Harbor agent adapters.

- Repository: mupt-ai/self-bench
- GitHub: https://github.com/mupt-ai/self-bench
- Human docs: https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc
- Complete Markdown: https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc/llms-full.txt

## Source Files

- `src/eval-main.ts`
- `src/evaluate.ts`
- `src/agent-smoke-main.ts`
- `src/agent-smoke.ts`
- `docs/evaluations.md`
- `tests/evaluate.test.ts`

---

---
title: "Evaluate with Harbor"
description: "Extract an export, run one task or the Codex matrix via self-bench-eval, resume completed jobs, and smoke Harbor agent adapters."
---

`self-bench-eval` is the published companion binary that turns a completed self-bench export into Harbor Codex trials. It accepts either the outer `.tar.gz` or an already-expanded task tree, fans out one Harbor job per `(task, model)` pair, writes `summary.json` under `--jobs`, and skips Harbor when that job already has a finished result. A single-task `harbor run` path remains available without the matrix helper. `self-bench-agent-smoke` is a separate install-only check of Harbor's pinned agent adapters, not a scoring run.

Generation-time Harbor **nop** / **oracle** gates belong to authoring, not this workflow. See [Validation and repair](/validation-and-repair).

<Warning>
Exports contain repository snapshots, held-out tests, and reference solutions. They are sensitive and unencrypted. Keep them private and extract only archives from a trusted self-bench deployment.
</Warning>

## Prerequisites

| Requirement | Role |
| --- | --- |
| `self-bench` package (`node` ≥ 22) | Provides `self-bench-eval` and `self-bench-agent-smoke` |
| Harbor `harbor[modal]==0.20.1.dev202608040148` | Task runner. Put `harbor` on `PATH` or pass `--harbor` |
| Python 3.12 + `uv` | Harbor install |
| `OPENAI_API_KEY` **or** Codex ChatGPT `auth.json` | Codex trial auth. API key wins when set |
| Modal account/token | Required for `--environment modal` (default) |
| Docker daemon | Required for `--environment docker` |

Install the pinned Harbor release:

```bash
uv tool install --python 3.12 'harbor[modal]==0.20.1.dev202608040148'
harbor --help
```

From a self-bench checkout after `bun run build`, the same binaries are `node dist/eval-main.js` and `node dist/agent-smoke-main.js`.

<Note>
`self-bench-eval` does not read `SELFBENCH_API_URL`, `SELFBENCH_HARBOR_ENVIRONMENT`, or `SELFBENCH_CODEX_AUTH_JSON`. Those keys belong to the API/worker and generation sandboxes. Evaluation talks to a local Harbor executable and uses `OPENAI_API_KEY` or a Codex `auth.json` file.
</Note>

## Get an export

The input is the SHA-256-verified outer archive from a completed run (`self-bench run --output` or `self-bench download`). Download is create-only: the destination path must not already exist. The API response must include `x-content-sha256`; a mismatch deletes the partial file.

<CodeGroup>

```bash title="Wait and download during submit"
self-bench run \
  --repo /absolute/path/to/your/repository \
  --easy-count 10 \
  --medium-count 10 \
  --hard-count 10 \
  --output ./self-bench-evals.tar.gz
```

```bash title="Download a finished run"
self-bench download RUN_ID ./self-bench-evals.tar.gz
```

</CodeGroup>

The outer archive is:

:::files
manifest.json
tasks/
├── TASK_ID.tar.gz
└── …
:::

Each inner archive is a compiled Harbor task under `harbor-task/` (`task.toml` schema 1.4, `instruction.md`, `environment/`, `tests/`, `solution/`). Full field layout is on [Export format](/export-format).

## Extract tasks

Use one of these layouts. `self-bench-eval --export` does the inner extract itself into `<jobs>/tasks/`. `--tasks` expects each immediate subdirectory to contain `task.toml`.

<Tabs>
<Tab title="Let self-bench-eval extract">

Pass the outer archive. On first use, the runner unpacks `tasks/*.tar.gz` and copies each `harbor-task/` to `<jobs>/tasks/<taskId>/`. If `<jobs>/tasks/` already has one or more `task.toml` directories, that tree is reused and the export is **not** re-read.

```bash
self-bench-eval --export ./self-bench-evals.tar.gz --jobs ./matrix-jobs
```

</Tab>
<Tab title="Expand every task">

Matches the README layout: each task directory *is* the Harbor task ( `task.toml` at the top of the folder).

```bash
mkdir -p ./self-bench-export ./self-bench-tasks
tar -xzf ./self-bench-evals.tar.gz -C ./self-bench-export
for archive in ./self-bench-export/tasks/*.tar.gz; do
  task_id="$(basename "$archive" .tar.gz)"
  mkdir -p "./self-bench-tasks/$task_id"
  tar -xzf "$archive" --strip-components=1 -C "./self-bench-tasks/$task_id"
done
```

Then pass `--tasks ./self-bench-tasks`.

</Tab>
<Tab title="Expand one task">

```bash
mkdir -p ./export ./selected-task
tar -xzf ./self-bench-evals.tar.gz -C ./export
TASK_ID="$(jq -r '.tasks[0].taskId' ./export/manifest.json)"
tar -xzf "./export/tasks/$TASK_ID.tar.gz" -C ./selected-task
```

Raw Harbor uses `--path ./selected-task/harbor-task`. The matrix helper uses `--tasks ./selected-task` because `harbor-task/task.toml` is the discovery signal.

</Tab>
</Tabs>

Discovery ignores files and directories that do not contain a readable `task.toml`. Empty inputs fail with `found no Harbor tasks in …`. Providing both `--export` and `--tasks` fails with `provide exactly one of exportPath or tasksPath`.

## What a trial mounts

A Codex (or other coding-agent) trial receives the base snapshot, `instruction.md`, and the compiled agent environment. It does not receive the held-out test patch or the reference solution. Harbor mounts `solution/` only for the explicit `oracle` agent.

Compiled `task.toml` also pins the agent network allowlist to `chatgpt.com`, `*.chatgpt.com`, `openai.com`, and `*.openai.com`, and collects `/opt/selfbench/agent.patch` as the trial artifact. Resource and timeout numbers come from the task definition, not from `self-bench-eval` flags.

See [Agent-visible boundary](/agent-visible-boundary) and [Harbor task compiler](/harbor-task-compiler).

## Run one task

This path calls Harbor directly. It does not write `summary.json` and does not implement self-bench's job-resume rules.

<Steps>
<Step title="Install Harbor and set API-key auth">

```bash
uv tool install --python 3.12 'harbor[modal]==0.20.1.dev202608040148'
export OPENAI_API_KEY=...
```

</Step>
<Step title="Run Codex against the harbor-task directory">

```bash
harbor run \
  --path ./selected-task/harbor-task \
  --agent codex \
  --model gpt-5.6-sol \
  --ak version=0.146.1 \
  --ak reasoning_effort=high \
  --env modal \
  --jobs-dir ./harbor-jobs \
  --yes
```

Codex is pinned to `@openai/codex@0.146.1` via `--ak version=0.146.1`. Reasoning is `high`.

</Step>
<Step title="Optional: one Harbor job for every extracted task">

Harbor can take the expanded tree and all three matrix models in one invocation (no per-job resume):

```bash
harbor run \
  --path ./self-bench-tasks \
  --agent codex \
  --model gpt-5.6-luna \
  --model gpt-5.6-terra \
  --model gpt-5.6-sol \
  --ak version=0.146.1 \
  --ak reasoning_effort=high \
  --env modal \
  --jobs-dir ./harbor-jobs \
  --n-concurrent 20 \
  --yes
```

</Step>
</Steps>

## Run the Codex matrix

`self-bench-eval` evaluates every discovered task with the fixed matrix `gpt-5.6-sol`, `gpt-5.6-terra`, and `gpt-5.6-luna` unless `--model` narrows the set. Work is the cartesian product `models × tasks`, executed with `--concurrency` (default `3`). Each Harbor subprocess uses `--n-concurrent 1`.

```bash
export OPENAI_API_KEY=...
self-bench-eval \
  --export ./self-bench-evals.tar.gz \
  --jobs ./matrix-jobs \
  --harbor harbor \
  --environment modal \
  --concurrency 20
```

Restrict models by repeating `--model`. Values must be members of the matrix; duplicates are dropped.

```bash
self-bench-eval \
  --tasks ./self-bench-tasks \
  --jobs ./matrix-jobs \
  --model gpt-5.6-sol \
  --model gpt-5.6-luna
```

### Flags

<ParamField body="export" type="string">
Outer self-bench `.tar.gz`. Mutually exclusive with `--tasks`. Present in usage and parsing; omitted from the `--help` options list.
</ParamField>

<ParamField body="tasks" type="string">
Directory whose child folders each contain `task.toml`. Mutually exclusive with `--export`.
</ParamField>

<ParamField body="jobs" type="string" required>
Job root. Created if missing. Holds Harbor job directories, optional `tasks/` materialization, and `summary.json`.
</ParamField>

<ParamField body="harbor" type="string">
Harbor executable. Default: `harbor`.
</ParamField>

<ParamField body="environment" type="string">
`modal` or `docker`. Default: `modal`. Any other value throws `--environment must be docker or modal`.
</ParamField>

<ParamField body="concurrency" type="string">
Positive integer of in-flight trials. Default: `3`.
</ParamField>

<ParamField body="model" type="string">
Repeatable. Allowed values: `gpt-5.6-sol`, `gpt-5.6-terra`, `gpt-5.6-luna`. Omit to run all three.
</ParamField>

<ParamField body="auth" type="string">
Codex ChatGPT `auth.json` used only when `OPENAI_API_KEY` is unset. Default path: `~/.codex/auth.json`.
</ParamField>

`--jobs` is required. `--concurrency` must be a positive integer. An empty model list (API use of `runMatrix`) throws `provide at least one evaluation model`.

### Authentication

| Condition | Behavior |
| --- | --- |
| `OPENAI_API_KEY` set (non-empty after trim) | Harbor inherits the process environment. `summary.json` records `auth: "openai-api-key"`. `--auth` is ignored. |
| `OPENAI_API_KEY` unset | Runner reads `--auth` or `~/.codex/auth.json`. File must be ChatGPT subscription JSON: `auth_mode` is `chatgpt` and `tokens` is an object. API-key-mode files are rejected. |
| Subscription child env | `OPENAI_API_KEY` is deleted. `CODEX_FORCE_AUTH_JSON=1` and `CODEX_AUTH_JSON_PATH` point at the auth file. `summary.json` records `auth: "codex-subscription"`. |

Auth is checked once before any trial. A missing or invalid subscription file aborts the process; it is not recorded as a per-trial exception.

### Harbor invocation

Each missing or incomplete trial runs:

```bash
harbor run \
  --path <taskDirectory> \
  --agent codex \
  --model <matrix-model> \
  --ak version=0.146.1 \
  --ak reasoning_effort=high \
  --env <docker|modal> \
  --job-name <taskId-model> \
  --jobs-dir <jobs> \
  --n-concurrent 1 \
  --max-retries 1 \
  --delete \
  --yes \
  --quiet
```

Job names are `${taskId}-${model}`, lowercased, with characters outside `[a-z0-9_.-]` replaced by `-`. The subprocess budget is four hours (`timeoutMs: 14400000`); a timeout SIGTERMs Harbor, then SIGKILLs after five seconds.

`--environment` selects Harbor's execution backend only. It does not change the compiled Dockerfiles. Pairing details for Docker versus Modal are on [Execution backends](/execution-backends).

### Pass / fail

`summarizeResult` reads the first Harbor `trial_results[]` entry, or the trial object itself.

| Outcome | Rule |
| --- | --- |
| `passed: true` | No exception **and** at least one numeric reward **and** (`rewards.reward >= 1` when that key exists, otherwise every numeric reward `>= 1`) |
| `passed: false` with `exception` | `exception_info.message`, string `exception`, or a thrown Harbor/process error |
| `passed: false` without exception | Rewards object empty, or a required reward `< 1` |
| Diagnostic zeros | Allowed only when `rewards.reward` is present and `>= 1` (for example `fail_to_pass_exit_code: 0`) |

If Harbor exits and `<jobs>/<jobName>` still has no finished trial, the summary is `Harbor produced no result (exit N)` plus the last 1,000 characters of stderr (or stdout). Spawn failures (Harbor missing from `PATH`) become the same `exception` field.

### Outputs

Progress is printed on **stderr**:

```text
[3/9] gpt-5.6-sol task-a: passed
[4/9] gpt-5.6-terra task-a: failed
[5/9] gpt-5.6-luna task-a: error
```

Status is `passed`, `failed`, or `error` (`exception` set).

<ResponseExample>

```json title="stdout"
{
  "trials": 9,
  "passed": 6,
  "failed": 3
}
```

</ResponseExample>

`<jobs>/summary.json` is rewritten at the end of every successful `runMatrix` call (`schemaVersion` 1):

<ResponseField name="schemaVersion" type="number">
Always `1`.
</ResponseField>
<ResponseField name="agent" type="string">
Always `codex`.
</ResponseField>
<ResponseField name="agentVersion" type="string">
Always `0.146.1`.
</ResponseField>
<ResponseField name="auth" type="string">
`openai-api-key` or `codex-subscription`.
</ResponseField>
<ResponseField name="reasoningEffort" type="string">
Always `high`.
</ResponseField>
<ResponseField name="models" type="string[]">
Models that were scheduled.
</ResponseField>
<ResponseField name="taskCount" type="number">
Discovered Harbor task directories.
</ResponseField>
<ResponseField name="trials" type="object[]">
One object per scheduled pair: `taskId`, `model`, `jobName`, `passed`, `rewards`, optional `exception`.
</ResponseField>

Harbor's own per-job tree is `<jobs>/<jobName>/` (`result.json` plus a trial subdirectory with optional `verifier/test-stdout.txt` and `verifier/test-stderr.txt`).

## Resume completed jobs

Re-run the same `--jobs` directory after a restart. Resume is per deterministic job name, not per export hash.

```mermaid
stateDiagram-v2
  [*] --> InspectJob
  InspectJob --> Reuse: finished trial result.json
  InspectJob --> Archive: missing, unfinished, or finished_at is null
  Archive --> HarborRun: rename to job.incomplete-<timestamp>
  HarborRun --> Summarize: trial result present
  HarborRun --> FailedTrial: no result or spawn error
  Reuse --> Summarize
```

| On disk | Action |
| --- | --- |
| One finished trial (or a single aggregate `trial_results` entry) | Skip Harbor. Summarize the stored trial. |
| Job directory with `finished_at: null` and no usable trial | Treat as missing. `tryReadHarborJobResult` returns nothing. |
| Incomplete or leftover directory | Rename to `<jobName>.incomplete-<Date.now()>` before starting Harbor. |
| No directory | Start Harbor. |
| `<jobs>/tasks/` already has `task.toml` dirs | Skip `--export` extraction. |

Multiple trial subdirectories plus a one-element aggregate `trial_results` fall back to the aggregate. Multiple trial directories without that aggregate throw and fail the process (not recorded as a soft trial exception).

Raw `harbor run` with several `--model` flags does **not** use this resume logic.

## Smoke Harbor agent adapters

`self-bench-agent-smoke` runs Harbor with `--install-only` for every adapter in the pinned Harbor roster (38 names, including `oracle`, `nop`, `codex`, and `pi`). It does not score tasks and does not pass `--model` or Codex `--ak` flags.

```bash
self-bench-agent-smoke \
  --task ./self-bench-tasks/TASK_ID \
  --jobs ./adapter-jobs \
  --environment modal \
  --concurrency 4
```

`--task` must be one Harbor task directory (the folder that contains `task.toml`). `--jobs` is required.

<ParamField body="task" type="string" required>
Expanded Harbor task directory.
</ParamField>
<ParamField body="jobs" type="string" required>
Job root. Reports land in `<jobs>/adapter-smoke/`.
</ParamField>
<ParamField body="harbor" type="string">
Harbor executable. Default: `harbor`.
</ParamField>
<ParamField body="environment" type="string">
`modal` or `docker`. Default: `modal`.
</ParamField>
<ParamField body="concurrency" type="string">
Positive integer of in-flight adapter checks. Default: `4`.
</ParamField>

Each adapter is invoked as:

```bash
harbor run \
  --path <task> \
  --agent <adapter> \
  --env <docker|modal> \
  --job-name install-<adapter>-<8-char-uuid> \
  --jobs-dir <jobs> \
  --install-only \
  --n-concurrent 1 \
  --max-retries 0 \
  --delete \
  --yes \
  --quiet
```

`installed` is `exitCode === 0`. Timeout is one hour. Harbor job names include a random suffix, so **resume is the JSON report**, not the Harbor job directory.

| Path | Meaning |
| --- | --- |
| `<jobs>/adapter-smoke/<adapter>.json` | Per-adapter result. If the file exists, the adapter is not re-run. Written with exclusive create (`wx`). |
| `<jobs>/adapter-smoke/summary.json` | `schemaVersion: 1`, `mode: "install-only"`, `environment`, counts, and the full `results` array |

<ResponseExample>

```json title="stdout"
{
  "adapters": 38,
  "installed": 36,
  "failed": 2
}
```

</ResponseExample>

Each result includes `agent`, `installed`, `exitCode`, `jobName`, and `outputTail` (last 2,000 characters of combined stdout/stderr).

<AccordionGroup>
<Accordion title="Pinned adapter roster">

`oracle`, `nop`, `acp`, `terminus-2`, `claude-code`, `copilot-cli`, `aider`, `cline-cli`, `codex`, `cortex-code`, `cursor-cli`, `gemini-cli`, `antigravity-cli`, `antigravity-sdk`, `rovodev-cli`, `goose`, `grok-build`, `hermes`, `kimi-code`, `kimi-cli`, `langgraph`, `deerflow`, `mini-swe-agent`, `nemo-agent`, `swe-agent`, `opencode`, `mimo`, `openclaw`, `openhands`, `openhands-sdk`, `pi`, `qwen-coder`, `devin`, `trae-agent`, `computer-1`, `eve`, `dspy-rlm`, `vibe`.

</Accordion>
</AccordionGroup>

## Jobs directory layout

After a matrix run that materialized an export:

:::files
matrix-jobs/
├── summary.json
├── tasks/
│   └── TASK_ID/
│       ├── task.toml
│       ├── instruction.md
│       ├── environment/
│       ├── tests/
│       └── solution/
├── task-id-gpt-5.6-sol/
│   ├── result.json
│   └── <trial>/
│       ├── result.json
│       └── verifier/
└── task-id-gpt-5.6-sol.incomplete-<timestamp>/   # only after an interrupted retry
:::

Smoke reports are isolated under `adapter-smoke/` in the same `--jobs` root if you reuse it. Prefer a separate jobs directory so matrix `summary.json` and adapter reports do not share a tree by accident.

## Errors

| Signal | Cause |
| --- | --- |
| `--jobs is required` | Missing jobs directory flag |
| `provide exactly one of exportPath or tasksPath` | Both or neither of `--export` / `--tasks` |
| `found no Harbor tasks in …` | No child directory with `task.toml`, or export `tasks/*.tar.gz` produced none |
| `--model must be one of gpt-5.6-sol, gpt-5.6-terra, gpt-5.6-luna` | Unknown `--model` |
| `--environment must be docker or modal` | Typo or unsupported backend |
| `--concurrency must be a positive integer` | Zero, negative, or non-integer |
| `… does not contain a ChatGPT subscription token set` | Subscription fallback file is missing `auth_mode: "chatgpt"` + `tokens` |
| `Harbor produced no result (exit N)` | Harbor exited without a finished trial (including the 4-hour kill) |
| `invalid Harbor result for task/model` | Trial payload is not an object |
| `expected one Harbor trial result …` | Ambiguous on-disk job layout (process abort) |
| Spawn `harbor` error in `exception` | Executable missing or not runnable |
| `downloaded export failed its SHA-256 integrity check` | `self-bench download` / `--output` only — not `self-bench-eval` |

Harbor `--delete` is always passed. Interrupted Docker networks or address-pool exhaustion are infrastructure failures; see [Troubleshooting](/troubleshooting).

Drop rejected tasks in the review UI and download a cleaned archive before evaluating if the exported set is larger than you want to score. See [Review exports](/review-exports).

## Next

<CardGroup>
<Card title="Companion commands" href="/companion-commands">
Flag lists and stdout contracts for `self-bench-eval`, `self-bench-agent-smoke`, and the other companion binaries.
</Card>
<Card title="Export format" href="/export-format">
Outer `tar.gz` layout, `manifest.json`, per-task `harbor-task` contents, and SHA-256 fields.
</Card>
<Card title="Agent-visible boundary" href="/agent-visible-boundary">
What a coding-agent trial mounts, what stays in `tests/` and `solution/`, and oracle-only gold.
</Card>
<Card title="Execution backends" href="/execution-backends">
Docker versus Modal sandboxes and how Harbor `--env` pairs with the compiled images.
</Card>
<Card title="Authentication" href="/authentication">
`OPENAI_API_KEY` precedence, Codex `auth.json` shape, and Modal tokens.
</Card>
<Card title="Troubleshooting" href="/troubleshooting">
409 export-not-ready, SHA-256 mismatch, Harbor infrastructure errors, leftover Docker resources.
</Card>
</CardGroup>