# Harbor task compiler

> How authored patches become a Harbor task: task.toml schema 1.4, agent and verifier Dockerfiles, toolchain layers, and dependency-setup.patch.

- Repository: mupt-ai/self-bench
- GitHub: https://github.com/mupt-ai/self-bench
- Human docs: https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc
- Complete Markdown: https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc/llms-full.txt

## Source Files

- `src/harbor-task.ts`
- `tests/harbor-task.test.ts`
- `src/contracts.ts`
- `src/temporal/activities.ts`
- `docs/task-construction.md`
- `Dockerfile.sandbox`

---

---
title: "Harbor task compiler"
description: "How authored patches become a Harbor task: task.toml schema 1.4, agent and verifier Dockerfiles, toolchain layers, and dependency-setup.patch."
---

`compileHarborTask` turns one authored directory (`definition.json`, `test.patch`, `gold.patch`) plus a Git checkout into a Harbor-native `harbor-task/` tree. It pins Harbor `schema_version` to `1.4` and SelfBench `compilerRevision` to `23`, writes `task.toml`, builds separate agent and verifier images from `ubuntu:24.04`, and extracts `dependency-setup.patch` only when the gold patch touches a recognized lockfile or package manifest.

Authoring sandboxes invoke the compiler through `sandbox-author`: exactly one task subdirectory under `SELFBENCH_TASK_OUTPUT`, then `tar -czf /work/task.tar.gz -C /work harbor-task`. Temporal stores `definition.json` as a sibling artifact. The Harbor bundle does not include `definition.json`. `refreshHarborTask` rewrites scripts, Dockerfiles, and `task.toml` after validation repair, coupling repair, and export without re-archiving the repository.

<Warning>
`Dockerfile.sandbox` is the authoring sandbox (Node 22, Pi, Codex, `gh`). Harbor trial images are generated by the compiler and start from `ubuntu:24.04`. Do not treat the sandbox image as the agent or verifier base.
</Warning>

## Compile inputs

<ParamField body="authoredDirectory" type="string" required>
Directory that must contain `definition.json` (schemaVersion 1), `test.patch`, and `gold.patch`.
</ParamField>

<ParamField body="repositoryDirectory" type="string" required>
Git checkout that contains `definition.baseCommit` (full 40-character SHA). The compiler runs `git cat-file -e <baseCommit>^{commit}` then `git archive --format=tar.gz`.
</ParamField>

<ParamField body="outputDirectory" type="string" required>
Destination wiped with `rm -rf` then populated as a Harbor task root (later packed as the `harbor-task/` member of a task archive).
</ParamField>

`loadAuthoredTask` parses `definition.json` with `taskDefinitionSchema`, rejects `workdir` / `testPaths` that escape the repository, and requires both patches to start with `diff --git `. It then path-checks **test** hunks only: the test patch must change at least one file, and no `b/` path may be `.git` or escape the repository.

| Input | Required shape | Compiler use |
| --- | --- | --- |
| `definition.json` | schemaVersion `1` | Timeouts, resources, toolchains, `{tests}` command, prompt, workdir, setup |
| `test.patch` | Git patch | Copied to `tests/test.patch`; drives `test.sh` excludes and held-out protection |
| `gold.patch` | Git patch | Copied to `solution/gold.patch`; scanned for dependency-manifest hunks |
| `baseCommit` | Object in the checkout | History-free `repo.tar.gz` snapshot for both images |

<Note>
Gold-patch paths are not escape-checked. Static audit and authoring rules still require gold and test paths to be disjoint; the compiler does not re-check that split.
</Note>

## Output layout

:::files
harbor-task/
├── task.toml
├── instruction.md
├── .selfbench-manifest.json
├── environment/
│   ├── Dockerfile
│   └── repo.tar.gz
├── tests/
│   ├── Dockerfile
│   ├── repo.tar.gz
│   ├── test.patch
│   ├── test.sh
│   └── dependency-setup.patch
└── solution/
    ├── gold.patch
    └── solve.sh
:::

`dependency-setup.patch` is written only when `dependencyManifestPatch(gold.patch)` is non-empty. Both `solve.sh` and `test.sh` are `chmod 0755`. `instruction.md` is `definition.prompt` trimmed plus a trailing newline.

<ResponseField name="generator" type="string">
Always `selfbench`.
</ResponseField>

<ResponseField name="harborSchemaVersion" type="string">
`1.4`, also written as `schema_version` in `task.toml`.
</ResponseField>

<ResponseField name="compilerRevision" type="number">
Integer `23`. Also stored as `[metadata].compiler_revision`.
</ResponseField>

<ResponseField name="taskId" type="string">
Copied from `definition.taskId`. `refreshHarborTask` rejects a definition whose `taskId` does not match the manifest.
</ResponseField>

<ResponseField name="difficulty" type="string">
`easy`, `medium`, or `hard`.
</ResponseField>

<ResponseField name="definitionSha256" type="string">
SHA-256 of `JSON.stringify(definition)` (no pretty-print).
</ResponseField>

<ResponseField name="testPatchSha256" type="string">
SHA-256 of the bytes currently in `tests/test.patch`.
</ResponseField>

<ResponseField name="goldPatchSha256" type="string">
SHA-256 of the bytes currently in `solution/gold.patch`.
</ResponseField>

## Compile versus refresh

```mermaid
flowchart LR
  subgraph authored["Authored directory"]
    def["definition.json"]
    testp["test.patch"]
    goldp["gold.patch"]
  end
  subgraph checkout["Pinned checkout"]
    archive["git archive baseCommit"]
  end
  subgraph bundle["harbor-task/"]
    meta["task.toml + instruction.md + .selfbench-manifest.json"]
    subgraph agentEnv["environment/"]
      adf["Dockerfile USER agent"]
      asnap["repo.tar.gz"]
    end
    subgraph verEnv["tests/"]
      vdf["Dockerfile stays root"]
      vsnap["repo.tar.gz"]
      held["test.patch + test.sh"]
      dep["dependency-setup.patch?"]
    end
    subgraph goldOnly["solution/ oracle only"]
      gp["gold.patch"]
      ss["solve.sh"]
    end
  end
  def --> meta
  testp --> held
  goldp --> gp
  goldp --> dep
  archive --> asnap
  archive --> vsnap
```

| Function | Rebuilds snapshot | Rewrites `instruction.md` / `gold.patch` / `solve.sh` | Rewrites Dockerfiles, `task.toml`, `test.sh`, dependency layer |
| --- | --- | --- | --- |
| `compileHarborTask` | Yes (`git archive`) | Yes | Yes |
| `refreshHarborTask` | No | No | Yes; deletes stale `tests/dependency-setup.patch` when gold no longer touches a manifest |

Repair stages write a new `tests/test.patch` (and sometimes a new definition), then call `refreshHarborTask`. Export unpacks each accepted bundle, refreshes `harbor-task/`, and re-packs that directory as `tasks/<taskId>.tar.gz`.

<Info>
A prompt change in a later definition does not update `instruction.md`. Only a full recompile rewrites the agent-visible instruction.
</Info>

## task.toml schema 1.4

The compiler emits this contract. Harbor then builds `environment/` for the agent and, because `environment_mode = "separate"`, builds `tests/` as the verifier image.

<RequestExample>
```toml title="Compiled task.toml"
schema_version = "1.4"
artifacts = ["/opt/selfbench/agent.patch"]

[task]
name = "selfbench/example"
version = "1.0.0"
description = "Reproduce example from its authentic engineer request."
keywords = ["software-engineering", "private-swe", "selfbench", "medium"]

[metadata]
selfbench_task_id = "example"
difficulty = "medium"
repo = "example/repo"
base_commit = "0123456789abcdef0123456789abcdef01234567"
workdir = "project"
source_pr = 1
compiler_revision = 23

[agent]
timeout_sec = 2400.0
user = "agent"
network_mode = "allowlist"
allowed_hosts = ["chatgpt.com", "*.chatgpt.com", "openai.com", "*.openai.com"]

[verifier]
timeout_sec = 1800.0
user = "root"
environment_mode = "separate"
network_mode = "public"

[[verifier.collect]]
service = "main"
user = "root"
timeout_sec = 300.0
command = "git --git-dir=/opt/selfbench/base.git --work-tree=/app add -A && git --git-dir=/opt/selfbench/base.git --work-tree=/app diff --cached --binary HEAD > /opt/selfbench/agent.patch"

[environment]
network_mode = "public"
build_timeout_sec = 1500.0
cpus = 4
memory_mb = 8192
storage_mb = 20480

[verifier.environment]
network_mode = "public"
build_timeout_sec = 1500.0
cpus = 4
memory_mb = 8192
storage_mb = 20480
```
</RequestExample>

`[task].version` is always `"1.0.0"` (Harbor task version, not the compiler revision). `timeout_sec` and `build_timeout_sec` are written as floats.

| `task.toml` field | Source |
| --- | --- |
| `[agent].timeout_sec` | `timeouts.agentSeconds` |
| `[verifier].timeout_sec` | `timeouts.setupSeconds + timeouts.testsSeconds` |
| `[[verifier.collect]].timeout_sec` | `min(timeouts.testsSeconds, 300)` |
| `[environment].build_timeout_sec` | `timeouts.setupSeconds + 600` |
| `[verifier.environment].build_timeout_sec` | `timeouts.setupSeconds + 600` |
| `cpus` / `memory_mb` / `storage_mb` | `resources.cpus` / `memoryMb` / `storageMb` on both environment tables |

Agent network is hardcoded to the ChatGPT / OpenAI allowlist. Verifier and image builds use `network_mode = "public"` so `setupCommand` and test-time installs can reach package registries.

Collect runs in the **agent** service (`service = "main"`) against `/opt/selfbench/base.git`, which exists only in the agent image. The resulting `/opt/selfbench/agent.patch` is the Harbor artifact the verifier consumes. `solution/` is for the Harbor `oracle` agent (`solve.sh` applies `gold.patch`); coding-agent trials do not mount it.

## Agent and verifier Dockerfiles

Both images share `toolchainDockerfile` + `repositoryDockerfile`, then diverge.

Shared repository layer:

1. `COPY repo.tar.gz` and extract to `/app`.
2. `git init`, identity `selfbench@local` / `selfbench`, commit message `base`.
3. `mkdir -p /opt/uv-cache` (`chmod 777`).
4. `cd /app/<workdir>` and `bash -lc '<setupCommand>'`.
5. If `python` is selected, `uv pip install --python <each .venv/bin/python> 'setuptools>=70' wheel`.

<CodeGroup>
```dockerfile title="environment/Dockerfile (agent)"
# toolchain + repository layers, then:
RUN useradd --create-home --shell /bin/bash agent
RUN git -C /app reset --hard -q HEAD \
    && git -C /app clean -fdq \
    && mkdir -p /opt/selfbench \
    && cp -a /app/.git /opt/selfbench/base.git \
    && chown -R agent:agent /app /home/agent /opt/uv-cache \
    && chown -R root:root /opt/selfbench \
    && chmod 700 /opt/selfbench
ENV UV_CACHE_DIR=/home/agent/.cache/uv \
    UV_NO_BUILD_ISOLATION=1
USER agent
WORKDIR /app
```

```dockerfile title="tests/Dockerfile (verifier)"
# toolchain + repository layers, optional gold dependency layer, then:
RUN useradd --create-home --shell /bin/bash verifier \
    && chown -R verifier:verifier /app /opt/uv-cache \
    && mkdir -p /opt/selfbench \
    && chmod 700 /opt/selfbench
ENV UV_CACHE_DIR=/home/verifier/.cache/uv \
    UV_NO_BUILD_ISOLATION=1
COPY test.patch test.sh /tests/
RUN chmod 700 /tests && chmod 600 /tests/test.patch && chmod +x /tests/test.sh
WORKDIR /app
```
</CodeGroup>

| Concern | Agent image | Verifier image |
| --- | --- | --- |
| Harbor user | `agent` | `root` (tests run via `runuser -u verifier`) |
| Base git for collect | `/opt/selfbench/base.git` (`chmod 700`, root-owned) | Empty `/opt/selfbench` only |
| Gold patch in Dockerfile | Never | Only the filtered `dependency-setup.patch` layer |
| Held-out tests | Absent | `COPY test.patch test.sh /tests/` |
| Build context | `environment/` (`Dockerfile` + `repo.tar.gz`) | `tests/` (`Dockerfile` + `repo.tar.gz` + tests + optional dep patch) |

The agent layer resets `/app` after `setupCommand` so the trial starts from a clean tree while `/opt/uv-cache` keeps warmed packages. The verifier leaves `/app` at the post-setup state (or post gold-dep setup + reset; see below).

## Toolchain layers

`definition.toolchains` is a non-empty list of `uv` | `python` | `node` | `bun` | `go` | `rust`. The compiler installs implied peers: `python` adds `uv`; `bun` adds `node`. Layers always emit in this order: `uv`, `python`, `node`, `bun`, `go`, `rust`.

Base image env on every task:

- `FROM ubuntu:24.04`
- `DEBIAN_FRONTEND=noninteractive`
- `UV_LINK_MODE=copy`, `UV_CACHE_DIR=/opt/uv-cache`, `UV_PYTHON_INSTALL_DIR=/usr/local/share/uv/python`, `UV_PYTHON_BIN_DIR=/usr/local/bin`
- `RUSTUP_HOME=/usr/local/rustup`, `CARGO_HOME=/usr/local/cargo`, `COREPACK_HOME=/opt/corepack`
- `PATH` prepends `/usr/local/go/bin` and `/usr/local/cargo/bin`
- apt: `bash` `build-essential` `ca-certificates` `curl` `git` `jq` `passwd` `pkg-config` `procps` `unzip` `xz-utils`

| Toolchain | Install |
| --- | --- |
| `uv` | `https://astral.sh/uv/install.sh` into `/usr/local/bin` (`UV_NO_MODIFY_PATH=1`) |
| `python` | `uv python install 3.11 3.12 3.13`; `python` and `python3` symlink to 3.12 |
| `node` | Node `v22.14.0` (amd64→`x64`, arm64→`arm64`); `corepack enable`; `PLAYWRIGHT_BROWSERS_PATH=/opt/playwright` |
| `bun` | `npm install --global bun@1.3.14` (after Node) |
| `go` | Go `1.25.0` tarball for `dpkg --print-architecture` |
| `rust` | rustup minimal profile, default toolchain `1.90.0`, no PATH mutation |

`setupCommand` must finish everything `testCommand` needs: frozen package-manager install, native builds, fixture generation, and browser/OS deps (for example `pnpm exec playwright install --with-deps chromium`). The compiler only exposes `PLAYWRIGHT_BROWSERS_PATH`; it does not install browsers.

## dependency-setup.patch

When gold changes a recognized manifest, the compiler copies **only those hunks** into `tests/dependency-setup.patch` and injects a verifier build layer:

```dockerfile
COPY dependency-setup.patch /tmp/selfbench-dependency-setup.patch
RUN git -C /app apply --binary --whitespace=nowarn /tmp/selfbench-dependency-setup.patch \
    && cd /app/<workdir> \
    && bash -lc '<setupCommand>' \
    && git -C /app reset --hard -q HEAD \
    && git -C /app clean -fdq \
    && rm /tmp/selfbench-dependency-setup.patch
```

The image then has gold-state dependencies while `/app` source is back at the base commit. Oracle can apply `gold.patch` on a clean tree without the agent image ever seeing gold. Matching uses the `b/` basename:

| Ecosystem | Basenames |
| --- | --- |
| JavaScript | `package.json`, `package-lock.json`, `npm-shrinkwrap.json`, `yarn.lock`, `pnpm-lock.yaml`, `bun.lock`, `bun.lockb`, `deno.lock` |
| Python | `pyproject.toml`, `uv.lock`, `poetry.lock`, `Pipfile`, `Pipfile.lock`, `requirements*.txt` |
| Go / Rust | `go.mod`, `go.sum`, `Cargo.toml`, `Cargo.lock` |

`src/package-handler.ts` is not a manifest. `apps/api/pyproject.toml` is. Implementation files in the same gold patch are dropped from the extracted patch.

## Verifier script

`tests/test.sh` is generated at compile/refresh time. It does **not** run `setupCommand` (that already ran at image build). Harbor collect must have written `/opt/selfbench/agent.patch` first.

<Steps>
<Step title="Apply the agent artifact">
Missing file → `patch_applied=0`. Empty file (typical nop) → skip apply, leave `patch_applied=1`. Non-empty file → `git apply --binary --whitespace=nowarn` with `--exclude` for every held-out path (union of `workdir`+`testPaths` and `b/` paths in `test.patch`).
</Step>
<Step title="Restore and overlay held-out tests">
`git restore` and `git clean` each protected path from `HEAD`, apply `/tests/test.patch`, then `chown root` and `chmod a-w,go+rX` those paths. Delete `/tests/test.patch` from the running container.
</Step>
<Step title="Run fail-to-pass, then pass-to-pass">
`cd /app/<workdir>`. Copy `/opt/uv-cache` to a verifier-owned temp dir. Substitute `{tests}` with shell-quoted `failToPass` or `passToPass` (empty `passToPass` becomes `true`). Each command runs as `verifier` with `UV_NO_BUILD_ISOLATION=1`. Fail-to-pass is executed twice; `deterministic=1` only if both succeed.
</Step>
<Step title="Write reward.json and exit 0">
Reward is `1` only when `patch_applied`, `fail_to_pass`, `pass_to_pass`, and `deterministic` are all `1`. The script always `exit 0`; Harbor reads `/logs/verifier/reward.json`.
</Step>
</Steps>

`run_verifier_command` retries up to 3 times with `10 * attempt` seconds of backoff when the log matches infrastructure noise (`ECONNRESET`, `ETIMEDOUT`, `ENOTFOUND`, `registry.npmjs`, `FetchError`, and similar). Assertion failures fail immediately. After each attempt it `pkill -KILL` processes owned by `verifier`.

<ResponseExample>
```json title="/logs/verifier/reward.json"
{
  "reward": 1,
  "patch_applied": 1,
  "fail_to_pass": 1,
  "pass_to_pass": 1,
  "deterministic": 1,
  "setup_completed": 1,
  "fail_to_pass_exit_code": 0,
  "fail_to_pass_repeat_exit_code": 0,
  "pass_to_pass_exit_code": 0
}
```
</ResponseExample>

`setup_completed` tracks whether the agent patch stage succeeded; it is not a second setup command. Temporal nop/oracle gates read these numeric fields (nop expects `fail_to_pass=0` and `pass_to_pass>=1`; oracle expects all four success bits).

Oracle solution script:

```bash
#!/bin/bash
set -euo pipefail
git -C /app apply --binary --whitespace=nowarn /solution/gold.patch
```

## Errors

| Message | When |
| --- | --- |
| `test.patch is not a Git patch` / `gold.patch is not a Git patch` | File does not start with `diff --git ` |
| `test.patch changes no files` | No `diff --git a/… b/…` hunks |
| `task path escapes repository: …` | `workdir` or `workdir`+`testPaths` resolves outside the repo root |
| `test patch path escapes repository: …` | A test `b/` path is `.git` or escapes the repo |
| `bundle task <id> does not match <id>` | `refreshHarborTask` manifest `taskId` ≠ definition `taskId` |
| `git cat-file` non-zero | `baseCommit` is not a commit in `repositoryDirectory` |
| Zod parse error | `definition.json` fails `taskDefinitionSchema` (including missing `{tests}`) |
| `authoring must produce exactly one task; found N` | `sandbox-author` saw zero or multiple task directories |

## Troubleshooting

| Symptom | Likely cause |
| --- | --- |
| Oracle fails apply / stale deps after a lockfile bump | Gold changed a manifest but the name is not in the basename list, so no `dependency-setup.patch` layer |
| Agent image contains gold | It must not. `environment/Dockerfile` never copies `gold.patch`. If you see gold in the agent context, the archive is not compiler output |
| Verifier cannot reach npm/PyPI | `setupCommand` or tests need public network; that is enabled on verifier/build, not on the agent allowlist |
| `{tests}` still in `test.sh` | Compile/refresh did not run after the definition change |
| `instruction.md` stale after repair | Refresh does not rewrite it; only `compileHarborTask` does |
| Playwright browsers missing | Not installed by the toolchain layer; they belong in `setupCommand` |
| Harbor infrastructure failure on build | Image build timeout is `setupSeconds + 600`; a long `setupCommand` can exceed it |
| Reward all zeros, `patch_applied=0` | Collect did not produce `/opt/selfbench/agent.patch`, or apply failed (including protected-path excludes) |

Harbor validation and eval invoke `harbor run --path <harbor-task directory>`. The worker uses `--agent nop` then `--agent oracle`. Downstream Codex trials use the same compiled tree.

## Related pages

<CardGroup>
<Card title="Task definition" href="/task-definition">
`definition.json` fields, `{tests}` placeholder rules, and `submit_task` outputs the compiler consumes.
</Card>
<Card title="Agent-visible boundary" href="/agent-visible-boundary">
What a trial mounts versus gold-only `solution/` and held-out tests.
</Card>
<Card title="Export format" href="/export-format">
Outer `tar.gz`, `manifest.json`, and per-task `harbor-task/` archives.
</Card>
<Card title="Validation and repair" href="/validation-and-repair">
Nop/oracle gates that read `reward.json`, then `refreshHarborTask` after a repair.
</Card>
<Card title="Evaluate with Harbor" href="/evaluate-with-harbor">
Run an extracted compiled task with `harbor run` or `self-bench-eval`.
</Card>
</CardGroup>