# Task definition

> definition.json fields, {tests} placeholder rules, toolchains, timeouts, resources, submit_task output files, and schemaVersion 1 constraints.

- Repository: mupt-ai/self-bench
- GitHub: https://github.com/mupt-ai/self-bench
- Human docs: https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc
- Complete Markdown: https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc/llms-full.txt

## Source Files

- `src/contracts.ts`
- `src/extensions/authoring.ts`
- `src/audit.ts`
- `src/skills/selfbench/SKILL.md`
- `src/harbor-task.ts`
- `tests/contracts.test.ts`

---

---
title: "Task definition"
description: "definition.json fields, {tests} placeholder rules, toolchains, timeouts, resources, submit_task output files, and schemaVersion 1 constraints."
---

An authored SelfBench task is one `schemaVersion` `1` `definition.json` plus two Git patches. The authoring tool `submit_task` writes those three files under `$SELFBENCH_TASK_OUTPUT/<taskId>/`. After the call, the sandbox compiles that directory into a Harbor bundle and the worker stores the JSON as `runs/<runId>/authoring/<candidateId>/definition.json`. The compiled `harbor-task/` tree does not include `definition.json`; the compiler copies `prompt` into `instruction.md` and maps timeouts, resources, and toolchains into `task.toml` and the Dockerfiles.

## Authored output

`submit_task` requires `SELFBENCH_TASK_OUTPUT`. It creates `$SELFBENCH_TASK_OUTPUT/<definition.taskId>/` (not recursively) and writes three files with exclusive create (`wx`):

:::files
$SELFBENCH_TASK_OUTPUT/<taskId>/
├── definition.json   # pretty-printed JSON, trailing newline
├── test.patch        # held-out tests; must start with "diff --git "
└── gold.patch        # non-test reference implementation; must start with "diff --git "
:::

The authoring sandbox sets `SELFBENCH_TASK_OUTPUT=/work/tasks`, then copies `/work/tasks/*/definition.json` to `/work/definition.json` and packs `/work/harbor-task` as `/work/task.tar.gz`. Authoring must produce **exactly one** task directory. A second `submit_task` for the same `taskId` fails because the directory and files already exist.

<ParamField body="definition" type="object" required>
`schemaVersion` `1` task object. Extra keys are rejected at `submit_task` (`additionalProperties: false`).
</ParamField>

<ParamField body="testPatch" type="string" required>
Binary-safe Git patch for held-out tests and minimal fixtures. Paths must stay inside the repository and must not overlap any path in `goldPatch`.
</ParamField>

<ParamField body="goldPatch" type="string" required>
Binary-safe Git patch for the implementation required by the prompt. Exclude tests, generated artifacts, formatting churn, and unrelated cleanup.
</ParamField>

<RequestExample>
```json title="submit_task arguments"
{
  "definition": {
    "schemaVersion": 1,
    "difficulty": "medium",
    "taskId": "project-pr-123",
    "repo": "example/project",
    "baseCommit": "0123456789abcdef0123456789abcdef01234567",
    "workdir": ".",
    "setupCommand": "bun install --frozen-lockfile",
    "testCommand": "bun test {tests}",
    "failToPass": ["tests/new-behavior.test.ts"],
    "passToPass": ["tests/existing-a.test.ts", "tests/existing-b.test.ts"],
    "testPaths": ["tests/new-behavior.test.ts"],
    "toolchains": ["node", "bun"],
    "sourcePr": 123,
    "sourceUrl": "https://github.com/example/project/pull/123",
    "prompt": "Implement the requested behavior...",
    "timeouts": {
      "setupSeconds": 900,
      "agentSeconds": 2400,
      "testsSeconds": 900
    },
    "resources": {
      "cpus": 4,
      "memoryMb": 8192,
      "storageMb": 20480
    }
  },
  "testPatch": "diff --git a/tests/new-behavior.test.ts b/tests/new-behavior.test.ts\n...",
  "goldPatch": "diff --git a/src/feature.ts b/src/feature.ts\n..."
}
```
</RequestExample>

<ResponseExample>
```text title="submit_task result"
Submitted project-pr-123.
```
</ResponseExample>

Call `submit_task` exactly once. Do not write an alternate task format.

## schemaVersion 1

`schemaVersion` is the literal integer `1`. That is the only accepted task-definition version. Persistence and compile parse the file with Zod `taskDefinitionSchema`. `submit_task` uses a matching TypeBox object that also locks `schemaVersion` to `1` and rejects unknown keys.

The worker then requires the authored `sourcePr`, `baseCommit`, and `difficulty` to match the assigned candidate. A mismatch rejects the candidate with `authored task does not match its assigned PR, base commit, and difficulty`.

Other `schemaVersion: 1` documents in the repo (export `manifest.json`, coupling reports, eval summaries) are not this object.

## Field reference

| Field | Type | Constraints |
| --- | --- | --- |
| `schemaVersion` | integer | Literal `1` |
| `difficulty` | string | `easy`, `medium`, or `hard` |
| `taskId` | string | `^[A-Za-z0-9][A-Za-z0-9._-]*$` |
| `repo` | string | Non-empty. Free-form label (owner/name or clone URL) |
| `baseCommit` | string | Full 40-character hex SHA |
| `workdir` | string | Non-empty repository-relative path. `.` is valid |
| `setupCommand` | string | Non-empty. Runs from `/app/<workdir>` during image build |
| `testCommand` | string | Must contain the literal `{tests}` placeholder |
| `failToPass` | string[] | At least one selector |
| `passToPass` | string[] | May be empty (easy has no minimum) |
| `testPaths` | string[] | At least one repository-relative held-out path |
| `toolchains` | enum[] | At least one of `uv`, `bun`, `go`, `node`, `python`, `rust` |
| `sourcePr` | integer | ≥ 1 |
| `sourceUrl` | string | Must be a URL when the definition is persisted |
| `prompt` | string | Non-empty standalone instruction |
| `timeouts.setupSeconds` | integer | ≥ 1 |
| `timeouts.agentSeconds` | integer | ≥ 1 |
| `timeouts.testsSeconds` | integer | ≥ 1 |
| `resources.cpus` | number | `> 0` (not required to be an integer) |
| `resources.memoryMb` | integer | ≥ 1 |
| `resources.storageMb` | integer | ≥ 1 |

<ParamField body="taskId" type="string" required>
Directory name under `SELFBENCH_TASK_OUTPUT` and the Harbor task name `selfbench/<taskId>`.
</ParamField>

<ParamField body="baseCommit" type="string" required>
Pinned history-free snapshot. The compiler runs `git cat-file -e <baseCommit>^{commit}` and `git archive` on that commit.
</ParamField>

<ParamField body="workdir" type="string" required>
Working directory relative to the extracted snapshot. The compiler resolves `workdir` and `posix.join(workdir, testPath)` against `/repo` and throws `task path escapes repository` if either leaves the tree. Use `.` or a subdirectory. Do not use `..` or an absolute path such as `/app`.
</ParamField>

<ParamField body="setupCommand" type="string" required>
Frozen, repository-native install and environment prep. Put dependency install, native builds, fixture generation, and browser/runtime install here — not in `testCommand`.
</ParamField>

<ParamField body="failToPass" type="string[]" required>
Selectors that must fail on the base snapshot plus the held-out tests, and pass after the gold implementation. Each value is substituted into `{tests}` as its own shell-quoted argument. Values may be paths or runner-native names.
</ParamField>

<ParamField body="passToPass" type="string[]">
Regression selectors that already pass at the base. An empty list compiles to the verifier command `true`.
</ParamField>

<ParamField body="testPaths" type="string[]" required>
Held-out paths relative to `workdir`. Combined with every path in `test.patch` to mark files the agent patch must not overwrite.
</ParamField>

<ParamField body="prompt" type="string" required>
Copied verbatim (trimmed, plus a trailing newline) to Harbor `instruction.md`. Must not mention the PR, commits, patches, test names, or the hidden implementation.
</ParamField>

## `{tests}` placeholder

`testCommand` is a single repository-native command template. At compile time every `{tests}` occurrence is replaced with the selected selectors, each single-quoted:

```text
bun test {tests}
+ failToPass ["tests/new-behavior.test.ts"]
→ bun test 'tests/new-behavior.test.ts'
```

Zod persistence only requires that the string **contain** `{tests}`. Static audit is stricter and is what accepts or rejects the candidate:

| Rule | Blocker |
| --- | --- |
| `{tests}` appears exactly once | `test command must contain "{tests}" exactly once` |
| The placeholder is not wrapped in matching quotes | `test command must expand "{tests}" as an unquoted shell argument list` |
| No `failToPass` or `passToPass` selector appears elsewhere in the command | `test command must not hard-code fail-to-pass or pass-to-pass paths outside "{tests}"` |

<Warning>
Do not write `bun test "{tests}"`, assign the placeholder to one scalar, chain two equivalent suites (`… {tests} && … {tests}`), or append a selected path beside the placeholder. Use one repository-native runner and bundler. If focused tests cannot be preflighted in the authoring sandbox, reject the candidate instead of guessing `testCommand`.
</Warning>

<CodeGroup>

```bash title="Bun"
bun test {tests}
```

```bash title="npm"
npm test -- {tests}
```

```bash title="pytest"
pytest {tests}
```

</CodeGroup>

Pytest, Go, Rust, and custom runners use their own selector syntax. The compiler does not rewrite the runner; it only substitutes the list.

## Toolchains

`toolchains` selects setup layers on `ubuntu:24.04`. The compiler always adds implied layers:

| Declared | Also installed | Image layer |
| --- | --- | --- |
| `uv` | — | `uv` into `/usr/local/bin` |
| `python` | `uv` | CPython 3.11, 3.12, 3.13 via `uv`; `python`/`python3` point at 3.12 |
| `node` | — | Node.js 22.14.0, Corepack, `PLAYWRIGHT_BROWSERS_PATH=/opt/playwright` |
| `bun` | `node` | `bun@1.3.14` via npm |
| `go` | — | Go 1.25.0 |
| `rust` | — | rustup minimal profile, toolchain 1.90.0 |

Install order is `uv`, `python`, `node`, `bun`, `go`, `rust`. Choose only the toolchains the clean checkout needs.

When `gold.patch` changes a recognized dependency manifest or lockfile, the **verifier** image applies a extracted `dependency-setup.patch`, re-runs `setupCommand`, then resets the tree to the base snapshot. The agent image never sees that layer. Manifest basenames include `package.json`, `package-lock.json`, `bun.lock` / `bun.lockb`, `pnpm-lock.yaml`, `yarn.lock`, `pyproject.toml`, `uv.lock`, `go.mod` / `go.sum`, and `Cargo.toml` / `Cargo.lock`.

For Playwright tests, install browsers and OS deps in `setupCommand` (for example `pnpm exec playwright install --with-deps chromium`). The shared `PLAYWRIGHT_BROWSERS_PATH` is available to the verifier user.

## Timeouts and resources

The schema has **no defaults**. The authoring prompt tells the model to use these values unless the repository needs more:

| Field | Authoring default |
| --- | --- |
| `timeouts.setupSeconds` | `900` |
| `timeouts.agentSeconds` | `2400` |
| `timeouts.testsSeconds` | `900` |
| `resources.cpus` | `4` |
| `resources.memoryMb` | `8192` |
| `resources.storageMb` | `20480` |

The Harbor compiler maps those fields as:

| Definition field | Harbor destination |
| --- | --- |
| `timeouts.agentSeconds` | `[agent] timeout_sec` |
| `timeouts.setupSeconds + timeouts.testsSeconds` | `[verifier] timeout_sec` |
| `min(timeouts.testsSeconds, 300)` | `[[verifier.collect]] timeout_sec` (agent-patch collection only) |
| `timeouts.setupSeconds + 600` | `[environment]` and `[verifier.environment] build_timeout_sec` |
| `resources.cpus` | both environment `cpus` |
| `resources.memoryMb` | both environment `memory_mb` |
| `resources.storageMb` | both environment `storage_mb` |

These limits apply to the compiled Harbor agent and verifier. They are not the authoring-sandbox timeout (`AUTHORING_TIMEOUT_MS` is two hours).

## Static audit beyond the schema

After compile, static audit reads `definition.json` together with both patches. Schema-valid JSON can still fail:

| Check | Requirement |
| --- | --- |
| Patch overlap | No path appears in both `gold.patch` and `test.patch` |
| Gold size | `easy`: ≥ 1 file / 20 changed lines; `medium`: ≥ 2 / 50; `hard`: ≥ 3 / 100 |
| Held-out tests | `test.patch` changes at least one file |
| `passToPass` count | `easy`: 0; `medium`: ≥ 1; `hard`: ≥ 2 |
| `{tests}` | Exactly once, unquoted, no hardcoded selected paths |

Changed-line counts are `+`/`-` hunk lines in the gold patch (`+++`/`---` headers excluded). `failToPass` already has a schema minimum of one.

## What validation repair may change

One validation-repair attempt may edit `/work/definition.json` and existing held-out test files only.

**Immutable:** `schemaVersion`, `difficulty`, `taskId`, `repo`, `baseCommit`, `workdir`, `sourcePr`, `sourceUrl`, `prompt`. Changing any of these throws `validation repair changed immutable definition field <name>`.

**Mutable:** `setupCommand`, `testCommand`, `failToPass`, `passToPass`, `testPaths`, `toolchains`, `timeouts`, `resources`.

Coupling repair does not rewrite the definition. After a successful validation repair, `refreshHarborTask` regenerates `tests/test.sh`, `task.toml`, and the Dockerfiles from the new definition.

## Errors

| Signal | Cause |
| --- | --- |
| `SELFBENCH_TASK_OUTPUT is not configured` | `submit_task` ran without the output root |
| `submit_task received an incomplete task` | Missing `taskId`, `testPatch`, or `goldPatch` |
| Exclusive-create / existing directory | Second submit for the same `taskId` |
| `authoring must produce exactly one task; found N` | Zero or multiple directories under the output root |
| Zod parse failure | Invalid `schemaVersion`, SHA, URL, empty arrays, or missing `{tests}` |
| `test.patch is not a Git patch` / `gold.patch is not a Git patch` | Patch does not start with `diff --git ` |
| `task path escapes repository` | `workdir` or a `testPaths` entry leaves the repo |
| `test.patch changes no files` | Empty or unparseable held-out patch |
| `authored task does not match its assigned PR, base commit, and difficulty` | Identity fields disagree with the candidate |
| Audit blockers joined by `; ` | Size, overlap, or `{tests}` rules failed |

## Next

<CardGroup>
<Card title="Harbor task compiler" href="/harbor-task-compiler">
How `definition.json` and the two patches become `task.toml`, Dockerfiles, and `dependency-setup.patch`.
</Card>
<Card title="Difficulty gates" href="/difficulty-gates">
easy / medium / hard file and line minima, fail-to-pass and pass-to-pass counts.
</Card>
<Card title="Validation and repair" href="/validation-and-repair">
Static audit, Harbor nop/oracle, determinism, and the one validation-repair attempt.
</Card>
<Card title="Agent-visible boundary" href="/agent-visible-boundary">
What a Harbor trial mounts, and which contract artifacts stay gold-only.
</Card>
<Card title="Export format" href="/export-format">
Outer `tar.gz` layout. Compiled tasks ship as `harbor-task/`, not as raw `definition.json`.
</Card>
</CardGroup>