# Agent-visible boundary

> What a Harbor trial mounts, what stays hidden, gold-only contract artifacts, coupling verdicts, and the held-out-test-only repair constraint.

- Repository: mupt-ai/self-bench
- GitHub: https://github.com/mupt-ai/self-bench
- Human docs: https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc
- Complete Markdown: https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc/llms-full.txt

## Source Files

- `src/coupling.ts`
- `src/codex-review.ts`
- `src/skills/selfbench/SKILL.md`
- `src/harbor-task.ts`
- `docs/task-construction.md`
- `tests/coupling.test.ts`

---

---
title: "Agent-visible boundary"
description: "What a Harbor trial mounts, what stays hidden, gold-only contract artifacts, coupling verdicts, and the held-out-test-only repair constraint."
---

A compiled Harbor task (`schema_version = "1.4"`, `compiler_revision = 23`) splits the workspace into an agent image, a separate verifier image, and a `solution/` tree that Harbor mounts only for the `oracle` agent. A coding-agent trial (`harbor run --agent codex`, or any adapter other than `oracle`) receives a history-free `git archive` of `baseCommit` plus `instruction.md`. It does not receive `tests/test.patch`, `tests/test.sh`, or `solution/gold.patch`.

<Warning>
The run export still contains snapshots, held-out tests, and the reference patch. Those files are operator-visible and unencrypted. They stay hidden from coding-agent trials, not from anyone who can read the archive.
</Warning>

## Trial mounts

`compileHarborTask` writes this Harbor bundle. `environment/Dockerfile` never copies `gold.patch`. `tests/Dockerfile` copies only verifier inputs. `task.toml` sets `[verifier] environment_mode = "separate"`.

:::files
harbor-task/
├── task.toml
├── instruction.md
├── .selfbench-manifest.json
├── environment/
│   ├── Dockerfile
│   └── repo.tar.gz
├── tests/
│   ├── Dockerfile
│   ├── repo.tar.gz
│   ├── test.patch
│   ├── test.sh
│   └── dependency-setup.patch   # only when gold changes a recognized lockfile
└── solution/
    ├── gold.patch
    └── solve.sh
:::

```mermaid
flowchart TB
  subgraph bundle ["harbor-task/"]
    instruction["instruction.md = definition.prompt"]
    envDir["environment/repo.tar.gz + Dockerfile"]
    testsDir["tests/test.patch + test.sh + Dockerfile"]
    solutionDir["solution/gold.patch + solve.sh"]
    toml["task.toml"]
  end

  subgraph agentTrial ["Coding-agent trial --agent other than oracle"]
    agentUser["USER agent, /app"]
    agentSnap["history-free base + setupCommand"]
    agentInst["instruction.md"]
    agentNet["network allowlist: chatgpt.com, openai.com"]
    agentOut["/opt/selfbench/agent.patch"]
  end

  subgraph verifierEnv ["Verifier environment_mode = separate"]
    verRoot["user = root"]
    verTests["/tests/test.patch mode 600"]
    verDep["dependency-setup.patch then git reset"]
    verReward["/logs/verifier/reward.json"]
  end

  subgraph oracleOnly ["harbor run --agent oracle"]
    solve["solve.sh applies /solution/gold.patch"]
  end

  instruction --> agentInst
  envDir --> agentSnap
  envDir --> verRoot
  toml --> agentUser
  toml --> agentNet
  toml --> agentOut
  testsDir --> verTests
  testsDir --> verDep
  testsDir --> verReward
  solutionDir --> solve
```

| Surface | Who sees it | Role |
| --- | --- | --- |
| `instruction.md` | Coding agent, nop, oracle | Standalone engineer request. Compiler writes `definition.prompt` only. |
| `environment/` image | Coding agent, nop | `repo.tar.gz` at `baseCommit`, fresh `git init`, one `base` commit, then `setupCommand`. |
| `tests/` image | Verifier only | Applies held-out tests after collecting `agent.patch`. |
| `solution/` | `oracle` only | `solve.sh` runs `git apply /solution/gold.patch`. Never mounted for coding-agent trials. |
| `task.toml` `[metadata]` | Harbor / operators | `selfbench_task_id`, `difficulty`, `repo`, `base_commit`, `workdir`, `source_pr`, `compiler_revision`. Not copied into `/app` by `environment/Dockerfile`. |
| `.selfbench-manifest.json` | Operators | SHA-256 of definition, test patch, and gold patch. |

Validation uses Harbor's built-in agents, not a coding adapter:

```bash
harbor run --path ./harbor-task --agent nop --env modal --yes
harbor run --path ./harbor-task --agent oracle --env modal --yes
```

A later eval trial uses a coding adapter (`codex` in `self-bench-eval`) against the same bundle. That trial still uses the agent image and `instruction.md` only.

## Agent-visible workspace

The agent image:

1. Extracts `environment/repo.tar.gz` into `/app` (no Git history).
2. Creates a new repo and commits `base`.
3. Runs `setupCommand` from `definition.workdir`.
4. Copies `/app/.git` to root-owned `/opt/selfbench/base.git` (`chmod 700`).
5. Switches to `USER agent`.

`task.toml` then constrains the running agent:

| Field | Value |
| --- | --- |
| `[agent].user` | `agent` |
| `[agent].timeout_sec` | `timeouts.agentSeconds` |
| `[agent].network_mode` | `allowlist` |
| `[agent].allowed_hosts` | `chatgpt.com`, `*.chatgpt.com`, `openai.com`, `*.openai.com` |
| `artifacts` | `/opt/selfbench/agent.patch` |

Harbor collects that artifact with the pinned `base.git`:

```bash
git --git-dir=/opt/selfbench/base.git --work-tree=/app add -A
git --git-dir=/opt/selfbench/base.git --work-tree=/app diff --cached --binary HEAD > /opt/selfbench/agent.patch
```

Image build and verifier networks are `public`. Only the coding-agent runtime uses the OpenAI/ChatGPT allowlist.

## What stays hidden

| Hidden from the coding agent | Where it lives | Why |
| --- | --- | --- |
| Held-out tests | `tests/test.patch` | Grader, not part of the request. |
| Verifier script and selectors | `tests/test.sh` | Expands `{tests}` into `failToPass` / `passToPass`. |
| Reference implementation | `solution/gold.patch` | Oracle only. |
| Gold lockfile preinstall | `tests/dependency-setup.patch` | Verifier image only; source is reset to `HEAD` after setup. |
| Provenance / session records | Authoring sandbox `/work/provenance.json` | Never compiled into the Harbor task. |
| Author and reviewer transcripts | Temporal artifacts under `runs/.../reviews` and `.../repairs` | Excluded from the export. |
| Completed commit and full Git history | Source checkout during authoring | Agent snapshot is `git archive` of `baseCommit`. |

`instruction.md` must not name pull-request numbers or URLs, commit hashes, branch names, test names, test paths, test commands, gold-patch content, or held-out-test content. `definition.json` still stores `sourcePr`, `sourceUrl`, and `baseCommit` for operators; those fields are not copied into `instruction.md`.

<Note>
If gold changes a recognized lockfile (`package.json`, `bun.lock`, `uv.lock`, `go.mod`, `Cargo.lock`, and the other names in `dependencyManifestPatch`), the verifier applies that subset, reruns `setupCommand`, then `git reset --hard` and deletes the patch. That keeps oracle dependencies honest without putting gold source into the agent image.
</Note>

## Held-out test protection at verify time

`tests/test.sh` grades `agent.patch`, not the gold patch:

1. Apply `/opt/selfbench/agent.patch` with `--exclude` for every path in `testPaths` and every path in `test.patch`.
2. `git restore` and `git clean` those paths back to `HEAD`.
3. Apply `/tests/test.patch`, then `chown root` and `chmod a-w` on each held-out path.
4. Delete `/tests/test.patch`.
5. Run fail-to-pass, repeat fail-to-pass (determinism), then pass-to-pass as user `verifier`.

Reward is 1 only when `patch_applied`, `fail_to_pass`, `pass_to_pass`, and `deterministic` are all 1. Validation interprets the same JSON as:

| Gate | Required `reward.json` |
| --- | --- |
| `nop` | `fail_to_pass === 0`, `pass_to_pass >= 1`, `setup_completed >= 1` |
| `oracle` | `patch_applied`, `fail_to_pass`, `pass_to_pass`, `deterministic`, and `setup_completed` all `>= 1` |

An agent that overwrites a held-out test file cannot keep that change. The verifier restores the path and re-applies the hidden patch.

## Gold-only contract artifacts

After nop/oracle pass, `reviewTask` scans added lines in `tests/test.patch` and keeps only artifacts that also appear in gold additions.

<ParamField body="schemaVersion" type="1" required>
`CouplingEvidence` version written into the review report.
</ParamField>

<ParamField body="artifacts[].category" type="endpoint_path | field_name | header_name | media_type" required>
Deterministic classifier. Quoted `/path` tokens are `endpoint_path`. `type/subtype` tokens are `media_type`. Header-like tokens on a `headers` / `content-type` line are `header_name`. Quoted object keys, bare `key:`, and `.member` names of length 3–64 are `field_name`.
</ParamField>

<ParamField body="artifacts[].introducedByGold" type="boolean" required>
Always `true` in emitted evidence. Candidates that never appear in gold additions are dropped.
</ParamField>

<ParamField body="artifacts[].presentInPrompt" type="boolean" required>
Verbatim `includes` for non-fields. `field_name` uses a word-boundary match so `id` does not hit `valid`.
</ParamField>

<ParamField body="artifacts[].presentInBase" type="boolean" required>
Exact string hit from `rg --fixed-strings` over the extracted `environment/repo.tar.gz` (`--hidden`, `--glob !.git/**`).
</ParamField>

<ParamField body="blockers" type="string[]" required>
Every artifact with `!presentInPrompt && !presentInBase`. Message shape: `held-out tests assert gold-only <category> "<artifact>" at <path>:patch-line-N`.
</ParamField>

Fixture literals that never appear in gold additions are ignored. `expect(status).toBe(200)` and `"org_test"` do not become artifacts unless gold introduced them.

```text
added test token
        |
        v
 present in gold additions?
    no --> ignore
   yes --> in prompt or base snapshot?
              yes --> not a blocker
               no --> couplingEvidence.blockers[]
```

The scanner does not invent helper names, SQL, or UI copy that are not quoted or dotted as above. The independent reviewer still has to judge those as `other`.

## Coupling verdicts

`reviewCouplingWithCodex` forces one `submit_review` call on `gpt-5.6-sol` (`COUPLING_REVIEW_MODEL`). Auth is `OPENAI_API_KEY`, or `SELFBENCH_PI_AUTH_JSON` / Pi `openai-codex` access when no API key is set.

<ResponseField name="verdict" type='"clean" | "coupled"'>
Reviewer claim. `resolveCouplingReview` may override `clean` to `coupled`.
</ResponseField>

<ResponseField name="findings[].disposition" type="enum">
`base_contract`, `prompt_contract`, `external_contract`, `gold_only`, or `not_contract`.
</ResponseField>

<ResponseField name="findings[].category" type="enum">
Same four scanner categories plus `other`.
</ResponseField>

<ResponseField name="counterexample" type="string">
Required. Must describe an alternative correct implementation and whether the tests would accept it.
</ResponseField>

`resolveCouplingReview` fails closed:

| Condition | Forced verdict | `reason` |
| --- | --- | --- |
| A blocker artifact has no finding with the exact same `artifact` string | `coupled` | `review did not resolve deterministic coupling evidence for: …` |
| Any finding has `disposition: "gold_only"` | `coupled` | `review identified gold-only artifacts: …` if the reviewer still said `clean` |
| Reviewer `verdict` is `coupled` | `coupled` | Reviewer `reason` |
| All blockers resolved and no `gold_only` | `clean` | Reviewer `reason` |

Allowed justifications for a blocker:

| Disposition | When it is valid |
| --- | --- |
| `prompt_contract` | The authentic request text requires that exact artifact. |
| `base_contract` | The extracted base snapshot already establishes it. A gold mention is not evidence. |
| `external_contract` | The request names a standard protocol that fixes the token independently of gold. |
| `not_contract` | Incidental test-language or framework syntax, not a product contract. |
| `gold_only` | Always `coupled`. |

`reviewTask` writes `runs/<runId>/reviews/<taskId>/<bundleSha12>/attempt-<n>.json` with the resolved verdict, `couplingEvidence`, and `reviewer: "gpt-5.6-sol"`. `accepted` is true only when the resolved verdict is `clean`.

Re-run the same gate on an extracted export:

```bash
self-bench-reaudit --tasks ./expanded-tasks --output ./coupling-review.json
```

The report is `provisional: true` and counts `clean`, `coupled`, and `errors`.

<AccordionGroup>
<Accordion title="Example fail-closed resolution">

A test asserts `routing_strategy` and gold added that field. The prompt says only “use the resolved strategy”. `baseArtifacts` is empty, so the blocker is `field_name "routing_strategy"`.

- Findings `[]` and `verdict: "clean"` → `coupled`, `missingArtifacts: ["routing_strategy"]`.
- Finding `{ artifact: "routing_strategy", disposition: "gold_only" }` → `coupled`.
- Finding `{ artifact: "routing_strategy", disposition: "external_contract" }` with `verdict: "clean"` → `clean` (only if the request actually names a protocol that fixes the field).

</Accordion>
</AccordionGroup>

## Held-out-test-only repair

A `coupled` review schedules **one** `repairTask`. After repair the candidate repeats `auditTask` → `validateTask` → `reviewTask`. A second coupled verdict rejects the candidate. There is no second coupling-repair cycle.

`repairTask` may Temporal-retry the sandbox up to `maximumAttempts: 4`. Exhaustion rejects with `test repair failed after activity retries`. That is infrastructure retry, not a second design pass.

### Coupling repair

The sandbox unpacks the Harbor task, applies the **current** `tests/test.patch` to the base snapshot, and runs Codex against `repairPrompt`.

Allowed edits: paths already present in the original `test.patch` (`assertRepairPaths`).

Forbidden:

- Application / gold paths
- New files outside that path set
- `instruction.md` / `definition.prompt`
- `solution/gold.patch`

Hard failures:

| Error | Cause |
| --- | --- |
| `repair changed files outside the held-out tests: <paths>` | Working tree touched a path not in the original test patch. |
| `repair produced no held-out test patch` | `git diff HEAD` is not a Git patch. |
| `repair left the held-out test patch unchanged` | Bytes equal the input `test.patch`. |
| `original held-out test patch changes no files` | Empty original patch. |

The prompt forbids deleting assertions just to silence the report. If no uncoupled public seam exists inside the allowed files, the agent must leave the tests in place and explain the blocker — that path then fails the unchanged-patch check and the candidate is rejected.

Default model: `SELFBENCH_REPAIR_MODEL` or `gpt-5.6-sol`. Auth: `OPENAI_API_KEY` or `SELFBENCH_CODEX_AUTH_JSON`. Offline companion:

```bash
self-bench-repair \
  --tasks ./expanded-tasks \
  --review ./coupling-review.json \
  --output ./repaired-tasks \
  --model gpt-5.6-sol
```

Only tasks whose review `resolution.verdict` is `coupled` are repaired. Clean tasks are copied unchanged unless `--task` selects one ID.

### Validation repair (harness, still not gold)

A failed nop/oracle gate is a different one-shot activity (`repairValidationTask`, `maximumAttempts: 1`).

It may edit the same held-out test paths **and** only these definition fields: `setupCommand`, `testCommand`, `failToPass`, `passToPass`, `testPaths`, `toolchains`, `timeouts`, `resources`.

It must not change `schemaVersion`, `difficulty`, `taskId`, `repo`, `baseCommit`, `workdir`, `sourcePr`, `sourceUrl`, or `prompt`. Paths must also look like tests (`tests/`, `__tests__`, `e2e`, `*.test.*`, and the other `isTestOnlyPath` patterns). Changing an immutable field throws `validation repair changed immutable definition field <key>`.

## Authoring vs evaluation

The authoring sandbox **does** see provenance, both commits, and the completed diff. That is how `goldPatch` and `testPatch` are split. The authored `prompt` is what later becomes `instruction.md`.

Held-out tests must exercise an existing public API, command, persistence boundary, or extension seam. They must not import gold-only private helpers or pin exact internal SQL, query counts, schema or index names, object identity, telemetry shape, incidental error wording, endpoint/response presentation, or UI copy/order unless the authentic request makes that artifact public. A second correct implementation with different files and helper names must still pass.

## Troubleshooting

| Symptom | Likely cause | What to check |
| --- | --- | --- |
| Review `accepted: false` with `gold-only` | Tests assert a token gold invented | Rewrite assertions onto a public seam, or reject the candidate. |
| `review did not resolve deterministic coupling evidence` | Reviewer omitted a blocker `artifact` string | Re-audit; findings must match blocker tokens exactly. |
| Oracle passes, coding agent still fails | Expected. Hidden tests are the grader. | Confirm `instruction.md` states the required behavior without naming tests. |
| Agent overwrote a test file and still failed | `test.sh` restored and re-applied `test.patch` | Do not treat test files as a writable contract. |
| `repair changed files outside the held-out tests` | Repair touched app code or a new file | Keep edits inside the original `test.patch` paths. |
| `Harbor <agent> infrastructure failure` | Harbor/`nop`/`oracle` crashed before rewards | Retry is automatic for validation; not a coupling verdict. |

## Next

<CardGroup>
<Card title="Validation and repair" href="/validation-and-repair">
Static audit, nop/oracle, determinism, one validation-repair attempt, one coupling-repair retry, then reject.
</Card>
<Card title="Harbor task compiler" href="/harbor-task-compiler">
How patches become `task.toml`, agent/verifier Dockerfiles, toolchain layers, and `dependency-setup.patch`.
</Card>
<Card title="Task definition" href="/task-definition">
`definition.json` fields, `{tests}` rules, and what `submit_task` is allowed to emit.
</Card>
<Card title="Evaluate with Harbor" href="/evaluate-with-harbor">
Extract an export and run coding-agent trials without mounting `solution/`.
</Card>
<Card title="Companion commands" href="/companion-commands">
`self-bench-reaudit` and `self-bench-repair` flags, defaults, and report JSON.
</Card>
<Card title="Export format" href="/export-format">
What the archive includes (tests and gold) and what it excludes (history, provenance, transcripts).
</Card>
</CardGroup>