# Validation and repair > Static audit, Harbor nop and oracle gates, determinism rerun, one validation-repair attempt, one coupling-repair retry, then reject. - Repository: mupt-ai/self-bench - GitHub: https://github.com/mupt-ai/self-bench - Human docs: https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc - Complete Markdown: https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc/llms-full.txt ## Source Files - `src/temporal/workflow.ts` - `src/temporal/activities.ts` - `src/audit.ts` - `src/validation-repair.ts` - `src/repair.ts` - `docs/task-construction.md` --- --- title: "Validation and repair" description: "Static audit, Harbor nop and oracle gates, determinism rerun, one validation-repair attempt, one coupling-repair retry, then reject." --- After authoring, `selfBenchRunWorkflow` accepts a candidate only when static audit, Harbor `nop` and `oracle` gates, and coupling review all pass. A Harbor gate failure schedules **one** `repairValidationTask` activity. A coupled review schedules **one** `repairTask` cycle. The next failure at the same gate rejects that candidate and does not replace it. The run phase stays `authoring` while each candidate moves through per-task statuses `auditing` → `validating` → `reviewing` → `accepted`. Both repair kinds set status `repairing`. Harbor infrastructure errors set `infrastructure_failed` instead of entering repair. ## Candidate pipeline ```mermaid stateDiagram-v2 [*] --> auditing: authored auditing --> rejected: static blockers auditing --> validating: audit.accepted validating --> infrastructure_failed: HarborInfrastructureFailure validating --> validation_repair: first nop/oracle failure validating --> rejected: Harbor failure after a repair validating --> reviewing: nop and oracle passed validation_repair --> rejected: activity threw or AuthorOutcome rejected validation_repair --> auditing: harness rewritten reviewing --> coupling_repair: first coupled verdict reviewing --> rejected: coupled after test repair reviewing --> accepted: clean coupling_repair --> rejected: activity retries exhausted or AuthorOutcome rejected coupling_repair --> auditing: tests rewritten ``` A candidate can receive both repairs in order: failed Harbor gates first, then failed coupling review. After `repairTask`, a new Harbor failure does **not** schedule `repairValidationTask`. After either repair, static audit runs again before Harbor. | Gate | Activity | Temporal `startToCloseTimeout` | Temporal `retry.maximumAttempts` | Workflow retries | | --- | --- | --- | --- | --- | | Static audit | `auditTask` | 7 hours | 4 | After each successful repair | | Harbor `nop` + `oracle` | `validateTask` | 7 hours | 4 | Once after validation repair; once after coupling repair | | Validation repair | `repairValidationTask` | 2 hours | **1** | None | | Coupling review | `reviewTask` | 7 hours | 4 | Once after coupling repair | | Coupling repair | `repairTask` | 7 hours | 4 | One scheduled cycle only | `repairValidationTask` is the only activity pinned to a single attempt. `repairTask` may Temporal-retry the sandbox, but the workflow still schedules that cycle only once. Cancellation on either repair activity cancels the run. ## Static audit `auditTask` reads `definition.json`, `solution/gold.patch`, and `tests/test.patch`, then calls `auditTaskDefinition`. The report is stored at `runs//audits//.json`. Rejection reason is the blockers joined with `"; "`. | Check | Rejects when | | --- | --- | | Patch overlap | Any path appears in both gold and held-out test patches | | Implementation files | Gold file count below the tier minimum | | Implementation lines | Gold `+`/`-` lines (excluding `+++`/`---`) below the tier minimum | | Held-out tests | Test patch changes no files | | Pass-to-pass count | `definition.passToPass.length` below the tier minimum | | Hard-coded selectors | `testCommand` with `{tests}` removed still contains a `failToPass` or `passToPass` path | | `{tests}` count | `testCommand` does not contain `{tests}` exactly once | | `{tests}` quoting | `testCommand` matches `(["'])\{tests\}\1` | Tier minima used by audit: | `difficulty` | Gold files | Gold changed lines | `passToPass` length | | --- | --- | --- | --- | | `easy` | 1 | 20 | 0 | | `medium` | 2 | 50 | 1 | | `hard` | 3 | 100 | 2 | `failToPass` must contain at least one selector because `taskDefinitionSchema` requires `.min(1)`. Audit does not re-check that. A passing easy audit with an empty `passToPass` list is valid; the Harbor verifier still needs a `pass_to_pass` reward of `1` (see below). Returned metrics: `implementationFiles`, `implementationChangedLines`, `testFiles`. ## Harbor nop and oracle `validateTask` extracts the Harbor bundle and runs two sequential `harbor run` jobs against `SELFBENCH_HARBOR_ENVIRONMENT`, falling back to `SELFBENCH_EXECUTION_BACKEND` (`docker` by default): ```bash harbor run \ --path \ --agent nop|oracle \ --env docker|modal \ --job-name --<8 hex> \ --jobs-dir /jobs \ --delete --yes --quiet ``` Each job has a 3-hour command timeout. Rewards come from `trial.verifier_result.rewards`. Any `trial.exception_info` fails the gate. Matching infrastructure exceptions (`AuthError` or known Docker/Modal/build failures) throw `HarborInfrastructureFailure` instead of becoming a validation failure. | Agent | Must hold | Meaning | | --- | --- | --- | | `nop` | `fail_to_pass === 0` | Held-out fail-to-pass tests fail on the base snapshot | | `nop` | `pass_to_pass >= 1` | Regression command succeeds (or `true` when `passToPass` is empty) | | `nop` | `setup_completed >= 1` | Verifier reached the test stage | | `oracle` | `patch_applied >= 1` | Gold / agent patch applied | | `oracle` | `fail_to_pass >= 1` | Fail-to-pass tests pass after gold | | `oracle` | `pass_to_pass >= 1` | Regression command succeeds | | `oracle` | `deterministic >= 1` | Fail-to-pass command passed a second time | | `oracle` | `setup_completed >= 1` | Verifier reached the test stage | Both agents must pass. Failure reason format: ```text Harbor gates failed: nop= (); oracle= () nop verifier tail: ... oracle verifier tail: ... ``` Checks logged: `patch_applied`, `fail_to_pass`, `pass_to_pass`, `deterministic`, `setup_completed`, `fail_to_pass_exit_code`, `fail_to_pass_repeat_exit_code`, `pass_to_pass_exit_code`. Verifier tails are truncated to the last 8 KB. Missing numeric rewards count as `0`. The compiled `tests/test.sh` applies an empty `nop` agent patch as a no-op (`-s` requires a non-empty `/opt/selfbench/agent.patch`), reapplies `tests/test.patch`, then runs fail-to-pass and pass-to-pass. An empty `passToPass` list compiles to the command `true`, so `pass_to_pass` can still be `1` on easy tasks. Each verifier command may retry up to three times on registry/network log patterns (`ECONNRESET`, `ETIMEDOUT`, `registry.npmjs`, and similar) with 10s / 20s backoff. Assertion failures fail immediately. A Harbor process non-zero exit or a classified infrastructure exception does not enter validation repair. The candidate status becomes `infrastructure_failed`. Sibling candidates continue; the run can still export whatever was accepted. ## Determinism rerun Determinism is not a third Harbor job. During the **oracle** trial, `tests/test.sh` runs the fail-to-pass command twice: 1. First success sets `fail_to_pass=1` and `fail_to_pass_exit_code=0`. 2. Immediate second success sets `deterministic=1` and `fail_to_pass_repeat_exit_code=0`. 3. Second failure leaves `deterministic=0` and records `fail_to_pass_repeat_exit_code`. `validateTask` requires `deterministic >= 1` on oracle only. `nop` does not check that reward. Flaky fail-to-pass tests therefore fail oracle and, on the first failure, enter validation repair with those diagnostics. ## Validation repair On the first `validation.accepted === false`, status becomes `repairing` and the workflow calls `repairValidationTask` with the failed `ValidationResult`. ### Sandbox The activity mounts the current Harbor bundle plus the original definition and the validation reason, then runs `dist/sandbox-validation-repair.bundle.js`. | Input | Path | | --- | --- | | Task archive | `/work/task.tar.gz` | | Original definition | `/work/original-definition.json` | | Diagnostics | `/work/validation-diagnostics.txt` (the Harbor reason, including bounded verifier tails) | | Repairer | `/work/sandbox-validation-repair.js` | | Output | Path | | --- | --- | | New definition | `/work/repaired-definition.json` | | New held-out patch | `/work/repaired-test.patch` | | Report | `/work/repair-report.json` | Sandbox wall clock is 2 hours. Agent inactivity timeout is 8 minutes. Inside the sandbox, `pi --print` runs for up to 90 minutes with `--tools read,bash,grep,find,ls`, `--thinking high`, and `--no-session`. Provider is `openai` when `OPENAI_API_KEY` is set, otherwise `openai-codex` with `SELFBENCH_PI_AUTH_JSON`. Model is `SELFBENCH_REPAIR_MODEL`, set from `run.authoring.model`. The working tree is the base snapshot from `tests/repo.tar.gz` with the current `tests/test.patch` already applied. `/work/gold.patch` and `/work/test.sh` are present for inspection only. ### Allowed edits `assertValidationRepair` enforces: **Immutable `definition.json` fields:** `schemaVersion`, `difficulty`, `taskId`, `repo`, `baseCommit`, `workdir`, `sourcePr`, `sourceUrl`, `prompt`. **Mutable `definition.json` fields** (named in the repair prompt): `setupCommand`, `testCommand`, `failToPass`, `passToPass`, `testPaths`, `toolchains`, `timeouts`, `resources`. **File paths:** every changed path must already appear in the original held-out test patch, and that original patch must be test-only. Test-only means a path segment matching `tests?`, `__tests__`, `e2e`, `integration`, `fixtures?`, `selfbench`, `selfbench-tests`, `selfbench_tests`, or `.selfbench-tests`, or a filename matching `(^|\.)(test|spec)\.[^.]+$`. Application code, `instruction.md`, and `solution/gold.patch` stay unchanged. Selecting additional existing pass-to-pass identifiers without editing those files is allowed. Leaving both the definition and the test patch unchanged fails the sandbox (`validation repair left the task unchanged`). ### After a successful repair The activity writes the new `tests/test.patch`, calls `refreshHarborTask` (rewrites `tests/test.sh`, `task.toml`, both Dockerfiles, and `dependency-setup.patch`), and stores a checkpoint: ```text runs//validation-repairs/// ├── definition.json ├── harbor-task.tar.gz └── attempt-/ ├── sandbox.log └── report.json ``` If both checkpoint objects already exist, the activity returns them without launching another model. An incomplete pair (definition xor bundle) throws. The workflow then re-runs `auditTask` and `validateTask`. A second Harbor failure rejects with `validation.reason` or `"validation rejected repaired harness"`. There is no second validation-repair. If the activity throws (other than cancellation), the candidate is rejected with `"validation repair failed after its single activity attempt"`. ## Coupling review and test repair ### Review `reviewTask` scans held-out tests for contract artifacts, greps the base snapshot, and runs `dist/sandbox-review.bundle.js` (15-minute sandbox, 5-minute inactivity timeout). The reviewer model constant is `gpt-5.6-sol`. Auth is `OPENAI_API_KEY` or `SELFBENCH_PI_AUTH_JSON`. `resolveCouplingReview` marks the task `coupled` when any of these hold: - the model verdict is `coupled` - a gold-introduced artifact is missing from the review findings - any finding has disposition `gold_only` `accepted` is `resolution.verdict === "clean"`. Reports land at `runs//reviews///attempt-.json`. Gold-only contract artifacts, what the Harbor trial mounts, and the held-out-test-only constraint are specified on [Agent-visible boundary](/agent-visible-boundary). ### Test repair On the first coupled verdict, status becomes `repairing` and the workflow calls `repairTask` with the review artifact. The sandbox runs `codex exec` (`--ephemeral`, `--dangerously-bypass-approvals-and-sandbox`) for up to 90 minutes inside a 2-hour / 8-minute-inactivity sandbox. Auth is `OPENAI_API_KEY` or `SELFBENCH_CODEX_AUTH_JSON`. Model is again `run.authoring.model`. `assertRepairPaths` allows edits only to files already listed in the original `tests/test.patch`. Unlike validation repair, this step cannot change `definition.json`. The sandbox must produce a **different** non-empty Git test patch. Successful output replaces only the Harbor bundle. Checkpoint: ```text runs//repairs// ├── harbor-task.tar.gz └── attempt-/ ├── sandbox.log └── report.json ``` An existing `harbor-task.tar.gz` checkpoint is reused. Manifest metadata records `testPatchSha256` plus `repair.model` and `repair.originalTestPatchSha256`. The workflow then repeats **audit → validate → review**. Any failure rejects: | Second-gate failure | Default reason | | --- | --- | | Audit | `"audit rejected repaired task"` | | Harbor | `"validation rejected repaired task"` | | Review | `"review rejected repaired task"` | | `repairTask` throw after Temporal retries | `"test repair failed after activity retries"` | ## Per-candidate terminals | `TaskProgress.status` | When | | --- | --- | | `rejected` | Authoring, audit, a second Harbor/review failure, or a repair that returns `kind: "rejected"` / throws | | `infrastructure_failed` | `HarborInfrastructureFailure` from `validateTask` after Temporal retries | | `accepted` | Audit, both Harbor agents, and a `clean` review all passed | Rejected and infrastructure-failed candidates are not replaced from the unused discovery pool. The run still exports accepted tasks. `true` only when both Harbor agents pass and the trial has no `exception_info`. `passed`, `result` artifact (`nop.json`), optional `output` artifact (`nop-verifier.log`). `passed`, `result` artifact (`oracle.json`), optional `output` artifact (`oracle-verifier.log`). Present when either agent failed. Passed to the validation-repair prompt as `/work/validation-diagnostics.txt`. Validation artifacts: ```text runs//validation///attempt-/ ├── nop.json ├── oracle.json ├── nop-verifier.log # when verifier output exists └── oracle-verifier.log ``` ## Offline companion commands The Temporal loop is the only path that sequences audit → Harbor → one validation repair → one coupling repair. Companion binaries operate on **already expanded** Harbor task directories. Run Harbor `nop` then `oracle` only. No static audit, no determinism job beyond the oracle reward, no repair. Directory of expanded Harbor tasks (each must contain `task.toml`). Harbor jobs directory. Also receives `validation-summary.json`. Harbor `--env`. Default: `modal` (not the worker's `SELFBENCH_HARBOR_ENVIRONMENT` default). Parallel tasks. Default: `10`. Each Harbor invocation adds `--n-concurrent 1 --max-retries 1` and a 4-hour timeout. ```bash self-bench-validate --tasks ./expanded --jobs ./harbor-jobs --environment modal ``` ```json { "total": 3, "accepted": 2, "rejected": 1, "errors": 0 } ``` A task is accepted when the same reward predicates as `validateTask` hold. Process failures become `status: "error"` entries, not Harbor infrastructure task statuses. Sandboxed coupling review only. Writes a report with `provisional: true`. ```bash self-bench-reaudit --tasks ./expanded --output ./coupling-review.json [--task TASK_ID] [--concurrency 10] ``` Offline **coupling** repair for tasks whose review verdict is `coupled`. Not validation repair. `--output` must be outside `--tasks`. Uncoupled tasks are copied unchanged unless `--task` is set. ```bash self-bench-repair \ --tasks ./expanded \ --review ./coupling-review.json \ --output ./repaired \ [--task TASK_ID] \ [--concurrency 9] \ [--model gpt-5.6-sol] ``` Re-run `self-bench-validate` and `self-bench-reaudit` on `./repaired` yourself. The companion does not repeat audit or Harbor. Full flags live on [Companion commands](/companion-commands). ## Troubleshooting | Symptom | Likely cause | | --- | --- | | Audit: gold/test overlap | Author included implementation paths in `test.patch` | | Audit: `{tests}` quoted or duplicated | `testCommand` must expand `{tests}` once as an unquoted argv list | | `nop` `fail_to_pass !== 0` | New tests already pass on the base snapshot | | `oracle` `fail_to_pass === 0` | Gold does not satisfy the selected fail-to-pass tests, or setup never reached them | | `oracle` `deterministic === 0` | Fail-to-pass is flaky; check `fail_to_pass_repeat_exit_code` and `oracle-verifier.log` | | `pass_to_pass === 0` | Listed regressions failed; an empty list should still yield `1` via `true` | | `setup_completed === 0` / `patch_applied === 0` | Agent patch missing or `git apply` failed (including test-patch reapply) | | `infrastructure_failed` | Harbor CLI crash, Docker/Modal/auth/build exception — not a harness bug | | Validation repair: "left the task unchanged" | Model did not edit allowed tests or mutable definition fields | | Validation repair: "test-only patch paths" | Original `test.patch` already touched product files | | Test repair: "outside the held-out tests" | Model created a new file instead of editing existing test-patch paths | | `"validation rejected repaired harness"` | The single validation-repair attempt did not produce a passing nop/oracle split | | `"review rejected repaired task"` | The single coupling-repair cycle is still `coupled` | `self-bench-validate --environment` defaults to `modal`. A worker using Docker Harbor (`SELFBENCH_EXECUTION_BACKEND=docker` without `SELFBENCH_HARBOR_ENVIRONMENT`) will not match that default. ## Related pages Tier file/line minima and which blockers are static versus Harbor-proven. What a trial mounts, gold-only artifacts, and why repair may edit only held-out tests. How `test.sh` encodes nop, oracle, and the fail-to-pass rerun. Run phases versus per-candidate statuses, including `rejected` and `infrastructure_failed`. `self-bench-validate`, `self-bench-reaudit`, and `self-bench-repair` flags. Export, Harbor infrastructure, and leftover Docker resources.