# Difficulty gates

> easy, medium, and hard eligibility: implementation file and line minima, fail-to-pass and pass-to-pass counts, and static audit blockers.

- Repository: mupt-ai/self-bench
- GitHub: https://github.com/mupt-ai/self-bench
- Human docs: https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc
- Complete Markdown: https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc/llms-full.txt

## Source Files

- `src/audit.ts`
- `src/contracts.ts`
- `src/skills/selfbench/SKILL.md`
- `docs/task-construction.md`
- `tests/audit.test.ts`

---

---
title: "Difficulty gates"
description: "easy, medium, and hard eligibility: implementation file and line minima, fail-to-pass and pass-to-pass counts, and static audit blockers."
---

`auditTaskDefinition` is the mechanical eligibility check for every authored Harbor task. After authoring, and again after either repair, Temporal `auditTask` reads `definition.json`, `solution/gold.patch`, and `tests/test.patch`. A clean report moves the candidate to Harbor nop/oracle validation. Any blocker rejects the candidate with `status: "auditing"` then `status: "rejected"` and `reason` set to the joined blocker list. Tiers `easy`, `medium`, and `hard` are size-and-structure eligibility rules, not measured model-success rates.

<Note>
`--easy-count`, `--medium-count`, and `--hard-count` are authoring budgets. Rejected candidates are not replaced. The export contains only tasks that pass static audit, Harbor validation, and independent review.
</Note>

## Tier thresholds

Thresholds are hardcoded. Discovery assigns the highest honest tier; authoring and repair must keep that assigned `difficulty`.

| `difficulty` | Gold implementation files | Gold changed lines | `failToPass` | `passToPass` |
| --- | ---: | ---: | ---: | ---: |
| `easy` | ≥ 1 | ≥ 20 | ≥ 1 | no minimum |
| `medium` | ≥ 2 | ≥ 50 | ≥ 1 | ≥ 1 |
| `hard` | ≥ 3 | ≥ 100 | ≥ 1 | ≥ 2 |

Every accepted task also needs a held-out test patch with no path overlap against gold, a `testCommand` that expands `{tests}` exactly once as an unquoted argument list, deterministic repository-native setup, and a passing nop/oracle split. Generated or vendored suitability is a review judgment, not a second size classifier. Git LFS, submodules, and service-heavy suites have no special path.

<Warning>
Static audit does **not** count `failToPass.length`. The ≥ 1 fail-to-pass rule is `taskDefinitionSchema` (`failToPass.min(1)`) plus Harbor nop. An empty `failToPass` never reaches `auditTaskDefinition` because definition parse fails first.
</Warning>

## What the size gate counts

The file and line minima apply only to **gold**. Held-out tests have a separate “changes at least one file” check.

| Metric | Source | How it is counted |
| --- | --- | --- |
| `implementationFiles` | `gold.patch` | One entry per `diff --git a/… b/<path>` dest path |
| `implementationChangedLines` | `gold.patch` | Every `+` or `-` line except `+++` / `---` headers |
| `testFiles` | `test.patch` | Same dest-path rule as gold |

Hunk headers, `diff --git` lines, and `---` / `+++` file headers do not count as changed lines. Additions and deletions both count. The authoring skill requires gold to contain only the non-test implementation core; if the author leaves tests, generated files, or cleanup in gold, those paths inflate the mechanical totals.

```text
diff --git a/src/a.ts b/src/a.ts     ← dest path counts as 1 file
--- a/src/a.ts                       ← ignored
+++ b/src/a.ts                       ← ignored
@@ -1,2 +1,3 @@                      ← ignored
-old                                ← 1 changed line
+new                                ← 1 changed line
+extra                              ← 1 changed line
```

GitHub provenance uses a coarser floor **before** discovery: merged non-bot PRs need `additions + deletions >= 20` and `changedFiles >= 1`. That filter is the easy floor on GitHub’s PR totals, not the gold-only audit metrics, and it is not raised for medium or hard. Local Pi, Claude Code, and Codex messages have no line filter; discovery must still reject a tiny implementation core.

## Fail-to-pass and pass-to-pass

| Field | Meaning | Enforced by |
| --- | --- | --- |
| `failToPass` | Selectors that fail on the base snapshot plus held-out tests, and pass after gold | Schema ≥ 1; Harbor nop/oracle |
| `passToPass` | Selectors that already pass on the base and must stay green | Audit length vs tier; Harbor nop/oracle |
| `testPaths` | Held-out files the verifier protects | Schema ≥ 1; not a size gate |
| `{tests}` | Placeholder substituted with the selected selectors | Schema: contains `{tests}`; audit: exactly once, unquoted, no hardcoded selector paths |

For Harbor, empty `passToPass` (legal on `easy`) becomes a no-op `true` in the verifier script. Medium and hard reject that empty list at static audit:

```text
medium mode requires at least 1 pass-to-pass regression tests
hard mode requires at least 2 pass-to-pass regression tests
```

`testCommand` examples:

| Command | Schema | Static audit |
| --- | --- | --- |
| `bun test {tests}` | pass | pass |
| `bun test "{tests}"` | pass | reject — quoted placeholder |
| `bun test {tests} {tests}` | pass | reject — placeholder not exactly once |
| `bun test {tests} && bun test tests/new.ts` | pass | reject if `tests/new.ts` is in `failToPass` or `passToPass` |

## Where the gates fire

```mermaid
stateDiagram-v2
  [*] --> authoring
  authoring --> rejected: no task or difficulty mismatch
  authoring --> auditing: submit_task
  auditing --> rejected: blockers
  auditing --> validating: accepted
  validating --> reviewing: nop and oracle pass
  validating --> repairing: one validation repair
  reviewing --> accepted
  reviewing --> repairing: one coupling repair
  repairing --> auditing: same difficulty
```

| Stage | What is locked | Failure |
| --- | --- | --- |
| Discovery | One `difficulty` per PR: highest honest tier | Shard over the per-tier cap is a hard error; empty pool after waves → run `blocked` |
| Authoring | `definition.difficulty` must equal `candidate.difficulty` | `authored task does not match its assigned PR, base commit, and difficulty` |
| `auditTask` | Gold files/lines, `passToPass` length, disjoint patches, `{tests}` shape | `reason` = `blockers.join("; ")` |
| Validation repair | Cannot change `difficulty`, identity, prompt, gold, or base | Immutable-field error, then reject |
| Coupling repair | Held-out tests only; then full re-audit | Same audit blockers apply to the repaired bundle |

Discovery prompt rule: assign **exactly one** tier and choose the **highest** the separable implementation core meets. A 100-line, 3-file change is hard even if the run still has empty easy slots.

Authoring must reject the candidate instead of retiering or weakening thresholds. `submit_task` writes `definition.json`, `gold.patch`, and `test.patch` once.

## Static audit blockers

`auditTask` stores the JSON report at:

```text
runs/<runId>/audits/<taskId>/<first 12 of bundle sha256>.json
```

<ResponseField name="accepted" type="boolean">
`true` only when `blockers` is empty.
</ResponseField>

<ResponseField name="blockers" type="string[]">
Human-readable reject reasons. Joined with `"; "` onto `TaskProgress.reason`.
</ResponseField>

<ResponseField name="metrics.implementationFiles" type="integer">
Gold dest-path count.
</ResponseField>

<ResponseField name="metrics.implementationChangedLines" type="integer">
Gold `+`/`-` line count excluding `+++`/`---`.
</ResponseField>

<ResponseField name="metrics.testFiles" type="integer">
Held-out dest-path count. `0` is always a blocker.
</ResponseField>

<AccordionGroup>
<Accordion title="Exact blocker strings">

| Condition | `blockers` entry |
| --- | --- |
| Same dest path in gold and tests | `gold and held-out test patches overlap: <paths>` |
| Too few gold files | `<tier> mode requires at least <n> implementation files; found <m>` |
| Too few gold lines | `<tier> mode requires at least <n> changed implementation lines; found <m>` |
| Empty test patch | `held-out test patch changes no files` |
| Too few `passToPass` | `<tier> mode requires at least <n> pass-to-pass regression tests` |
| Selector path appears outside `{tests}` | `test command must not hard-code fail-to-pass or pass-to-pass paths outside "{tests}"` |
| `{tests}` missing or repeated | `test command must contain "{tests}" exactly once` |
| `"{tests}"` or `'{tests}'` | `test command must expand "{tests}" as an unquoted shell argument list` |

</Accordion>
</AccordionGroup>

<ResponseExample>
```json
{
  "accepted": false,
  "blockers": [
    "medium mode requires at least 50 changed implementation lines; found 49"
  ],
  "metrics": {
    "implementationFiles": 2,
    "implementationChangedLines": 49,
    "testFiles": 1
  }
}
```
</ResponseExample>

`self-bench-validate` reruns Harbor nop/oracle only. `self-bench-reaudit` reruns test-to-gold coupling review. Neither command reruns `auditTaskDefinition`.

## Requested budgets

<ParamField body="easy-count" type="integer" default="0">
CLI authoring budget for `easy`. Combined total across tiers must be 1–100.
</ParamField>

<ParamField body="medium-count" type="integer" default="0">
CLI authoring budget for `medium`.
</ParamField>

<ParamField body="hard-count" type="integer" default="0">
CLI authoring budget for `hard`.
</ParamField>

`runRequestSchema.candidateCounts` is the same shape: each tier `0–100`, sum `1–100`. HTTP `POST /v1/runs` accepts `candidateCounts` and Temporal status exposes `requestedByDifficulty`.

```bash
self-bench run \
  --repo /absolute/path/to/repository \
  --easy-count 5 \
  --medium-count 10 \
  --hard-count 5 \
  --output ./self-bench-tasks.tar.gz
```

Discovery expands until each requested tier is filled or the pool is exhausted (`MAX_DISCOVERED_CANDIDATES` is 300). Exhaustion is terminal:

```text
candidate pool exhausted; missing easy=2, medium=0, hard=1
```

The run phase becomes `blocked`. That is a coverage failure, not a size-audit failure.

## Diagnose a rejected candidate

<Steps>
<Step title="Read the per-task reason">
`self-bench status <runId>` (or `GET /v1/runs/<runId>`) lists `tasks[]` with `difficulty`, `status`, and `reason`. Audit rejects look like the blocker strings above. Difficulty-mismatch rejects say the authored task does not match the assigned PR, base commit, and difficulty.
</Step>
<Step title="Compare gold metrics to the assigned tier">
Open the audit JSON for that `taskId`. `metrics.implementationFiles` and `metrics.implementationChangedLines` must meet the assigned `difficulty`, not a lower one. Do not retier the definition.
</Step>
<Step title="Check patch split and testCommand">
Confirm gold and tests share no dest paths, `test.patch` is non-empty, `{tests}` appears once and unquoted, and selected selectors are not repeated in the rest of `testCommand`.
</Step>
<Step title="Do not expect a replacement slot">
A rejected hard candidate does not free a hard budget for another PR. If discovery never found enough honest candidates, the whole run blocks instead of exporting a partial set.
</Step>
</Steps>

## Related pages

<CardGroup>
<Card title="Run lifecycle" href="/run-lifecycle">
Discovery shards, candidate budgets versus accepted tasks, and `blocked` / `rejected` terminals.
</Card>
<Card title="Validation and repair" href="/validation-and-repair">
Harbor nop/oracle, determinism, one validation repair, one coupling repair, then reject.
</Card>
<Card title="Task definition" href="/task-definition">
`definition.json` fields, `{tests}` rules, and `schemaVersion` 1 constraints.
</Card>
<Card title="Provenance sources" href="/provenance-sources">
Session roots, merged GitHub PRs, and the GitHub 20-line / 1-file ingest floor.
</Card>
<Card title="Troubleshooting" href="/troubleshooting">
Empty provenance and blocked candidate-pool failures.
</Card>
</CardGroup>