self-bench Documentation

Pages

1. OverviewWhat self-bench generates, who should run it, the CLI/API/worker surfaces, and the first docs routes after install.
2. InstallationPrerequisites, global install, Modal and GitHub auth, required environment variables, and the first successful CLI help check.
3. QuickstartShortest path: start the Modal-backed local stack, submit a tiered run with --output, extract accepted Harbor tasks, and verify one Codex trial.
4. Provenance sourcesWhere self-bench collects human requests: Pi, Claude Code, and Codex session roots, merged GitHub pull requests, sanitization rules, and the empty-corpus failure.
5. Difficulty gateseasy, medium, and hard eligibility: implementation file and line minima, fail-to-pass and pass-to-pass counts, and static audit blockers.
6. Run lifecycleTemporal phases, per-candidate statuses, eight-shard discovery, candidate budgets versus accepted tasks, and blocked, failed, or cancelled terminals.
7. Validation and repairStatic audit, Harbor nop and oracle gates, determinism rerun, one validation-repair attempt, one coupling-repair retry, then reject.
8. Agent-visible boundaryWhat a Harbor trial mounts, what stays hidden, gold-only contract artifacts, coupling verdicts, and the held-out-test-only repair constraint.
9. Start the local stackself-bench up and down: Compose services, --backend docker or modal, --modal-config, named volumes, and what happens when the worker stops.
10. Build a benchmarkself-bench run: --repo GitHub origin, 1-100 tier budgets, --model, --run-id, HEAD pin, provenance upload, and --output wait plus SHA-256 download.
11. Manage runslist, status, cancel, and download against SELFBENCH_API_URL, including phase polling, create-only writes, and integrity headers.
12. Evaluate with HarborExtract an export, run one task or the Codex matrix via self-bench-eval, resume completed jobs, and smoke Harbor agent adapters.
13. Review exportsOpen the API-hosted review UI or a local archive, inspect accepted tasks, drop tasks, and download a cleaned tar.gz.
14. Deploy Temporal CloudSplit API and worker processes, GCS artifacts, Temporal TLS and API key, Modal tokens, and matching image digest plus task queue.
15. CLI referenceself-bench commands, flags, defaults, run-id rules, wait and download semantics, and command-level errors.
16. Companion commandsFlags, defaults, and outputs for self-bench-eval, self-bench-validate, self-bench-reaudit, self-bench-repair, and self-bench-agent-smoke.
17. HTTP APIRoutes, Bearer auth, status codes, run submission schema, provenance upload limits, export headers, and review-asset serving.
18. Configuration referenceSELFBENCH_* environment keys, defaults, validation, loopback token rule, artifact backends, and Temporal connection fields.
19. Task definitiondefinition.json fields, {tests} placeholder rules, toolchains, timeouts, resources, submit_task output files, and schemaVersion 1 constraints.
20. Harbor task compilerHow authored patches become a Harbor task: task.toml schema 1.4, agent and verifier Dockerfiles, toolchain layers, and dependency-setup.patch.
21. Export formatOuter tar.gz layout, manifest.json fields, per-task harbor-task contents, SHA-256 checks, and what the archive excludes.
22. Execution backendsDocker versus Modal sandboxes: images, default concurrency, resource limits, inactivity timeouts, and Harbor environment pairing.
23. AuthenticationOPENAI_API_KEY precedence, Pi and Codex subscription JSON fallbacks, GH_TOKEN, API Bearer token, and Modal profile versus token secrets.
24. TroubleshootingNo-provenance, blocked candidate pool, 409 export-not-ready, SHA-256 mismatch, Harbor infrastructure failures, and leftover Docker resources.
25. ContributingClone, bun install, bun run validate, source CLI, API/worker/review dev servers, CI, package verification, and npm tag publish.