# Export format

> Outer tar.gz layout, manifest.json fields, per-task harbor-task contents, SHA-256 checks, and what the archive excludes.

- Repository: mupt-ai/self-bench
- GitHub: https://github.com/mupt-ai/self-bench
- Human docs: https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc
- Complete Markdown: https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc/llms-full.txt

## Source Files

- `src/temporal/activities.ts`
- `src/harbor-task.ts`
- `review/src/export-loader.ts`
- `review/src/types.ts`
- `src/artifacts.ts`
- `docs/evaluations.md`
- `docs/task-construction.md`

---

---
title: "Export format"
description: "Outer tar.gz layout, manifest.json fields, per-task harbor-task contents, SHA-256 checks, and what the archive excludes."
---

`buildExport` writes one unencrypted gzip tar after Temporal reaches the `exporting` phase. The archive contains only tasks that finished `accepted`. The worker stores it as `runs/<runId>/export/attempt-<n>/selfbench-<runId>.tar.gz` with `contentType` `application/gzip`. `GET /v1/runs/:runId/export` streams that object; `self-bench download` and `self-bench run --output` write it locally after a streamed SHA-256 check.

<Warning>
The archive includes history-free repository snapshots, held-out tests, and the gold patch. It is not encrypted. Treat it as private benchmark material from a trusted SelfBench deployment.
</Warning>

## Outer archive

The outer tar is created with `tar -czf … -C <root> manifest.json tasks`. The `tasks/` directory is always present, even when `acceptedCount` is `0`.

:::files
selfbench-<runId>.tar.gz
├── manifest.json
└── tasks/
    ├── <taskId>.tar.gz
    └── <taskId>.tar.gz
:::

| Path | Role |
| --- | --- |
| `manifest.json` | UTF-8 JSON, pretty-printed, trailing newline. Schema `1`. |
| `tasks/<taskId>.tar.gz` | One native Harbor task. The filename stem is `taskId`. |

Download names:

| Surface | Filename |
| --- | --- |
| API `Content-Disposition` | `selfbench-<runId>.tar.gz` |
| Review UI cleaned download | `selfbench-<runId>-cleaned.tar.gz` |

`taskId` matches `^[A-Za-z0-9][A-Za-z0-9._-]*$`. The review loader requires a `tasks/<taskId>.tar.gz` entry for every `manifest.tasks[]` row and fails with `export is missing tasks/<taskId>.tar.gz` when one is absent.

## manifest.json

`schemaVersion` is the literal `1`. `acceptedCount` is `tasks.length` after the accepted-only filter. `candidateCounts` is the **requested** authoring budget from the run (`easy` / `medium` / `hard`), not the number of accepted tasks.

<ParamField body="schemaVersion" type="number" required>
Always `1`.
</ParamField>

<ParamField body="runId" type="string" required>
Workflow id: `^[a-z0-9][a-z0-9-]{2,62}$`.
</ParamField>

<ParamField body="candidateCounts" type="object" required>
`{ easy, medium, hard }` integers from the run request (each 0–100, sum 1–100).
</ParamField>

<ParamField body="repository" type="object" required>
`{ url, commit }`. `commit` is the 40-character HEAD pin of the source checkout at submit time. Per-task snapshots use each task’s `baseCommit`, which can differ.
</ParamField>

<ParamField body="version" type="object" required>
Build identity copied from the run request.
</ParamField>

<ParamField body="acceptedCount" type="number" required>
Number of packed task archives.
</ParamField>

<ParamField body="tasks" type="array" required>
`{ taskId, sha256 }[]`. `sha256` is the lowercase hex digest of that inner `tasks/<taskId>.tar.gz` **after** `refreshHarborTask` and re-tar.
</ParamField>

`version` fields:

| Field | Type | Meaning |
| --- | --- | --- |
| `selfbenchCommit` | 40-char SHA | SelfBench image / CLI commit used for the run |
| `executionBackend` | `"docker"` \| `"modal"` | Sandbox backend that authored and validated |
| `sandboxImage` | string | Worker sandbox image |
| `schema` | `1` | Run-request schema |

The run’s `authoring` block (`provider`, `model`, `reasoningEffort`) is **not** copied into `manifest.json`.

<ResponseExample>
```json manifest.json
{
  "schemaVersion": 1,
  "runId": "sb-20260814-example",
  "candidateCounts": { "easy": 10, "medium": 10, "hard": 10 },
  "repository": {
    "url": "https://github.com/acme/payments",
    "commit": "0123456789abcdef0123456789abcdef01234567"
  },
  "version": {
    "selfbenchCommit": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
    "executionBackend": "modal",
    "sandboxImage": "selfbench-sandbox:local",
    "schema": 1
  },
  "acceptedCount": 2,
  "tasks": [
    {
      "taskId": "payments-pr-1842",
      "sha256": "b3c1c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0"
    },
    {
      "taskId": "payments-pr-1901",
      "sha256": "d4e2d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1"
    }
  ]
}
```
</ResponseExample>

## Per-task Harbor archive

Each inner archive is `tar -czf … -C <expanded> harbor-task`. Extract with `--strip-components=1` if you want `task.toml` at the destination root. `self-bench-eval` copies `harbor-task/` to `<jobs>/tasks/<taskId>/`.

Before packing, `buildExport` loads the stored `definition.json` artifact and runs `refreshHarborTask` so `task.toml`, Dockerfiles, `tests/test.sh`, optional `tests/dependency-setup.patch`, and `.selfbench-manifest.json` match the accepted definition (including any validation-repair edits).

:::files
<taskId>.tar.gz
└── harbor-task/
    ├── task.toml
    ├── instruction.md
    ├── .selfbench-manifest.json
    ├── environment/
    │   ├── Dockerfile
    │   └── repo.tar.gz
    ├── tests/
    │   ├── Dockerfile
    │   ├── repo.tar.gz
    │   ├── test.patch
    │   ├── test.sh
    │   └── dependency-setup.patch   # only when gold changes a lockfile/manifest
    └── solution/
        ├── gold.patch
        └── solve.sh
:::

| Path | Packed? | Notes |
| --- | --- | --- |
| `task.toml` | always | Harbor `schema_version` `"1.4"`. Name `selfbench/<taskId>`. |
| `instruction.md` | always | Trimmed `definition.prompt` plus a trailing newline. |
| `.selfbench-manifest.json` | always | Compiler sidecar; not a Harbor input. |
| `environment/Dockerfile` | always | Agent image. Does not copy `gold.patch`. |
| `environment/repo.tar.gz` | always | `git archive` of `definition.baseCommit`. |
| `tests/Dockerfile` | always | Verifier image. Copies `test.patch` and `test.sh`. |
| `tests/repo.tar.gz` | always | Same snapshot bytes as `environment/repo.tar.gz`. |
| `tests/test.patch` | always | Held-out tests. Must start with `diff --git `. |
| `tests/test.sh` | always | Substitutes `{tests}`; mode `0755`. |
| `tests/dependency-setup.patch` | optional | Gold hunks for recognized lockfiles/manifests only. Removed on refresh when gold no longer changes them. |
| `solution/gold.patch` | always | Reference implementation. Harbor mounts `solution/` only for the oracle agent. |
| `solution/solve.sh` | always | `git apply` of `gold.patch`; mode `0755`. |
| `definition.json` | **not packed** | Stored as a separate run artifact and used at refresh time. The review UI shows `Missing definition.json` unless a later packer added one. |

`.selfbench-manifest.json` fields:

| Field | Meaning |
| --- | --- |
| `generator` | `"selfbench"` |
| `harborSchemaVersion` | `"1.4"` |
| `compilerRevision` | Integer written by the compiler (currently `23`) |
| `taskId` | Must match the accepted definition |
| `difficulty` | `easy` \| `medium` \| `hard` |
| `definitionSha256` | SHA-256 of `JSON.stringify(definition)` |
| `testPatchSha256` | SHA-256 of `tests/test.patch` |
| `goldPatchSha256` | SHA-256 of `solution/gold.patch` |

`refreshHarborTask` refuses a bundle whose sidecar `taskId` does not match the definition.

`repo.tar.gz` is a `git archive --format=tar.gz` of the task base commit. It has no `.git` directory. The agent and verifier Dockerfiles unpack it and create a fresh repo with a single `base` commit.

## SHA-256 checks

Digests are lowercase hex SHA-256 from Node `crypto`. They detect accidental corruption. They are not signatures.

```mermaid
flowchart TB
  subgraph Store["Artifact store"]
    Put["putFile computes sha256 and sizeBytes"]
    Get["get / openRead reject digest or size mismatch"]
  end
  subgraph HTTP["GET /v1/runs/:runId/export"]
    H1["content-type from ArtifactRef"]
    H2["content-length = sizeBytes"]
    H3["x-content-sha256 = sha256"]
  end
  subgraph Clients["Clients"]
    CLI["self-bench download / --output"]
    Review["Review UI loadExport"]
    Eval["self-bench-eval --export"]
  end
  subgraph Interior["Inside the tar"]
    Inner["manifest.tasks[].sha256 of each inner tar.gz"]
  end
  Put --> Get --> H1
  Get --> H2
  Get --> H3
  H3 --> CLI
  HTTP --> Review
  CLI -->|"does not read"| Inner
  Review -->|"does not verify"| Inner
  Eval -->|"lists tasks/*.tar.gz; ignores manifest"| Inner
```

### Outer archive (authoritative for download)

The store records `{ uri, sha256, sizeBytes, contentType }` on `putFile`. Local and GCS reads recompute the digest and byte length and throw `artifact integrity check failed: <uri>` on mismatch.

When `status.export` is present, the API streams that object with:

| Header | Value |
| --- | --- |
| `content-type` | `application/gzip` |
| `content-length` | `status.export.sizeBytes` |
| `content-disposition` | `attachment; filename="selfbench-<runId>.tar.gz"` |
| `x-content-sha256` | `status.export.sha256` |

`self-bench download RUN_ID OUTPUT.tar.gz` and `self-bench run --output OUTPUT.tar.gz`:

1. `GET /v1/runs/<runId>/export` (Bearer `SELFBENCH_API_TOKEN` when set).
2. Require `x-content-sha256` and a response body. Missing header → `SelfBench API returned an export without integrity metadata`.
3. Create `OUTPUT` with exclusive `wx` (will not overwrite an existing file).
4. Hash the stream while writing.
5. On digest mismatch, delete `OUTPUT` and throw `downloaded export failed its SHA-256 integrity check`.
6. On success, print `{ "runId", "output" }`.

`--output` implies `--wait`. Download starts only after phase `complete`.

The review UI fetches the same route and gunzips the blob. It does not re-hash against `x-content-sha256`. Integrity for that path is the store check on `openRead`.

### Per-task digests (manifest only)

Each `tasks[].sha256` is `sha256(readFile(tasks/<taskId>.tar.gz))` after refresh. No current consumer (`loadExport`, `self-bench-eval`, Harbor) checks those values. A cleaned review archive copies the original inner bytes and keeps the original per-task digests; it only rewrites `acceptedCount` and the `tasks` array.

Because `manifest.json` lives inside the same tar as the files it names, a rewritten archive can update both together. Extract only from a SelfBench API you trust, then keep the file private.

## What the archive excludes

The export is accepted Harbor tasks plus the run pin. Everything else stays in the artifact store or never left the authoring sandbox.

| Excluded | Where it actually lives |
| --- | --- |
| Rejected, blocked, or `infrastructure_failed` candidates | Run status `tasks[]`, not the tar |
| Git history, remotes, worktrees, uncommitted files | Snapshots are `git archive` of `baseCommit` only |
| Local Pi / Claude Code / Codex sessions | Sanitized JSONL at `runs/<runId>/input/provenance.jsonl` |
| Per-candidate provenance JSON | `runs/<runId>/provenance/<candidateId>.json` |
| Author, reviewer, and repair transcripts / sandbox logs | `runs/<runId>/authoring/`, `reviews/`, `validation/`, `*-repairs/` |
| Stored `definition.json` | `runs/<runId>/authoring/<candidateId>/definition.json` (and repair checkpoints) |
| Static audit and coupling-review reports | `runs/<runId>/audits/`, `runs/<runId>/reviews/` |
| Harbor nop / oracle job artifacts | Validation attempt prefixes |
| Source transcripts inside gold or test patches | Authoring skill forbids them; not a separate export file |

`candidateCounts` in the manifest can be larger than `acceptedCount`. Rejected candidates are not replaced and are not packed.

## Extract a task

<Steps>
<Step title="Download a verified archive">
```bash
self-bench download "$RUN_ID" ./self-bench-evals.tar.gz
# or: self-bench run --repo /path/to/repo --easy-count 1 --output ./self-bench-evals.tar.gz
```
Confirm the CLI printed `{ "runId", "output" }` and the file exists.
</Step>
<Step title="Unpack the outer tar">
```bash
mkdir -p ./export
tar -xzf ./self-bench-evals.tar.gz -C ./export
jq '.acceptedCount, .tasks[0].taskId' ./export/manifest.json
```
</Step>
<Step title="Unpack one Harbor task">
```bash
TASK_ID="$(jq -r '.tasks[0].taskId' ./export/manifest.json)"
mkdir -p "./selected-task"
tar -xzf "./export/tasks/${TASK_ID}.tar.gz" -C ./selected-task
test -f ./selected-task/harbor-task/task.toml
```
Point Harbor at `./selected-task/harbor-task`.
</Step>
</Steps>

<CodeGroup>
```bash extract-all-for-harbor
mkdir -p ./export ./self-bench-tasks
tar -xzf ./self-bench-evals.tar.gz -C ./export
for archive in ./export/tasks/*.tar.gz; do
  task_id="$(basename "$archive" .tar.gz)"
  mkdir -p "./self-bench-tasks/${task_id}"
  tar -xzf "$archive" --strip-components=1 -C "./self-bench-tasks/${task_id}"
done
```

```bash self-bench-eval
self-bench-eval --export ./self-bench-evals.tar.gz --jobs ./matrix-jobs
```
</CodeGroup>

`self-bench-eval --export` extracts every `tasks/*.tar.gz` that contains `harbor-task/task.toml`. It does not read `manifest.json` and does not compare per-task SHA-256. If `<jobs>/tasks/` already has task directories, it reuses them and does not unpack again.

The review loader is stricter: missing `manifest.json` throws `export is missing manifest.json`, and extra inner archives that are not listed in `manifest.tasks` are ignored.

## Cleaned review archives

`buildCleanExport` writes a new outer gzip tar:

1. Drop rows whose `taskId` is marked for removal.
2. Set `acceptedCount` to the kept count.
3. Copy kept `tasks/<taskId>.tar.gz` bytes unchanged.
4. Leave `runId`, `candidateCounts`, `repository`, and `version` as they were.

It does not re-run `refreshHarborTask` and does not recompute inner SHA-256 values.

## Errors

| Signal | Cause |
| --- | --- |
| `409` `{ "error": "run export is not ready" }` | Status has no `export` yet (`queued`, `discovering`, `authoring`, `exporting`, or a failed/blocked/cancelled terminal). |
| `SelfBench API returned an export without integrity metadata` | Response lacked `x-content-sha256` or a body. |
| `downloaded export failed its SHA-256 integrity check` | Stream digest ≠ header. Destination file is deleted. |
| `EEXIST` from `open(…, "wx")` | Output path already exists. Choose a new path. |
| `export is missing manifest.json` | Review UI / `loadExport` on a non-export gzip. |
| `export is missing tasks/<taskId>.tar.gz` | Manifest row with no matching inner archive. |
| `artifact integrity check failed: <uri>` | Stored object bytes no longer match the ArtifactRef. |
| `bundle task <id> does not match <taskId>` | Export refresh: sidecar `taskId` ≠ stored definition. |

Compare a local file to the status object when you did not use the CLI:

```bash
# status.export.sha256 from GET /v1/runs/:runId
shasum -a 256 ./self-bench-evals.tar.gz
```

## Related pages

<CardGroup>
<Card title="Evaluate with Harbor" href="/evaluate-with-harbor">
Extract an export and run one task or the Codex matrix.
</Card>
<Card title="Review exports" href="/review-exports">
Inspect accepted tasks, drop rows, and download a cleaned tar.gz.
</Card>
<Card title="Harbor task compiler" href="/harbor-task-compiler">
How patches become `task.toml`, Dockerfiles, and `dependency-setup.patch`.
</Card>
<Card title="Task definition" href="/task-definition">
`definition.json` fields used at export refresh time.
</Card>
<Card title="Manage runs" href="/manage-runs">
Download, wait, and integrity headers on `SELFBENCH_API_URL`.
</Card>
<Card title="Troubleshooting" href="/troubleshooting">
`409` export-not-ready and SHA-256 mismatch.
</Card>
</CardGroup>