# Provenance sources

> Where self-bench collects human requests: Pi, Claude Code, and Codex session roots, merged GitHub pull requests, sanitization rules, and the empty-corpus failure.

- Repository: mupt-ai/self-bench
- GitHub: https://github.com/mupt-ai/self-bench
- Human docs: https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc
- Complete Markdown: https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc/llms-full.txt

## Source Files

- `src/provenance.ts`
- `src/github.ts`
- `src/cli.ts`
- `src/extensions/discovery.ts`
- `tests/provenance.test.ts`
- `docs/task-construction.md`

---

---
title: "Provenance sources"
description: "Where self-bench collects human requests: Pi, Claude Code, and Codex session roots, merged GitHub pull requests, sanitization rules, and the empty-corpus failure."
---

`self-bench run` collects the human-request corpus on the operator machine, sanitizes it, and uploads NDJSON to `POST /v1/provenance?runId=...` before the Temporal workflow starts. Discovery later ranks candidates only from that uploaded corpus. A model-authored prompt may restate a retained request; it may not invent behavior from the implementation or tests.

The CLI concatenates two collectors, then fails closed if both return nothing:

1. Local user turns from Pi, Claude Code, and Codex session trees under `$HOME` that mention the `--repo` checkout or one of its Git worktrees.
2. Exact titles and optional bodies from up to 500 merged, non-bot GitHub pull requests that clear a coarse size floor.

```mermaid
flowchart LR
  subgraph Operator["Operator machine"]
    HOME["$HOME session roots<br/>.pi / .claude / .codex"]
    GH["gh pr list --state merged --limit 500"]
    CLI["self-bench run"]
    HOME --> CLI
    GH --> CLI
    CLI -->|0 messages| FAIL["Error: no sanitized local-session<br/>or GitHub pull-request provenance was found"]
  end
  subgraph API["API artifact store"]
    UP["POST /v1/provenance"]
    STORE["runs/{runId}/input/provenance.jsonl"]
    UP --> STORE
  end
  subgraph Worker["Discovery sandbox"]
    SHARD["8 shards: index % 8"]
    FILE["/work/provenance.jsonl"]
    BIND["Match sourceType + sessionId + messageIndex"]
    SHARD --> FILE --> BIND
  end
  CLI -->|"application/x-ndjson ≤ 100 MiB"| UP
  STORE --> SHARD
```

## Collection during `self-bench run`

`self-bench run --repo PATH` pins `origin` and `HEAD`, then collects provenance locally. Uncommitted files are not part of the pin. `origin` must be `https://github.com/owner/repo` or `git@github.com:owner/repo` (optional `.git`). SSH remotes are rewritten to HTTPS before GitHub collection.

<ParamField body="--repo" type="string" required>
Git checkout whose `origin` is a GitHub URL. Worktree paths from `git worktree list --porcelain` become the local-session association needles.
</ParamField>

<ParamField body="HOME" type="string">
Session scan root. The CLI uses `process.env.HOME`, then `os.homedir()`. Tests and CI that set `HOME` only see sessions under that directory.
</ParamField>

Local collection and repository resolution run in parallel. GitHub collection runs next, using the resolved origin. The uploaded corpus is local messages followed by GitHub messages.

<RequestExample>
```bash
self-bench run \
  --repo /absolute/path/to/repository \
  --easy-count 5 \
  --medium-count 10 \
  --hard-count 5 \
  --output ./self-bench-tasks.tar.gz
```
</RequestExample>

<ResponseExample>
```json
{
  "runId": "sb-20260814153000-a1b2c3d4",
  "provenanceMessages": 42,
  "localProvenanceMessages": 18,
  "githubPullRequestMessages": 24
}
```
</ResponseExample>

`provenanceMessages` is the uploaded count after sanitization. It is not an accepted-task count. Discovery, authoring, validation, review, and audit run remotely after this upload.

<Note>
The CLI is the supported collector. A direct `POST /v1/provenance` stores whatever NDJSON you send; the API does not re-run session scans or the empty-corpus check.
</Note>

## Local session roots

`collectRepositoryProvenance` walks these directories under `$HOME`, recursively, and reads only `*.jsonl` files. Missing roots are skipped.

| `sourceType` | Root under `$HOME` | Session id when present in the file |
| --- | --- | --- |
| `pi` | `.pi/agent/sessions` | record `type=session` field `id` |
| `claude-code` | `.claude/projects` | record field `sessionId` |
| `codex` | `.codex/sessions` | `session_meta.payload.id` |
| `codex` | `.codex/archived_sessions` | same Codex parser |

If a file has no embedded session id, the CLI uses the filename (`basename`).

A file is kept only when its path **or** raw contents include a worktree path from the `--repo` checkout, or that path with `/` replaced by `-`. The dash encoding matches Claude Code project directories such as `-Users-you-src-project`. A Codex or Pi transcript that merely mentions the absolute worktree path in a user turn is also kept.

Each matching file is parsed as JSON, a `{ "messages": [...] }` wrapper, or JSONL. The collector then keeps **user** turns only:

| Format | How user text is read |
| --- | --- |
| `codex` | `event_msg` payloads with `user_message`. If none exist, `response_item` messages with `role=user`. |
| `claude-code` | records with `type` `user` or `assistant` and a nested `message`. Records that set `sourceToolAssistantUUID` are dropped (tool-echoed user turns). |
| `pi` | records with `type=message` and a nested `message`. |
| `generic` | top-level `role`/`content`, else nested `message`. |

The CLI tags files by the root they came from (`pi`, `claude-code`, or `codex`). `generic` is accepted later by `submit_discovery`; the CLI session walk does not emit it.

`messageIndex` counts only retained user turns in that session (0-based). Empty strings after trim and redaction are dropped.

## Merged GitHub pull requests

`collectGitHubPullRequestProvenance` runs local `gh`:

```bash
gh pr list \
  --repo owner/repo \
  --state merged \
  --limit 500 \
  --json number,title,body,url,author,isDraft,additions,deletions,changedFiles
```

`owner/repo` is the lowercased GitHub repository from `origin`. `gh` must be on `PATH` and authenticated with read access. A non-zero `gh` exit is a hard CLI error (`gh exited N: ...`), not the empty-corpus message.

A listed PR becomes one corpus record when **all** of the following hold:

| Check | Rule |
| --- | --- |
| Draft | `isDraft` is not `true` |
| Author | `author.login` is a string, `author.is_bot` is not `true`, and `login` does not end with `[bot]` (case-insensitive) |
| Identity | positive `number`, non-empty `url`, non-empty trimmed `title` |
| Size floor | `additions + deletions >= 20` and `changedFiles >= 1` |
| URL bind | `url` is `https://github.com/owner/repo/pull/{number}` for the same repository |

This size floor is the easy-tier line minimum only. Discovery still assigns `easy` / `medium` / `hard` from the implementation core.

Record content:

- Body present and `length <= 12000` after trim → `{title}\n\n{body}`
- Missing body, or body longer than 12 000 characters → **title only** (the body is omitted, not truncated)

```json
{
  "sourceType": "github-pull-request",
  "sessionId": "github:example/project#42",
  "messageIndex": 0,
  "content": "Add the public routing API\n\nExpose routing through the documented client.",
  "sourcePr": 42,
  "sourceUrl": "https://github.com/example/project/pull/42"
}
```

HTTPS and SSH origins both bind to the same lowercased `owner/repo`. A PR URL from another repository, or a number that does not match the URL, is rejected.

## Record shape

Every uploaded line is one JSON object:

<ResponseField name="sourceType" type="string">
`pi` | `claude-code` | `codex` | `generic` | `github-pull-request`
</ResponseField>

<ResponseField name="sessionId" type="string">
Local session id, filename fallback, or `github:{owner/repo}#{number}`.
</ResponseField>

<ResponseField name="messageIndex" type="integer">
Non-negative index of the retained user turn. GitHub records always use `0`.
</ResponseField>

<ResponseField name="content" type="string">
Sanitized request text. Discovery copies this onto the candidate; it does not keep the discovery agent's `request` string.
</ResponseField>

<ResponseField name="sourcePr" type="integer">
GitHub records only. Positive pull-request number.
</ResponseField>

<ResponseField name="sourceUrl" type="string">
GitHub records only. Canonical `https://github.com/owner/repo/pull/{n}` URL.
</ResponseField>

Local and GitHub records are not cross-deduplicated. The same wording can appear once as a session turn and again as a PR title/body.

## Sanitization

User text is passed through `redactSecrets`, then dropped if it looks like injected harness context.

Dropped when the trimmed string **starts with** any of:

| Prefix | Typical source |
| --- | --- |
| `# AGENTS.md instructions` | injected agent instructions |
| `# Review Guidelines` | review boilerplate |
| `<environment_context>` | host environment dump |
| `<permissions instructions>` | permission preamble |
| `<collaboration_mode>` | collaboration preamble |
| `<skills_instructions>` | skill injection |
| `<apps_instructions>` | app injection |
| `<plugins_instructions>` | plugin injection |
| `<skill name=` | inlined skill body |
| `Base directory for this skill:` | skill path banner |
| `## Memory` | memory block |

Redaction replacements (not a general secret scanner):

| Pattern | Replacement |
| --- | --- |
| PEM private-key block | `[REDACTED PRIVATE KEY]` |
| `Authorization: Bearer …` | `Authorization: Bearer [REDACTED]` |
| `AKIA` / `ASIA` + 16 alphanumerics | `AWS_ACCESS_KEY_ID=[REDACTED]` |
| `AWS_SECRET_ACCESS_KEY=` / `:` value | `AWS_SECRET_ACCESS_KEY=[REDACTED]` |
| `npm_` + 16+ alphanumerics | `npm_[REDACTED]` |
| `glpat-` + 16+ token chars | `glpat-[REDACTED]` |
| `password` / `passwd` / `pwd` assignment | `password=[REDACTED]` |
| `database_url` / `db_url` assignment | `DATABASE_URL=[REDACTED]` |
| `postgres(ql)`, `mysql`, `mariadb`, `mongodb(+srv)`, `redis` URLs | `[REDACTED DATABASE URL]` |
| `dari_` / `sk-` + 16+ token chars | prefix + `[REDACTED]` |
| `ghp_` / `github_pat_` + 16+ token chars | `github_[REDACTED]` |

Local messages are then deduplicated on `sourceType`, `sessionId`, `messageIndex`, and `content`.

<Warning>
Redaction is a fixed regex list. Session files and PR bodies can still contain secrets that do not match these patterns. Treat the uploaded corpus as sensitive run input, not as scrubbed public data.
</Warning>

## Upload

:::endpoint POST /v1/provenance
Store the sanitized corpus for a run.

**Query**

| Name | Type | Constraint |
| --- | --- | --- |
| `runId` | string | `^[a-z0-9][a-z0-9-]{2,62}$` |

**Request**

- `Content-Type: application/x-ndjson`
- Body: one JSON object per line
- Limit: **100 MiB** (`100 * 1024 * 1024`). Larger bodies fail with `request body exceeds … bytes` (HTTP 500). Other API JSON routes use a 10 MiB default.

**Auth**

`Authorization: Bearer $SELFBENCH_API_TOKEN` when the API token is configured. Unauthenticated if the token is unset.

**Storage key**

`runs/{runId}/input/provenance.jsonl`

**Success:** `201` with an artifact ref (`uri`, `sha256`, `sizeBytes`, `contentType`). `self-bench run` posts that ref on `POST /v1/runs`.
:::

Invalid `runId` is HTTP 400. The API does not parse or validate line schema at upload time.

## How discovery uses the corpus

The workflow splits the uploaded lines across **eight** shards (`index % 8`). Each shard sandbox receives `/work/provenance.jsonl` and must call `submit_discovery` exactly once. An empty candidate list is valid for a thin shard.

Discovery rules encoded in the shard prompt and enforced after the tool call:

- Prefer local `pi`, `claude-code`, and `codex` turns when they clearly describe the same change as a merged PR on `SOURCE_REPO_URL`.
- `github-pull-request` is valid fallback provenance and may be used **only** for its own `sourcePr` and `sourceUrl`.
- Select a request only by an exact `sourceType` + `sessionId` + `messageIndex` present in that shard. Invented ids fail with `candidate … references unknown provenance`.
- Pairing a GitHub record with a different PR fails `assertProvenanceMatchesPullRequest`.
- Repository names or PR URLs inside message text are context only. `sourceUrl` / `sourcePr` must belong to the run's origin.
- Candidate `request` is replaced with the corpus `content` before authoring.

Authoring mounts the staged record at `/work/provenance.json`:

```json
{
  "source": { "sourceType": "codex", "sessionId": "session-1", "messageIndex": 0, "content": "…" },
  "messages": [{ "role": "user", "content": "…" }]
}
```

The authoring skill may use that file only to preserve the original intent. The Harbor export rebuilds `harbor-task/` from the accepted definition and patches; it does **not** include `provenance.jsonl`, session files, or author/reviewer transcripts.

## Empty corpus and related failures

### CLI: no retained messages

If local collection and GitHub collection both return zero messages, `self-bench run` throws **before** upload:

```text
no sanitized local-session or GitHub pull-request provenance was found
```

Typical causes:

| Symptom | What to check |
| --- | --- |
| No local hits | Sessions live under a different `$HOME`. Files are not `*.jsonl`. Transcript never mentions the worktree path (and the file path is not the dash-encoded worktree). |
| Only injected turns | Remaining user text starts with an injected prefix and was dropped. |
| No GitHub hits | No merged PRs in the `gh` window, only drafts/bots, titles empty, or every PR is under 20 changed lines / 0 files. |
| `gh` not usable | Missing binary, auth, or repo read access. You see `gh exited …`, not the empty-corpus string. |
| Unsupported `origin` | Non-GitHub remote fails in `resolveRepository` before GitHub collection. |

<Steps>
<Step title="Confirm the checkout and origin">
Run `git -C "$REPO" remote get-url origin` and `git -C "$REPO" worktree list --porcelain`. The origin must be GitHub. Every `worktree ` path is a local-session needle.
</Step>
<Step title="Confirm session files exist">
Look for `*.jsonl` under `$HOME/.pi/agent/sessions`, `$HOME/.claude/projects`, `$HOME/.codex/sessions`, and `$HOME/.codex/archived_sessions`. Confirm the worktree path or its dash-encoded form appears in the file path or contents.
</Step>
<Step title="Confirm merged human PRs">
Run the same `gh pr list --state merged --limit 500 --json …` command the CLI uses. Drafts, `*[bot]` authors, and PRs with fewer than 20 changed lines never enter the corpus.
</Step>
</Steps>

### After upload: corpus present but no candidates

A non-empty upload can still leave discovery with nothing it may pair to a merged PR on the origin. When a wave returns no new unique `sourcePr` values, the workflow enters `blocked` with:

```text
candidate pool exhausted; missing easy=N, medium=N, hard=N
```

That is a later terminal than the CLI empty-corpus error. Shard failures that are not cancellation or non-retryable activity errors contribute zero candidates and can produce the same blocked pool.

## Related pages

<CardGroup>
<Card title="Build a benchmark" href="/build-a-benchmark">
`self-bench run` flags, HEAD pin, provenance upload, and `--output` wait.
</Card>
<Card title="Run lifecycle" href="/run-lifecycle">
Eight-shard discovery, candidate budgets, and the blocked pool terminal.
</Card>
<Card title="Difficulty gates" href="/difficulty-gates">
Implementation-line and test-count minima applied after a request is paired.
</Card>
<Card title="HTTP API" href="/http-api">
`POST /v1/provenance` among the other authenticated run routes.
</Card>
<Card title="Authentication" href="/authentication">
Local `gh` / `GH_TOKEN` used for GitHub reads after collection.
</Card>
<Card title="Troubleshooting" href="/troubleshooting">
No-provenance, blocked candidate pool, and related operator failures.
</Card>
</CardGroup>