# Authentication

> OPENAI_API_KEY precedence, Pi and Codex subscription JSON fallbacks, GH_TOKEN, API Bearer token, and Modal profile versus token secrets.

- Repository: mupt-ai/self-bench
- GitHub: https://github.com/mupt-ai/self-bench
- Human docs: https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc
- Complete Markdown: https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc/llms-full.txt

## Source Files

- `src/subscription-auth.ts`
- `src/codex-review.ts`
- `src/evaluate.ts`
- `src/config.ts`
- `docs/operations.md`
- `tests/subscription-auth.test.ts`

---

---
title: "Authentication"
description: "OPENAI_API_KEY precedence, Pi and Codex subscription JSON fallbacks, GH_TOKEN, API Bearer token, and Modal profile versus token secrets."
---

Self-bench never hosts a model provider account. Operators supply credentials; the HTTP API holds only `SELFBENCH_API_TOKEN`, and the Temporal worker injects a **single selected model credential** (plus `GH_TOKEN` on clone stages) into each sandbox. `OPENAI_API_KEY` wins whenever it is non-empty after trim. ChatGPT subscription JSON is a compatibility fallback for existing Pi and Codex logins.

<Warning>
Compose binds the API to `0.0.0.0` inside the container and defaults `SELFBENCH_API_TOKEN` to `selfbench-local`. Export the same token on the host before `self-bench run`, or every `/v1/*` call returns `401`.
</Warning>

## Credential map

| Secret | Who reads it | Required when | Fallback |
| --- | --- | --- | --- |
| `OPENAI_API_KEY` | Worker, companion sandboxes, `self-bench-eval` | Recommended for discovery, authoring, review, repair, and Harbor Codex trials | Trimmed; whitespace-only is treated as unset |
| `SELFBENCH_PI_AUTH_JSON` | Worker via `loadPiModelAuth()` | `OPENAI_API_KEY` unset and a Pi stage runs | `~/.pi/agent/auth.json` |
| `SELFBENCH_CODEX_AUTH_JSON` | Worker via `loadCodexModelAuth()` | `OPENAI_API_KEY` unset and coupling repair runs | `CODEX_AUTH_JSON_PATH` or `~/.codex/auth.json` |
| `GH_TOKEN` | CLI `gh` process; worker `githubToken()` | Private clones and in-sandbox `gh`; merged-PR collection on the host | `gh auth token` on the worker; host `gh auth login` for `self-bench run` |
| `SELFBENCH_API_TOKEN` | API and CLI | API host is not loopback; always set in Compose | Unset is allowed only on `127.0.0.1`, `::1`, or `localhost` |
| `MODAL_TOKEN_ID` / `MODAL_TOKEN_SECRET` | Worker Modal SDK | Cloud workers without a mounted profile | `~/.modal.toml` mounted at `/root/.modal.toml` |
| `SELFBENCH_TEMPORAL_API_KEY` | API and worker Temporal clients | Temporal Cloud | Local Temporal has no API key |

## Ownership

```mermaid
flowchart TB
  subgraph host [Operator host]
    CLI["self-bench CLI"]
    GH["gh auth login / GH_TOKEN"]
    Key["OPENAI_API_KEY"]
    PiFile["~/.pi/agent/auth.json"]
    CodexFile["~/.codex/auth.json"]
    ModalFile["~/.modal.toml"]
    ApiTok["SELFBENCH_API_TOKEN"]
  end

  subgraph compose [Compose / cloud processes]
    API["API: node dist/api-main.js"]
    Worker["Worker: node dist/temporal/worker-main.js"]
  end

  subgraph sandboxes [Per-stage sandbox secrets]
    PiStages["OPENAI_API_KEY or SELFBENCH_PI_AUTH_JSON"]
    CodexStage["OPENAI_API_KEY or SELFBENCH_CODEX_AUTH_JSON"]
    GitSecret["GH_TOKEN on discover and author only"]
  end

  CLI -->|"Bearer"| API
  GH --> CLI
  ApiTok --> API
  ApiTok --> CLI
  Key --> Worker
  PiFile --> Worker
  CodexFile --> Worker
  ModalFile --> Worker
  GH --> Worker
  Worker --> PiStages
  Worker --> CodexStage
  Worker --> GitSecret
```

Compose gives the API the shared Temporal and artifact settings only. `GH_TOKEN`, `OPENAI_API_KEY`, Modal tokens, and the two subscription JSON variables are attached to the **worker** service. The API never receives the Docker socket or model credentials.

## Model authentication

`openAiApiKey()` reads `process.env.OPENAI_API_KEY`, trims it, and returns `undefined` when the result is empty. Both `loadPiModelAuth()` and `loadCodexModelAuth()` take that key first.

```mermaid
flowchart LR
  env["OPENAI_API_KEY trimmed"]
  env -->|set| api["provider openai + apiKey"]
  env -->|unset| pi["Pi: openai-codex + stripped authJson"]
  env -->|unset| cd["Codex: stripped ChatGPT tokens"]
```

<Note>
Subscription JSON is a compatibility path. Operations docs recommend `OPENAI_API_KEY` for discovery, authoring, coupling review, constrained repair, and model evaluation.
</Note>

<ParamField body="OPENAI_API_KEY" type="string">
Trimmed OpenAI API key. When set, Pi sandboxes use `--provider openai` and Codex review posts to `https://api.openai.com/v1/responses` with `Authorization: Bearer <key>`.
</ParamField>

<ParamField body="SELFBENCH_PI_AUTH_JSON" type="string">
Inline Pi `auth.json`. Used only when `OPENAI_API_KEY` is unset.
</ParamField>

<ParamField body="SELFBENCH_CODEX_AUTH_JSON" type="string">
Inline Codex `auth.json`. Used only when `OPENAI_API_KEY` is unset.
</ParamField>

<ParamField body="CODEX_AUTH_JSON_PATH" type="string">
Filesystem path for Codex subscription JSON. Default `~/.codex/auth.json`. Also the default `--auth` path for `self-bench-eval`.
</ParamField>

### Pi subscription JSON

`loadPiSubscriptionAuth()` reads `SELFBENCH_PI_AUTH_JSON` or `~/.pi/agent/auth.json`. The file may contain other providers; only the `openai-codex` object is forwarded.

Required shape:

```json
{
  "openai-codex": {
    "type": "oauth",
    "access": "<access-token>",
    "refresh": "<refresh-token>"
  }
}
```

Anything else throws `Pi auth does not contain an openai-codex subscription credential`. Tests assert that sibling keys such as `anthropic` never leave the host.

In discovery and authoring sandboxes, the bootstrap script writes that JSON to `$HOME/.pi/agent/auth.json` (`0600`) and selects `--provider openai-codex` when `OPENAI_API_KEY` is absent. The file is deleted on `EXIT`.

Coupling review (`sandbox-review`) also uses Pi auth. With only `authJson`, `reviewCouplingWithCodex()` posts to `https://chatgpt.com/backend-api/codex/responses`, sets `ChatGPT-Account-Id` from the JWT claim `https://api.openai.com/auth.chatgpt_account_id`, and sends `Originator: selfbench`. Missing `access` throws `Pi auth does not contain an OpenAI Codex subscription access token`. Missing account id throws `OpenAI Codex subscription token has no ChatGPT account ID`. Neither key nor `authJson` throws `OpenAI model authentication is required`.

If both `apiKey` and `authJson` reach the reviewer, the API key wins and the public Responses endpoint is used.

### Codex subscription JSON

`loadCodexSubscriptionAuth()` reads `SELFBENCH_CODEX_AUTH_JSON`, else `CODEX_AUTH_JSON_PATH`, else `~/.codex/auth.json`. `assertCodexSubscriptionAuth()` requires `auth_mode === "chatgpt"` and a `tokens` object. `auth_mode: "apikey"` is rejected even when `tokens` is present.

Forwarded payload:

```json
{
  "auth_mode": "chatgpt",
  "tokens": { },
  "last_refresh": "<optional string copied when present>"
}
```

Coupling **repair** (`sandbox-repair`, `self-bench-repair`) is the Codex-auth consumer. It writes `SELFBENCH_CODEX_AUTH_JSON` to `~/.codex/auth.json` (`0600`), or `{ "OPENAI_API_KEY": "<key>" }` when the API key is set. Missing both throws `OPENAI_API_KEY or SELFBENCH_CODEX_AUTH_JSON is required`.

## Stage-to-secret matrix

| Stage | Loader | Secret injected |
| --- | --- | --- |
| Discovery shard | `loadPiModelAuth()` + `githubToken()` | `OPENAI_API_KEY` or `SELFBENCH_PI_AUTH_JSON`, plus `GH_TOKEN` when resolved |
| Author candidate | same | same |
| Validation repair | `loadPiModelAuth()` | `OPENAI_API_KEY` or `SELFBENCH_PI_AUTH_JSON` |
| Coupling review / `self-bench-reaudit` | `loadPiModelAuth()` | `OPENAI_API_KEY` or `SELFBENCH_PI_AUTH_JSON` |
| Coupling repair / `self-bench-repair` | `loadCodexModelAuth()` | `OPENAI_API_KEY` or `SELFBENCH_CODEX_AUTH_JSON` |
| Harbor nop / oracle | none | No model secret; Harbor environment only |
| `self-bench-eval` matrix | `openAiApiKey()` then Codex file | `OPENAI_API_KEY`, or `CODEX_FORCE_AUTH_JSON=1` + `CODEX_AUTH_JSON_PATH` |

Docker sandboxes merge `secrets` into container env. Modal sandboxes wrap the same map in `client.secrets.fromObject(...)` and do not put those values in the non-secret `env` object.

## GitHub

Two different surfaces use GitHub credentials.

### Host CLI: merged pull requests

`self-bench run` calls `gh pr list --repo <owner/repo> --state merged --limit 500` on the **operator host**. Self-bench does not pass a token into that process. Authenticate with `gh auth login`, or export `GH_TOKEN` so the `gh` CLI picks it up. Write access is not required. Empty local-session plus GitHub corpora fail with `no sanitized local-session or GitHub pull-request provenance was found`.

### Worker sandboxes: clone and in-sandbox `gh`

`githubToken()` returns `process.env.GH_TOKEN` if that variable is set (not trimmed), otherwise `gh auth token` with `allowFailure: true`. A non-zero `gh` exit or empty stdout yields `undefined`; discovery and authoring then omit `GH_TOKEN`.

When present, bootstrap writes `https://x-access-token:${GH_TOKEN}@github.com` to `$HOME/.git-credentials` (`0600`) and enables `credential.helper store` before `git clone` of `SOURCE_REPO_URL`. The credentials file is removed on `EXIT`. Public HTTPS clones work without a token.

<Tip>
Compose interpolates `GH_TOKEN: ${GH_TOKEN:-}`. An empty host variable leaves the worker without a usable GitHub token. Export `GH_TOKEN="$(gh auth token)"` before `self-bench up`.
</Tip>

Provenance sanitization redacts `Authorization: Bearer …`, `sk-…`, and `ghp_` / `github_pat_` forms. That is not a general secret scanner.

## API Bearer token

`loadConfig()` requires `SELFBENCH_API_TOKEN` (min length 1) when `SELFBENCH_API_HOST` is not `127.0.0.1`, `::1`, or `localhost`. Compose sets `SELFBENCH_API_HOST=0.0.0.0`, so the token is mandatory for the local stack.

<ParamField body="SELFBENCH_API_TOKEN" type="string">
Shared secret. The CLI sends `Authorization: Bearer ${SELFBENCH_API_TOKEN}` only when the variable is set. The API compares with `crypto.timingSafeEqual` after stripping a leading `Bearer ` (case-insensitive). Unequal lengths fail closed.
</ParamField>

<ParamField body="SELFBENCH_API_URL" type="string">
CLI base URL. Default `http://127.0.0.1:8080`.
</ParamField>

Unauthenticated routes (even when a token is configured):

| Method | Path | Purpose |
| --- | --- | --- |
| `GET` | `/healthz` | `{ "ok": true }` |
| `GET` | `/` | Review UI `index.html` |
| `GET` | `/assets/*` | Review UI static assets |

Every other route, including `/v1/provenance`, `/v1/runs`, status, cancel, and export, returns `401` `{ "error": "unauthorized" }` when a token is configured and the header is missing or wrong.

<RequestExample>
```bash
export SELFBENCH_API_TOKEN="$(openssl rand -hex 24)"
export SELFBENCH_API_URL=http://127.0.0.1:8080
curl -sS -H "Authorization: Bearer $SELFBENCH_API_TOKEN" "$SELFBENCH_API_URL/v1/runs"
```
</RequestExample>

<ResponseExample>
```json
{
  "error": "unauthorized"
}
```
</ResponseExample>

README generates a 24-byte hex token. Compose default is `selfbench-local` when the host variable is unset — the CLI will not send a header unless you export the same value.

## Modal profile versus token secrets

`self-bench up --backend modal` sets `SELFBENCH_MODAL_CONFIG_PATH` to `~/.modal.toml` unless `--modal-config PATH` overrides it. `--modal-config` with `--backend docker` fails: `--modal-config requires --backend modal`.

Compose mounts that path at `/root/.modal.toml:ro`. Docker-backend `up` leaves `SELFBENCH_MODAL_CONFIG_PATH` unset, so the mount source is `/dev/null`.

Worker startup calls `removeEmptyModalCredentialOverrides()` **before** `loadConfig()`. Empty or whitespace `MODAL_TOKEN_ID` and `MODAL_TOKEN_SECRET` are deleted so they cannot mask a valid mounted profile. Non-empty values are kept for secret-manager / Temporal Cloud workers.

<CodeGroup>
```bash title="Local Modal profile"
modal token new
self-bench up --backend modal
# or:
self-bench up --backend modal --modal-config /absolute/path/to/.modal.toml
```

```text title="Cloud worker tokens"
SELFBENCH_EXECUTION_BACKEND=modal
SELFBENCH_HARBOR_ENVIRONMENT=modal
MODAL_TOKEN_ID=...
MODAL_TOKEN_SECRET=...
```
</CodeGroup>

`ModalSandboxExecutor` constructs `new ModalClient()` with no explicit credentials; the Modal SDK uses the profile or the token pair from the process environment.

## Harbor evaluation

`self-bench-eval` prefers `OPENAI_API_KEY`. If it is unset, it validates `--auth FILE` (default `~/.codex/auth.json`) with `assertCodexSubscriptionAuth()` before launching Harbor.

Subscription trials delete `OPENAI_API_KEY` from the child env and set:

- `CODEX_FORCE_AUTH_JSON=1`
- `CODEX_AUTH_JSON_PATH=<auth file>`

`summary.json` records `"auth": "openai-api-key"` or `"auth": "codex-subscription"`. Compiled Harbor tasks allowlist `chatgpt.com`, `*.chatgpt.com`, `openai.com`, and `*.openai.com` on the agent network.

```bash
export OPENAI_API_KEY=...
self-bench-eval --export ./self-bench-evals.tar.gz --jobs ./matrix-jobs --environment modal

# compatibility:
# unset OPENAI_API_KEY
# self-bench-eval --export ./self-bench-evals.tar.gz --jobs ./matrix-jobs --auth ~/.codex/auth.json
```

Harbor nop/oracle validation does not inject model credentials. `self-bench-agent-smoke` does not implement a separate auth loader.

## Local setup

<Steps>
<Step title="Authenticate GitHub and Modal">
```bash
gh auth login
modal token new
```
Read access to merged pull requests is enough.
</Step>
<Step title="Export worker and API secrets">
```bash
export OPENAI_API_KEY=...
export GH_TOKEN="$(gh auth token)"
export SELFBENCH_API_TOKEN="$(openssl rand -hex 24)"
```
Keep `SELFBENCH_API_TOKEN` in the same shell that later runs `self-bench run`.
</Step>
<Step title="Start the stack and confirm the API">
```bash
self-bench up --backend modal
export SELFBENCH_API_URL=http://127.0.0.1:8080
curl -sS "$SELFBENCH_API_URL/healthz"
curl -sS -H "Authorization: Bearer $SELFBENCH_API_TOKEN" "$SELFBENCH_API_URL/v1/runs"
```
Expect `{ "ok": true }` then a JSON array (possibly empty). A missing Bearer against Compose yields `unauthorized`.
</Step>
</Steps>

## Cloud worker secrets

A Temporal Cloud worker from the same image digest needs the model, GitHub, and Modal tokens. The API process on Cloud Run (or equivalent) needs `SELFBENCH_API_TOKEN`, GCS, and Temporal TLS + `SELFBENCH_TEMPORAL_API_KEY` — not `OPENAI_API_KEY` or `GH_TOKEN`.

```text
# worker
SELFBENCH_EXECUTION_BACKEND=modal
SELFBENCH_HARBOR_ENVIRONMENT=modal
MODAL_TOKEN_ID=...
MODAL_TOKEN_SECRET=...
GH_TOKEN=...
OPENAI_API_KEY=...
# optional compatibility:
# SELFBENCH_PI_AUTH_JSON=...
# SELFBENCH_CODEX_AUTH_JSON=...
```

Use separate least-privilege identities. Sandboxes execute a trusted repository checkout plus a short-lived model credential; this is not a malware-analysis boundary.

## Failures

| Symptom | Cause |
| --- | --- |
| `SELFBENCH_API_TOKEN is required when the API listens beyond loopback` | `SELFBENCH_API_HOST` is not `127.0.0.1` / `::1` / `localhost` and the token is missing |
| `401` `unauthorized` | CLI token unset or mismatched with Compose (`selfbench-local` vs generated hex) |
| `Pi auth does not contain an openai-codex subscription credential` | Fallback file missing `openai-codex` oauth `access` + `refresh` |
| `… does not contain a ChatGPT subscription token set` | Codex file is `auth_mode: "apikey"` or lacks `tokens` |
| `OpenAI model authentication is required` | Review sandbox received neither key nor Pi `authJson` |
| `OPENAI_API_KEY or SELFBENCH_PI_AUTH_JSON is required` | Review / validation-repair entrypoint |
| `OPENAI_API_KEY or SELFBENCH_CODEX_AUTH_JSON is required` | Coupling-repair entrypoint |
| Private `git clone` fails in discover/author | Worker `GH_TOKEN` empty; Compose passed through an unset host variable |
| Modal SDK ignores mounted `~/.modal.toml` | Empty `MODAL_TOKEN_*` leaked into the worker (fixed by startup strip) |
| Harbor Codex trial cannot authenticate | Matrix has no `OPENAI_API_KEY` and no valid ChatGPT `auth.json` |

## Next

<CardGroup>
<Card title="Installation" href="/installation">
Global install, Modal and GitHub login, required exports, first `self-bench` help check.
</Card>
<Card title="Configuration reference" href="/configuration-reference">
`SELFBENCH_*` keys, loopback token rule, artifact backends, Temporal fields.
</Card>
<Card title="HTTP API" href="/http-api">
Bearer-protected routes, status codes, provenance limits, export integrity headers.
</Card>
<Card title="Start the local stack" href="/start-local-stack">
`self-bench up` backends, `--modal-config`, named volumes.
</Card>
<Card title="Evaluate with Harbor" href="/evaluate-with-harbor">
`self-bench-eval` API-key path and `--auth` ChatGPT fallback.
</Card>
<Card title="Deploy Temporal Cloud" href="/deploy-temporal-cloud">
Split API/worker secrets, Modal tokens, Temporal TLS and API key.
</Card>
</CardGroup>