# Execution backends

> Docker versus Modal sandboxes: images, default concurrency, resource limits, inactivity timeouts, and Harbor environment pairing.

- Repository: mupt-ai/self-bench
- GitHub: https://github.com/mupt-ai/self-bench
- Human docs: https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc
- Complete Markdown: https://grok-wiki.com/public/docs/mupt-ai-self-bench-62fff2be9bcc/llms-full.txt

## Source Files

- `src/sandbox.ts`
- `src/docker-executor.ts`
- `src/modal-executor.ts`
- `src/modal-auth.ts`
- `src/config.ts`
- `src/cli.ts`
- `Dockerfile.sandbox`

---

---
title: "Execution backends"
description: "Docker versus Modal sandboxes: images, default concurrency, resource limits, inactivity timeouts, and Harbor environment pairing."
---

`createSandboxExecutor` selects `DockerSandboxExecutor` or `ModalSandboxExecutor` from `SELFBENCH_EXECUTION_BACKEND` (`docker` by default). Those sandboxes run discovery, authoring, coupling review, and repair. Harbor nop and oracle gates use a second knob, `SELFBENCH_HARBOR_ENVIRONMENT`, passed as `harbor run --env`. `self-bench up --backend` writes both knobs to the same value and records `executionBackend` plus `sandboxImage` on each run.

## Two knobs, one local pairing

| Knob | Controls | Local `self-bench up` | Config default |
| --- | --- | --- | --- |
| `SELFBENCH_EXECUTION_BACKEND` | Worker `SandboxExecutor` | `--backend` | `docker` |
| `SELFBENCH_HARBOR_ENVIRONMENT` | Worker `harbor run --env` | same `--backend` | falls back to the execution backend |

`self-bench up` always pairs them. A Temporal Cloud worker or a manually composed stack can set the two independently. If either is `docker`, worker startup runs `docker info` and `docker compose version` (30s each) before polling Temporal.

Companion CLIs do **not** read those worker keys. `self-bench-eval`, `self-bench-validate`, and `self-bench-agent-smoke` default `--environment modal` on the machine that invokes Harbor.

<Warning>
`self-bench-eval --environment` defaults to `modal` even when the generating worker used Docker. Pass `--environment docker` to match a Docker-validated export.
</Warning>

```mermaid
flowchart TB
  subgraph cli [CLI and Compose]
    UP["self-bench up --backend"]
    EXEC["SELFBENCH_EXECUTION_BACKEND"]
    HARB["SELFBENCH_HARBOR_ENVIRONMENT"]
    UP --> EXEC
    UP --> HARB
  end

  subgraph worker [Worker process]
    LOAD["loadConfig"]
    FACTORY["createSandboxExecutor"]
    DX["DockerSandboxExecutor"]
    MX["ModalSandboxExecutor"]
    VAL["validateTask"]
    EXEC --> LOAD
    HARB --> LOAD
    LOAD --> FACTORY
    FACTORY -->|"kind docker"| DX
    FACTORY -->|"kind modal"| MX
    LOAD --> VAL
  end

  subgraph dockerHost [Host Docker daemon]
    SIMG["selfbench-sandbox:local"]
    CTR["selfbench-RUN-STAGE-suffix container + volume"]
    DX --> SIMG
    DX --> CTR
  end

  subgraph modalCloud [Modal]
    APP["apps.fromName selfbench"]
    MSB["sandboxes.create workdir /work"]
    MX --> APP
    MX --> MSB
  end

  subgraph harborRt [Harbor from the worker image]
    HCMD["harbor run --env docker|modal"]
    TOML["task.toml cpus / memory_mb / storage_mb"]
    VAL --> HCMD
    HCMD --> TOML
  end
```

## Choose a backend

<Tabs>
<Tab title="Modal">
<Steps>
<Step title="Authenticate Modal">
Use a profile at `~/.modal.toml` (`modal token new`) or set `MODAL_TOKEN_ID` and `MODAL_TOKEN_SECRET`. Empty token env vars are deleted at worker startup so they cannot override a mounted profile.
</Step>
<Step title="Start the stack">

```bash
self-bench up --backend modal
```

Override the profile path when it is not `~/.modal.toml`:

```bash
self-bench up --backend modal --modal-config /absolute/path/to/.modal.toml
```

`--modal-config` is rejected unless `--backend` is `modal`.
</Step>
<Step title="Verify">
The CLI prints `SelfBench is running with the modal backend at http://127.0.0.1:8080`. The worker log includes `activity concurrency 20` unless `SELFBENCH_ACTIVITY_CONCURRENCY` overrides it.
</Step>
</Steps>
</Tab>
<Tab title="Docker">
<Steps>
<Step title="Build the sandbox image and start the stack">

```bash
self-bench up --backend docker
```

This builds `Dockerfile.sandbox` as `selfbench-sandbox:local`, then `docker compose up -d --build`. Compose pins `SELFBENCH_DOCKER_IMAGE` to `selfbench-sandbox:local`.
</Step>
<Step title="Confirm the socket">
The worker service runs as `root`, joins `DOCKER_GID` (default `0`), and mounts `/var/run/docker.sock`. The API service does not receive the socket or model credentials.
</Step>
<Step title="Verify">
The CLI prints `SelfBench is running with the docker backend at http://127.0.0.1:8080`. The worker log includes `activity concurrency 1` unless `SELFBENCH_ACTIVITY_CONCURRENCY` overrides it.
</Step>
</Steps>
</Tab>
</Tabs>

<Info>
`--backend` defaults to `docker`. The README quickstart uses `--backend modal` so sandboxes leave the host.
</Info>

## Backend comparison

| Property | Docker | Modal |
| --- | --- | --- |
| Executor | `DockerSandboxExecutor` | `ModalSandboxExecutor` |
| Image | `SELFBENCH_DOCKER_IMAGE` (`selfbench-sandbox:local`) | `SELFBENCH_MODAL_IMAGE` (`node:22-bookworm`) plus inline install layers |
| App / environment | host Docker daemon | `SELFBENCH_MODAL_APP` (`selfbench`), optional `SELFBENCH_MODAL_ENVIRONMENT` |
| Default activity concurrency | `1` | `20` |
| CPU / memory defaults | `--cpus 4`, `--memory 8192m` | `cpu: 4`, `memoryMiB: 8192` |
| Storage limit | not applied by the executor | not applied by the executor |
| Hard timeout | `SandboxRequest.timeoutMs` on `docker start --attach` | same value on `sandboxes.create({ timeoutMs })` |
| Inactivity timeout | `runCommand` kills the attach process | executor timer calls `sandbox.terminate()` |
| Extra idle cap | none | `idleTimeoutMs = min(timeoutMs, 10 minutes)` |
| Workdir | `/work` named volume | `/work` |
| Secrets | merged into container `--env` | `secrets.fromObject` on `exec` |
| Missing required output after exit `0` | throws | omitted; caller diagnoses from stdout/stderr |
| Cleanup | `docker rm --force` and `docker volume rm --force` | `sandbox.terminate()` |
| `close()` | no-op | `ModalClient.close()` |

`SELFBENCH_ACTIVITY_CONCURRENCY` is an integer from 1 to 100. It becomes Temporal `maxConcurrentActivityTaskExecutions`. Docker defaults to 1 because sandboxes share the host.

## Images

### Docker: `Dockerfile.sandbox`

`self-bench up --backend docker` builds:

```bash
docker build -f Dockerfile.sandbox -t selfbench-sandbox:local .
```

`up --backend modal` skips that build. The image is `node:22-bookworm` plus:

| Install | Pin |
| --- | --- |
| apt | `bash` `ca-certificates` `curl` `git` `jq` `ripgrep` `unzip` `xz-utils` |
| GitHub CLI | `gh` 2.89.0 (`amd64` and `arm64` SHA-256 checked) |
| Pi | `@earendil-works/pi-coding-agent@0.84.0` |
| Codex | `@openai/codex@0.146.1` |

`WORKDIR` is `/work`. Compose does not interpolate `SELFBENCH_DOCKER_IMAGE`; the local stack always uses `selfbench-sandbox:local`. A standalone worker (Temporal Cloud) honors the env var / default.

### Modal: registry base plus dockerfile commands

`ModalSandboxExecutor` calls `images.fromRegistry(SELFBENCH_MODAL_IMAGE)` and then the same tool installs as `Dockerfile.sandbox`, with two differences:

- `gh` is the **amd64** 2.89.0 tarball only (SHA `d0422caade520530e76c1c558da47daebaa8e1203d6b7ff10ad7d6faba3490d8`).
- `PATH` is forced to `/root/.local/bin` plus the usual Unix locations.

The Modal app is created if missing. Each sandbox is tagged `run_id` and `stage`.

### Worker image (not the sandbox)

`Dockerfile` builds the API/worker image (`selfbench:local` in Compose). It ships the Docker CLI 29.4.0 and `harbor[modal]==0.20.1.dev202608040148`. Harbor validation therefore runs **inside the worker container**, not inside `selfbench-sandbox:local`.

## Sandbox contract

Every executor implements:

```ts
run(request: SandboxRequest, options?: SandboxRunOptions): Promise<SandboxResult>
close(): void
```

<ParamField body="runId" type="string" required>
Workflow or companion run id. Docker names use the first 20 characters; Modal uses 24. Both append stage (16 chars) and an 8-character UUID suffix.
</ParamField>

<ParamField body="stage" type="string" required>
Activity label. Docker names become `selfbench-{runId}-{stage}-{suffix}` (lowercase `[a-z0-9_.-]`). Modal names omit the `selfbench-` prefix and keep mixed case `[a-zA-Z0-9._-]`.
</ParamField>

<ParamField body="command" type="string[]" required>
Executed in `/work`. Authoring uses `bash -lc` around the Pi bootstrap; review/repair invoke bundled Node scripts.
</ParamField>

<ParamField body="files" type="SandboxFile[]">
Written before exec. Docker rejects any path that is not under `/work/`.
</ParamField>

<ParamField body="outputPaths" type="string[]">
Copied after the process exits. Docker throws if the process exits 0 without a listed path.
</ParamField>

<ParamField body="environment" type="Record<string, string>">
Non-secret env (repo URL, model name, output paths).
</ParamField>

<ParamField body="secrets" type="Record<string, string>">
Short-lived credentials only: `OPENAI_API_KEY` or a stage-specific `SELFBENCH_PI_AUTH_JSON` / `SELFBENCH_CODEX_AUTH_JSON`, plus `GH_TOKEN` when cloning.
</ParamField>

<ParamField body="timeoutMs" type="number" required>
Hard wall clock for the sandbox process.
</ParamField>

<ParamField body="inactivityTimeoutMs" type="number">
No stdout/stderr for this long raises `InactivityTimeoutError` (`{scope} produced no output for {timeoutMs}ms`).
</ParamField>

<ParamField body="cpu" type="number">
Defaults to `4` when omitted.
</ParamField>

<ParamField body="memoryMiB" type="number">
Defaults to `8192` when omitted.
</ParamField>

`SandboxRunOptions.signal` force-removes the Docker container or terminates the Modal sandbox. `onProgress` reports stdout/stderr byte counts for Temporal heartbeats.

Stdout/stderr are capped at 8 MiB (`RollingOutput`). Older output is replaced with `[selfbench: earlier output truncated]`.

## Resource limits

There are two resource surfaces. Do not treat them as one.

### Authoring-family sandboxes

Discovery, authoring, review, and repair omit `cpu` / `memoryMiB` in the worker and therefore get **4 CPU and 8192 MiB**. `self-bench-repair` sets those same values explicitly. Neither executor sets a disk quota.

### Harbor task resources

Authored `definition.json` `resources` compile into `task.toml`:

```toml
[environment]
cpus = 4
memory_mb = 8192
storage_mb = 20480

[verifier.environment]
cpus = 4
memory_mb = 8192
storage_mb = 20480
```

Authoring prompt defaults are 4 CPU, 8192 MB memory, 20480 MB storage, and timeouts 900 / 2400 / 900 seconds (setup / agent / tests). Those Harbor limits apply when Harbor builds agent and verifier environments, not when `DockerSandboxExecutor` creates the authoring container.

<Note>
`storageMb` is a Harbor `task.toml` field. Changing it does not add a Docker `--storage-opt` or a Modal disk request on the SelfBench sandbox.
</Note>

## Timeouts

| Stage | `stage` value | Hard timeout | Inactivity | Temporal start-to-close |
| --- | --- | --- | --- | --- |
| Discovery shard | `discover-{wave}-{shardIndex}` | 45 min | 8 min | 1 hour, 3 attempts |
| Author | `author-{candidateId}` | 2 hours | 8 min | 7 hours, 4 attempts |
| Harbor nop / oracle | Harbor CLI, not `SandboxExecutor` | 3 hours per `harbor run` | — | same 7-hour activity budget |
| Validation repair | `validation-repair-{taskId}` | 2 hours | 8 min | 2 hours, 1 attempt |
| Coupling review | `review-{taskId}` | 15 min | 5 min | 7 hours, 4 attempts |
| Coupling repair | `repair-{taskId}` | 2 hours | 8 min | 7 hours, 4 attempts |
| `self-bench-reaudit` | task id | 15 min | unset | n/a |
| `self-bench-repair` | task id | 2 hours | unset | n/a |
| Companion Harbor runs | `harbor run` | 4 hours | — | n/a |

Modal also sets `idleTimeoutMs` to `min(timeoutMs, 10 minutes)` on every sandbox create. Review's 15-minute hard timeout is therefore paired with a 10-minute Modal idle cap.

Discovery uses eight independently retryable shards. Heartbeat timeout for proxied activities is 10 minutes.

Harbor infrastructure failures (`HarborInfrastructureFailure`, including non-zero `harbor` exits and Harbor infrastructure errors in the job result) mark the candidate `infrastructure_failed` after activity retries rather than failing the whole workflow.

## Harbor environment pairing

Worker validation:

```bash
harbor run \
  --path TASK_DIR \
  --agent nop|oracle \
  --env "$SELFBENCH_HARBOR_ENVIRONMENT" \
  --job-name ... \
  --jobs-dir ... \
  --delete --yes --quiet
```

<CodeGroup>
```bash title="Paired local stack"
self-bench up --backend modal    # execution=modal, harbor=modal
self-bench up --backend docker   # execution=docker, harbor=docker
```

```bash title="Independent worker env"
SELFBENCH_EXECUTION_BACKEND=modal
SELFBENCH_HARBOR_ENVIRONMENT=modal
# or mix, e.g. modal sandboxes + docker Harbor (requires Docker on the worker)
```

```bash title="Companion Harbor (defaults to modal)"
self-bench-validate --tasks ./tasks --jobs ./jobs --environment docker
self-bench-eval --export ./export.tar.gz --jobs ./jobs --environment modal
self-bench-agent-smoke --task ./task --jobs ./jobs --environment modal
```
</CodeGroup>

Companion defaults:

| Command | `--environment` default | `--concurrency` default |
| --- | --- | --- |
| `self-bench-validate` | `modal` | `10` |
| `self-bench-eval` | `modal` | `3` |
| `self-bench-agent-smoke` | `modal` | `4` |

`self-bench-eval` and `self-bench-validate` pass `--n-concurrent 1 --max-retries 1` into each Harbor job. The worker gate does not pass those flags.

Each submitted run stores:

```json
{
  "version": {
    "executionBackend": "docker",
    "sandboxImage": "selfbench-sandbox:local",
    "schema": 1
  }
}
```

Modal runs record `sandboxImage` as `SELFBENCH_MODAL_IMAGE` (the registry base), not the layered Modal image id.

## Configuration

<ParamField body="SELFBENCH_EXECUTION_BACKEND" type="docker | modal">
Worker sandbox backend. Default `docker`.
</ParamField>

<ParamField body="SELFBENCH_HARBOR_ENVIRONMENT" type="docker | modal">
Harbor `--env`. Default: same as `SELFBENCH_EXECUTION_BACKEND`.
</ParamField>

<ParamField body="SELFBENCH_ACTIVITY_CONCURRENCY" type="integer 1–100">
Overrides the Docker `1` / Modal `20` default.
</ParamField>

<ParamField body="SELFBENCH_DOCKER_IMAGE" type="string">
Docker sandbox image. Default `selfbench-sandbox:local`. Compose hardcodes that tag.
</ParamField>

<ParamField body="SELFBENCH_MODAL_IMAGE" type="string">
Modal registry base. Default `node:22-bookworm`. Not set in `compose.yaml`; the default applies unless the worker env adds it.
</ParamField>

<ParamField body="SELFBENCH_MODAL_APP" type="string">
Modal app name. Default `selfbench`.
</ParamField>

<ParamField body="SELFBENCH_MODAL_ENVIRONMENT" type="string">
Optional Modal environment passed to `apps.fromName` and `secrets.fromObject`.
</ParamField>

<ParamField body="SELFBENCH_MODAL_CONFIG_PATH" type="string">
Compose bind-mount source for `/root/.modal.toml`. `up --backend modal` sets this to `~/.modal.toml` or `--modal-config`. Otherwise Compose mounts `/dev/null`.
</ParamField>

<ParamField body="MODAL_TOKEN_ID / MODAL_TOKEN_SECRET" type="string">
Optional token pair. Blank values are stripped in `worker-main` before `loadConfig`.
</ParamField>

<RequestExample>
```bash
# Cloud worker (same image digest as the API)
export SELFBENCH_EXECUTION_BACKEND=modal
export SELFBENCH_HARBOR_ENVIRONMENT=modal
export SELFBENCH_MODAL_APP=selfbench
export MODAL_TOKEN_ID=...
export MODAL_TOKEN_SECRET=...
node dist/temporal/worker-main.js
```
</RequestExample>

<ResponseExample>
```text
SelfBench worker polling default/selfbench-dev with activity concurrency 20
```
</ResponseExample>

## Cleanup and leftover Docker resources

On success or failure, `DockerSandboxExecutor` force-removes the container and its named volume, then deletes the host temp root (`selfbench-docker-*`). A host or daemon crash can leave `selfbench-*` containers and volumes.

```bash
docker ps -a --filter name=selfbench-
docker volume ls --filter name=selfbench-
docker rm --force "$(docker ps -aq --filter name=selfbench-)"
docker volume rm --force $(docker volume ls -q --filter name=selfbench-)
```

Modal sandboxes are terminated in a `finally` block. `self-bench down` stops Compose services and keeps the `selfbench_artifacts` and `selfbench_temporal-postgres` named volumes.

<Warning>
Sandboxes receive a source checkout plus a short-lived model credential. Use SelfBench only with repositories you trust to execute.
</Warning>

## Errors

| Signal | Meaning |
| --- | --- |
| `--backend must be "docker" or "modal"` | `self-bench up` flag is not one of those two values |
| `--modal-config requires --backend modal` | profile override used with Docker |
| `--environment must be docker or modal` | companion Harbor flag invalid |
| `sandbox path must be beneath /work` | Docker file or output path rejected |
| `sandbox {id} exited successfully without output {path}` | Docker exit 0, missing `outputPaths` entry |
| `InactivityTimeoutError` | no process output for 8 min (agents) or 5 min (review) |
| `Harbor … infrastructure failure` | Harbor CLI or job-level infrastructure error |
| Worker fails `docker info` | execution or Harbor is `docker` but the daemon is unreachable |

## Next

<CardGroup>
<Card title="Start the local stack" href="/start-local-stack">
`self-bench up` / `down`, Compose services, `--modal-config`, and named volumes.
</Card>
<Card title="Configuration reference" href="/configuration-reference">
Full `SELFBENCH_*` table, loopback token rule, and Temporal fields.
</Card>
<Card title="Authentication" href="/authentication">
`OPENAI_API_KEY` precedence, subscription JSON fallbacks, and Modal profile versus tokens.
</Card>
<Card title="Evaluate with Harbor" href="/evaluate-with-harbor">
`self-bench-eval` / Harbor `--env` after an export.
</Card>
<Card title="Deploy Temporal Cloud" href="/deploy-temporal-cloud">
Modal worker, matching image digest, and task queue.
</Card>
<Card title="Troubleshooting" href="/troubleshooting">
Harbor infrastructure failures and leftover Docker resources.
</Card>
</CardGroup>