# Health and metrics

> GET /healthz and /readyz semantics, Prometheus GET /metrics labels, audit/access logging paths, and docker compose logs for sandboxd and Traefik.

- Repository: tastyeffectco/sandboxes
- GitHub: https://github.com/tastyeffectco/sandboxes
- Human docs: https://grok-wiki.com/public/docs/tastyeffectco-sandboxes-f551c1a2e9a0
- Complete Markdown: https://grok-wiki.com/public/docs/tastyeffectco-sandboxes-f551c1a2e9a0/llms-full.txt

## Source Files

- `control-plane/internal/api/handlers.go`
- `control-plane/internal/metrics/metrics.go`
- `control-plane/internal/api/api.go`
- `control-plane/internal/audit/audit.go`
- `control-plane/cmd/sandboxd/main.go`
- `AGENTS.md`

---

---
title: "Health and metrics"
description: "GET /healthz and /readyz semantics, Prometheus GET /metrics labels, audit/access logging paths, and docker compose logs for sandboxd and Traefik."
---

`sandboxd` exposes three operator-facing HTTP probes on the control-plane mux — `GET /healthz`, `GET /readyz`, and `GET /metrics` — plus structured request logs on stderr and a shared Traefik JSON access log that drives idle detection. Audit rows land in SQLite; there is no audit read API.

## Probe endpoints

All three routes register on the API mux in `control-plane/internal/api/api.go`. Each handler (except the raw Prometheus handler) runs inside the `observe` middleware, which increments `sandboxd_api_requests_total` and records `sandboxd_api_request_duration_seconds` with the route pattern as the `endpoint` label.

| Route | Purpose | Auth on external path |
| --- | --- | --- |
| `GET /healthz` | Liveness — process is up | Exempt (no bearer token) |
| `GET /readyz` | Readiness — SQLite + Docker daemon | Exempt |
| `GET /metrics` | Prometheus scrape | **404** (loopback only) |

The auth middleware (`control-plane/internal/auth/middleware.go`) treats `/healthz` and `/readyz` as exempt on Traefik-routed traffic. `/metrics` is explicitly blocked on the external path; only direct loopback calls to `SANDBOXED_API_BIND` can scrape it.

### Liveness: `GET /healthz`

:::endpoint GET /healthz
Process liveness. Does not touch SQLite or Docker.
:::

<ResponseExample>

```http
HTTP/1.1 200 OK

ok
```

</ResponseExample>

The handler always returns **200** with the plain-text body `ok\n`. Use this for “is the HTTP server accepting connections?” — not for Docker socket or database health.

<RequestExample>

```bash
curl -s "http://127.0.0.1:9090/healthz"
```

</RequestExample>

### Readiness: `GET /readyz`

:::endpoint GET /readyz
Readiness. Fails unless SQLite answers a ping and `docker info` succeeds in this request.
:::

`handleReadyz` runs two synchronous checks on every call:

1. **SQLite** — `Store.DB().PingContext(r.Context())`. Failure → **503** JSON `{"error":"sqlite ping: ..."}`.
2. **Docker** — `Docker.Info`, which shells out to `docker info --format {{.ServerVersion}}`. Failure → **503** JSON `{"error":"docker info: ..."}`. Success also feeds `sandboxd_docker_command_duration_seconds{op="info"}`.

When both pass, the response is **200** with body `ready\n`.

<ResponseExample>

```http
HTTP/1.1 200 OK

ready
```

</ResponseExample>

<RequestExample>

```bash
curl -s "http://127.0.0.1:9090/readyz"
```

</RequestExample>

<Warning>

A **503** on `readyz` with `docker info: exit status 1` almost always means the control-plane container cannot reach the host Docker socket. Confirm `/var/run/docker.sock` is mounted and the daemon is running. See the troubleshooting page for the full checklist.

</Warning>

<Note>

Orchestrators should use **`/healthz` for liveness** and **`/readyz` for readiness**. Do not point a liveness probe at `readyz` — a transient Docker blip would restart the pod unnecessarily.

</Note>

## Prometheus: `GET /metrics`

The scrape handler uses a dedicated registry (`metrics.Registry`) and `promhttp.HandlerFor` — not the default global Prometheus registry.

### Scrape access

| Caller | Result |
| --- | --- |
| Host loopback (`curl http://127.0.0.1:9090/metrics`) | **200** — standard Prometheus text exposition |
| Traefik / LAN without loopback | **404** — middleware blocks external `/metrics` |

Bind is controlled by compose: host `${SANDBOXED_API_BIND:-127.0.0.1:9090}` → container `:9000` (`SANDBOXD_ADDR` default `0.0.0.0:9000` inside the container).

### Metric families and labels

Metrics are defined in `control-plane/internal/metrics/metrics.go`. API middleware buckets HTTP status into `1xx` … `5xx` for the `code` label on `sandboxd_api_requests_total`.

#### Build and inventory

| Metric | Type | Labels | Notes |
| --- | --- | --- | --- |
| `sandboxd_build_info` | Gauge (=1) | `version`, `git_commit` | Set at startup from build metadata |
| `sandboxd_sandboxes_total` | Gauge | `status` | `creating`, `running`, `stopped`, `error` — refreshed on create/destroy and at boot |

#### API and Docker CLI

| Metric | Type | Labels |
| --- | --- | --- |
| `sandboxd_api_requests_total` | Counter | `endpoint`, `method`, `code` |
| `sandboxd_api_request_duration_seconds` | Histogram | `endpoint`, `method` |
| `sandboxd_docker_command_duration_seconds` | Histogram | `op` — e.g. `run`, `inspect`, `exec`, `rm`, `info` |
| `sandboxd_docker_command_errors_total` | Counter | `op` |

The `endpoint` label is the registered route pattern (e.g. `GET /readyz`, `POST /v1/sandboxes/{id}/tasks`), not the resolved path with IDs.

#### Reconciler

| Metric | Type | Labels |
| --- | --- | --- |
| `sandboxd_reconciler_runs_total` | Counter | — |
| `sandboxd_reconciler_last_duration_seconds` | Gauge | — |
| `sandboxd_reconciler_orphans_total` | Gauge | `kind` — e.g. `container`, `mount` |

#### Idle, pressure, wake, activity

| Metric | Type | Labels |
| --- | --- | --- |
| `sandboxd_idle_reaper_runs_total` | Counter | — |
| `sandboxd_idle_reaper_stops_total` | Counter | `reason` |
| `sandboxd_pressure_reaper_runs_total` | Counter | — |
| `sandboxd_pressure_reaper_stops_total` | Counter | `band` |
| `sandboxd_mem_available_percent` | Gauge | — |
| `sandboxd_mem_available_bytes` | Gauge | — |
| `sandboxd_wakes_total` | Counter | `outcome` — `success`, `admission_denied`, `start_failed`, `tcp_ready_timeout`, `not_found`, `error` |
| `sandboxd_wake_duration_seconds` | Histogram | — |
| `sandboxd_wakes_refused_active` | Gauge | 0/1 pressure gate |
| `sandboxd_inflight_exec_total` | Gauge | — |
| `sandboxd_access_log_lag_seconds` | Gauge | Lag between now and newest parsed Traefik access line |

#### Snapshots, preview auth, git push

| Metric | Type | Labels |
| --- | --- | --- |
| `sandboxd_snapshots_taken_total` | Counter | `outcome` — `ok`, `error` |
| `sandboxd_snapshot_restores_total` | Counter | `outcome` |
| `sandboxd_snapshotter_runs_total` | Counter | — |
| `sandboxd_snapshot_last_duration_seconds` | Gauge | — |
| `sandboxd_snapshot_last_size_bytes` | Gauge | — |
| `sandboxd_forward_auth_duration_seconds` | Histogram | — |
| `sandboxd_preview_access_total` | Counter | `result` — `allowed`, `denied` |
| `sandboxd_git_push_total` | Counter | `outcome` — `ok`, `failed` |
| `sandboxd_nginx_reloads_total` | Counter | `outcome` |

#### Egress (non-OSS compose)

These collectors exist in code but the **portable docker-compose build sets `egressMgr = nil`** and does not start journald/nft pollers. In OSS, `sandboxd_egress_sources_active` stays **0** and egress counters are unused.

| Metric | Labels |
| --- | --- |
| `sandboxd_egress_connections_total` | `sandbox_id`, `dst_port_bucket` |
| `sandboxd_egress_drops_total` | `reason` |
| `sandboxd_egress_log_lag_seconds` | — |
| `sandboxd_git_hosts_refresh_runs_total` | `outcome` |
| `sandboxd_abuse_list_refresh_runs_total` | `outcome` |

<Info>

Optional Traefik connection polling (`SANDBOXD_POLLER_METRIC_RE`, `SANDBOXD_POLLER_URL`, default `http://127.0.0.1:8082/metrics`) is a separate scrape target for long-lived WebSocket activity. Without that env, sandboxd logs **fallback mode** and relies on the access-log tailer alone.

</Info>

## Log and audit paths

```text
SANDBOXED_DATA_DIR/          (default /var/lib/sandboxed)
├── state/
│   ├── sandboxd.db          SQLite — sandboxes + audit_log
│   └── traefik-tail.offset  Access-log tailer checkpoint
└── workspaces/              Per-sandbox bind mounts

SANDBOXED_LOG_DIR/           (default /var/lib/sandboxed/log)
└── traefik-access.log       Traefik JSON access log (shared mount)
```

Both paths default under `/var/lib/sandboxed` and are bind-mounted **symmetrically** into `traefik` and `sandboxd` (`docker-compose.yml`, `.env.example`).

### Traefik access log (activity signal)

Traefik writes JSON access lines to `filePath: /var/lib/sandboxed/log/traefik-access.log` (`traefik/traefik.yml`). Override the directory with `SANDBOXED_LOG_DIR`; sandboxd reads the file via `SANDBOXD_ACCESS_LOG` (compose default: `${SANDBOXED_LOG_DIR}/traefik-access.log`).

A background **access-log tailer** (`control-plane/internal/activity/tailer.go`):

- Matches `RequestHost` against `s-{id}-{port}.preview.{PREVIEW_DOMAIN}`
- Calls `BumpLastActive` on matching sandbox rows (feeds the idle reaper)
- Persists read offset to `SANDBOXED_DATA_DIR/state/traefik-tail.offset` (`SANDBOXD_TAILER_OFFSET` override)
- Updates `sandboxd_access_log_lag_seconds` from each line’s `StartUTC`

Fields parsed per line: `RequestHost`, `RequestMethod`, `OriginStatus`, `StartUTC`, `RouterName`. Malformed lines are skipped.

### sandboxd request logs

Every HTTP request passes through `logging.Middleware`, which emits **JSON slog** lines to **stderr** with `request_id`, `method`, `path`, `status`, `duration_ms`. In compose, collect them with `docker compose logs sandboxd`. Set `SANDBOXD_DEBUG` for debug-level verbosity.

Traefik process logs are separate JSON on stderr (`log.format: json` in `traefik/traefik.yml`) — use `docker compose logs traefik`.

### Audit log (SQLite)

Privileged actions append one row to `audit_log` in `sandboxd.db` (`control-plane/migrations/0004_external_identity.sql`). There is **no HTTP read API**; operators query with `sqlite3` on the host.

| Column | Meaning |
| --- | --- |
| `at` | Unix seconds |
| `actor_kind` | `service`, `operator`, `system`, `unknown` |
| `actor_name` | Token name or `loopback` |
| `actor_ip` | Client IP (respects `X-Forwarded-For` on external path) |
| `external_user_id` | Upstream user when relevant |
| `action` | Stable action string |
| `target` | Sandbox ID, external user/project id, etc. |
| `detail` | JSON blob |

Writes are **best-effort** (failures log a warning, never fail the API response). `preview.access_allowed` uses **sampled** writes (at most one row per minute per `sub|sandbox` key).

Representative `action` values:

| Action | Trigger |
| --- | --- |
| `sandbox.create`, `sandbox.destroy`, `sandbox.exec`, `sandbox.stop` | Lifecycle / exec |
| `sandbox.wake` | Wake handler |
| `sandbox.purge`, `sandbox.claim` | Purge / claim |
| `sandbox.snapshot.create`, `sandbox.snapshot.restore` | Manual snapshots |
| `task.create`, `task.cancel` | v1 tasks API |
| `file.put` | v1 workspace file write |
| `snapshot.create`, `snapshot.delete` | v1 library snapshots |
| `preview.session_issued`, `preview.access_denied`, `preview.access_allowed` | Private preview auth |
| `external_user.purge`, `external_project.purge` | Bulk purge |
| `auth.token_invalid` | Failed bearer token |

`POST /sandbox/{id}/exec` audits **only `cmd[0]`** — never the full argv.

<RequestExample>

```bash
sqlite3 /var/lib/sandboxed/state/sandboxd.db \
  "SELECT datetime(at,'unixepoch'), actor_kind, actor_name, action, target
   FROM audit_log ORDER BY id DESC LIMIT 20;"
```

</RequestExample>

## Compose log operations

<Steps>

<Step title="Follow control-plane logs">

```bash
docker compose logs -f sandboxd
```

Look for `component` fields (`api`, `idle-reaper`, `pressure-reaper`, `access-log-tailer`, `reconcile`, `wake`) and `http` access lines with `request_id`.

</Step>

<Step title="Follow edge router logs">

```bash
docker compose logs -f traefik
```

Router/service churn, TLS, and provider errors appear here — not in sandboxd.

</Step>

<Step title="Verify probes after install">

```bash
curl -s "http://127.0.0.1:9090/healthz"   # ok
curl -s "http://127.0.0.1:9090/readyz"   # ready
curl -s "http://127.0.0.1:9090/metrics" | head
```

</Step>

</Steps>

When `traefik/dynamic/api.yml` is present, the same probes are reachable at `http://api.preview.<PREVIEW_DOMAIN>/healthz` without a bearer token even when API auth is enabled.

## Request flow (probes vs activity)

```mermaid
sequenceDiagram
  participant Op as Operator / kubelet
  participant SD as sandboxd
  participant DB as SQLite
  participant DK as Docker daemon
  participant TF as Traefik
  participant AL as traefik-access.log

  Op->>SD: GET /healthz
  SD-->>Op: 200 ok

  Op->>SD: GET /readyz
  SD->>DB: Ping
  SD->>DK: docker info
  SD-->>Op: 200 ready

  TF->>AL: JSON access line
  SD->>AL: tailer reads RequestHost
  SD->>DB: BumpLastActive
```

## Related pages

<CardGroup>

<Card title="Installation" href="/installation">
Install verification uses `healthz` / `readyz` after `docker compose up`.
</Card>

<Card title="API authentication" href="/api-authentication">
Exempt probes, loopback operator path, and why `/metrics` is not exposed externally.
</Card>

<Card title="Configuration reference" href="/configuration-reference">
`SANDBOXED_DATA_DIR`, `SANDBOXED_LOG_DIR`, `SANDBOXED_API_BIND`, and poller env keys.
</Card>

<Card title="Wake, idle, and pressure" href="/wake-idle-reapers">
How access-log bumps and reaper metrics tie into stop-on-idle behavior.
</Card>

<Card title="Troubleshooting" href="/troubleshooting">
`readyz` / Docker socket failures and compose log probes.
</Card>

<Card title="Control plane API (legacy)" href="/legacy-api-reference">
Full route list including health and metrics endpoints.
</Card>

</CardGroup>