# Troubleshooting

> readyz/docker socket failures, port 80 conflicts (HTTP_PORT), ULID validation, warming-page stalls, userns-remap seed errors, preview spin-up, and compose log probes.

- Repository: tastyeffectco/sandboxes
- GitHub: https://github.com/tastyeffectco/sandboxes
- Human docs: https://grok-wiki.com/public/docs/tastyeffectco-sandboxes-f551c1a2e9a0
- Complete Markdown: https://grok-wiki.com/public/docs/tastyeffectco-sandboxes-f551c1a2e9a0/llms-full.txt

## Source Files

- `AGENTS.md`
- `README.md`
- `control-plane/internal/api/handlers.go`
- `control-plane/internal/wake/handler.go`
- `install.sh`
- `.env.example`

---

---
title: "Troubleshooting"
description: "readyz/docker socket failures, port 80 conflicts (HTTP_PORT), ULID validation, warming-page stalls, userns-remap seed errors, preview spin-up, and compose log probes."
---

sandboxd exposes `GET /healthz` and `GET /readyz` on `SANDBOXED_API_BIND` (default `127.0.0.1:9090`); Traefik publishes previews on `HTTP_PORT` (default `80`). Most install and runtime failures show up as a non-`ready` readiness probe, a compose bind error on port 80, a `400` with `id must be a ULID`, a stuck **Spinning up your app!** page, or `seed:` / permission errors during workspace provisioning.

## Quick checks

<Steps>
<Step title="Probe the control plane">

```bash
curl -s http://127.0.0.1:9090/healthz   # expect: ok
curl -s http://127.0.0.1:9090/readyz    # expect: ready
```

`healthz` only confirms the process is up. `readyz` additionally requires SQLite and a recent successful `docker info`.

</Step>
<Step title="Inspect the stack">

```bash
docker compose ps
docker compose logs -f sandboxd
docker compose logs -f traefik
docker ps --filter label=sandboxed.managed=true
```

</Step>
<Step title="Confirm Docker from the host">

```bash
docker info
```

`install.sh` falls back to `sudo docker` when the current user cannot reach the daemon.

</Step>
</Steps>

| Symptom | Likely cause | First move |
| --- | --- | --- |
| `readyz` → `503` with `docker info:` | Socket unreachable or daemon down | `docker info`; verify `/var/run/docker.sock` in compose |
| `docker compose up` fails on `:80` | Port 80 in use | Set `HTTP_PORT=8088` in `.env`, `docker compose up -d` |
| `id must be a ULID` | Non-ULID `id` on create | Omit `id` or pass a valid ULID |
| **Spinning up your app!** loops | Wake OK but nothing listens on the port, or slow start | `POST .../exec` or tasks API; check sandbox logs |
| `seed:` / permission on create | `userns-remap` without `SANDBOXED_USERNS=host` | Keep default `SANDBOXED_USERNS=host` |
| Preview 404 / not_found | Wrong id or Host casing (handled internally) | `GET /sandbox/{id}`; use canonical preview host |

## Health and readiness

| Endpoint | HTTP | Body | Meaning |
| --- | --- | --- | --- |
| `GET /healthz` | 200 | `ok\n` | Process alive; no Docker or DB check |
| `GET /readyz` | 200 | `ready\n` | SQLite `Ping` OK and `docker info` succeeded |
| `GET /readyz` | 503 | JSON error | `sqlite ping: …` or `docker info: …` |

Both probes are auth-exempt when API auth is enabled.

<Warning>
Orchestrators that only hit `healthz` can mark the stack healthy while `readyz` is still failing — sandbox create, exec, and wake all need a working Docker socket.
</Warning>

### `readyz` and Docker socket failures

`sandboxd` shells out to the host `docker` CLI over the mounted socket (`docker-compose.yml` bind-mounts `/var/run/docker.sock` into the `sandboxd` service). `handleReadyz` runs `Store.DB().PingContext` then `Docker.Info`, which executes `docker info --format {{.ServerVersion}}`.

<RequestExample>

```bash
curl -s -w "\nHTTP %{http_code}\n" http://127.0.0.1:9090/readyz
```

</RequestExample>

<ResponseExample>

```text
docker info: exit status 1
HTTP 503
```

</ResponseExample>

<Steps>
<Step title="Verify the daemon on the host">

```bash
docker info
# or, if install.sh selected sudo:
sudo docker info
```

</Step>
<Step title="Confirm the socket inside sandboxd">

```bash
docker compose exec sandboxd ls -l /var/run/docker.sock
docker compose exec sandboxd docker info --format '{{.ServerVersion}}'
```

</Step>
<Step title="Restart the control plane">

```bash
docker compose restart sandboxd
curl -s http://127.0.0.1:9090/readyz
```

</Step>
</Steps>

<Note>
`install.sh` probes `docker info` before compose and prints `using 'sudo docker'` when the user is not in the `docker` group. The in-container `sandboxd` process still uses the socket mount — fix host daemon access, not only your shell alias.
</Note>

## Port 80 conflicts (`HTTP_PORT`)

Traefik maps host port `${HTTP_PORT:-80}` to container port `80`. If another service owns 80, compose fails at bind time or previews hit the wrong process.

<ParamField body="HTTP_PORT" type="number" default="80">
Host port for Traefik HTTP. When not `80`, every preview URL needs the suffix, e.g. `http://s-<id>-3000.preview.localhost:8088`.
</ParamField>

<Steps>
<Step title="Pick a free port">

```bash
# example: use 8088
echo 'HTTP_PORT=8088' >> .env
```

</Step>
<Step title="Recreate the stack">

```bash
docker compose up -d
```

</Step>
<Step title="Hit previews with the suffix">

```bash
ID=<your-ulid>
curl -s -H "Host: s-${ID}-3000.preview.localhost" http://127.0.0.1:8088/
```

</Step>
</Steps>

The installer summary prints the correct `PORTSUFFIX` when `HTTP_PORT != 80`.

## ULID validation

`POST /sandbox` accepts an optional `id`. Empty `id` triggers server-side `newULID()`. A supplied `id` must parse via `ulid.Parse`; otherwise the handler returns `400` with message `id must be a ULID`.

| Input | Result |
| --- | --- |
| omit `id` | Auto-generated Crockford Base32 ULID |
| `"01ARZ3NDEKTSV4RRFFQ69G5FAV"` | Accepted if valid |
| `"demo01"`, UUIDs, arbitrary strings | `400` — `id must be a ULID` |

```bash
# recommended: omit id
curl -s -XPOST http://127.0.0.1:9090/sandbox \
  -H 'content-type: application/json' \
  -d '{"ports":[3000]}'
```

<Warning>
The post-install banner in `install.sh` still shows `"id":"demo01"` as an example — that value is **not** a ULID and will fail validation. Omit `id` or capture the generated id from the response.
</Warning>

Sandbox primary keys in SQLite are ULIDs; preview hostnames embed the same id (`s-{id}-{port}.preview.{domain}`).

## Warming-page stalls

Stopped sandboxes have no Traefik priority-100 router. The file-provider catch-all in `traefik/dynamic/wake.yml` (priority `1`) forwards matching preview hosts to `http://sandboxd:9000`. The wake handler starts the container and returns an HTML page titled **Spinning up your app!** with a meta-refresh (default **2** seconds). After refresh, the sandbox’s Docker labels should register a priority-100 route and Traefik proxies to the dev server.

```mermaid
sequenceDiagram
  participant Browser
  participant Traefik
  participant sandboxd
  participant Docker
  participant Sandbox as s-{id} container

  Browser->>Traefik: GET preview Host (stopped)
  Traefik->>sandboxd: catch-all priority 1
  sandboxd->>Docker: docker start s-{id}
  sandboxd->>Sandbox: TCP probe :port (up to 8s)
  sandboxd-->>Browser: 200 Spinning up + meta-refresh
  Note over Traefik,Sandbox: Labels publish priority-100 router
  Browser->>Traefik: refresh
  Traefik->>Sandbox: proxy to listening port
```

### When the page keeps appearing

| Cause | What to verify |
| --- | --- |
| Nothing listening on the exposed port | Process bound to the port you declared in `ports` (e.g. `3000`) |
| Slow boot (install, agent task) | Wait longer than one refresh; default TCP-ready wait is **8s** (`SANDBOXD_WAKE_TCP_READY_TIMEOUT_SECONDS`) — timeout is informational; wake still succeeds and serves the refresh page |
| Sandbox `error` or `creating` | `GET /sandbox/{id}` → `status`, `error_message` |
| Memory admission denied | **Almost ready…** page (503) with `Retry-After`; host memory pressure |
| Wrong preview port in URL | Port in hostname must match an exposed port |

```bash
# is the sandbox up?
curl -s http://127.0.0.1:9090/sandbox/$ID | jq .status

# start a minimal server on the declared port
curl -s -XPOST http://127.0.0.1:9090/sandbox/$ID/exec \
  -H 'content-type: application/json' \
  -d '{"cmd":["bash","-lc","cd ~/workspace && python3 -m http.server 3000"]}'
```

### Machine-readable wake headers

White-label HTML hides internal reasons; correlate in logs or DevTools:

| Header | When |
| --- | --- |
| `X-Wake-Error` | Non-admission failure (`not_found`, `start_failed`, `creating`, …) |
| `X-Retry-After-Reason` | Admission / pressure denial |
| `Retry-After` | Seconds before retry (busy page, default 30) |

Prometheus label `wakes_total{result="tcp_ready_timeout"}` increments when the TCP probe times out; the user still gets the refresh page.

## Userns-remap and seed errors

Workspace directories live under `SANDBOXED_DATA_DIR/workspaces/<id>`. On first create, `loopback.Provision` runs a one-shot seed container that copies `/opt/sandbox-skel` and `chown`s to `sandbox:sandbox`. With Docker `userns-remap`, seeding as root inside the default user namespace maps to a high host UID that cannot write a host-owned workspace dir.

Default: `SANDBOXED_USERNS=host` on infra containers (`traefik`, `sandboxd`) and on sandbox + seed `docker run` (`--userns host`). On a daemon **without** userns-remap this is a no-op; with remap it keeps `chown` deterministic.

| Failure shape | Fix |
| --- | --- |
| `seed: …` / permission denied on create | Ensure `.env` / compose passes `SANDBOXED_USERNS=host` (default) |
| Intentionally use daemon userns | Set `SANDBOXED_USERNS=` empty (see `ARCHITECTURE.md`) — only if you understand ownership on the data dir |

```bash
docker compose exec sandboxd printenv SANDBOXED_USERNS
# expect: host
```

After changing userns policy, recreate affected sandboxes or purge workspaces that were seeded with wrong ownership.

## Preview spin-up

End-to-end preview recovery:

<Steps>
<Step title="Confirm row state">

```bash
curl -s http://127.0.0.1:9090/sandbox/$ID
```

Expect `running` for an active server, or `stopped` before the first wake.

</Step>
<Step title="Wake via preview (local)">

```bash
# add :$HTTP_PORT when HTTP_PORT is not 80
curl -sv -H "Host: s-${ID}-3000.preview.localhost" http://127.0.0.1:${HTTP_PORT:-80}/ 2>&1 | head -30
```

Browsers lowercase `Host`; the wake handler uppercases the captured id before DB lookup.

</Step>
<Step title="Programmatic wake">

```bash
curl -s -XPOST http://127.0.0.1:9090/wake/$ID \
  -H 'Accept: application/json'
```

</Step>
<Step title="Ensure Traefik sees the sandbox network">

```bash
docker inspect s-$ID --format '{{json .NetworkSettings.Networks}}'
```

Sandboxes join `SANDBOXED_NETWORK` (default `sandboxed_net`).

</Step>
</Steps>

| Preview symptom | Check |
| --- | --- |
| `not_found` / error page | Id typo; sandbox purged; `GET /sandbox/{id}` |
| Redirect to sign-in | `visibility=private` without preview cookie — see private previews |
| Connection refused on `:80` | `HTTP_PORT` mismatch |
| Works with `curl -H Host` but not browser | DNS: `*.localhost` resolves to `127.0.0.1` on most browsers; production needs real wildcard DNS |

## Compose log probes

| Command | Use |
| --- | --- |
| `docker compose logs -f sandboxd` | Create, wake, reaper, `docker start` / `seed:` failures |
| `docker compose logs -f traefik` | Router registration, catch-all vs per-sandbox priority |
| `docker compose ps` | `traefik` / `sandboxd` restarts, port mapping |
| `docker logs s-<ID>` | In-sandbox dev server or `runtimed` |
| `tail -f ${SANDBOXED_LOG_DIR}/traefik-access.log` | Per-request routing (shared log dir mount) |

```bash
# follow wake-related lines
docker compose logs -f sandboxd 2>&1 | rg -i 'wake|seed|docker start|admission'

# last 100 lines without follow
docker compose logs --tail=100 sandboxd
```

<Info>
Access logs path defaults to `${SANDBOXED_LOG_DIR}/traefik-access.log` (see `SANDBOXD_ACCESS_LOG` in compose). `install.sh` chmods the log dir `0777` so Traefik can write.
</Info>

## Related pages

<CardGroup>
<Card title="Installation" href="/installation">
Prerequisites, `install.sh`, `.env` bootstrap, and first `healthz` / `readyz` checks.
</Card>
<Card title="Observability" href="/observability">
Probe semantics, Prometheus metrics, and structured logging paths.
</Card>
<Card title="Wake, idle, and pressure" href="/wake-idle-reapers">
Stop-on-idle, wake admission, keepalive, and warming-page behavior in depth.
</Card>
<Card title="Preview routing" href="/preview-routing">
Traefik priorities, catch-all wake router, and `PREVIEW_DOMAIN` rules.
</Card>
<Card title="Workspaces and isolation" href="/workspaces-persistence">
Bind mounts, skeleton seeding, `SANDBOXED_USERNS`, and storage layout.
</Card>
<Card title="Configuration reference" href="/configuration-reference">
`HTTP_PORT`, `SANDBOXED_DATA_DIR`, wake timeouts, and reaper tuning env keys.
</Card>
</CardGroup>