# Troubleshooting

> Common install and runtime failures: health checks, auth cookies, Valkey connectivity, storage misconfiguration, and MCP connection errors.

- Repository: Paca-AI/paca
- GitHub: https://github.com/Paca-AI/paca
- Human docs: https://grok-wiki.com/public/docs/paca-ai-paca-f238b2ab3d25
- Complete Markdown: https://grok-wiki.com/public/docs/paca-ai-paca-f238b2ab3d25/llms-full.txt

## Source Files

- `deploy/README.md`
- `SECURITY.md`
- `services/api/docker-entrypoint.sh`
- `services/realtime/README.md`
- `docs/guides/getting-started.md`
- `apps/mcp/README.md`

---

---
title: "Troubleshooting"
description: "Common install and runtime failures: health checks, auth cookies, Valkey connectivity, storage misconfiguration, and MCP connection errors."
---

Paca surfaces install and runtime failures through container health probes (`/api/healthz`, `/healthz`), API bootstrap logs (`bootstrap: ...`), and service-specific startup checks (Valkey `PING`, storage `EnsureBucket`, required env validation). Use the sections below to isolate which dependency or configuration surface is failing before changing secrets or restarting the full stack.

## Quick diagnostic checklist

<Steps>
<Step title="Confirm the gateway is reachable">

From the host where you run clients or browsers:

```bash
curl -s http://localhost/api/healthz
```

<ResponseExample>

```json
{"status":"ok"}
```

</ResponseExample>

A `200` with `{"status":"ok"}` means the nginx gateway and API are up. Connection refused or `502`/`504` usually means the gateway container started before upstreams were healthy, or `nginx/gateway.conf` is missing from the install directory.

</Step>

<Step title="Inspect container health and logs">

```bash
docker compose ps
docker compose logs api --tail 50
docker compose logs realtime --tail 50
docker compose logs valkey --tail 20
```

Look for `bootstrap:` errors on API startup, `cache: ping:` for Valkey, `bootstrap: ensure storage bucket:` for MinIO/S3, and `Missing required environment variable` in the realtime service.

</Step>

<Step title="Verify required secrets in .env">

Production compose fails fast when required variables are absent. At minimum, confirm these are set and non-empty:

| Variable | Used by |
|---|---|
| `JWT_SECRET` | API — token signing |
| `ADMIN_PASSWORD` | API — seeded admin account |
| `POSTGRES_PASSWORD` or `DATABASE_URL` | API, ai-agent |
| `STORAGE_ACCESS_KEY_ID`, `STORAGE_SECRET_ACCESS_KEY` | API, MinIO |
| `ENCRYPTION_KEY` | API, ai-agent — plugin/agent secret encryption |

Generate secrets with `openssl rand -hex 32`. `ENCRYPTION_KEY` must be a 64-character lowercase hex string (32 bytes).

</Step>
</Steps>

## Health check failures

### API container unhealthy

The API health probe calls `GET /api/healthz` inside the container. The handler returns `200` with `{"status":"ok"}` and performs no dependency checks — if the probe fails, the process is not listening or bootstrap never completed.

| Symptom | Likely cause | Fix |
|---|---|---|
| API stuck in `starting` for 40s+ then `unhealthy` | Bootstrap failed before HTTP server started | Read `docker compose logs api` for `bootstrap:` prefix |
| `JWT_SECRET is required` at compose parse time | Missing `.env` entry | Set `JWT_SECRET` in `.env` and restart |
| `bootstrap: cache: ping:` | Valkey unreachable at `REDIS_URL` | Ensure `valkey` is healthy; confirm `REDIS_URL=redis://valkey:6379/0` in Docker networks |
| `bootstrap: auto-migrate:` | PostgreSQL connection or migration error | Verify `DATABASE_URL`; check postgres health and credentials |
| `bootstrap: ensure storage bucket:` | MinIO down or credential mismatch | Confirm `minio` is healthy; align `STORAGE_*` with `MINIO_ROOT_USER` / `MINIO_ROOT_PASSWORD` |

<Note>

The API applies database migrations automatically on every startup. Migration failures block the health endpoint from ever becoming available.

</Note>

### Gateway returns 502 or connection refused

The gateway mounts `./nginx/gateway.conf` read-only. If the file is missing, nginx fails to start or serves no upstream routes.

```bash
ls -la nginx/gateway.conf
curl -s http://localhost/api/healthz
```

Download both release artifacts together when setting up manually:

```bash
curl -fsSL https://github.com/Paca-AI/paca/releases/latest/download/docker-compose.yml -o docker-compose.yml
curl -fsSL https://github.com/Paca-AI/paca/releases/latest/download/gateway.conf -o nginx/gateway.conf
```

### Realtime container unhealthy

Realtime exposes `GET /healthz` on port `3001` (independent of Socket.IO). Compose waits for API health before starting realtime.

| Symptom | Likely cause | Fix |
|---|---|---|
| `Missing required environment variable: REDIS_URL` | Env not passed to realtime service | Set `REDIS_URL` in `.env` (default in compose: `redis://valkey:6379/0`) |
| `Missing required environment variable: API_URL` | Internal API URL unset | Production compose sets `API_URL=http://api:8080` automatically |
| Realtime logs `valkey subscriber error` | Valkey connection dropped | Restart `valkey`; verify network and `REDIS_URL` match API |

## Auth cookie failures

Paca session auth uses HttpOnly cookies set by `POST /api/v1/auth/login`:

| Cookie | Path | Notes |
|---|---|---|
| `access_token` | `/` | Sent on API and realtime requests |
| `refresh_token` | `/api/v1/auth/refresh` | Only sent to the refresh endpoint |

<ParamField body="COOKIE_SECURE" type="boolean" default="true (production)">
When `true`, browsers only send cookies over HTTPS. Set `COOKIE_SECURE=false` only when serving over plain HTTP (local dev). Production `.env.production.example` defaults to `true`.
</ParamField>

### Login succeeds but subsequent requests return 401

| Symptom | Likely cause | Fix |
|---|---|---|
| Login works once, then all API calls 401 | `JWT_SECRET` changed after tokens were issued | Log out and log in again; avoid rotating `JWT_SECRET` without clearing sessions |
| Browser never stores cookies | `COOKIE_SECURE=true` over `http://` | Set `COOKIE_SECURE=false` for HTTP deployments |
| Realtime connects then immediately disconnects | Missing or expired `access_token` cookie | Confirm `withCredentials: true` on Socket.IO; re-login if access token expired (default `JWT_ACCESS_TTL=15m`) |
| `POST /api/v1/auth/refresh` returns missing token | Refresh cookie not sent | Refresh cookie path is `/api/v1/auth/refresh` only — call refresh against the gateway API base, not a different host/port |

### Realtime auth rejected

Socket.IO auth resolves the JWT from `handshake.auth.token` first, then the `access_token` cookie. The realtime service does not validate JWTs locally — it calls `GET /api/v1/users/me/global-permissions` on the API. Any non-200 response produces `invalid or expired token`.

```ts
const socket = io("http://localhost", {
  path: "/ws/socket.io",
  withCredentials: true,
});
```

<Warning>

If `CORS_ORIGINS` on the realtime service does not include your browser origin, the Socket.IO handshake fails before auth runs. Production compose defaults `CORS_ORIGINS` to `PUBLIC_URL`.

</Warning>

## Valkey connectivity

Valkey (Redis-compatible) is required by the API, realtime, and ai-agent services. The API pings Valkey at bootstrap; failure prevents startup.

| Service | Env var | Default (bundled compose) |
|---|---|---|
| API | `REDIS_URL` | `redis://valkey:6379/0` |
| Realtime | `REDIS_URL` | `redis://valkey:6379/0` |
| AI agent | `VALKEY_URL` | Same as `REDIS_URL` in compose |

### API fails with `cache: ping`

1. Confirm Valkey is running: `docker compose ps valkey`
2. Test from the API container network: Valkey healthcheck uses `valkey-cli ping`
3. For host-side dev (API on host, infra in Docker): set `REDIS_URL=redis://localhost:6379/0`

### Realtime events not arriving

Realtime subscribes to the `paca.events` Pub/Sub channel published by the API. Symptoms and checks:

| Symptom | Check |
|---|---|
| Web UI stale until manual refresh | `docker compose logs realtime` — look for `subscribed to valkey channel` |
| `valkey subscriber reconnecting` loops | Valkey restart or network partition; check Valkey logs and volume |
| Socket connected but no task/doc events | Client must `emit("join", { projectId })` after connect; events route only to joined project rooms |

```bash
docker compose logs realtime | rg "valkey|subscribe|auth failed"
docker compose logs api | rg "redis connected"
```

## Storage misconfiguration

The API uses S3-compatible storage for attachments and presigned uploads. With bundled MinIO, the API auto-creates the bucket on startup (`EnsureBucket`). With AWS S3 (`STORAGE_PROVIDER=s3`), the bucket must already exist and the API skips auto-creation.

### Startup failure: `bootstrap: ensure storage bucket`

| Cause | Fix |
|---|---|
| MinIO not healthy | Start or scale up `minio`; wait for `mc ready local` healthcheck |
| Credential mismatch | `STORAGE_ACCESS_KEY_ID` / `STORAGE_SECRET_ACCESS_KEY` must match MinIO `MINIO_ROOT_USER` / `MINIO_ROOT_PASSWORD` |
| Wrong endpoint | Docker network: `STORAGE_ENDPOINT=minio:9000`; host-side dev: `localhost:9000` |
| Using S3 but MinIO still running | Set `STORAGE_PROVIDER=s3`, real AWS credentials, and start with `--scale minio=0` |

### Upload/download URL failures in the browser

Presigned URLs are rewritten from the internal MinIO host (`minio:9000`) to `STORAGE_PUBLIC_URL` so browsers reach storage through the gateway `/storage/` proxy. The gateway rewrites the `Host` header to `minio:9000` for signature validation.

| Misconfiguration | Symptom |
|---|---|
| `STORAGE_PUBLIC_URL` does not match your public hostname | Presigned URLs point at unreachable host |
| `STORAGE_PUBLIC_URL` missing `/storage` suffix | Browser requests bypass gateway MinIO proxy |
| `minio` scaled to 0 but `STORAGE_PROVIDER=minio` | 502 on `/storage/` routes |

<Tabs>
<Tab title="Bundled MinIO">

```bash
STORAGE_PROVIDER=minio
STORAGE_ENDPOINT=minio:9000
STORAGE_PUBLIC_URL=http://localhost/storage
STORAGE_BUCKET=paca
STORAGE_ACCESS_KEY_ID=<matches MINIO_ROOT_USER>
STORAGE_SECRET_ACCESS_KEY=<matches MINIO_ROOT_PASSWORD>
STORAGE_USE_SSL=false
```

</Tab>
<Tab title="AWS S3">

```bash
STORAGE_PROVIDER=s3
STORAGE_ENDPOINT=
STORAGE_PUBLIC_URL=
STORAGE_REGION=us-east-1
STORAGE_BUCKET=your-existing-bucket
STORAGE_ACCESS_KEY_ID=<aws-key>
STORAGE_SECRET_ACCESS_KEY=<aws-secret>
STORAGE_USE_SSL=true
```

Start without MinIO: `docker compose --env-file .env up -d --scale minio=0`

</Tab>
</Tabs>

### Plugin directory permissions

The API entrypoint creates `/plugins`, `/plugins-frontend`, and `/plugins-mcp` with ownership `app:app` before starting the binary. Permission errors on plugin volumes usually appear at container start, not at first plugin install.

## MCP connection errors

The `@paca-ai/paca-mcp` server runs via `npx` and authenticates with `X-API-Key`. It exits immediately if required env vars are missing.

<ParamField body="PACA_API_KEY" type="string" required>
API key from **Settings → API Keys**. Required at startup; missing key prints an error and exits with code 1.
</ParamField>

<ParamField body="PACA_API_URL" type="string" default="http://localhost:8080">
Base URL prepended to paths like `/api/v1/projects`. MCP requests use `${PACA_API_URL}/api/v1/...`.
</ParamField>

<ParamField body="PACA_AGENT_ID" type="string">
When set, `PACA_PROJECT_ID` is also required. Agent mode uses the server `AGENT_API_KEY` value, not a per-agent key.
</ParamField>

### Common MCP failures

| Error | Cause | Fix |
|---|---|---|
| `PACA_API_KEY environment variable is required` | Key not in MCP client config | Add `PACA_API_KEY` to the `env` block; restart the MCP client |
| `Connection refused` / `ECONNREFUSED` | API not reachable at `PACA_API_URL` | For production Docker (API not host-published), use the gateway URL: `PACA_API_URL=http://localhost` not `:8080` |
| `API request failed: 401` | Invalid, revoked, or expired API key | Generate a new key in Paca settings; check for trailing whitespace |
| `API request failed: 403` | Key valid but missing permissions | Assign global/project roles; set `PACA_PROJECT_ID` for project-scoped tools |
| `PACA_PROJECT_ID is required when using PACA_AGENT_ID` | Agent mode incomplete | Set both `PACA_AGENT_ID` and `PACA_PROJECT_ID` |
| `npx: command not found` | Node.js not installed | Install Node.js 18+ |
| `Cannot find package '@paca-ai/paca-mcp'` | npm registry unreachable | Check network; verify with `npx @paca-ai/paca-mcp --version` |
| No project tools listed | Global user mode without project scope | Set `PACA_PROJECT_ID` to enable `list_tasks`, `create_task`, etc. |

<CodeGroup>

```bash title="Verify API reachability (production via gateway)"
curl -s http://localhost/api/healthz
```

```bash title="Verify API reachability (dev API on host port)"
curl -s http://localhost:8080/api/healthz
```

```bash title="Test MCP package"
npx @paca-ai/paca-mcp --version
```

</CodeGroup>

<AccordionGroup>
<Accordion title="Agent mode key confusion">

Agent mode expects `PACA_API_KEY` to be the server-wide `AGENT_API_KEY` from the Paca deployment environment, combined with `PACA_AGENT_ID` and `PACA_PROJECT_ID`. The MCP server sends `X-Agent-ID` to impersonate the agent. A user's personal API key works for user mode only.

</Accordion>

<Accordion title="Plugin MCP tools missing">

At startup the MCP server calls `GET /api/v1/plugins` and loads enabled plugins with `mcp.remoteEntryUrl`. Failures to fetch plugins log warnings but core tools still load. If plugin tools are absent, confirm the plugin is enabled in Paca and the API key can read plugins.

</Accordion>

<Accordion title="Debug logging">

```bash
export DEBUG="*"
npx @paca-ai/paca-mcp
```

</Accordion>
</AccordionGroup>

## Install and upgrade failures

### Compose project rename (volume migration)

The compose project was renamed from `paca-prod` to `paca`. Existing volumes (`paca-prod_postgres_data`, etc.) are not automatically attached. After stopping the old stack, copy data into new volume names before starting the new project, or accept a fresh install with empty data.

### Required env validation at compose time

Docker Compose interpolates `${JWT_SECRET:?...}` and `${ADMIN_PASSWORD:?...}` — missing values fail before containers start. Generate a complete `.env` via the install script or `.env.production.example`.

### AI agent optional but failing

Skip the ai-agent when Docker socket access or resources are unavailable:

```bash
docker compose --env-file .env up -d --scale ai-agent=0
```

When enabled, `AGENT_API_KEY` and `INTERNAL_API_KEY` must match across `api` and `ai-agent` services, and `ENCRYPTION_KEY` must be identical so agent LLM keys stored encrypted in PostgreSQL can be decrypted.

### Rate limiting on API

The gateway applies `100r/s` per IP with burst 50 on `/api/`. Sustained automated probing returns `429`. This is expected under aggressive health polling or load tests — back off request rates.

## Log signals reference

| Log prefix / message | Service | Meaning |
|---|---|---|
| `bootstrap: cache: ping:` | API | Valkey unreachable at `REDIS_URL` |
| `bootstrap: ensure storage bucket:` | API | MinIO/S3 bucket creation or head failed |
| `bootstrap: auto-migrate:` | API | PostgreSQL migration error |
| `redis connected` | API | Valkey connection OK |
| `schema migrations applied` | API | DB ready |
| `valkey session client connected` | Realtime | Session Redis client OK |
| `subscribed to valkey channel` | Realtime | Pub/Sub on `paca.events` active |
| `socket auth failed` | Realtime | Token rejected by API permissions check |
| `PACA_API_KEY environment variable is required` | MCP | Client env misconfigured |

## Related pages

<CardGroup>
<Card title="Installation" href="/installation">
Prerequisites, install script, manual compose setup, and required secrets.
</Card>
<Card title="Quickstart" href="/quickstart">
First successful run and health endpoint verification.
</Card>
<Card title="Configuration reference" href="/configuration-reference">
Full environment variable catalog across all services.
</Card>
<Card title="Connect MCP server" href="/connect-mcp">
Client-specific MCP setup for Claude Desktop, VS Code, and other hosts.
</Card>
<Card title="Deploy production" href="/deploy-production">
Production topology, storage backends, and scaling optional services.
</Card>
<Card title="Upgrade and migration" href="/upgrade-migration">
Image upgrades, automatic migrations, and volume migration steps.
</Card>
</CardGroup>