# Troubleshooting

> Auth failures, invalid resume selectors, provider 401s, network retry, worker recovery, and connection-mode failure probes.

- Repository: PrimeIntellect-ai/prime-agent
- GitHub: https://github.com/PrimeIntellect-ai/prime-agent
- Human docs: https://grok-wiki.com/public/docs/primeintellect-ai-prime-agent-3367c32760b1
- Complete Markdown: https://grok-wiki.com/public/docs/primeintellect-ai-prime-agent-3367c32760b1/llms-full.txt

## Source Files

- `packages/coding-agent/test/suite/regressions/4435-auth-error-login-guidance.test.ts`
- `packages/coding-agent/test/suite/regressions/3317-network-connection-lost-retry.test.ts`
- `packages/coding-agent/test/suite/regressions/4722-invalid-resume-selector.test.ts`
- `packages/coding-agent/test/suite/regressions/4491-provider-stale-after-401.test.ts`
- `packages/coding-agent/test/suite/regressions/4603-worker-recovery.test.ts`
- `packages/coding-agent/docs/agent-connection.md`

---

---
title: "Troubleshooting"
description: "Auth failures, invalid resume selectors, provider 401s, network retry, worker recovery, and connection-mode failure probes."
---

Runtime failures in Prime Agent surface on three boundaries: provider auth and stream errors on the agent session, saved-session resume selectors at CLI startup, and local daemon/worker transport under `AgentConnection`. Most user-facing recovery paths are `/login`, retry settings, `--resume` / Agents View, and `prime-agent doctor` / `shutdown --force`.

## Failure map

| Symptom | Boundary | Primary signals | First recovery |
|---------|----------|-----------------|----------------|
| `401` / `403`, invalid API key, expired token | Provider stream → `AgentSession` | `stopReason: "error"`, optional `provider_stream_failure` diagnostic, `auth_stale` | `/login`, then re-prompt |
| `Network connection lost.` or other transient provider errors | Auto-retry | `auto_retry_start` / `auto_retry_end` | Wait for backoff; adjust `retry.*` if exhausted |
| `No session found matching '…'` / ambiguous resume | CLI session resolver | `SessionSelectorNotFoundError`, `SessionSelectorAmbiguousError` | Use suggested ID, suffix, or left-arrow session browser |
| Lost daemon socket, session closed, stale supervisor | `DaemonAgentConnection` | `connection_status`, `closed`, session closed reasons | Reopen from Agents View; `list` / `attach`; `doctor` / `shutdown --force` |
| Worker crash / supervisor generation fence | Resident worker recovery | New worker descriptor, `supervisor_generation_stale`, `session_resynced` | Let recovery finish; reattach; do not replay uncertain mutations |

## Auth failures and login guidance

When an assistant message ends with `stopReason: "error"` and the error text looks like authentication failure, the session appends:

```text
Run /login to update credentials.
```

Classification treats a message as authentication-related when it matches any of:

- status tokens `401` or `403`
- phrases such as `unauthorized`, `forbidden`, `invalid API key`, `authentication failed`, `expired` / `invalid token`, `access denied`, `permission denied`

If the error already contains `/login`, guidance is not duplicated.

### Startup / credential-missing messages

| Situation | Message pattern | Action |
|-----------|-----------------|--------|
| No models resolve | `No models available.` + provider login help | `/login` or set a provider API key |
| Model selected but no key | `No API key found for <provider>.` | `/login` or env / `auth.json` for that provider |
| Explicit auth failure | `Authentication failed for "<provider>"…` | `/login` |

Provider login help points at the local docs under the install’s `providers.md` and `models.md`, and at interactive `/login` (OAuth or API key). Credentials live in `~/.prime/agent/auth.json` (mode `0600`); auth-file entries take priority over environment variables.

```bash
# Interactive recovery
/login
/model

# Or BYOK via environment (provider-specific)
export ANTHROPIC_API_KEY=sk-ant-...
export OPENAI_API_KEY=sk-...
export PRIME_API_KEY=...
prime-agent
```

<Check>
After a successful `/login`, re-send the prompt. Auth status for the provider should no longer report `source: "stale"` / `label: "expired"`.
</Check>

## Provider 401s and stale credentials

Concrete provider auth failures are detected from:

1. Structured diagnostics: `diagnostics[]` entry with `type: "provider_stream_failure"` and `details.kind: "auth"` with status `401` or `403`
2. Bare error text patterns such as `401 status code (no body)` or `401 Unauthorized` combined with auth-related wording

Stream failures are classified into kinds including `auth`, `rate_limit`, `overloaded`, `server_error`, `invalid_request`, `refusal`, `safety`, and `unknown`. Permanent structured kinds (`auth`, `invalid_request`, `refusal`) are not retried indefinitely after the first retry attempt is already in progress.

### Auth stale lifecycle

```text
provider returns 401/403
        │
        ▼
capture AuthSourceToken for current provider source
        │
        ├─ retry.enabled ──► one auto-retry (auth) then mark stale if still failing
        │
        └─ retry disabled / cancelled / exhausted ──► mark auth stale immediately
        │
        ▼
auth_stale event
authStorage: configured=false, source="stale", label="expired"
API key lookup returns undefined for that source
errorMessage includes "Run /login to update credentials."
```

Behavior under repeated 401s (with default-style retry):

| Condition | Observed behavior |
|-----------|-------------------|
| Structured `auth` 401 with retries enabled | At most one auto-retry for that auth failure class; then `auth_stale` and credentials cleared for the failed source |
| Bare `401 status code (no body)` (e.g. `prime-inference`) | Still classified as auth; emits `auth_stale` with `sourceTokens` (often `source: "runtime"`) for daemon clients |
| Retry cancelled during backoff (`abortRetry`) | `auto_retry_end` with `finalError: "Retry cancelled"`; captured auth sources still marked stale |
| Credentials replaced mid-backoff | Prior failed source tokens are marked stale; fresh key is not left as the dead source |
| Final error after 401 then non-auth (e.g. 500) | Captured auth sources still go stale; final assistant error may be the non-auth text but still gets login guidance when stale-mark runs |
| `retry.enabled: false` | No `auto_retry_start`; single request; concrete auth failure still marks stale |

<Warning>
Stale means the runtime will not keep using the failed credential source. Update credentials with `/login` (or a new env / auth-file key) before expecting further provider calls for that provider.
</Warning>

## Network and auto-retry

Agent-level auto-retry runs on `agent_end` when the last assistant message is retryable (or a still-eligible concrete auth failure). Transient text such as `Network connection lost.` is retryable: the session emits `auto_retry_start`, strips the failed assistant message from live agent state (history retains it), waits with exponential backoff, then calls `agent.continue()`.

### Retry settings

Configure under global or project settings (`~/.prime/agent/settings.json` or `.prime/agent/settings.json`):

| Key | Type | Default | Role |
|-----|------|---------|------|
| `retry.enabled` | boolean | `true` | Master switch for agent-level auto-retry |
| `retry.maxRetries` | number | `3` | Maximum agent-level attempts after the first failure |
| `retry.baseDelayMs` | number | `2000` | Backoff base: delay = `baseDelayMs * 2^(attempt-1)` (2s, 4s, 8s, …) |
| `retry.provider.timeoutMs` | number | SDK default | Provider/SDK request timeout |
| `retry.provider.maxRetries` | number | SDK default | Provider/SDK-level retries (separate from agent-level) |
| `retry.provider.maxRetryDelayMs` | number | `60000` | Cap on server-requested retry delay; longer delays fail immediately. `0` disables the cap |

```json
{
  "retry": {
    "enabled": true,
    "maxRetries": 3,
    "baseDelayMs": 2000,
    "provider": {
      "timeoutMs": 3600000,
      "maxRetries": 0,
      "maxRetryDelayMs": 60000
    }
  }
}
```

### What is not auto-retried

| Case | Reason |
|------|--------|
| Context overflow | Handled by compaction, not retry |
| Faux provider queue exhausted (`No more faux responses queued`) | Terminal fixture state |
| `agent_lifecycle_failure` diagnostics | Not a provider transient error |
| Structured permanent provider failure after a retry already started | Avoids spinning on `auth` / `invalid_request` / `refusal` |
| Successful assistant message mid-turn | Retry counter resets on non-error assistant messages |

### Events (interactive, JSON, RPC, SDK)

```json
{
  "type": "auto_retry_start",
  "attempt": 1,
  "maxAttempts": 3,
  "delayMs": 2000,
  "errorMessage": "Network connection lost."
}
```

```json
{
  "type": "auto_retry_end",
  "success": true,
  "attempt": 1
}
```

On final failure, `success` is `false` and `finalError` carries the last error text (or `Retry cancelled` when aborted). RPC exposes `set_auto_retry` to toggle the agent-level switch at runtime.

## Invalid resume selectors

`prime-agent --resume <selector>` and related session open paths resolve selectors through the session resolver (not the daemon attach name path).

### Selector resolution order

1. Path-like selector (`/`, `\`, or `.jsonl` suffix) → open as path  
2. Exact normalized ID match in local project sessions  
3. Exact match in all sessions  
4. Unique prefix/suffix partial match (local, then all)  
5. Otherwise `SessionSelectorNotFoundError`, optionally with a closest-ID suggestion  

Normalization strips hyphens and lowercases IDs. Hex session list UIs often show a 12-character suffix; that suffix is accepted when it uniquely matches.

### Errors and CLI output

| Error | When | User-facing recovery |
|-------|------|----------------------|
| `SessionSelectorNotFoundError` | No match | Message `No session found matching '<selector>'`. If a single closest ID is within edit distance ≤ `max(1, floor(len/5))` and not tied, CLI adds `Did you mean '<id>'?` |
| `SessionSelectorAmbiguousError` | Multiple prefix/suffix matches | Lists matching IDs (and names when present); no auto-pick |
| Missing `--resume` value misuse | Flag without selector where required | Browse with left-arrow / session picker |

```bash
# Mistyped ID → exit 1 with suggestion when confident
prime-agent --resume <almost-right-id>

# Accept the 12-char suffix shown in session lists when unique
prime-agent --resume aaaaaaaaaaaa

# Open picker then send a prompt after selection
prime-agent --resume -- "continue this work"

# Continue most recent without a selector
prime-agent --continue
```

Exact normalized ID wins over partial prefix/suffix matches. Suggestions are suppressed for short selectors (`< 4` normalized chars), ties, or low-confidence distances.

Interactive fallback: open Prime Agent and use left-arrow (or `/resume`) to browse sessions. Daemon Agents View requires the daemon path (not `--no-daemon`).

## Worker recovery and daemon process faults

Resident interactive sessions run in isolated workers under a detached supervisor. Closing the TUI detaches the client; it does not stop the worker.

### Worker crash recovery

- A worker crash is scoped to one root session tree.  
- Recovery retries after **250 ms**, **1 s**, and **5 s**; three failures mark that root failed.  
- Recovery reaps the old process group, appends a visible recovery marker to the transcript, restores the root under the same active-session ID, and does **not** replay uncertain side effects.  
- Only the **current supervisor generation** may replace a crashed resident worker; obsolete generations are fenced.

### Generation fencing

Workers authenticate to the supervisor with a per-worker token and the current supervisor generation. Commands from a displaced generation fail with `supervisor_generation_stale` and must not insert public journal entries for the old owner.

### Client recovery commands

```bash
prime-agent list
prime-agent attach <agent>
prime-agent agents
prime-agent status
prime-agent doctor
prime-agent doctor --fix
prime-agent stop <agent>
prime-agent shutdown
prime-agent shutdown --force
```

| Command | Use when |
|---------|----------|
| `list` / `attach` | Reattach to a live resident worker after client disconnect |
| `status` | Inspect background services |
| `doctor` / `doctor --fix` | Diagnose or repair service / socket state (including orphan sockets) |
| `shutdown --force` | Stop supervisor + workers; required for incompatible/stale daemons that refuse idle replacement, and for unresponsive process groups |

`shutdown --force` serializes shutdown admission and reclaims unrenewed live leases so concurrent force-shutdowns do not race destructively.

## Connection-mode failure probes

`AgentConnection` is the UI/SDK client boundary. Interactive mode normally uses `DaemonAgentConnection`; SDK and explicit fallbacks may use `InProcessAgentConnection`. Failure behavior differs by transport.

### Daemon version probe

Startup/connect probes the local socket (`probeDaemonVersion`):

| Status | Meaning |
|--------|---------|
| `absent` | No daemon accepted a connect within the short probe timeouts |
| `current` | Hello matches client protocol version, schema id, and app version |
| `stale` | Connected but version/schema/app mismatch, or no recognizable hello |

A stale daemon that cannot be auto-replaced (busy sessions) raises `StaleDaemonError` with both daemon and client identity lines and instructs:

```bash
prime-agent shutdown --force
```

then retry the original command.

### Daemon process inventory statuses

`daemon ps` / related discovery classifies sockets as:

| Status | Meaning |
|--------|---------|
| `current` | Compatible live daemon |
| `stale` | Live but version-incompatible |
| `unreachable` | Socket present but not healthy; force shutdown may be required |
| `orphan-file` | Socket file without a live daemon; safe to remove with doctor/fix paths |

### Connection closed / session closed messages

`DaemonAgentConnection` formats fatal messages with session ID, session file (when known), and a diagnostic log path under `~/.prime/agent/logs/` (or the default agent log).

| Situation | Message intent |
|-----------|----------------|
| Socket loss without recovery hook | `Lost connection to the Prime Agent daemon…` transcript remains; restart / Agents View |
| Session `killed` / `shutdown` / `completed` / `replaced` / `update` | Reason-specific closed text + reopen guidance |
| Update reconnect timeout | Daemon restarted for update; window failed to restore before timeout |
| Snapshot transfer failure | Failed attach/resync snapshot recovery with snapshot and recovery causes |

With `recoverDaemon` configured (normal interactive path), transient socket loss triggers bounded reconnect: stable client identity + last event cursor `{ generation, sequence }`, reattach, then `session_resynced` / replacement snapshot. Generation changes invalidate bare sequence comparison; missing replay is non-fatal—the attach snapshot is the durable baseline.

### Mutation idempotency on reconnect

Mutating daemon commands are journaled by `clientId + commandId`:

- Completed command replay returns the stored result  
- Received-but-not-durable results report **uncertain** and are not blindly re-executed  
- Clients acknowledge durable results so journals can compact  

Do not assume a client method promise is a general remote workflow API; treat uncertain results as requiring operator inspection of session state.

### In-process vs daemon

| Mode | Failure surface |
|------|-----------------|
| `DaemonAgentConnection` | Socket, supervisor, worker generation, snapshot streaming, journal |
| `InProcessAgentConnection` | Same process as runtime; no daemon reconnect/journal—errors are local runtime/provider failures |

## Diagnostic artifacts

| Artifact | Location / command | When useful |
|----------|--------------------|-------------|
| Daemon / worker / client / provider logs | `~/.prime/agent/logs/` | Connection closed messages cite the path |
| Agent default log | `getAgentLogPath()` under agent dir | Message-handler and recovery noise |
| TUI layout debug | Hidden `/debug` → `~/.prime/agent/prime-agent-debug.log` | Rendering / width issues (not provider auth) |
| Session JSONL | `~/.prime/agent/sessions/` | Resume path, transcript after worker recovery |
| Settings | `~/.prime/agent/settings.json`, project `.prime/agent/settings.json` | Retry and related knobs |
| Auth store | `~/.prime/agent/auth.json` | Credential presence after `/login` |

## Quick recovery procedures

<Steps>
  <Step title="Provider 401 or auth_stale">
    Confirm the assistant error includes `Run /login to update credentials.` Run `/login` for the provider (OAuth or API key), select a model with `/model` if needed, and re-prompt. For BYOK, update the provider env var or `auth.json` entry and restart if the process still holds a stale runtime key.
  </Step>
  <Step title="Network / transient stream errors">
    Leave `retry.enabled` true and wait for `auto_retry_*` cycles. If retries exhaust, raise `retry.maxRetries` or lower `retry.baseDelayMs` only after confirming the provider is healthy. Cap long server-requested delays with `retry.provider.maxRetryDelayMs`.
  </Step>
  <Step title="Resume selector rejected">
    Copy the suggested session ID from the error, use the list’s 12-character suffix when unique, or run `prime-agent --resume` / left-arrow browse. For ambiguous prefixes, lengthen the selector until one session matches.
  </Step>
  <Step title="Lost daemon connection or stuck workers">
    Run `prime-agent list` and `prime-agent attach <agent>`. If services are unhealthy, `prime-agent doctor` then `doctor --fix`. For incompatible or stuck daemons, `prime-agent shutdown --force`, then start again. Transcripts remain on disk under the sessions directory.
  </Step>
</Steps>

## Related pages

<CardGroup>
  <Card title="Authentication and providers" href="/authentication-providers">
    Login paths, API keys, OAuth, and BYOK boundaries.
  </Card>
  <Card title="Settings and provider keys" href="/settings-providers">
    Provider registration, model selection, and 401 stale-provider recovery.
  </Card>
  <Card title="Run daemon-backed sessions" href="/daemon-sessions">
    Detach/reattach, resume selectors, and worker recovery verification.
  </Card>
  <Card title="Agent connection modes" href="/agent-connection">
    Daemon vs in-process adapters, snapshots, and reconnect semantics.
  </Card>
  <Card title="Sessions and runtime" href="/sessions-runtime">
    Session lifecycle, events, and session-scoped vs durable state.
  </Card>
  <Card title="Session configuration" href="/session-configuration">
    Config keys including retry-related session surfaces.
  </Card>
</CardGroup>