# Providers and models

> Built-in and dynamic providers, scoped model order and refresh, models.json hot reload, and provider-retry message behavior.

- Repository: earendil-works/pi
- GitHub: https://github.com/earendil-works/pi
- Human docs: https://grok-wiki.com/public/docs/earendil-works-pi-7860a70e44d1
- Complete Markdown: https://grok-wiki.com/public/docs/earendil-works-pi-7860a70e44d1/llms-full.txt

## Source Files

- `packages/coding-agent/test/agent-session-dynamic-provider.test.ts`
- `packages/coding-agent/test/suite/agent-session-model-extension.test.ts`
- `packages/coding-agent/test/suite/regressions/3217-scoped-model-order.test.ts`
- `packages/coding-agent/test/suite/regressions/6999-models-json-hot-reload.test.ts`
- `packages/coding-agent/test/suite/regressions/7153-scoped-models-refresh.test.ts`
- `packages/coding-agent/test/suite/regressions/6949-unavailable-scoped-model.test.ts`

---

---
title: "Providers and models"
description: "Built-in and dynamic providers, scoped model order and refresh, models.json hot reload, and provider-retry message behavior."
---

`ModelRuntime` is the coding-agent surface that composes built-in catalogs, `~/.pi/agent/models.json`, extension-registered providers, and credential availability into a single model list for sessions, CLI selection, and SDK embedding.

## Runtime surface

| Surface | Role |
|---------|------|
| `ModelRuntime` | Owns providers, model snapshots, refresh, auth checks, and streaming |
| `ModelRegistry` | Sync compatibility facade for extensions; delegates to `ModelRuntime` |
| `ModelConfig` | Immutable load/validate of `models.json` |
| `FileModelsStore` | Locked cache at `models-store.json` for dynamic remote catalogs |
| `composeModelProvider` | Layers built-in → `models.json` → extension → `modelOverrides` |

Default paths (under the agent config dir, typically `~/.pi/agent/`):

| Path | Contents |
|------|----------|
| `models.json` | User provider/model config (hot-reloaded on refresh) |
| `models-store.json` | Cached dynamic catalogs for offline reuse |
| `auth.json` | API keys and OAuth credentials (see [Authentication](/authentication)) |

Create a runtime:

```ts
import { ModelRuntime, createAgentSession } from "@earendil-works/pi-coding-agent";

const modelRuntime = await ModelRuntime.create({
  // modelsPath defaults to ~/.pi/agent/models.json
  // allowModelNetwork: true  // optional create-time network catalog refresh
  // refreshOnCreate: false   // skip initial refresh; static models still load
});

const available = await modelRuntime.getAvailable();
const model = modelRuntime.getModel("anthropic", "claude-opus-4-8");
```

<ParamField body="allowModelNetwork" type="boolean">
When `true` and not offline, create-time refresh may hit the network. Default `false`.
</ParamField>

<ParamField body="refreshOnCreate" type="boolean">
When not `false`, `create()` calls `refresh()`. Static catalogs remain usable if skipped.
</ParamField>

<ParamField body="modelsPath" type="string | null">
Path to `models.json`. `null` disables the file. Default: agent dir `models.json`.
</ParamField>

Offline: `--offline` or `PI_OFFLINE=1` disables network model operations (`PI_OFFLINE` also forces `modelNetworkEnabled` off).

## Provider composition

Each provider id is recomposed from up to four layers:

```text
builtin / native extension base
        │
        ▼
  models.json (baseUrl, models, apiKey, oauth: "radius", …)
        │
        ▼
  extension registerProvider config (legacy config or native Provider)
        │
        ▼
  models.json modelOverrides  ← topmost user-config layer
```

Rules:

- **Untouched built-in**: if only a built-in exists (no `models.json` entry, no extension overlay), the native provider is used as-is so auth/login/stream behavior stays exact.
- **Native extension provider**: `registerProvider(provider: Provider)` or `registerNativeProvider` replaces the base for that id and clears any prior legacy extension config for the same id.
- **Legacy config form**: `registerProvider(name, config)` merges defined fields over a previous registration; `undefined` fields are preserved.
- **models.json overrides**: `modelOverrides` apply after custom-model upserts and extension model lists.
- **Broken recompose**: composition errors are recorded on the runtime; a healthy base falls back when present.

Availability: models appear in `/model`, `cycleModel`, and `--list-models` only when the provider has configured auth (stored credential, environment key, runtime key, or resolved `models.json` `apiKey` presence). Shell-command `apiKey` values are treated as configured for availability without executing the command.

## Built-in providers

Built-ins come from `@earendil-works/pi-ai` catalogs. Most wrap with remote catalog refresh and cache into `models-store.json`. Radius stays special: custom gateways can be declared in `models.json` with `"oauth": "radius"` and a gateway `baseUrl`.

Default model ids per known provider live in `defaultModelPerProvider` (used when building fallback model objects for unknown ids on a known provider). Auth is separate: OAuth via `/login`, API keys via env/`auth.json`, or ambient cloud credentials. See [Authentication](/authentication).

CLI selection:

```bash
pi --provider anthropic --model claude-opus-4-8
pi --model openai/gpt-5.5
pi --model sonnet:high
pi --list-models
pi --list-models sonnet
```

## Dynamic providers

### models.json

Add or override providers under `~/.pi/agent/models.json`. Schema is validated by `ModelConfig` (`providers` record). Supported API types include `openai-completions`, `openai-responses`, `anthropic-messages`, and `google-generative-ai`.

Minimal local example:

```json
{
  "providers": {
    "ollama": {
      "baseUrl": "http://localhost:11434/v1",
      "api": "openai-completions",
      "apiKey": "ollama",
      "compat": {
        "supportsDeveloperRole": false,
        "supportsReasoningEffort": false
      },
      "models": [{ "id": "llama3.1:8b", "reasoning": true }]
    }
  }
}
```

| Field | Notes |
|-------|--------|
| `baseUrl` | Required for non-built-in custom models |
| `api` | Provider or per-model |
| `apiKey` | Literal, `$ENV`, `${ENV}`, `!command`, or omit if auth is elsewhere |
| `oauth` | `"radius"` only; requires `baseUrl` |
| `headers` / `authHeader` | Same value-resolution rules as `apiKey` |
| `models` | Upsert by model `id` onto base catalog |
| `modelOverrides` | Patch built-in or extension models by id |
| `compat` | Provider/API compatibility flags (merged model-over-provider) |

Model defaults when omitted: `name` → `id`, `reasoning` → `false`, `input` → `["text"]`, `contextWindow` → `128000`, `maxTokens` → `16384`, zero costs.

### Extension registration

Extensions call `pi.registerProvider` in two forms:

```ts
// Native pi-ai Provider (preferred for custom auth/stream)
pi.registerProvider(createProvider({ id: "native-local", /* … */ }));

// Legacy config: override or add models
pi.registerProvider("anthropic", { baseUrl: "https://proxy.example.com" });
pi.registerProvider("my-provider", {
  baseUrl: "https://api.example.com",
  api: "openai-completions",
  apiKey: "$MY_API_KEY",
  models: [/* … */],
  refreshModels: async (ctx) => [/* dynamic list */],
});
```

Registration timing:

| When | Behavior |
|------|----------|
| Extension factory (top-level) | Available at session construction and `--list-models` |
| `session_start` handler | Applied after bind; active model re-read from runtime |
| Slash command handler | Applied immediately; no full resource reload |

On register/unregister, the session refreshes the **current** model object from the runtime so `baseUrl`/catalog changes apply to the next stream without a restart.

`unregisterProvider(name)` removes extension overlays and recomposes the built-in/`models.json` base.

## Scoped models

Scoped models limit which models are available for cycling (`cycleModel` / Ctrl+P) and optional default selection.

### Sources

Patterns come from, in practice:

1. CLI `--models <patterns>` (comma-separated)
2. Else settings `enabledModels` (same pattern language)

```bash
pi --models claude-sonnet,claude-haiku,gpt-4o
pi --models "github-copilot/*"
pi --models sonnet:high,haiku:low
```

Pattern rules (`resolveModelScope` / `resolveModelScopeFromModels`):

- Exact `provider/id` or unambiguous bare id
- Glob (`*`, `?`, `[…]`) against `provider/id` or id
- Optional `:thinkingLevel` suffix (`off`, `minimal`, `low`, `medium`, `high`, `xhigh`, `max`)
- Prefer alias ids over dated variants when partial-matching
- Unmatched patterns warn; order of **matched** models follows pattern list order

Startup without an existing session and without `--model`: if a scope exists, prefer the saved default when it is in scope; otherwise use the first scoped model (and its explicit thinking level when set). Explicit `--thinking` wins over scoped thinking levels.

### Session order and cycling

`AgentSession.setScopedModels` stores an ordered list. `cycleModel`:

- Uses scoped list when non-empty (filtering to currently available ids)
- Otherwise cycles all available models
- Explicit scoped `thinkingLevel` overrides the session level; `undefined` inherits the current session preference (then clamps to model capability)

Interactive **Model Configuration** (`ScopedModelsSelectorComponent`):

- Session-only until save (settings `enabledModels`)
- Reorder enabled ids (order is preserved in the `/model` scoped tab)
- Toggle/enable-all/clear
- Persist only via the save keybinding

### Unavailable scoped entries

Enabled ids that no longer resolve in the catalog still appear as `provider/id [unavailable] ✗`. They can be toggled off and persisted. A partial scope is **not** cleared merely because one enabled id is missing.

### Background refresh in selectors

Opening model selectors:

1. Renders the current `getAvailableSnapshot()` immediately
2. Starts `modelRuntime.refresh({ signal })` with a 15s abort timeout
3. Updates the list when refresh completes
4. Shows status: refreshing → refreshed / timed out / per-catalog errors
5. Closing the selector aborts the in-flight refresh

## models.json hot reload

`ModelRuntime.refresh()` always reloads `models.json` from disk via `ModelConfig.load`, recomposes providers, refreshes dynamic catalogs (when network allowed), and rebuilds the availability snapshot.

Implications:

- Edit `models.json` during a session; open `/model` (or any path that calls refresh) to pick up changes—no process restart
- Invalid schema/parse errors surface through `getError()`; prior healthy state may remain until a successful load
- Missing file is empty config (not an error)
- Extension `registerProvider` triggers a non-network `refresh({ allowNetwork: false })` so composition stays consistent

```ts
const result = await modelRuntime.refresh({ signal, allowNetwork: true, providers: ["openrouter"] });
// result.aborted, result.errors: Map<providerId, Error>
```

## Auto-retry and provider retry messages

Two related retry layers exist.

### Agent-turn auto-retry

`AgentSession` retries assistant errors classified by `isRetryableAssistantError` (from `@earendil-works/pi-ai`), **except** context-overflow errors (those go through compaction).

Settings (`settings.retry`):

| Key | Default | Meaning |
|-----|---------|---------|
| `enabled` | `true` | Master switch |
| `maxRetries` | `3` | Max agent-turn retry attempts |
| `baseDelayMs` | `2000` | Exponential backoff base (2s, 4s, 8s, …) |

Events:

- `auto_retry_start` — `attempt`, `maxAttempts`, `delayMs`, `errorMessage` (provider text preserved)
- `auto_retry_end` — `success`, `attempt`, optional `finalError` (`"Retry cancelled"` if aborted)

Explicit provider retry guidance is treated as retryable. Verified examples include OpenAI help-center retry text and Bedrock “Try your request again” payloads: when `retry.enabled` is true, the agent re-prompts after backoff instead of failing the turn on the first error.

### Provider/SDK retry settings

Nested `settings.retry.provider`:

| Key | Default | Meaning |
|-----|---------|---------|
| `timeoutMs` | unset | Provider request timeout |
| `maxRetries` | unset | SDK/provider-level retries |
| `maxRetryDelayMs` | `60000` | Cap for server-requested delays |

Project and global provider retry objects merge field-wise (project overrides individual keys without wiping the rest of the nested object). See [Settings](/settings).

RPC: `set_auto_retry` / `abort_retry` control the agent-turn layer.

## Interactive and CLI commands

| Command / flag | Behavior |
|----------------|----------|
| `/model` | Model selector; snapshot first, then hot-reload refresh |
| Model configuration UI | Scope enable/order/save to `enabledModels` |
| `/login` / `/logout` | Credentials; then availability refresh |
| `--models` | Scope patterns for this run |
| `--list-models [search]` | Print available models (optional fuzzy filter) |
| `--offline` / `PI_OFFLINE` | No network catalog refresh |

## SDK notes

```ts
const modelRuntime = await ModelRuntime.create();
const available = await modelRuntime.getAvailable();

const { session } = await createAgentSession({
  model: available[0],
  thinkingLevel: "medium",
  scopedModels: [{ model: available[0], thinkingLevel: "high" }, { model: available[1] }],
  modelRuntime,
});

await session.setModel(available[1]); // requires configured auth
await session.cycleModel();           // respects scoped list order
session.setScopedModels(/* … */);
```

`setModel` throws if the provider has no configured API key/auth. Extension `ctx.setModel` returns `false` when auth is missing instead of throwing.

## Failure modes

| Symptom | Likely cause |
|---------|----------------|
| Model missing from `/model` | No auth for provider; or catalog not refreshed |
| `models.json` ignored after edit | Selector/refresh not opened; wait for background refresh |
| `Invalid models.json schema` | `ModelConfig` validation failure—fix schema and re-open `/model` |
| `Provider "x": …` in errors | Composition error; base may still be used |
| Scoped list empty after patterns | Patterns matched nothing (warnings on CLI resolve) |
| Unavailable ✗ entries stick | Expected; remove and save to clean `enabledModels` |
| Retry not firing | `retry.enabled: false`, non-retryable error, or overflow path |
| Refresh shows cached only | Timeout (15s), `aborted`, or network/`PI_OFFLINE` |

## Related pages

<CardGroup>
  <Card title="Authentication" href="/authentication">
    API keys, OAuth login, credential storage, and ambient auth.
  </Card>
  <Card title="Settings" href="/settings">
    Settings load/reload and merge rules for `retry` and `enabledModels`.
  </Card>
  <Card title="Extensions" href="/extensions">
    Extension registration, including `registerProvider` lifecycle.
  </Card>
  <Card title="SDK" href="/sdk">
    Embed with custom models, tools, and settings.
  </Card>
  <Card title="SDK examples" href="/sdk-examples">
    Copy-paste recipes including custom model selection.
  </Card>
  <Card title="Troubleshooting" href="/troubleshooting">
    Retry, network, and credential failure patterns.
  </Card>
</CardGroup>