# Model and Provider Abstraction

> The provider-neutral LLM surface: model metadata, stream contracts, lazy provider registration, API-key resolution, OAuth hooks, and custom provider wiring.

- Repository: earendil-works/pi
- GitHub: https://github.com/earendil-works/pi
- Human wiki: https://grok-wiki.com/public/wiki/earendil-works-pi-121d322b171c
- Complete Markdown: https://grok-wiki.com/public/wiki/earendil-works-pi-121d322b171c/llms-full.txt

## Source Files

- `packages/ai/src/types.ts`
- `packages/ai/src/api-registry.ts`
- `packages/ai/src/stream.ts`
- `packages/ai/src/models.ts`
- `packages/ai/src/providers/register-builtins.ts`
- `packages/ai/src/env-api-keys.ts`
- `packages/ai/src/oauth.ts`
- `packages/coding-agent/src/core/model-registry.ts`

---

<details>
<summary>Relevant source files</summary>
The following files were used as context for generating this wiki page:
- [packages/ai/src/types.ts](packages/ai/src/types.ts)
- [packages/ai/src/api-registry.ts](packages/ai/src/api-registry.ts)
- [packages/ai/src/stream.ts](packages/ai/src/stream.ts)
- [packages/ai/src/models.ts](packages/ai/src/models.ts)
- [packages/ai/src/models.generated.ts](packages/ai/src/models.generated.ts)
- [packages/ai/src/providers/register-builtins.ts](packages/ai/src/providers/register-builtins.ts)
- [packages/ai/src/env-api-keys.ts](packages/ai/src/env-api-keys.ts)
- [packages/ai/src/oauth.ts](packages/ai/src/oauth.ts)
- [packages/ai/src/utils/oauth/types.ts](packages/ai/src/utils/oauth/types.ts)
- [packages/ai/src/utils/oauth/index.ts](packages/ai/src/utils/oauth/index.ts)
- [packages/ai/src/utils/event-stream.ts](packages/ai/src/utils/event-stream.ts)
- [packages/coding-agent/src/core/model-registry.ts](packages/coding-agent/src/core/model-registry.ts)
- [packages/coding-agent/src/core/auth-storage.ts](packages/coding-agent/src/core/auth-storage.ts)
- [packages/coding-agent/src/core/extensions/types.ts](packages/coding-agent/src/core/extensions/types.ts)
- [packages/coding-agent/src/core/resolve-config-value.ts](packages/coding-agent/src/core/resolve-config-value.ts)
</details>

# Model and Provider Abstraction

This page explains pi’s provider-neutral LLM surface: how model metadata is represented, how streaming calls are routed by API shape rather than vendor name, how built-in providers are lazily registered, and how the coding agent layers custom providers, API keys, headers, and OAuth on top.

Generation note: no local `STRATEGY.md`, `docs/solutions/**`, or prior generated wiki files were present in this checkout. Page shape guidance came from the bundled Compound Engineering wiki guidance, while implementation claims below are sourced from repository code.

## Architecture at a Glance

pi separates **model metadata** from **stream implementation**. A `Model` carries the requested provider, model id, API type, base URL, pricing, capabilities, and compatibility hints. Runtime calls use the model’s `api` to look up an API provider, so several commercial, local, proxy, or extension-backed providers can share one protocol adapter.

```mermaid
flowchart TD
  subgraph Metadata["Model metadata"]
    Generated["packages/ai/src/models.generated.ts\nMODELS"]
    Models["packages/ai/src/models.ts\ngetModels(), getModel()"]
    AgentRegistry["packages/coding-agent/src/core/model-registry.ts\nbuilt-ins + models.json + extensions"]
  end

  subgraph Runtime["Provider-neutral runtime"]
    Stream["packages/ai/src/stream.ts\nstream(), complete(), streamSimple()"]
    ApiRegistry["packages/ai/src/api-registry.ts\napi -> stream handlers"]
    Builtins["packages/ai/src/providers/register-builtins.ts\nlazy built-in adapters"]
  end

  subgraph Auth["Request auth"]
    AuthStorage["auth-storage.ts\nruntime/stored/OAuth/env/fallback"]
    EnvKeys["env-api-keys.ts\nknown env and ambient auth"]
    OAuth["utils/oauth/*\nlogin, refresh, getApiKey, modifyModels"]
  end

  Generated --> Models --> AgentRegistry
  AgentRegistry --> Stream
  Stream --> ApiRegistry --> Builtins
  AgentRegistry --> AuthStorage
  AuthStorage --> EnvKeys
  AuthStorage --> OAuth
```

Sources: [packages/ai/src/models.generated.ts:1-24](), [packages/ai/src/models.ts:1-37](), [packages/ai/src/stream.ts:1-58](), [packages/ai/src/api-registry.ts:23-98](), [packages/ai/src/providers/register-builtins.ts:162-406](), [packages/coding-agent/src/core/model-registry.ts:384-630]()

## Core Types and Model Metadata

`types.ts` defines open-ended API and provider identifiers. Known APIs and providers are enumerated, but `Api` and `Provider` also admit strings, which is what makes extension and custom-provider wiring possible without changing the core type list.

A `Model<TApi>` contains:

| Field group | Purpose |
| --- | --- |
| Identity | `id`, `name`, `provider`, `api` |
| Routing | `baseUrl`, optional `headers`, optional protocol-specific `compat` |
| Capabilities | `reasoning`, `thinkingLevelMap`, `input` |
| Limits | `contextWindow`, `maxTokens` |
| Cost tracking | per-million-token `input`, `output`, `cacheRead`, `cacheWrite` |

Compatibility metadata is intentionally protocol-specific. `OpenAICompletionsCompat`, `OpenAIResponsesCompat`, and `AnthropicMessagesCompat` let custom or proxy providers override behavior such as max-token field names, reasoning formats, cache retention, session-affinity headers, and Anthropic tool/cache behavior without creating a new stream adapter.

Sources: [packages/ai/src/types.ts:6-50](), [packages/ai/src/types.ts:365-432](), [packages/ai/src/types.ts:538-566]()

## Built-in Model Registry

Built-in metadata is generated into `MODELS` and loaded into an in-memory `Map<provider, Map<modelId, Model<Api>>>` at module initialization. `getProviders()`, `getModels(provider)`, and `getModel(provider, modelId)` expose that generated catalog. Utility functions then compute usage cost from model pricing and normalize thinking-level support.

The generated file is data, not behavior: each entry satisfies the `Model` shape and records the API adapter to use, such as `bedrock-converse-stream`, plus provider, base URL, reasoning support, input modalities, cost, context window, and max tokens.

Sources: [packages/ai/src/models.generated.ts:1-24](), [packages/ai/src/models.ts:1-91]()

## Stream Contracts

The central runtime contract is `StreamFunction<TApi, TOptions>`. It must return an `AssistantMessageEventStream`; after invocation, request/model/runtime failures are expected to be represented as stream events rather than thrown. A stream begins with `start`, can emit text, thinking, and tool-call deltas, and ends with either `done` or `error`.

`complete()` and `completeSimple()` are thin convenience wrappers over streaming: they call `stream()` or `streamSimple()` and await the stream’s final `result()`.

Sources: [packages/ai/src/types.ts:84-210](), [packages/ai/src/types.ts:277-359](), [packages/ai/src/utils/event-stream.ts:21-80](), [packages/ai/src/stream.ts:25-58]()

### Full vs. Simple Streaming

| API | Purpose |
| --- | --- |
| `stream(model, context, options)` | Provider-specific options through `ProviderStreamOptions`. |
| `complete(model, context, options)` | Await final `AssistantMessage` from `stream()`. |
| `streamSimple(model, context, options)` | Common options plus `reasoning` and `thinkingBudgets`. |
| `completeSimple(model, context, options)` | Await final `AssistantMessage` from `streamSimple()`. |

Sources: [packages/ai/src/types.ts:190-210](), [packages/ai/src/stream.ts:25-58]()

## API Provider Registry

`api-registry.ts` maps API identifiers to stream implementations. Registration stores both `stream` and `streamSimple`, optionally tagged with a `sourceId` so dynamically registered handlers can be removed later. The wrapper functions enforce that `model.api` matches the registered API before dispatching.

This means provider names and API shapes are separate. A custom provider can reuse `openai-completions`, `openai-responses`, `anthropic-messages`, or another registered API as long as its model metadata points at that API and its base URL/auth settings are valid.

Sources: [packages/ai/src/api-registry.ts:23-98](), [packages/ai/src/types.ts:538-566]()

## Lazy Built-in Provider Registration

Importing `stream.ts` imports `providers/register-builtins.ts`, whose module-level `registerBuiltInApiProviders()` call registers built-in API adapters. The adapters are lazy wrappers: the initial registry entry exists immediately, but the provider module is imported only when a request uses that API. Lazy load failures are converted into an `error` stream event with an `AssistantMessage` containing `stopReason: "error"`.

Built-in registered APIs include Anthropic Messages, OpenAI Completions, Mistral Conversations, OpenAI Responses, Azure OpenAI Responses, OpenAI Codex Responses, Google Generative AI, Google Vertex, and Bedrock Converse Stream. Bedrock uses a node-only import path helper and supports an override module for nonstandard runtime loading.

Sources: [packages/ai/src/stream.ts:1-23](), [packages/ai/src/providers/register-builtins.ts:89-92](), [packages/ai/src/providers/register-builtins.ts:162-204](), [packages/ai/src/providers/register-builtins.ts:206-406]()

## API Key and Header Resolution

There are two layers of auth resolution:

1. `packages/ai` knows common environment variables and ambient provider auth markers.
2. `packages/coding-agent` combines stored auth, OAuth, environment variables, `models.json`, and extension-provided request configuration.

`env-api-keys.ts` maps known providers to environment variables. Anthropic checks `ANTHROPIC_OAUTH_TOKEN` before `ANTHROPIC_API_KEY`; Vertex can report authenticated status through Application Default Credentials plus project/location; Bedrock can report authenticated status through AWS profile, IAM keys, bearer token, ECS, or web identity environment sources.

Sources: [packages/ai/src/env-api-keys.ts:91-221]()

`AuthStorage.getApiKey()` resolves credentials in this order: runtime override, stored API key, stored OAuth token with refresh, environment variable, and optional fallback resolver. `ModelRegistry.getApiKeyAndHeaders()` then combines that result with provider-level `apiKey`, provider headers, model headers, and static model headers. If `authHeader` is enabled, it injects `Authorization: Bearer <apiKey>` and returns an error if no key is available.

Sources: [packages/coding-agent/src/core/auth-storage.ts:454-522](), [packages/coding-agent/src/core/model-registry.ts:685-721](), [packages/coding-agent/src/core/resolve-config-value.ts:10-136]()

## OAuth Hooks

OAuth support is provider-neutral. An `OAuthProviderInterface` supplies:

| Hook | Responsibility |
| --- | --- |
| `login(callbacks)` | Run the provider’s login flow and return credentials to persist. |
| `refreshToken(credentials)` | Refresh expired credentials. |
| `getApiKey(credentials)` | Convert credentials into the API key/token string used for requests. |
| `modifyModels(models, credentials)` | Optionally rewrite model metadata, such as base URLs, after login. |

The OAuth registry starts with built-in Anthropic, GitHub Copilot, and OpenAI Codex providers. Custom OAuth providers can be registered, unregistered, reset to built-ins, and queried. `getOAuthApiKey()` refreshes expired credentials before returning the request token.

Sources: [packages/ai/src/utils/oauth/types.ts:43-72](), [packages/ai/src/utils/oauth/index.ts:29-151](), [packages/ai/src/oauth.ts:1]()

The coding-agent model registry applies OAuth in two places: while loading models, it lets stored OAuth providers modify the combined built-in/custom model list; when a dynamic provider includes `oauth`, it registers the OAuth provider under the dynamic provider name and can apply `modifyModels` after credentials exist.

Sources: [packages/coding-agent/src/core/model-registry.ts:384-410](), [packages/coding-agent/src/core/model-registry.ts:860-930]()

## Custom Provider Wiring

Custom providers can be declared in `models.json` or registered dynamically by extensions. The schema accepts provider-level `name`, `baseUrl`, `apiKey`, `api`, `headers`, `compat`, `authHeader`, `models`, and `modelOverrides`. Model definitions can specify model identity, API/base URL overrides, reasoning support, input modalities, cost, context window, max tokens, headers, and compatibility overrides.

For non-built-in providers with models, validation requires `baseUrl`, `apiKey`, and an API at the provider or model level. Built-in providers can inherit API and base URL defaults from existing models. Parsed custom models default missing `name` to `id`, `input` to `["text"]`, cost to zeros, `contextWindow` to `128000`, and `maxTokens` to `16384`.

Sources: [packages/coding-agent/src/core/model-registry.ts:144-202](), [packages/coding-agent/src/core/model-registry.ts:459-630]()

### Dynamic Extension Registration

Extensions expose `pi.registerProvider(name, config)` and `pi.unregisterProvider(name)`. Registration can add a new provider, override an existing provider’s base URL, include OAuth support, or supply a `streamSimple` handler for a custom API. During initial extension load, provider registrations are queued and then flushed when the runner binds context; afterward, calls take effect immediately.

In `ModelRegistry.registerProvider()`, providers with models replace all current models for that provider. Providers with only `baseUrl` or `headers` update existing models/request config. If `streamSimple` is provided, the registry also registers an API provider under a source id of `provider:<providerName>`, allowing refresh/unregister flows to rebuild dynamic API registrations cleanly.

Sources: [packages/coding-agent/src/core/extensions/types.ts:1248-1382](), [packages/coding-agent/src/core/extensions/runner.ts:294-336](), [packages/coding-agent/src/core/model-registry.ts:789-957]()

## Implementation Responsibilities

| Module | Responsibility |
| --- | --- |
| `packages/ai/src/types.ts` | Provider-neutral model, context, message, options, stream, compatibility, and OAuth-facing types. |
| `packages/ai/src/models.ts` | Load generated model metadata and expose catalog helpers. |
| `packages/ai/src/api-registry.ts` | Register API-shape stream handlers and enforce model/API matching. |
| `packages/ai/src/stream.ts` | Public stream/complete entry points that resolve adapters by `model.api`. |
| `packages/ai/src/providers/register-builtins.ts` | Lazy registration and reset of built-in API adapters. |
| `packages/ai/src/env-api-keys.ts` | Known provider environment-variable and ambient-auth discovery. |
| `packages/ai/src/utils/oauth/*` | OAuth provider registry, credential refresh, and token extraction interfaces. |
| `packages/coding-agent/src/core/model-registry.ts` | Merge built-ins, `models.json`, dynamic providers, OAuth model mutation, request auth, and custom headers. |
| `packages/coding-agent/src/core/auth-storage.ts` | Persisted/runtime/env/OAuth/fallback credential resolution. |

Sources: [packages/ai/src/types.ts:84-210](), [packages/ai/src/models.ts:1-91](), [packages/ai/src/api-registry.ts:23-98](), [packages/ai/src/stream.ts:1-58](), [packages/coding-agent/src/core/model-registry.ts:335-957](), [packages/coding-agent/src/core/auth-storage.ts:454-522]()

## Summary

pi’s abstraction is BYOC/BYOK-friendly because the stable boundary is the API protocol plus model metadata, not a hardcoded hosted provider. Built-ins are registered lazily, custom providers can reuse or add API handlers, auth can come from runtime keys, stored credentials, OAuth, environment variables, command-backed config values, or model config, and OAuth hooks can adjust model metadata after login. Sources: [packages/ai/src/api-registry.ts:23-98](), [packages/coding-agent/src/core/model-registry.ts:789-957]()