# Local and custom providers

> OpenAI-compatible custom endpoints (LLM_SERVER_*), Ollama local or cloud, config path YAML, legacy and preserve reasoning flags, and aggregator endpoints such as OpenRouter and DeepInfra.

- Repository: vxcontrol/pentagi
- GitHub: https://github.com/vxcontrol/pentagi
- Human docs: https://grok-wiki.com/public/docs/vxcontrol-pentagi-eca0cf4de5f5
- Complete Markdown: https://grok-wiki.com/public/docs/vxcontrol-pentagi-eca0cf4de5f5/llms-full.txt

## Source Files

- `backend/pkg/providers/custom/custom.go`
- `backend/pkg/providers/ollama/ollama.go`
- `backend/pkg/providers/ollama/config.yml`
- `examples/configs/custom-openai.provider.yml`
- `examples/configs/openrouter.provider.yml`
- `examples/configs/ollama-cloud.provider.yml`
- `backend/docs/ollama.md`
- `.env.example`

---

---
title: "Local and custom providers"
description: "OpenAI-compatible custom endpoints (LLM_SERVER_*), Ollama local or cloud, config path YAML, legacy and preserve reasoning flags, and aggregator endpoints such as OpenRouter and DeepInfra."
---

PentAGI registers two BYOK/BYOC-style providers for endpoints you control: **`custom`** (`ProviderCustom`) speaks the OpenAI Chat Completions protocol via `LLM_SERVER_*`, and **`ollama`** (`ProviderOllama`) talks to a local Ollama server or Ollama Cloud via `OLLAMA_SERVER_*`. Both load per-agent model and sampling settings from YAML at `LLM_SERVER_CONFIG_PATH` / `OLLAMA_SERVER_CONFIG_PATH` (or built-in defaults), then expose the same agent option keys as first-party providers.

## When each provider activates

At process start, `backend/pkg/providers/providers.go` builds default agent configs for every provider type, then instantiates only those that pass connection gates:

| Provider type | Gate | Default name |
| --- | --- | --- |
| `custom` | `LLM_SERVER_URL` is set **and** either `LLM_SERVER_MODEL` or `LLM_SERVER_CONFIG_PATH` is set | `custom` |
| `ollama` | `OLLAMA_SERVER_URL` is set | `ollama` |

<Warning>
Setting only `LLM_SERVER_URL` without a model or config path does **not** register the custom provider. Ollama needs only a non-empty server URL.
</Warning>

```mermaid
flowchart LR
  subgraph env [Environment]
    LLM["LLM_SERVER_*"]
    OLL["OLLAMA_SERVER_*"]
  end
  subgraph runtime [Provider factory]
    GateC{"URL and model or config?"}
    GateO{"URL set?"}
    Custom["custom.New<br/>openai.LLM client"]
    Ollama["ollama.New<br/>ollama.LLM + api.Client"]
  end
  subgraph yaml [Agent YAML]
    CfgC["LLM_SERVER_CONFIG_PATH<br/>or empty skeleton"]
    CfgO["OLLAMA_SERVER_CONFIG_PATH<br/>or embedded config.yml"]
  end
  LLM --> GateC
  GateC -->|yes| Custom
  CfgC --> Custom
  OLL --> GateO
  GateO -->|yes| Ollama
  CfgO --> Ollama
```

## Custom OpenAI-compatible provider

Implementation: `backend/pkg/providers/custom`. The client is `langchaingo/llms/openai` pointed at `LLM_SERVER_URL` with bearer token `LLM_SERVER_KEY` and base model `LLM_SERVER_MODEL`. HTTP traffic uses the shared proxy-aware client from `system.GetHTTPClient`.

### Environment variables

| Variable | Config field | Default | Role |
| --- | --- | --- | --- |
| `LLM_SERVER_URL` | `LLMServerURL` | *(empty)* | Base URL of the OpenAI-compatible API (must include path prefix such as `/v1` when the upstream expects it) |
| `LLM_SERVER_KEY` | `LLMServerKey` | *(empty)* | API key / bearer token |
| `LLM_SERVER_MODEL` | `LLMServerModel` | *(empty)* | Default model for all agents when YAML does not override |
| `LLM_SERVER_CONFIG_PATH` | `LLMServerConfig` | *(empty)* | Path to per-agent YAML; empty uses a skeleton of empty agent blocks |
| `LLM_SERVER_PROVIDER` | `LLMServerProvider` | *(empty)* | Optional model-name prefix for LiteLLM-style proxies (`prefix/model`) |
| `LLM_SERVER_LEGACY_REASONING` | `LLMServerLegacyReasoning` | `false` | Reasoning parameter wire format |
| `LLM_SERVER_PRESERVE_REASONING` | `LLMServerPreserveReasoning` | `false` | Keep reasoning content across multi-turn tool calls |

Docker Compose also supports a **host path** mount variable (maps into the container config tree):

| Host / compose variable | Container path (default target) |
| --- | --- |
| `PENTAGI_LLM_SERVER_CONFIG_PATH` | `/opt/pentagi/conf/custom.provider.yml` (default host file: `./example.custom.provider.yml`) |

Set the in-container path on `LLM_SERVER_CONFIG_PATH` to match the mount, for example:

```bash
PENTAGI_LLM_SERVER_CONFIG_PATH=./examples/configs/openrouter.provider.yml
LLM_SERVER_CONFIG_PATH=/opt/pentagi/conf/custom.provider.yml
```

### Default call options

When building the provider config, custom defaults are:

- `temperature`: `1.0`
- `top_p`: `1.0`
- `n`: `1`
- `max_tokens`: `16384`
- `model`: `LLM_SERVER_MODEL` when that env is non-empty

YAML agent blocks override these per agent type.

### Reasoning flags

When `LLM_SERVER_LEGACY_REASONING` is **`false`** (default), the OpenAI client enables modern reasoning options:

- `WithUsingReasoningMaxTokens()`
- `WithModernReasoningFormat()`

When **`true`**, those options are omitted so the client uses the legacy string-style `reasoning_effort` path. Use `true` for official OpenAI reasoning-compatible endpoints when modern object-shaped reasoning fails.

When `LLM_SERVER_PRESERVE_REASONING` is **`true`**, the client sets `WithPreserveReasoningContent()`. Enable this for providers (for example Moonshot-class APIs) that reject multi-turn tool-call histories if assistant reasoning content is stripped.

| Setting | `false` (default) | `true` |
| --- | --- | --- |
| `LLM_SERVER_LEGACY_REASONING` | Modern structured reasoning (`max_tokens` object style) | Legacy `reasoning_effort` string style |
| `LLM_SERVER_PRESERVE_REASONING` | Do not keep reasoning in conversation history | Preserve reasoning content for subsequent turns |

### Model discovery and LiteLLM prefix

On init, custom calls `provider.LoadModelsFromHTTP(baseURL, apiKey, httpClient, prefix)`:

- `GET {baseURL}/models` with optional `Authorization: Bearer …`
- 3-second timeout
- On failure, the models list falls back to empty (provider still starts)

When `LLM_SERVER_PROVIDER` is set (for example `moonshot`):

- Discovery **filters** model IDs that start with `prefix/`
- Stored names strip the prefix
- `ModelWithPrefix` re-applies `prefix/model` at call time via `ApplyModelPrefix`

That lets one YAML file work for both direct vendor URLs and a LiteLLM (or similar) proxy that namespaces models.

### Config path behavior

```text
LLM_SERVER_CONFIG_PATH empty?
  yes → pconfig.EmptyProviderConfigRaw (all agent keys present, empty objects)
  no  → os.ReadFile(path) → pconfig.LoadConfigData
```

Agent keys in YAML (same surface as other providers):

`simple`, `simple_json`, `primary_agent`, `assistant`, `generator`, `refiner`, `adviser`, `reflector`, `searcher`, `enricher`, `coder`, `installer`, `pentester`

Common fields: `model`, `temperature`, `top_p`, `n`, `max_tokens`, `json`, `reasoning` (`effort` and/or `max_tokens`), `price` (`input` / `output`).

## Ollama provider (local and cloud)

Implementation: `backend/pkg/providers/ollama`. Uses `langchaingo/llms/ollama` for inference and the official `ollama/api` client for list/show/pull.

### Environment variables

| Variable | Config field | Default | Role |
| --- | --- | --- | --- |
| `OLLAMA_SERVER_URL` | `OllamaServerURL` | *(empty)* | Local base URL or cloud `https://ollama.com` |
| `OLLAMA_SERVER_API_KEY` | `OllamaServerAPIKey` | *(empty)* | Required for Ollama Cloud; leave empty for local |
| `OLLAMA_SERVER_MODEL` | `OllamaServerModel` | *(empty)* | Default model; also injected into default call options |
| `OLLAMA_SERVER_CONFIG_PATH` | `OllamaServerConfig` | *(empty)* | Custom agent YAML; empty uses embedded `config.yml` |
| `OLLAMA_SERVER_PULL_MODELS_TIMEOUT` | `OllamaServerPullModelsTimeout` | `600` | Pull timeout in seconds |
| `OLLAMA_SERVER_PULL_MODELS_ENABLED` | `OllamaServerPullModelsEnabled` | `false` | Auto-pull models from YAML + base model on startup |
| `OLLAMA_SERVER_LOAD_MODELS_ENABLED` | `OllamaServerLoadModelsEnabled` | `false` | Query server model list for UI/catalog |

Docker host mount:

| Host / compose variable | Container path (default target) |
| --- | --- |
| `PENTAGI_OLLAMA_SERVER_CONFIG_PATH` | `/opt/pentagi/conf/ollama.provider.yml` (default host file: `./example.ollama.provider.yml`) |

### Local vs cloud

<Tabs>
  <Tab title="Local server">
```bash
OLLAMA_SERVER_URL=http://localhost:11434
# or from another Compose service: http://ollama-server:11434
OLLAMA_SERVER_API_KEY=
OLLAMA_SERVER_MODEL=llama3.1:8b-instruct-q8_0
OLLAMA_SERVER_PULL_MODELS_ENABLED=false
OLLAMA_SERVER_LOAD_MODELS_ENABLED=false
```

Local inference needs no API key. Pricing fields on models are left unset (`Price: nil`) when listing from the server.
  </Tab>
  <Tab title="Ollama Cloud">
```bash
OLLAMA_SERVER_URL=https://ollama.com
OLLAMA_SERVER_API_KEY=your_ollama_cloud_api_key
OLLAMA_SERVER_MODEL=gpt-oss:120b
# Multi-agent assignments (paid / multi-model):
OLLAMA_SERVER_CONFIG_PATH=/opt/pentagi/conf/ollama-cloud.provider.yml
```

Non-empty `OLLAMA_SERVER_API_KEY` is passed as `ollama.WithAPIKey`. Free-tier usage typically pins a single concurrent model via `OLLAMA_SERVER_MODEL`; multi-agent YAML (for example `examples/configs/ollama-cloud.provider.yml`) assigns cloud-tagged models per agent.
  </Tab>
</Tabs>

### Default agent config and pull behavior

If `OLLAMA_SERVER_CONFIG_PATH` is empty, the provider embeds `backend/pkg/providers/ollama/config.yml` (sampling defaults only—no per-agent `model` keys; agents inherit `OLLAMA_SERVER_MODEL`).

Build defaults: `n: 1`, `max_tokens: 32768`, model from `OLLAMA_SERVER_MODEL`.

When `OLLAMA_SERVER_PULL_MODELS_ENABLED=true`:

1. Collect unique models from the base model plus every model named in the agent config.
2. For each model, `Show` with a 10s timeout; if missing, `Pull` within `OLLAMA_SERVER_PULL_MODELS_TIMEOUT` (or 10 minutes if timeout ≤ 0).
3. Any pull error fails provider construction.

When `OLLAMA_SERVER_LOAD_MODELS_ENABLED=true`, the provider lists models via the Ollama API (10s timeout). When false, the available list is only the base model.

`ModelWithPrefix` is a passthrough (no LiteLLM-style prefix for Ollama).

## Aggregator and example endpoints

Point the custom provider at any OpenAI-compatible gateway. Repository examples under `examples/configs/`:

| Endpoint pattern | Example base URL | Example config file |
| --- | --- | --- |
| OpenRouter | `https://openrouter.ai/api/v1` | `openrouter.provider.yml` |
| DeepInfra | `https://api.deepinfra.com/v1/openai` | `deepinfra.provider.yml` |
| OpenAI-compatible direct | `https://api.openai.com/v1` | `custom-openai.provider.yml` |
| Novita | `https://api.novita.ai/openai` | `novita.provider.yml` |
| Azure OpenAI-compatible | vendor-specific | `azure-openai.provider.yml` |
| vLLM (local) | your `vllm serve` base URL | `vllm-qwen*.provider.yml` |
| Ollama Cloud multi-model | `https://ollama.com` | `ollama-cloud.provider.yml` |

### Wire OpenRouter or DeepInfra

<Steps>
  <Step title="Choose aggregator credentials">
    Obtain an API key from OpenRouter, DeepInfra, or another OpenAI-compatible aggregator. No first-party vendor key is required for the custom path beyond that gateway key.
  </Step>
  <Step title="Mount agent YAML">
    Copy or reference `examples/configs/openrouter.provider.yml` or `deepinfra.provider.yml`. Under Docker, set `PENTAGI_LLM_SERVER_CONFIG_PATH` to the host path so it mounts at `/opt/pentagi/conf/custom.provider.yml`.
  </Step>
  <Step title="Set custom LLM env">
```bash
LLM_SERVER_URL=https://openrouter.ai/api/v1
# or: https://api.deepinfra.com/v1/openai
LLM_SERVER_KEY=your_api_key
LLM_SERVER_MODEL=
LLM_SERVER_CONFIG_PATH=/opt/pentagi/conf/custom.provider.yml
LLM_SERVER_PROVIDER=
LLM_SERVER_LEGACY_REASONING=false
LLM_SERVER_PRESERVE_REASONING=false
```
  Leave `LLM_SERVER_MODEL` empty when every agent specifies `model` in YAML. Set `LLM_SERVER_PROVIDER` only when a proxy namespaces models (for example LiteLLM).
  </Step>
  <Step title="Restart and verify">
    Restart the PentAGI container or process. Confirm the `custom` provider appears in Settings and passes provider/agent tests before creating a flow.
  </Step>
</Steps>

### OpenRouter-style agent snippet

From `examples/configs/openrouter.provider.yml`—models use `vendor/model` IDs and optional `reasoning` / `price` blocks:

```yaml
primary_agent:
  model: "openai/gpt-5"
  n: 1
  max_tokens: 6000
  reasoning:
    effort: medium
  price:
    input: 1.25
    output: 10.0

generator:
  model: "anthropic/claude-sonnet-4.5"
  n: 1
  max_tokens: 12000
  reasoning:
    max_tokens: 4000
  price:
    input: 3.0
    output: 15.0
```

### DeepInfra-style agent snippet

From `examples/configs/deepinfra.provider.yml`—IDs follow DeepInfra’s `org/model` naming:

```yaml
simple:
  model: "Qwen/Qwen3-Next-80B-A3B-Instruct"
  temperature: 0.7
  top_p: 0.95
  n: 1
  max_tokens: 4000
  price:
    input: 0.14
    output: 1.4

pentester:
  model: "moonshotai/Kimi-K2-Instruct-0905"
  temperature: 1.0
  n: 1
  max_tokens: 6000
  price:
    input: 0.4
    output: 2.0
```

### Official OpenAI via custom path

When using the custom provider against `https://api.openai.com/v1` with `custom-openai.provider.yml`:

```bash
LLM_SERVER_URL=https://api.openai.com/v1
LLM_SERVER_KEY=your_openai_api_key
LLM_SERVER_CONFIG_PATH=/opt/pentagi/conf/custom.provider.yml
LLM_SERVER_LEGACY_REASONING=true
```

Prefer the dedicated OpenAI provider (`OPEN_AI_KEY`) when you do not need the custom path; use custom when you share one OpenAI-compatible stack for aggregators and self-hosted gateways.

### LiteLLM proxy with the same YAML

```bash
# Direct vendor
LLM_SERVER_URL=https://api.moonshot.ai/v1
LLM_SERVER_CONFIG_PATH=/opt/pentagi/conf/moonshot.provider.yml
LLM_SERVER_PROVIDER=

# Same YAML through LiteLLM
LLM_SERVER_URL=http://litellm-proxy:4000
LLM_SERVER_CONFIG_PATH=/opt/pentagi/conf/moonshot.provider.yml
LLM_SERVER_PROVIDER=moonshot
```

With `LLM_SERVER_PROVIDER=moonshot`, a YAML model `kimi-2.5` is sent as `moonshot/kimi-2.5`.

## Quick reference: local Ollama

```bash
# Install and pull models on the host (or Ollama container)
ollama pull llama3.1:8b-instruct-q8_0

# Fast startup (static config)
OLLAMA_SERVER_URL=http://localhost:11434
OLLAMA_SERVER_MODEL=llama3.1:8b-instruct-q8_0
OLLAMA_SERVER_PULL_MODELS_ENABLED=false
OLLAMA_SERVER_LOAD_MODELS_ENABLED=false

# Optional: auto-pull + discover
OLLAMA_SERVER_PULL_MODELS_ENABLED=true
OLLAMA_SERVER_PULL_MODELS_TIMEOUT=900
OLLAMA_SERVER_LOAD_MODELS_ENABLED=true
```

Example local multi-agent files: `ollama-llama318b.provider.yml`, `ollama-qwen332b-fp16-tc.provider.yml`, `ollama-qwq32b-fp16-tc.provider.yml`.

## Operational notes

<AccordionGroup>
  <Accordion title="Custom provider experimental surface">
    `LLM_SERVER_*` is documented in the repo as an experimental wiring surface that may evolve. Prefer stable first-party providers when a dedicated adapter exists; use custom for aggregators, proxies, and self-hosted OpenAI-compatible servers.
  </Accordion>
  <Accordion title="Proxy">
    Both providers use `system.GetHTTPClient(cfg)`, so a global `PROXY_URL` applies to model HTTP traffic when configured.
  </Accordion>
  <Accordion title="Vertex AI workaround">
    There is no first-class Vertex AI adapter. Supported path: expose Vertex through an OpenAI-compatible gateway that preserves chat and tool-call behavior, then set `LLM_SERVER_URL`, `LLM_SERVER_KEY`, and `LLM_SERVER_MODEL` (or a config path).
  </Accordion>
  <Accordion title="vLLM and smaller local models">
    Production-style local stacks often use vLLM with Qwen-class models via `LLM_SERVER_*` and the `vllm-qwen*.provider.yml` templates (thinking vs non-thinking). See the dedicated deployment page for hardware and supervision flags.
  </Accordion>
</AccordionGroup>

## Failure modes

| Symptom | Likely cause | Mitigation |
| --- | --- | --- |
| Custom provider missing from UI | Gate failed: no URL, or URL without model **and** config path | Set `LLM_SERVER_URL` plus `LLM_SERVER_MODEL` and/or `LLM_SERVER_CONFIG_PATH` |
| Startup fails reading config | Bad path or unreadable file | Fix path; under Docker use `PENTAGI_*_CONFIG_PATH` host mount + in-container `*_CONFIG_PATH` |
| Reasoning / tool-call errors | Wrong wire format or stripped reasoning | Toggle `LLM_SERVER_LEGACY_REASONING`; enable `LLM_SERVER_PRESERVE_REASONING` for multi-turn reasoning APIs |
| Models empty in custom UI | `/models` fetch failed or timed out | Provider still runs; list models manually in YAML; check URL auth and reachability |
| Ollama connection errors | Server down or wrong URL | Ensure Ollama is listening; use Docker network hostname when applicable |
| Ollama model not found | Not pulled | `ollama pull <model>` or enable `OLLAMA_SERVER_PULL_MODELS_ENABLED` |
| Slow Ollama startup | Pull and/or load enabled | Disable load/pull after first successful warm-up for static deployments |
| LiteLLM 404 on model | Missing or wrong prefix | Align `LLM_SERVER_PROVIDER` with proxy namespace |

## Related pages

<CardGroup>
  <Card title="Configure LLM providers" href="/configure-llm-providers">
    First-party env keys, UI profiles, and testing providers before flows.
  </Card>
  <Card title="Provider configuration schema" href="/provider-config-schema">
    Per-agent YAML fields: model, sampling, reasoning, price, extra_body.
  </Card>
  <Card title="Example provider configs" href="/example-provider-configs">
    Copy-paste YAML for vLLM, Ollama, OpenRouter, DeepInfra, Azure, and more.
  </Card>
  <Card title="Deploy with vLLM and Qwen" href="/vllm-qwen-deployment">
    Local/air-gapped inference: serve flags, LLM_SERVER wiring, thinking configs.
  </Card>
  <Card title="Environment variables" href="/environment-variables">
    Full Config struct and `.env.example` reference.
  </Card>
</CardGroup>