# Deploy with vLLM and Qwen

> Air-gapped style local inference: hardware matrix, vLLM serve flags, LLM_SERVER_* wiring, thinking vs non-thinking provider YAML, and supervision flags recommended for sub-32B models.

- Repository: vxcontrol/pentagi
- GitHub: https://github.com/vxcontrol/pentagi
- Human docs: https://grok-wiki.com/public/docs/vxcontrol-pentagi-eca0cf4de5f5
- Complete Markdown: https://grok-wiki.com/public/docs/vxcontrol-pentagi-eca0cf4de5f5/llms-full.txt

## Source Files

- `examples/guides/vllm-qwen35-27b-fp8.md`
- `examples/configs/vllm-qwen3.5-27b-fp8.provider.yml`
- `examples/configs/vllm-qwen3.5-27b-fp8-no-think.provider.yml`
- `examples/configs/vllm-qwen3.6-27b-fp8.provider.yml`
- `.env.example`
- `backend/pkg/providers/custom/custom.go`
- `README.md`

---

---
title: "Deploy with vLLM and Qwen"
description: "Air-gapped style local inference: hardware matrix, vLLM serve flags, LLM_SERVER_* wiring, thinking vs non-thinking provider YAML, and supervision flags recommended for sub-32B models."
---

PentAGI runs fully local multi-agent flows by pointing the **Custom** OpenAI-compatible provider at a vLLM OpenAI API server (`/v1`) serving Qwen dense FP8 weights. The integration path is `backend/pkg/providers/custom` plus `LLM_SERVER_*` (or a UI Custom provider), agent sampling from `examples/configs/vllm-*.provider.yml`, and optional execution monitor / planning flags for models under ~32B parameters.

```text
┌─────────────────────┐     OpenAI /v1      ┌──────────────────────────┐
│  PentAGI (Custom)   │ ──────────────────► │  vLLM serve              │
│  LLM_SERVER_*  or   │  chat/completions   │  Qwen/*-FP8              │
│  UI provider YAML   │ ◄────────────────── │  reasoning + tool parsers│
└──────────┬──────────┘                     └──────────────────────────┘
           │ agents: primary, pentester, coder, …
           │ supervision: EXECUTION_MONITOR_*, AGENT_PLANNING_STEP_*
           ▼
    Docker sandboxes / tools / pgvector memory
```

## Supported example configs

| File | Model | Thinking default |
| --- | --- | --- |
| `examples/configs/vllm-qwen3.5-27b-fp8.provider.yml` | `Qwen/Qwen3.5-27B-FP8` | On for primary/coding; off for simple/searcher/enricher/reflector |
| `examples/configs/vllm-qwen3.5-27b-fp8-no-think.provider.yml` | same | Off for all agents |
| `examples/configs/vllm-qwen3.6-27b-fp8.provider.yml` | `Qwen/Qwen3.6-27B-FP8` | Same pattern as 3.5 thinking |
| `examples/configs/vllm-qwen3.6-27b-fp8-no-think.provider.yml` | same | Off for all agents |
| `examples/configs/vllm-qwen3.6-35b-a3b-fp8.provider.yml` | `Qwen/Qwen3.6-35B-A3B-FP8` | Thinking default |
| `examples/configs/vllm-qwen3.6-35b-a3b-fp8-no-think.provider.yml` | same | Non-thinking |
| `examples/configs/vllm-qwen332b-fp16.provider.yml` | `Qwen/Qwen3-32B` | Sampling-only (no `chat_template_kwargs`) |

Primary operational guide for hardware, serve flags, and curl checks: `examples/guides/vllm-qwen35-27b-fp8.md`.

## Model and hardware matrix

**Qwen3.5-27B-FP8** (guide baseline): 27B dense, hybrid ~75% Gated DeltaNet / ~25% Gated Attention, native context **262,144** tokens (YaRN up to ~1,010,000), FP8 W8A8 block size 128. Treat as a VLM family member: the vision encoder still consumes VRAM unless skipped with `--language-model-only`.

FP8 W8A8 acceleration needs GPU **Compute Capability ≥ 8.9** (Ada Lovelace, Hopper, Blackwell). On Ampere (A100, RTX 3090, etc.) FP8 falls back to W8A16 via Marlin with lower throughput.

| Configuration | Total VRAM | Max context | FP8 mode | Notes |
| --- | --- | --- | --- | --- |
| 2× RTX 5090 (64 GB) | 64 GB | ≤131k | W8A8 | Good |
| **4× RTX 5090 (128 GB)** | **128 GB** | **262k native** | **W8A8** | **Tested ~30 GB/GPU @ util 0.75** |
| 1× H100 SXM (80 GB) | 80 GB | 262k | W8A8 | Single GPU |
| 2× H100 SXM (160 GB) | 160 GB | 262k | W8A8 | Excellent |
| 4× A100 80GB (320 GB) | 320 GB | 262k | W8A16 | Slower fallback |

### Host prerequisites

- Linux (Ubuntu 22.04+ recommended)
- CUDA 12.1+, NVIDIA drivers 535+
- Python 3.9–3.12
- Multi-GPU: NCCL **2.27.3+**; Blackwell tensor-parallel needs `NCCL_P2P_DISABLE=1`

```bash
nvidia-smi
nvcc --version
```

## Install vLLM

The guide states `qwen3_5` architecture is **not** recognized in stable vLLM at time of writing; use **nightly** until vLLM v0.17.0+ lands that support.

<Tabs>
<Tab title="uv">
```bash
curl -LsSf https://astral.sh/uv/install.sh | sh
uv pip install vllm --torch-backend=auto --extra-index-url https://wheels.vllm.ai/nightly
python -c "import vllm; print(vllm.__version__)"
```
</Tab>
<Tab title="pip">
```bash
pip install vllm --pre --extra-index-url https://wheels.vllm.ai/nightly
python -c "import vllm; print(vllm.__version__)"
```
</Tab>
<Tab title="Docker">
```bash
docker pull vllm/vllm-openai:nightly
```
</Tab>
</Tabs>

## vLLM serve flags

Recommended parameters (4× RTX 5090 tested profile):

| Flag | Value | Role |
| --- | --- | --- |
| `--model` | `Qwen/Qwen3.5-27B-FP8` | Hugging Face id (swap for 3.6 / 35B-A3B variants) |
| `--tensor-parallel-size` | `4` (or omit for 1 GPU) | One shard per GPU |
| `--max-model-len` | `262144` | Native context |
| `--max-num-batched-tokens` | `4096` | Lower inter-token latency for chat |
| `--block-size` | `128` | Match FP8 block size |
| `--gpu-memory-utilization` | `0.75` | KV-cache headroom |
| `--language-model-only` | flag | Skip vision encoder; reclaim ~2–4 GB KV |
| `--enable-prefix-caching` | flag | Cache repeated system prompts |
| `--reasoning-parser` | `qwen3` | Strip/parse thinking for Qwen3.x |
| `--tool-call-parser` | `qwen3_xml` | Avoid infinite `!!!!` with long contexts |
| `--attention-backend` | `FLASHINFER` | Ada / Hopper / Blackwell |
| `--speculative-config` | `{"method":"qwen3_next_mtp","num_speculative_tokens":1}` | MTP; keep tokens at **1** |
| `-O3` | flag | `torch.compile` max opt |

<CodeGroup>
```bash title="Single GPU (e.g. H100 / H200)"
vllm serve Qwen/Qwen3.5-27B-FP8 \
  --max-model-len 262144 \
  --max-num-batched-tokens 4096 \
  --block-size 128 \
  --gpu-memory-utilization 0.75 \
  --language-model-only \
  --enable-prefix-caching \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_xml \
  --attention-backend FLASHINFER \
  --speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":1}' \
  -O3 \
  --host 127.0.0.1 \
  --port 8000
```

```bash title="Multi-GPU 4× RTX 5090 (Blackwell)"
NCCL_P2P_DISABLE=1 vllm serve Qwen/Qwen3.5-27B-FP8 \
  --tensor-parallel-size 4 \
  --max-model-len 262144 \
  --max-num-batched-tokens 4096 \
  --block-size 128 \
  --gpu-memory-utilization 0.75 \
  --language-model-only \
  --enable-prefix-caching \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_xml \
  --attention-backend FLASHINFER \
  --speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":1}' \
  -O3 \
  --host 127.0.0.1 \
  --port 8000
```
</CodeGroup>

<Warning>
`NCCL_P2P_DISABLE=1` is required for Blackwell (RTX 5090) when `tensor-parallel-size > 1` to avoid NCCL hangs. Prefer NCCL 2.27.3+.
</Warning>

Optional server-wide non-thinking default:

```bash
vllm serve Qwen/Qwen3.5-27B-FP8 \
  --default-chat-template-kwargs '{"enable_thinking": false}' \
  # ... remaining flags
```

Per-request control still uses `chat_template_kwargs.enable_thinking` (not a root-level field).

### Multi-turn thinking hygiene

Historical assistant turns should keep **final answers only**, not prior `<think>...</think>` blocks. vLLM’s chat template handles this when you use the stock OpenAI path; custom history builders must strip thinking tags.

### Extended context (YaRN)

Native 262k is enough for most flows. For ~1M tokens:

```bash
VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 vllm serve Qwen/Qwen3.5-27B-FP8 \
  --hf-overrides '{"text_config": {"rope_parameters": {"mrope_interleaved": true, "mrope_section": [11, 11, 10], "rope_type": "yarn", "rope_theta": 10000000, "partial_rotary_factor": 0.25, "factor": 4.0, "original_max_position_embeddings": 262144}}}' \
  --max-model-len 1010000 \
  # ... other parameters
```

YaRN uses a **static** scale (short prompts can pay a quality cost). Prefer `factor=2.0` for ~524k if you need half that extension.

## Verify the inference server

<Steps>
<Step title="Thinking mode (default)">
```bash
curl "http://127.0.0.1:8000/v1/chat/completions" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3.5-27B-FP8",
    "messages": [{"role": "user", "content": "hey! what is the weather in Moscow?"}],
    "temperature": 1.0,
    "top_k": 20,
    "top_p": 0.95,
    "min_p": 0.0,
    "presence_penalty": 1.5,
    "repetition_penalty": 1.0
  }'
```
Expect `<think>` content in the stream/body when reasoning is on.
</Step>
<Step title="Non-thinking mode">
```bash
curl "http://127.0.0.1:8000/v1/chat/completions" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3.5-27B-FP8",
    "messages": [{"role": "user", "content": "hey! what is the weather in Beijing?"}],
    "temperature": 0.7,
    "top_k": 20,
    "top_p": 0.8,
    "min_p": 0.0,
    "presence_penalty": 1.5,
    "repetition_penalty": 1.0,
    "chat_template_kwargs": {"enable_thinking": false}
  }'
```
Expect a direct answer without think tags.
</Step>
</Steps>

## Wire PentAGI: `LLM_SERVER_*` and Custom provider

The custom adapter (`backend/pkg/providers/custom/custom.go`) builds an OpenAI client from:

| Env | Config field | Purpose |
| --- | --- | --- |
| `LLM_SERVER_URL` | `LLMServerURL` | Base OpenAI API root (include `/v1`) |
| `LLM_SERVER_KEY` | `LLMServerKey` | Bearer token; use any non-empty placeholder if vLLM auth is off |
| `LLM_SERVER_MODEL` | `LLMServerModel` | Default model if agent YAML omits `model` |
| `LLM_SERVER_CONFIG_PATH` | `LLMServerConfig` | Path **inside the process** to AgentsConfig YAML |
| `LLM_SERVER_PROVIDER` | `LLMServerProvider` | Optional model name prefix (LiteLLM-style); usually empty for raw vLLM |
| `LLM_SERVER_LEGACY_REASONING` | `LLMServerLegacyReasoning` | Default `false` → modern structured reasoning opts |
| `LLM_SERVER_PRESERVE_REASONING` | `LLMServerPreserveReasoning` | Default `false`; set `true` if the backend requires `reasoning_content` on later turns |

Compose mounts the host YAML via `PENTAGI_LLM_SERVER_CONFIG_PATH` → container `/opt/pentagi/conf/custom.provider.yml` (see `docker-compose.yml`). Set the in-container path on `LLM_SERVER_CONFIG_PATH`.

### Example `.env` for host vLLM + Docker PentAGI

```bash
# Reachable FROM the pentagi container (not container-local loopback if vLLM is on the host)
LLM_SERVER_URL=http://<host-reachable-ip>:8000/v1
LLM_SERVER_KEY=dummy
LLM_SERVER_MODEL=Qwen/Qwen3.5-27B-FP8
LLM_SERVER_CONFIG_PATH=/opt/pentagi/conf/custom.provider.yml
LLM_SERVER_LEGACY_REASONING=false
LLM_SERVER_PRESERVE_REASONING=false

# Host path bind-mounted into the container
PENTAGI_LLM_SERVER_CONFIG_PATH=./examples/configs/vllm-qwen3.5-27b-fp8.provider.yml
```

<Note>
If PentAGI runs in Docker and vLLM binds `127.0.0.1` on the host, the container cannot use `http://127.0.0.1:8000`. Bind vLLM to a host interface the container can reach, or use a host-gateway URL your Docker setup provides. UI “Base URL” must also resolve from the **PentAGI server process**, not only from the browser.
</Note>

Defaults when agent options omit sampling: temperature `1.0`, top_p `1.0`, n `1`, max_tokens `16384` (`BuildProviderConfig`). Provider YAML typically raises `max_tokens` to `32768`.

### UI Custom provider (no restart)

1. Open **Settings → Providers → Add Provider**
2. **Type**: Custom  
3. **Base URL**: `http://127.0.0.1:8000/v1` (or the server-reachable URL)  
4. **API Key**: `dummy`  
5. **Configuration**: paste contents of a `examples/configs/vllm-*.provider.yml`  
6. Save, then create a flow and select that provider  

Smoke task: `Scan localhost port 80` and watch tool/LLM logs.

## Thinking vs non-thinking provider YAML

Thinking is controlled with `extra_body.chat_template_kwargs.enable_thinking`, not a top-level API field.

| Mode | File pattern | Behavior |
| --- | --- | --- |
| **Thinking (quality)** | `vllm-*-fp8.provider.yml` | Primary/assistant/generator/refiner/adviser and coder/installer/pentester leave thinking **on** (no `extra_body`); simple / simple_json / searcher / enricher / reflector force `enable_thinking: false` |
| **Non-thinking (latency)** | `*-no-think.provider.yml` | Every agent sets `enable_thinking: false` |

### Sampling profiles (Qwen recommendations, applied in YAML)

| Mode | temperature | top_p | top_k | presence_penalty | Agents (thinking YAML) |
| --- | --- | --- | --- | --- | --- |
| Thinking, general | 1.0 | 0.95 | 20 | 1.5 | primary_agent, assistant, generator, refiner, adviser |
| Thinking, coding | 0.6 | 0.95 | 20 | 0.0 | coder, installer, pentester |
| Non-thinking, general | 0.7 | 0.8 | 20 | 1.5 | simple, simple_json, searcher, enricher |
| Non-thinking, reasoning-style | 1.0 | 0.95–1.0 | 20–40 | 1.5–2.0 | used heavily in `*-no-think` for specialist agents |

Shared defaults in examples: `repetition_penalty: 1.0`, `min_p: 0.0`, `n: 1`, `max_tokens: 32768`. Set `json: true` on `simple_json`.

Snippet (thinking config forces non-think only on lightweight agents):

```yaml
simple:
  model: "Qwen/Qwen3.5-27B-FP8"
  temperature: 0.7
  top_k: 20
  top_p: 0.8
  max_tokens: 32768
  extra_body:
    chat_template_kwargs:
      enable_thinking: false

primary_agent:
  model: "Qwen/Qwen3.5-27B-FP8"
  temperature: 1.0
  top_k: 20
  top_p: 0.95
  max_tokens: 32768
  # thinking enabled by default (no extra_body)

coder:
  model: "Qwen/Qwen3.5-27B-FP8"
  temperature: 0.6
  top_k: 20
  top_p: 0.95
  presence_penalty: 0.0
  max_tokens: 32768
```

## Supervision for sub-32B local models

README guidance for open-source models **&lt; 32B** (including Qwen3.5-27B-FP8): enable **execution monitoring** and **agent planning** for production-grade autonomous runs. Hard tool-call caps always apply.

| Variable | Default | Sub-32B recommendation |
| --- | --- | --- |
| `EXECUTION_MONITOR_ENABLED` | `false` | `true` — mentor (adviser) on same-tool / total-tool thresholds |
| `EXECUTION_MONITOR_SAME_TOOL_LIMIT` | `5` | start at `5`; raise for exploratory tasks |
| `EXECUTION_MONITOR_TOTAL_TOOL_LIMIT` | `10` | start at `10` |
| `AGENT_PLANNING_STEP_ENABLED` | `false` | `true` — 3–7 step plan before pentester/coder/installer |
| `MAX_GENERAL_AGENT_TOOL_CALLS` | `100` | keep or raise for long engagements |
| `MAX_LIMITED_AGENT_TOOL_CALLS` | `20` | keep defaults unless limited agents thrash |

```bash
EXECUTION_MONITOR_ENABLED=true
EXECUTION_MONITOR_SAME_TOOL_LIMIT=5
EXECUTION_MONITOR_TOTAL_TOOL_LIMIT=10
AGENT_PLANNING_STEP_ENABLED=true
MAX_GENERAL_AGENT_TOOL_CALLS=100
MAX_LIMITED_AGENT_TOOL_CALLS=20
```

**Trade-offs** (repo-reported for Qwen3.5-27B-FP8): ~2–3× tokens and wall time; ~2× result quality / less looping. Prefer the **thinking** provider YAML so adviser/planner use higher reasoning sampling; optionally give `adviser` a stronger model later without changing the rest of the stack.

Always-on layers (no env toggle): reflector after failed tool-call generations, barrier tools (`done`, `ask`), and hard max iterations for general vs limited agents.

## Performance (repo benchmarks)

On **4× RTX 5090**, guide figures for the 27B FP8 stack:

| Metric | Value |
| --- | --- |
| Prompt processing | ~13,000 tok/s |
| Completion | ~650 tok/s |
| Concurrent flows | ~12 stable |
| VRAM | ~30 GB/GPU at `--gpu-memory-utilization 0.75` |
| Context | full 262k when VRAM allows |

`examples/tests/vllm-qwen332b-fp16-report.md` also records high agent tool-call pass rates for `Qwen/Qwen3.5-27B-FP8` under the project’s agent test harness (not a substitute for end-to-end pentest QA).

## Troubleshooting

| Symptom | Cause | Fix |
| --- | --- | --- |
| `Unknown architecture 'qwen3_5'` | Stable vLLM | Install nightly wheels / image |
| Multi-GPU hang | Blackwell P2P | `NCCL_P2P_DISABLE=1`; upgrade `nvidia-nccl-cu12` |
| `enable_thinking` ignored | Wrong JSON shape | Nest under `chat_template_kwargs` (YAML: `extra_body.chat_template_kwargs`) |
| Infinite `!!!!` | Wrong tool parser | `--tool-call-parser qwen3_xml` (not `qwen3_coder`) |
| OOM | Context / util too high | Lower `--max-model-len` (e.g. `131072`) or `--gpu-memory-utilization` |
| Speculative decode errors | Unstable MTP depth | `num_speculative_tokens: 1` only |
| Custom provider empty / wrong models | Config path not mounted | Align `PENTAGI_LLM_SERVER_CONFIG_PATH` host file with `LLM_SERVER_CONFIG_PATH` container path |
| Connection refused from Docker | URL is container loopback | Point `LLM_SERVER_URL` at a host-reachable address |

## Air-gapped checklist

1. Pre-pull or offline-cache `Qwen/Qwen3.5-27B-FP8` (or chosen variant) and the vLLM nightly image/wheels.  
2. Run vLLM on the GPU host; no cloud LLM keys required.  
3. Configure Custom provider / `LLM_SERVER_*` only.  
4. Use thinking YAML + supervision flags for &lt;32B quality.  
5. Optionally disable external search engines if the network must stay closed (search tools fail closed without keys/endpoints; see search configuration pages).

## Related pages

<CardGroup>
<Card title="Local and custom providers" href="/local-and-custom-providers">
OpenAI-compatible endpoints, LLM_SERVER_* flags, Ollama, aggregators.
</Card>
<Card title="Example provider configs" href="/example-provider-configs">
Copy-paste YAML for vLLM, Ollama, OpenRouter, DeepInfra, Azure.
</Card>
<Card title="Provider configuration schema" href="/provider-config-schema">
AgentsConfig fields: model, sampling, json, reasoning, extra_body.
</Card>
<Card title="Agents and supervision" href="/agents-and-supervision">
Specialist agents, mentor monitor, planning step, tool-call limits.
</Card>
<Card title="Environment variables" href="/environment-variables">
Full Config / .env.example including supervision and custom LLM keys.
</Card>
<Card title="Installation" href="/installation">
Docker Compose core stack, volumes, SSL, overlays.
</Card>
</CardGroup>