# Reasoner vLLM configuration

> vllm serve flags: hf-overrides architectures, tensor-parallel-size, mm-encoder-tp-mode, async-scheduling, allowed-local-media-path, media-io-kwargs, VLLM_USE_DEEP_GEMM, and vLLM/cu130 version pairs.

- Repository: NVIDIA/cosmos
- GitHub: https://github.com/NVIDIA/cosmos
- Human docs: https://grok-wiki.com/public/docs/nvidia-cosmos-82de3e90abd9
- Complete Markdown: https://grok-wiki.com/public/docs/nvidia-cosmos-82de3e90abd9/llms-full.txt

## Source Files

- `README.md`
- `cookbooks/cosmos3/README.md`
- `cookbooks/cosmos3/reasoner/README.md`
- `cookbooks/cosmos3/reasoner/run_with_vllm.ipynb`

---

---
title: "Reasoner vLLM configuration"
description: "vllm serve flags: hf-overrides architectures, tensor-parallel-size, mm-encoder-tp-mode, async-scheduling, allowed-local-media-path, media-io-kwargs, VLLM_USE_DEEP_GEMM, and vLLM/cu130 version pairs."
---

Cosmos 3 Reasoner production inference runs through `vllm serve` on gated Hugging Face checkpoints (`nvidia/Cosmos3-Nano`, `nvidia/Cosmos3-Super`), with the `vllm-cosmos3` plugin registering `Cosmos3ReasonerForConditionalGeneration` and cookbook-tested flags for multimodal tensor parallelism, local `file://` media, and video frame ingestion.

## Install and CUDA version pairs

Create a Python 3.13 venv and install a **matched** `torch` backend and `vllm` wheel. vLLM does not ship wheels for every CUDA minor version, so `--torch-backend=auto` is unreliable for Reasoner serving — pin the pair that matches `nvidia-smi` driver CUDA.

| Driver CUDA | `uv` torch backend | `vllm` version |
| --- | --- | --- |
| 13.x | `cu130` | `0.21.0` |
| 12.x | `cu128` | `0.19.1` |

<Tabs>
<Tab title="CUDA 13 (cu130)">

```bash
uv venv --python 3.13 --seed --managed-python
source .venv/bin/activate

uv pip install --torch-backend=cu130 "vllm==0.21.0" \
  "vllm-cosmos3 @ git+https://github.com/NVIDIA/cosmos-framework.git#subdirectory=packages/vllm-cosmos3"
```

</Tab>
<Tab title="CUDA 12.x (cu128)">

```bash
uv venv --python 3.13 --seed --managed-python
source .venv/bin/activate

uv pip install --torch-backend=cu128 "vllm==0.19.1" \
  "vllm-cosmos3 @ git+https://github.com/NVIDIA/cosmos-framework.git#subdirectory=packages/vllm-cosmos3"
```

</Tab>
</Tabs>

The Reasoner notebook also installs `transformers-cosmos3` from a local `cosmos-framework` checkout (`packages/transformers-cosmos3` alongside `packages/vllm-cosmos3`). The git URL install path above is sufficient for a minimal server; clone the framework when you need the full notebook dependency set.

<Warning>
Installing `cu130` wheels on a CUDA 12.x driver yields `torch.cuda.is_available() == False` and server startup failures. Switch to the `cu128` / `vllm==0.19.1` pair instead of using `--torch-backend=auto`.
</Warning>

Authenticate to Hugging Face before the first serve (gated model repos):

```bash
uvx hf@latest auth login
```

## Reference `vllm serve` commands

Cookbooks use a **full** flag set for image and video Reasoner workloads. The root README quickstart omits some multimodal flags; prefer the cookbook commands below for parity with `run_with_vllm.ipynb` and `cookbooks/cosmos3/reasoner/README.md`.

### Cosmos3-Nano (single GPU)

```bash
CUDA_VISIBLE_DEVICES=0 \
vllm serve nvidia/Cosmos3-Nano \
  --hf-overrides '{"architectures": ["Cosmos3ReasonerForConditionalGeneration"]}' \
  --tensor-parallel-size 1 \
  --mm-encoder-tp-mode data \
  --async-scheduling \
  --allowed-local-media-path "$(dirname "$(pwd)")" \
  --media-io-kwargs '{"video": {"num_frames": -1}}' \
  --port 8000
```

When launching from `cookbooks/cosmos3/reasoner/`, `$(dirname "$(pwd)")` resolves to `cookbooks/cosmos3`, covering `file://` paths under that tree.

### Cosmos3-Super (4 GPUs)

```bash
CUDA_VISIBLE_DEVICES=0,1,2,3 \
vllm serve nvidia/Cosmos3-Super \
  --hf-overrides '{"architectures": ["Cosmos3ReasonerForConditionalGeneration"]}' \
  --tensor-parallel-size 4 \
  --mm-encoder-tp-mode data \
  --async-scheduling \
  --allowed-local-media-path "$COSMOS3_MEDIA_ROOT" \
  --media-io-kwargs '{"video": {"num_frames": -1}}' \
  --port 8001
```

Set `COSMOS3_MEDIA_ROOT` to the cookbook media root (for example `…/cosmos/cookbooks/cosmos3` in `run_with_vllm.ipynb`).

| Checkpoint | Typical GPUs | `--tensor-parallel-size` | Default port in docs |
| --- | --- | ---: | ---: |
| `nvidia/Cosmos3-Nano` | 1 | `1` | `8000` |
| `nvidia/Cosmos3-Super` | 4 | `4` | `8001` (notebook) |

<Note>
The first server start compiles CUDA graphs and can take several minutes. Poll readiness with `curl -fsS http://127.0.0.1:<port>/health` or `curl http://localhost:<port>/v1/models`.
</Note>

```text
Client (OpenAI /v1/chat/completions)
        │
        ▼
vllm serve nvidia/Cosmos3-{Nano|Super}
  + vllm-cosmos3 → Cosmos3ReasonerForConditionalGeneration
  + mm-encoder / media-io / allowed-local-media-path
        │
        ▼
Text output (Qwen3-VL-compatible multimodal messages in)
```

## Serve flag reference

| Flag | Role |
| --- | --- |
| `--hf-overrides '{"architectures": ["Cosmos3ReasonerForConditionalGeneration"]}'` | Loads the Reasoner head; without it, the omnimodal checkpoint default architecture is wrong for text-only Reasoner serving. |
| `--tensor-parallel-size` | Shards model weights across GPUs (`1` for Nano, `4` for Super in cookbooks). |
| `--mm-encoder-tp-mode data` | Uses data parallelism for the multimodal visual encoder. |
| `--async-scheduling` | Enables async scheduling (used in all Reasoner cookbook serve lines). |
| `--allowed-local-media-path` | Required prefix allowlist when requests use local `file://` image or video URLs. |
| `--media-io-kwargs '{"video": {"num_frames": -1}}'` | Lets the processor consider all available frames before downstream frame sampling. |
| `--port` | HTTP port for the OpenAI-compatible API (cookbook examples use `8000` or `8001`). |

<ParamField body="--hf-overrides" type="JSON string" required>
Overrides Hugging Face config at load time. Reasoner serving sets `architectures` to `Cosmos3ReasonerForConditionalGeneration`, registered by the `vllm-cosmos3` plugin.
</ParamField>

<ParamField body="--tensor-parallel-size" type="integer" required>
Number of GPUs for tensor-parallel inference. Must align with `CUDA_VISIBLE_DEVICES` count and available GPU memory for the chosen checkpoint.
</ParamField>

<ParamField body="--mm-encoder-tp-mode" type="string">
Cookbooks set `data` for data-parallel multimodal encoder execution alongside tensor-parallel language weights.
</ParamField>

<ParamField body="--async-scheduling" type="boolean (flag)">
Present on all documented Reasoner serve commands; omit only if you are deliberately matching a minimal README-only experiment.
</ParamField>

<ParamField body="--allowed-local-media-path" type="filesystem path">
Directory prefix allowed for local media paths in chat messages. Must be a parent of every `file://` path you send. Remote `https://` URLs do not require this flag but still need network access from the server.
</ParamField>

<ParamField body="--media-io-kwargs" type="JSON string">
Server-side media I/O options. Cookbooks pass `{"video": {"num_frames": -1}}` so video ingestion does not cap frames before the client or processor applies sampling.
</ParamField>

### Minimal serve (README quickstart)

The repository README documents a shorter Nano command without `--tensor-parallel-size`, `--mm-encoder-tp-mode`, or `--media-io-kwargs`:

```bash
vllm serve nvidia/Cosmos3-Nano \
  --hf-overrides '{"architectures": ["Cosmos3ReasonerForConditionalGeneration"]}' \
  --async-scheduling \
  --allowed-local-media-path / \
  --port 8000
```

Use the full cookbook flag set when serving local videos or matching benchmark/cookbook behavior.

## Environment variables

### `VLLM_USE_DEEP_GEMM`

If the vLLM build reports DeepGEMM as unavailable, disable it before starting the server:

```bash
export VLLM_USE_DEEP_GEMM=0
vllm serve nvidia/Cosmos3-Nano ...
```

### Process and build helpers

| Variable | Purpose |
| --- | --- |
| `CUDA_VISIBLE_DEVICES` | Restricts which GPUs the server binds (`0` for Nano, `0,1,2,3` for Super in cookbooks). |
| `TMPDIR` | Notebook sets `/tmp/${USER:-vllm}-vllm` for vLLM temp files. |
| `VLLM_PORT` / `VLLM_LOG_FILE` | Notebook overrides for background `setsid` launch and log tailing. |

<Tip>
When invoking `.venv/bin/vllm` without activating the venv, keep `.venv/bin` on `PATH`. FlashInfer JIT builds shell out to `ninja`, which lives in the venv.
</Tip>

## Client configuration (server complement)

The server exposes OpenAI-compatible `/v1/chat/completions`. Clients use `api_key="EMPTY"` and `base_url="http://localhost:<port>/v1"`.

- Resolve the model id dynamically: `client.models.list().data[0].id`.
- Image and video content follow Qwen3-VL-style message blocks (`image_url`, `video_url`).
- For local video in the notebook, convert paths with `Path(...).resolve().as_uri()` and ensure the path stays under `--allowed-local-media-path`.
- Per-request video sampling can be passed via `extra_body`, for example `{"mm_processor_kwargs": {"fps": 4, "do_sample_frames": True}}`.

Reasoning-style outputs use a prompt suffix with `redacted_reasoning` tags; sampling defaults differ with and without that format — see the sampling parameters page.

## Verification

<Steps>
<Step title="Confirm GPU visibility">

```bash
.venv/bin/python - <<'PY'
import torch
print("torch:", torch.__version__)
print("torch cuda:", torch.version.cuda)
print("cuda available:", torch.cuda.is_available())
print("device count:", torch.cuda.device_count())
PY
```

</Step>
<Step title="Start the server">

Run the Nano or Super command from above with matching `CUDA_VISIBLE_DEVICES` and `--tensor-parallel-size`.

</Step>
<Step title="Check the API">

```bash
curl -fsS http://127.0.0.1:8000/health
curl http://127.0.0.1:8000/v1/models
```

</Step>
<Step title="Send a smoke request">

```python
import openai

client = openai.OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")
model = client.models.list().data[0].id

response = client.chat.completions.create(
    model=model,
    messages=[{
        "role": "user",
        "content": [
            {"type": "image_url", "image_url": {"url": "https://example.com/image.jpg"}},
            {"type": "text", "text": "Caption the image in detail."},
        ],
    }],
    max_tokens=4096,
    seed=0,
)
print(response.choices[0].message.content)
```

</Step>
</Steps>

Published Reasoner serving metrics (TTFT, request latency, throughput at concurrency 1/64/128/256) for `nvidia/Cosmos3-Nano` via vLLM are in the inference benchmarks doc; concurrency there is **client** concurrency, not `--tensor-parallel-size`.

## Related pages

<CardGroup>
<Card title="Run Reasoner with vLLM" href="/run-reasoner-vllm">
End-to-end install, serve, chat completion, and reasoning-format prompts.
</Card>
<Card title="Cookbook environment setup" href="/cookbook-environment">
Shared vLLM + vllm-cosmos3 install, CUDA tags, and GPU verification.
</Card>
<Card title="Sampling and prompt parameters" href="/sampling-and-prompt-parameters">
Reasoner `top_p` / `temperature` tables and Qwen3-VL message shape.
</Card>
<Card title="Troubleshooting" href="/troubleshooting">
CUDA/driver mismatches, `uv` backend errors, and `VLLM_USE_DEEP_GEMM` workaround.
</Card>
<Card title="Inference benchmarks" href="/inference-benchmarks">
Cosmos3-Nano Reasoner vLLM TTFT and throughput under load.
</Card>
</CardGroup>