# Run Reasoner with vLLM

> Install vllm-cosmos3 plugin, serve Cosmos3ReasonerForConditionalGeneration with mm-encoder and media-io-kwargs, Qwen3-VL-compatible chat messages, and reasoning-format prompt suffix.

- Repository: NVIDIA/cosmos
- GitHub: https://github.com/NVIDIA/cosmos
- Human docs: https://grok-wiki.com/public/docs/nvidia-cosmos-82de3e90abd9
- Complete Markdown: https://grok-wiki.com/public/docs/nvidia-cosmos-82de3e90abd9/llms-full.txt

## Source Files

- `README.md`
- `cookbooks/cosmos3/README.md`
- `cookbooks/cosmos3/reasoner/README.md`
- `cookbooks/cosmos3/reasoner/run_with_vllm.ipynb`
- `cookbooks/cosmos3/reasoner/assets/robotics_next_action.mp4`
- `cookbooks/cosmos3/reasoner/assets/temporal_localization_1.mp4`

---

---
title: "Run Reasoner with vLLM"
description: "Install vllm-cosmos3 plugin, serve Cosmos3ReasonerForConditionalGeneration with mm-encoder and media-io-kwargs, Qwen3-VL-compatible chat messages, and reasoning-format prompt suffix."
---

The Cosmos 3 Reasoner runs as an OpenAI-compatible `vllm serve` process: the `vllm-cosmos3` plugin registers `Cosmos3ReasonerForConditionalGeneration`, multimodal inputs use Qwen3-VL-style chat messages, and clients call `POST /v1/chat/completions` for text outputs from images and videos.

## Architecture

```mermaid
sequenceDiagram
    participant Client as OpenAI client
    participant API as vLLM /v1
    participant Plugin as vllm-cosmos3
    participant Model as Cosmos3ReasonerForConditionalGeneration

    Client->>API: POST /v1/chat/completions
    Note over Client,API: image_url / video_url + text prompt
    API->>Plugin: Cosmos3ReasonerForConditionalGeneration
    Plugin->>Model: mm encoder + AR decode
    Model-->>API: text tokens
    API-->>Client: choices[0].message.content
```

| Component | Role |
| --- | --- |
| `vllm-cosmos3` | Registers Reasoner architecture and processors from `cosmos-framework` |
| `Cosmos3ReasonerForConditionalGeneration` | Autoregressive Reasoner path (text out only) |
| `--mm-encoder-tp-mode data` | Data-parallel visual encoder for multimodal workloads |
| `--media-io-kwargs` | Server-side video frame ingestion before downstream sampling |

<Note>
Reasoner vLLM loads only the understanding path. For image/video/sound/action generation, use vLLM-Omni instead.
</Note>

## Prerequisites

| Requirement | Detail |
| --- | --- |
| OS / GPU | Linux; NVIDIA Ampere, Hopper, or Blackwell |
| Package manager | `uv` (see cookbook environment guide) |
| Hugging Face | Gated access to `nvidia/Cosmos3-Nano` and/or `nvidia/Cosmos3-Super` |
| CUDA pairing | `cu130` + `vllm==0.21.0` (CUDA 13 driver) or `cu128` + `vllm==0.19.1` (CUDA 12.x) |

Authenticate before the first download:

```bash
uvx hf@latest auth login
```

## Install vLLM and vllm-cosmos3

<Steps>
<Step title="Create a Python 3.13 venv">

From the `cosmos` repository root (or any working directory):

```bash
uv venv --python 3.13 --seed --managed-python
source .venv/bin/activate
```

</Step>
<Step title="Install the CUDA-matched vLLM stack">

<Tabs>
<Tab title="CUDA 13 (cu130)">

```bash
uv pip install --torch-backend=cu130 "vllm==0.21.0" \
  "vllm-cosmos3 @ git+https://github.com/NVIDIA/cosmos-framework.git#subdirectory=packages/vllm-cosmos3"
```

</Tab>
<Tab title="CUDA 12.x (cu128)">

```bash
uv pip install --torch-backend=cu128 "vllm==0.19.1" \
  "vllm-cosmos3 @ git+https://github.com/NVIDIA/cosmos-framework.git#subdirectory=packages/vllm-cosmos3"
```

</Tab>
</Tabs>

The notebook path can also install from a local `packages/cosmos3` checkout (`transformers-cosmos3` + `vllm-cosmos3`) after cloning `cosmos-framework`.

</Step>
<Step title="Verify GPU visibility">

```bash
.venv/bin/python - <<'PY'
import torch
print("cuda available:", torch.cuda.is_available())
if torch.cuda.is_available():
    print("device 0:", torch.cuda.get_device_name(0))
PY
```

</Step>
</Steps>

<Warning>
`--torch-backend=auto` is unreliable for vLLM wheels. Match the install pair to `nvidia-smi` driver CUDA or `torch.cuda.is_available()` returns `False`.
</Warning>

If the build reports DeepGEMM unavailable:

```bash
export VLLM_USE_DEEP_GEMM=0
```

When invoking `.venv/bin/vllm` without activating the venv, keep `.venv/bin` on `PATH` so FlashInfer can find `ninja`.

## Serve the Reasoner

Override Hugging Face `architectures` so vLLM loads the Reasoner class instead of the full omnimodal checkpoint default.

### Cosmos3-Nano (single GPU)

From the repo root, with media under `cookbooks/cosmos3` reachable via `file://` URLs:

```bash
CUDA_VISIBLE_DEVICES=0 \
vllm serve nvidia/Cosmos3-Nano \
  --hf-overrides '{"architectures": ["Cosmos3ReasonerForConditionalGeneration"]}' \
  --tensor-parallel-size 1 \
  --mm-encoder-tp-mode data \
  --async-scheduling \
  --allowed-local-media-path "$(dirname "$(pwd)")" \
  --media-io-kwargs '{"video": {"num_frames": -1}}' \
  --port 8000
```

### Cosmos3-Super (4 GPUs)

The Reasoner cookbook notebook defaults to Super with tensor parallelism:

```bash
CUDA_VISIBLE_DEVICES=0,1,2,3 \
vllm serve nvidia/Cosmos3-Super \
  --hf-overrides '{"architectures": ["Cosmos3ReasonerForConditionalGeneration"]}' \
  --tensor-parallel-size 4 \
  --mm-encoder-tp-mode data \
  --async-scheduling \
  --allowed-local-media-path /path/to/cookbooks/cosmos3 \
  --media-io-kwargs '{"video": {"num_frames": -1}}' \
  --port 8001
```

| Flag | Purpose |
| --- | --- |
| `--hf-overrides '{"architectures": ["Cosmos3ReasonerForConditionalGeneration"]}'` | Select Reasoner weights and config |
| `--tensor-parallel-size` | GPU count for model parallelism (`1` Nano, `4` Super in cookbooks) |
| `--mm-encoder-tp-mode data` | Data parallelism for the multimodal visual encoder |
| `--async-scheduling` | Async request scheduling (recommended in cookbooks) |
| `--allowed-local-media-path` | Parent directory allowed for `file://` image/video paths in requests |
| `--media-io-kwargs '{"video": {"num_frames": -1}}'` | Let the processor see all frames before downstream frame sampling |

<Info>
First startup compiles CUDA graphs and can take several minutes. Poll readiness with `curl -fsS http://127.0.0.1:8000/health` or `curl http://localhost:8000/v1/models`.
</Info>

## Query with chat completions

Clients use the OpenAI SDK against `http://localhost:<port>/v1` with any API key (for example `EMPTY`).

### Image caption

```python
import openai

client = openai.OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")
model = client.models.list().data[0].id

response = client.chat.completions.create(
    model=model,
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://github.com/nvidia-cosmos/cosmos-dependencies/raw/refs/heads/assets/cosmos3/inputs/vision/robot_153.jpg"
                    },
                },
                {"type": "text", "text": "Caption the image in detail."},
            ],
        }
    ],
    max_tokens=4096,
    seed=0,
)
print(response.choices[0].message.content)
```

### Local video (file URL)

Resolve cookbook assets to `file://` URIs. The server path in `--allowed-local-media-path` must contain the file.

```python
from pathlib import Path

video_path = "cookbooks/cosmos3/reasoner/assets/temporal_localization_1.mp4"
video_url = Path(video_path).resolve().as_uri()

response = client.chat.completions.create(
    model=model,
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "video_url", "video_url": {"url": video_url}},
                {"type": "text", "text": "Describe the video in detail."},
            ],
        }
    ],
    max_tokens=4096,
    extra_body={"mm_processor_kwargs": {"fps": 4, "do_sample_frames": True}},
)
```

Bundled examples under `cookbooks/cosmos3/reasoner/assets/` include `robotics_next_action.mp4`, `temporal_localization_1.mp4`, `video_caption.mp4`, and task-specific images for grounding and planning.

## Qwen3-VL-compatible messages

Reasoner requests follow Qwen3-VL message conventions: multimodal content is an ordered list of typed parts (`image_url`, `video_url`, `text`). Optional system role:

```json
[
  {
    "role": "system",
    "content": [{"type": "text", "text": "You are a helpful assistant."}]
  },
  {
    "role": "user",
    "content": [
      {"type": "video_url", "video_url": {"url": "https://example.com/video.mp4"}},
      {"type": "text", "text": "List notable events with approximate timestamps."}
    ]
  }
]
```

| Content type | Field shape |
| --- | --- |
| `image_url` | `{"url": "<https or file:// path>"}` |
| `video_url` | `{"url": "<https or file:// path>"}` |
| `text` | `{"text": "<prompt>"}` |

For per-request video frame rate and sampling, pass `extra_body`:

<ParamField body="mm_processor_kwargs" type="object">
Server-side multimodal processor overrides. Cookbooks use `{"fps": 4, "do_sample_frames": true}` for video understanding workloads.
</ParamField>

## Reasoning-format prompt suffix

For chain-of-thought style outputs, append this instruction to the user text (tags are literal in model outputs):

```text
Answer the question using the following format:

<think>
Your reasoning.
</think>

Write your final answer immediately after the </think> tag.
```

Embodied examples (for example `robotics_next_action.mp4`) use the same pattern inline. Parse structured JSON after `</think>` when the task requests trajectories or boxes.

## Sampling parameters

| Parameter | Without reasoning | With reasoning |
| --- | ---: | ---: |
| `temperature` | `0.7` | `0.6` |
| `top_p` | `0.8` | `0.95` |
| `top_k` | `20` | `20` |
| `repetition_penalty` | `1.0` | `1.0` |
| `presence_penalty` | `1.5` | `0.0` |

Reasoning-heavy cookbook cells pass the “with reasoning” set explicitly:

```python
client.chat.completions.create(
    model=model,
    messages=[...],
    max_tokens=4096,
    temperature=0.6,
    top_p=0.95,
    presence_penalty=0.0,
    extra_body={"top_k": 20, "repetition_penalty": 1.0},
)
```

## Verification

| Check | Command / signal |
| --- | --- |
| Server up | `curl http://localhost:8000/v1/models` returns model list |
| Health | `curl -fsS http://127.0.0.1:8000/health` succeeds |
| End-to-end | Chat completion returns non-empty `choices[0].message.content` |

Published Nano Reasoner serving metrics (TTFT, latency, throughput at concurrency 1/64/128/256) are in the repository inference benchmarks doc.

## Cookbook notebook

`cookbooks/cosmos3/reasoner/run_with_vllm.ipynb` installs the stack, launches Super on four GPUs by default (`VLLM_PORT=8001`), waits on `/health`, and runs captioning, temporal localization, embodied reasoning, grounding, action CoT, and related workflows. To use Nano, change only the server launch cell (`nvidia/Cosmos3-Nano`, `--tensor-parallel-size 1`, one GPU) and client `base_url` port; prompts resolve `MODEL` from `client.models.list()`.

## Related pages

<CardGroup>
<Card title="Cookbook environment" href="/cookbook-environment">
Shared uv setup, CUDA tags, HF auth, and vLLM install for Reasoner and Generator backends.
</Card>
<Card title="Reasoner vLLM configuration" href="/reasoner-vllm-configuration">
Flag reference for serve options, DeepGEMM, and CUDA version pairs.
</Card>
<Card title="Sampling and prompt parameters" href="/sampling-and-prompt-parameters">
Reasoner sampling tables, message schema, and reasoning-format details.
</Card>
<Card title="Reasoner cookbooks" href="/reasoner-cookbooks">
End-to-end notebook workflows and bundled `assets/` media.
</Card>
<Card title="Quickstart" href="/quickstart">
Minimal first-run Reasoner serve plus chat completion.
</Card>
<Card title="Troubleshooting" href="/troubleshooting">
CUDA/driver mismatches, `torch.cuda` failures, and `VLLM_USE_DEEP_GEMM`.
</Card>
</CardGroup>