# Reasoner cookbook recipes

> Runnable workflows for captioning, temporal localization, embodied/common-sense reasoning, 2D grounding, describe-anything, action CoT, physical plausibility, and situation understanding with bundled media assets.

- Repository: NVIDIA/cosmos
- GitHub: https://github.com/NVIDIA/cosmos
- Human docs: https://grok-wiki.com/public/docs/nvidia-cosmos-82de3e90abd9
- Complete Markdown: https://grok-wiki.com/public/docs/nvidia-cosmos-82de3e90abd9/llms-full.txt

## Source Files

- `cookbooks/cosmos3/reasoner/README.md`
- `cookbooks/cosmos3/reasoner/run_with_vllm.ipynb`
- `cookbooks/cosmos3/reasoner/run_with_cosmos_framework.ipynb`
- `cookbooks/cosmos3/reasoner/assets/common_sense_reasoning.mp4`
- `cookbooks/cosmos3/reasoner/assets/action_cot_driving_scene.mp4`
- `cookbooks/cosmos3/reasoner/assets/physical_plausibility.mp4`
- `README.md`

---

---
title: "Reasoner cookbook recipes"
description: "Runnable workflows for captioning, temporal localization, embodied/common-sense reasoning, 2D grounding, describe-anything, action CoT, physical plausibility, and situation understanding with bundled media assets."
---

The Cosmos 3 Reasoner cookbooks under `cookbooks/cosmos3/reasoner/` ship two runnable notebooks—`run_with_vllm.ipynb` (OpenAI chat completions over image/video, default **Cosmos3-Super** on 4 GPUs) and `run_with_cosmos_framework.ipynb` (JSON inputs through `cosmos_framework.scripts.inference`, **Cosmos3-Nano** on 1 GPU)—plus checked-in MP4/PNG assets in `assets/` and a quickstart in `README.md`. Environment setup for both backends is centralized in the shared [Cookbook environment setup](/cookbook-environment) page.

## Cookbook layout

```text
cookbooks/cosmos3/reasoner/
├── README.md                          # Quickstarts for Framework and vLLM
├── run_with_vllm.ipynb                # Image + video recipes (Super, vLLM)
├── run_with_cosmos_framework.ipynb  # Text + image recipes (Nano, Framework)
└── assets/                            # Bundled media for notebook cells
    ├── video_caption.mp4
    ├── temporal_localization_1.mp4
    ├── temporal_localization_2.mp4
    ├── robotics_next_action.mp4
    ├── drive_scene_next_action.mp4
    ├── assisted_task_next_action.mp4
    ├── common_sense_reasoning.mp4
    ├── action_cot_driving_scene.mp4
    ├── physical_plausibility.mp4
    ├── situation_understanding.mp4
    ├── robot_planning.png
    ├── grounding_2d.png
    ├── describe_anything.png
    └── action_cot_trajectory.png
```

<Note>
Cosmos Framework Reasoner currently treats `vision_path` as a PIL **image** input. Video-heavy recipes (captioning over MP4, temporal localization, embodied video reasoning, physical plausibility, situation understanding) run in `run_with_vllm.ipynb`. Image workflows overlap across both notebooks.
</Note>

## Choose a backend

| Goal | Notebook | Model (default) | API / entrypoint | Media |
| --- | --- | --- | --- | --- |
| Full image + video recipe set, production-style serving | `run_with_vllm.ipynb` | `nvidia/Cosmos3-Super`, TP=4 | `vllm serve` + OpenAI `chat.completions` | Local `file://` URLs under `cookbooks/cosmos3` via `--allowed-local-media-path` |
| Native PyTorch inference, JSON batch inputs, parsed overlays in-notebook | `run_with_cosmos_framework.ipynb` | `Cosmos3-Nano`, 1 GPU | `python -m cosmos_framework.scripts.inference` | `vision_path` (image URL or local path); outputs in `reasoner_text.txt` |
| Fast CLI smoke test | `README.md` | Nano (either backend) | Copy-paste commands without opening a notebook | Remote robot image URL or local assets |

Transformers-based Reasoner inference is listed as coming soon in the cookbooks README; use vLLM or Cosmos Framework today.

## Recipe catalog

Recipes map to the Reasoner use-case table in the repository README. Assets column lists files under `cookbooks/cosmos3/reasoner/assets/` unless noted.

| Workflow | Modality | Asset(s) | vLLM notebook | Framework notebook |
| --- | --- | --- | --- | --- |
| Image caption (detail) | Image | Remote `robot_153.jpg` (vLLM); same URL in Framework JSON | Image Caption | `image_caption_detail.json` |
| Video caption | Video | `video_caption.mp4` | Video Caption | — |
| Temporal localization — action segments | Video | `temporal_localization_1.mp4` | Temporal Localization | — |
| Temporal localization — event timeline | Video | `temporal_localization_2.mp4` | Event Timeline | — |
| Temporal localization — timestamp query | Video | `temporal_localization_2.mp4` (reused) | Timestamp Query | — |
| Temporal localization — interval QA | Video | `temporal_localization_2.mp4` (reused) | Interval Question | — |
| Embodied — robotics next action | Video | `robotics_next_action.mp4` | Robotics Next Action | — |
| Embodied — drive scene next action | Video | `drive_scene_next_action.mp4` | Drive Scene Next Action | — |
| Embodied — robot planning | Image | `robot_planning.png` | Robot Planning | `robot_planning.json` |
| Embodied — assisted task next action | Video | `assisted_task_next_action.mp4` | Assisted Task Next Action | — |
| Common-sense reasoning | Video | `common_sense_reasoning.mp4` | Common Sense Reasoning | — |
| 2D grounding | Image | `grounding_2d.png` | 2D Grounding (+ box overlay) | `ground_load_bbox.json` |
| Describe anything | Image | `describe_anything.png` | Describe Anything | `describe_marked_subjects.json` |
| Action CoT — gripper trajectory | Image | `action_cot_trajectory.png` | Trajectory Coordinates (+ point overlay) | `trajectory_bowl.json` |
| Action CoT — flower task trajectory | Image | `robot_planning.png` | Trajectory Coordinates (second cell) | `trajectory_flower.json` |
| Action CoT — driving scene | Video | `action_cot_driving_scene.mp4` | Driving Scene | — |
| Physical plausibility | Video | `physical_plausibility.mp4` | Physical Plausibility Analysis | — |
| Situation understanding | Video | `situation_understanding.mp4` | Situation Understanding | — |
| Text-only smoke test | Text | — | — | `nano_text.json` |
| Image one-liner smoke test | Image | Remote `robot_153.jpg` | — | `nano_image.json` |

## Shared request conventions

Reasoner calls follow **Qwen3-VL-compatible** chat messages: vision items use `image_url` or `video_url`; prompts are sibling `text` parts in the user message.

### Sampling defaults

| Parameter | Without explicit reasoning | With `redacted_reasoning` / `<think>` prompts |
| --- | ---: | ---: |
| `temperature` | `0.7` | `0.6` |
| `top_p` | `0.8` | `0.95` |
| `top_k` | `20` | `20` |
| `repetition_penalty` | `1.0` | `1.0` |
| `presence_penalty` | `1.5` | `0.0` |

Notebook cells commonly set `max_tokens=4096` (vLLM) or `max_new_tokens: 4096` (Framework JSON). Action CoT and driving-scene cells in the vLLM notebook pass the “with reasoning” sampler values explicitly.

### Explicit reasoning format

Append this instruction when you want chain-of-thought before the final answer:

```text
Answer the question using the following format:

<think>
Your reasoning.
</think>

Write your final answer immediately after the </think> tag.
```

Common-sense, assisted-task, robotics next-action, trajectory, and driving-scene cells use this pattern.

### Video processor kwargs (vLLM)

Video examples pass frame sampling through `extra_body`:

```python
extra_body={"mm_processor_kwargs": {"fps": 4, "do_sample_frames": True}}
```

The vLLM server is started with `--media-io-kwargs '{"video": {"num_frames": -1}}'` so the encoder can consume full frame streams when needed.

## Run with vLLM (`run_with_vllm.ipynb`)

<Steps>
<Step title="Install vLLM and cosmos3 plugins">

Clone or reuse `packages/cosmos3` from [NVIDIA/cosmos-framework](https://github.com/NVIDIA/cosmos-framework.git), create `.venv`, and install the CUDA-matched pair (`cu130` → `vllm==0.21.0`, `cu128` → `vllm==0.19.1`) plus `transformers-cosmos3` and `vllm-cosmos3` from the framework checkout. See [Cookbook environment setup](/cookbook-environment) § vLLM.

</Step>
<Step title="Launch Cosmos3-Super">

Default notebook server (4× GPU):

```bash
CUDA_VISIBLE_DEVICES=0,1,2,3 \
vllm serve nvidia/Cosmos3-Super \
  --hf-overrides '{"architectures": ["Cosmos3ReasonerForConditionalGeneration"]}' \
  --tensor-parallel-size 4 \
  --mm-encoder-tp-mode data \
  --async-scheduling \
  --allowed-local-media-path "$(dirname "$(pwd)")" \
  --media-io-kwargs '{"video": {"num_frames": -1}}' \
  --port 8001
```

Set `--allowed-local-media-path` to `cookbooks/cosmos3` (parent of `reasoner/`) so `file://` URLs for bundled assets resolve. For **Nano** on one GPU, use the README quickstart on port `8000` with `--tensor-parallel-size 1`.

</Step>
<Step title="Query with OpenAI client">

```python
import openai
from pathlib import Path

client = openai.OpenAI(api_key="EMPTY", base_url="http://localhost:8001/v1")
MODEL = client.models.list().data[0].id

video_path = "cookbooks/cosmos3/reasoner/assets/video_caption.mp4"
video_url = Path(video_path).resolve().as_uri()

response = client.chat.completions.create(
    model=MODEL,
    messages=[{
        "role": "user",
        "content": [
            {"type": "video_url", "video_url": {"url": video_url}},
            {"type": "text", "text": "Describe the video in detail."},
        ],
    }],
    max_tokens=4096,
    extra_body={"mm_processor_kwargs": {"fps": 4, "do_sample_frames": True}},
)
print(response.choices[0].message.content)
```

Helper functions in the notebook: `asset_path(name)` → `assets/<name>`, `asset_url(name)` → `file://` URI for requests.

</Step>
</Steps>

<Warning>
Only change the **server launch cell** and **client `base_url` port** when switching checkpoints; prompts resolve `MODEL` dynamically via `client.models.list()`.
</Warning>

### Structured outputs in-notebook

| Recipe | Parse helper | Coordinate space |
| --- | --- | --- |
| 2D grounding | `parse_boxes()` — strips markdown fences, loads JSON array/object | Boxes normalized **0–1000**, scaled to image pixels for PIL overlay |
| Action CoT trajectories | `parse_points()` — extracts `point_2d` entries | Pixel coordinates drawn on image preview |

Grounding prompt (shared with Framework): `Locate the accurate bounding box of the load as a whole. Return a json.`

Describe-anything prompt expects JSON with keys `subject_id`, `category`, and `caption`.

## Run with Cosmos Framework (`run_with_cosmos_framework.ipynb`)

<Steps>
<Step title="Install framework">

From repo root, clone into `packages/cosmos3` and sync training extras (required by the inference import path):

```bash
export GIT_LFS_SKIP_SMUDGE=1
cd packages/cosmos3
uv sync --all-extras --group=cu130-train   # or cu128-train on CUDA 12.x
```

Override with `export COSMOS3_UV_GROUP=cu128-train` before running the notebook on CUDA 12.x drivers.

</Step>
<Step title="Write Reasoner JSON inputs">

Every input requires `"model_mode": "reasoner"` and **`"enable_sound": false`** to avoid strict validation failures on the current Reasoner path.

```json
{
  "model_mode": "reasoner",
  "name": "nano_image",
  "prompt": "Describe what is happening in this image in one sentence.",
  "vision_path": "https://github.com/nvidia-cosmos/cosmos-dependencies/raw/refs/heads/assets/cosmos3/inputs/vision/robot_153.jpg",
  "enable_sound": false
}
```

Capability inputs are written under `packages/cosmos3/outputs/cookbooks/cosmos3/reasoner/nano/inputs/capabilities/` (paths configurable via `COSMOS3_OUTPUT_ROOT` / `COSMOS3_INPUT_DIR`).

</Step>
<Step title="Run inference">

```bash
cd packages/cosmos3
COSMOS_TRAINING=false CUDA_VISIBLE_DEVICES=0 \
MASTER_ADDR=127.0.0.1 MASTER_PORT=29501 RANK=0 WORLD_SIZE=1 LOCAL_RANK=0 \
.venv/bin/python -m cosmos_framework.scripts.inference \
  --parallelism-preset=latency \
  -i outputs/cookbooks/cosmos3/reasoner/nano/inputs/nano_image.json \
  -o outputs/cookbooks/cosmos3/reasoner/nano/cosmos_framework_nano_image \
  --checkpoint-path Cosmos3-Nano \
  --seed=0 \
  --benchmark
```

Text output path:

```text
<output_dir>/<name>/reasoner_text.txt
```

The notebook’s display helper renders prompt, media, model text, and parsed boxes or trajectory points side by side.

</Step>
</Steps>

<Action CoT JSON inputs set `"do_sample": true` with `temperature: 0.6`, `top_p: 0.95`, `top_k: 20`, `repetition_penalty: 1.0`, `presence_penalty: 0.0` to match the “with reasoning” sampler table.

<Info>
The Reasoner README notes scaling **Nano → Super** with `.venv/bin/torchrun` for Framework; the shipped framework notebook cells run **Nano on a single GPU** (`WORLD_SIZE=1`). For Super-scale Framework runs, see [Run Reasoner with Cosmos Framework](/run-reasoner-cosmos-framework).
</Info>

## Prompt patterns by workflow

### Captioning

| Target | Prompt (representative) |
| --- | --- |
| Image | `Caption the image in detail.` |
| Video | `Describe the video in detail.` |

### Temporal localization

| Variant | Prompt highlights | Time format |
| --- | --- | --- |
| Action segments | List action segments; JSON with `start`, `end`, `caption` | `seconds` |
| Event timeline | Notable events in JSON | `mm:ss.ff` |
| Timestamp query | Natural-language event description → interval | `mm:ss.ff`, keys `start`, `end` |
| Interval QA | `What happened between 00:05.64 and 00:17.49?` | — |

### Embodied reasoning

| Scenario | Prompt (abbrev.) |
| --- | --- |
| Robotics next action | `What can be the next immediate action?` + reasoning format |
| Drive scene | Autonomous-vehicle planner: observe critical objects, next action and trajectory |
| Robot planning | `The task is to put flower into the red bottle. Generate a plan consisting of subtasks...` |
| Assisted task | Overall printer-cartridge task + current step `place old ink_cartridge` → next action with reasoning format |

### Common sense, physical plausibility, situation

| Workflow | Prompt (representative) |
| --- | --- |
| Common sense | `Can the countertop support the weight of the juicers?` + reasoning format |
| Physical plausibility | Judge object permanence / trajectories; answer `(A) Possible` or `(B) Impossible`; ignore simulation quality and experimental “rising wall” |
| Situation understanding | `What is the person doing with the skillet? What will the person likely do next in this situation?` |

### Action CoT

| Variant | Task string | Output shape |
| --- | --- | --- |
| Bowl trajectory | `Move the pink bowl to the right` | JSON `{"point_2d": [x, y], "label": "gripper trajectory"}` + reasoning format |
| Flower trajectory | `Put flower into the red bottle` | Same JSON schema on `robot_planning.png` |
| Driving scene | Step-by-step critical objects for safe navigation | Reasoning format over `action_cot_driving_scene.mp4` |

## Verification signals

| Check | vLLM | Cosmos Framework |
| --- | --- | --- |
| Server ready | `curl -fsS http://127.0.0.1:8001/health` (notebook default port) | Inference exits 0; `reasoner_text.txt` exists |
| Model loaded | `client.models.list()` returns Cosmos checkpoint id | Log shows checkpoint download / load for `Cosmos3-Nano` |
| Media found | `asset_path(...)` assert passes at notebook start | `vision_path` resolves (local asset or HTTPS URL) |
| Structured parse | Red boxes / trajectory dots render in notebook | Display cell shows overlay when JSON parses |

<Check>
Successful first run: image caption cell prints multi-sentence description; Framework smoke test writes non-empty `reasoner_text.txt` under the configured output root.
</Check>

## Related pages

<CardGroup>
<Card title="Cookbook environment setup" href="/cookbook-environment">
Shared uv/Docker setup, HF auth, CUDA `cu130`/`cu128` pairs, and vLLM plugin install for Reasoner notebooks.
</Card>
<Card title="Run Reasoner with vLLM" href="/run-reasoner-vllm">
Production `vllm serve` flags, Qwen3-VL message shape, and reasoning-format suffix reference.
</Card>
<Card title="Run Reasoner with Cosmos Framework" href="/run-reasoner-cosmos-framework">
JSON input schema, `enable_sound`, latency preset, and Super multi-GPU torchrun patterns.
</Card>
<Card title="Sampling and prompt parameters" href="/sampling-and-prompt-parameters">
Reasoner sampler tables and structured JSON / chat message conventions.
</Card>
<Card title="Reasoner and Generator" href="/reasoner-and-generator">
When to use Reasoner (text out) vs Generator surfaces.
</Card>
<Card title="Choose an integration" href="/choose-integration">
Pick vLLM vs Cosmos Framework vs Diffusers by deployment goal.
</Card>
</CardGroup>