# Run Reasoner with Cosmos Framework

> Build reasoner JSON inputs (model_mode, vision_path, enable_sound), run cosmos_framework.scripts.inference with latency preset, and read reasoner_text.txt outputs; scale Nano to Super via torchrun.

- Repository: NVIDIA/cosmos
- GitHub: https://github.com/NVIDIA/cosmos
- Human docs: https://grok-wiki.com/public/docs/nvidia-cosmos-82de3e90abd9
- Complete Markdown: https://grok-wiki.com/public/docs/nvidia-cosmos-82de3e90abd9/llms-full.txt

## Source Files

- `cookbooks/cosmos3/reasoner/README.md`
- `cookbooks/cosmos3/reasoner/run_with_cosmos_framework.ipynb`
- `cookbooks/cosmos3/README.md`
- `cookbooks/cosmos3/reasoner/assets/robot_planning.png`
- `cookbooks/cosmos3/reasoner/assets/describe_anything.png`
- `README.md`

---

---
title: "Run Reasoner with Cosmos Framework"
description: "Build reasoner JSON inputs (model_mode, vision_path, enable_sound), run cosmos_framework.scripts.inference with latency preset, and read reasoner_text.txt outputs; scale Nano to Super via torchrun."
---

Cosmos3 Reasoner inference on the native PyTorch path runs through `cosmos_framework.scripts.inference` in a [Cosmos Framework](https://github.com/NVIDIA/cosmos-framework) checkout (`packages/cosmos3`). Each job takes one JSON input file (`-i`), writes text under `{output_dir}/{name}/reasoner_text.txt`, and uses `--parallelism-preset=latency` with `COSMOS_TRAINING=false` for cookbook Reasoner runs.

## Prerequisites

| Requirement | Notes |
| --- | --- |
| Linux + NVIDIA GPU | Ampere, Hopper, or Blackwell per project support matrix |
| `uv`, `git`, `git-lfs` | Framework install uses `uv sync` |
| Hugging Face access | Gated Cosmos3 repos; `uvx hf@latest auth login` or `HF_TOKEN` |
| Framework checkout access | HTTPS or SSH to `NVIDIA/cosmos-framework` |
| Disk | Nano download plus CUDA deps can use tens of GiB |

Full shared setup (CUDA `cu130-train` / `cu128-train` pairing, clone path, GPU verify) lives on the cookbook environment page. Install the framework venv at `packages/cosmos3/.venv` before running commands below.

<Steps>
<Step title="Install Cosmos Framework">

From the `cosmos` repo root:

```bash
mkdir -p packages
git clone https://github.com/NVIDIA/cosmos-framework.git packages/cosmos3
cd packages/cosmos3

export GIT_LFS_SKIP_SMUDGE=1
uv sync --all-extras --group=cu130-train   # use cu128-train on CUDA 12.x drivers
```

</Step>
<Step title="Verify GPU">

```bash
cd packages/cosmos3
.venv/bin/python - <<'PY'
import torch
print("cuda available:", torch.cuda.is_available())
print("device count:", torch.cuda.device_count())
PY
```

</Step>
</Steps>

## Reasoner input JSON

Point `-i` at a single JSON file. Cookbook Reasoner inputs set `model_mode` to `"reasoner"` and include `enable_sound: false` — shipped examples fail argument validation without it.

### Core fields

| Field | Required | Type | Role |
| --- | --- | --- | --- |
| `model_mode` | yes | string | Must be `"reasoner"` |
| `name` | yes | string | Output subdirectory name under `-o` |
| `prompt` | yes | string | User instruction or task text |
| `enable_sound` | yes (today) | bool | Set `false` for current Reasoner path |
| `vision_path` | no | string | HTTP(S) URL or local path to an image |

### Optional sampling fields

Capability prompts in `run_with_cosmos_framework.ipynb` also use:

| Field | Example use |
| --- | --- |
| `max_new_tokens` | `4096` for long captions and structured outputs |
| `do_sample` | `true` for trajectory / chain-of-thought prompts |
| `temperature` | `0.6` with reasoning-format prompts |
| `top_p` | `0.95` with reasoning |
| `top_k` | `20` |
| `repetition_penalty` | `1.0` |
| `presence_penalty` | `0.0` with reasoning (vs `1.5` without in README sampling table) |

<ParamField body="model_mode" type="string" required>
Must be `"reasoner"`. Selects autoregressive text output (not Generator diffusion modes).
</ParamField>

<ParamField body="vision_path" type="string">
Image URL or filesystem path. Framework Reasoner currently treats this as a PIL image input. Video Reasoner workflows belong on the vLLM path.
</ParamField>

<ParamField body="enable_sound" type="boolean" required>
Set `false` for all current cookbook Reasoner JSON. Omitting or setting `true` triggers strict argument-validation failure on the shipped path.
</ParamField>

### Minimal examples

Text-only:

```json
{
  "model_mode": "reasoner",
  "name": "nano_text",
  "prompt": "Describe a modern robotics research laboratory in one sentence.",
  "enable_sound": false
}
```

Image-conditioned:

```json
{
  "model_mode": "reasoner",
  "name": "robot_image",
  "prompt": "Describe what is happening in this image in one sentence.",
  "vision_path": "https://github.com/nvidia-cosmos/cosmos-dependencies/raw/refs/heads/assets/cosmos3/inputs/vision/robot_153.jpg",
  "enable_sound": false
}
```

Local cookbook assets (for example `robot_planning.png`, `describe_anything.png`, `grounding_2d.png`) resolve under `cookbooks/cosmos3/reasoner/assets/`.

## Run inference

Work from the framework checkout (`cd packages/cosmos3`). The entrypoint is `python -m cosmos_framework.scripts.inference` (or `.venv/bin/torchrun` for multi-GPU Super).

```text
cosmos repo
└── packages/cosmos3/          # COSMOS3_REPO, run commands here
    ├── .venv/
    └── outputs/cookbooks/cosmos3/reasoner/...
```

### Cosmos3-Nano (single GPU)

Nano fits one GPU. Export distributed env vars even for `WORLD_SIZE=1` (cookbook pattern):

```bash
cd packages/cosmos3

mkdir -p outputs/cookbooks/cosmos3/reasoner/inputs
cat > outputs/cookbooks/cosmos3/reasoner/inputs/robot_image.json <<'JSON'
{
  "model_mode": "reasoner",
  "name": "robot_image",
  "prompt": "Describe what is happening in this image in one sentence.",
  "vision_path": "https://github.com/nvidia-cosmos/cosmos-dependencies/raw/refs/heads/assets/cosmos3/inputs/vision/robot_153.jpg",
  "enable_sound": false
}
JSON

COSMOS_TRAINING=false CUDA_VISIBLE_DEVICES=0 \
MASTER_ADDR=127.0.0.1 MASTER_PORT=29501 RANK=0 WORLD_SIZE=1 LOCAL_RANK=0 \
.venv/bin/python -m cosmos_framework.scripts.inference \
  --parallelism-preset=latency \
  -i outputs/cookbooks/cosmos3/reasoner/inputs/robot_image.json \
  -o outputs/cookbooks/cosmos3/reasoner/nano/cosmos_framework_image \
  --checkpoint-path Cosmos3-Nano \
  --seed=0 \
  --benchmark
```

| CLI flag | Reasoner cookbook value | Notes |
| --- | --- | --- |
| `--parallelism-preset` | `latency` | Generator audiovisual examples use `throughput`; Reasoner cookbooks use `latency` |
| `-i` | path to input JSON | One sample per invocation |
| `-o` | output run directory | Per-sample folder is `{name}/` inside this directory |
| `--checkpoint-path` | `Cosmos3-Nano` or `Cosmos3-Super` | Short names; weights download from Hugging Face on first run |
| `--seed` | `0` | Reproducibility |
| `--benchmark` | optional | Writes `benchmark.json` next to sample outputs with timing aggregates |
| `COSMOS_TRAINING` | `false` | Set for Reasoner cookbook runs |

### Cosmos3-Super (multi-GPU)

`Cosmos3-Super` (64B) needs multiple GPUs. The Reasoner cookbook points to `.venv/bin/torchrun`; the audiovisual Framework notebook uses four processes for Super (`COSMOS3_NUM_GPUS=4`, `CUDA_VISIBLE_DEVICES=0,1,2,3`).

```bash
cd packages/cosmos3

export CUDA_VISIBLE_DEVICES=0,1,2,3
export COSMOS3_NUM_GPUS=4
export COSMOS3_MASTER_ADDR=127.0.0.1
export COSMOS3_MASTER_PORT=29502

COSMOS_TRAINING=false \
.venv/bin/torchrun \
  --nproc-per-node="$COSMOS3_NUM_GPUS" \
  --master-addr="$COSMOS3_MASTER_ADDR" \
  --master-port="$COSMOS3_MASTER_PORT" \
  -m cosmos_framework.scripts.inference \
  --parallelism-preset=latency \
  -i outputs/cookbooks/cosmos3/reasoner/inputs/robot_image.json \
  -o outputs/cookbooks/cosmos3/reasoner/super/cosmos_framework_image \
  --checkpoint-path Cosmos3-Super \
  --seed=0 \
  --benchmark
```

<Note>
Match `--nproc-per-node` to the number of visible GPUs. vLLM Reasoner serves Super with `--tensor-parallel-size 4` on four GPUs; align Framework GPU count with your hardware and memory headroom.
</Note>

## Output layout

For `-o outputs/.../cosmos_framework_image` and input `"name": "robot_image"`:

```text
outputs/.../cosmos_framework_image/
├── benchmark.json              # when --benchmark is set (run-level aggregates)
└── robot_image/
    └── reasoner_text.txt       # model text output
```

<ResponseField name="reasoner_text.txt" type="string">
Plain-text Reasoner completion for the prompt (and vision conditioning when `vision_path` is set). Read with `cat` or your notebook display helper.
</ResponseField>

With `--benchmark`, inspect timing:

```bash
cat outputs/cookbooks/cosmos3/reasoner/nano/cosmos_framework_image/benchmark.json
```

The notebook prints the `average` object from that file when present.

## Capability workflows (image)

`run_with_cosmos_framework.ipynb` builds inputs under `outputs/.../reasoner/nano/inputs/` and `inputs/capabilities/`, then runs Nano with the same CLI pattern. Bundled assets drive local `vision_path` values.

| Input `name` | Task | Asset / vision |
| --- | --- | --- |
| `image_caption_detail` | Detailed caption | Remote robot image URL |
| `robot_planning` | Subtask plan for manipulation | `assets/robot_planning.png` |
| `ground_load_bbox` | 2D bounding box JSON | `assets/grounding_2d.png` |
| `describe_marked_subjects` | Describe marked subjects (JSON) | `assets/describe_anything.png` |
| `trajectory_bowl` / `trajectory_flower` | 2D gripper trajectory + `redacted_reasoning` format | `action_cot_trajectory.png`, `robot_planning.png` |

Trajectory prompts append the reasoning-format block from the main README (`redacted_reasoning` tags) and set sampling fields (`do_sample`, `temperature`, `top_p`, etc.).

<Warning>
Framework Reasoner expects **images** via `vision_path`. Video assets under `cookbooks/cosmos3/reasoner/assets/*.mp4` are exercised in `run_with_vllm.ipynb`, not the Framework image path.
</Warning>

## Environment overrides

| Variable | Default | Purpose |
| --- | --- | --- |
| `COSMOS3_REPO` | `<cosmos>/packages/cosmos3` | Framework checkout root |
| `COSMOS3_UV_GROUP` | `cu130-train` | `cu128-train` on CUDA 12.x drivers |
| `COSMOS3_OUTPUT_ROOT` | `.../reasoner/nano` under framework outputs | Notebook output base |
| `CUDA_VISIBLE_DEVICES` | `0` (Nano) | GPU selection |
| `HF_HOME` | `~/.cache/huggingface` | Model cache location |
| `COSMOS_TRAINING` | `false` for Reasoner | Disables training code paths for inference |

## Troubleshooting

| Symptom | Likely fix |
| --- | --- |
| `cuda available: False` after `uv sync` | Wrong `COSMOS3_UV_GROUP` for driver; use `cu128-train` on CUDA 12.x |
| Reasoner JSON validation error | Add `"enable_sound": false` |
| Video input not working | Use [Run Reasoner with vLLM](/run-reasoner-vllm) for video; keep Framework on images |
| `uv` / `--torch-backend` errors | Upgrade `uv` (≥ 0.11.3 per cookbook notes); see troubleshooting page |
| First run slow / large download | `Cosmos3-Nano` / `Cosmos3-Super` fetch from Hugging Face; ensure `HF_TOKEN` or `hf auth login` |

## Related pages

<CardGroup>
<Card title="Cookbook environment setup" href="/cookbook-environment">
Clone Framework, `uv sync` train groups, and verify GPU before Reasoner runs.
</Card>
<Card title="Reasoner and Generator" href="/reasoner-and-generator">
MoT surfaces: Reasoner (text out) vs Generator (vision/sound/action out).
</Card>
<Card title="Run Reasoner with vLLM" href="/run-reasoner-vllm">
OpenAI-compatible Reasoner for image and video, including Super on four GPUs.
</Card>
<Card title="Reasoner cookbook recipes" href="/reasoner-cookbooks">
Captioning, grounding, temporal localization, and video workflows with bundled media.
</Card>
<Card title="Sampling and prompt parameters" href="/sampling-and-prompt-parameters">
Reasoner sampling tables and `redacted_reasoning` prompt suffix.
</Card>
<Card title="Choose an integration" href="/choose-integration">
When to pick Framework vs vLLM vs Diffusers for research or serving.
</Card>
</CardGroup>