# Sampling and prompt parameters

> Generator prompt-upsampling defaults, Reasoner sampling tables (with/without reasoning), structured JSON prompt schema, Qwen3-VL message shape, and redacted_reasoning format instruction.

- Repository: NVIDIA/cosmos
- GitHub: https://github.com/NVIDIA/cosmos
- Human docs: https://grok-wiki.com/public/docs/nvidia-cosmos-82de3e90abd9
- Complete Markdown: https://grok-wiki.com/public/docs/nvidia-cosmos-82de3e90abd9/llms-full.txt

## Source Files

- `README.md`
- `cookbooks/cosmos3/generator/audiovisual/assets/prompts/text2video/robot_pouring_water_audio.json`
- `cookbooks/cosmos3/generator/audiovisual/assets/negative_prompts/image2video/neg_prompt.json`
- `cookbooks/cosmos3/generator/audiovisual/run_with_diffusers.ipynb`
- `cookbooks/cosmos3/reasoner/run_with_vllm.ipynb`

---

---
title: "Sampling and prompt parameters"
description: "Generator prompt-upsampling defaults, Reasoner sampling tables (with/without reasoning), structured JSON prompt schema, Qwen3-VL message shape, and redacted_reasoning format instruction."
---

Cosmos 3 splits prompt and sampling configuration across two surfaces: the **Generator** accepts dense structured JSON (often produced by prompt upsampling) plus diffusion denoising parameters (`guidance_scale`, `num_inference_steps`, `flow_shift`, `seed`), while the **Reasoner** uses Qwen3-VL-compatible chat messages with autoregressive sampling (`temperature`, `top_p`, `top_k`, penalties) and an optional `redacted_reasoning` output format for chain-of-thought tasks.

## Generator prompt upsampling

Prompt upsampling expands a short scene description into a dense structured JSON prompt suitable for diffusion conditioning. The repository documents these **upsampling** defaults (autoregressive text generation used to author prompts, not diffusion sampling):

| Parameter | Value |
| --- | ---: |
| `max_tokens` | `20000` |
| `temperature` | `0.7` |
| `top_p` | `0.8` |
| `top_k` | `20` |
| `repetition_penalty` | `1.0` |
| `presence_penalty` | `1.5` |
| `seed` | `3407` |

Checked-in examples under `cookbooks/cosmos3/generator/audiovisual/assets/prompts/` are already upsampled; you can reuse them directly or regenerate prompts with the same parameter profile.

## Generator diffusion sampling

Audiovisual cookbooks share a `FIXED_SAMPLING` block for **diffusion** generation across Diffusers, vLLM-Omni, and Cosmos Framework paths:

| Parameter | Cookbook default | Notes |
| --- | ---: | --- |
| `num_steps` / `num_inference_steps` | `35` | Denoising steps |
| `guidance` / `guidance_scale` | `6.0` | Classifier-free guidance; use `guidance_scale`, not `true_cfg_scale` |
| `shift` / `flow_shift` | `10.0` | UniPC scheduler flow shift (Diffusers sets `flow_shift` on `UniPCMultistepScheduler`) |
| `fps` | `24` | Output frame rate |
| `num_frames` | `189` | ~7.9 s at 24 FPS |
| `resolution` | `"720"` | Maps to 1280×720 with `aspect_ratio` `"16,9"` |
| `aspect_ratio` | `"16,9"` | Comma-separated form used in payloads |
| `seed` | `1234` (Diffusers), `0` (vLLM-Omni / Framework) | Backend-specific in notebooks |

Supported payload resolution mapping in notebooks:

| `resolution` | `aspect_ratio` | Height × width |
| --- | --- | --- |
| `720` | `16,9` | 720 × 1280 |
| `256` | `16,9` | 192 × 320 |

README quickstart examples may use different values (for example `guidance_scale=4.0`, `num_frames=81` in curl samples). Treat cookbook `FIXED_SAMPLING` as the reference for asset-driven runs; override per request for experiments.

<Warning>
Action Generator workflows (forward/inverse dynamics) use `guidance_scale=1.0` in action cookbooks — not the audiovisual `6.0` default.
</Warning>

### Template toggles

Cookbooks disable built-in prompt templates so structured JSON carries resolution and duration:

```python
add_resolution_template=False
add_duration_template=False
```

vLLM-Omni passes the same flags in `extra_params`:

```json
{"use_resolution_template": false, "use_duration_template": false, "guardrails": true}
```

## Structured JSON prompt schema

Generator audiovisual prompts are JSON objects checked in under `cookbooks/cosmos3/generator/audiovisual/assets/prompts/{text2image,text2video,image2video}/`. They are passed to the model as **compact JSON strings** (`json.dumps(..., separators=(",", ":"))`), not as raw prose.

### Top-level fields

| Field | Typical use | Present in |
| --- | --- | --- |
| `subjects` | Array of scene entities with per-subject attributes | All modes |
| `background_setting` | Environment description | All modes |
| `lighting` | `conditions`, `direction`, `shadows`, `illumination_effect` | All modes |
| `aesthetics` | `composition`, `color_scheme`, `mood_atmosphere`, `patterns` | All modes |
| `cinematography` | Camera framing, motion, DOF, lens | All modes |
| `style_medium`, `artistic_style`, `context` | Medium and narrative context | All modes |
| `text_and_signage_elements` | On-screen text (often `[]`) | All modes |
| `resolution` | `{"H": int, "W": int}` | All modes |
| `aspect_ratio` | e.g. `"16,9"` (comma, not colon) | All modes |
| `actions` | Timed action beats (`time`, `description`) | Video prompts |
| `segments` | Shot segments with `time_range`, `key_changes`, `camera` | Video prompts |
| `transitions` | Edit transitions (often `[]`) | Video prompts |
| `temporal_caption` | Dense timeline prose | `text2video`, `image2video` |
| `comprehensive_t2i_caption` | Single rich caption | `text2image` |
| `duration`, `fps` | Clip timing metadata | Video prompts |
| `audio_description` | Sound design prose | Prompts with sound |
| `subject_details` | Extra keyed detail (e.g. fabric, pins) | Some `text2image` |
| `quadrant_scan` | Spatial quadrant descriptions | Some `text2image` |
| `physical_realism` | Physics constraints prose | Some negative prompts |

### Per-subject fields

Each `subjects[]` entry commonly includes:

`description`, `appearance_details`, `relationship`, `location`, `relative_size`, `orientation`, `pose`, `action`, `state_changes`, and optional humanoid fields (`clothing`, `expression`, `gender`, `age`, `skin_tone_and_texture`, `facial_features`, `number_of_subjects`, `number_of_arms`, `number_of_legs`). Image prompts may add `number_of_hands`, `number_of_fingers`.

### Negative prompts

Video modes (`text2video`, `image2video`) pair a structured positive prompt with a structured **negative** JSON under `assets/negative_prompts/{mode}/neg_prompt.json`. The negative schema mirrors the positive layout (subjects, lighting, cinematography, `temporal_caption`, etc.) and describes artifacts to avoid. **Text-to-image** runs use an empty `negative_prompt` string in cookbooks.

### Serialization and payloads

```text
prompt file (.json)
    → compact_json_file() → payload["prompt"] (string)
    → pipeline / API (json.dumps(prompt) at call site)
```

Notebook `create_payload()` writes a sidecar JSON per use case:

```json
{
  "model_mode": "text2video",
  "name": "t2v_nano_noaudio",
  "prompt": "{...compact structured JSON...}",
  "negative_prompt": "{...compact negative JSON or \"\"}",
  "enable_sound": false,
  "num_steps": 35,
  "guidance": 6.0,
  "shift": 10.0,
  "fps": 24,
  "num_frames": 189,
  "resolution": "720",
  "aspect_ratio": "16,9",
  "seed": 1234
}
```

Image-to-video payloads add `vision_path` (relative image path). Preview helpers resolve captions via `temporal_caption`, `comprehensive_t2i_caption`, or `extra.prompt`.

## Reasoner sampling parameters

Reasoner outputs are autoregressive text. Documented defaults depend on whether the task uses explicit chain-of-thought formatting.

### Without reasoning

Use for captioning, VQA, grounding, and other direct-answer tasks. vLLM cookbook calls often omit explicit sampling kwargs and rely on server defaults, with `max_tokens=4096` and `seed=0` on image examples.

| Parameter | Value |
| --- | ---: |
| `temperature` | `0.7` |
| `top_p` | `0.8` |
| `top_k` | `20` |
| `repetition_penalty` | `1.0` |
| `presence_penalty` | `1.5` |

### With reasoning

Use when the user prompt includes the `redacted_reasoning` format block (see below). Cookbooks and Framework inputs set:

| Parameter | Value |
| --- | ---: |
| `temperature` | `0.6` |
| `top_p` | `0.95` |
| `top_k` | `20` |
| `repetition_penalty` | `1.0` |
| `presence_penalty` | `0.0` |

vLLM example (OpenAI client):

```python
client.chat.completions.create(
    model=MODEL,
    messages=[...],
    max_tokens=4096,
    temperature=0.6,
    top_p=0.95,
    presence_penalty=0.0,
    extra_body={"top_k": 20, "repetition_penalty": 1.0},
)
```

Cosmos Framework Reasoner JSON can mirror the same fields (`do_sample`, `temperature`, `top_p`, `top_k`, `repetition_penalty`, `presence_penalty`, `max_new_tokens`) on trajectory and planning inputs.

### Video multimodal processing

Video requests in the vLLM Reasoner notebook commonly pass:

```python
extra_body={"mm_processor_kwargs": {"fps": 4, "do_sample_frames": True}}
```

Pair with server `--media-io-kwargs '{"video": {"num_frames": -1}}'` so the encoder can see all frames before downstream sampling.

## Qwen3-VL message shape

Reasoner serving follows **Qwen3-VL-compatible** chat conventions. Production vLLM uses `Cosmos3ReasonerForConditionalGeneration` with an OpenAI-compatible `/v1/chat/completions` API.

### Recommended layout

Include an optional `system` message, then a `user` message whose `content` is a **multimodal array** (vision first, then text):

```json
[
  {
    "role": "system",
    "content": [{"type": "text", "text": "You are a helpful assistant."}]
  },
  {
    "role": "user",
    "content": [
      {"type": "video_url", "video_url": "https://example.com/video.mp4"},
      {"type": "text", "text": "List the notable events with approximate timestamps."}
    ]
  }
]
```

### URL forms used in cookbooks

| Modality | Content block | URL shape |
| --- | --- | --- |
| Image | `{"type": "image_url", "image_url": {"url": "<url>"}}` | HTTPS or `file://` from `Path(...).resolve().as_uri()` |
| Video | `{"type": "video_url", "video_url": {"url": "<url>"}}` | Same; local video requires `--allowed-local-media-path` on the server |

README shows a flat `video_url` string form for illustration; cookbook clients use the nested `{"url": ...}` object — match the cookbook shape when copying runnable examples.

Image-only quickstarts may omit `system` and use a single `user` turn:

```python
messages=[{
    "role": "user",
    "content": [
        {"type": "image_url", "image_url": {"url": image_url}},
        {"type": "text", "text": "Caption the image in detail."},
    ],
}]
```

## `redacted_reasoning` format instruction

For chain-of-thought Reasoner tasks (embodied next-action, action CoT trajectories, planning), append this block to the **user text** (after the task-specific instruction):

```text
Answer the question using the following format:

<think>
Your reasoning.
</think>

Write your final answer immediately after the </think> tag.
```

The model emits reasoning inside `redacted_reasoning` tags; parsers split on `</think>` and read JSON or prose from the suffix. Notebook helpers strip fenced code and extract JSON arrays for trajectory visualization.

<Note>
Apply the **with reasoning** sampling table whenever this format block is present. Set `presence_penalty` to `0.0` (not `1.5`) so reasoning tokens are not suppressed.
</Note>

Compact single-line variant (equivalent intent):

```text
Answer the question using the following format: <think> Your reasoning. </think> Write your final answer immediately after the </think> tag.
```

## Quick reference: which parameters where

```text
┌─────────────────────────────────────────────────────────────────┐
│ Generator (diffusion)                                           │
│  • Structured JSON prompt + optional negative JSON              │
│  • num_inference_steps, guidance_scale, flow_shift, seed        │
│  • num_frames, fps, size/height/width                           │
│  • Upsampling (separate AR pass): README table, seed 3407       │
├─────────────────────────────────────────────────────────────────┤
│ Reasoner (autoregressive)                                       │
│  • Qwen3-VL messages (image_url / video_url + text)             │
│  • temperature, top_p, top_k, repetition/presence penalties     │
│  • Optional redacted_reasoning wrapper + with-reasoning table   │
│  • mm_processor_kwargs for video frame sampling                 │
└─────────────────────────────────────────────────────────────────┘
```

## Related pages

<CardGroup>
<Card title="Input and output specifications" href="/input-output-specifications">
Resolution tiers, frame counts, prompt length limits, and output formats that bound sampling choices.
</Card>
<Card title="Run Generator with Diffusers" href="/run-generator-diffusers">
`Cosmos3OmniPipeline` calls with structured JSON prompts and scheduler `flow_shift`.
</Card>
<Card title="Run Reasoner with vLLM" href="/run-reasoner-vllm">
Serve Reasoner and send Qwen3-VL chat completions with local media paths.
</Card>
<Card title="vLLM-Omni API reference" href="/vllm-omni-api-reference">
Request fields for `guidance_scale`, `flow_shift`, `extra_params`, and curl `--form-string` rules.
</Card>
<Card title="Reasoner cookbooks" href="/reasoner-cookbooks">
Runnable captioning, grounding, CoT, and situation-understanding examples with bundled media.
</Card>
<Card title="Audiovisual cookbooks" href="/audiovisual-cookbooks">
End-to-end Generator notebooks and asset layout under `assets/prompts`.
</Card>
</CardGroup>