# Audiovisual cookbook recipes

> End-to-end notebooks for text-to-image, text-to-video, image-to-video with optional sound across Diffusers, Cosmos Framework, and vLLM-Omni; asset layout under assets/prompts and assets/images.

- Repository: NVIDIA/cosmos
- GitHub: https://github.com/NVIDIA/cosmos
- Human docs: https://grok-wiki.com/public/docs/nvidia-cosmos-82de3e90abd9
- Complete Markdown: https://grok-wiki.com/public/docs/nvidia-cosmos-82de3e90abd9/llms-full.txt

## Source Files

- `cookbooks/cosmos3/generator/audiovisual/README.md`
- `cookbooks/cosmos3/generator/audiovisual/run_with_diffusers.ipynb`
- `cookbooks/cosmos3/generator/audiovisual/run_with_cosmos_framework.ipynb`
- `cookbooks/cosmos3/generator/audiovisual/run_with_vllm_omni.ipynb`
- `cookbooks/cosmos3/generator/audiovisual/assets/prompts/image2video/coastal_road_audio.json`
- `cookbooks/cosmos3/generator/audiovisual/assets/prompts/text2image/robot_draping.json`
- `README.md`

---

---
title: "Audiovisual cookbook recipes"
description: "End-to-end notebooks for text-to-image, text-to-video, image-to-video with optional sound across Diffusers, Cosmos Framework, and vLLM-Omni; asset layout under assets/prompts and assets/images."
---

The Generator audiovisual cookbooks under `cookbooks/cosmos3/generator/audiovisual/` ship three Jupyter walkthroughs plus checked-in structured JSON prompts and conditioning images. Each notebook runs the same eight use-case keys (`ASSET_SETS`) through a different backend: in-process `Cosmos3OmniPipeline` (Diffusers), `torchrun -m cosmos_framework.scripts.inference` (Cosmos Framework), or HTTP calls to a running vLLM-Omni server (`/v1/images/generations` and `/v1/videos/sync`).

## Cookbook location

| Artifact | Path |
| --- | --- |
| Folder README and CLI quickstarts | `cookbooks/cosmos3/generator/audiovisual/README.md` |
| Diffusers notebook | `run_with_diffusers.ipynb` |
| Cosmos Framework notebook | `run_with_cosmos_framework.ipynb` |
| vLLM-Omni notebook | `run_with_vllm_omni.ipynb` |
| Shared environment guide | `cookbooks/cosmos3/README.md` |

Run notebooks from the audiovisual folder (or with that folder as the working directory) so relative `assets/...` paths resolve. The root `README.md` examples table links each notebook to nbviewer renders on `main`.

## Three backends at a glance

| Backend | Notebook | Runtime | Primary APIs |
| --- | --- | --- | --- |
| Diffusers | `run_with_diffusers.ipynb` | Dedicated venv + Jupyter kernel `Cosmos3 Diffusers (Python 3.13)` | `Cosmos3OmniPipeline.from_pretrained`, `export_to_video` / `encode_video` |
| Cosmos Framework | `run_with_cosmos_framework.ipynb` | Framework checkout `packages/cosmos3` (or `packages/cosmos-framework`) + `uv sync` | `torchrun -m cosmos_framework.scripts.inference` |
| vLLM-Omni | `run_with_vllm_omni.ipynb` | External Docker or host server on port 8000 | `POST /v1/images/generations`, `POST /v1/videos/sync` via `curl` |

<Note>
All three notebooks share the same `ASSET_SETS` manifest and default 720p 16:9 sampling block. Nano examples run first; Super covers text-to-image plus text-to-video and image-to-video **without** audio only.
</Note>

```text
assets/prompts/*.json  +  assets/images/*.jpg (I2V)
            │
            ▼
    create_payload(use_case)  →  outputs/notebooks/{backend}/payloads/{use_case}.json
            │
            ├── diffusers/     → Cosmos3OmniPipeline  → .png / .mp4
            ├── pytorch/       → torchrun inference  → framework output tree
            └── vllm_omni/     → curl OpenAI-compat  → .png / .mp4
```

## Asset layout

Checked-in inputs live under `assets/`. Prompts are modality-scoped; image-to-video pairs each have a matching JPEG under `assets/images/image2video/`.

:::files
cookbooks/cosmos3/generator/audiovisual/
├── README.md
├── run_with_diffusers.ipynb
├── run_with_cosmos_framework.ipynb
├── run_with_vllm_omni.ipynb
├── assets/
│   ├── prompts/
│   │   ├── text2image/
│   │   │   └── robot_draping.json
│   │   ├── text2video/
│   │   │   ├── robot_kitchen.json
│   │   │   ├── robot_pouring_water_audio.json
│   │   │   └── car_colliding.json
│   │   └── image2video/
│   │       ├── car_driving.json
│   │       ├── coastal_road_audio.json
│   │       └── humanoid_robot.json
│   ├── images/image2video/
│   │   ├── car_driving.jpg
│   │   ├── coastal_road_audio.jpg
│   │   └── humanoid_robot.jpg
│   └── negative_prompts/
│       ├── text2video/neg_prompt.json
│       └── image2video/neg_prompt.json
└── outputs/notebooks/          # created at runtime (gitignored in practice)
    ├── diffusers/
    ├── pytorch/
    └── vllm_omni/
:::

### Prompt files wired into notebooks

The notebooks’ `ASSET_SETS` table selects these prompt/image pairs (extra JSON under `assets/prompts/` such as `car_colliding.json` and `humanoid_robot.json` are available for custom payloads but are not in the default manifest):

| Use-case key | Model | Mode | Prompt | Image (I2V only) | Sound |
| --- | --- | --- | --- | --- | --- |
| `t2i` | Cosmos3-Nano | text2image | `text2image/robot_draping.json` | — | off |
| `t2i_super` | Cosmos3-Super | text2image | `text2image/robot_draping.json` | — | off |
| `t2v_nano_noaudio` | Cosmos3-Nano | text2video | `text2video/robot_kitchen.json` | — | off |
| `t2vs` | Cosmos3-Nano | text2video | `text2video/robot_pouring_water_audio.json` | — | on |
| `i2v_nano_noaudio` | Cosmos3-Nano | image2video | `image2video/car_driving.json` | `car_driving.jpg` | off |
| `i2vs` | Cosmos3-Nano | image2video | `image2video/coastal_road_audio.json` | `coastal_road_audio.jpg` | on |
| `t2v_super_noaudio` | Cosmos3-Super | text2video | `text2video/robot_kitchen.json` | — | off |
| `i2v_super_noaudio` | Cosmos3-Super | image2video | `image2video/car_driving.json` | `car_driving.jpg` | off |

Video modes load a modality-specific negative prompt from `assets/negative_prompts/{text2video|image2video}/neg_prompt.json`. Text-to-image runs pass an empty negative prompt string.

## Structured JSON prompts

Generator cookbooks expect **structured JSON** scene specifications, not a single free-text line. Prompts are passed to inference as compact JSON strings (`json.dumps(..., separators=(",", ":"))`).

### Common fields

| Field | Role |
| --- | --- |
| `subjects` | Array of scene entities with description, appearance, pose, action, spatial placement |
| `background_setting` | Environment description |
| `lighting`, `aesthetics`, `cinematography` | Look, mood, camera behavior |
| `actions`, `segments` | Time-ranged motion beats |
| `temporal_caption` | Video: single narrative caption across time |
| `comprehensive_t2i_caption` | Image: consolidated caption (e.g. `robot_draping.json`) |
| `audio_description` | Sound design text when generating synchronized audio |
| `resolution` | `{"W": 1280, "H": 720}` in checked-in assets |
| `aspect_ratio` | `"16,9"` (comma-separated, matches cookbook payload convention) |
| `duration`, `fps` | Present on video prompt JSON (e.g. `"7s"`, `24`) |

Sound-enabled runs use prompts that include a rich `audio_description` (engine/tire noise, impacts, braking screech, etc.). The `enable_sound` / `generate_sound` flag on the inference call must still be set to actually emit audio.

### Inference payload shape

Each use case writes a runner payload JSON under `outputs/notebooks/{backend}/payloads/{use_case}/{use_case}.json`:

| Field | Default (all notebooks) | Notes |
| --- | --- | --- |
| `model_mode` | `text2image`, `text2video`, or `image2video` | Drives pipeline branch |
| `prompt` | Compact JSON string from asset file | |
| `negative_prompt` | Compact JSON or `""` | Empty for T2I |
| `enable_sound` | per `ASSET_SETS` | Maps to `enable_sound` (Diffusers/Framework) or `generate_sound` (vLLM-Omni) |
| `num_steps` | `35` | |
| `guidance` | `6.0` | CFG strength |
| `shift` | `10.0` | UniPC `flow_shift` |
| `fps` | `24` | |
| `num_frames` | `189` (video); `1` for Framework T2I | ~7.9 s at 24 fps |
| `resolution` | `"720"` | Maps to 720×1280 when `aspect_ratio` is `"16,9"` |
| `aspect_ratio` | `"16,9"` | |
| `seed` | `1234` (Diffusers), `0` (Framework, vLLM-Omni) | |
| `vision_path` | relative path | Image2video only; resolved beside payload file |

Image2video payloads store `vision_path` relative to the payload directory so Framework and vLLM-Omni can resolve local files inside the mounted workspace.

## Default sampling and templates

All backends disable resolution and duration templates in cookbook runs:

- Diffusers: `add_resolution_template=False`, `add_duration_template=False`
- vLLM-Omni `extra_params`: `"use_resolution_template": false`, `"use_duration_template": false`, `"guardrails": true`

Diffusers additionally sets `UniPCMultistepScheduler` with `flow_shift` from the payload (default `10.0`).

## Notebook workflow pattern

Every use-case section follows the same three steps:

<Steps>
<Step title="Create payload">
Run `create_payload("<use_case_key>", backend="diffusers"|"pytorch"|"vllm_omni")` to materialize payload JSON and print paths. Environment variables such as `COSMOS3_DIFFUSERS_T2I_INPUT` / `_OUTPUT` are set for Framework bash cells.
</Step>
<Step title="Run inference">
Execute the backend-specific run cell (`run_diffusers_payload`, `torchrun` bash, or `run_vllm_payload`).
</Step>
<Step title="View results">
Call `view_run(output_dir)` to embed PNG or MP4 in the notebook.
</Step>
</Steps>

Default output root: `cookbooks/cosmos3/generator/audiovisual/outputs/notebooks/` (override with `COSMOS3_AUDIOVISUAL_OUTPUT_ROOT`). Generated media paths:

`outputs/notebooks/{diffusers|pytorch|vllm_omni}/{use_case_key}/{use_case_key}.png` or `.mp4`

## Run with Diffusers

**Environment:** `COSMOS3_DIFFUSERS_VENV` (default `<cosmos-root>/.venv-cosmos3-diffusers`), `COSMOS3_TORCH_BACKEND` (`cu130` or `cu128`), Hugging Face auth.

<Steps>
<Step title="Install">
`uv venv` + `uv pip install` diffusers (git), torch, accelerate, `cosmos_guardrail`, ipykernel; register kernel `cosmos3-diffusers`.
</Step>
<Step title="Switch kernel">
Select **Cosmos3 Diffusers (Python 3.13)** and run the restore-environment cell so `sys.executable` matches the venv.
</Step>
<Step title="Generate">
`Cosmos3OmniPipeline.from_pretrained("nvidia/Cosmos3-Nano"|"nvidia/Cosmos3-Super")` with `torch.bfloat16`, `flow_shift` on scheduler, then `run_diffusers_payload`. Video with sound uses `encode_video` when `result.sound` is present.
</Step>
</Steps>

**CLI quickstart** (from folder README): load `assets/prompts/text2video/robot_kitchen.json`, `num_frames=189`, `1280x720`, `enable_sound=False`, `export_to_video` to `/tmp/cosmos3_t2v_diffusers.mp4`.

## Run with Cosmos Framework

**Environment:** clone `cosmos-framework` to `packages/cosmos3`, `uv sync --all-extras --group=cu130-train` (or `cu128-train` via `COSMOS3_UV_GROUP`), `COSMOS3_NUM_GPUS` (default `4`), `CUDA_VISIBLE_DEVICES`.

Inference command shape (each use case):

```bash
torchrun --nproc-per-node="$COSMOS3_NUM_GPUS" \
  -m cosmos_framework.scripts.inference \
  --parallelism-preset=throughput \
  -i "$COSMOS3_PYTORCH_<USE_CASE>_INPUT" \
  -o "$COSMOS3_PYTORCH_<USE_CASE>_OUTPUT" \
  --checkpoint-path "Cosmos3-Nano"   # or Cosmos3-Super
  --seed=0
```

**CLI quickstart:** single-GPU text-to-video from `assets/prompts/text2video/robot_kitchen.json` to `/tmp/cosmos3_t2v_framework` with `--checkpoint-path Cosmos3-Nano`.

<Warning>
Headless Linux may need `libxcb1`, `libgl1`, `libglib2.0-0` if imports fail with `libxcb.so.1` errors (documented in notebooks and root README).
</Warning>

## Run with vLLM-Omni

The notebook does **not** start the server; it assumes Docker (or host) `vllm serve` is already listening.

| Modality | Endpoint | Response |
| --- | --- | --- |
| Text-to-image | `POST {base}/v1/images/generations` | JSON with `data[0].b64_json` → PNG |
| Text-to-video / image-to-video | `POST {base}/v1/videos/sync` | Raw `video/mp4` body |

<ParamField body="COSMOS3_VLLM_NANO_BASE_URL" type="string">
Base URL for Nano (default `http://localhost:8000`). Normalized to `{url}/v1`.
</ParamField>

<ParamField body="COSMOS3_VLLM_SUPER_BASE_URL" type="string">
Separate base URL when Nano and Super run on different ports.
</ParamField>

Video form fields mirror Diffusers: `prompt`, `negative_prompt`, `size` (`1280x720`), `num_frames`, `fps`, `num_inference_steps`, `guidance_scale`, `flow_shift`, `seed`, `extra_params` JSON. Image-to-video adds `input_reference=@/path/to/image`. Audio: `generate_sound=true` and `sound_duration` derived from `num_frames / fps`.

**Server quickstart** (README): `docker run` `vllm/vllm-omni:cosmos3` serving `nvidia/Cosmos3-Nano` with `--allowed-local-media-path /` and workspace mount; Super adds `--tensor-parallel-size` and `--enable-layerwise-offload`.

For image-to-video via HTTP only, post to `/v1/videos/sync` with `files={"input_reference": ...}` as in the README Python snippet.

## Verification signals

| Check | Success signal |
| --- | --- |
| Diffusers GPU cell | `cuda available: True`, expected kernel python path |
| Framework `uv sync` | `$COSMOS3_UV_ENV/bin/python` exists |
| vLLM endpoint cell | Prints `images/generations` and `videos/sync` URLs per model |
| Generation | `wrote .../outputs/notebooks/.../{use_case}.mp4` or `.png` |
| Viewer | Inline video/image in notebook; no `_preview`/`_browser` suffix pollution |

## Prerequisites (all notebooks)

- Linux with NVIDIA GPU, gated Hugging Face models (`uvx hf auth login` or `HF_TOKEN`)
- `uv >= 0.11.3` and CUDA backend tag matching driver (`cu130` / `cu128`)
- Cosmos Framework / vLLM paths: access to `NVIDIA/cosmos-framework` where applicable
- Sufficient disk for weights, uv cache, and outputs

Centralized setup steps: [Cookbook environment setup](/cookbook-environment).

## Related pages

<CardGroup>
<Card title="Cookbook environment setup" href="/cookbook-environment">
Shared uv/Docker setup for Diffusers, Framework, and vLLM-Omni backends.
</Card>
<Card title="Run Generator with Diffusers" href="/run-generator-diffusers">
`Cosmos3OmniPipeline` flags, scheduler, and export paths.
</Card>
<Card title="Run Generator with Cosmos Framework" href="/run-generator-cosmos-framework">
`torchrun` inference, parallelism presets, checkpoint paths.
</Card>
<Card title="Run Generator with vLLM-Omni" href="/run-generator-vllm-omni">
Docker serve, tensor parallel, guardrails, deploy config.
</Card>
<Card title="Diffusers pipeline reference" href="/diffusers-pipeline-reference">
Pipeline modes and call arguments.
</Card>
<Card title="vLLM-Omni API reference" href="/vllm-omni-api-reference">
OpenAI-compatible video/image endpoints and form fields.
</Card>
<Card title="Sampling and prompt parameters" href="/sampling-and-prompt-parameters">
Structured JSON schema and template toggles.
</Card>
<Card title="Input and output specifications" href="/input-output-specifications">
Resolution tiers, frame counts, and sound output specs.
</Card>
<Card title="Choose an integration" href="/choose-integration">
When to pick Diffusers vs Framework vs vLLM-Omni.
</Card>
</CardGroup>