# Diffusers pipeline reference

> Cosmos3OmniPipeline.from_pretrained modes (text-to-image, text-to-video, image-to-video, text-to-video-with-sound), key call arguments, export_to_video, and torch-backend install pairing.

- Repository: NVIDIA/cosmos
- GitHub: https://github.com/NVIDIA/cosmos
- Human docs: https://grok-wiki.com/public/docs/nvidia-cosmos-82de3e90abd9
- Complete Markdown: https://grok-wiki.com/public/docs/nvidia-cosmos-82de3e90abd9/llms-full.txt

## Source Files

- `README.md`
- `cookbooks/cosmos3/generator/audiovisual/README.md`
- `cookbooks/cosmos3/generator/audiovisual/run_with_diffusers.ipynb`
- `cookbooks/cosmos3/generator/audiovisual/assets/prompts/text2video/car_colliding.json`
- `cookbooks/cosmos3/generator/audiovisual/assets/images/image2video/car_driving.jpg`

---

---
title: "Diffusers pipeline reference"
description: "Cosmos3OmniPipeline.from_pretrained modes (text-to-image, text-to-video, image-to-video, text-to-video-with-sound), key call arguments, export_to_video, and torch-backend install pairing."
---

`Cosmos3OmniPipeline` is the Hugging Face Diffusers entry point for Cosmos 3 Generator audiovisual workflows. It loads full omnimodal checkpoints (`nvidia/Cosmos3-Nano`, `nvidia/Cosmos3-Super`), runs diffusion with a UniPC scheduler and configurable `flow_shift`, accepts compact JSON scene prompts, and returns frame tensors you save as PNG (single frame) or MP4 via `export_to_video` or `encode_video` when audio is enabled.

## Checkpoints and `from_pretrained`

Cookbooks map logical model names to Hugging Face IDs:

| Cookbook name | `from_pretrained` ID |
| --- | --- |
| `Cosmos3-Nano` | `nvidia/Cosmos3-Nano` |
| `Cosmos3-Super` | `nvidia/Cosmos3-Super` |

The omnimodal Nano and Super checkpoints cover text-to-image, text-to-video, image-to-video, and sound-capable video generation in one pipeline. Specialized Super variants (`Cosmos3-Super-Text2Image`, `Cosmos3-Super-Image2Video`) exist for task-focused serving; the audiovisual Diffusers cookbook uses the full omni checkpoints above.

<ParamField body="model_id" type="string" required>
Hugging Face repo id, e.g. `nvidia/Cosmos3-Nano`.
</ParamField>

<ParamField body="torch_dtype" type="torch.dtype">
Use `torch.bfloat16` (BF16 is the tested Generator precision).
</ParamField>

<ParamField body="device_map" type="string">
Optional in minimal quickstarts: `device_map="cuda"` loads weights directly on GPU. The cookbook instead calls `pipe.to("cuda")` after load.
</ParamField>

<ParamField body="safety_checker" type="object | None">
Cookbook sets `safety_checker=None` with `enable_safety_checker=True` so guardrails still run through the pipeline’s safety path.
</ParamField>

<ParamField body="token" type="string | None">
Pass `HF_TOKEN` when not using `hf auth login`; gated Cosmos3 repos require Hugging Face authentication.
</ParamField>

<CodeGroup>
```python title="README quickstart"
pipe = Cosmos3OmniPipeline.from_pretrained(
    "nvidia/Cosmos3-Nano",
    torch_dtype=torch.bfloat16,
    device_map="cuda",
)
```

```python title="Cookbook pattern"
pipe = Cosmos3OmniPipeline.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    safety_checker=None,
    enable_safety_checker=True,
    token=os.environ.get("HF_TOKEN") or None,
)
pipe.to("cuda")
```
</CodeGroup>

<Note>
The first run downloads weights from Hugging Face. Diffusion at 720p with 35 steps is compute-heavy; long per-step times on the first generation are expected, not a hang.
</Note>

## Generation modes

README documents four Diffusers modes. The audiovisual notebook implements them with internal `model_mode` values and `enable_sound`:

| Diffusers mode | Notebook `model_mode` | Conditioning | `num_frames` | Sound |
| --- | --- | --- | ---: | --- |
| Text-to-image | `text2image` | Prompt only | `1` | Off |
| Text-to-video | `text2video` | Prompt + optional negative JSON | `189` (default) | `enable_sound=False` |
| Image-to-video | `image2video` | Prompt + `image=` + negative JSON | `189` | `enable_sound=False` |
| Text-to-video-with-sound | `text2video` | Same as T2V | `189` | `enable_sound=True` |

Image-to-video-with-sound uses `image2video` with `enable_sound=True`. Super cookbook sections run T2V and I2V without audio; Nano runs the full matrix including sound.

```text
                    ┌─────────────────┐
  JSON prompt ─────►│ Cosmos3Omni     │
  negative (video)  │ Pipeline.__call__│──► result.video (frames)
  image (i2v)       └────────┬────────┘    result.sound (optional)
                             │
              enable_sound=True & sound present
                             ▼
                    encode_video (AAC in MP4)
              else
                             ▼
                    export_to_video (silent MP4)
              text2image (num_frames=1)
                             ▼
                    result.video[0].save (.png)
```

## Install and `--torch-backend` pairing

Create a Python 3.13 venv and install Diffusers from the upstream git ref plus media dependencies. Pin `--torch-backend` to the CUDA major version your NVIDIA driver supports.

| Driver CUDA | `--torch-backend` | Notes |
| --- | --- | --- |
| 13.x | `cu130` | Default in cookbooks (`COSMOS3_TORCH_BACKEND=cu130`). |
| 12.x | `cu128` | Use on CUDA 12.8 drivers. |
| Detect at install | `auto` | README quickstart option; uv picks a matching `torch` wheel. Without any pin, uv may install `cu130` and `torch.cuda.is_available()` returns `False` on older drivers. |

<Tabs>
<Tab title="Cookbook (explicit backend)">

```bash
uv venv --python 3.13 --seed --managed-python
source .venv/bin/activate

uv pip install --torch-backend=cu130 \
  "diffusers @ git+https://github.com/huggingface/diffusers.git" \
  accelerate av cosmos_guardrail huggingface_hub \
  imageio imageio-ffmpeg torch torchvision transformers
```

Set `export COSMOS3_TORCH_BACKEND=cu128` before the same command on CUDA 12.x systems.

</Tab>
<Tab title="README quickstart (auto backend)">

```bash
uv venv --python 3.13 --seed --managed-python
source .venv/bin/activate
uv pip install --torch-backend=auto \
  "diffusers @ git+https://github.com/huggingface/diffusers.git" \
  accelerate av cosmos_guardrail huggingface_hub \
  imageio imageio-ffmpeg torch torchvision transformers
```

</Tab>
</Tabs>

<Warning>
Requires `uv >= 0.11.3` for `cu130` backend recognition. Headless Linux may need `libxcb1`, `libgl1`, and `libglib2.0-0` before importing the pipeline.
</Warning>

Verify GPU visibility after install:

```bash
python - <<'PY'
import torch
print("torch:", torch.__version__, "cuda:", torch.version.cuda)
print("cuda available:", torch.cuda.is_available())
PY
```

## Scheduler

Replace the pipeline scheduler before generation. Cookbooks use `UniPCMultistepScheduler` with `flow_shift` (default **10.0**, aligned with vLLM-Omni `flow_shift`):

```python
from diffusers.schedulers.scheduling_unipc_multistep import UniPCMultistepScheduler

pipe.scheduler = UniPCMultistepScheduler.from_config(
    pipe.scheduler.config, flow_shift=10.0
)
```

Per-run payloads can override `shift`; the runner re-applies `flow_shift=payload["shift"]` before each call.

## `__call__` parameters

Shared cookbook defaults (`FIXED_SAMPLING`):

| Parameter | Default | Role |
| --- | ---: | --- |
| `num_inference_steps` | `35` | Diffusion steps |
| `guidance_scale` | `6.0` | Classifier-free guidance |
| `fps` | `24` | Output frame rate (video modes) |
| `num_frames` | `189` | ~7.9 s at 24 FPS for T2V/I2V |
| `seed` | `1234` | Via `torch.Generator(device="cuda").manual_seed(...)` |

<ParamField body="prompt" type="string" required>
Scene description. Cookbooks pass **compact JSON** (`json.dumps(..., separators=(",", ":"))`) from files under `assets/prompts/{text2image,text2video,image2video}/`. Plain text strings also work in the README minimal example.
</ParamField>

<ParamField body="negative_prompt" type="string">
Structured JSON for video modes, loaded from `assets/negative_prompts/{mode}/neg_prompt.json`. Text-to-image uses `negative_prompt=""`.
</ParamField>

<ParamField body="image" type="PIL.Image | None">
Required for image-to-video. Load with `load_image(path)`; cookbook sets `image=None` for text-only modes.
</ParamField>

<ParamField body="height" type="int">
With `width`, derived from payload `resolution` + `aspect_ratio`. Cookbook supports `720` + `16,9` → **720×1280** and `256` + `16,9` → **192×320**.
</ParamField>

<ParamField body="width" type="int">
See `height`.
</ParamField>

<ParamField body="num_frames" type="int">
`1` for text-to-image; `189` for standard video examples.
</ParamField>

<ParamField body="fps" type="float">
Output FPS for video modes; passed to export helpers.
</ParamField>

<ParamField body="num_inference_steps" type="int">
Diffusion step count.
</ParamField>

<ParamField body="guidance_scale" type="float">
CFG strength.
</ParamField>

<ParamField body="enable_sound" type="bool">
`True` for text-to-video-with-sound (and image-to-video-with-sound). Requires a checkpoint with sound modules; when `result.sound` is present, mux audio with `encode_video`.
</ParamField>

<ParamField body="add_resolution_template" type="bool">
Cookbook sets `False` when height/width are explicit.
</ParamField>

<ParamField body="add_duration_template" type="bool">
Cookbook sets `False` when `num_frames` and `fps` are explicit.
</ParamField>

<ParamField body="generator" type="torch.Generator">
CUDA generator for reproducible seeds.
</ParamField>

### Text-to-image

```python
result = pipe(
    prompt=payload["prompt"],
    negative_prompt="",
    num_frames=1,
    height=720,
    width=1280,
    num_inference_steps=35,
    guidance_scale=6.0,
    add_resolution_template=False,
    add_duration_template=False,
    generator=generator,
)
result.video[0].save("output.png")
```

README notes single-frame generation returns a PIL-accessible frame via `result.video[0]`; the cookbook saves PNG from that tensor.

### Text-to-video and image-to-video

```python
from diffusers.utils import load_image

image = load_image("assets/images/image2video/car_driving.jpg")  # i2v only

result = pipe(
    prompt=json.dumps(prompt_dict, separators=(",", ":")),
    negative_prompt=json.dumps(neg_dict, separators=(",", ":")),
    image=image,  # None for t2v
    num_frames=189,
    height=720,
    width=1280,
    fps=24,
    num_inference_steps=35,
    guidance_scale=6.0,
    enable_sound=False,
    add_resolution_template=False,
    add_duration_template=False,
    generator=generator,
)
```

### Text-to-video-with-sound

Set `enable_sound=True`. After `__call__`, if `result.sound is not None`, mux with `encode_video` using the pipeline sound tokenizer sample rate; otherwise fall back to silent `export_to_video`.

## Exporting outputs

| Output | Helper | When |
| --- | --- | --- |
| Silent MP4 | `export_to_video(result.video, path, fps=24, macro_block_size=1)` | Video modes without audio |
| MP4 + AAC | `encode_video(result.video, fps=..., output_path=..., audio=result.sound, audio_sample_rate=pipe.sound_tokenizer.config.sampling_rate)` | `enable_sound=True` and sound returned |
| PNG | `result.video[0].save(path)` | `num_frames=1` |

Stereo AAC at 48 kHz is the documented sound output spec when generated with video. Match `fps` in export calls to the `fps` passed into `pipe()`.

<Check>
Success signal: MP4 or PNG written to disk; cookbook prints `generated in X.Xs` and `wrote {path}`. For sound runs, confirm `result.sound is not None` before calling `encode_video`.
</Check>

## Structured prompt assets

Prompts under `cookbooks/cosmos3/generator/audiovisual/assets/prompts/` are JSON scene specs (subjects, cinematography, `temporal_caption`, `resolution`, `fps`, etc.). Example text-to-video prompt: `assets/prompts/text2video/car_colliding.json` (720p, 24 FPS, 7s duration in metadata). Image-to-video pairs prompts like `assets/prompts/image2video/car_driving.json` with conditioning images such as `assets/images/image2video/car_driving.jpg`.

Pass prompts to the pipeline as **serialized JSON strings**, not raw dicts, to match cookbook and vLLM-Omni parity.

## Operational notes

- **Guardrails**: `cosmos_guardrail` is an install dependency; safety checking stays enabled in cookbook loads.
- **Memory**: Switching Nano ↔ Super in one process deletes the cached pipeline and calls `torch.cuda.empty_cache()` before reloading.
- **Upstream API docs**: Mode-specific examples and additional kwargs are documented in [Cosmos 3 Diffusers API](https://huggingface.co/docs/diffusers/main/en/api/pipelines/cosmos3).

## Related pages

<CardGroup>
<Card title="Run Generator with Diffusers" href="/run-generator-diffusers">
Step-by-step install, scheduler setup, and first MP4 from the audiovisual cookbook.
</Card>
<Card title="Cookbook environment setup" href="/cookbook-environment">
Shared `cu130`/`cu128` matrix, HF auth, and GPU verification for all backends.
</Card>
<Card title="Sampling and prompt parameters" href="/sampling-and-prompt-parameters">
Structured JSON prompt schema, negative prompts, and upsampling defaults.
</Card>
<Card title="Input and output specifications" href="/input-output-specifications">
Resolution tiers, frame counts, aspect ratios, and sound output specs.
</Card>
<Card title="Audiovisual cookbook recipes" href="/audiovisual-cookbooks">
Full `run_with_diffusers.ipynb` walkthrough for every Nano/Super asset set.
</Card>
<Card title="Troubleshooting" href="/troubleshooting">
CUDA/driver mismatches, `torch.cuda` false negatives, libxcb, and uv backend errors.
</Card>
</CardGroup>