# Troubleshooting

> CUDA/driver mismatches, NGC container selection, torch.cuda unavailable fixes, libxcb headless imports, uv version and --torch-backend errors, and VLLM_USE_DEEP_GEMM workaround.

- Repository: NVIDIA/cosmos
- GitHub: https://github.com/NVIDIA/cosmos
- Human docs: https://grok-wiki.com/public/docs/nvidia-cosmos-82de3e90abd9
- Complete Markdown: https://grok-wiki.com/public/docs/nvidia-cosmos-82de3e90abd9/llms-full.txt

## Source Files

- `README.md`
- `cookbooks/cosmos3/README.md`
- `cookbooks/cosmos3/generator/audiovisual/run_with_diffusers.ipynb`
- `cookbooks/cosmos3/reasoner/run_with_vllm.ipynb`

---

---
title: "Troubleshooting"
description: "CUDA/driver mismatches, NGC container selection, torch.cuda unavailable fixes, libxcb headless imports, uv version and --torch-backend errors, and VLLM_USE_DEEP_GEMM workaround."
---

Cosmos 3 setup failures usually trace to a CUDA driver and PyTorch wheel mismatch, an outdated `uv` that cannot parse framework config or accept `cu130`, missing X11 libraries on headless hosts, or vLLM Reasoner builds that require disabling DeepGEMM. The tables below map symptoms to backend-specific fixes for Diffusers, Cosmos Framework `uv sync`, vLLM, and the `vllm/vllm-omni:cosmos3` container.

## Symptom map

```text
Symptom                          Likely cause                    Fix surface
─────────────────────────────────────────────────────────────────────────────
"The NVIDIA driver is too old"   torch cu130 on CUDA 12 driver  --torch-backend / COSMOS3_*
cuda available: False            Same mismatch                    cu128 / cu128-train
--torch-backend invalid/cu129    uv < 0.11.3                      uv self update
libxcb.so.1 missing              Headless X11 libs                apt-get libxcb1 ...
DeepGEMM unavailable             vLLM build / GPU combo           VLLM_USE_DEEP_GEMM=0
vllm serve fails after install   cu130+vllm 0.21 on CUDA 12       cu128 + vllm 0.19.1
```

## CUDA driver and PyTorch alignment

Supported driver CUDA lines are **13.x** (recommended) and **12.x** (12.8 wheels). The driver CUDA reported by `nvidia-smi` and the CUDA baked into PyTorch must agree on the **major** version.

| Check | Command |
| --- | --- |
| Driver CUDA | `nvidia-smi` (top-right CUDA Version) |
| PyTorch CUDA | `python -c "import torch; print(torch.version.cuda)"` |
| GPU visible | `python -c "import torch; print(torch.cuda.is_available())"` |

| Driver CUDA | Diffusers / generic `uv pip` | Cosmos Framework `uv sync` | vLLM Reasoner |
| --- | --- | --- | --- |
| 13.x | `--torch-backend=cu130` or `COSMOS3_TORCH_BACKEND=cu130` | `--group=cu130-train` or `COSMOS3_UV_GROUP=cu130-train` | `--torch-backend=cu130 "vllm==0.21.0"` |
| 12.x | `--torch-backend=cu128` or `COSMOS3_TORCH_BACKEND=cu128` | `--group=cu128-train` or `COSMOS3_UV_GROUP=cu128-train` | `--torch-backend=cu128 "vllm==0.19.1"` |

<Warning>
vLLM does not publish wheels for every CUDA minor version. For Reasoner installs, **`--torch-backend=auto` is not reliable** — always pick the `cu130`/`vllm==0.21.0` or `cu128`/`vllm==0.19.1` pair that matches your driver.
</Warning>

<Tip>
For **Diffusers** Generator installs only, `uv pip install --torch-backend=auto` lets `uv` detect the NVIDIA driver and select a matching `torch`/`torchvision` build. Without it, `uv pip install torch` defaults to the newest CUDA wheel (`cu130`), which fails on pre-CUDA-13 drivers.
</Tip>

## `torch.cuda.is_available()` is False

### Typical error

```text
The NVIDIA driver on your system is too old
```

PyTorch was built for a newer CUDA than the host driver supports. `uv pip install torch` without `--torch-backend` pulls CUDA 13 (`cu130`) by default.

<Steps>
<Step title="Confirm the mismatch">

Run both checks and compare major CUDA versions:

```bash
nvidia-smi
python -c "import torch; print(torch.__version__, torch.version.cuda, torch.cuda.is_available())"
```

</Step>
<Step title="Reinstall torch with a matching backend">

<Tabs>
<Tab title="Diffusers venv">

```bash
source .venv/bin/activate
uv pip install --torch-backend=auto torch torchvision
# or pin explicitly:
# uv pip install --torch-backend=cu128 torch torchvision
```

Set `COSMOS3_TORCH_BACKEND=cu128` before re-running the Diffusers cookbook install cell if you use the notebook env vars.

</Tab>
<Tab title="Cosmos Framework">

```bash
export COSMOS3_UV_GROUP=cu128-train   # CUDA 12.x driver
cd packages/cosmos3
uv sync --all-extras --group="$COSMOS3_UV_GROUP"
```

Default notebooks use `cu130-train`; change the group **before** the configuration cell on CUDA 12.x systems.

</Tab>
<Tab title="vLLM Reasoner">

```bash
source .venv/bin/activate
# CUDA 12.x driver:
uv pip install --torch-backend=cu128 "vllm==0.19.1" \
  "vllm-cosmos3 @ git+https://github.com/NVIDIA/cosmos-framework.git#subdirectory=packages/vllm-cosmos3"
```

</Tab>
</Tabs>

</Step>
<Step title="Verify GPU access">

```bash
.venv/bin/python - <<'PY'
import torch
print("torch:", torch.__version__)
print("torch cuda:", torch.version.cuda)
print("cuda available:", torch.cuda.is_available())
print("device count:", torch.cuda.device_count())
if torch.cuda.is_available():
    print("device 0:", torch.cuda.get_device_name(0))
PY
```

Expect `cuda available: True` and at least one device.

</Step>
</Steps>

## NGC base container selection

When you run inside NVIDIA NGC PyTorch images instead of a local `uv` venv, match the container tag to your target CUDA line:

| Target CUDA | NGC image |
| --- | --- |
| CUDA 13 | `nvcr.io/nvidia/pytorch:25.09-py3` |
| CUDA 12 | `nvcr.io/nvidia/pytorch:25.06-py3` |

Pair the container with the same `cu130` / `cu128` backend choices as bare-metal installs. Generator production serving via vLLM-Omni uses the separate prebuilt image `vllm/vllm-omni:cosmos3` (not the NGC PyTorch tags).

## vLLM-Omni Docker

| Issue | Remediation |
| --- | --- |
| Server not ready | Wait for `Application startup complete.` in container logs |
| Local media not found | Mount host paths and pass `--allowed-local-media-path` covering those files |
| Super OOM | Add `--tensor-parallel-size 4 --enable-layerwise-offload` for `nvidia/Cosmos3-Super` |

Minimal Nano serve pattern:

```bash
docker pull vllm/vllm-omni:cosmos3

docker run --runtime nvidia --gpus all \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -v "$(pwd):/workspace" \
  -p 8000:8000 \
  --ipc=host \
  vllm/vllm-omni:cosmos3 \
  vllm serve nvidia/Cosmos3-Nano \
  --omni \
  --model-class-name Cosmos3OmniDiffusersPipeline \
  --allowed-local-media-path / \
  --port 8000
```

Verify the API: `curl http://localhost:8000/v1/models`.

## Headless import: `libxcb.so.1`

On headless servers and minimal containers, importing Diffusers pipelines or Framework modules can fail because a dependency links against X11/graphics libraries that are not installed:

```text
libxcb.so.1: cannot open shared object file
```

Install the system packages:

```bash
apt-get install -y libxcb1 libgl1 libglib2.0-0
```

Cookbooks for Diffusers and Cosmos Framework audiovisual generation document this under prerequisites.

## `uv` version and `--torch-backend` errors

Cosmos Framework requires **`uv >= 0.11.3`** (enforced in its `pyproject.toml`). Older `uv` builds fail in two common ways:

| Error pattern | Cause |
| --- | --- |
| Parse failure on `[tool.uv.audit]` | `uv` too old for framework `pyproject.toml` |
| `a value is required for '--torch-backend'` | Flag present but value rejected |
| Accepted backends stop at `cu129` | `uv` predates `cu130` support |

Upgrade:

```bash
uv self update
# or reinstall from https://astral.sh/uv
```

Confirm version: `uv --version` (must be ≥ 0.11.3).

### Environment variables for notebooks

| Variable | Default | Use on CUDA 12.x |
| --- | --- | --- |
| `COSMOS3_TORCH_BACKEND` | `cu130` | `cu128` (Diffusers) |
| `COSMOS3_UV_GROUP` | `cu130-train` | `cu128-train` (Framework) |

Export these **before** the notebook configuration or install cells.

### Framework install: git-LFS mirror failures

If `uv sync` fails on optional git-LFS test artifacts in the Framework mirror:

```bash
export GIT_LFS_SKIP_SMUDGE=1
uv sync --all-extras --group=cu130-train
```

## `VLLM_USE_DEEP_GEMM` workaround

vLLM Reasoner serving may report that **DeepGEMM is unavailable** on your GPU or build. Disable DeepGEMM before `vllm serve`:

```bash
export VLLM_USE_DEEP_GEMM=0

vllm serve nvidia/Cosmos3-Nano \
  --hf-overrides '{"architectures": ["Cosmos3ReasonerForConditionalGeneration"]}' \
  --async-scheduling \
  --allowed-local-media-path / \
  --port 8000
```

Apply the same export in notebook shells that launch the background vLLM server.

<Note>
When launching with `.venv/bin/vllm` without activating the venv, ensure `.venv/bin` is on `PATH`. FlashInfer’s JIT kernel build invokes `ninja`, which lives in the venv.
</Note>

## Backend quick reference

| Backend | CUDA pairing | `auto` backend? |
| --- | --- | --- |
| Diffusers Generator | `COSMOS3_TORCH_BACKEND` / `--torch-backend` | Yes (`auto` recommended in README quickstart) |
| Cosmos Framework | `COSMOS3_UV_GROUP` (`cu130-train` / `cu128-train`) | No — pick group explicitly |
| vLLM Reasoner | `cu130`+`vllm==0.21.0` or `cu128`+`vllm==0.19.1` | No |
| vLLM-Omni Generator | Prebuilt `vllm/vllm-omni:cosmos3` image | N/A (container CUDA is fixed) |

## Related pages

<CardGroup>
<Card title="Installation" href="/installation">
Prerequisites, CUDA driver pairing, venv and Docker setup, and first verification commands.
</Card>
<Card title="Cookbook environment setup" href="/cookbook-environment">
Shared uv/Docker paths for every backend, HF auth, and GPU verification probes.
</Card>
<Card title="Reasoner vLLM configuration" href="/reasoner-vllm-configuration">
`vllm serve` flags, tensor parallelism, `VLLM_USE_DEEP_GEMM`, and cu130/cu128 version pairs.
</Card>
<Card title="Diffusers pipeline reference" href="/diffusers-pipeline-reference">
`Cosmos3OmniPipeline` modes and `--torch-backend` install pairing.
</Card>
</CardGroup>