# Inference benchmarks

> Published latency tables for Cosmos3-Nano/Super Generator (PyTorch, vLLM-Omni, Diffusers by GPU/resolution/TP) and Reasoner vLLM serving metrics (TTFT, throughput at concurrency tiers).

- Repository: NVIDIA/cosmos
- GitHub: https://github.com/NVIDIA/cosmos
- Human docs: https://grok-wiki.com/public/docs/nvidia-cosmos-82de3e90abd9
- Complete Markdown: https://grok-wiki.com/public/docs/nvidia-cosmos-82de3e90abd9/llms-full.txt

## Source Files

- `inference_benchmarks.md`
- `README.md`
- `cookbooks/cosmos3/generator/audiovisual/run_with_diffusers.ipynb`
- `cookbooks/cosmos3/generator/audiovisual/run_with_vllm_omni.ipynb`

---

---
title: "Inference benchmarks"
description: "Published latency tables for Cosmos3-Nano/Super Generator (PyTorch, vLLM-Omni, Diffusers by GPU/resolution/TP) and Reasoner vLLM serving metrics (TTFT, throughput at concurrency tiers)."
---

Published latency and serving numbers for Cosmos3 live in the repository root file `inference_benchmarks.md`. Generator sections report diffusion-path latency in **seconds** for text-to-image (t2i), text-to-video (t2v), and image-to-video (i2v) across **PyTorch**, **vLLM-Omni**, and **Diffusers**. The Reasoner section reports **vLLM** autoregressive serving metrics—time to first token (TTFT), end-to-end request latency, and throughput—at client concurrency tiers 1, 64, 128, and 256. Results are filled in incrementally; **empty cells mean that GPU, engine, resolution, or tensor-parallel combination has not been measured yet**, not that it is unsupported.

## Benchmark inventory

| Section | Surface | Checkpoint | Metrics |
| --- | --- | --- | --- |
| Cosmos3-Nano generator | Generator (diffusion) | `nvidia/Cosmos3-Nano` (16B) | Latency (s) by GPU, engine, resolution tier, TP width |
| Cosmos3-Super generator | Generator (diffusion) | `nvidia/Cosmos3-Super` (64B) | Same modalities and engines at larger scale |
| Cosmos3-Nano reasoner | Reasoner (VLM) | `nvidia/Cosmos3-Nano` | TTFT, request latency (ms), req/s and tok/s at concurrency 1/64/128/256 |

<Info>
Canonical tables with all measured cells are maintained in [`inference_benchmarks.md`](https://github.com/nvidia/cosmos/blob/main/inference_benchmarks.md) on the main branch. This page documents methodology, column semantics, and the published numbers as of the current checkout.
</Info>

## How to read generator tables

### Column semantics

Generator table headers use **`{resolution}p/{gpu_count}`**:

| Column pattern | Meaning |
| --- | --- |
| `256p/1`, `480p/1`, `720p/1` | Single-GPU run at 320×192, 832×480, or 1280×720 |
| `256p/4`, `720p/8`, etc. | Tensor parallelism across 4 or 8 GPUs |

### Engines compared

| Engine | What the number measures | Resolution coverage in tables |
| --- | --- | --- |
| **PyTorch** | Average generation (sampling) time from OSS reference inference; CUDA Graphs enabled where supported | All resolution/TP columns where measured |
| **vLLM-Omni** | Total pipeline time | Primarily **720p** on supported GPUs |
| **Diffusers** | End-to-end time via Hugging Face `Cosmos3OmniPipeline` without custom CUDA graphs | **256p/1**, **480p/1**, **720p/1** (single-GPU only) |

### Shared workload settings

All generator runs use **BF16**, **batch size 1**, and **identical prompts, seeds, and sampler settings** across engines where noted. Video workloads follow the standard Cosmos3 profile: **189 frames at 24 FPS** (about 7.9 seconds), matching cookbook defaults such as `num_frames=189` in audiovisual Generator notebooks.

<Warning>
vLLM-Omni figures are tied to the upcoming public vLLM-Omni release and may change before GA. Values marked with **(*)** on H100 NVL are pre-release measurements.
</Warning>

## Cosmos3-Nano generator

### Text-to-video (t2v)

Latency in **seconds**.

| GPU | Engine | 256p/1 | 256p/4 | 256p/8 | 480p/1 | 480p/4 | 480p/8 | 720p/1 | 720p/4 | 720p/8 |
| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| **RTX PRO 6000 Blackwell** | PyTorch | | | | | | | 786.37 | 225.45 | |
| | vLLM-Omni | | | | | | | 369.67 | 114.30 | 68.66 |
| | Diffusers | 11.20 | | | 112.00 | | | 392.00 | | |
| **H20** | PyTorch | | | | | | | 931.39 | 268.88 | 157.71 |
| | vLLM-Omni | | | | | | | | | |
| | Diffusers | 30.20 | | | 258.00 | | | 926.00 | | |
| **H100 NVL** | PyTorch | | | 3.95 | 84.12 | | | 297.27 | 94.15 | 61.63 |
| | vLLM-Omni | | | | | | | 311.13 | 88.25(*) | 54.01(*) |
| | Diffusers | 11.00 | | | 90.00 | | | 324.20 | | |
| **H200 NVL** | PyTorch | | | | | | | 244.39 | 77.35 | 45.70 |
| | vLLM-Omni | | | | | | | | | |
| | Diffusers | 9.00 | | | 74.00 | | | 276.20 | | |
| **H100 80GB HBM3** | PyTorch | 7.61 | | | 59.83 | | | 207.78 | | |
| | vLLM-Omni | | | | | | | | | |
| | Diffusers | 9.00 | | | 68.00 | | | 240.00 | | |
| **H200 141GB HBM3** | PyTorch | | 3.34 | 3.19 | | | 13.97 | 214.28 | 67.48 | 41.26 |
| | vLLM-Omni | | | | | | | | | |
| | Diffusers | 9.00 | | | 67.00 | | | 239.60 | | |
| **B200** | PyTorch | 4.56 | 2.78 | 2.79 | | | | 114.85 | 39.75 | 26.27 |
| | vLLM-Omni | | | | | | | 107.84 | 35.29 | 22.87 |
| | Diffusers | 7.00 | | | 36.80 | | | 117.00 | | |
| **B300** | PyTorch | | | | | | | | | |
| | vLLM-Omni | | | | | | | | | |
| | Diffusers | 39.40 | | | 63.40 | | | 139.40 | | |

On **B200** at **720p/8**, vLLM-Omni (22.87 s) is roughly **5× faster** than Diffusers single-GPU (117.00 s) for the same modality. **720p/4** tensor parallelism cuts Nano t2v latency sharply versus **720p/1** on the same GPU (for example, B200 PyTorch: 114.85 s → 39.75 s).

### Image-to-video (i2v)

Latency in **seconds**.

| GPU | Engine | 256p/1 | 256p/4 | 256p/8 | 480p/1 | 480p/4 | 480p/8 | 720p/1 | 720p/4 | 720p/8 |
| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| **RTX PRO 6000 Blackwell** | PyTorch | | | | | | | 788.80 | 226.25 | 127.79 |
| | vLLM-Omni | | | | | | | 375.01 | 119.27 | 73.57 |
| | Diffusers | 12.00 | | | 112.00 | | | 397.00 | | |
| **H20** | PyTorch | | | | | | | | | 158.10 |
| | vLLM-Omni | | | | | | | | | |
| | Diffusers | 31.00 | | | 258.00 | | | 925.00 | | |
| **H100 NVL** | PyTorch | | | 3.99 | 84.50 | 28.69 | | 298.57 | 95.76 | 60.58 |
| | vLLM-Omni | | | | | | | 286.33 | 92.23(*) | 58.02(*) |
| | Diffusers | 11.00 | | | 91.00 | | | 325.20 | | |
| **H200 NVL** | PyTorch | | | | | | | 246.62 | 77.69 | 45.99 |
| | vLLM-Omni | | | | | | | | | |
| | Diffusers | 9.00 | | | 74.00 | | | 275.20 | | |
| **H100 80GB HBM3** | PyTorch | 7.64 | | | | | | 207.87 | | |
| | vLLM-Omni | | | | | | | | | |
| | Diffusers | 9.00 | | | 68.00 | | | 239.80 | | |
| **H200 141GB HBM3** | PyTorch | | 3.37 | 3.17 | | | 14.07 | 214.80 | 67.14 | 41.00 |
| | vLLM-Omni | | | | | | | | | |
| | Diffusers | 9.00 | | | 67.20 | | | 240.00 | | |
| **B200** | PyTorch | 4.60 | 2.77 | 2.81 | | | 9.66 | 113.90 | 40.01 | 26.58 |
| | vLLM-Omni | | | | | | | 110.19 | 37.76 | 25.68 |
| | Diffusers | | | | | | | 116.00 | | |
| **B300** | PyTorch | | | | | | | | | |
| | vLLM-Omni | | | | | | | | | |
| | Diffusers | 28.60 | | | 65.60 | | | 139.60 | | |

### Text-to-image (t2i)

Latency in **seconds**. t2i workloads are shortest; they are useful for sanity-checking a GPU stack before running full 189-frame video jobs.

| GPU | Engine | 256p/1 | 256p/4 | 256p/8 | 480p/1 | 480p/4 | 480p/8 | 720p/1 | 720p/4 | 720p/8 |
| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| **RTX PRO 6000 Blackwell** | PyTorch | | | | | | | 7.12 | 3.18 | 2.70 |
| | vLLM-Omni | | | | | | | 4.99 | 2.32 | 1.96 |
| | Diffusers | 2.00 | | | 4.00 | | | 5.00 | | |
| **H20** | PyTorch | | | | | | | | | |
| | vLLM-Omni | | | | | | | | | |
| | Diffusers | 3.00 | | | 6.00 | | | 10.00 | | |
| **H100 NVL** | PyTorch | | 2.45 | | | | | 4.21 | 2.57 | 2.64 |
| | vLLM-Omni | | | | | | | 3.44 | 1.83 | 1.90 |
| | Diffusers | 3.00 | | | 3.00 | | | 4.00 | | |
| **H200 NVL** | PyTorch | | | | | | | 3.58 | 2.62 | 2.64 |
| | vLLM-Omni | | | | | | | | | |
| | Diffusers | 3.00 | | | 3.00 | | | 4.00 | | |
| **H100 80GB HBM3** | PyTorch | 3.01 | | | | | | 3.45 | | |
| | vLLM-Omni | | | | | | | | | |
| | Diffusers | 3.00 | | | 3.00 | | | 4.00 | | |
| **H200 141GB HBM3** | PyTorch | | 2.59 | 2.70 | | 2.78 | | 3.28 | 2.84 | 2.77 |
| | vLLM-Omni | | | | | | | | | |
| | Diffusers | 3.00 | | | 3.00 | | | 4.00 | | |
| **B200** | PyTorch | | | 2.59 | 2.75 | | 2.56 | 2.87 | 2.58 | 2.62 |
| | vLLM-Omni | | | | | | | 1.77 | 2.20 | 3.41 |
| | Diffusers | | | | | | | 3.00 | | |
| **B300** | PyTorch | | | | | | | | | |
| | vLLM-Omni | | | | | | | | | |
| | Diffusers | 36.20 | | | | | | 41.00 | | |

<Tip>
At **256p**, multi-GPU tensor parallelism on **B300** can underperform single-GPU due to small-workload TP overhead; single-GPU is recommended at that resolution tier.
</Tip>

## Cosmos3-Super generator

Super benchmarks use the same modalities, engines, and column layout as Nano. Expect **longer runtimes** than Nano at the same resolution because of the **64B** checkpoint. Early Super coverage is narrower: **vLLM-Omni** and **Diffusers** runs exist primarily on **B200** and select **H200** configurations.

### Text-to-video (t2v)

| GPU | Engine | 256p/1 | 256p/4 | 256p/8 | 480p/1 | 480p/4 | 480p/8 | 720p/1 | 720p/4 | 720p/8 |
| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| **RTX PRO 6000 Blackwell** | PyTorch | | | | | | | | | 427.16 |
| | vLLM-Omni | | | | | | | | | |
| | Diffusers | | | | | | | | | |
| **H20** | PyTorch | | | | | | | | | 492.41 |
| | vLLM-Omni | | | | | | | | | |
| | Diffusers | | | | | | | | | |
| **H100 NVL** | PyTorch | | | | | 101.27 | | | 330.04 | 186.19 |
| | vLLM-Omni | | | | | | | | | |
| | Diffusers | | | | | | | | | |
| **H200 NVL** | PyTorch | | | | | | | | 258.34 | 139.37 |
| | vLLM-Omni | | | | | | | | | |
| | Diffusers | 33.00 | | | 286.80 | | | 1036.00 | | |
| **H100 80GB HBM3** | PyTorch | | | | | | | | | |
| | vLLM-Omni | | | | | | | | | |
| | Diffusers | | | | | | | | | |
| **H200 141GB HBM3** | PyTorch | | | | | 70.27 | 41.78 | | 224.43 | 123.49 |
| | vLLM-Omni | | | | | | | | | |
| | Diffusers | 31.00 | | | 251.60 | | | 886.20 | | |
| **B200** | PyTorch | | 5.59 | | 114.38 | 35.73 | 21.39 | 407.50 | 118.38 | 65.93 |
| | vLLM-Omni | | | | | | | 390.28 | 113.31 | 62.11 |
| | Diffusers | | | | 127.20 | | | 414.40 | | |
| **B300** | PyTorch | | | | | | | | | |
| | vLLM-Omni | | | | | | | | | |
| | Diffusers | 54.20 | | | 155.40 | | | 424.80 | | |

### Image-to-video (i2v) and text-to-image (t2i) highlights

| Modality | GPU | Engine | Notable measurement |
| --- | --- | --- | --- |
| i2v 720p/8 | B200 | PyTorch | 65.91 s |
| i2v 720p/8 | B200 | vLLM-Omni | 64.82 s |
| i2v 720p/1 | H200 NVL | Diffusers | 1034.60 s |
| t2i 720p/1 | B200 | vLLM-Omni | 6.02 s |
| t2i 720p/4 | B200 | PyTorch | 4.28 s |

Full Super **i2v** and **t2i** grids are in `inference_benchmarks.md` under [Cosmos3-Super Generator](https://github.com/nvidia/cosmos/blob/main/inference_benchmarks.md#cosmos3-super-generator).

## Cosmos3-Nano reasoner (vLLM)

Reasoner benchmarks measure **autoregressive text generation** from vision and text inputs—not diffusion sampling. All runs use checkpoint **`nvidia/Cosmos3-Nano`** served through **vLLM**, with metrics collected by the **AIPerf** client.

### Workload matrix

Each GPU section in `inference_benchmarks.md` contains **four** workload tables:

| Workload label | Input tokens | Output tokens | Video FPS | Typical use |
| --- | ---: | ---: | ---: | --- |
| Input 50 / Output 1 / Video 1 FPS | 50 | 1 | 1 | Minimal single-token response |
| Input 50 / Output 1 / Video 2 FPS | 50 | 1 | 2 | Higher video sampling rate |
| Input 50 / Output 100 / Video 1 FPS | 50 | 100 | 1 | Short caption / VQA-style answer |
| Input 50 / Output 100 / Video 2 FPS | 50 | 100 | 2 | Longer answer with denser video input |

### Metrics

| Metric | Unit | Interpretation |
| --- | --- | --- |
| **Time To First Token (TTFT)** | ms | Latency until the first output token is emitted; lower is better |
| **Request Latency** | ms | End-to-end time per request; for Output 1 workloads, equals TTFT |
| **Request Throughput** | Req/s | Completed requests per second; higher is better |
| **Output Token Throughput** | Tok/s | Generated tokens per second; for Output 1, matches request throughput |
| **Request Count** | requests | Total requests in the benchmark window |

<ParamField body="concurrency" type="integer">
Client-side simultaneous requests issued by AIPerf (1, 64, 128, or 256). This is **not** vLLM tensor-parallel GPU count.
</ParamField>

### Cross-GPU comparison (Input 50 / Output 100 / Video 1 FPS)

Representative serving numbers at **concurrency 64**—a mid-tier load for captioning-style **100-token** outputs:

| GPU | TTFT (ms) | Request latency (ms) | Req/s | Tok/s |
| --- | ---: | ---: | ---: | ---: |
| RTX PRO 6000 Blackwell | 2280.43 | 9309.93 | 6.85 | 684.76 |
| H20 | 6607.80 | 18990.48 | 3.37 | 336.55 |
| H100 NVL | 2890.81 | 9192.19 | 6.94 | 694.37 |
| H200 NVL | 1948.06 | 5284.58 | 12.07 | 1206.86 |
| H100 80GB HBM3 (SXM) | 1720.73 | 5251.56 | 12.14 | 1213.61 |
| H200 141GB HBM3 | 2060.05 | 4965.09 | 12.84 | 1284.38 |
| B200 | 1106.35 | 2736.53 | 23.28 | 2328.01 |
| B300 | 1070.93 | 2657.06 | 23.96 | 2396.35 |

At **concurrency 1** on the same workload, **B300** reaches **203.29 Tok/s** with **490.11 ms** request latency versus **71.22 Tok/s** on RTX PRO 6000 Blackwell.

### Concurrency scaling pattern

TTFT grows roughly linearly with client concurrency because queued prefills compete for the same server. For **Output 1** workloads, request latency equals TTFT at every concurrency tier. For **Output 100** workloads, request latency exceeds TTFT because generation continues after the first token.

Example — **B200**, Input 50 / Output 100 / Video 1 FPS:

| Metric | C=1 | C=64 | C=128 | C=256 |
| --- | ---: | ---: | ---: | ---: |
| TTFT (ms) | 115.27 | 1106.35 | 2111.97 | 2549.79 |
| Request latency (ms) | 553.01 | 2736.53 | 5001.20 | 9279.25 |
| Output tok/s | 180.16 | 2328.01 | 2523.07 | 2701.08 |

Throughput often peaks between **concurrency 128 and 256** before latency dominates SLA budgets.

### Per-GPU tables

<AccordionGroup>
<Accordion title="RTX PRO 6000 Blackwell, H20, H100 NVL, H200 NVL">
Full four-workload tables for these GPUs are in `inference_benchmarks.md` under [Cosmos3-Nano Reasoner](https://github.com/nvidia/cosmos/blob/main/inference_benchmarks.md#cosmos3-nano-reasoner).
</Accordion>
<Accordion title="H100 80GB HBM3 (SXM), H200 141GB HBM3, B200, B300">
Full four-workload tables for datacenter Blackwell and Hopper SKUs are in the same Reasoner section of `inference_benchmarks.md`.
</Accordion>
</AccordionGroup>

## Choosing hardware and integration from benchmarks

```text
                    ┌─────────────────────────────────────┐
                    │     Goal: pick stack from tables    │
                    └─────────────────┬───────────────────┘
                                      │
          ┌───────────────────────────┼───────────────────────────┐
          ▼                           ▼                           ▼
   Lowest latency t2v          Fastest iteration          Production VLM QPS
   at 720p (datacenter)        (single GPU, t2i)          (Reasoner, high C)
          │                           │                           │
          ▼                           ▼                           ▼
   B200/B300 + vLLM-Omni         Diffusers 256p/1            B200/B300 vLLM
   720p/8 TP (Nano ~23–27s)     (Nano ~2–11s t2i)          (23+ tok/s @ C=64)
          │                           │
          └───────── PyTorch OSS if you need Framework parity ─────────┘
```

| Goal | Favor | Example from tables |
| --- | --- | --- |
| Fastest Nano **720p t2v** | vLLM-Omni + multi-GPU TP | B200 720p/8: **22.87 s** (vLLM-Omni) vs **114.85 s** (PyTorch 720p/1) |
| Simplest single-GPU video | Diffusers | B200 720p/1 t2v: **117.00 s** |
| Highest Reasoner **tok/s** | B200/B300 at concurrency 64–256 | B300 Output 100 / V1 FPS @ C=64: **2396 Tok/s** |
| Super quality at scale | B200 + TP | Super t2v B200 720p/8: **62.11 s** (vLLM-Omni) |

<Note>
Benchmarks characterize reference workloads under controlled settings. Your prompts, guardrails, resolution, frame count, and server flags will shift absolute numbers. Use tables for **relative** comparisons across GPUs, engines, and TP widths on the same modality.
</Note>

## Reproducing or extending measurements

<Steps>
<Step title="Match generator workload settings">
Use BF16, batch size 1, 189 frames at 24 FPS for video, and fixed seeds/prompts across PyTorch, vLLM-Omni, and Diffusers runs. Align resolution tiers with 256p (320×192), 480p (832×480), and 720p (1280×720).
</Step>
<Step title="Run the integration under test">
- **PyTorch / Framework:** OSS inference benchmarking path; Framework cookbooks also support `--benchmark` on `cosmos_framework.scripts.inference` for local timing.
- **vLLM-Omni:** Docker `vllm/vllm-omni:cosmos3` with `--tensor-parallel-size` for Super; see audiovisual vLLM-Omni notebook.
- **Diffusers:** `Cosmos3OmniPipeline` end-to-end without CUDA graphs.
</Step>
<Step title="Run Reasoner serving benchmarks">
Serve `nvidia/Cosmos3-Nano` with Reasoner architecture overrides (`Cosmos3ReasonerForConditionalGeneration`), then drive load with the AIPerf client at concurrency 1, 64, 128, and 256.
</Step>
<Step title="Publish new cells">
Add results to `inference_benchmarks.md` in the appropriate Generator or Reasoner table; leave cells empty until a combination is actually measured.
</Step>
</Steps>

## Related pages

<CardGroup>
<Card title="Choose an integration" href="/choose-integration">
Compare PyTorch, vLLM-Omni, Diffusers, and Framework paths that these benchmarks measure.
</Card>
<Card title="Input and output specifications" href="/input-output-specifications">
Resolution tiers, frame counts, and precision defaults that define generator workloads.
</Card>
<Card title="Run Generator with vLLM-Omni" href="/run-generator-vllm-omni">
Tensor parallelism and Docker serve flags behind vLLM-Omni benchmark configurations.
</Card>
<Card title="Run Generator with Diffusers" href="/run-generator-diffusers">
Cosmos3OmniPipeline setup for Diffusers end-to-end latency numbers.
</Card>
<Card title="Run Reasoner with vLLM" href="/run-reasoner-vllm">
Serve Cosmos3-Nano Reasoner—the stack used for TTFT and throughput tables.
</Card>
<Card title="Reasoner vLLM configuration" href="/page-reasoner-vllm-configuration">
Serve flags (TP, mm-encoder, media-io-kwargs) that affect Reasoner serving performance.
</Card>
</CardGroup>