# vLLM-Omni API reference

> OpenAI-compatible endpoints (/v1/images/generations, /v1/videos, /v1/videos/sync), request fields (prompt, size, num_frames, guidance_scale, extra_params), action_mode values, and curl --form-string constraints.

- Repository: NVIDIA/cosmos
- GitHub: https://github.com/NVIDIA/cosmos
- Human docs: https://grok-wiki.com/public/docs/nvidia-cosmos-82de3e90abd9
- Complete Markdown: https://grok-wiki.com/public/docs/nvidia-cosmos-82de3e90abd9/llms-full.txt

## Source Files

- `README.md`
- `cookbooks/cosmos3/generator/audiovisual/README.md`
- `cookbooks/cosmos3/generator/audiovisual/run_with_vllm_omni.ipynb`
- `cookbooks/cosmos3/generator/action/README.md`
- `cookbooks/cosmos3/generator/action/run_fd_with_vllm.ipynb`

---

---
title: "vLLM-Omni API reference"
description: "OpenAI-compatible endpoints (/v1/images/generations, /v1/videos, /v1/videos/sync), request fields (prompt, size, num_frames, guidance_scale, extra_params), action_mode values, and curl --form-string constraints."
---

Cosmos 3 Generator production serving exposes an OpenAI-compatible HTTP API on the vLLM-Omni server (`vllm serve … --omni --model-class-name Cosmos3OmniDiffusersPipeline`). Vision generation uses `POST /v1/images/generations` (JSON) and `POST /v1/videos/sync` (multipart, blocking MP4). Action and long-running jobs use asynchronous `POST /v1/videos` with `GET /v1/videos/{id}` polling and optional `GET /v1/videos/{id}/content` for the rendered video.

<Note>
The `vllm/vllm-omni:cosmos3` Docker image ships all modalities (text-to-image, text-to-video, image-to-video, video-to-video, video-with-sound, action). A PR-branch install of vLLM-Omni currently covers only text-to-image, text-to-video, and image-to-video until upstream merges complete.
</Note>

## Endpoint map

| Method | Path | Use when | Response |
| --- | --- | --- | --- |
| `POST` | `/v1/images/generations` | Text-to-image | JSON with base64 PNG in `data[0].b64_json` |
| `POST` | `/v1/videos/sync` | Text/image/video generation; forward dynamics (video-only output) | Raw MP4 bytes (`Accept: video/mp4`) |
| `POST` | `/v1/videos` | Policy, inverse dynamics, or chunked action jobs that return action data | JSON job handle with `id` |
| `GET` | `/v1/videos/{id}` | Poll async job | JSON with `status`, `progress`, and `action` when complete |
| `GET` | `/v1/videos/{id}/content` | Download completed video | Raw MP4 bytes |
| `GET` | `/v1/models` | Verify server readiness | OpenAI-style model list |

```text
Client                          vLLM-Omni (port 8000)
  |-- POST /v1/images/generations --> JSON { data[].b64_json }
  |-- POST /v1/videos/sync ---------> MP4 body (blocking)
  |-- POST /v1/videos -------------> { id }
  |       +-- GET /v1/videos/{id} --> { status, progress, action? }
  |       +-- GET .../content -----> MP4 body
```

## Prerequisites

Start the server with local media paths allowed when using `action_path` or mounted cookbook assets:

```bash
docker run --runtime nvidia --gpus all \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -v "$(pwd):/workspace" \
  -p 8000:8000 \
  --ipc=host \
  vllm/vllm-omni:cosmos3 \
  vllm serve nvidia/Cosmos3-Nano \
  --omni \
  --model-class-name Cosmos3OmniDiffusersPipeline \
  --allowed-local-media-path / \
  --port 8000
```

The API is ready when logs show `Application startup complete.` Confirm with `curl http://localhost:8000/v1/models`.

For Super (64B), add `--tensor-parallel-size` and optionally `--enable-layerwise-offload`. CFG and Ulysses parallelism use `--cfg-parallel-size` and `--ulysses-degree`; set request `guidance_scale` for CFG strength (do not use `true_cfg_scale`).

## `POST /v1/images/generations`

:::endpoint POST /v1/images/generations Text-to-image; returns base64-encoded PNG in JSON.

This endpoint accepts **`Content-Type: application/json`** (not multipart). Cosmos-specific image options go in `extra_args`, not `extra_params`.

<ParamField body="prompt" type="string" required>
Positive text prompt. Cookbooks often pass a JSON-encoded structured prompt string.
</ParamField>

<ParamField body="size" type="string" required>
Output resolution as `widthxheight` (for example `1280x720` for 720p 16:9).
</ParamField>

<ParamField body="n" type="integer">
Number of images; cookbooks use `1`.
</ParamField>

<ParamField body="num_inference_steps" type="integer">
Diffusion denoising steps.
</ParamField>

<ParamField body="guidance_scale" type="number">
Classifier-free guidance scale for Cosmos 3. Use this field with `--cfg-parallel-size`; do not send `true_cfg_scale`.
</ParamField>

<ParamField body="flow_shift" type="number">
Scheduler flow-shift value (Diffusers cookbooks commonly use `10.0`).
</ParamField>

<ParamField body="seed" type="integer">
Reproducibility seed.
</ParamField>

<ParamField body="extra_args" type="object">
Cosmos 3 image options such as `use_resolution_template` and `guardrails`.
</ParamField>

<ResponseField name="data" type="array">
Each element includes `b64_json` (PNG bytes, base64-encoded).
</ResponseField>

:::

<RequestExample>

```bash title="curl text-to-image"
curl -sS -X POST http://localhost:8000/v1/images/generations \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "A robotics laboratory with a manipulator arm.",
    "size": "1280x720",
    "n": 1,
    "num_inference_steps": 35,
    "guidance_scale": 6.0,
    "flow_shift": 10.0,
    "seed": 0,
    "extra_args": {
      "use_resolution_template": false,
      "guardrails": true
    }
  }'
```

</RequestExample>

## `POST /v1/videos/sync`

:::endpoint POST /v1/videos/sync Blocking video generation; returns MP4 bytes directly.

Send **`multipart/form-data`**. Set `Accept: video/mp4` to receive the encoded file in the response body.

| Mode | Required inputs | Notable fields |
| --- | --- | --- |
| Text-to-video | `prompt` | `size`, `num_frames`, `fps`, `guidance_scale` |
| Image-to-video | `prompt`, `input_reference` (image file) | Same sampling fields |
| Video-to-video | `prompt`, `input_reference` (video file) | `extra_params`: `condition_frame_indexes_vision`, `condition_video_keep` |
| Video with sound | `prompt` | `generate_sound=true`, `sound_duration` (seconds, often `num_frames / fps`) |
| Forward dynamics | `input_reference` (image), action in `extra_params` | `action_mode`: `forward_dynamics` |

<ParamField body="prompt" type="string" required>
Positive text prompt (plain string or JSON-encoded structured prompt).
</ParamField>

<ParamField body="negative_prompt" type="string">
Concepts or artifacts to avoid.
</ParamField>

<ParamField body="size" type="string">
Output resolution as `widthxheight` (for example `1280x720`, `832x480`, `320x192`).
</ParamField>

<ParamField body="num_frames" type="string | integer">
Video length in frames. Supported range is 5–300; default in model settings is 189. Action cookbooks often set `action_chunk_size + 1`.
</ParamField>

<ParamField body="fps" type="string | integer">
Frame rate: 10, 16, 24, or 30 (default 24). AV forward dynamics uses 10; DROID 15; UMI 20 in action examples.
</ParamField>

<ParamField body="num_inference_steps" type="string | integer">
Diffusion denoising steps (README example: 35; action examples: 30).
</ParamField>

<ParamField body="guidance_scale" type="string | number">
CFG scale. Audiovisual examples use `6.0`; action examples use `1.0`.
</ParamField>

<ParamField body="flow_shift" type="string | number">
Scheduler flow-shift (commonly `10.0`).
</ParamField>

<ParamField body="seed" type="string | integer">
Reproducibility seed.
</ParamField>

<ParamField body="max_sequence_length" type="integer">
Maximum prompt tokens kept for conditioning (Cosmos 3 default `512`). Longer prompts are truncated with a warning.
</ParamField>

<ParamField body="input_reference" type="file">
Uploaded image or video for image-to-video, video-to-video, and action conditioning. Use curl `-F input_reference=@/path/to/file`.
</ParamField>

<ParamField body="generate_sound" type="string">
Set to `true` to mux a stereo AAC soundtrack (48 kHz) into the output MP4.
</ParamField>

<ParamField body="sound_duration" type="string">
Soundtrack length in seconds; cookbooks set `num_frames / fps` to three decimal places.
</ParamField>

<ParamField body="extra_params" type="string (JSON)">
JSON-encoded Cosmos 3 options (see table below). Serialize compactly when using curl `--form-string`.
</ParamField>

:::

<RequestExample>

```bash title="curl text-to-video"
curl -sS -X POST http://localhost:8000/v1/videos/sync \
  -H "Accept: video/mp4" \
  --form-string "prompt=A small warehouse robot moves a blue box across a clean floor." \
  --form-string "negative_prompt=blurry, distorted, low quality" \
  --form-string "size=1280x720" \
  --form-string "num_frames=81" \
  --form-string "fps=24" \
  --form-string "num_inference_steps=35" \
  --form-string "guidance_scale=4.0" \
  --form-string "seed=42" \
  -o cosmos3_t2v_output.mp4
```

</RequestExample>

<RequestExample>

```python title="requests text-to-video with extra_params"
import json
import requests

response = requests.post(
    "http://localhost:8000/v1/videos/sync",
    data={
        "prompt": json.dumps({"scene": "robot kitchen"}),
        "negative_prompt": json.dumps({"avoid": "blur"}),
        "size": "1280x720",
        "num_frames": "189",
        "fps": "24",
        "num_inference_steps": "35",
        "guidance_scale": "6.0",
        "flow_shift": "10.0",
        "seed": "0",
        "extra_params": json.dumps({
            "use_resolution_template": False,
            "use_duration_template": False,
            "guardrails": True,
        }),
    },
    headers={"Accept": "video/mp4"},
)
response.raise_for_status()
open("/tmp/cosmos3_t2v.mp4", "wb").write(response.content)
```

</RequestExample>

## `POST /v1/videos` (async jobs)

:::endpoint POST /v1/videos Asynchronous video/action jobs with polling.

Use this endpoint when the response includes **predicted action** data (policy or inverse dynamics) or when generation time exceeds comfortable HTTP timeouts. The initial response is JSON with an `id`. Poll `GET /v1/videos/{id}` until `status` is `completed`, then fetch video bytes from `GET /v1/videos/{id}/content` if needed. Terminal failure states are `failed` and `cancelled`.

| Mode | `action_mode` in `extra_params` | `input_reference` | Primary output |
| --- | --- | --- | --- |
| Policy | `policy` | Image + instruction | Video + action chunk (read from completed job JSON) |
| Inverse dynamics | `inverse_dynamics` | Video + instruction | Predicted ego/action trajectory in `action` |
| Forward dynamics | `forward_dynamics` | Image + action chunk | Video (`/content`); action may be absent |

Cookbook forward-dynamics flows POST the same multipart fields as sync, but target `/v1/videos` and poll every two seconds until completion.

:::

### Async job lifecycle

```mermaid
sequenceDiagram
    participant Client
    participant API as vLLM-Omni /v1/videos
    Client->>API: POST multipart (prompt, extra_params, input_reference)
    API-->>Client: { id }
    loop Until completed
        Client->>API: GET /v1/videos/{id}
        API-->>Client: { status, progress }
    end
    Client->>API: GET /v1/videos/{id}/content
    API-->>Client: MP4 bytes
    Note over Client: action fields read from final GET /v1/videos/{id}
```

## `extra_params` reference

Pass `extra_params` as a **single JSON object** serialized into one form field. Keys used in Cosmos 3 cookbooks and the root README:

| Key | Applies to | Purpose |
| --- | --- | --- |
| `action_mode` | Action | `policy`, `inverse_dynamics`, or `forward_dynamics` |
| `domain_name` | Action | Embodiment selector: `av`, `droid_lerobot`, `umi`, `bridge_orig_lerobot`, `camera_pose`, and variants in vLLM-Omni online-serving examples |
| `action_chunk_size` | Action | Trajectory length in action steps (AV example: 60; DROID/UMI: 16) |
| `raw_action_dim` | Action | Action vector width (inverse dynamics AV example: `9`; UMI: `10`) |
| `action_path` | Action | Server-readable path to a JSON action file (requires `--allowed-local-media-path`) |
| `action` | Action | Inline JSON array of action rows (used in forward-dynamics notebooks instead of `action_path`) |
| `image_size` | Action | Conditioning resolution tier (for example `480`) |
| `view_point` | Action | Camera/view identifier (for example `ego_view`) |
| `condition_frame_indexes_vision` | Video-to-video | Which source frames stay as clean vision conditioning |
| `condition_video_keep` | Video-to-video | Conditioning retention policy for source video |
| `use_resolution_template` | Vision | When `false`, do not wrap prompt with resolution template |
| `use_duration_template` | Vision | When `false`, do not wrap prompt with duration template |
| `guardrails` | All | When `false`, skip prompt screening and face blurring for that request |

<Warning>
Use curl `--form-string` for `prompt`, `negative_prompt`, and `extra_params`. With `-F`, curl treats `;` as a MIME parameter separator and **silently truncates** values that contain semicolons (common inside JSON).
</Warning>

<Tip>
Reserve `-F` for file uploads such as `input_reference=@/path/to/image.jpg`. Text fields that are safe only as `--form-string` should not be sent with `-F`.
</Tip>

### Action `extra_params` examples

**AV forward dynamics** (async job, 60-step chunk at 10 FPS):

```json
{
  "action_mode": "forward_dynamics",
  "domain_name": "av",
  "action_chunk_size": 60,
  "image_size": 480,
  "view_point": "ego_view",
  "action": [[...]],
  "guardrails": false
}
```

Form fields alongside `extra_params`: `num_frames` = `61` (`action_chunk_size + 1`), `fps` = `10`, `guidance_scale` = `1.0`, `flow_shift` = `10.0`, plus `input_reference` image upload.

**AV inverse dynamics** (async job, video input):

```json
{
  "action_mode": "inverse_dynamics",
  "domain_name": "av",
  "action_chunk_size": 60,
  "image_size": 480,
  "view_point": "ego_view",
  "raw_action_dim": 9,
  "guardrails": false
}
```

On completion, read `action` from the final `GET /v1/videos/{id}` JSON (notebooks expect `action.data` for inverse dynamics).

**DROID / UMI forward dynamics** use `domain_name` values `droid_lerobot` and `umi` respectively, with `action_chunk_size` 16 and embodiment-specific FPS (15 for DROID, 20 for UMI in cookbook specs).

## Embodiment quick reference

Action cookbooks align `domain_name`, frame counts, and action dimensionality as follows:

| Embodiment | `domain_name` (examples) | Action dim | Typical `action_chunk_size` | FPS in cookbooks |
| --- | --- | ---: | ---: | ---: |
| Autonomous vehicle | `av` | 9D ego pose | 60 | 10 |
| DROID | `droid_lerobot` | 10D (9D pose + 1D gripper) | 16 | 15 |
| UMI | `umi` | 10D | 16 | 20 |

See the action modality page for full embodiment semantics and additional `domain_name` values referenced in vLLM-Omni online-serving examples.

## Guardrails

Cosmos 3 enables safety guardrails by default (prompt screening and face blurring). Disable per request:

```bash
curl -sS -X POST http://localhost:8000/v1/videos/sync \
  --form-string "prompt=A small warehouse robot moves a blue box." \
  --form-string 'extra_params={"guardrails":false,"use_resolution_template":false,"use_duration_template":false}' \
  -o cosmos3_t2v.mp4
```

To disable guardrails server-wide (models not loaded; per-request `guardrails: true` cannot re-enable), pass a deploy config with `model_config.guardrails: false` on `vllm serve … --deploy-config no_guardrails.yaml`.

## Resolution and sampling defaults

| Setting | Supported / default |
| --- | --- |
| Resolution tiers | 256p (`320x192`), 480p (`832x480`), 720p (`1280x720`); default tier 480p |
| Aspect ratios | 16:9, 4:3, 1:1, 3:4, 9:16; default 16:9 |
| Frame count | 5–300; default 189 |
| Frame rates | 10, 16, 24, 30 FPS; default 24 |
| Prompt length | Fewer than 300 words recommended for generation prompts |

External API field definitions also appear in the [vLLM-Omni Image Generation API](https://docs.vllm.ai/projects/vllm-omni/en/latest/serving/image_generation_api/) and [Videos API](https://docs.vllm.ai/projects/vllm-omni/en/latest/serving/videos_api/) documentation; Cosmos-specific behavior is concentrated in `extra_params`, `extra_args`, and `action_mode`.

## Related pages

<CardGroup>
<Card title="Run Generator with vLLM-Omni" href="/run-generator-vllm-omni">
Docker server startup, tensor parallelism, CFG/Ulysses, and deploy-config guardrail disable.
</Card>
<Card title="Run Generator action workflows" href="/run-generator-action">
Forward and inverse dynamics multipart requests with `domain_name` and chunked robotics generation.
</Card>
<Card title="Action modality" href="/action-modality">
Embodiment dimensions, policy/inverse/forward dynamics semantics, and `domain_name` conditioning.
</Card>
<Card title="Input and output specifications" href="/input-output-specifications">
Resolution tiers, frame rates, vision conditioning frame counts, and sound output format.
</Card>
<Card title="Sampling and prompt parameters" href="/sampling-and-prompt-parameters">
Structured JSON prompts, upsampling defaults, and template toggles mirrored in `extra_params`.
</Card>
<Card title="Audiovisual cookbooks" href="/audiovisual-cookbooks">
End-to-end `run_with_vllm_omni.ipynb` recipes for image and video with optional sound.
</Card>
</CardGroup>