# Run Generator action workflows

> Forward dynamics (image + action trajectory) and inverse dynamics (video + instruction) across Framework torchrun and vLLM-Omni multipart /v1/videos requests with domain_name and action_mode extra_params.

- Repository: NVIDIA/cosmos
- GitHub: https://github.com/NVIDIA/cosmos
- Human docs: https://grok-wiki.com/public/docs/nvidia-cosmos-82de3e90abd9
- Complete Markdown: https://grok-wiki.com/public/docs/nvidia-cosmos-82de3e90abd9/llms-full.txt

## Source Files

- `cookbooks/cosmos3/generator/action/README.md`
- `cookbooks/cosmos3/generator/action/run_fd_with_cosmos_framework.ipynb`
- `cookbooks/cosmos3/generator/action/run_fd_with_vllm.ipynb`
- `cookbooks/cosmos3/generator/action/run_id_with_cosmos_framework.ipynb`
- `cookbooks/cosmos3/generator/action/assets/actions/av_traj_left.json`
- `cookbooks/cosmos3/generator/action/assets/videos/av_0.mp4`

---

---
title: "Run Generator action workflows"
description: "Forward dynamics (image + action trajectory) and inverse dynamics (video + instruction) across Framework torchrun and vLLM-Omni multipart /v1/videos requests with domain_name and action_mode extra_params."
---

Cosmos3-Nano Generator action workflows live under `cookbooks/cosmos3/generator/action/`: JSONL input specs drive `cosmos_framework.scripts.inference` (native PyTorch), while the same specs map to multipart `POST /v1/videos` jobs on vLLM-Omni with `extra_params.action_mode` and `extra_params.domain_name`. Forward dynamics (`forward_dynamics`) rolls out video from a conditioning image plus an action trajectory; inverse dynamics (`inverse_dynamics`) predicts ego-motion from an input video and instruction.

## Modes and embodiments

| Mode | Framework `model_mode` | vLLM `extra_params.action_mode` | Inputs | Outputs (cookbooks) |
| --- | --- | --- | --- | --- |
| Forward dynamics | `forward_dynamics` | `forward_dynamics` | Start image (`vision_path`) + action file (`action_path`) or inline `action` | `vision.mp4` per run |
| Inverse dynamics | `inverse_dynamics` | `inverse_dynamics` | Input video (`vision_path`), no `action_path` | `sample_outputs.json` with predicted action |

Embodiment conditioning uses `domain_name` (and usually `view_point`). Checked-in examples:

| Embodiment | `domain_name` | Action dim | Chunk / duration (cookbooks) |
| --- | --- | --- | --- |
| Autonomous vehicle | `av` | 9D ego pose | 60 actions, 10 FPS → `num_frames` 61 |
| DROID (LeRobot) | `droid_lerobot` | 10D (9D pose + gripper) | 16 actions per chunk, 5 autoregressive chunks |
| UMI | `umi` | 10D | 16 actions per chunk, all chunks in `assets/actions/umi.json` |

Pose semantics (9D translation + 6D rotation, grasp encoding) are documented in the action cookbook README; AV trajectories use `rot6d`, `backward_framewise`, and `translation_scale=1.35` when visualizing.

## Cookbook layout and assets

```text
cookbooks/cosmos3/generator/action/
├── README.md
├── run_fd_with_cosmos_framework.ipynb   # FD: AV, DROID, UMI (Framework)
├── run_id_with_cosmos_framework.ipynb   # ID: AV (Framework)
├── run_fd_with_vllm.ipynb               # FD: AV, DROID, UMI (vLLM-Omni)
├── run_id_with_vllm.ipynb               # ID: AV (vLLM-Omni)
└── assets/
    ├── images/          # av_0.jpg, umi.png, …
    ├── actions/         # av_traj_*.json, umi.json
    ├── videos/          # av_0.mp4, av_1.mp4 (inverse dynamics)
    └── droid_lerobot_example/   # LeRobot sample for robotics FD
```

Environment setup (HF auth, CUDA groups, Framework clone, vLLM-Omni Docker) is centralized in the [Cosmos3 cookbooks environment setup](/cookbook-environment); action notebooks assume that baseline.

## JSONL input spec (both backends)

Each line is one JSON object. Framework reads `model_mode`; vLLM maps the same run to `extra_params.action_mode`.

**Forward dynamics (AV example)** — one shared start frame, three trajectories:

```json
{
  "name": "av_left",
  "model_mode": "forward_dynamics",
  "domain_name": "av",
  "view_point": "ego_view",
  "vision_path": "/abs/path/to/assets/images/av_0.jpg",
  "action_path": "/abs/path/to/assets/actions/av_traj_left.json",
  "action_chunk_size": 60,
  "fps": 10,
  "image_size": 480,
  "prompt": "You are an autonomous vehicle planning system.",
  "seed": 0
}
```

**Inverse dynamics (AV)** — video only; action is predicted:

```json
{
  "name": "av_inverse_0",
  "model_mode": "inverse_dynamics",
  "domain_name": "av",
  "view_point": "ego_view",
  "vision_path": "/abs/path/to/assets/videos/av_0.mp4",
  "action_chunk_size": 60,
  "fps": 10,
  "image_size": 480,
  "prompt": "You are an autonomous vehicle planning system.",
  "seed": 0
}
```

<Note>
Framework resolves relative paths against the JSONL file directory. vLLM notebooks write **absolute** `vision_path` values because request cells read records directly; action trajectories for FD are embedded in `extra_params.action` (loaded from `action_path`) rather than sent as a separate upload.
</Note>

## Run with Cosmos Framework

Prerequisites: Linux + NVIDIA GPU, Hugging Face access to `Cosmos3-Nano`, Framework checkout (notebooks default `packages/cosmos3`), `uv sync --all-extras --group=cu130-train` (or `cu128-train` per driver).

### Entry command

The README quickstart uses `torchrun`; notebooks invoke the same module with single-process distributed env vars:

```bash
torchrun --nproc-per-node=1 \
  -m cosmos_framework.scripts.inference \
  --parallelism-preset=latency \
  -i <path/to/spec.jsonl> \
  -o <output_dir> \
  --checkpoint-path Cosmos3-Nano \
  --image_size 480 \
  --video-save-quality 8 \
  --seed 0
```

AV forward dynamics (batch JSONL):

```bash
export COSMOS3_AV_FD_INPUT=.../inputs/action_forward_dynamics_av_custom.jsonl
export COSMOS3_AV_FD_OUTPUT=.../action_forward_dynamics_av_custom

CUDA_VISIBLE_DEVICES=0 \
MASTER_ADDR=127.0.0.1 MASTER_PORT=29500 RANK=0 WORLD_SIZE=1 LOCAL_RANK=0 \
  .venv/bin/python -m cosmos_framework.scripts.inference \
  --parallelism-preset=latency \
  -i "$COSMOS3_AV_FD_INPUT" \
  -o "$COSMOS3_AV_FD_OUTPUT" \
  --checkpoint-path Cosmos3-Nano \
  --image_size 480 \
  --video-save-quality 8 \
  --seed 0 \
  --benchmark
```

Inverse dynamics uses the same flags with `action_inverse_dynamics_av_custom.jsonl` and writes **`sample_outputs.json`** per run (not `vision.mp4`):

```text
<output_dir>/<name>/sample_outputs.json
  └── outputs[0].content["action"]   # [T-1, 9] for AV
```

### Autoregressive robotics and UMI

DROID and UMI forward dynamics run **one JSONL line per 16-action chunk**. Chunk 0 conditions on a ground-truth image; later chunks condition on the **last frame** of the previous chunk’s generated video (extracted with ffmpeg in the notebook). Robotics uses `--no-guardrails` and `domain_name: droid_lerobot`; UMI uses `domain_name: umi`, `image_size: 256`, and `fps: 20`.

Default Framework output root (unless `COSMOS3_OUTPUT_ROOT` is set):

```text
packages/cosmos3/outputs/cookbooks/cosmos3/generator/action/
├── inputs/…jsonl
└── action_forward_dynamics_av_custom/<name>/vision.mp4
```

## Run with vLLM-Omni

### Start server

From the `cosmos` repo root (mount repo + HF cache; allow local media):

```bash
docker rm -f cosmos3-vllm-omni-notebook 2>/dev/null || true

docker run -d --name cosmos3-vllm-omni-notebook \
  --runtime nvidia --gpus '"device=0"' \
  -e CUDA_DEVICE_ORDER=PCI_BUS_ID \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -v "$PWD:/workspace" \
  -p 8001:8000 --ipc=host \
  vllm/vllm-omni:cosmos3 \
  vllm serve nvidia/Cosmos3-Nano \
    --omni \
    --model-class-name Cosmos3OmniDiffusersPipeline \
    --allowed-local-media-path / \
    --port 8000 \
    --init-timeout 1800

curl http://localhost:8001/v1/models
```

Notebooks default `COSMOS3_VLLM_BASE_URL=http://localhost:8001` and outputs under `outputs/cosmos3_action_vllm/`.

### Multipart `POST /v1/videos`

Action jobs use the **async** video endpoint (poll until `status=completed`). Forward dynamics then downloads MP4 bytes from `/v1/videos/{id}/content`; inverse dynamics reads predicted action from the completed job JSON.

```mermaid
sequenceDiagram
  participant NB as action notebook
  participant API as vLLM-Omni /v1/videos
  participant FS as run_dir

  NB->>API: POST multipart (prompt, num_frames, extra_params, input_reference)
  API-->>NB: job id
  loop until completed
    NB->>API: GET /v1/videos/{id}
    API-->>NB: status, progress, action (ID)
  end
  alt forward_dynamics
    NB->>API: GET /v1/videos/{id}/content
    API-->>FS: vision.mp4
  else inverse_dynamics
    NB->>FS: action.json, sample_outputs.json
  end
```

<Warning>
Use `curl --form-string` for `prompt` and `extra_params`, not `-F`, when values contain `;` (see root README). Encode `extra_params` as a single JSON string.
</Warning>

### `extra_params` for action

| Field | Forward dynamics | Inverse dynamics |
| --- | --- | --- |
| `action_mode` | `forward_dynamics` | `inverse_dynamics` |
| `domain_name` | e.g. `av`, `droid_lerobot`, `umi` | e.g. `av` |
| `action_chunk_size` | Matches JSONL | Matches JSONL |
| `image_size` | e.g. `480` (AV/DROID), `256` (UMI) | e.g. `480` |
| `view_point` | e.g. `ego_view` or dataset viewpoint | e.g. `ego_view` |
| `action` | 2D array loaded from `action_path` | omitted |
| `raw_action_dim` | omitted (FD) | `9` for AV ID notebooks |
| `guardrails` | `false` in cookbook requests | `false` in cookbook requests |

Top-level form fields used in notebooks (in addition to `input_reference` file upload):

<ParamField body="prompt" type="string" required>
Instruction or domain prompt (AV: autonomous-vehicle planning string; robotics: dataset `ai_caption`).
</ParamField>

<ParamField body="num_frames" type="integer" required>
Set to `action_chunk_size + 1` (e.g. 61 for AV).
</ParamField>

<ParamField body="fps" type="integer" required>
Embodiment frame rate (AV: 10; UMI: 20; DROID: from dataset `conditioning_fps`).
</ParamField>

<ParamField body="num_inference_steps" type="integer">
Cookbooks use `30`.
</ParamField>

<ParamField body="guidance_scale" type="float">
Cookbooks use `1.0` for action runs.
</ParamField>

<ParamField body="flow_shift" type="float">
Cookbooks use `10.0`.
</ParamField>

<ParamField body="extra_params" type="string (JSON)" required>
Stringified JSON object; must include `action_mode` and `domain_name`.
</ParamField>

**Forward dynamics** uploads the conditioning **image** as `input_reference`. **Inverse dynamics** uploads the **video** (`video/mp4`). Robotics/UMI FD omit `size`/`width`/`height`; the server derives canvas from `image_size` with aspect-preserving padding.

Example FD `extra_params` payload shape:

```json
{
  "action_mode": "forward_dynamics",
  "domain_name": "av",
  "action_chunk_size": 60,
  "image_size": 480,
  "view_point": "ego_view",
  "action": [[...9 floats...], ...],
  "guardrails": false
}
```

Root README notes FD may also use synchronous `POST /v1/videos/sync` when only video is returned; the action cookbooks standardize on async `POST /v1/videos` plus `/content` download for parity with inverse-dynamics job polling.

### vLLM output layout

```text
outputs/cosmos3_action_vllm/
├── inputs/action_forward_dynamics_av_custom.jsonl
├── action_forward_dynamics_av_custom/<name>/
│   ├── response.json
│   ├── final.json
│   └── vision.mp4
├── action_inverse_dynamics_av_custom/<name>/
│   ├── response.json
│   ├── final.json
│   ├── action.json
│   └── sample_outputs.json
└── action_forward_dynamics_robotics_custom/ …
```

## Verification

<Steps>
<Step title="Confirm environment">
HF login succeeds; Framework `.venv` or vLLM container sees GPU; `curl http://localhost:8001/v1/models` returns Cosmos3-Nano metadata (vLLM path).
</Step>
<Step title="Run AV forward dynamics">
Three trajectories (`av_forward`, `av_left`, `av_right`) from one `av_0.jpg` produce three `vision.mp4` files under `action_forward_dynamics_av_custom/`.
</Step>
<Step title="Run AV inverse dynamics">
`av_0.mp4` and `av_1.mp4` produce `sample_outputs.json` per run; visualize predicted 9D poses with the notebook pose utilities.
</Step>
<Step title="Optional robotics / UMI">
DROID: 5×16 frames autoregressive with last-frame conditioning; UMI: chunks stitched to one preview MP4. Expect guardrails disabled and `--no-guardrails` on Framework robotics runs.
</Step>
</Steps>

<Check>
Success signals: JSONL specs print in notebook setup cells; Framework exits 0 with `vision.mp4` or `sample_outputs.json` under the configured output root; vLLM jobs reach `status: completed` and FD runs save non-empty `vision.mp4`.
</Check>

## Framework vs vLLM-Omni

| Concern | Cosmos Framework | vLLM-Omni |
| --- | --- | --- |
| Mode field | `model_mode` in JSONL | `action_mode` in `extra_params` |
| Action input | `action_path` on disk | `action` array inside `extra_params` (FD) |
| Inference API | `python -m cosmos_framework.scripts.inference` | `POST /v1/videos` + poll + optional `/content` |
| Default outputs | Under Framework checkout `outputs/cookbooks/...` | `outputs/cosmos3_action_vllm/` |
| Best for | Training-aligned batch runs, `--benchmark`, `--no-guardrails` CLI | OpenAI-compatible serving, same prompts as production |

## Related pages

<CardGroup>
<Card title="Action modality" href="/action-modality">
Action token semantics, embodiment dimensions, and `domain_name` conditioning.
</Card>
<Card title="Action cookbook recipes" href="/action-cookbooks">
Notebook index with asset paths and output directories.
</Card>
<Card title="Run Generator with Cosmos Framework" href="/run-generator-cosmos-framework">
General Framework inference flags and parallelism presets.
</Card>
<Card title="Run Generator with vLLM-Omni" href="/run-generator-vllm-omni">
Server startup, guardrails, and vision endpoints.
</Card>
<Card title="vLLM-Omni API reference" href="/vllm-omni-api-reference">
`/v1/videos` fields, `action_mode` values, and curl constraints.
</Card>
<Card title="Cookbook environment setup" href="/cookbook-environment">
Shared uv/Docker setup for Framework and vLLM-Omni.
</Card>
</CardGroup>