# Action cookbook recipes > Forward-dynamics and inverse-dynamics notebooks for AV, DROID, and UMI with checked-in trajectories, LeRobot sample data, and Framework vs vLLM-Omni output directories. - Repository: NVIDIA/cosmos - GitHub: https://github.com/NVIDIA/cosmos - Human docs: https://grok-wiki.com/public/docs/nvidia-cosmos-82de3e90abd9 - Complete Markdown: https://grok-wiki.com/public/docs/nvidia-cosmos-82de3e90abd9/llms-full.txt ## Source Files - `cookbooks/cosmos3/generator/action/README.md` - `cookbooks/cosmos3/generator/action/run_fd_with_cosmos_framework.ipynb` - `cookbooks/cosmos3/generator/action/run_fd_with_vllm.ipynb` - `cookbooks/cosmos3/generator/action/run_id_with_cosmos_framework.ipynb` - `cookbooks/cosmos3/generator/action/run_id_with_vllm.ipynb` - `cookbooks/cosmos3/generator/action/assets/images/av_0.jpg` - `README.md` --- --- title: "Action cookbook recipes" description: "Forward-dynamics and inverse-dynamics notebooks for AV, DROID, and UMI with checked-in trajectories, LeRobot sample data, and Framework vs vLLM-Omni output directories." --- The `cookbooks/cosmos3/generator/action/` tree ships four Jupyter notebooks that run **Cosmos3-Nano** action Generator workflows: forward dynamics (image + trajectory → video) for AV, DROID, and UMI, and inverse dynamics (AV video → predicted ego-motion) on two backends—native **Cosmos Framework** (`cosmos_framework.scripts.inference`) and **vLLM-Omni** (`POST /v1/videos`). Checked-in assets under `assets/` supply trajectories, images, videos, and a trimmed LeRobot DROID sample; each notebook writes JSONL specs under an `inputs/` subtree and artifacts under backend-specific output roots. ## Notebook map | Task | Backend | Notebook | Primary outputs | | --- | --- | --- | --- | | Forward dynamics | Cosmos Framework | `run_fd_with_cosmos_framework.ipynb` | Per-run `vision.mp4`; robotics/UMI stitched MP4s | | Forward dynamics | vLLM-Omni | `run_fd_with_vllm.ipynb` | Same layout under `outputs/cosmos3_action_vllm/` | | Inverse dynamics | Cosmos Framework | `run_id_with_cosmos_framework.ipynb` | Per-run `sample_outputs.json` (predicted action) | | Inverse dynamics | vLLM-Omni | `run_id_with_vllm.ipynb` | Same + API debug JSON (`response.json`, `final.json`, `action.json`) | Environment setup (HF auth, CUDA groups, Docker server, framework clone) is centralized in the [Cookbook environment setup](/cookbook-environment) page; the action folder README links to those sections for each backend. ## Forward vs inverse dynamics | Mode | `model_mode` / `action_mode` | Inputs | Model output | Cookbook embodiments | | --- | --- | --- | --- | --- | | Forward dynamics | `forward_dynamics` | Start image (`vision_path`) + action JSON (`action_path`) | Generated video (`vision.mp4`) | AV (3 trajectories), DROID (autoregressive chunks), UMI (autoregressive chunks) | | Inverse dynamics | `inverse_dynamics` | Input video only (`vision_path`; no `action_path`) | Predicted action in `sample_outputs.json` | AV only (`av_0.mp4`, `av_1.mp4`) | Forward dynamics rolls out future observations conditioned on ego or end-effector motion. Inverse dynamics recovers the ego-motion trajectory that explains each input driving clip. ## Embodiment parameters The action README defines pose semantics and generation timing per embodiment: | Embodiment | `domain_name` | Action dim | Chunk size in cookbooks | FPS | `image_size` | | --- | --- | --- | --- | --- | --- | | Autonomous vehicle | `av` | 9D ego pose | 60 | 10 | 480 | | DROID (LeRobot) | `droid_lerobot` | 10D (9D pose + 1D gripper) | 16 per chunk, 5 chunks (80 frames) | 15 (from dataset) | 480 | | UMI | `umi` | 10D | 16 per chunk; checked-in `umi.json` has 32 rows → 2 chunks | 20 | 256 | Action rows are JSON arrays of floats. AV trajectories use `rot6d` rotation, `backward_framewise` pose convention, and `translation_scale=1.35` when converting absolute camera poses via `cosmos_framework.data.vfm.action.pose_utils.pose_abs_to_rel`. ## Checked-in assets :::files cookbooks/cosmos3/generator/action/ ├── README.md ├── run_fd_with_cosmos_framework.ipynb ├── run_fd_with_vllm.ipynb ├── run_id_with_cosmos_framework.ipynb ├── run_id_with_vllm.ipynb └── assets/ ├── images/ │ ├── av_0.jpg # AV FD start frame │ ├── av_1.jpg │ └── umi.png # UMI FD start frame ├── videos/ │ ├── av_0.mp4 # AV ID inputs │ ├── av_1.mp4 │ └── umi.mp4 ├── actions/ │ ├── av_traj_forward.json │ ├── av_traj_left.json │ ├── av_traj_right.json │ └── umi.json # 32×10D UMI actions └── droid_lerobot_example/ # LeRobot v3.0 sample (1 episode, 3 cameras) ├── meta/info.json ├── data/chunk-000/file-000.parquet └── videos/observation.image.*/ ::: DROID forward-dynamics notebooks load this tree with `DROIDLeRobotDataset` from `cosmos_framework.data.vfm.action.datasets`, extract 16-action chunks, write per-chunk action JSON under `inputs/`, and use the dataset’s `ai_caption` as the generation prompt. ## Output directories Backends use separate default roots so Framework and vLLM runs do not collide. | Backend | Default root | Override env var | | --- | --- | --- | | Cosmos Framework | `packages/cosmos3/outputs/cookbooks/cosmos3/generator/action/` | `COSMOS3_OUTPUT_ROOT` | | vLLM-Omni | `/outputs/cosmos3_action_vllm/` | `COSMOS3_VLLM_OUTPUT_ROOT` | Both layouts place derived specs in `inputs/` and run artifacts in named subfolders: ```text / ├── inputs/ │ ├── action_forward_dynamics_av_custom.jsonl │ ├── action_forward_dynamics_robotics_custom.jsonl │ ├── action_forward_dynamics_umi_custom.jsonl │ ├── action_inverse_dynamics_av_custom.jsonl │ ├── robotics_droid_action_chunk_XX.json │ ├── umi_action_chunk_XX_10d.json │ └── … conditioning PNGs for autoregressive chunks ├── action_forward_dynamics_av_custom//vision.mp4 ├── action_forward_dynamics_robotics_custom//vision.mp4 │ └── robotics_action_cond_stitched.mp4 ├── action_forward_dynamics_umi_custom//vision.mp4 │ └── umi_action_cond_stitched.mp4 └── action_inverse_dynamics_av_custom//sample_outputs.json ``` Framework notebooks also export `COSMOS3_*_INPUT` / `COSMOS3_*_OUTPUT` env vars for bash inference cells. vLLM notebooks default `COSMOS3_VLLM_BASE_URL` to `http://localhost:8001` (Docker maps container port 8000 → host 8001). ## JSONL input spec Each inference run is one JSON object per line. Shared fields across action cookbooks: | Field | Forward dynamics | Inverse dynamics | | --- | --- | --- | | `name` | Run identifier (output subfolder) | Same | | `model_mode` | `forward_dynamics` | `inverse_dynamics` | | `domain_name` | `av`, `droid_lerobot`, or `umi` | `av` | | `vision_path` | Absolute path to start image | Absolute path to input video | | `action_path` | Absolute path to action JSON array | Omitted | | `action_chunk_size` | 60 (AV), 16 (robotics/UMI) | 60 | | `fps` | 10 / 15 / 20 | 10 | | `image_size` | 480 or 256 (UMI) | 480 | | `view_point` | `ego_view` or dataset viewpoint | `ego_view` | | `prompt` | Task text (AV planning string; DROID caption; UMI: `"mouse arrangement"`) | AV planning string | | `seed` | Per-run seed | `0` | AV forward dynamics reuses one start image (`assets/images/av_0.jpg`) with three trajectories (`av_forward`, `av_left`, `av_right`). ## Run with Cosmos Framework Clone or reuse `packages/cosmos3`, run `uv sync --all-extras --group=cu130-train` (or `cu128-train` on CUDA 12.x drivers), authenticate to Hugging Face, and set `COSMOS3_CHECKPOINT_PATH=Cosmos3-Nano` if needed. See [Run Generator with Cosmos Framework](/run-generator-cosmos-framework). - Forward: `run_fd_with_cosmos_framework.ipynb` (AV, DROID autoregressive, UMI autoregressive) - Inverse: `run_id_with_cosmos_framework.ipynb` (AV videos only) AV forward dynamics invokes the framework entrypoint once per JSONL file: ```bash .venv/bin/python -m cosmos_framework.scripts.inference \ --parallelism-preset=latency \ -i "$COSMOS3_AV_FD_INPUT" \ -o "$COSMOS3_AV_FD_OUTPUT" \ --checkpoint-path "$COSMOS3_CHECKPOINT_PATH" \ --video-save-quality 8 \ --image_size 480 \ --seed 0 \ --benchmark ``` DROID and UMI forward dynamics loop five and two chunks respectively, calling inference per chunk with `--no-guardrails`, then ffmpeg-extract the last generated frame as the next chunk’s conditioning image. Robotics uses `--parallelism-preset=latency` and writes `robotics_action_cond_stitched.mp4` (80 frames). UMI stitches to `umi_action_cond_stitched.mp4` (32 frames from two 16-action chunks). Inverse dynamics runs the same entrypoint against `action_inverse_dynamics_av_custom.jsonl` and reads predicted actions from `//sample_outputs.json` at `outputs[0].content["action"]`. Framework notebooks clone `https://github.com/NVIDIA/cosmos-framework.git` into `packages/cosmos3` when missing and set `GIT_LFS_SKIP_SMUDGE=1` during `uv sync` so LeRobot LFS blobs in upstream test data do not break installs. ## Run with vLLM-Omni From the `cosmos` repo root, start the official image (mount HF cache and workspace, allow local media): ```bash docker rm -f cosmos3-vllm-omni-notebook 2>/dev/null || true docker run -d --name cosmos3-vllm-omni-notebook \ --runtime nvidia --gpus '"device=0"' \ -e CUDA_DEVICE_ORDER=PCI_BUS_ID \ -v ~/.cache/huggingface:/root/.cache/huggingface \ -v "$PWD:/workspace" \ -p 8001:8000 --ipc=host \ vllm/vllm-omni:cosmos3 \ vllm serve nvidia/Cosmos3-Nano \ --omni \ --model-class-name Cosmos3OmniDiffusersPipeline \ --allowed-local-media-path / \ --port 8000 curl http://localhost:8001/v1/models ``` `run_fd_with_vllm.ipynb` / `run_id_with_vllm.ipynb` resolve `COSMOS_ROOT`, default `COSMOS3_VLLM_OUTPUT_ROOT` to `outputs/cosmos3_action_vllm`, and poll `COSMOS3_VLLM_BASE_URL`. Forward dynamics posts multipart jobs to `POST /v1/videos` with `input_reference` (image) and JSON `extra_params` embedding the action array inline (not only `action_path`): `forward_dynamics` or `inverse_dynamics`. Embodiment key: `av`, `droid_lerobot`, or `umi`. Must match trajectory length (60 for AV, 16 for robotics/UMI chunks). Forward dynamics only: JSON action trajectory loaded from `action_path`. Canvas short side; robotics omits explicit `size` so vLLM pads to aspect ratio. e.g. `ego_view` or DROID dataset `viewpoint`. Inverse dynamics AV runs set `9`. Cookbooks set `false` for reproducible demos; robotics FD explicitly disables guardrails. Top-level form fields used in notebooks: `prompt`, `num_frames` (`action_chunk_size + 1`), `fps`, `num_inference_steps` (30), `guidance_scale` (1.0), `flow_shift` (10.0), `seed`, `extra_params` (JSON string). Forward dynamics polls `GET /v1/videos/{id}` until `completed`, then downloads `GET /v1/videos/{id}/content` to `/vision.mp4`. Inverse dynamics uses the same async job flow; predicted actions land in `sample_outputs.json` (and optional `action.json`). DROID/UMI autoregressive loops match the Framework pattern: last frame → next chunk’s PNG via ffmpeg. ```mermaid sequenceDiagram participant NB as action notebook participant API as vLLM-Omni :8001 participant FS as output run_dir NB->>API: POST /v1/videos (image + extra_params) API-->>NB: job id loop until completed NB->>API: GET /v1/videos/{id} API-->>NB: status / progress end NB->>API: GET /v1/videos/{id}/content API-->>FS: vision.mp4 (FD) or action payload (ID) NB->>FS: sample_outputs.json (ID) ``` Inverse dynamics returns an action chunk in the job result; the README notes policy and inverse modes use asynchronous `POST /v1/videos` (not `/v1/videos/sync`). Forward dynamics can use sync in general deployments; these cookbooks use the async poll pattern for all video jobs. ## Autoregressive robotics and UMI Both forward-dynamics backends share the same chunking model: ```text Chunk 0: GT image + action JSON → vision.mp4 └─ extract frame N ──► conditioning PNG for chunk 1 Chunk 1..K-1: repeat with updated vision_path Stitch: concat per-chunk MP4s → *_stitched.mp4 ``` | Example | Chunks | Actions per chunk | Total generated frames | Stitched artifact | | --- | --- | --- | --- | --- | | DROID LeRobot | 5 | 16 | 80 | `robotics_action_cond_stitched.mp4` | | UMI | 2 (from 32-row `umi.json`) | 16 | 32 | `umi_action_cond_stitched.mp4` | DROID chunk 0 saves the ground-truth first frame from `DROIDLeRobotDataset` to `robotics_droid_autoregressive_input_chunk_00.png`. Later chunks use paths `robotics_droid_autoregressive_input_chunk_XX.png` populated from the previous chunk’s generated video. ## Verification signals | Check | Expected signal | | --- | --- | | vLLM server | `curl http://localhost:8001/v1/models` returns model metadata | | Framework GPU | Notebook verify cell reports `cuda available: True` | | AV FD | Three runs under `action_forward_dynamics_av_custom/*/vision.mp4` | | AV ID | `sample_outputs.json` per `av_inverse_0` / `av_inverse_1` with `outputs[0].content.action` | | DROID FD | Five chunk folders + optional 80-frame stitched MP4 | | UMI FD | Two chunk folders + 32-frame stitched MP4 | Trajectory visualization cells plot AV ego paths (3D frustum + bird’s-eye) from input JSON (FD) or predicted actions (ID). ## Choosing a backend | Goal | Prefer | | --- | --- | | Match training/inference stack, batch JSONL, local checkpoint control | Cosmos Framework notebooks | | OpenAI-compatible HTTP API, Docker-only deploy, no framework venv | vLLM-Omni notebooks | | Embed action inline in HTTP without mounting `action_path` on server | vLLM (`extra_params.action` array) | | Per-chunk subprocess control, `--no-guardrails` CLI flag | Framework (robotics/UMI loops) | Detailed API fields and curl constraints live on [vLLM-Omni API reference](/vllm-omni-api-reference) and [Run Generator action workflows](/run-generator-action). Action token semantics and embodiment dimensions are documented on [Action modality](/action-modality). ## Related pages Shared uv/Docker setup for Framework and vLLM-Omni before running action notebooks. Forward and inverse dynamics CLI/API patterns beyond the notebook walkthrough. Action token semantics, embodiment dimensions, and `domain_name` conditioning. Server flags, guardrails, and multipart video endpoints used by `run_*_with_vllm.ipynb`. `cosmos_framework.scripts.inference` parallelism presets and checkpoint paths. `POST /v1/videos`, `extra_params`, and `action_mode` values.