# Automation

> Embedded automation server: accessibility snapshots, widget driving, record/replay, deterministic screenshots, and agent command surface.

- Repository: vercel-labs/native
- GitHub: https://github.com/vercel-labs/native
- Human docs: https://grok-wiki.com/public/docs/vercel-labs-native-8ccc3580636a
- Complete Markdown: https://grok-wiki.com/public/docs/vercel-labs-native-8ccc3580636a/llms-full.txt

## Source Files

- `tools/native-sdk/automation.zig`
- `src/runtime/automation_commands.zig`
- `src/runtime/automation_snapshot.zig`
- `src/runtime/automation_widget_dispatch.zig`
- `skill-data/automation/SKILL.md`

---

---
title: "Automation"
description: "Embedded automation server: accessibility snapshots, widget driving, record/replay, deterministic screenshots, and agent command surface."
---

Every Native SDK app can embed an automation server that publishes runtime state into `.zig-cache/native-sdk-automation/` and consumes a FIFO queue of `command-<n>.txt` entries. The `native automate` CLI is the agent and smoke-test surface: wait for readiness, assert on accessibility snapshots, drive retained-canvas widgets through real platform input paths, capture deterministic CPU-reference screenshots, round-trip bridge JSON, and launch session record/replay journals.

<Warning>
Automation is not browser DOM automation. It reports runtime, window, tray, audio, and widget state; drives retained-canvas (`gpu_surface`) widgets; and can reload WebViews or dispatch bridge requests. For DOM testing of the optional WebView path, use the frontend framework's tests or a browser tool against the dev server.
</Warning>

## Architecture

```mermaid
flowchart LR
  subgraph CLI["native automate"]
    Wait["wait / assert"]
    Drive["widget-* / tray / bridge"]
    Shot["screenshot"]
    Sess["record / replay"]
  end

  subgraph Dropbox[".zig-cache/native-sdk-automation/"]
    Snap["snapshot.txt\nwindows.txt\naccessibility.txt"]
    Queue["command-N.txt\n(max 8, FIFO)"]
    Art["screenshot-*.png\nbridge-response.txt\nprovenance.txt"]
  end

  subgraph Runtime["Runtime + automation.Server"]
    Pub["publish() per frame"]
    Drain["takeCommand() one per frame"]
    Dispatch["automation_widget_dispatch\nplatform events"]
  end

  Wait --> Snap
  Drive --> Queue
  Shot --> Queue
  Queue --> Drain
  Drain --> Dispatch
  Pub --> Snap
  Pub --> Art
  Sess -->|"NATIVE_SDK_SESSION_*"| Runtime
```

| Layer | Ownership |
| --- | --- |
| Dropbox directory | Created by the **running app**, never by the CLI |
| Path resolution | Relative to the CLI's **current working directory** (run from the app project root) |
| Protocol handshake | `protocol=N` in the snapshot header; CLI refuses mismatched versions |
| Liveness | `publisher_pid` checked against the live process table; dead publishers are stale |
| Queue cadence | One command consumed per presented frame; delete entry = consumption ack |

Current dropbox protocol version is **6** (queued `command-<n>.txt` entries, max depth **8**). Bump history lives in `src/automation/protocol.zig`.

## Enable automation

Build or run with the compile-time gate:

```bash
zig build run -Dplatform=macos -Dautomation=true
# or, when using the managed CLI:
native build -Dautomation=true
```

Wire the server into `Runtime.init`:

```zig
const server = native_sdk.automation.Server.init(
    io,
    ".zig-cache/native-sdk-automation",
    "My App",
);
var runtime = native_sdk.Runtime.init(.{
    .platform = my_platform,
    .automation = server,
});
```

Apps built without `-Dautomation=true` ignore automation files. The directory name passed to `Server.init` must match where the CLI looks (default `.zig-cache/native-sdk-automation`).

## Dropbox files

| File | Role |
| --- | --- |
| `snapshot.txt` | App readiness, protocol, diagnostics, windows, views, widgets, tray, audio, errors, optional `frame_profile` |
| `accessibility.txt` | Accessibility tree summary (roles and accessible names) |
| `windows.txt` | Window list: `window @w{id} "{title}" focused={bool}` |
| `command-<n>.txt` | Queued command lines; CLI claims next sequence exclusively |
| `bridge-response.txt` | Last bridge response JSON |
| `provenance.txt` | Last provenance query result |
| `screenshot-<view-label>.png` | Atomic PNG from the reference renderer |

<Note>
`label=` on a widget replaces visible text as the accessible name. Snapshot greps and screen readers see the label, not the text — do not label an element whose visible text your assertions match.
</Note>

### Snapshot header fields

The ready line is shaped like:

```text
ready=true protocol=6 frame=… commands=… runtime_uptime_ns=… dispatch_errors=… dropped_trace_records=… publisher_pid=… markup_watch=armed|off
```

| Field | Meaning |
| --- | --- |
| `protocol` | Dropbox handshake version baked into both CLI and app |
| `dispatch_errors` | Lifetime count of handler/update errors degraded without killing the app |
| `publisher_pid` | Publishing process; CLI refuses dead pids as stale |
| `markup_watch` | `armed` when hot-reload watch is active (typically Debug + `watch_path`) |

Recent degraded errors appear as indented lines:

```text
  error event=<tag> name=<ErrorName> timestamp_ns=… detail="…"
```

While `profile on` is active, a greppable `frame_profile` line lists per-stage p50/p90/max microseconds and sample counts for `rebuild`, `layout`, `reconcile`, `emit`, `a11y`, `plan`, `patch`, `encode`, `present`, `host_decode`, and `host_draw`.

## Standard workflow

<Steps>
  <Step title="Start the app with automation">
    Build/run with `-Dautomation=true` from the app project directory so the dropbox path matches the CLI cwd.
  </Step>
  <Step title="Wait for readiness">
    ```bash
    native automate wait
    ```
    Blocks until `snapshot.txt` contains `ready=true` from a live publisher.
  </Step>
  <Step title="Assert or inspect">
    ```bash
    native automate assert 'gpu_nonblank=true' 'role=button name="Reset"'
    native automate assert --absent 'error event='
    native automate snapshot
    native automate list
    ```
  </Step>
  <Step title="Drive UI and capture">
    Use widget verbs, tray actions, bridge JSON, and `screenshot` against view labels and bare widget ids from the snapshot (`#id` prints without the `#` in CLI args).
  </Step>
</Steps>

Successful command delivery prints:

```text
delivered <action> -> <absolute automation dir>
```

The CLI waits for the app to consume the queue entry. A dead or frozen app exits non-zero instead of silently dropping the command.

## CLI command surface

Entry point: `native automate <command>` (repository-built: `zig-out/bin/native automate …`).

### Session and readiness

| Command | Behavior |
| --- | --- |
| `wait` | Poll until `ready=true` with a live publisher |
| `assert [--absent] [--timeout-ms 30000] <pattern>…` | Regex match against full `snapshot.txt` (poll 100ms); `--absent` inverts |
| `list` | Print `windows.txt` after liveness check |
| `snapshot` | Print `snapshot.txt` after liveness check |
| `record --out <journal> -- <app…>` | Launch app with `NATIVE_SDK_SESSION_RECORD` |
| `replay <journal> [--verify\|--no-verify] -- <app…>` | Launch with `NATIVE_SDK_SESSION_REPLAY` (+ verify flag) |

### Window, focus, chrome

| Command | Behavior |
| --- | --- |
| `reload` | WebView reload request |
| `resize <w> <h> [scale]` | Main-window resize (+ optional scale) |
| `menu-command <id>` | Menu command event |
| `native-command <id> [view-label]` | Native view command |
| `shortcut <id>` | Shortcut command |
| `tray-action <item-id>` | Status-item dropdown row (`tray-item #id` in snapshot) |
| `focus <view-label>` | Focus a view |
| `focus-next` / `focus-previous` | Cycle visible enabled views |
| `profile on\|off` | Toggle per-stage frame timing in snapshot |

### Widget driving

Widget verbs target a `gpu_surface` view by **label** and a widget by **bare numeric id** across all open windows.

| Command | Behavior |
| --- | --- |
| `widget-action <view> <id> <action> [value]` | Semantic actions (see below) |
| `widget-click <view> <id>` | Synthetic pointer down+up (one atomic gesture batch) |
| `widget-hold <view> <id>` | Press-and-hold: down → hold timer fire → suppressed release (`on_hold`) |
| `widget-context-press <view> <id>` | Secondary click; presents context menu or immediate `on_hold` |
| `widget-context-menu <view> <id> <item-index>` | Invoke declared menu item (0-based; skips OS tracking loop) |
| `widget-drag <view> <id> <sx> <ex> [sy ey]` | Pointer drag across ratios (default y = 0.5) |
| `widget-wheel <view> <id> <delta-y>` | Wheel on interactive/scrollable targets only |
| `widget-key <view> <key> [text]` | Key to focused widget; chords like `cmd+c`, `ctrl+shift+arrowleft` |

#### `widget-action` kinds

| Action | Notes |
| --- | --- |
| `focus` | Focus target |
| `press` | Enter-key press path |
| `toggle` | Space-key toggle path |
| `increment` / `decrement` | Step keys from widget |
| `set_text` / `set-text` | Real typing path: focus, select-all, text-input event (TEA `on_input` stays consistent) |
| `set_selection` / `set-selection` | Requires value |
| `set_composition` / `commit_composition` / `cancel_composition` | IME composition path (requires set_text support) |
| `select` | Selection target |
| `drag` | Value is drag delta |
| `drop_files` / `drop-files` | Requires path value |
| `dismiss` | Dismiss path |

Unsupported actions and bad targets land in the snapshot error ring (for example `automation.widget_wheel` with `WheelTargetUnknown`, `WheelTargetNotInteractive`, `WheelTargetHasEmptyBounds`).

### Screenshots

```bash
native automate screenshot <view-label> [scale]
```

Renders the named `gpu_surface` canvas through the **deterministic CPU reference renderer** (same pixel path as Linux software presentation) to:

```text
.zig-cache/native-sdk-automation/screenshot-<view-label>.png
```

Written as temp file + rename (presence implies complete PNG).

| Property | Behavior |
| --- | --- |
| Default scale | `1` (independent of display backing scale) |
| Same-machine determinism | Unchanged scene → byte-identical PNGs |
| Layout | Live retained scene + platform text measurement (e.g. CoreText on macOS) |
| Glyph raster | Bundled faces via reference renderer (not OS font rasterizer) |
| Cross-machine | Text widths can differ with OS text metrics; compare on one machine or assert dimensions/changedness |
| WebView pixels | **Not** captured |
| OS `screencapture` | Not a substitute — may silently return wallpaper without Screen Recording permission |

### Bridge

```bash
native automate bridge '{"id":"smoke","command":"native.ping","payload":{"source":"automation"}}'
```

Origin is `zero://inline`. The app bridge policy must allow that origin or the call fails with `permission_denied`.

| Bridge error | Meaning |
| --- | --- |
| `unknown_command` | No handler or wrong name |
| `permission_denied` | Origin/permission policy |
| `handler_failed` | Handler error or invalid JSON |
| `payload_too_large` | Exceeded bridge limits |

### Provenance and markup write-back

```bash
native automate provenance <view-label> <widget-id>
native automate provenance <view-label> at <x> <y>
```

Reports authored markup location into `provenance.txt`: file, byte span, line:column, template instantiation chain, iteration keys. Zig-builder widgets report `authored=zig` (not editable via write-back).

```bash
native automate edit <view> <id> set-attr <name> <value>
native automate edit <view> <id> remove-attr <name>
native automate edit <view> <id> set-text <text...>
```

Flow: provenance → refusal ladder → checked span edit → whole-closure validation → file write. Hot reload (when `markup_watch=armed`) picks up the change; no reload command required.

Refusals (file untouched):

- Widget not markup-authored
- Disk file hash differs from what the app loaded (concurrent edit guard)
- Edit fails markup validation or would break the import closure
- App not watching markup (`watching=true` required)

## Record and replay

Session journals capture platform events, effect results, checkpoints, and screenshot marks from **launch** (init-time effects must be recorded).

```bash
native automate record --out session.journal -- ./zig-out/bin/my-app
native automate replay session.journal --verify -- ./zig-out/bin/my-app
native automate replay session.journal --no-verify -- ./zig-out/bin/my-app
```

| Environment variable | Effect |
| --- | --- |
| `NATIVE_SDK_SESSION_RECORD` | Stream journal from first dispatched event |
| `NATIVE_SDK_SESSION_REPLAY` | Headless null-platform replay of the journal |
| `NATIVE_SDK_SESSION_VERIFY` | `1` (default) verify fingerprints/screenshots; `0` skip |

Replay refuses truncated journals (unclean exit leaves no end record). Child exit code propagates; verification failures exit non-zero with mismatch reports.

## Assertions

Prefer `assert` over `snapshot | grep` chains: it polls and prints missing patterns plus a snapshot tail on timeout.

```bash
native automate assert 'gpu_nonblank=true' 'role=button name="Reset"' 'count: 0'
native automate assert --timeout-ms 10000 '4 open'
native automate assert --absent 'error event=' 'dispatch_errors=[1-9]'
```

Regex subset: literals, `.`, postfix `* + ?`, anchors `^ $`, classes `[a-z]` / `[^0-9]`, `\d \w \s` (and uppercase negations). No groups or alternation — pass multiple patterns. Quote patterns in single quotes.

## Scope

**Can verify**

- Automation-enabled app reached `ready=true`
- App name, source kind/size, window metadata
- Widget roles, names, bounds, focus, selection, scroll, tray rows, audio mirror
- Bridge round-trips and builtin command dispatch
- Retained-canvas pixels via reference screenshots
- Deterministic session replay with checkpoint verification

**Cannot verify**

- WebView/DOM pixel capture or arbitrary DOM queries
- Browser network assertions
- Exhaustive UI coverage (smoke layer, not a full E2E suite)

## Troubleshooting

| Symptom | Check |
| --- | --- |
| `wait` times out with no snapshot | App running? `-Dautomation=true`? Runner passes `automation` into `Runtime.init`? Correct cwd? |
| `no automation dir at <path>` | CLI cwd differs from app launch cwd; dir is app-created only |
| Protocol mismatch | Rebuild stale CLI or app (`native version`); both must share protocol N |
| Stale instance warning | Publisher started before newest `zig-out/bin` binary; kill leftover and relaunch |
| Command seems ignored | Confirm `delivered …` line; queue full (max 8) retries then fails loudly |
| Bridge `permission_denied` | Allow origin `zero://inline` for smoke tests |
| Wheel/click failures | Aim at interactive widget ids from snapshot; read `error event=` lines |
| Screenshot empty expectations | Confirm `gpu_surface` view label; WebView content is out of scope |
| Edit refused | Need markup-authored widget, matching file hash, validation success, `markup_watch=armed` |

```bash
# Inspect live dropbox
cat .zig-cache/native-sdk-automation/snapshot.txt
# Extra runtime tracing
zig build run -Dautomation=true -Dtrace=all
```

## CI and smoke patterns

Repository examples wire automation into smoke steps, for example:

```bash
zig build test-webview-smoke -Dplatform=macos
zig build test-writeback-smoke   # provenance + edit loop (macOS)
```

A minimal smoke loop:

1. Build with `-Dautomation=true` (and `-Djs-bridge=true` when testing bridge)
2. Start the app in a GUI-capable session (or Xvfb on Linux)
3. `native automate wait`
4. `native automate assert` on critical patterns
5. Optional: `bridge`, `screenshot`, widget verbs
6. Fail on timeout, protocol mismatch, or unexpected response

Scaffolds from `native init --full` ship a CI workflow that pairs null-platform `zig build test` with a Linux Xvfb job using `wait` / `assert` / non-empty screenshot.

## Agent surface

The shipped skill pack at `skill-data/automation/SKILL.md` is discoverable via `native skills` and describes the same command vocabulary for agents verifying a running app. Automation is provider-neutral file IPC: any local agent that can run shell commands against the project cwd can drive it without a hosted connector.

## Related pages

<CardGroup>
  <Card title="Testing" href="/testing">
    Full-loop UI tests via native test, headless truth drivers, and frame-level verification.
  </Card>
  <Card title="Testing in CI" href="/testing-ci">
    Gate scripts, platform truth drivers, and reproducible headless/hosted checks.
  </Card>
  <Card title="CLI reference" href="/cli-reference">
    native automate flags, outputs, and failure modes alongside other CLI verbs.
  </Card>
  <Card title="Agent skills" href="/agent-skills">
    How skill packs (including automation) are discovered and applied.
  </Card>
  <Card title="Bridge and builtin commands" href="/bridge">
    Bridge payloads, origins, permissions, and handlers used by automate bridge.
  </Card>
  <Card title="Runtime and effects" href="/runtime-effects">
    Effect channel behavior that session record/replay journals and feeds.
  </Card>
</CardGroup>