# Testing in CI

> CI workflows, gate scripts, platform truth drivers, and eval harness wiring for reproducible headless and hosted checks.

- Repository: vercel-labs/native
- GitHub: https://github.com/vercel-labs/native
- Human docs: https://grok-wiki.com/public/docs/vercel-labs-native-8ccc3580636a
- Complete Markdown: https://grok-wiki.com/public/docs/vercel-labs-native-8ccc3580636a/llms-full.txt

## Source Files

- `.github/workflows/ci.yml`
- `docs/src/app/testing/ci/layout.tsx`
- `scripts/gate.sh`
- `tools/linux-truth/drive.sh`
- `evals/README.md`

---

---
title: "Testing in CI"
description: "CI workflows, gate scripts, platform truth drivers, and eval harness wiring for reproducible headless and hosted checks."
---

Native SDK continuous integration is defined by `.github/workflows/ci.yml` (PR and `main` pushes), the local tiered gate `scripts/gate.sh`, platform smoke scripts under `.github/scripts/`, containerized or SSH-based live-truth drivers in `tools/linux-truth` and `tools/windows-truth`, and the eval harness under `evals/` (typechecked in CI; model runs stay manual). Zig is pinned at **0.16.0** via `mlugg/setup-zig@v2`. Workflow permissions are `contents: read` on CI; separate workflows handle CEF packaging and releases.

## CI jobs

| Job | Runner | What it runs |
| --- | --- | --- |
| `zig` | `ubuntu-latest` | `zig build test`, `zig build validate` |
| `macos-webview` | `macos-14` | WebView system-link + smoke, package signing, GPU dashboard/components smokes |
| `macos-gpu-perf` | `macos-14` | `zig build test-gpu-dashboard-perf` (isolated from correctness smokes) |
| `linux-webkitgtk` | `ubuntu-latest` | `zig build test-webview-system-link -Dplatform=linux` (+ GTK/WebKitGTK apt deps) |
| `windows-webview` | `windows-2022` | `zig build test-webview-system-link -Dplatform=windows` |
| `cef-platform-tooling` | `ubuntu-latest` | `zig build test-tooling` |
| `npm-package` | `ubuntu-latest` | `version:check`, `scripts:check` under `packages/native-sdk` (Node 24) |
| `native-examples` | `ubuntu-latest` | `zig build test-examples-native` (+ GTK deps) |
| `linux-canvas-smoke` | `ubuntu-latest` | `.github/scripts/linux-canvas-smoke.sh` (Xvfb + WebKitGTK) |
| `windows-canvas-smoke` | `ubuntu-latest` (30 min) | `.github/scripts/windows-canvas-smoke.sh` (Wine + Xvfb + xdotool) |
| `windows-effects-smoke` | `ubuntu-latest` (30 min) | `.github/scripts/windows-effects-smoke.sh` (Wine effects path) |
| `frontend-examples` | `ubuntu-latest` | `zig build test-examples-frontends` |
| `mobile-examples` | `ubuntu-latest` | `zig build test-examples-mobile` |
| `scaffold` | `ubuntu-latest` | `native init` / `test` / `check` / `eject` + frontend scaffold YAML parse |
| `evals-typecheck` | `ubuntu-latest` | `pnpm typecheck` in `evals/` (Node 22, pnpm 10.23.0) |
| `docs` | `ubuntu-latest` | `pnpm check` in `docs/` |

```text
.github/workflows/
  ci.yml              # PR + main (table above)
  cef-runtime.yml     # workflow_dispatch: CEF multi-platform packaging
  release.yml         # maintainer release path
.github/scripts/
  linux-canvas-smoke.sh
  windows-canvas-smoke.sh
  windows-effects-smoke.sh
scripts/gate.sh       # local affected / full gate
tools/linux-truth/    # container Xvfb live-truth
tools/windows-truth/  # real Windows desktop live-truth (SSH)
evals/                # agent authoring harness (manual runs)
```

### macOS budgets in CI

Shared `macos-14` runners are noisier than a quiet dev box. CI **widens latency budgets without relaxing correctness assertions**:

| Variable | Used by | CI value | Local default (dev) |
| --- | --- | --- | --- |
| `NATIVE_SDK_SMOKE_BUDGET_MS` | GPU dashboard/components smokes | `1500` | `150` (first-frame / automation-ready ceiling) |
| `NATIVE_SDK_PERF_BUDGET_MS` | `test-gpu-dashboard-perf` | `1500` | `300` (p90 cold first-frame) |
| `NATIVE_SDK_PERF_INPUT_BUDGET_MS` | same | `500` | `100` (p90 steady-state input) |

`macos-gpu-perf` is a **separate job** so runner noise is visible in isolation and never blocks WebView/GPU correctness smokes. Perf harness details: `scripts/perf-gpu-dashboard.sh` (defaults: 5 cold launches, 5 interactions; p90 nearest-rank).

### Canvas and effects smokes

**Linux** (`.github/scripts/linux-canvas-smoke.sh`) builds `examples/ui-inbox` with `-Dplatform=linux -Dweb-engine=system -Dautomation=true`, runs under `xvfb-run`, and asserts:

1. `ready=true` (readiness budget **90 s** on shared runners — measured ~27 s stall after EGL init)
2. `gpu_nonblank=true` and `gpu_backend=software`
3. automation `widget-click` on “Add task” → snapshot contains `4 open`
4. non-empty `screenshot-inbox-canvas.png`

Env overrides used in CI/local containers:

- `WEBKIT_DISABLE_SANDBOX_THIS_IS_DANGEROUS=1` — bubblewrap needs unprivileged user namespaces; ubuntu-24.04 AppArmor otherwise kills the web process before a snapshot appears
- `GTK_A11Y=none` — avoids ~25 s GDBus a11y bus lookup under Xvfb with no session bus

**Windows canvas** (Wine on Ubuntu): same ui-inbox assertions plus real XTEST pointer/keyboard via `xdotool` (only coverage of Win32 pointer/keyboard mapping in the software path). Re-execs under Xvfb when `DISPLAY` is empty (`1280x800x24`).

**Windows effects** (Wine): drives `examples/effects-probe` — `fx.spawn` → streamed lines, `effects_wake` in the app log (not inferred from frame ticks alone), `fx.cancel` freezes the line count.

Failures dump snapshot + app log via script-local `fail()` / `diagnostics()` (scripts deliberately avoid bare `set -e` so grep misses do not silent-exit).

### Scaffold job

After `zig build`:

- Slim zero-config: `native init` → no `build.zig` / `build.zig.zon` → `native test -Dplatform=null` → `native check` → `native eject` → in-app `zig build test -Dplatform=null`
- Full frontends (`native`, `next`, `vite`, `react`, `svelte`, `vue`): `native init --frontend … --full` → `zig build test -Dplatform=null` → `native validate app.zon` → non-empty `.github/workflows/ci.yml` must parse as YAML

### Related workflows

- **`cef-runtime.yml`**: manual multi-platform CEF prepare/release (not on every PR).
- **`release.yml`**: maintainer packaging/distribution (see packaging and updates docs).

## Local gate (`scripts/gate.sh`)

The local gate mirrors CI-shaped checks without requiring every remote job. Every step runs even after failures; exit code is non-zero if any step failed. Steps are timed and summarized.

```bash
scripts/gate.sh fast [base-ref]                  # affected-only (default base: main)
scripts/gate.sh full [base-ref] [--all] [--perf] # full local CI-shaped set
```

Diff set: `git diff --name-only` against `git merge-base <base-ref> HEAD`, plus untracked files.

### Path → suite mapping (`fast`)

| Changed paths | Steps |
| --- | --- |
| `src/**`, `build.zig`, `build.zig.zon`, `build/**`, `tools/**`, `tests/**`, `assets/**` | Root `zig build test` + `validate`, **all** example suites (frontends, native, mobile), render-bench ratchet (macOS) |
| `src/platform/macos/**` | Above + `cef-host-link` (`test-webview-cef-link` / Chromium host compile+link; skips with warning if CEF layout missing) |
| `examples/<name>/**` | That example’s suite only (`test-example-<name>`, or in-dir `zig build test` for `browser`/`hello`/`webview`; mobile group → `test-examples-mobile`) |
| `docs/**` only | Docs `pnpm` install + `check` only |
| Other (README, `.github`, packages, scripts, skills, changelog…) | Root suites only |

Example suite names are **derived from** `addExampleTestStep("test-example-…")` in `build.zig` so the gate never drifts from the build graph.

### Full tier

Always: root test + validate, all example suites, CEF host link step, markup check over every `examples/**/*.native` file, render-bench ratchet on macOS.

macOS-only smokes: `test-gpu-surface-smoke`, `test-gpu-dashboard-smoke`, `test-gpu-components-smoke`, `test-webview-smoke`, `test-native-shell-smoke`, `test-canvas-preview-smoke`, `test-writeback-smoke`.

| Flag / env | Effect |
| --- | --- |
| `--perf` | `test-gpu-dashboard-perf` (macOS; opt-in, load-sensitive) |
| `--all` | Force docs check even if `docs/` unchanged |
| `NATIVE_SDK_SKIP_BENCH_CHECK=1` | Skip render-bench budget ratchet on a noisy box |

**Render-bench ratchet** (`bench-check`): `zig build bench-render -Doptimize=ReleaseFast -- --check tools/bench-render-budgets.txt`. Headless, in-process, calibrated on Apple Silicon; trips on order-of-magnitude regressions (~1.3× headroom), not machine noise. Runs by default on framework diffs (`fast`) and always on `full` (macOS).

Docs check installs with frozen lockfile and uses `NEXT_DIST_DIR=.next-gate` so it never corrupts a running docs dev server’s `.next` cache.

<Check>
Run `scripts/gate.sh fast` before finishing a change. It maps your diff to the suites that cover it; full CI still runs the remote matrix on push/PR.
</Check>

## Platform truth drivers

Live-truth loops build and drive **real** showcase binaries (not only `-Dplatform=null`) to record layout, input, resize clamps, multi-window, and screenshots.

### Linux (`tools/linux-truth`)

Container image bundles Zig, GTK4, WebKitGTK headers, and Xvfb. Repo mounts read-only at `/src`; builds go to volume `/work`; artifacts land in `/out`.

```bash
tools/linux-truth/run-all.sh [image|up|sync|recon|drive|suites|all]
```

| Step | Role |
| --- | --- |
| `image` / `up` | Build image; start long-lived container `native-sdk-linux-truth` |
| `sync` | rsync `/src` → `/work` (never writes back); refuses if container mounts a different checkout |
| `recon` | Build showcase apps (`-Dplatform=linux -Dweb-engine=system -Dautomation=true`), launch under Xvfb, dump snapshot/widgets/views + engine + X11 screenshots |
| `drive` | Per-app scenarios: clicks, set-text, wheel, resize, min-size clamp (`drive.sh`) |
| `suites` | `zig build test`, `validate`, `test-webview-system-link -Dplatform=linux`, `test-examples-native` |

Automation transport: apps with `-Dautomation=true` publish `.zig-cache/native-sdk-automation/snapshot.txt` and consume ordered `command-*.txt` entries (deleted on ack). Helpers live in `lib.sh` (`launch_app`, `send`, `widget_id`, engine `shot` vs X `xshot`).

Copy artifacts: `docker cp native-sdk-linux-truth:/out <dest>`.

### Windows (`tools/windows-truth`)

Real Windows 11 desktop over SSH (not Wine). Requires unlocked interactive desktop; GUI work hops through `schtasks /IT`. Artifacts under `%TEMP%\native-truth-out\`.

```powershell
powershell -NoProfile -File tools\windows-truth\run-all.ps1 [recon|drive|effects|record|writeback|package|all]
```

Steps cover recon, interaction drive, effects/clipboard, record/replay, markup write-back, and packaged-artifact launch — deeper than CI’s Wine smokes.

## Eval harness and CI boundary

`evals/` grades agent authoring of Native apps (scaffold + skill delivery + deterministic graders + optional LLM judge). **CI only typechecks** the harness (`evals-typecheck` → `pnpm typecheck`). Real eval runs are local/manual — no model calls in CI.

```bash
cd evals && pnpm install
pnpm typecheck                 # what CI runs
pnpm eval --list
pnpm eval --dry-run            # scaffold + graders, no model
pnpm eval --sandbox            # Linux lane in Vercel Sandbox microVMs
pnpm eval templates-settings-app
```

| Surface | Notes |
| --- | --- |
| Live snapshots | Default needs macOS GUI; use `--skip-live` or `--sandbox` elsewhere |
| Auth for real runs | `AI_GATEWAY_API_KEY` (gateway-routed agent); sandbox also needs Vercel OIDC or `VERCEL_TOKEN` + team/project IDs |
| Outputs | `results/<timestamp>/<case>/` (`result.json`, transcript, screenshots); workspaces under `.workspaces/` (gitignored) |
| Permissions | Default allowlist for `zig` / `native-sdk` / file tools; deny `evals/cases/**` and `evals/results/**` so agents cannot read grading configs |

Cases live under `evals/cases/<name>/eval.json`. Prompts describe requirements only; graders assert markup grammar, tests, and automation snapshot roles.

## App-generated CI (`native init --full`)

Scaffolds with `--full` (and web-frontend templates) write `.github/workflows/ci.yml` into the **app** repo. Zero-config slim scaffolds omit CI; copy recipes when needed.

Typical generated shape uses `NATIVE_SDK_PATH` (default `../native-sdk`) so `build.zig.zon` path deps resolve after cloning the framework onto the runner.

| Tier | Command / environment | Proves |
| --- | --- | --- |
| Logic | `zig build test -Dplatform=null` | Model/Msg/update, markup, layout without display |
| Linux smoke | build `-Dplatform=linux -Dautomation=true` under Xvfb + `native automate` | Real window, GPU surface, widget semantics |
| macOS | framework CI / optional local GUI runners | AppKit windows + same automate surface |

Deterministic screenshots: `native automate screenshot <view-label>` uses the CPU reference renderer at scale 1 (byte-identical for unchanged scenes on the same machine). Prefer goldens from the same runner image across OS versions.

```bash
rm -rf .zig-cache/native-sdk-automation   # never assert against a prior run
native automate wait
native automate assert 'gpu_nonblank=true' 'role=button name="Reset"'
native automate assert --absent 'error event=' 'dispatch_errors=[1-9]'
native automate screenshot main-canvas
```

`automate assert` polls until every regex matches (or `--absent` until all are gone). Default timeout 30000 ms; CI-friendly exit codes and snapshot tail on failure.

## Environment and knobs reference

| Variable | Where | Purpose |
| --- | --- | --- |
| `NATIVE_SDK_SMOKE_BUDGET_MS` | GPU smokes | First-frame / automation-ready latency budget (ms) |
| `NATIVE_SDK_PERF_BUDGET_MS` | GPU perf | p90 cold first-frame budget |
| `NATIVE_SDK_PERF_INPUT_BUDGET_MS` | GPU perf | p90 input latency budget |
| `NATIVE_SDK_PERF_LAUNCHES` / `NATIVE_SDK_PERF_INTERACTIONS` | perf script | Sample counts (default 5 / 5) |
| `NATIVE_SDK_SKIP_BENCH_CHECK` | `gate.sh` | Skip render-bench ratchet |
| `WEBKIT_DISABLE_SANDBOX_THIS_IS_DANGEROUS` | Linux canvas smoke | WebKit under restricted runners |
| `GTK_A11Y` | Linux canvas smoke | Avoid a11y bus stall under Xvfb |
| `WINEPREFIX` / `WINEDEBUG` | Wine smokes | Isolated prefix; quiet Wine logs |
| `AI_GATEWAY_API_KEY` | evals real runs | Model gateway |
| `NATIVE_SDK_EVAL_MODEL` / `NATIVE_SDK_EVAL_JUDGE_MODEL` | evals | Override coder/judge gateway slugs |
| `NATIVE_SDK_PATH` | generated app CI | Framework checkout relative path |

## Failure modes and verification signals

| Symptom | Likely cause | Signal |
| --- | --- | --- |
| Canvas smoke: snapshot never `ready=true` | Sandbox/a11y env, cold-start stall | Script diagnostics dump snapshot + app log; readiness 90 s on Linux CI |
| GPU smoke budget fail on CI only | Shared runner noise | Raise `NATIVE_SDK_*_BUDGET_MS` in **CI env only**; keep local defaults strict |
| Gate `cef-host-link` skipped with warning | No CEF layout on disk | `zig build && ./zig-out/bin/native cef install` |
| linux-truth sync refuses | Container mounts another checkout | Recreate with `run-all.sh up` from the intended tree |
| Wine smoke flaky input | Focus/window manager under Xvfb | windows-canvas dumps X window list + snapshot |
| evals CI green but authoring broken | CI never runs models | Run `pnpm eval` / `--sandbox` locally |
| Docs gate fails only after docs edit | Path-gated by design | `pnpm --dir docs check` or `gate.sh full --all` |

**Green signals:** gate summary `all steps green`; smoke scripts print `PASS: …`; CI job matrix green on PR; scaffold job leaves valid YAML workflows; `pnpm typecheck` clean under `evals/`.

## Next

<CardGroup>
  <Card title="Testing" href="/testing">
    Headless full-loop tests, TestHarness, and local smoke step names without the CI matrix.
  </Card>
  <Card title="Automation" href="/automation">
    Snapshot protocol, widget drive, screenshots, and `native automate` commands used by smokes and truth drivers.
  </Card>
  <Card title="Contributing" href="/contributing">
    Local setup, agent guidance, and when to run the gate before landing a change.
  </Card>
  <Card title="CLI reference" href="/cli-reference">
    `native test`, `check`, `automate`, and build flags used in scripts and scaffolds.
  </Card>
  <Card title="Platform support and security" href="/platform-security">
    Host capability matrix and how Linux/Windows live-truth relate to platform claims.
  </Card>
  <Card title="Web engines" href="/web-engines">
    System WebKitGTK, CEF host link checks, and Chromium install paths that CI and the gate touch.
  </Card>
</CardGroup>