# Testing

> Unit tests (Vitest), Playwright Electron E2E, computer-use smoke scripts, and SSH-localhost E2E env flags documented in test helpers.

- Repository: stablyai/orca
- GitHub: https://github.com/stablyai/orca
- Human docs: https://grok-wiki.com/public/docs/stablyai-orca-2036d532bf1c
- Complete Markdown: https://grok-wiki.com/public/docs/stablyai-orca-2036d532bf1c/llms-full.txt

## Source Files

- `package.json`
- `config/vitest.config.ts`
- `tests/playwright.config.ts`
- `tests/e2e/helpers/orca-app.ts`
- `tests/e2e/ssh-localhost.spec.ts`
- `config/scripts/computer-use-smoke.mjs`
- `.env.e2e`

---

---
title: "Testing"
description: "Unit tests (Vitest), Playwright Electron E2E, computer-use smoke scripts, and SSH-localhost E2E env flags documented in test helpers."
---

Orca’s test surface spans Vitest unit tests under `src/` and `config/scripts/`, Playwright Electron E2E in `tests/e2e/` (built with `electron-vite --mode e2e` so `window.__store` is exposed), optional Vitest-driven `computer` CLI E2E, a `smoke:computer` script for desktop accessibility snapshots, and an opt-in localhost SSH spec gated by `ORCA_E2E_SSH_LOCALHOST`.

## Test layers

```text
pnpm test                    Vitest (Node) — src/**, config/scripts/**
       │
       ├─ PR CI (pr.yml)     lint, typecheck, pnpm test, pnpm build
       │
pnpm test:e2e                Playwright _electron.launch → out/main/index.js
       │                     globalSetup: electron-vite build --mode e2e + seeded git repo
       ├─ electron-headless  default; grepInvert @headful
       └─ electron-headful   ORCA_E2E_HEADFUL / @headful specs (pointer, WebGL)

pnpm test:e2e:computer       Vitest — tests/e2e/computer-*.e2e.ts (ORCA_COMPUTER_E2E=1)
pnpm smoke:computer          CLI smoke — config/scripts/computer-use-smoke.mjs

ORCA_E2E_SSH_LOCALHOST=1     ssh-localhost.spec.ts (+ build:relay in globalSetup)
```

<Note>
E2E is not part of the default PR verify job. It runs on a dedicated workflow (scheduled, `workflow_dispatch`, and from release cutting) with a separate build artifact and five-way sharding on Ubuntu.
</Note>

## Unit tests (Vitest)

| Item | Value |
| --- | --- |
| Command | `pnpm test` |
| Config | `config/vitest.config.ts` |
| Environment | `node` |
| Includes | `src/**/*.test.ts`, `src/**/*.test.tsx`, `config/scripts/**/*.test.mjs` |
| Pre-step | `ensure-native-runtime.mjs --runtime=node` (rebuilds `better-sqlite3` and `node-pty` for system Node) |

Vitest defines `ORCA_FEATURE_WALL_ENABLED: 'true'` and aliases `@renderer` / `@` to `src/renderer/src`. Tests live next to implementation (`src/main/`, `src/relay/`, renderer store slices, CLI paths, and similar).

PR CI (`/.github/workflows/pr.yml`) runs `pnpm rebuild better-sqlite3` before `pnpm test` because postinstall targets Electron’s ABI while Vitest executes under system Node.

### Renderer store slice tests

Pure Zustand slice behavior belongs in `src/renderer/src/store/slices/*.test.ts` (local `createTestStore()` helpers), not in Playwright specs. Reserve Electron E2E for DOM interaction, IPC, persistence, or multi-worktree lifecycle that unit tests cannot model faithfully.

## Playwright Electron E2E

| Script | Behavior |
| --- | --- |
| `pnpm run test:e2e` | `ensure:electron-runtime`, then Playwright project `electron-headless` |
| `pnpm run test:e2e:headful` | Same config, project `electron-headful` (`@headful` tests only) |
| `SKIP_BUILD=1 pnpm run test:e2e` | Reuse existing `out/` if `out/main/index.js` exists |

Config: `tests/playwright.config.ts` (`@stablyai/playwright-test`). Specs live in `tests/e2e/*.spec.ts`. Shared fixtures: `tests/e2e/helpers/orca-app.ts`.

### E2E build mode and `window.__store`

`.env.e2e` sets `VITE_EXPOSE_STORE=true`. That flag is applied only when the renderer is built with:

```bash
npx electron-vite build --mode e2e
```

`globalSetup` (`tests/e2e/global-setup.ts`) runs that build unless `SKIP_BUILD=1` and `out/main/index.js` already exist. A plain `pnpm build` / `pnpm build:electron-vite` **does not** expose the store; specs then hang on `waitForFunction(() => Boolean(window.__store))`.

Fast iteration:

<Steps>
<Step title="Build once for E2E">
Run `pnpm exec electron-vite build --mode e2e`.
</Step>
<Step title="Run tests without rebuild">
Run `SKIP_BUILD=1 pnpm run test:e2e` (optionally filter a file).
</Step>
</Steps>

### Lifecycle: setup, fixture, teardown

**globalSetup**

1. Builds Electron output (`--mode e2e`).
2. When `ORCA_E2E_SSH_LOCALHOST=1`, also runs `pnpm run build:relay` for localhost SSH relay deployment.
3. Creates a temp git repo (initial commit + secondary worktree on branch `e2e-secondary`) and writes its path to `orca-e2e-test-repo-path.txt` under the system temp directory.

**Per-test fixture** (`orca-app.ts`)

- Launches `_electron` against `out/main/index.js` with an isolated `ORCA_E2E_USER_DATA_DIR`.
- Seeds completed-onboarding `orca-data.json` by default (`dismissOnboarding: true`; `onboarding.spec.ts` can opt out).
- Strips `ELECTRON_RUN_AS_NODE` from the child env (host shells sometimes set it and break Playwright launch).
- Adds the seeded repo via `window.api.repos.add`, enables external worktree visibility, waits for `workspaceSessionReady`, and activates the primary worktree.
- Tears down: closes the app, cleans E2E daemons, removes userData.

**globalTeardown** removes the temp repo, sibling `orca-e2e-worktree-*` dirs, and the path file.

### Playwright projects and timing

| Setting | Value |
| --- | --- |
| `timeout` | 120_000 ms per test |
| `expect.timeout` | 10_000 ms |
| `fullyParallel` | `true` (headful drag/WebGL suites use `test.describe.serial` where needed) |
| `workers` | `2` on CI; default locally |
| `retries` | `0` |
| Trace / screenshot | `retain-on-failure` / `only-on-failure` |

| Project | `testMatch` | Filter |
| --- | --- | --- |
| `electron-headless` | `**/*.spec.ts` | `grepInvert: /@headful/` |
| `electron-headful` | `**/*.spec.ts` | `grep: /@headful/` |

Tag specs with `@headful` when they need a visible window, real pointer capture, or WebGL (for example terminal pane resize drags and dead-terminal reproduction). `ORCA_E2E_FORCE_HEADFUL=1` runs any spec headful without retagging.

On Linux CI, headless launch args disable GPU subprocess paths (`tests/e2e/helpers/electron-launch-args.ts`); runners use `xvfb-run` because Chromium still needs a display even when `ORCA_E2E_HEADLESS` suppresses `mainWindow.show()`.

### E2E authoring rules

From `tests/e2e/AGENTS.md`:

- Use `window.__store` and IPC for **setup** only.
- Final assertions must be **DOM-visible** (`getByRole`, `toBeVisible`, `toHaveText`, etc.).
- Do not assert store round-trips as a substitute for UI correctness.

Headless mode still drives the real DOM over CDP; store-only expectations miss render-layer regressions.

## E2E environment variables

Variables below are read in `tests/e2e/helpers/orca-app.ts`, `tests/e2e/global-setup.ts`, or `tests/e2e/ssh-localhost.spec.ts` unless noted.

### Harness and debugging

<ParamField body="SKIP_BUILD" type="string">
When set (any value) and `out/main/index.js` exists, `globalSetup` skips `electron-vite build --mode e2e`.
</ParamField>

<ParamField body="ORCA_E2E_USER_DATA_DIR" type="string">
Set by the fixture to a per-test temp dir. Main/preload read this for isolated persistence (`src/main/e2e-config.ts`, `src/preload/e2e-config.ts`).
</ParamField>

<ParamField body="ORCA_E2E_HEADLESS" type="string">
`1` suppresses showing the main window during headless project runs.
</ParamField>

<ParamField body="ORCA_E2E_HEADFUL" type="string">
`1` on headful project runs; also honored when `ORCA_E2E_FORCE_HEADFUL=1`.
</ParamField>

<ParamField body="ORCA_E2E_FORCE_HEADFUL" type="string">
`1` forces a visible window for any spec without switching Playwright projects.
</ParamField>

<ParamField body="ORCA_E2E_SLOWMO_MS" type="number">
Milliseconds of Playwright `slowMo` between actions (parsed at worker scope; invalid values log a warning and fall back to `0`).
</ParamField>

<ParamField body="ORCA_E2E_FORWARD_APP_LOGS" type="string">
`1` forwards Electron child stdout/stderr into the test runner (used on CI: `ORCA_E2E_FORWARD_APP_LOGS=1`).
</ParamField>

<ParamField body="ORCA_E2E_RECORD_VIDEO" type="string">
`1` records WebM under the test output directory (creates the dir first on Windows).
</ParamField>

### Localhost SSH E2E

<Warning>
`tests/e2e/ssh-localhost.spec.ts` is skipped unless `ORCA_E2E_SSH_LOCALHOST=1`. It requires a running local `sshd`, non-interactive key/agent auth, POSIX hook scripts, and (by default) remote agent hooks enabled. It is not run in standard CI shards.
</Warning>

| Variable | Role |
| --- | --- |
| `ORCA_E2E_SSH_LOCALHOST` | `1` enables the suite; triggers `build:relay` in `globalSetup` |
| `ORCA_E2E_SSH_HOST` | SSH host (default `127.0.0.1` when `ORCA_E2E_SSH_CONFIG_HOST` unset) |
| `ORCA_E2E_SSH_PORT` | Port (default `22`; validated 1–65535) |
| `ORCA_E2E_SSH_USER` | Username (falls back to `USER` / `USERNAME` / `os.userInfo()`) |
| `ORCA_E2E_SSH_CONFIG_HOST` | Use SSH config `Host` alias instead of explicit host |
| `ORCA_E2E_SSH_IDENTITY_FILE` | Optional identity file path |
| `ORCA_RELAY_PATH` | Relay bundle path; fixture sets `out/relay` when unset and SSH localhost mode is on |
| `ORCA_FEATURE_REMOTE_AGENT_HOOKS` | Skip when explicitly `0`; unset or non-empty non-`0` keeps remote hook tests enabled |

Example local run:

```bash
ORCA_E2E_SSH_LOCALHOST=1 pnpm run test:e2e tests/e2e/ssh-localhost.spec.ts
```

The spec connects to localhost SSH, registers a remote repo, exercises remote PTY output, agent-hook env vars (`ORCA_AGENT_HOOK_*`, `ORCA_PANE_KEY`), Codex hook POST relay, and keyboard interrupt behavior.

## Computer-use tests

Two complementary paths exercise desktop accessibility through the `orca computer` CLI.

### Smoke script (`pnpm smoke:computer`)

`config/scripts/computer-use-smoke.mjs` requires a built CLI at `out/cli/index.js` (`pnpm build:cli` first).

| Flag / env | Purpose |
| --- | --- |
| `--apps` | Comma-separated app names (default from `ORCA_COMPUTER_SMOKE_APPS` or Finder, TextEdit, Spotify, Slack, Microsoft Edge) |
| `--session` | Session id (default `computer-smoke-<pid>`) |
| `--screenshot` | Include screenshots in `get-app-state` |
| `ORCA_COMPUTER_SMOKE_USER_DATA_PATH` | Overrides dev userData (`orca-dev` per platform) |

The script lists running apps, snapshots preferred targets that are actually running, prints element/line counts, and exits `0` when no preferred apps are open (not a failure). Failures increment a counter and exit `1`.

### Vitest computer E2E (`pnpm test:e2e:computer`)

Config: `tests/e2e/vitest.config.ts` — includes `tests/e2e/computer-*.e2e.ts`, 60s test/hook timeouts, Node environment.

| File | Platform gate |
| --- | --- |
| `computer-mac.e2e.ts` | `darwin` + `ORCA_COMPUTER_E2E=1` |
| `computer-linux.e2e.ts` | Linux + `ORCA_COMPUTER_E2E=1` (+ `ACCESSIBILITY_ENABLED=1` on scheduled CI) |
| `computer-windows.e2e.ts` | Windows + `ORCA_COMPUTER_E2E=1` |

Helpers in `tests/e2e/helpers/computer-driver.ts` invoke `config/scripts/orca-dev` (or `out/cli/index.js` on Windows) unless `ORCA_COMPUTER_CLI` overrides the command. macOS specs drive TextEdit (`list-apps`, `get-app-state`, `click`, `type-text`, `paste-text`, hotkeys).

Workflow `.github/workflows/computer-e2e.yml`:

- **PR**: `verify:computer-native` + `build:cli` smoke on Ubuntu/Windows matrix only.
- **Scheduled / manual**: full platform E2E on macOS 14, Ubuntu 22.04 (xvfb + dbus), and Windows with `ORCA_COMPUTER_E2E=1`.

## CI summary

| Workflow | What runs |
| --- | --- |
| `pr.yml` → `verify` | `pnpm test` (Vitest), typecheck, lint, full `pnpm build` |
| `e2e.yml` | Separate build job uploads `out/`; five shards run `SKIP_BUILD=1 ORCA_E2E_FORWARD_APP_LOGS=1 pnpm run test:e2e` under `xvfb-run`; failed shards upload `test-results/` traces |
| `computer-e2e.yml` | Native verify smoke on PR; platform computer Vitest on schedule/manual |
| `release-cut.yml` | Calls `e2e.yml` before release steps |

## Common failures

| Symptom | Likely cause | Fix |
| --- | --- | --- |
| All E2E hang at `window.__store` | `out/` built without `--mode e2e` | `pnpm exec electron-vite build --mode e2e` or full `pnpm run test:e2e` without `SKIP_BUILD` |
| Vitest fails loading `better-sqlite3` | Native module built for Electron ABI | `pnpm rebuild better-sqlite3` (PR CI does this automatically) |
| Linux E2E Chromium startup errors | GPU under Xvfb | Already mitigated via launch args + xvfb; check `ORCA_E2E_FORWARD_APP_LOGS=1` logs |
| SSH localhost skipped | Gate env unset | `ORCA_E2E_SSH_LOCALHOST=1` + local `sshd` + keys |
| Computer E2E skipped | Opt-in unset | `ORCA_COMPUTER_E2E=1` and platform-specific prerequisites |

<Tip>
Filter a single spec during development: `SKIP_BUILD=1 pnpm run test:e2e tests/e2e/worktree.spec.ts`. For headful debugging: `ORCA_E2E_FORCE_HEADFUL=1 ORCA_E2E_SLOWMO_MS=250 pnpm run test:e2e:headful tests/e2e/terminal-panes.spec.ts`.
</Tip>

## Related pages

<CardGroup>
<Card title="Develop and build" href="/develop-and-build">
Native runtime rebuilds, `electron-vite` build graph, and `build:computer-macos` artifacts that computer tests depend on.
</Card>
<Card title="Contributing" href="/contributing">
PR expectations, cross-platform constraints, and maintainer workflows that trigger E2E and release checks.
</Card>
<Card title="SSH remote worktrees" href="/ssh-remote-worktrees">
Product behavior exercised by the localhost SSH E2E spec (targets, relay, remote PTY, agent hooks).
</Card>
<Card title="CLI browser and computer reference" href="/cli-browser-computer-reference">
`orca computer` commands used by smoke and Vitest computer E2E.
</Card>
<Card title="Troubleshooting" href="/troubleshooting">
Runtime reachability, native dependency, and platform-specific failure modes that overlap with test setup.
</Card>
</CardGroup>