# Codebase invariants, testscript stubs, and E2E suite

> Enforce single-owner and layer rules in internal/structure, write deterministic CLI testscript tests with internal/stubtest, and run real-VM E2E suites.

- Repository: devvmhq/devvm
- GitHub: https://github.com/devvmhq/devvm
- Human docs: https://grok-wiki.com/public/docs/devvmhq-devvm-4242986d2853
- Complete Markdown: https://grok-wiki.com/public/docs/devvmhq-devvm-4242986d2853/llms-full.txt

## Source Files

- `internal/structure/structure_test.go`
- `internal/structure/owners_test.go`
- `internal/structure/rules_test.go`
- `internal/cli/script_test.go`
- `internal/stubtest/stubtest.go`
- `tests/e2e/sandbox_test.go`
- `mise.toml`

---

---
title: "Codebase invariants, testscript stubs, and E2E suite"
description: "Enforce single-owner and layer rules in internal/structure, write deterministic CLI testscript tests with internal/stubtest, and run real-VM E2E suites."
---

`devvm` enforces architectural boundaries, single-owner concerns, CLI conventions, and runtime correctness across three verification tiers: static AST and linter gates (`internal/structure` and `.golangci.yml`), deterministic CLI integration scripts backed by a multi-call host binary stub (`internal/cli/script_test.go` and `internal/stubtest`), and a multi-slot real-VM end-to-end harness (`tests/e2e`).

## Verification gate and `mise` tasks

Every change must pass `mise run fmt && mise run lint && mise run test` before and after implementation. Changes that affect VM lifecycle, image builds, networking, mounts, or guest integration also run `mise run e2e` against real Lima VMs.

| Task | Command / Script | Behavior and Verification Scope |
| --- | --- | --- |
| `mise run fmt` | `golangci-lint fmt` | Formats Go packages with `gofumpt` and `goimports` (local prefix `github.com/devvmhq/devvm`). |
| `mise run lint` | Depends on `lint:code`, `lint:docs` | Runs all code, workflow, shell, and documentation checks. |
| `mise run lint:code` | `golangci-lint`, `actionlint`, `zizmor`, `shellcheck` | Runs Go linters, verifies every `forbidigo` rule fires on `testdata/lint/*.go`, checks GitHub workflows offline with `actionlint` and `zizmor --offline .`, and lints all repo shell scripts (excluding `testdata/script` and `.agents/skills/security-audit`) with `shellcheck --external-sources`. |
| `mise run lint:docs` | `markdownlint-cli2`, `vale`, `lychee`, `go test`, `hugo` | Lints Markdown and prose voice, checks offline links and fragments, runs `TestDocLinks` (`./internal/msg`) and `TestCommandLines` (`./internal/cli`), and builds the Hugo site with `--panicOnWarning`. |
| `mise run test` | `scripts/test.sh` | Runs unit tests, `internal/structure`, CLI `testscript` suites, and `TestHarness*` E2E runner unit tests with `-race -cover -timeout 30m` in a throwaway `HOME`, then merges unit and CLI coverage against `testdata/coverage/<os>.txt`. |
| `mise run coverage` | `scripts/test.sh --raise` | Runs the full test suite and updates `testdata/coverage/<os>.txt` for any package whose coverage increased, adding new packages and removing deleted ones. |
| `mise run vuln` | `govulncheck ./...` | Scans dependencies for known vulnerabilities. |
| `mise run deadcode` | `go run golang.org/x/tools/cmd/deadcode@... ./...` | Lists unreachable functions outside test entry points (`backendtest`, `stubtest`, `upgradetest`, `msg.IDOf`, `msg.NamesOf`, `msg.Recorder`, `msg.Fake`). |
| `mise run cli-docs` | `go run ./internal/cli/docgen docs/content/docs/cli` | Regenerates the CLI reference pages checked by `TestReferenceIsCurrent`. |
| `mise run e2e` | `go test -tags e2e -count=1 -timeout 0 -run '^TestE2E$' -args` | Runs real-VM end-to-end sections in `tests/e2e`. |

### Merged unit and CLI coverage baselines

`scripts/test.sh` sets `CGO_ENABLED=1` (required by `-race`), creates an isolated temporary `HOME` and XDG directory tree, unsets `SSH_AUTH_SOCK`, `NO_COLOR`, `FORCE_COLOR`, and `CLICOLOR_FORCE`, and exports `CLI_GOCOVERDIR=$cov/cli`.

When `CLI_GOCOVERDIR` is set, `TestScript` in `internal/cli/script_test.go` compiles the `devvm` binary with `-cover -covermode=atomic -coverpkg=github.com/devvmhq/devvm/...` and assigns each parallel `.txtar` script its own subdirectory (`$CLI_GOCOVERDIR/script.*`) as `GOCOVERDIR` so concurrent `devvm` invocations never race on Go's coverage metadata emission. After `gotestsum` finishes, `scripts/test.sh` merges `$cov/unit` and all non-empty `$cov/cli/*` directories with `go tool covdata merge` and passes `go tool covdata percent` output to `scripts/coverage.sh`:

- **Baseline check (`check` mode):** Compares each package's statement coverage against `testdata/coverage/darwin.txt` or `testdata/coverage/linux.txt`. Fails if any package drops more than `COVERAGE_TOLERANCE` (default `1` percentage point) below its baseline, if a new package has no baseline entry, or if a deleted package remains in the baseline.
- **Baseline update (`raise` mode):** Raises baseline percentages for packages that beat their recorded line, adds new packages, drops removed packages, and never lowers an existing baseline entry within the 1-point tolerance.

---

## Codebase invariants in `internal/structure` and `.golangci.yml`

`internal/structure` parses every non-generated `.go` file in the repository across all OS build tags (skipping hidden directories, `testdata`, `docs`, and `node_modules`) and enforces structural rules that standard linters cannot express. `TestStructure` checks the live repository against `devvm()`, while `TestRulesFail` in `internal/structure/rules_test.go` runs synthetic `fstest.MapFS` test cases against every rule and verifies that every layer in `devvm().layers` has a test case that breaks it.

```mermaid
flowchart TD
    subgraph Entry["Entry & CLI Layer"]
        CMD["cmd/devvm"]
        CLI["internal/cli/..."]
        UI["internal/ui"]
    end

    subgraph Core["VM & Domain Services"]
        VM["internal/vm"]
        DOMAIN["internal/build, tools, netpolicy, secrets,\nproject, gitprotect, hostsafety,\nsshconfig, sshagent, trust, vmssh"]
        AGENTLOGIN["internal/agentlogin"]
        DEVCONTAINER["internal/devcontainer"]
    end

    subgraph BackendLayer["Backend Abstraction & Drivers"]
        REG["internal/backend/registry"]
        BACKEND["internal/backend"]
        LIMA["internal/backend/lima"]
    end

    subgraph StateAndPaths["State, Images & Messages"]
        STATE["internal/state"]
        IMAGE["internal/image"]
        PATHS["internal/paths"]
        MSG["internal/msg"]
    end

    subgraph Standalone["Stdlib-Only & Leaf Packages"]
        PROXY["internal/proxy/..."]
        DAEMON["internal/daemon"]
        LEAVES["assets, locales, packaging, internal/platform,\ninternal/testbuild, internal/hostgit, internal/shell,\ninternal/vmname, internal/fetch, internal/jsonc, internal/redact"]
    end

    CMD --> CLI
    CMD --> UI
    CLI --> UI
    CLI --> VM
    CLI --> REG
    CLI --> LIMA
    VM --> DOMAIN
    VM --> REG
    VM --> BACKEND
    REG --> LIMA
    REG --> BACKEND
    DOMAIN --> BACKEND
    DOMAIN --> STATE
    AGENTLOGIN --> STATE
    AGENTLOGIN --> PROXY
    DEVCONTAINER --> IMAGE
    PROXY --> DAEMON
    STATE --> PATHS
    IMAGE --> PATHS
    PATHS --> MSG
    MSG --> LEAVES
```

### Structural rule categories

`check(m, devvm())` in `internal/structure/check_test.go` evaluates five rule categories and formats violations as `<rule>: <file>:<line>: <message>`:

| Rule ID | Invariant Enforced | Configuration in `devvm()` |
| --- | --- | --- |
| `max-methods` | Caps any struct or interface at `24` methods (value and pointer receivers combined; embedded interfaces count toward their own type, not the embedding interface). | `maxMethods: 24`, with explicit caps in `largeTypes` for `internal/paths.Dirs` (`58`) and `internal/ui.UI` (`46`). If a `largeTypes` entry shrinks below its cap or drops to `<= 24`, the check fails until `largeTypes` is lowered or the entry is removed. |
| `no-init` | Forbids `func init()` across all production and test files. Initialization happens in constructors or where used. | Checked unconditionally by `checkInit(m)`. |
| `package-vars` | Bans mutable package-level `var` declarations in production and test files. | Permits `var _`, `//go:embed`, `Err*`/`err*` sentinels from `errors.New`/`fmt.Errorf`, and `regexp.MustCompile`, `template.Must`, or `sync.Once*` calls. All other read-only tables, fixtures, and `-ldflags` targets must be listed in `allowedVars` with a reason (`ldflags`, `table`, `fixture`, `derived`). Stale entries fail the test. |
| `layers` | Enforces directional package import rules across non-test files (test files may wire packages together). | 18 `layer` rules with `from`, `exceptFrom`, `to`, and `exceptTo` package glob patterns (`...` and `pkg/...`). |
| `owners` | Ensures each critical operation, command, environment variable, path builder, string literal, separator, and state reader has a single owner in non-test files. | `commands`, `execSites`, `guardOwner`, `envOwner`, `envSettings`, `envOwners`, `literals`, `separators`, `pathsOwner`, and `callers`. |

### Package layer rules

The 18 layer rules in `internal/structure/structure_test.go` enforce strict upward and downward boundaries:

1. **UI ownership:** Only `cmd/devvm` and `internal/cli/...` may import `internal/ui`. All other packages emit messages, ask questions, and run logged commands through the `msg.Reporter` interface.
2. **VM orchestration:** Only `internal/cli/...` may import `internal/vm`, and only `cmd/devvm` may import `internal/cli/...`.
3. **Backend isolation:** `internal/backend/...` cannot import `internal/vm`, `internal/build`, `internal/tools`, `internal/netpolicy`, `internal/secrets`, `internal/project`, `internal/gitprotect`, `internal/hostsafety`, `internal/sshconfig`, `internal/sshagent`, `internal/trust`, or `internal/vmssh`. Only `internal/cli/...` and `internal/backend/registry` may import `internal/backend/lima`.
4. **Test-only packages:** `internal/backend/backendtest`, `internal/upgrade/upgradetest`, and `internal/stubtest/...` cannot be imported by non-test files outside their own trees.
5. **Stub isolation:** `internal/stubtest/...` may import only `internal/msg` and its own subpackages. `internal/stubtest/stub` and `internal/stubtest/stubstate` may import nothing from `devvm` except `internal/stubtest/stubstate`, because the `stub` binary is compiled as a standalone executable replacing host tools.
6. **Proxy and leaf boundaries:**
   - `internal/proxy/...` imports only the Go standard library and `internal/daemon` (enforced by both `structure` and `depguard`).
   - Leaf packages (`assets`, `locales`, `packaging`, `internal/platform`, `internal/msg/gen`, `internal/testbuild`, `internal/hostgit`, `internal/shell`, `internal/vmname`, `internal/daemon`, `internal/fetch`, `internal/jsonc`, `internal/redact`) import zero `devvm` packages.
   - `internal/agentlogin` imports only `internal/fetch`, `internal/jsonc`, `internal/proxy`, and `internal/state`.
   - `internal/msg` imports only `locales`.
   - `internal/paths` imports only `internal/msg` and `internal/vmname`.
   - `internal/state` imports only `internal/msg`, `internal/paths`, `internal/platform`, and `internal/vmname`.
   - `internal/image` imports only `assets`, `internal/msg`, and `internal/paths`.
   - `internal/devcontainer` imports only `internal/image`, `internal/jsonc`, `internal/msg`, and `internal/shell`.

### Single-owner checks (`owners` and `forbidigo`)

When a second copy of a parser, path builder, or system call would introduce drift or a security bug, `internal/structure/owners_test.go` and `forbidigo` in `.golangci.yml` enforce a single owner:

| Concern | Enforced By | Single Owner | Rule Details |
| --- | --- | --- | --- |
| Terminal output & process exit | `forbidigo` | `internal/ui` (`fmt.Print*`, `os.Stdout`, `os.Stderr`), `cmd/devvm` (`os.Exit`) | All packages return errors; only `internal/ui` writes to terminal streams and only `cmd/devvm` calls `os.Exit`. |
| Root context creation | `forbidigo` | `cmd/devvm`, `internal/ui` | Bans `context.Background` and `context.TODO` elsewhere unless annotated with `//nolint:forbidigo // <why>` for work that must outlive the caller. |
| Host hardware & free space | `forbidigo` | `internal/platform` (`platform.Host`) | Bans `runtime.NumCPU` and `unix`/`syscall` `Statfs`, `Fstatfs`, `Sysinfo`, `SysctlUint64` elsewhere so test builds can override host sizing. |
| Test build detection | `forbidigo` | `internal/cli` (`readOverrides` in `internal/cli/overrides.go`) | Bans `testbuild.On` outside `internal/cli`; other packages receive overrides as struct fields. |
| Process signaling by PID | `forbidigo` | `internal/daemon`, `internal/platform` | Bans `os.FindProcess`, `syscall.Kill`, `unix.Kill`, and `os.Process.Kill`/`Signal` elsewhere; `internal/daemon` verifies PID files and process arguments before signaling. |
| HTTP clients & transports | `forbidigo` | `internal/fetch` (`fetch.NewClient`) | Bans `http.Client`, `http.DefaultClient`, `http.Transport`, `http.Get`, etc. outside `internal/fetch` and stdlib-only `internal/proxy`. |
| Atomic file writes | `forbidigo` | `internal/state` (`state.WriteFile`, `state.WriteFileMode`) | Bans `os.WriteFile`, `os.Create`, and `os.O_TRUNC` outside `internal/state`, `internal/proxy` (`writeAtomic`), `internal/msg/gen`, and `internal/stubtest`. |
| Process environment mutation | `forbidigo` | `internal/daemon/status.go` (`daemon.Ready`) | Bans `os.Setenv`, `os.Unsetenv`, `os.Clearenv` (and `syscall` equivalents) outside tests so child processes never inherit mutated global env vars. |
| Host `git` execution | `structure` (`commands`) | `internal/hostgit` | Any `exec.Command` or `exec.CommandContext` invoking `"git"` (literal or constant) outside `internal/hostgit` fails. |
| `os/exec` construction sites | `structure` (`execSites`) | 19 allowlisted files in `devvmExecSites()` | Outside tests, only files listed in `execSites` may call `exec.Command` or `exec.CommandContext`. Removing the call without removing the file from `execSites` also fails. |
| Backend type assertions | `structure` (`guardOwner`) | `internal/backend/guard.go` (`backend.As`) | Bans type assertions and type switches on any type in `internal/backend` or `internal/backend/...` outside `guard.go` because registry backends are wrapped in a guard struct. |
| `HOME` & `DEVVM_*` env reads | `structure` (`envOwner`, `envOwners`, `envSettings`) & `forbidigo` | `internal/paths/dirs.go` (`HOME`), `internal/cli/overrides.go` (`DEVVM_*`) | Matches any call or field named `Getenv` or `LookupEnv` (case-insensitive). `HOME` is read only in `internal/paths/dirs.go`; `DEVVM_*` variables are read only in `internal/cli/overrides.go` or the exact files listed in `devvmEnvSettings()`. |
| Shell quoting & VM name regex | `structure` (`literals`) | `internal/shell` (`shell.Quote`), `internal/vmname` (`vmname.IsVM`) | Bans string literals containing `'\''` outside `internal/shell` and `[0-9a-f]{14}` outside `internal/vmname`. |
| Image layer separator | `structure` (`separators`) | `internal/image` (`image.Name`) | Bans `strings.*` calls taking `"+"` (literal or constant) outside `internal/image`. |
| Data & config folder paths | `structure` (`pathsOwner`) | `internal/paths` | Bans `filepath.Join`, `os.MkdirTemp`, or `os.CreateTemp` on a `.Data` or `.Config` selector outside `internal/paths`. |
| State, finder & agent store callers | `structure` (`callers`) | `internal/vm/finder.go`, `internal/vm/inventory.go`, `internal/vm/agents.go`, `internal/secrets/secrets.go` | Restricts external calls to `state.ReadHash` and `project.Parents` (`finder.go`), `state.ReadAll` (`inventory.go`), `paths.Dirs.AgentSettings` (`agents.go`), and `paths.Dirs.AgentLogins` (`secrets.go`). |

<Note>
Every `forbidigo` rule in `.golangci.yml` has a corresponding violation line ending with `// want forbidigo` in `testdata/lint/forbidigo.go`. During `mise run lint:code`, `golangci-lint run --enable-only forbidigo` runs against `./testdata/lint/` and fails unless the reported lines match the `// want forbidigo` annotations 1-to-1.
</Note>

### CLI rule and agreement tests

In addition to `internal/structure`, unit tests across `internal/cli`, `internal/image`, and `internal/msg` verify cross-cutting invariants:

- **Command tree and flag conventions (`internal/cli`):** `TestEveryCommandsHelp`, `TestPersistentFlags`, `TestFlagNames`, `TestFlagNamesHaveOneMeaning`, `TestDurationFlags`, `TestCommandNames`, `TestUseMatchesArgs`, `TestUnknownWord`, and `TestUsageErrorsEveryCommand` walk the Cobra command tree to enforce naming, help text, flag types, and exit code `2` on usage errors.
- **JSON output schemas (`internal/cli/json_test.go`):** `TestShowCommandsTakeJSON` verifies that every display command accepts `--json` (unless listed in `withoutJSON`), and `TestJSONShapes` locks every `"version": 1` field path and type so fields are never renamed or removed without bumping `jsonVersion`.
- **Command-line occurrences (`TestCommandLines`):** Scans all Markdown docs, `README.md`, `AGENTS.md`, `locales/en.toml`, Go comments/strings, `assets/`, `install.sh`, `.txtar` scripts, and skills to verify that every `devvm` command, subcommand, flag, and placeholder is valid.
- **Interactive prompts (`TestQuestionsDeclareAnswers`):** Scans non-test ASTs to ensure every `msg.Question` passed to `Ask`, `Confirm`, `Answer`, or `Secret` declares a valid flag/command via `AnsweredBy(...)` or a refusal message via `RefusedWith(...)`.
- **Shared tools and messages (`internal/image`, `internal/msg`):** `TestEveryPartHasAPolicy`, `TestSharedToolsPolicies`, `TestConfigTools`, `TestBuiltinPolicies`, `TestPreludeUsers`, `TestEveryMessageIsUsed`, `TestDocLinks`, and `TestReferenceIsCurrent` keep `image.SharedTools`, `assets/shared-tools/mise.toml`, `locales/en.toml`, `internal/msg/docs.go`, and `docs/content/docs/cli/` synchronized.

---

## Deterministic CLI tests (`TestScript` and `internal/stubtest`)

CLI integration tests live in `testdata/script/*.txtar` and are executed by `TestScript` in `internal/cli/script_test.go` using `github.com/rogpeppe/go-internal/testscript`.

Run a single testscript file by name:

```sh
mise exec -- go test ./internal/cli -run TestScript/<name>
```

### Test builds and environment overrides

`TestScript` (and `tests/e2e`) compiles `./cmd/devvm` with:

```sh
-ldflags "-X github.com/devvmhq/devvm/internal/testbuild.testBuild=yes"
```

Only binaries linked with `testBuild=yes` read the test override environment variables in `readOverrides` (`internal/cli/overrides.go`). Production release builds ignore all test override variables.

| Override Variable | Field in `cli.Overrides` | What It Replaces in Test Builds |
| --- | --- | --- |
| `DEVVM_HOST_MISE` | `HostMise` | Path to `devvm`'s pinned host `mise` binary (pointed at the `mise` stub in CLI tests). |
| `DEVVM_KVM` | `Overrides.KVM` | Path to Linux's `/dev/kvm` character device (pointed at `$STUB_DIR/kvm` in CLI tests). |
| `DEVVM_UBUNTU_URL` | `Overrides.UbuntuURL` | Base URL for Ubuntu cloud image releases (`file://$ROOT/ubuntu` in CLI tests, copied from `internal/ubuntu/testdata/resolute`). |
| `DEVVM_UBUNTU_RELEASE` | `Overrides.UbuntuRelease` | Pins a specific `release-YYYYMMDD` folder instead of selecting the newest release (`e2eUbuntuRelease` in E2E). |
| `DEVVM_UNREVIEWED` | `Overrides.Unreviewed` | Comma-separated `backend:feature` pairs allowed to run before emulation review completes. |
| `DEVVM_TEST_CPUS` | `Overrides.Host.CPUs` | Host CPU count read by `platform.Host` (default `8` in `TestScript`). |
| `DEVVM_TEST_MEMORY_GIB` | `Overrides.Host.MemoryGiB` | Host RAM in GiB read by `platform.Host` (default `16` in `TestScript`). |
| `DEVVM_TEST_FREE_GIB` | `Overrides.Host.FreeGiB` | Free disk space in GiB read by `platform.Host` (default `500` in `TestScript`). |
| `DEVVM_TEST_LIST_TIMEOUT` | `Overrides.Host.ListTimeout` | Timeout for backend instance listings (`10m` in `TestScript`). |
| `DEVVM_PIPED_ANSWERS` | `PipedAnswers` | When `1`, reads prompt answers line-by-line from non-TTY `stdin` instead of refusing with `no_input`. |
| `DEVVM_PICK_ON_PIPE` | `PickOnPipe` | When `1`, runs the interactive picker when only `stdin` is a TTY (used with `testscript`'s `ttyin`). |
| `DEVVM_HISTORY_MAX` | `HistoryMax` | Maximum byte size for agent history exports (replaces the 2 GiB cap). |
| `DEVVM_REPO_URL`, `DEVVM_RELEASE_KEY`, `DEVVM_RELEASE_CA`, `DEVVM_TEST_VERSION` | `Release.*` | Points `devvm upgrade` at the local TLS `upgradetest` server, custom Minisign public key, CA PEM file, and current binary version. |

### `TestScript` sandbox environment and custom commands

For each `.txtar` script, `sandbox()` creates `/tmp/dvt.*` (keeping Unix domain socket paths within `paths.SocketMax`), prepends the built `devvm` directory and the `stubtest` binary directory to `PATH`, pre-populates `$DATA/settings.json` with `{"features": []}`, and sets `DEVVM_MESSAGES=names` so assertions match message IDs and key-value parameters (`stderr '^devvm: flag_unknown command=devvm flag=--nope$'`) rather than prose wording.When a script ends, a deferred cleanup kills any background `_proxy`, `_ports`, `_links` daemons and `ssh -M` control masters tied to that sandbox root before removing `/tmp/dvt.*`.

| Environment Variable in `.txtar` | Sandbox Path / Value |
| --- | --- |
| `HOME` | `/tmp/dvt.<id>/home` |
| `XDG_DATA_HOME`, `DATA`, `VMS`, `RUN` | `/tmp/dvt.<id>/data`, `.../data/devvm`, `.../data/devvm/vms`, `.../data/devvm/run` |
| `XDG_CONFIG_HOME`, `CFG` | `/tmp/dvt.<id>/config`, `.../config/devvm` |
| `DEVVM_LIMA_HOME`, `LIMA_HOME` | `/tmp/dvt.<id>/lima` |
| `STUB_DIR`, `STUB_LOG` | `/tmp/dvt.<id>/stub`, `/tmp/dvt.<id>/stub/calls` |
| `DEVVM_SECRET_SERVICE` | `devvm-test` |
| `DEVVM_NO_RELEASE_CHECK` | `1` |
| `MISE_STUB_LOCK` | `<repo>/testdata/mise.lock` |
| `DEVVM_BIN`, `DEVVM_REPO` | Built `devvm` directory and repository checkout root |

`scriptCmds()` in `internal/cli/script_test.go` registers custom commands for `.txtar` scripts:

| Command | Syntax | Behavior |
| --- | --- | --- |
| `project` | `project NAME [git]` | Creates `$WORK/NAME`, `chdir`s into it, and runs `git init -q` when `git` is passed. |
| `vm_name` | `[!] vm_name VAR NAME` | Finds the single VM in `$VMS/*/vm.json` matching `^devvm-NAME-[0-9a-f]+$`, sets `$VAR` to the VM name, and sets `${VAR}_dir` to `$VMS/<hash>`. Negated (`!`), passes when the count is not `1`. |
| `vm_state` | `vm_state GLOB` | Reads the single `vm.json` matched by `GLOB` and prints sorted, flattened `KEY=VALUE` lines (`net.on=true`, `tools.names=node claude`, `links.links.0.name=db`). |
| `json_set` | `json_set GLOB KEY=VALUE...` | Modifies or creates JSON files matching `GLOB`. `KEY=VALUE` sets a string; `KEY:=JSON` sets a parsed JSON value; dotted keys (`net.on:=false`) traverse or create nested objects. |
| `catg` | `[!] catg GLOB` | Writes the single file matching `GLOB` to `stdout`; fails if `0` or `>1` files match (negated: fails if `1` file matches). |
| `stub_calls` | `[!] stub_calls` | Prints `$STUB_LOG` to `stdout`. Negated (`! stub_calls`), fails if any stub calls were logged. |
| `stub_reset` | `stub_reset` | Truncates `$STUB_LOG` and clears `testscript`'s `stdout` and `stderr` buffers. |
| `wait_file` | `wait_file FILE` | Polls every `50ms` up to `60s` (`waitFileTimeout`) for `FILE` to exist; fails immediately if all background commands have already exited. |
| `cp_exe` | `cp_exe SRC DST` | Copies an executable with mode `0755` while holding `syscall.ForkLock.RLock()`, preventing `ETXTBSY` (`text file busy`) on Linux when parallel scripts fork subprocesses. |
| `free_port` | `free_port VAR... \| free_port -range FIRST LAST` | Binds `:0` across all interfaces, closes the listener, and assigns unused TCP port numbers (or two consecutive ports with `-range`) to the named variables. |
| `listen_any` | `listen_any VAR` | Listens on `:0` for the lifetime of the script and sets `$VAR` to the bound port. |
| `release_server` | `release_server [-badsig] LATEST [VERSION...]` | Starts an in-process TLS release server (`upgradetest.Start`), signs `SHA256SUMS` with a test Minisign key (or a mismatched key with `-badsig`), and exports `DEVVM_REPO_URL`, `DEVVM_RELEASE_KEY`, and `DEVVM_RELEASE_CA`. |

### Multi-call stub binary (`internal/stubtest`)

`internal/stubtest/build.go` compiles `github.com/devvmhq/devvm/internal/stubtest/stub` once into `$TMPDIR/devvm-stubs-<hash>`, where `<hash>` is the first 16 hex characters of the SHA-256 over `runtime.Version()`, `GOOS/GOARCH`, and the non-test `.go` files in `internal/stubtest/stub` and `internal/stubtest/stubstate`. In the same folder, it creates symlinks to `stub` for every command returned by `stub names`, and prunes any `devvm-stubs-*` directories older than 24 hours.

```mermaid
sequenceDiagram
    participant TS as TestScript / Go Test
    participant Devvm as devvm (testBuild=yes)
    participant Stub as stub binary (argv[0] dispatch)
    participant State as $STUB_DIR/state.json (Flock)

    TS->>Stub: stub vm devvm-web-0123456789abcd Running
    Stub->>State: stubstate.Update(STUB_DIR) under state.lock
    TS->>TS: touch $STUB_DIR/start-fails (optional switch)
    TS->>Devvm: exec devvm ls
    Devvm->>Stub: PATH resolves limactl -> stub
    Stub->>Stub: Append "limactl list --json" to $STUB_LOG (O_APPEND)
    Stub->>State: stubstate.Read(STUB_DIR) under state.lock
    State-->>Stub: Instances, Keychain, Agent, Excluded
    Stub-->>Devvm: JSON instance list
    Devvm-->>TS: stdout / stderr (DEVVM_MESSAGES=names)
    TS->>TS: stub_calls -> verify $STUB_LOG
```

#### Stubbed commands and `stub` control subcommands

The `stub` binary dispatches on `filepath.Base(os.Args[0])` across 18 host commands plus the `stub` control interface:

- **Virtualization & containers:** `limactl`, `qemu-img`, `qemu-system-aarch64`, `qemu-system-x86_64`, `docker`
- **SSH & credentials:** `ssh`, `ssh-add`, `ssh-agent`, `ssh-keygen`, `security`
- **Toolchains, agents & editors:** `mise`, `opencode`, `code`, `open`
- **Host OS utilities:** `chflags`, `tmutil`, `sw_vers` (`STUB_SW_VERS`), `uname` (`STUB_UNAME_S`, `STUB_UNAME_M`)

Every stub invocation appends a single line (`<name> <args...>`) to `$STUB_LOG` in one `O_APPEND` write so concurrent calls never interleave and sensitive arguments (such as proxy URLs with passwords passed to `limactl edit`) are never forwarded to another child process's argv.

Persistent state across stub calls is stored in `$STUB_DIR/state.json`, protected by `syscall.Flock` on `$STUB_DIR/state.lock` (`internal/stubtest/stubstate`). Scripts inspect or mutate this state with `exec stub ...`:

```txtar
# Pre-populate a running Lima instance and inspect stub calls
exec stub vm devvm-app-0123456789abcd Running
exec stub size devvm-app-0123456789abcd 4 8589934592 107374182400
exec stub vms
stdout '^devvm-app-0123456789abcd Running$'
```

| `stub` Subcommand | Arguments | Effect on `$STUB_DIR/state.json` |
| --- | --- | --- |
| `stub names` | None | Prints all registered stub command names, sorted one per line. |
| `stub vm` | `NAME STATUS` | Creates or updates Lima instance `NAME` with `STATUS` (`Running`, `Stopped`, `Broken`) and default sizing (`4` CPUs, `8 GiB` RAM, `100 GiB` disk). |
| `stub vms` | None | Prints all non-empty Lima instances as sorted `NAME STATUS` lines. |
| `stub size` | `NAME CPUS MEMORY DISK` | Updates CPU count and memory/disk sizes (in bytes) for existing instance `NAME`. |
| `stub keychain` | `ACCOUNT` | Prints the stored secret for `ACCOUNT`, or exits `1` if missing. |
| `stub keychain-set` | `ACCOUNT` | Reads a secret value from `stdin` (trimming trailing newlines) and stores it under `ACCOUNT`. |
| `stub agent` | None | Prints keys held by the stubbed `ssh-agent` in `ssh-add -l` format, or exits `1` if no agent is running. |
| `stub agent-stop` | None | Clears the running `ssh-agent` state (`st.Agent = nil`). |

#### Fault-injection switches and guest simulation

Tests inject failures, hangs, or custom payloads by writing switch files into `$STUB_DIR`:

- **Switch naming conventions:** `<cmd>-fails`, `<cmd>-hangs`, `<cmd>-fails-on`, and `<cmd>-waits-on`. When a stub hits a `-hangs` or `-waits-on` switch, `c.hang()` touches `$STUB_DIR/hanging` (so `wait_file $STUB_DIR/hanging` can synchronize before sending `kill -INT`) and sleeps `30s`.
- **Guest shell simulation (`internal/stubtest/stub/guest.go`):** Both `limactl shell` and `ssh` route guest commands to `guest.go`. During VM setup and builds, the guest stub records tar streams into `$STUB_DIR/sync.tar` and `$STUB_DIR/packages-in.tar`, emits fake tool images or OCI disk artifacts, and supports targeted fault switches such as `$STUB_DIR/hostkeys-fails`, `$STUB_DIR/net-direct`, and `$STUB_DIR/interrupt-on`.
- **Real Lima contract and golden verification:** `TestLimactlStubContract` (`internal/backend/lima/contract_test.go`) runs an identical multi-step scenario against both the `limactl` stub and real `limactl` dummy instances, asserting identical `limactl list --json` state after every step. Golden fixtures in `testdata/golden/darwin/` and `testdata/golden/linux/` (captured via `mise run e2e -- --golden`) verify both `devvm`'s parsers and the stubs' output shapes in `internal/backend/lima/golden_test.go` and `internal/platform`.

### Using `stubtest` and `msg.Fake` in Go unit tests

Go unit tests use `stubtest.New(t, only...)` to place all stubs (or only the named command symlinks, leaving host tools like `git` intact) first on `PATH` with a per-test `STUB_DIR`:

```go
// Example unit test setup with stubtest.New and msg.Fake
s := stubtest.New(t, "limactl", "ssh")
s.VM(t, "devvm-web-0123456789abcd", "Running")
s.Touch(t, "start-fails")

var rec msg.Recorder
rep := &msg.Fake{OnWarn: rec.Add}
```

- `stubtest.ExecRunner{}` implements `backend.Runner` (`Run`, `Output`, `Attach`) by running `exec.Cmd` directly without a UI status line or run log.
- `msg.IDOf(err)` and `msg.NamesOf(err)` extract the message ID (`"no_vm"`) or `names`-mode string (`"no_vm dir=/path"`) from a returned error.

---

## Real-VM E2E test suite (`tests/e2e`)

`tests/e2e` (built only when `-tags e2e` is passed) runs `devvm` against real Lima VMs. `TestE2E` in `tests/e2e/main_test.go` requires `-timeout 0` so `go test`'s default 10-minute timeout never aborts a section mid-check.

<CodeGroup>
```sh title="Run via mise"
# Run default sections (all except desktop and from)
mise run e2e

# Run specific sections
mise run e2e -- --only images,gc

# Run all sections including --from and all base features across 3 parallel slots
mise run e2e -- --from --slots 3

# Keep reusable VMs running between iterations on a single section
mise run e2e -- --only project --reuse
```

```sh title="Run via go test directly"
# Run from tests/e2e without a package argument so go test streams output live
cd tests/e2e && CGO_ENABLED=0 go test -tags e2e -count=1 -timeout 0 -run '^TestE2E$' -args --only gc
```
</CodeGroup>

### E2E CLI flags

Flags are registered on `flag.CommandLine` in `tests/e2e/options_test.go` and passed after `--` (with `mise run e2e`) or `-args` (with `go test`):

<ParamField body="--only" type="string">
  Comma-separated list of sections to run (executed in canonical `sections()` order). Fails if an unknown section name is given.
</ParamField>

<ParamField body="--from" type="bool" default="false">
  Runs every section (including `desktop` and `from`) and enables `--every-feature` so base builds with all optional features in `assets/features/*.sh`.
</ParamField>

<ParamField body="--clean" type="bool" default="false">
  Deletes cached `devvm-img-*` Lima instances, `$DATA/devvm/images`, and `$DATA/devvm/tools` in the target sandbox before running.
</ParamField>

<ParamField body="--every-feature" type="bool" default="false">
  Writes `config.toml` with every feature in `assets/features/*.sh` enabled (uses the `features` timing tier). Automatically enabled when `--from` or `--only=from` is selected.
</ParamField>

<ParamField body="--newest-base" type="bool" default="false">
  Runs `images`, `project`, and `tools` in a dedicated `-newest` sandbox against Ubuntu's latest live cloud image and mirror packages instead of `e2eUbuntuRelease`, reporting upstream drift on failure. Cannot be combined with `--slots` or `--slot`.
</ParamField>

<ParamField body="--golden" type="bool" default="false">
  Captures anonymized golden tool outputs into `testdata/golden/<os>/` from `TestLimactlStubContract` and the `images` and `project` sections.
</ParamField>

<ParamField body="--reuse" type="bool" default="false">
  Retains VMs created via `s.Reuse(...)` at the end of the run and reuses them on subsequent runs if still `Running`.
</ParamField>

<ParamField body="--slots" type="int" default="1">
  Spreads the selected sections across `1` to `9` parallel slot sandboxes, seeding slots `1..N-1` from slot `0` before launching child `TestE2E` processes. Must be run from slot `0`.
</ParamField>

<ParamField body="--slot" type="int" default="0">
  Runs in slot `K`'s isolated sandbox (`0` to `9`), using that slot's cached images and port offset (`+100 * K`).
</ParamField>

### Sandbox isolation, locks, and multi-slot seeding

Each slot `K` (`0` by default) runs in an isolated sandbox (`tests/e2e/sandbox_test.go`) and refuses to start if its resolved `LIMA_HOME` matches `~/.lima` or `~/.local/share/devvm/lima`.

| Resource | Slot `0` | Slot `K` (`1..9`) | `--newest-base` |
| --- | --- | --- | --- |
| Lima home (`LIMA_HOME`, `DEVVM_LIMA_HOME`) | `~/.lt-e2e-go` (or `$DEVVM_E2E_LIMA_HOME`) | `~/.lt-e2e-go-K` | `~/.lt-e2e-go-newest` |
| State & work dir (`XDG_DATA_HOME`, `XDG_CONFIG_HOME`) | `/tmp/devvm-go-e2e` | `/tmp/devvm-go-e2e-K` | `/tmp/devvm-go-e2e-newest` |
| Keychain service (`DEVVM_SECRET_SERVICE`) | `devvm-go-e2e` | `devvm-go-e2e-K` | `devvm-go-e2e-newest` |
| PID symlink lock | `/tmp/devvm-go-e2e.lock` | `/tmp/devvm-go-e2e-K.lock` | `/tmp/devvm-go-e2e-newest.lock` |
| Fixed port offset (`sb.ports`) | `+0` | `+100 * K` | `+1000` |
| Timings directory (`sb.tdir`) | `/tmp/devvm-go-e2e/timings/<tier>` | `/tmp/devvm-go-e2e/timings/<tier>` (shared) | `/tmp/devvm-go-e2e/timings/<tier>-newest` |

Before building `devvm` into `$WORK/src-go/devvm`, the sandbox setup performs three checks:

1. **Host prerequisite check (`checkTools`):** Verifies `limactl`, `pkill`, and all tools declared by the selected sections (and ensures `openssl` on macOS is OpenSSL 3 rather than `/usr/bin/openssl` LibreSSL). Missing tools fail immediately with platform-specific `brew`, `apt-get`, or `dnf` install commands.
2. **Stale-safe PID lock (`takeLock`):** Creates an atomic symlink `/tmp/devvm-go-e2e[-K].lock -> <pid>`. If the symlink exists, `takeLock` checks `syscall.Kill(pid, 0)` and reclaims the lock if the previous process is no longer alive.
3. **Pinned tool lock verification (`pinLock`):** Verifies that the first line of `tests/e2e-mise.lock` matches `# devvm e2e: shared-tools/mise.toml sha256:<hex>` for the current `assets/shared-tools/mise.toml`. If `mise.toml` changed without running `scripts/bump.sh e2e-tools`, setup aborts immediately.

When `--slots N` (`N > 1`) is passed, `plan()` in `tests/e2e/slots_test.go` schedules sections longest-first (using recorded durations from `/tmp/devvm-go-e2e/timings/<tier>/` or the section's fallback `guess`), grouping `from` and `link` together because `link` in the `features` tier boots an Alpine client from the image built by `from`. Before spawning slot subprocesses, `seed()` clones slot `0`'s `devvm-img-*` Lima instances, EROFS tool images, caches, and cached `.deb` packages into slots `1..N-1` using copy-on-write APFS `clonefile` on macOS or reflinks on Linux.

### Check windows, heartbeats, and the check-count guard

`runner` (`tests/e2e/runner_test.go`) wraps every setup phase and `s.Check(...)` in a timed `window`:

- **Time limits (`limitOf`):** Each check gets `3x` its longest duration from the section's last complete run, with a minimum floor of `2m` (`limitFloor`), or `10m` (`limitDefault`) if no prior timing exists. Checks marked with `Builds()` and inter-check setup steps get at least `1h` (`limitBuild`).
- **Process-group termination:** Every command executed via `s.Run` runs in its own process group (`Setpgid: true`). When a window's deadline expires, the runner sends `SIGTERM` to the process group and follows with `SIGKILL` after `30s` (`killAfter`), recording the last heartbeat line in the failure output.
- **Heartbeat:** Every `1m`, a background goroutine prints `... <elapsed> since the last check: <lastLine>`, pulling the most recently modified line across `devvm`'s run logs and `$WORK/check.log`.
- **Check options (`Flaky`, `Builds`, `Needs`):**
  - `Flaky(reason)` retries a check once on failure, marking the output and timing record with `retried once`.
  - `Needs(backend.Feature...)` (or section-wide `s.Needing(...)`) queries `devvm doctor --json` once per run and records `SKIP` if the active backend reports `none` or `unreviewed` for a required feature.
- **Check-count guard (`sectionEnd`):** Each section writes its check timings to `timings/<tier>/<section>` and a SHA-256 hash of `tests/e2e/*.go` to `timings/<tier>/<section>.sum`. If a section completes with the same `tests/e2e/*.go` hash as the previous run but executes fewer checks, `sectionEnd` flags the run as `FAILED` (`<section> ran M checks, but its last complete run ran N: it ended early`) and preserves the previous timings file.

### E2E sections and `*S` check API

`sections()` in `tests/e2e/sections_test.go` defines the 10 E2E sections in execution order:

| Section | File | Default Run | Fallback Guess | Primary Verification Scope |
| --- | --- | --- | --- | --- |
| `images` | `tests/e2e/images_test.go` | Yes | `600s` | Base and custom layer builds, rebuilds, failed script cleanup, `SIGINT` during build, and offline replay from recorded `.deb` package sets. |
| `project` | `tests/e2e/project_test.go` | Yes | `900s` | Project VM launch, mounts, `ssh` config and host-key pinning, `.git` locking, `TestE2EExecQuoting`, `devcontainer.json`, `agent import`/`save`, and `ssh-agent` modes. |
| `clone` | `tests/e2e/clone_test.go` | Yes | `120s` | `devvm --clone` isolated repositories, branch syncing, and switching between mounted folders and clones. |
| `gc` | `tests/e2e/gc_test.go` | Yes | `120s` | `devvm gc` removal of stale layer images and unused shared tool images while preserving `img-*` project VMs. |
| `tools` | `tests/e2e/tools_test.go` | Yes | `600s` | Upstream tool resolution with host `mise`, read-only EROFS mounts for `node` and `dotnet`, global installs into `$HOME`, and `.nvmrc`/`global.json` overrides. |
| `network` | `tests/e2e/network_test.go` | Yes | `600s` | `allowlist`, `open`, `internet`, and `none` tiers, host `_proxy` TLS interception, per-VM CAs, secret placeholder injection, corporate proxy traversal, and `agent login` flows. |
| `desktop` | `tests/e2e/desktop_test.go` | `--from` / `--only` | `300s` | `desktop` layer VNC relay, authentication, clipboard filtering, and `TestE2EDesktop`. |
| `nix` | `tests/e2e/nix_test.go` | Yes | `900s` | `nix` layer build behind a corporate proxy, multi-user `nix-daemon` confinement, `flake.nix` + `direnv` activation, and blocked fixed-output derivations. |
| `from` | `tests/e2e/from_test.go` | `--from` / `--only` | `1800s` | OCI base conversion (`node:24` Debian glibc, `alpine:3` OpenRC musl, `ubuntu:26.04`) without a running Docker daemon, musl tool images, and layer compatibility. |
| `link` | `tests/e2e/link_test.go` | Yes | `900s` | Inter-VM links (`127.77.*.*`), `publish`/`unpublish`, 256-connection concurrency, `sshd` `PermitListen`/`GatewayPorts` hardening, and cross-tier enforcement. |

To add a check or a new section, call `s.Check` inside `<name>Section(s *S)`:

```go
func gcSection(s *S) {
	s.Check("gc deletes a tool image no VM uses, with its record", func() error {
		if Exists(old) || Exists(old+".json") {
			return errors.New("the old tool image or its record is still there")
		}
		return nil
	})
}
```

## Related pages

<CardGroup cols={2}>
  <Card title="Backend contract and Lima driver" href="/backend-architecture">
    Explore the four-role backend.Backend interface, capability guard checks, and the backendtest.Conformance suite.
  </Card>
  <Card title="VM lifecycle and state" href="/vm-lifecycle-and-state">
    Understand how vm.json sections, layout migrations in internal/state, and paths.Dirs ownership work.
  </Card>
  <Card title="Pinning, release signing, and supply-chain verification" href="/release-and-supply-chain-pipeline">
    Maintain SHA-256 download pins, e2e-mise.lock, e2eUbuntuRelease, and Minisign release signatures with scripts/bump.sh.
  </Card>
  <Card title="Configuration (config.toml) and --json schema reference" href="/configuration-and-json-reference">
    Review config.toml settings, environment variables, state directory layout, and version 1 --json schemas.
  </Card>
</CardGroup>