# Build System, Runtime Images & Testing

> How the full repo is built with Bazel and rules_zig, how glibc runtime SquashFS images are packaged and selected via the libc platform property, how standalone binaries embed compressed kernel and initramfs payloads, and the layered testing strategy: unit tests, Docker Linux e2e, VM e2e, and the LLVM tblgen smoke benchmark.

- Repository: hermeticbuild/actiond
- GitHub: https://github.com/hermeticbuild/actiond
- Human wiki: https://grok-wiki.com/public/wiki/hermeticbuild-actiond-796c0ee40e63
- Complete Markdown: https://grok-wiki.com/public/wiki/hermeticbuild-actiond-796c0ee40e63/llms-full.txt

## Source Files

- `MODULE.bazel`
- `BUILD.bazel`
- `vm/BUILD.bazel`
- `runtimes/BUILD.bazel`
- `runtimes/glibc_runtime_repo.bzl`
- `tools/e2e.sh`
- `e2e/run_llvm_vm_smoke.sh`
- `e2e/llvm_tblgen_smoke.sh`

---

<details>
<summary>Relevant source files</summary>
The following files were used as context for generating this wiki page:

- [MODULE.bazel](MODULE.bazel)
- [BUILD.bazel](BUILD.bazel)
- [vm/BUILD.bazel](vm/BUILD.bazel)
- [runtimes/BUILD.bazel](runtimes/BUILD.bazel)
- [runtimes/glibc_runtime_repo.bzl](runtimes/glibc_runtime_repo.bzl)
- [src/BUILD.bazel](src/BUILD.bazel)
- [src/embedded_payload.zig](src/embedded_payload.zig)
- [cmd/darwin_actiond/BUILD.bazel](cmd/darwin_actiond/BUILD.bazel)
- [cmd/linux_actiond/BUILD.bazel](cmd/linux_actiond/BUILD.bazel)
- [cmd/linux_actiond_guest/BUILD.bazel](cmd/linux_actiond_guest/BUILD.bazel)
- [platforms/BUILD.bazel](platforms/BUILD.bazel)
- [e2e/BUILD.bazel](e2e/BUILD.bazel)
- [test/BUILD.bazel](test/BUILD.bazel)
- [tools/e2e.sh](tools/e2e.sh)
- [e2e/llvm_tblgen_smoke.sh](e2e/llvm_tblgen_smoke.sh)
- [e2e/run_llvm_vm_smoke.sh](e2e/run_llvm_vm_smoke.sh)
- [.bazelrc](.bazelrc)
</details>

# Build System, Runtime Images & Testing

This page explains how `hermeticbuild/actiond` is built with Bazel and `rules_zig`, how glibc runtime SquashFS images are assembled and selected at execution time, how standalone binaries embed compressed payloads, and the full layered testing strategy from unit tests through the LLVM tblgen VM smoke benchmark.

Understanding these layers together is important because a change in any one area—kernel config, runtime image content, or embedded payload format—affects all three others. The build, packaging, and test systems are tightly coupled around the same artifacts.

---

## Build System

### Bazel with rules_zig

The entire repository is built with Bazel. The primary language is Zig, configured via [`rules_zig`](https://github.com/aherrmann/rules_zig). The Zig toolchain is pinned to version `0.16.0` and is declared to require the `//platforms:local_execution` constraint, ensuring Zig compile actions are never sent to a remote executor.

```python
# MODULE.bazel
bazel_dep(name = "rules_zig", version = "0.15.1")

zig.toolchain(
    extra_exec_compatible_with = ["//platforms:local_execution"],
    zig_version = "0.16.0",
)
```

The `//platforms:local_execution_platform` platform carries `exec_properties = {"no-remote-exec": "1"}` and is registered as an execution platform.

Sources: [MODULE.bazel:3-6](), [platforms/BUILD.bazel:21-31]()

### Core Bazel dependencies

| Dependency | Purpose |
|---|---|
| `rules_zig` | Zig toolchain, `zig_binary`, `zig_library`, `zig_test` |
| `linux.bzl` | VM kernel build (`linux()` macro, compact pre-built repos) |
| `llvm` | `llvm-objcopy`, `llvm-strip`, `llvm-tblgen` and LLVM toolchains |
| `zstd` | Zstandard compression (patched to remove `pthread` dependency) |
| `apple_support` / Obj-C | macOS `Virtualization.framework` bridge in `cmd/darwin_actiond` |
| `bazel_lib` | `platform_transition_binary` for cross-compiling guest binary |
| `rules_cc` / `rules_shell` | C/ObjC libraries, shell test rules |

Sources: [MODULE.bazel:3-12]()

### Remote config

`.bazelrc` defines a `--config=remote` flag that sets `--jobs=500` and adds `@llvm//:rbe_platform` as an extra execution platform. Large builds—especially kernel and LLVM packages—are recommended to use this flag:

```
common:remote --jobs=500
common:remote --extra_execution_platforms=@llvm//:rbe_platform
```

Sources: [.bazelrc:5-6]()

---

## VM Kernel

The Linux kernel is built by `linux.bzl` from source archive `linux-6.18.2` declared in `MODULE.bazel`. A custom kernel filesystem module, `actiondfs` (located at `kernel/actiondfs/`), is compiled in via `linux_kernel.extra_source`. The kernel uses `allnoconfig` with a custom `//vm:linux.config` to minimize size.

```python
# vm/BUILD.bazel
linux(
    name = "linux_kernel",
    compact_repos = {"aarch64": "@actiond_vm_compact"},
    config = "@actiond_vm_kconfig//:kconfig",
    config_mode = "allnoconfig",
    extra_kconfigs = {"//kernel/actiondfs:Kconfig": "fs/actiondfs"},
    extra_srcs = ["//kernel/actiondfs:srcs"],
    image_format = "Image",
    source_repo = "@linux_6_18_2",
)
```

The raw `Image` is then zstd-compressed to produce `//vm:linux_kernel_zst`. A pre-built compact repo (`@actiond_vm_compact`) is provided so that the kernel does not need to be built from source in typical CI runs.

### initramfs

The VM initramfs is built at `//vm:initramfs`. It is a `cpio.zst` archive containing a single statically-linked aarch64 binary—the guest agent `linux-actiond-guest-aarch64`—produced from `cmd/linux_actiond_guest`. The binary is cross-compiled to `//platforms:linux_aarch64` via `platform_transition_binary` and stripped with `llvm-strip`.

Sources: [vm/BUILD.bazel:1-42](), [cmd/linux_actiond_guest/BUILD.bazel]()

---

## Glibc Runtime SquashFS Images

### Repository rule: `glibc_deb_runtime`

Each versioned glibc runtime is declared using the `glibc_deb_runtime` repository rule from `runtimes/glibc_runtime_repo.bzl`. The rule:

1. Downloads `.deb` packages from Ubuntu's package archives (supporting `.xz`, `.zst`, and `.gz` payload compression).
2. Extracts `data.tar.*` into a `root/` directory.
3. Strips documentation, locales, man pages, and gconv tables to minimize image size.
4. Writes a `runtime_manifest.json` describing the runtime's name, architecture, source `.deb` URLs, mount points, and ELF interpreter paths.

```json
// runtime_manifest.json (generated)
{
  "name": "glibc2.35",
  "arch": "aarch64",
  "debs": ["https://ports.ubuntu.com/...libc6_2.35-..._arm64.deb"],
  "mounts": [
    ["root/lib", "/lib"],
    ["root/lib64", "/lib64"],
    ["root/usr/lib", "/usr/lib"],
    ["root/etc", "/etc"]
  ],
  "interpreters": ["/lib/ld-linux-aarch64.so.1"]
}
```

Sources: [runtimes/glibc_runtime_repo.bzl:5-75]()

### Declared runtime versions

Six glibc runtimes are declared in `MODULE.bazel`:

| Name | glibc | Arch | Source |
|---|---|---|---|
| `glibc_2_31_aarch64` | 2.31 | aarch64 | Ubuntu Ports (focal) |
| `glibc_2_31_x86_64` | 2.31 | x86_64 | Ubuntu Archive (focal) |
| `glibc_2_35_aarch64` | 2.35 | aarch64 | Ubuntu Ports (jammy) |
| `glibc_2_35_x86_64` | 2.35 | x86_64 | Ubuntu Archive (jammy) |
| `glibc_2_39_aarch64` | 2.39 | aarch64 | Ubuntu Ports (noble) |
| `glibc_2_39_x86_64` | 2.39 | x86_64 | Ubuntu Archive (noble) |

Sources: [MODULE.bazel:93-148]()

### SquashFS packaging and architecture selection

`runtimes/BUILD.bazel` defines two `genrule` targets—`runtimes_squashfs_aarch64` and `runtimes_squashfs_x86_64`—that pack all three versioned runtimes for their respective architecture into a single SquashFS image using `//tools:sqfs_pack`. The image directory layout is:

```text
runtimes-tree/
├── common/
│   └── root/
│       ├── etc/hosts
│       └── etc/nsswitch.conf
└── libc/
    ├── glibc2.31/<arch>/root/  + runtime_manifest.json
    ├── glibc2.35/<arch>/root/  + runtime_manifest.json
    └── glibc2.39/<arch>/root/  + runtime_manifest.json
```

The public `//runtimes:runtimes_squashfs` alias uses a `select()` to pick the correct architecture-specific image:

```python
# runtimes/BUILD.bazel
alias(
    name = "runtimes_squashfs",
    actual = select({
        ":target_aarch64": ":runtimes_squashfs_aarch64",
        ":target_arm64":   ":runtimes_squashfs_aarch64",
        ":target_x86_64":  ":runtimes_squashfs_x86_64",
        "//conditions:default": ":runtimes_squashfs_aarch64",
    }),
)
```

Sources: [runtimes/BUILD.bazel:1-30]()

### libc platform property

Clients select the desired glibc version at action dispatch time via the `libc` exec property. The stress e2e workspace uses:

```
--remote_default_exec_properties=libc=glibc2.35
```

The server reads this property, locates the corresponding subtree within the mounted SquashFS image, and applies the overlay mounts from `runtime_manifest.json` before executing the action.

Sources: [tools/e2e.sh:152-153]()

---

## Standalone Binary Packaging

For deployment without separate runtime files, `actiond` supports **standalone** binaries that have the kernel, initramfs, and runtimes SquashFS embedded directly inside the executable. Extraction logic lives in `src/embedded_payload.zig`.

### Darwin (macOS/Mach-O)

The `darwin-standalone-payload-sections` `cc_library` uses `-Wl,-sectcreate` linker flags to inject three Mach-O sections into the `__ACTIOND` segment:

```python
# cmd/darwin_actiond/BUILD.bazel
cc_library(
    name = "darwin-standalone-payload-sections",
    linkopts = [
        "-Wl,-sectcreate,__ACTIOND,__kernel,$(location //vm:linux_kernel_zst)",
        "-Wl,-sectcreate,__ACTIOND,__initramfs,$(location //vm:initramfs)",
        "-Wl,-sectcreate,__ACTIOND,__runtimes,$(location //runtimes:runtimes_squashfs_aarch64)",
    ],
    ...
)
```

The resulting binary is codesigned with `darwin-actiond.entitlements`.

Sources: [cmd/darwin_actiond/BUILD.bazel:43-64]()

### Linux (ELF)

The Linux standalone binary uses `llvm-objcopy` to inject the runtimes SquashFS as an ELF section named `.actiond.runtimes`:

```python
# cmd/linux_actiond/BUILD.bazel
"$(execpath @llvm//tools:llvm-objcopy)" \
  --add-section .actiond.runtimes="$(location //runtimes:runtimes_squashfs)" "$@"
```

Sources: [cmd/linux_actiond/BUILD.bazel:13-26]()

### Payload extraction (`src/embedded_payload.zig`)

At runtime, `embedded_payload.zig` reads the binary's own on-disk format:
- **Mach-O**: walks `LC_SEGMENT_64` commands to find sections in `__ACTIOND` by name.
- **ELF**: parses the section header table to locate `.actiond.runtimes`.

Extracted payloads are written to `<root>/embedded/<name>-<sha256hex>` with content-addressed deduplication: if a file of the correct size already exists at that path, extraction is skipped.

```
const mach_o_segment_name  = "__ACTIOND";
const mach_o_kernel_section = "__kernel";
const mach_o_initramfs_section = "__initramfs";
const mach_o_runtimes_section  = "__runtimes";
const elf_runtimes_section = ".actiond.runtimes";
```

Sources: [src/embedded_payload.zig:10-21](), [src/embedded_payload.zig:43-105]()

---

## Artifact Relationships

```text
MODULE.bazel (glibc_deb_runtime declarations)
        │
        ▼
@glibc_2_3{1,5,9}_{aarch64,x86_64}
  .deb download → root/ tree + runtime_manifest.json
        │
        ▼
runtimes/BUILD.bazel → runtimes_squashfs_{aarch64,x86_64}.sqfs
                                  │
               ┌──────────────────┼───────────────────────────┐
               │                  │                            │
        vm/BUILD.bazel      cmd/linux_actiond           cmd/darwin_actiond
         (vm_bundle)         (standalone: ELF section)   (standalone: Mach-O section)
          ├── linux_kernel_zst
          ├── initramfs.cpio.zst
          └── runtimes_squashfs
```

---

## Testing Strategy

### Layer 1: Unit tests

Unit tests cover the Zig library source directly. The `zig_test` target in `src/BUILD.bazel` shares the same source list as the `zig_library`:

```
bazel test //src:unit_tests
```

`embedded_payload.zig` contains inline tests for Mach-O extraction, ELF extraction, and the null case where a binary has no embedded payloads.

Sources: [src/BUILD.bazel:30-63](), [src/embedded_payload.zig:262-410]()

### Layer 2: Repository build checks

`tools/e2e.sh build` runs the full build and test sweep:

```bash
run_bazel test //src:unit_tests
run_bazel build //cmd/linux_actiond_guest:linux-actiond-guest-aarch64
run_bazel build //vm:initramfs
run_bazel build //runtimes:runtimes_squashfs
run_bazel build //vm:linux_kernel --nobuild
run_bazel build //tools:e2e_action_tool_linux_{aarch64,x86_64}
run_bazel build //...
run_bazel test //...
```

Sources: [tools/e2e.sh:175-184]()

### Layer 3: Docker Linux e2e

For macOS developers who need to validate the Linux code path, `tools/docker/run_linux_e2e.sh` wraps the `linux` mode inside a Docker container. `tools/e2e.sh linux` itself requires a Linux host.

### Layer 4: Stress workspace e2e

The `test/` directory is a standalone Bazel workspace. The `stress_workload` macro generates:
- Bare file inputs (default 160)
- Source directory inputs (8 directories × 32 files)
- Nested individual file inputs (8 groups × 96 files)
- Multiple actions sharing the same output directory (to measure tree-artifact reuse)

The e2e harness builds a platform-specific `e2e_action_tool` binary, copies it into `test/tool/action-tool`, then executes:

```bash
bazel build //:stress_all \
  --remote_executor="grpc://${endpoint}" \
  --remote_cache="grpc://${endpoint}" \
  --remote_default_exec_properties=libc=glibc2.35 \
  --noremote_accept_cached \
  --spawn_strategy=remote \
  --genrule_strategy=remote
```

Sources: [tools/e2e.sh:143-170](), [test/BUILD.bazel]()

#### Linux mode

Starts `linux-actiond serve --listen=... --root=... --runtime-image=<sqfs>` on the local Linux host, then runs the stress workspace against it. Standalone mode passes no `--runtime-image`; the binary self-extracts.

Sources: [tools/e2e.sh:186-230]()

#### VM mode

macOS-only. Creates an ext4 disk image for the CAS (`//server/cas.ext4`), then starts:

```bash
darwin-actiond serve-vm \
  --listen=... --root=... \
  --kernel=Image.zst --initramfs=initramfs.cpio.zst \
  --runtime-image=runtimes.sqfs \
  --cas-image=cas.ext4 \
  --memory-mib=1024 --cpus=4
```

Waits up to 90 seconds for the port to become ready, then runs the stress workspace.

Sources: [tools/e2e.sh:232-284]()

### Layer 5: LLVM tblgen smoke benchmark

The most realistic and performance-significant test builds `@llvm-project//llvm:llvm-tblgen` (≈5,341 configured actions) via the VM executor. The workflow is split into two phases to isolate actiondfs performance:

```
Phase 1: Warmup  (--noremote_accept_cached)
  Build //e2e:llvm_exec_warmup  [wraps llvm-min-tblgen; ~2,403 actions]
  → populates the VM CAS and action cache

Phase 2: Measured  (--remote_accept_cached)
  Build @llvm-project//llvm:llvm-tblgen
  → only the incremental actions beyond warmup are executed from scratch
```

All builds use `@llvm//platforms:linux_arm64_musl` as both target and host platform, producing Linux aarch64 musl binaries. This avoids glibc runtime actions and allows exec tools to run inside the VM guest.

`--noremote_cache_compression` is required because actiond does not yet implement remote cache compression.

```bash
# e2e/llvm_tblgen_smoke.sh (inner smoke)
bazel build @llvm-project//llvm:llvm-tblgen \
  --platforms=@llvm//platforms:linux_arm64_musl \
  --host_platform=@llvm//platforms:linux_arm64_musl \
  --remote_executor=grpc://127.0.0.1:8998 \
  --noremote_cache_compression \
  --noremote_accept_cached \
  --spawn_strategy=remote
```

Sources: [e2e/llvm_tblgen_smoke.sh:30-73]()

#### run_llvm_vm_smoke.sh

`e2e/run_llvm_vm_smoke.sh` automates the full before/after cycle:
1. Builds the `darwin-actiond-standalone` server package with `--config=remote`.
2. Creates a fresh ext4 CAS image.
3. Starts the VM and waits for the guest to be ready (polls `actiondfs_stats.txt`).
4. Runs `llvm_tblgen_smoke.sh` and records the measured server log slice.
5. Parses timing data via `test/parse_timings.py` and writes a Markdown summary.
6. Optionally runs a mac-host local baseline (same target, no remote executor) for comparison.
7. Records the output root at `/tmp/actiond-last-llvm-vm-smoke-path`.

Sources: [e2e/run_llvm_vm_smoke.sh:1-55](), [e2e/run_llvm_vm_smoke.sh:57-130]()

### Testing summary

| Level | Command | What it exercises |
|---|---|---|
| Unit tests | `bazel test //src:unit_tests` | Zig library correctness, payload extraction |
| Build checks | `tools/e2e.sh build` | All targets build; no broken deps |
| Linux e2e | `tools/docker/run_linux_e2e.sh` | Linux chroot executor + runtime mount |
| VM e2e | `tools/e2e.sh vm` | Virtualization.framework VM, vsock, actiondfs, CAS |
| Standalone e2e | `ACTIOND_E2E_STANDALONE=1 tools/e2e.sh vm` | Self-extracting payload at startup |
| LLVM smoke | `e2e/run_llvm_vm_smoke.sh` | End-to-end REAPI performance against real LLVM build |

---

## Closing Summary

The actiond build pipeline is organized as a single Bazel workspace that produces kernel, initramfs, runtime images, and host-side server binaries from hermetic, content-addressed inputs. Glibc runtimes are downloaded from Ubuntu `.deb` archives, packed into SquashFS images with per-version directory trees, and selected at build time by CPU architecture and at execution time by the `libc` exec property. Standalone binaries carry all three payloads (kernel, initramfs, runtimes) as native binary sections—Mach-O `__ACTIOND` segment on macOS, `.actiond.runtimes` ELF section on Linux—extracted on first run to a content-addressed cache within the server root. The testing strategy layers Zig unit tests, full repo build validation, Docker-wrapped Linux e2e, macOS VM e2e (using Virtualization.framework), and a two-phase LLVM tblgen smoke benchmark that serves as the canonical performance regression gate.