# Native Layer — Rust Crates & N-API Addon
> The ~27k-line Rust underbelly: pi-natives (N-API cdylib), pi-shell (embedded bash/PTY), pi-ast (tree-sitter summarizer), pi-iso (workspace isolation), and how each module eliminates fork/exec on the hot path.
- Repository: can1357/oh-my-pi
- GitHub: https://github.com/can1357/oh-my-pi
- Human wiki: https://grok-wiki.com/public/wiki/can1357-oh-my-pi-64b0ce1ccc45
- Complete Markdown: https://grok-wiki.com/public/wiki/can1357-oh-my-pi-64b0ce1ccc45/llms-full.txt
## Source Files
- `crates/pi-natives/Cargo.toml`
- `crates/pi-natives/build.rs`
- `crates/pi-shell/Cargo.toml`
- `crates/pi-ast/Cargo.toml`
- `crates/pi-iso/Cargo.toml`
- `packages/natives`
---
Relevant source files
The following files were used as context for generating this wiki page:
- [crates/pi-natives/Cargo.toml](crates/pi-natives/Cargo.toml)
- [crates/pi-natives/build.rs](crates/pi-natives/build.rs)
- [crates/pi-natives/src/lib.rs](crates/pi-natives/src/lib.rs)
- [crates/pi-natives/src/shell.rs](crates/pi-natives/src/shell.rs)
- [crates/pi-natives/src/pty.rs](crates/pi-natives/src/pty.rs)
- [crates/pi-natives/src/grep.rs](crates/pi-natives/src/grep.rs)
- [crates/pi-shell/Cargo.toml](crates/pi-shell/Cargo.toml)
- [crates/pi-shell/src/lib.rs](crates/pi-shell/src/lib.rs)
- [crates/pi-shell/src/shell.rs](crates/pi-shell/src/shell.rs)
- [crates/pi-shell/src/minimizer.rs](crates/pi-shell/src/minimizer.rs)
- [crates/pi-shell/src/minimizer/engine.rs](crates/pi-shell/src/minimizer/engine.rs)
- [crates/pi-ast/Cargo.toml](crates/pi-ast/Cargo.toml)
- [crates/pi-ast/src/lib.rs](crates/pi-ast/src/lib.rs)
- [crates/pi-ast/src/summary.rs](crates/pi-ast/src/summary.rs)
- [crates/pi-ast/src/language/mod.rs](crates/pi-ast/src/language/mod.rs)
- [crates/pi-iso/Cargo.toml](crates/pi-iso/Cargo.toml)
- [crates/pi-iso/src/lib.rs](crates/pi-iso/src/lib.rs)
- [crates/pi-iso/src/apfs.rs](crates/pi-iso/src/apfs.rs)
- [crates/pi-iso/src/overlayfs.rs](crates/pi-iso/src/overlayfs.rs)
- [packages/natives/native/index.js](packages/natives/native/index.js)
- [packages/natives/native/loader-state.js](packages/natives/native/loader-state.js)
# Native Layer — Rust Crates & N-API Addon
The oh-my-pi toolchain offloads performance-critical work to a Rust layer that surfaces as a single compiled N-API addon (`pi_natives.*.node`) loaded by the Node/Bun JS runtime. Rather than shelling out to `grep`, `bash`, or filesystem utilities on the hot path, the JS frontend calls directly into the in-process Rust binary, eliminating fork/exec latency for operations that happen hundreds of times per session.
This page covers the four Rust library crates that make up the native layer — **pi-natives** (the N-API cdylib), **pi-shell** (embedded bash interpreter), **pi-ast** (tree-sitter code summarizer), and **pi-iso** (workspace isolation PAL) — plus the JS loader in `packages/natives` that validates and attaches the compiled addon at startup.
---
## Crate Structure at a Glance
```text
┌─────────────────────────────────────────────────────────────┐
│ packages/natives (JS/TS) │
│ native/index.js ──loadNative()── loader-state.js │
└─────────────────────────┬───────────────────────────────────┘
│ require("pi_natives..node")
▼
┌─────────────────────────────────────────────────────────────┐
│ crates/pi-natives (cdylib) │
│ appearance · ast · clipboard · fd · glob · grep · html │
│ highlight · iso · keys · power · ps · pty · shell · text │
│ sixel · summary · task · tokens · workspace │
└────┬──────────────┬──────────────┬───────────────────────── ┘
│ │ │
▼ ▼ ▼
crates/pi-shell crates/pi-ast crates/pi-iso
(brush embed) (tree-sitter) (CoW / overlay)
```
Each inner crate has no `[lib] crate-type` override — they are normal `rlib` dependencies. Only `pi-natives` declares `crate-type = ["cdylib"]`, making it the single N-API boundary.
Sources: [crates/pi-natives/Cargo.toml:9-10]()
---
## pi-natives — The N-API cdylib
### Module Inventory
`pi-natives` re-exports every capability through a flat module tree. The full list from `src/lib.rs` is:
| Module | Responsibility |
|---|---|
| `appearance` | macOS dark-mode observation |
| `ast` | `ast-grep` structural search/edit |
| `clipboard` | Read/write system clipboard (arboard) |
| `fd` | File-descriptor utilities |
| `fs_cache` | Filesystem scan cache (`DashMap`-backed) |
| `glob` | Glob file discovery (globset + ignore) |
| `grep` | Ripgrep-backed content search |
| `highlight` | Syntax highlighting (syntect) |
| `html` | HTML-to-Markdown conversion |
| `keys` | Terminal key-sequence parsing |
| `iso` | N-API shim for `pi-iso` backends |
| `power` | macOS power-assert / display-sleep prevention |
| `prof` | Flame-graph profiling (inferno) |
| `ps` | Process listing |
| `pty` | PTY-backed interactive sessions (portable-pty) |
| `shell` | Brush-based shell execution shim |
| `sixel` | SIXEL image encoding for terminal display |
| `summary` | Code structural summary shim for `pi-ast` |
| `task` | Cancellation / abort token wiring for N-API |
| `text` | ANSI-aware width measurement and wrapping |
| `tokens` | Tiktoken token counting |
| `workspace` | Workspace listing |
Sources: [crates/pi-natives/src/lib.rs:24-50]()
### Version Sentinel
The addon exports a single zero-cost function whose JS name encodes the package version:
```rust
#[napi(js_name = "__piNativesV15_2_1")]
pub const fn pi_natives_version_sentinel() {}
```
The JS loader checks for the presence of `__piNativesV{major}_{minor}_{patch}` before handing the binding to callers. If the name is absent (e.g., a locked Windows file left a stale `.node` from a previous install), the loader surfaces a clear load-time error instead of a silent ` is not a function` crash. The sentinel name is bumped by `scripts/release.ts` on every release.
Sources: [crates/pi-natives/src/lib.rs:54-71]()
### Build Script
`build.rs` does two things: calls `napi_build::setup()` to wire N-API header generation, and concatenates all per-tool filter definition TOML files under `src/shell/minimizer/defs/*.toml` into a single `OUT_DIR/builtin_filters.toml` that gets embedded at compile time. Each `.toml` file is watched independently so only relevant filter changes trigger a rebuild.
Sources: [crates/pi-natives/build.rs:8-64]()
### JS Loader (`packages/natives`)
The loader in `packages/natives/native/loader-state.js` resolves which `.node` file to load through several stages:
1. **Compiled-binary detection**: checks for embedded addon (Bun standalone binaries), `PI_COMPILED` env var, or bunfs URL patterns.
2. **Variant selection**: on x64, tries `pi_natives.-modern.node` (AVX2), then `pi_natives.-baseline.node`, then the default.
3. **Windows staging**: on Windows, mirrors the `.node` from `node_modules` to a version-pinned path in `~/.omp/natives//` so concurrent processes never lock each other's file.
4. **Sentinel validation**: requires the version-matching export to exist before returning the binding.
Supported platform tags are: `linux-x64`, `linux-arm64`, `darwin-x64`, `darwin-arm64`, `win32-x64`.
Sources: [packages/natives/native/loader-state.js:33-148]()
---
## pi-shell — Embedded Bash Interpreter
### Design: No Fork/Exec for Shell Builtins
`pi-shell` embeds the [brush](https://github.com/reubeno/brush) Bash-compatible interpreter as a Rust library. Shell commands are executed by calling into `BrushShell::run_string` in-process rather than forking `/bin/bash`. This removes the process-creation round trip from the hot path for short shell commands and enables reliable cancellation.
The public API surface (`src/lib.rs`) re-exports `Shell`, `execute_shell`, `execute_shell_streams`, `ShellOptions`, `ShellRunOptions`, and `ShellRunResult`.
Sources: [crates/pi-shell/src/lib.rs:1-13]()
### Session Model
`Shell` holds a single `Arc>>`. The first `Shell::run` call boots a fresh `BrushShell` via `create_session` and caches it. Subsequent calls reuse the same session (state, environment, working directory). The session is evicted when the command's `ExecutionControlFlow` is not `Normal` (e.g., `ExitShell`).
```rust
pub struct Shell {
session: Arc>>,
abort_state: ShellAbortState,
config: ShellConfig,
}
```
Sources: [crates/pi-shell/src/shell.rs:123-127](), [crates/pi-shell/src/shell.rs:1154-1162]()
### Session Initialization
`create_session` builds a brush shell with no profile or rc loading (`ProfileLoadBehavior::Skip`, `RcLoadBehavior::Skip`), copies the parent-process environment while explicitly filtering out shell-internals that would pollute the embedded shell (`BASH_FUNC_*%%`, `PS1`, `HISTFILE`, `SHLVL`, and about 40 others), disables the `exec` and `suspend` builtins, and registers custom `sleep` and `timeout` builtins that honour the Tokio cancellation token. On Windows, it additionally deduplicates PATH entries.
Sources: [crates/pi-shell/src/shell.rs:471-551](), [crates/pi-shell/src/shell.rs:1096-1152]()
### Output Streaming and Cancellation
For each command, `run_shell_command` creates an `os_pipe` and redirects both stdout and stderr through it. A concurrent Tokio task drains the pipe, emitting UTF-8-decoded chunks to an `mpsc::UnboundedSender`. On Unix, the pipe file descriptor is set to `O_NONBLOCK` and wrapped in a `tokio::io::unix::AsyncFd` so the drain loop can park on readiness without blocking the runtime thread.
Cancellation is cooperative at three levels:
1. A `CancellationToken` passed into `BrushShell::run_string` stops the brush interpreter.
2. A cancel bridge task wakes when the token fires and cancels the reader.
3. A process-cancel bridge issues SIGTERM (wave 0, 75 ms grace), then SIGKILL (waves 1-2, 150 ms) to any descendant PIDs that the brush interpreter spawned as external processes.
Sources: [crates/pi-shell/src/shell.rs:596-681]()
### Output Minimizer
The minimizer is an opt-in post-processor that rewrites verbose command output before it reaches the JS caller. The engine is inert unless a `MinimizerConfig` is supplied; compound or piped commands are always passed through unchanged.
```rust
pub enum MinimizerMode {
None,
WholeCommand,
}
```
For single commands, `engine::mode_for` runs the detector and, if a registered filter matches, switches to `WholeCommand` mode: the output is buffered (up to `max_capture_bytes`, default 4 MiB) and the matching filter rewrites it after the command exits. The original text is returned alongside the minimized text so the JS session layer can persist the original via its `ArtifactManager`.
Built-in filter definitions are `.toml` files under `src/shell/minimizer/defs/` that get concatenated and embedded at build time. Filters exist for `git`, `cargo`, `bun`, `go`, `docker`, `gh`, Python, Ruby, lint tools, cloud CLIs, and more.
Sources: [crates/pi-shell/src/minimizer.rs:1-6](), [crates/pi-shell/src/minimizer/engine.rs:17-43]()
### PTY Sessions
When the JS caller needs a full interactive terminal (e.g., for commands that require a TTY), `pi-natives/src/pty.rs` creates a `portable_pty` session instead. The `PtySession` N-API class exposes `start`, `writeInput`, `resize`, and `kill`. Output is streamed via a N-API threadsafe function. Control messages (input, resize, kill) are delivered through a `std::sync::mpsc` channel to a dedicated reader thread.
Sources: [crates/pi-natives/src/pty.rs:1-80]()
---
## pi-ast — Tree-sitter Code Summarizer
### Language Registry
`pi-ast` bundles over 50 tree-sitter grammars as workspace dependencies (Rust, TypeScript, Python, Go, Java, C/C++, Kotlin, Swift, Dart, Zig, Nix, Haskell, TLA+, Odin, Verilog, and many others). Each grammar is exposed through the `SupportLang` enum. The `language/mod.rs` module implements the `ast-grep-core` `Language` and `LanguageExt` traits for every entry using a macro that keeps the boilerplate minimal.
Sources: [crates/pi-ast/Cargo.toml:18-75](), [crates/pi-ast/src/language/mod.rs:20-45]()
### Structural Summarization
`pi-ast::summary::summarize_code` is the main entry point used by the `summarizeCode` N-API export. Given source text and an optional language hint or file path, it:
1. Resolves the tree-sitter grammar from the language alias or file extension.
2. Parses the source and short-circuits to a passthrough result on syntax errors.
3. Walks the AST, building a list of `LineSpan` ranges to elide — function bodies longer than `min_body_lines` (default 4) and block comments longer than `min_comment_lines` (default 6).
4. Returns a `SummaryResult` carrying a `Vec` where each segment is tagged `"kept"` or `"elided"`.
```rust
pub struct SummaryResult {
pub language: Option,
pub parsed: bool,
pub elided: bool,
pub total_lines: u32,
pub segments: Vec,
}
```
The caller can reconstruct a token-efficient representation of any source file by emitting kept segments and substituting a placeholder comment for elided spans.
Sources: [crates/pi-ast/src/summary.rs:15-53](), [crates/pi-ast/src/summary.rs:61-80]()
---
## pi-iso — Workspace Isolation PAL
### Purpose
When the toolchain needs to run a task in an isolated copy of the workspace (e.g., an agent that may mutate files), `pi-iso` provides a platform-abstracted copy-on-write mechanism. The goal is to give the task a writable view of the workspace without a full deep copy — only blocks actually written are duplicated.
### Backend Hierarchy
`pi-iso` defines the `IsolationBackend` trait and ships eight implementations:
| `BackendKind` | Mechanism | Platform |
|---|---|---|
| `Apfs` | `clonefile(2)` recursive reflink | macOS |
| `Btrfs` | `btrfs subvolume snapshot` | Linux + btrfs |
| `Zfs` | ZFS dataset snapshot + clone | Linux/FreeBSD/macOS + ZFS |
| `LinuxReflink` | `FICLONE` ioctl per-file reflink | Linux (btrfs, XFS, bcachefs) |
| `Overlayfs` | kernel `overlay` fs + `fuse-overlayfs` fallback | Linux |
| `WindowsBlockClone` | `FSCTL_DUPLICATE_EXTENTS_TO_FILE` | Windows (NTFS/ReFS) |
| `Projfs` | Windows Projected File System | Windows |
| `Rcopy` | `git worktree` or recursive copy | All (universal fallback) |
The auto-selection order is platform-specific:
- **macOS**: `Apfs → Zfs → Rcopy`
- **Linux**: `Btrfs → Zfs → LinuxReflink → Overlayfs → Rcopy`
- **Windows**: `WindowsBlockClone → Projfs → Rcopy`
Sources: [crates/pi-iso/src/lib.rs:47-140]()
### Backend Trait
```rust
pub trait IsolationBackend: Send + Sync {
fn kind(&self) -> BackendKind;
fn probe(&self) -> ProbeResult;
fn start(&self, lower: &Path, merged: &Path) -> IsoResult<()>;
fn stop(&self, merged: &Path) -> IsoResult<()>;
async fn diff(&self, lower: &Path, merged: &Path) -> IsoResult;
}
```
`start`/`stop` are synchronous because the platform primitives they wrap (`mount`, `clonefile`, `PrjStartVirtualizing`) are blocking syscalls; callers are expected to drive them from `spawn_blocking`. `diff` is async because it may walk large trees or spawn `git`.
Every backend is compiled into every build as a unit struct. On platforms where it isn't native, `probe()` returns `available: false` and `start` returns `IsoError::Unavailable` — so the N-API shim can mirror the user's `task.isolation.mode` setting without platform-conditional code on the JS side.
Sources: [crates/pi-iso/src/lib.rs:211-246](), [crates/pi-iso/src/lib.rs:253-277]()
### Diff Surface
For git-backed workspaces (all agent tasks in omp), `IsolationBackend::diff` delegates to `git diff`, producing output that is byte-identical to what `git apply` consumes. For non-git trees (only reachable via `Rcopy`), it walks both trees using `(size, mtime)` as a cheap pre-filter before doing a content comparison.
Sources: [crates/pi-iso/src/lib.rs:1-20]()
### Fallback Resolution
The `resolve(preferred: Option)` function encodes the full probe-and-fallback sequence: it checks the caller's preferred backend first, then iterates the platform's auto-order list, skipping backends whose `probe()` reports unavailable. The result is a `Resolution` struct carrying the winning `kind`, a full `candidates` list for retry-on-`IsoError::Unavailable`, and a `fell_back` flag.
Sources: [crates/pi-iso/src/lib.rs:316-373]()
---
## Hot-Path Invariants
The no-fork guarantee holds because:
1. **Shell execution** (`pi-shell`) calls `BrushShell::run_string` — an in-process Rust function call — not `execvp`. External commands launched by the shell do fork child processes, but those are controlled by the tool and rare on the hot path.
2. **Search** (`pi-natives::grep`) calls the ripgrep library crates (`grep-regex`, `grep-searcher`, `ignore`) directly with `rayon` parallelism, never spawning `rg`.
3. **AST operations** (`pi-ast`) run the tree-sitter parser in-process with zero child processes.
4. **Isolation setup** (`pi-iso`) uses OS-level CoW syscalls (`clonefile`, `FICLONE`, `mount overlay`) that are single syscalls, not child programs — except `Rcopy` which may call `git worktree add` as a one-time setup.
The minimizer is the only component that intentionally defers work (buffering output) to save downstream token cost, and it is strictly opt-in with an explicit byte cap to bound peak memory.
Sources: [crates/pi-natives/src/lib.rs:17-19](), [crates/pi-shell/src/shell.rs:569-588]()