# Benchmarking

> Measure `rift create` performance with `create` and `compare` Cargo benches, sample counts, JSON output schema, and candidate comparison workflow.

- Repository: anomalyco/rift
- GitHub: https://github.com/anomalyco/rift
- Human docs: https://grok-wiki.com/public/docs/anomalyco-rift-ea3fd5dbf662
- Complete Markdown: https://grok-wiki.com/public/docs/anomalyco-rift-ea3fd5dbf662/llms-full.txt

## Source Files

- `README.md`
- `crates/core/benches/create.rs`
- `crates/core/benches/compare.rs`
- `crates/core/benches/AUTORESEARCH.md`
- `crates/core/Cargo.toml`

---

---
title: "Benchmarking"
description: "Measure `rift create` performance with `create` and `compare` Cargo benches, sample counts, JSON output schema, and candidate comparison workflow."
---

The `rift` crate (`crates/core`) ships two standalone Cargo bench binaries—`create` and `compare`—that time the production `Manager::create_with_options` path against a real workspace directory. Both benches use `harness = false`, so they run as ordinary binaries invoked through `cargo bench` rather than through Criterion. The `create` bench records per-sample elapsed milliseconds; `compare` orchestrates that bench across multiple candidate checkouts and ranks results by median time.

## What gets measured

The `create` bench exercises the same code path the CLI uses for workspace creation, with two deliberate isolations from unrelated overhead:

| Phase | Included in timing? | Notes |
| --- | --- | --- |
| `Manager::init` on the workload | No | Runs once before the sample loop |
| `Manager::create_with_options` | Yes | Production copy strategy and registry registration |
| Postcreate hooks | No | `HookMode::Skip` disables `.rift.toml` hook execution |
| `Manager::remove` + `Manager::gc` | No | Runs after each sample to reset state for the next run |

Each timed sample creates a uniquely named child workspace (`benchmark-{pid}-{run_id}-{sample}`), then removes it and runs garbage collection so later samples start from a clean registry state.

<Note>
The benchmark uses the production filesystem strategy selected at runtime. On macOS that means APFS `clonefile`; on Linux btrfs that means subvolume snapshots; on reflink-capable Linux filesystems that means per-file reflinks. Results are not comparable across platforms.
</Note>

## Bench targets

Both targets are declared in `crates/core/Cargo.toml`:

```toml
[[bench]]
name = "create"
harness = false

[[bench]]
name = "compare"
harness = false
```

Run them from the repository root or from `crates/core`. `cargo bench` compiles with release optimizations by default.

## `create` benchmark

### Command

```bash
cargo bench --bench create -- /path/to/workspace [--samples N] [--output /path/to/result.json]
```

<ParamField body="workspace" type="path" required>
  Directory to initialize and use as the create source. Must exist. The bench canonicalizes this path before timing.
</ParamField>

<ParamField body="--samples" type="integer" default="1">
  Number of independent create operations to time. Must be greater than zero.
</ParamField>

<ParamField body="--output" type="path">
  Optional file path. When set, writes a JSON result document after all samples complete.
</ParamField>

### Stdout

Each sample prints a tab-separated progress line:

```text
create	1/10	37.842 ms	/path/to/.rifts/benchmark-12345-...
```

When `--samples` is greater than 1, a summary line follows:

```text
median	37.800 ms	min 37.600 ms	max 38.100 ms
```

On failure the bench prints `benchmark failed: {error}` to stderr and exits with code `1`.

### Establish a baseline

Measure multiple independent creations and persist machine-readable output outside the workload directory so result files do not alter future measurements:

```bash
mkdir -p /path/to/results
cargo bench --bench create -- /path/to/linux \
  --samples 10 \
  --output /path/to/results/baseline.json
```

Use `median_ms` as the primary comparison metric. Keep `samples_ms` to detect noisy runs.

## `create` JSON output schema

When `--output` is set, the bench writes a single JSON object:

<ResponseField name="benchmark" type="string">
  Always `"create"`.
</ResponseField>

<ResponseField name="timestamp_ms" type="integer">
  Wall-clock milliseconds since Unix epoch at write time.
</ResponseField>

<ResponseField name="platform" type="string">
  `std::env::consts::OS` value (for example `"linux"`, `"macos"`).
</ResponseField>

<ResponseField name="source" type="string">
  Canonicalized workload directory path.
</ResponseField>

<ResponseField name="samples_ms" type="number[]">
  Elapsed milliseconds for each sample, in run order.
</ResponseField>

<ResponseField name="median_ms" type="number">
  Median of `samples_ms`. For even sample counts, the average of the two middle values.
</ResponseField>

<ResponseField name="min_ms" type="number">
  Minimum value in `samples_ms`.
</ResponseField>

<ResponseField name="max_ms" type="number">
  Maximum value in `samples_ms`.
</ResponseField>

<ResponseField name="cleanup_passed" type="boolean">
  Always `true` when the bench exits successfully. The `compare` bench rejects candidates where this is `false`.
</ResponseField>

<RequestExample>

```json
{
  "benchmark": "create",
  "timestamp_ms": 1718640000123,
  "platform": "macos",
  "source": "/path/to/linux",
  "samples_ms": [37.8, 38.1, 37.6, 37.9, 38.0],
  "median_ms": 37.8,
  "min_ms": 37.6,
  "max_ms": 38.1,
  "cleanup_passed": true
}
```

</RequestExample>

## `compare` benchmark

The `compare` bench runs each candidate's `create` bench against the same workload, collects per-candidate JSON results, and writes a ranked summary.

### Command

```bash
cargo bench --bench compare -- /path/to/workload \
  --candidate /path/to/rift \
  [--candidate /path/to/rift-other] \
  --output /path/to/results \
  [--samples N]
```

<ParamField body="workload" type="path" required>
  Workload directory passed through to every candidate's `create` bench. Canonicalized before execution.
</ParamField>

<ParamField body="--candidate" type="path" required>
  Repeatable. Each path must be a Cargo workspace root containing `Cargo.toml` and the `create` bench target. At least one candidate is required.
</ParamField>

<ParamField body="--output" type="path" required>
  Output directory. Created if missing, then canonicalized. Receives per-candidate JSON files and `summary.json`.
</ParamField>

<ParamField body="--samples" type="integer" default="10">
  Sample count forwarded to each candidate's `create` bench.
</ParamField>

### How candidates are invoked

For each `--candidate` path, `compare` runs:

```bash
cargo bench --locked --bench create -- <workload> --samples <N> --output <output>/candidate-NN.json
```

with `current_dir` set to the candidate checkout. Candidates are numbered in invocation order (`candidate-01.json`, `candidate-02.json`, …). If any candidate's `cargo bench` exits non-zero, or `cleanup_passed` is not `true` in its result file, the comparison fails.

### Stdout

Per candidate:

```text
running	/path/to/rift-git
```

After all candidates complete, ranked results:

```text
1	34.900 ms	+0.00%	/path/to/rift-registry
2	39.400 ms	+12.92%	/path/to/rift
summary	/path/to/results/round-01/summary.json
```

Ranking uses `median_ms`. `difference_from_fastest_percent` is `(median_ms / fastest_median - 1) * 100`.

### Output directory layout

```text
/path/to/results/round-01/
  candidate-01.json
  candidate-02.json
  candidate-03.json
  summary.json
```

## `compare` JSON output schema

### Per-candidate files (`candidate-NN.json`)

Same schema as the `create` bench output.

### Summary file (`summary.json`)

<ResponseField name="benchmark" type="string">
  Always `"create"`.
</ResponseField>

<ResponseField name="timestamp_ms" type="integer">
  Wall-clock milliseconds since Unix epoch at summary write time.
</ResponseField>

<ResponseField name="platform" type="string">
  OS of the machine running `compare` (not per-candidate).
</ResponseField>

<ResponseField name="source" type="string">
  Canonicalized workload path.
</ResponseField>

<ResponseField name="samples" type="integer">
  Sample count used for every candidate in this round.
</ResponseField>

<ResponseField name="candidates" type="object[]">
  Ranked list sorted by ascending `median_ms`.
</ResponseField>

Each entry in `candidates`:

<ResponseField name="rank" type="integer">
  1-based rank by `median_ms`.
</ResponseField>

<ResponseField name="candidate" type="string">
  Canonicalized path to the candidate checkout.
</ResponseField>

<ResponseField name="result" type="string">
  Path to the corresponding `candidate-NN.json` file.
</ResponseField>

<ResponseField name="median_ms" type="number">
  Median create time for this candidate.
</ResponseField>

<ResponseField name="min_ms" type="number">
  Minimum sample time.
</ResponseField>

<ResponseField name="max_ms" type="number">
  Maximum sample time.
</ResponseField>

<ResponseField name="difference_from_fastest_percent" type="number">
  Percent slower than the fastest candidate's `median_ms`. The fastest candidate is `0.0`.
</ResponseField>

<RequestExample>

```json
{
  "benchmark": "create",
  "timestamp_ms": 1718640000456,
  "platform": "linux",
  "source": "/path/to/linux",
  "samples": 10,
  "candidates": [
    {
      "rank": 1,
      "candidate": "/path/to/rift-registry",
      "result": "/path/to/results/round-01/candidate-03.json",
      "median_ms": 34.9,
      "min_ms": 34.1,
      "max_ms": 36.0,
      "difference_from_fastest_percent": 0.0
    },
    {
      "rank": 2,
      "candidate": "/path/to/rift",
      "result": "/path/to/results/round-01/candidate-01.json",
      "median_ms": 39.4,
      "min_ms": 38.8,
      "max_ms": 40.6,
      "difference_from_fastest_percent": 12.9
    }
  ]
}
```

</RequestExample>

## Candidate comparison workflow

Use the `compare` bench to evaluate implementation changes across isolated checkouts. A typical layout keeps one unchanged baseline, several experimental siblings, one shared workload, and results stored outside all checkouts:

```text
/path/to/rift              unchanged baseline and comparison runner
/path/to/rift-git          one experimental hypothesis
/path/to/rift-registry     another experimental hypothesis
/path/to/linux             shared workload checkout
/path/to/results           benchmark evidence
```

<Steps>
<Step title="Preserve the benchmark framework">
Commit or otherwise preserve the benchmark targets in the base `rift` checkout before creating candidates. Every candidate must contain the `create` bench and its dependencies.
</Step>

<Step title="Create experimental candidates">
Create sibling checkouts from the same Git revision. Rift itself can provision them:

```bash
cd /path/to/rift
rift init --here .
rift create --name rift-git --into /path/to
rift create --name rift-registry --into /path/to
```

Use the paths printed by `rift create` if storage layout differs from the example.
</Step>

<Step title="Verify correctness in each changed candidate">
Before benchmarking, run tests in every candidate that received code changes:

```bash
cargo test --workspace --locked
```

Exclude candidates whose tests fail from the comparison invocation.
</Step>

<Step title="Run the comparison from the baseline checkout">
Keep the base `rift` checkout unchanged and include it as a `--candidate` for baseline ranking:

```bash
cd /path/to/rift
cargo bench --bench compare -- /path/to/linux \
  --candidate /path/to/rift \
  --candidate /path/to/rift-git \
  --candidate /path/to/rift-registry \
  --samples 10 \
  --output /path/to/results/round-01
```
</Step>

<Step title="Inspect results and decide next steps">
Read `summary.json` and per-candidate `samples_ms` arrays. Apply the decision rules below before promoting a winner or starting a follow-up round.
</Step>
</Steps>

For follow-up rounds among finalists, increase sample count rather than re-measuring clearly unsuccessful candidates:

```bash
cargo bench --bench compare -- /path/to/linux \
  --candidate /path/to/rift \
  --candidate /path/to/rift-registry \
  --candidate /path/to/rift-combined \
  --samples 30 \
  --output /path/to/results/round-02-finalists
```

<Warning>
On Linux, the workload must reside on supported btrfs or native reflink-capable storage for production copy-on-write behavior. On the first benchmark run against an ordinary btrfs directory, `init` may convert it into a subvolume before any timed samples—that conversion is outside the measured interval but affects what subsequent runs measure.
</Warning>

## Interpreting results

| Rule | Action |
| --- | --- |
| Tests fail in a candidate | Exclude the candidate; do not benchmark it |
| `cleanup_passed` is not `true` | Reject the candidate's benchmark result |
| Lower `median_ms` than baseline | Candidate is a timing improvement candidate |
| Raw `samples_ms` overlap substantially | Treat small median differences as inconclusive; rerun finalists with more `--samples` |
| Multiple independent winners | Combine changes in a fresh `rift-*` candidate from the baseline, then benchmark the combination against individual winners |
| Round complete | Keep the results directory unchanged as experiment evidence |

<Tip>
Assign one clearly stated hypothesis per experimental `rift-*` candidate so measured differences can be attributed to specific code changes.
</Tip>

## Limitations

The benchmark framework handles measurement, JSON persistence, and candidate ranking. It does not automate hypothesis selection, parallel code changes, test execution, candidate exclusion, or promotion decisions—those remain manual or agent-driven steps in the auto-research workflow documented in `crates/core/benches/AUTORESEARCH.md`.

Postcreate hooks are skipped during timing (`HookMode::Skip`). Benchmark results do not reflect hook execution cost; see the postcreate hooks page if hook latency matters for your workload.

## Related pages

<CardGroup>
<Card title="Create workspaces" href="/create-workspaces">
CLI flags and behavior for `rift create`, including filtered copy, `--copy-all`, and hook modes.
</Card>
<Card title="Copy strategies and platforms" href="/copy-strategies">
Platform-specific copy-on-write backends that determine what the benchmark actually measures.
</Card>
<Card title="Development" href="/development">
Build and test the Rust workspace with `cargo test --workspace --locked` before benchmarking candidates.
</Card>
<Card title="Postcreate hooks" href="/postcreate-hooks">
Hook configuration and execution—the benchmark skips hooks via `HookMode::Skip`.
</Card>
</CardGroup>
