# Build, test, and troubleshooting

> `cargo build/test` targets, musl release builds, feature flags (`db`, `otel`, `sentry`), E2E shell scripts, live MCP tests, and common failure signals from integration tests.

- Repository: Parcha-ai/ati
- GitHub: https://github.com/Parcha-ai/ati
- Human docs: https://grok-wiki.com/public/docs/parcha-ai-ati-a9d4398f11fa
- Complete Markdown: https://grok-wiki.com/public/docs/parcha-ai-ati-a9d4398f11fa/llms-full.txt

## Source Files

- `README.md`
- `Cargo.toml`
- `scripts/test_skills_e2e.sh`
- `scripts/test_proxy_server_e2e.sh`
- `scripts/e2e/README.md`
- `tests/mcp_live_test.rs`
- `src/core/error.rs`

---

---
title: "Build, test, and troubleshooting"
description: "`cargo build/test` targets, musl release builds, feature flags (`db`, `otel`, `sentry`), E2E shell scripts, live MCP tests, and common failure signals from integration tests."
---

ATI ships as the `agent-tools-interface` Rust crate (`ati` library, `ati` binary) with optional compile-time features, a layered `cargo test` suite (~790 cases), and bash E2E harnesses that exercise real `ati proxy` processes. Release Linux artifacts are static musl binaries; CI runs the default test sweep plus proxy, skills, full-stack, and OTel integration scripts against a debug build.

## Build commands

| Goal | Command | Output |
|------|---------|--------|
| Dev binary | `cargo build` | `target/debug/ati` |
| Optimized binary | `cargo build --release` | `target/release/ati` |
| Static Linux (sandbox) | `cargo build --release --target x86_64-unknown-linux-musl` | `target/x86_64-unknown-linux-musl/release/ati` |
| Install from crates.io | `cargo install agent-tools-interface` | `~/.cargo/bin/ati` |

Linux musl builds need `musl-tools` on x86_64; `aarch64-unknown-linux-musl` release builds use `cross` in `.github/workflows/release.yml`. Tagged releases also run `cargo auditable build` for supply-chain attestations.

The release profile in `Cargo.toml` optimizes for small static binaries: `opt-level = "z"`, `lto = true`, `codegen-units = 1`, `strip = true`.

<CodeGroup>
```bash title="Default build"
cargo build --release
```

```bash title="Production proxy (Sentry + OTel)"
cargo build --release --features sentry,otel
```

```bash title="Proxy with Postgres persistence"
cargo build --release --features db --target x86_64-unknown-linux-musl
```
</CodeGroup>

<ParamField body="--features" type="comma-separated flags">
Optional compile-time features. Default build has no features enabled.
</ParamField>

## Feature flags

| Feature | Enables | Runtime behavior |
|---------|---------|------------------|
| *(default)* | Core CLI + proxy | No Sentry, OTel, or Postgres driver compiled in |
| `db` | `sqlx`, `moka` | `ATI_DB_URL` connects a pool; `--migrate` applies embedded SQL; `/health` reports `db: "connected"` |
| `otel` | OpenTelemetry SDK + OTLP HTTP exporter | Spans/metrics when `OTEL_EXPORTER_OTLP_ENDPOINT` is set; see `docs/OTEL.md` |
| `sentry` | `sentry`, `sentry-tracing` | Error reporting when `SENTRY_DSN` or `GREP_SENTRY_DSN` is set |

<Note>
`db` uses `sqlx` with `runtime-tokio-rustls` only (no MySQL compiled into the binary). `otel` uses HTTP/protobuf OTLP, not gRPC, to keep musl artifacts lean (~500 KiB measured delta in `Cargo.toml` comments).
</Note>

Build without `db` but set `ATI_DB_URL` → `connect_optional()` returns `DbError::FeatureDisabled` with message *"ATI built without `db` feature; rebuild with --features db to use ATI_DB_URL"*. Build without `otel` but set `OTEL_EXPORTER_OTLP_ENDPOINT` → startup warning pointing at `--features otel`. Same pattern for Sentry DSN without `sentry`.

Release matrix (`.github/workflows/release.yml`) publishes musl and macOS variants with suffixes `""`, `-sentry`, and `-sentry-otel`. Pre-release tags (`v0.8.0-rc.*`) upload assets but skip crates.io/PyPI publish.

## Test layers

```mermaid
flowchart TB
  subgraph unit["Unit tests (src/**/mod tests)"]
    U["~304 in lib crate"]
  end
  subgraph integration["Integration tests (tests/*.rs)"]
    I["wiremock + tempfile + tower::ServiceExt"]
    S["assert_cmd + CARGO_BIN_EXE_ati subprocess"]
  end
  subgraph optional["Operator / CI opt-in"]
    L["mcp_live_test --ignored"]
    D["control_plane_db_test --features db"]
    K["keys_live_test + ATI_DB_URL_TEST"]
  end
  subgraph e2e["Bash E2E (scripts/)"]
    E1["test_proxy_e2e.sh"]
    E2["test_skills_e2e.sh"]
    E3["test_proxy_server_e2e.sh"]
    E4["test_full_stack_e2e.sh"]
    E5["test_otel_e2e.sh"]
  end
  unit --> integration
  integration --> e2e
  optional -.-> integration
```

### Default `cargo test`

```bash
cargo test                    # full suite; RUSTFLAGS=-Dwarnings in CI
cargo test --quiet            # CI uses this in .github/workflows/ci.yml
cargo test --test manifest_test
cargo test test_parse_parallel_manifest   # single test by name
cargo test mcp_client         # pattern across crates
```

A clean run reports **~790 passing tests** across the library crate and `tests/*.rs` integration files (README badge: 791). `tests/mcp_live_test.rs` contributes **8 ignored** network tests that do not run in the default sweep.

### Integration test patterns

| Pattern | Where | Purpose |
|---------|-------|---------|
| `wiremock::MockServer` | `tests/proxy_test.rs`, `proxy_server_test.rs`, `http_test.rs`, … | Mock upstream HTTP without TCP |
| `tower::ServiceExt::oneshot` | Proxy router tests | In-process axum handlers, no bind |
| `assert_cmd` + `env!("CARGO_BIN_EXE_ati")` | `skill_fetch_test.rs`, `provider_load_test.rs`, … | Real CLI subprocess with env overrides |
| `tempfile::TempDir` | Most integration tests | Isolated `ATI_DIR` / manifests |
| `tests/common/mod.rs` | Shared builders | `test_provider()`, `temp_manifests()`, JWT helpers |

### Live MCP tests

Network-dependent cases in `tests/mcp_live_test.rs` are `#[ignore]` so CI and local sweeps stay offline-safe.

```bash
cargo test --test mcp_live_test -- --ignored --nocapture
```

<Warning>
Live tests need `npx` in PATH and credentials from env or `~/.claude.json` / `~/.github-token` (GitHub, Linear, Sentry MCP servers). They hit real MCP endpoints and external APIs.
</Warning>

Ignored cases include GitHub stdio (`@modelcontextprotocol/server-github`), Linear HTTP MCP, Sentry MCP, DeepWiki HTTP, and ATI `McpClient` integration paths. Unit-level SSE parsing tests in the same file **are not** ignored and run on every `cargo test`.

### Postgres-backed tests (feature `db`)

```bash
# In-process: no Postgres required
cargo test --test db_optional_test

# Live DB: opt in with ATI_TEST_DB_URL
export ATI_TEST_DB_URL=postgres://postgres:test@localhost:5440/postgres
cargo test --features db --test control_plane_db_test

# Virtual keys E2E
export ATI_DB_URL_TEST=postgres://...
cargo test --features db --test keys_live_test
```

`db_optional_test` verifies `DbState::Disabled` when `ATI_DB_URL` is unset/empty and that migrations are a no-op on disabled state. Live suites skip cleanly when the env URL is missing.

## E2E shell scripts

| Script | What it validates | Binary default | CI |
|--------|-------------------|----------------|-----|
| `scripts/test_proxy_e2e.sh` | `ati run` through mock Python proxy (`ATI_PROXY_URL`) | `./target/debug/ati` | Yes |
| `scripts/test_skills_e2e.sh` | Skill registry, HTTP/OpenAPI/MCP manifests, proxy `/skills` | `target/release/ati` (builds if missing) | Yes (`ATI_BIN=./target/debug/ati`) |
| `scripts/test_proxy_server_e2e.sh` | Real `ati proxy` → mock upstream round-trip | `./target/debug/ati` | Yes |
| `scripts/test_full_stack_e2e.sh` | Sig-verify, SIGHUP rotation, passthrough, WebSocket, `ati edge` (~67 cases) | `target/release/ati` | Yes (needs `websockets==15.0.1`) |
| `scripts/test_otel_e2e.sh` | OTLP HTTP export with `--features otel` | `./target/debug/ati` | Yes |
| `scripts/test_cleanup_e2e.sh` | Audit, rate limiter, plan mode | `target/release/ati` | No (local) |
| `scripts/test_skill_fetch_e2e.sh` | Remote GCS skillati shape | `./target/debug/ati` | No (needs GCP creds) |

<Steps>
<Step title="Run the CI-equivalent E2E stack locally">

```bash
cargo build
bash scripts/test_proxy_e2e.sh
ATI_BIN=./target/debug/ati bash scripts/test_skills_e2e.sh
bash scripts/test_proxy_server_e2e.sh
pip install --user 'websockets==15.0.1'
ATI_BIN=./target/debug/ati bash scripts/test_full_stack_e2e.sh --pr all
bash scripts/test_otel_e2e.sh
```

</Step>
<Step title="Full-stack harness with post-mortem">

```bash
cargo build --release --bin ati
bash scripts/test_full_stack_e2e.sh --pr all --keep-tmpdir
# Inspect /tmp/ati-e2e-XXXXXX proxy logs
```

Filter groups: `--pr 96` (sig-verify), `--pr 97` (`ati edge`), `--pr 98` (WebSocket). See `scripts/e2e/README.md` for port map (18910–18931 on `127.0.0.1`) and scenario matrix.

</Step>
</Steps>

<Tip>
Set `ATI_BIN` to point E2E scripts at a specific build. CI builds debug once, then reuses it for all harnesses except the Sentry smoke build (`target/sentry/debug/ati`) so feature-specific binaries do not overwrite the default E2E binary.
</Tip>

### What full-stack E2E catches that `cargo test` cannot

`scripts/e2e/README.md` documents gaps in-process tests miss: middleware ordering (sig-verify before JWT), real `SIGHUP` + `ArcSwap` secret rotation, 50 concurrent in-flight requests across rotation, atomic `ati edge rotate-keyring`, fake `op` subprocess via `--op-path`, and real WebSocket upgrade/subprotocol negotiation.

## CI jobs

`.github/workflows/ci.yml` runs on every push/PR to `main`:

| Job | Command / action |
|-----|------------------|
| Test and E2E | `cargo test --quiet`, `cargo build`, Sentry smoke, all E2E scripts above |
| cargo-deny | `cargo deny check` |
| Formatting | `cargo fmt --all --check` |
| Clippy | `cargo clippy --all-targets -- -D warnings` |
| Coverage | `cargo llvm-cov` |
| cargo-audit | `cargo audit` |

`SKIP_OTEL_E2E=1` skips the OTel harness when needed.

## Structured errors and exit codes

CLI failures map through `src/core/error.rs`. With `--output json`, stderr emits a structured object; `--verbose` adds an error chain.

| `error.code` | Typical message signal | Exit code |
|--------------|------------------------|-----------|
| `tool.not_found` | `unknown tool` | 1 |
| `auth.scope_denied` | `scope`, `access denied` | 3 |
| `auth.expired` | `expired` | 3 |
| `auth.missing_key` | `key not found`, `missing key`, `no keys found` | 3 |
| `provider.timeout` | `timeout` | 4 |
| `provider.upstream_error` | `upstream`, `bad gateway`, `mcp error` | 4 |
| `provider.not_found` | `provider` + `not found` | 4 |
| `input.missing_arg` | `missing`, `required` | 2 |
| `input.invalid_value` | `invalid`, `parse` | 2 |
| `rate.exceeded` | `rate limit` | 5 |
| `tool.execution_failed` | *(fallback)* | 1 |

<Info>
Use `ati run ... --verbose` for full error chains on stderr. JSON mode always prints `error.code`, `error.message`, and `error.exit_code`; verbose JSON adds `error.chain`.
</Info>

## Troubleshooting

### Build failures

| Symptom | Likely cause | Fix |
|---------|--------------|-----|
| `linker ... musl-gcc not found` | Missing musl toolchain | `sudo apt-get install musl-tools` (x86_64 Linux) |
| `failed to run custom build command` for `ring` | Cross-compile env | Use release workflow’s `cross` for `aarch64-unknown-linux-musl` |
| `multiple versions of crate opentelemetry` | Mixed OTel crate versions | Bump `opentelemetry`, `opentelemetry_sdk`, `opentelemetry-otlp`, `tracing-opentelemetry` together (`Cargo.toml` note) |
| Clippy CI failure | `RUSTFLAGS=-Dwarnings` | Run `cargo clippy --all-targets -- -D warnings` locally |

### Test failures

| Symptom | Likely cause | Fix |
|---------|--------------|-----|
| `Address already in use` on 18910–18931 | Stale proxy from aborted E2E | Re-run harness (orchestrator kills ports on trap); or `ss -tln` / `lsof` |
| `python3 -c 'import websockets' failed` | Missing dep for full-stack | `pip install --user websockets==15.0.1` |
| `ATI built without db feature` in proxy logs | `ATI_DB_URL` set on default binary | Rebuild with `--features db` or unset `ATI_DB_URL` |
| `enabled sentry client` grep fails in CI | Built without `sentry` or bad DSN | Build `cargo build --features sentry --target-dir target/sentry` |
| MCP live test skip / fail | Not using `--ignored` or no token | `cargo test --test mcp_live_test -- --ignored`; set `GITHUB_PERSONAL_ACCESS_TOKEN`, `LINEAR_API_KEY`, etc. |
| `connect_optional_set_env_without_feature_errors` | Expected on default build | Either unset `ATI_DB_URL` or build with `--features db` |

### Runtime / operator signals

| Symptom | Check |
|---------|-------|
| Tool missing at runtime | `ati tool list`; manifest under `$ATI_DIR/manifests/`; scope in JWT |
| Proxy 401 vs 403 on passthrough | 403 often sig-verify (unsigned); 401 JWT — full-stack Group F tests ordering |
| `/health` shows `db: "disabled"` | Normal without `ATI_DB_URL`; set URL + `db` feature for persistence |
| OTel spans absent | `OTEL_EXPORTER_OTLP_ENDPOINT`, `--features otel`, 10s flush window (`docs/OTEL.md`) |
| Sentry events dropped on CLI exit | Regression if transport flush skipped; CI greps for `client close; request transport to shut down` |

### Manifest and registry errors

Integration tests in `tests/manifest_test.rs` cover TOML parsing for auth types, OpenAPI overrides, MCP fields, and `auth_generator` blocks. Load-time failures surface as manifest/registry errors before any network call — fix the `.toml` under `manifests/` and re-run `ati tool list` to confirm registration.

## Local quality gates before push

<CodeGroup>
```bash title="Fast loop"
cargo test
cargo fmt --all --check
cargo clippy --all-targets -- -D warnings
```

```bash title="Pre-release proxy check"
cargo build --release --features sentry,otel
bash scripts/test_full_stack_e2e.sh --pr all
```
</CodeGroup>

Set `RUST_LOG=debug` or `ati=debug` for tracing during failing E2E or proxy runs.

## Related pages

<CardGroup>
<Card title="Installation" href="/installation">
Pre-built musl binaries, `cargo install`, and platform prerequisites.
</Card>
<Card title="Deploy proxy server" href="/deploy-proxy-server">
Run `ati proxy` with optional `--features db` and production flags.
</Card>
<Card title="Environment variables" href="/environment-variables">
`ATI_DB_URL`, OTel, Sentry, JWT, and proxy env contract.
</Card>
<Card title="Security and production" href="/security-and-production">
Threat model, proxy hardening, and observability in production.
</Card>
</CardGroup>