# Security and production

> Threat model, proxy hardening (JWT enforce, download allowlist, SSRF), sig-verify modes, optional Postgres audit/virtual keys, OTel observability, and edge keyring rotation.

- Repository: Parcha-ai/ati
- GitHub: https://github.com/Parcha-ai/ati
- Human docs: https://grok-wiki.com/public/docs/parcha-ai-ati-a9d4398f11fa
- Complete Markdown: https://grok-wiki.com/public/docs/parcha-ai-ati-a9d4398f11fa/llms-full.txt

## Source Files

- `docs/SECURITY.md`
- `src/core/sig_verify.rs`
- `src/core/file_manager.rs`
- `docs/PERSISTENCE.md`
- `docs/OTEL.md`
- `src/cli/edge.rs`
- `migrations/20260501000001_init_persistence.sql`

---

---
title: "Security and production"
description: "Threat model, proxy hardening (JWT enforce, download allowlist, SSRF), sig-verify modes, optional Postgres audit/virtual keys, OTel observability, and edge keyring rotation."
---

ATI production security centers on **where credentials live** (sandbox keyring vs proxy host), **how every request is authenticated** (JWT, optional DB-backed `Ati-Key`, HMAC sig-verify for passthrough), and **what egress the proxy allows** (SSRF checks and download host allowlists on `file_manager:download`). The sections below map those controls to exact env vars, CLI flags, and operator workflows.

## Threat model and trust boundaries

ATI exposes tools to untrusted agent sandboxes. The security model splits on execution mode:

| Mode | Credential location | Primary risk |
|------|---------------------|--------------|
| **Local** (`ATI_PROXY_URL` unset) | AES-256-GCM `keyring.enc` decrypted in-process; one-shot session key at `/run/ati/.key` (unlinked after first read) | Keys exist inside the sandbox process (`mlock`, `madvise(DONTDUMP)`, zeroize on drop — best-effort) |
| **Proxy** (`ATI_PROXY_URL` set) | Real API keys only on the proxy host | Sandbox holds manifests, scopes, and `ATI_SESSION_TOKEN` — no keyring material |

```mermaid
flowchart TB
  subgraph sandbox["Agent sandbox"]
    ATI_CLI["ati run / ati tool"]
    TOKEN["ATI_SESSION_TOKEN JWT or Ati-Key"]
  end
  subgraph proxy_host["Proxy host (~/.ati/)"]
    PROXY["ati proxy"]
    KEYRING["keyring.enc + manifests"]
    DB["Postgres optional"]
  end
  subgraph upstream["Upstream APIs"]
    APIs["HTTP / MCP / OpenAPI providers"]
  end
  ATI_CLI -->|"POST /call (proxy mode)"| PROXY
  TOKEN --> PROXY
  PROXY --> KEYRING
  PROXY --> DB
  PROXY --> APIs
```

**Local-mode mitigations** include: no secrets in environment variables; session key file deleted after first read; encrypted keyring at rest; `mlock` / `MADV_DONTDUMP` on secret pages. **Known limitations**: without seccomp, `ptrace` can read process memory; brief race on `/run/ati/.key` before first read; `mlock` may fail silently if `RLIMIT_MEMLOCK` is low.

**Proxy-mode mitigations** include: zero key material in the sandbox; JWT scope enforcement on `/call`, `/mcp`, discovery routes, and SkillATI; scope-filtered listings so agents cannot enumerate tools they are not allowed to use. **Known limitations**: `ATI_PROXY_URL` is visible in the environment (not secret); proxy host compromise equals credential compromise; added network latency.

Scope behavior is **strict when JWT is configured** and **unrestricted dev mode when it is not** — production proxies must configure `ATI_JWT_PUBLIC_KEY` or `ATI_JWT_SECRET` so `auth_middleware` rejects missing or invalid `Authorization: Bearer` tokens.

## Production proxy hardening

Treat the following as a minimum bar for an Internet-facing `ati proxy`:

<Steps>
<Step title="Enable JWT validation">
Set ES256 or HS256 verification env vars (`ATI_JWT_PUBLIC_KEY`, or `ATI_JWT_SECRET`, plus optional `ATI_JWT_ISSUER` / `ATI_JWT_AUDIENCE`). Issue scoped tokens with `ati token issue` and pass them to sandboxes as `ATI_SESSION_TOKEN`.

Verify `/health` reports `"auth": "jwt"` (not `"disabled"`).
</Step>
<Step title="Restrict file_manager downloads">
Set both egress controls:

```bash
export ATI_SSRF_PROTECTION=1
export ATI_DOWNLOAD_ALLOWLIST=v3b.fal.media,*.googleapis.com,raw.githubusercontent.com
```

`file_manager:download` always runs SSRF validation via `validate_url_not_private` before fetch. The allowlist is separate: when `ATI_DOWNLOAD_ALLOWLIST` is **unset**, any non-private host is permitted — acceptable only for local dev.
</Step>
<Step title="Roll out HMAC sig-verify">
For passthrough or edge deployments, load `sandbox_signing_shared_secret` into the keyring, start with `--sig-verify-mode log`, observe structured `sig_verify` logs, then switch to `enforce` (see below).
</Step>
<Step title="Optional Postgres for virtual keys">
Build with `--features db`, set `ATI_DB_URL`, run `ati proxy --migrate`, and set `ATI_ADMIN_TOKEN` for `/admin/keys/*`. Use `Authorization: Ati-Key` for per-job credentials with immediate revocation.
</Step>
</Steps>

### JWT enforcement

When `jwt_config` is present at proxy startup, `auth_middleware` requires `Authorization: Bearer <jwt>` on all routes except:

- `/health` and `/.well-known/jwks.json`
- Passthrough routes matched by `PassthroughRouter` (authenticated by sig-verify instead)

| Claim / behavior | Effect |
|------------------|--------|
| `scope` (space-delimited) | Tool, skill, provider, category wildcards; `help` scope for `/help` |
| `exp` | Hard expiry — expired tokens rejected |
| Legacy `tool:github_search` | Normalized to colon namespaced form without weakening checks |

`Authorization: Ati-Key <raw>` is evaluated **before** JWT when both are present, so per-job DB keys get correct audit attribution.

<Warning>
If JWT keys are not configured, the proxy runs in **unrestricted dev mode** and allows unauthenticated requests (unless an `Ati-Key` header is sent and the `db` feature is enabled).
</Warning>

### Download allowlist (`ATI_DOWNLOAD_ALLOWLIST`)

Comma-separated host patterns (case-insensitive). Pattern forms:

| Pattern | Matches |
|---------|---------|
| `example.com` | Exact host |
| `*.fal.media` | `v3b.fal.media`, `cdn.fal.media`, and `fal.media` — not suffix tricks like `evilfal.media` |
| `*` | Everything (defeats the allowlist) |

Violations return HTTP **403** with `HostNotAllowed` on `POST /call` for `file_manager:download`.

### SSRF protection (`ATI_SSRF_PROTECTION`)

Used by the generic HTTP executor (`core/http.rs`) for hand-written and OpenAPI-backed HTTP tools:

| Value | Behavior |
|-------|----------|
| unset | No SSRF check (dev default) |
| `warn` | Log warning, allow request |
| `1` or `true` | Block loopback, RFC1918, link-local, `.internal`, `.local`, and DNS-resolved private IPs |

`file_manager:download` applies SSRF checks **regardless** of this env var. MCP and CLI subprocess tools are outside this SSRF path today.

## HMAC sandbox signature verification

Sig-verify is axum middleware on every non-exempt path. It validates outbound sandbox traffic using a shared secret from the keyring entry `sandbox_signing_shared_secret` (hex-decoded when valid hex, otherwise raw UTF-8 bytes).

### Signing format

Clients send:

```http
X-Sandbox-Signature: t=<unix_ts>,s=<hex_hmac>
```

Optional `X-Sandbox-Job-Id` is logged only. HMAC-SHA256 covers `{t}.{method}.{path}` (constant-time compare on digest).

### Modes (`--sig-verify-mode`)

| Mode | Request handling | Response |
|------|------------------|----------|
| `log` (default) | Always allow | Structured log: `valid`, `reason`, `job_id`, `method`, `path` |
| `warn` | Always allow | Adds `X-Signature-Status: <reason>` |
| `enforce` | **403 Forbidden** on invalid/missing signature | Body is the reason string (`missing_signature`, `hmac_mismatch`, `expired_timestamp_drift=Ns`, etc.) |

<ParamField body="--sig-drift-seconds" type="integer" default="60">
Maximum `|now - t|` in seconds before `ExpiredTimestamp`.
</ParamField>

<ParamField body="--sig-exempt-paths" type="string">
Comma-separated globs. Defaults include `/health`, `/healthz`, `/root/*`, `/npm/*`, `/otel/*`, `/.well-known/jwks.json`.
</ParamField>

Startup gates:

- **`enforce` without secret** — proxy refuses to start (every request would 403).
- **Passthrough + `log`/`warn` without secret** — error log: passthrough is effectively unauthenticated.

On **SIGHUP**, the proxy reloads `keyring.enc` and hot-swaps the sig-verify secret via `ArcSwapOption` without restart. Transient reload failures **preserve** the previous secret so a disk glitch does not mass-403 traffic in enforce mode.

```text
Request path (non-exempt):
  sig_verify_middleware → auth_middleware → handler
Passthrough (host+path match, not a named ATI route):
  sig_verify only (JWT skipped)
```

## Optional Postgres persistence

Persistence is **compile-time** (`--features db`) and **runtime** (`ATI_DB_URL`) gated. Without both, behavior matches a non-DB proxy.

### Schema and current write paths

Migration `20260501000001_init_persistence.sql` creates:

| Table | Purpose | Written today |
|-------|---------|---------------|
| `ati_keys` | Virtual keys (scope arrays, TTL, counters) | Yes — via `KeyStore` |
| `ati_deleted_keys` | Soft-delete snapshots on revoke | Yes |
| `ati_audit_log` | Admin mutations (`key.create`, `key.revoke`, …) | Yes |
| `ati_call_log` | Per-request proxy audit rows | **Schema only** — proxy audit still appends to `~/.ati/audit.jsonl` via `core/audit` |

<Info>
Per-call Postgres audit (`ati_call_log` inserts on every `/call`) is planned; operators should use `ati audit tail|search` on the JSONL file until DB call logging ships.
</Info>

### Virtual keys (implemented)

Orchestrators issue one-shot credentials:

:::endpoint POST /admin/keys/issue
Requires `ATI_DB_URL`, `db` feature build, and `Authorization: Bearer <ATI_ADMIN_TOKEN>`. Body includes `user_id`, `alias`, scope arrays (`tools`, `providers`, `categories`, `skills`), optional `expires_in_secs`. Response includes `raw_key` (prefix `ati-key_`), `hash`, `alias`, `expires_at` — **raw key is shown once**; DB stores `sha256(raw)` only.
:::

Revocation: `DELETE /admin/keys/{hash}`, bulk revoke, `LISTEN ati_key_revoked` for cross-pod cache invalidation (30s moka TTL on lookups).

Sandboxes authenticate with:

```http
Authorization: Ati-Key ati-key_<random>
```

Claims are synthesized from the row so existing scope checks unchanged.

### Startup and health

| Condition | Result |
|-----------|--------|
| `ATI_DB_URL` unset | `db: "disabled"` in `/health` |
| URL set, DB reachable, `--migrate` | Migrations applied; `db: "connected"` |
| URL set, DB unreachable | Proxy **exits** at startup |
| URL set, build without `db` feature | Proxy **exits** with rebuild instructions |

<Note>
`/health` `db: connected` reflects pool configuration at startup, not continuous Postgres liveness.
</Note>

## Observability (OpenTelemetry)

OTel is behind `--features otel`. When `OTEL_EXPORTER_OTLP_ENDPOINT` is unset, the layer is not installed (zero runtime overhead). When set on a non-otel build, ATI logs a warning to rebuild with the feature.

### Spans and metrics

| Span | Where | Key attributes |
|------|-------|----------------|
| `http.server.request` | Outermost proxy middleware | `http.route`, `http.request.method`, `http.response.status_code`, `url.path` |
| `proxy.call` | `POST /call` | `tool` |
| `proxy.mcp` | `POST /mcp` | `jsonrpc.method` |
| `proxy.help` | `POST /help` | — |
| `passthrough.request` | Passthrough fallback | `route`, `upstream` |

| Metric | Type | Labels |
|--------|------|--------|
| `ati.proxy.requests` | counter | `http.route`, `http.request.method`, `http.response.status_class` |
| `ati.proxy.request_duration_ms` | histogram | same |
| `ati.upstream.errors` | counter | `provider`, `error_kind` |

W3C `traceparent` is extracted on inbound proxy requests and injected on outbound HTTP/MCP HTTP calls. Passthrough strips inbound sandbox trace headers before upstream injection.

Production build pattern:

```bash
cargo build --release --features sentry,otel
export OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp.example.com/otlp
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic ..."
export OTEL_SERVICE_NAME=ati-proxy
export ENVIRONMENT_TIER=production
ati proxy --port 8090
```

`/otel/*` routes are sig-verify exempt (aligned with typical Caddy `forward_auth` skips).

## Edge keyring rotation

`ati edge` commands target static-IP egress VMs that hold the production keyring. They shell out to the 1Password CLI (`op`) — field `label` becomes keyring entry name.

<Steps>
<Step title="Bootstrap a fresh VM">
```bash
ati edge bootstrap-keyring \
  --vault "Production" \
  --item "ATI Edge Keyring" \
  --ati-dir /var/lib/ati \
  --op-token-file "${CREDENTIALS_DIRECTORY}/op-token"
```

Writes `keyring.enc` (AES-256-GCM) and `.keyring-key` (persistent 32-byte session key, mode `0600`) via atomic tempfile + `rename`.
</Step>
<Step title="Rotate credentials without restart">
```bash
ati edge rotate-keyring \
  --vault "Production" \
  --item "ATI Edge Keyring" \
  --ati-dir /var/lib/ati \
  --service ati
```

Re-pulls from 1Password, atomically replaces `keyring.enc` (session key unchanged), sends **SIGHUP** to the systemd `MainPID`. The proxy reloads API keys and `sandbox_signing_shared_secret` for sig-verify. Use `--no-signal` when rotating before first start.
</Step>
</Steps>

Expected 1Password item fields include provider API keys and `sandbox_signing_shared_secret` for HMAC enforcement.

## Binary supply chain

Release verification (from `docs/SECURITY.md`):

```bash
gh attestation verify ./ati --repo Parcha-ai/ati
sha256sum -c ati-x86_64-unknown-linux-musl.sha256
cargo audit bin ./ati
```

CI runs `cargo-deny` (registry allowlist, no git deps, RustSec advisories, license allowlist) and pins GitHub Actions to full commit SHAs.

## Failure modes

| Symptom | Likely cause | Mitigation |
|---------|--------------|------------|
| `401` on all proxy calls | Missing/expired JWT or revoked `Ati-Key` | Re-issue token; check `ati token validate` |
| `403` + `missing_signature` | `enforce` mode without `X-Sandbox-Signature` | Fix sandbox signer or use `log` during rollout |
| `403` + `HostNotAllowed` | URL host outside `ATI_DOWNLOAD_ALLOWLIST` | Add pattern or fix agent URL |
| `403` + `PrivateUrl` | SSRF block on internal target | Expected — do not disable in production |
| Proxy won't start with `ATI_DB_URL` | DB down or non-`db` binary | Fix connectivity or use `-db` release artifact |
| Passthrough works without auth | `log`/`warn` + no signing secret | Set secret + `--sig-verify-mode enforce` |
| Stale API keys after rotation | SIGHUP failed | Check `systemctl` MainPID; restart proxy |

## Related pages

<CardGroup>
<Card title="Execution modes" href="/execution-modes">
Local vs proxy credential placement and mode auto-detection.
</Card>
<Card title="Deploy proxy server" href="/deploy-proxy-server">
Bind, passthrough, sig-verify flags, and systemd deployment patterns.
</Card>
<Card title="Configure JWT and keys" href="/configure-jwt-and-keys">
Token issuance, JWKS, and orchestrator scope patterns.
</Card>
<Card title="File manager operations" href="/file-manager-operations">
Download/upload tools, allowlist examples, and upload destinations.
</Card>
<Card title="Environment variables" href="/environment-variables">
Full runtime contract including SSRF, OTel, and DB vars.
</Card>
<Card title="Build, test, and troubleshooting" href="/build-test-and-troubleshooting">
Feature flags (`db`, `otel`, `sentry`) and E2E verification.
</Card>
</CardGroup>