# Metrics & Sandbox Collector

> Reconciler latency/result metrics plus the custom Prometheus collector that surfaces per-sandbox phase, age, and warm-pool stats.

- Repository: kubernetes-sigs/agent-sandbox
- GitHub: https://github.com/kubernetes-sigs/agent-sandbox
- Human wiki: https://grok-wiki.com/public/wiki/kubernetes-sigs-agent-sandbox-c3f2597a654a
- Complete Markdown: https://grok-wiki.com/public/wiki/kubernetes-sigs-agent-sandbox-c3f2597a654a/llms-full.txt

## Source Files

- `internal/metrics/metrics.go`
- `internal/metrics/sandbox_collector.go`
- `internal/metrics/metrics_test.go`
- `internal/metrics/sandbox_collector_test.go`

---

<details>
<summary>Relevant source files</summary>
The following files were used as context for generating this wiki page:
- [internal/metrics/metrics.go](internal/metrics/metrics.go)
- [internal/metrics/sandbox_collector.go](internal/metrics/sandbox_collector.go)
- [internal/metrics/metrics_test.go](internal/metrics/metrics_test.go)
- [internal/metrics/sandbox_collector_test.go](internal/metrics/sandbox_collector_test.go)
- [cmd/agent-sandbox-controller/main.go](cmd/agent-sandbox-controller/main.go)
- [extensions/controllers/sandboxclaim_controller.go](extensions/controllers/sandboxclaim_controller.go)
- [api/v1beta1/sandbox_types.go](api/v1beta1/sandbox_types.go)
</details>

# Metrics & Sandbox Collector

The `internal/metrics` package owns every Prometheus surface that the `agent-sandbox-controller` exposes beyond the metrics that controller-runtime emits automatically. It bundles two independent concerns: a small set of histogram and counter recorders that the `SandboxClaim` reconciler calls when a claim crosses lifecycle boundaries, and a custom Prometheus collector (`SandboxCollector`) that derives point-in-time `agent_sandboxes` gauge series on every scrape by listing `Sandbox` objects through the controller-runtime cache.

This page documents both halves, the labels they emit, where reconciler code drives them, and the trade-offs the collector makes when it runs against a large cache.

## Package Layout and Registration

All metrics, descriptors, and the collector live in a single package and register themselves into the global `controller-runtime` Prometheus registry that backs the metrics HTTP server configured by the controller manager.

| File | Responsibility |
|---|---|
| `internal/metrics/metrics.go` | Declares histogram, counter, and gauge vectors; defines launch-type / annotation constants; registers everything in `init()`; provides `Record*` helpers. |
| `internal/metrics/sandbox_collector.go` | Defines `SandboxCollector`, the `AgentSandboxesDesc` descriptor, its `AgentSandboxesMetricKey` aggregation key, and `RegisterSandboxCollector`. |
| `internal/metrics/tracing.go` | Unrelated OpenTelemetry `Instrumenter` (out of scope for this page, but shares the package). |
| `internal/metrics/metrics_test.go`, `sandbox_collector_test.go`, `testmain_test.go` | Recorder unit tests, fake-client collector tests, and a `goleak` `TestMain`. |

Vector metrics and `BuildInfo` are registered up-front in `init()` so they are visible on `/metrics` even before any reconcile has occurred. The custom collector is registered later, from `main.go`, once the controller manager has a usable client:

```go
// cmd/agent-sandbox-controller/main.go:234
// Register the custom Sandbox metric collector globally.
asmetrics.RegisterSandboxCollector(mgr.GetClient(), mgr.GetLogger().WithName("sandbox-collector"))
```

`RegisterSandboxCollector` is idempotent: it swallows `prometheus.AlreadyRegisteredError` and logs at info level instead, so re-invocations during test setup or manager restarts do not crash the process.

Sources: [internal/metrics/metrics.go:38-139](internal/metrics/metrics.go), [internal/metrics/sandbox_collector.go:61-71](internal/metrics/sandbox_collector.go), [cmd/agent-sandbox-controller/main.go:179-234](cmd/agent-sandbox-controller/main.go)

## Component Map

```mermaid
flowchart LR
    subgraph Reconciler["extensions/controllers/sandboxclaim_controller.go"]
        Init["initializeAnnotations<br/>stamps ObservabilityAnnotation"]
        Adopt["tryAdoptSandbox<br/>warm path"]
        ColdCreate["create cold Sandbox"]
        ReadyTx["recordCreationLatencyMetric<br/>on Ready transition"]
    end

    subgraph MetricsPkg["internal/metrics"]
        Helpers["Record* helpers<br/>metrics.go"]
        Vectors["HistogramVec / CounterVec / GaugeFunc<br/>metrics.go"]
        Collector["SandboxCollector<br/>sandbox_collector.go"]
        Desc["AgentSandboxesDesc"]
    end

    subgraph CR["controller-runtime"]
        Reg["metrics.Registry"]
        Server["metricsserver on :metricsAddr"]
    end

    APIServer[("Kubernetes API<br/>v1beta1.SandboxList")]

    Init --> Helpers
    Adopt --> Helpers
    ColdCreate --> Helpers
    ReadyTx --> Helpers
    Helpers --> Vectors
    Vectors --> Reg
    Collector --> Desc
    Collector -->|"List with<br/>UnsafeDisableDeepCopy"| APIServer
    Collector --> Reg
    Reg --> Server
```

The reconciler never touches the collector; the collector never touches the reconciler. They meet only through Prometheus' scrape protocol via the shared `metrics.Registry`.

Sources: [internal/metrics/metrics.go:132-139](internal/metrics/metrics.go), [internal/metrics/sandbox_collector.go:62-107](internal/metrics/sandbox_collector.go), [extensions/controllers/sandboxclaim_controller.go:285-300](extensions/controllers/sandboxclaim_controller.go)

## Reconciler-Driven Metrics

Four vector metrics and one build-info gauge are pre-registered in `init()` and updated through small package-level helpers. The helpers all live in `metrics.go` and exist so callers do not have to know the exact label order.

### Metric Catalog

| Name | Type | Labels | Helper | What it measures |
|---|---|---|---|---|
| `agent_sandbox_claim_startup_latency_ms` | HistogramVec | `launch_type`, `sandbox_template` | `RecordClaimStartupLatency` | End-to-end ms from the *webhook* first observing the claim to `SandboxClaim` Ready. |
| `agent_sandbox_claim_controller_startup_latency_ms` | HistogramVec | `launch_type`, `sandbox_template` | `RecordClaimControllerStartupLatency` | ms from the *controller* first observing the claim to `SandboxClaim` Ready (excludes admission lag). |
| `agent_sandbox_creation_latency_ms` | HistogramVec | `namespace`, `launch_type`, `sandbox_template` | `RecordSandboxCreationLatency` | ms from `Sandbox` creation to underlying Pod Ready. For warm launches this collapses to controller synchronization overhead. |
| `agent_sandbox_claim_creation_total` | CounterVec | `namespace`, `sandbox_template`, `launch_type`, `warmpool_name`, `pod_condition` | `RecordSandboxClaimCreation` | One increment per claim that produces a Sandbox; emitted at warm adoption and at cold creation. |
| `agent_sandbox_build_info` | GaugeFunc, constant `1` | `git_version`, `git_commit`, `build_date`, `go_version`, `compiler`, `platform` | n/a (constant) | Build provenance, populated from `internal/version` at process start. |

Latency histograms share two carefully tuned bucket sets. Claim histograms span `100ms`..`240000ms`; sandbox-creation histograms extend further to `600000ms` (10 minutes) because they cover cold Pod scheduling and image pull:

```go
// internal/metrics/metrics.go:48
Buckets: []float64{100, 250, 500, 750, 1000, 1250, 1500, 2000, 2500, 5000, 10000, 30000, 60000, 120000, 240000},
```

Sources: [internal/metrics/metrics.go:38-130](internal/metrics/metrics.go), [internal/metrics/metrics.go:141-161](internal/metrics/metrics.go)

### Launch-Type Vocabulary

Three constants normalize the `launch_type` label across every metric:

```go
// internal/metrics/metrics.go:26
LaunchTypeWarm    = "warm"    // Pod from a SandboxWarmPool
LaunchTypeCold    = "cold"    // Pod not from a SandboxWarmPool
LaunchTypeUnknown = "unknown" // Used when Sandbox is nil during failure
```

The claim controller derives the value from the resulting `Sandbox`: a non-empty `agents.x-k8s.io/pod-name` annotation marks an adopted warm pod; absence of that annotation marks a cold creation; a nil `*Sandbox` (typically during failure paths) falls back to `unknown`. See `getLaunchType` in `extensions/controllers/sandboxclaim_controller.go:1335-1343`.

### Where Recorders Fire

Two timing annotations bracket the claim lifecycle. The webhook stamps `agents.x-k8s.io/webhook-first-observed-at` (`WebhookAnnotation`) when it first admits the claim, and `SandboxClaimReconciler.initializeAnnotations` stamps `agents.x-k8s.io/controller-first-observed-at` (`ObservabilityAnnotation`) on the first reconcile pass:

```go
// extensions/controllers/sandboxclaim_controller.go:287
needObservabilityPatch := claim.Annotations[asmetrics.ObservabilityAnnotation] == ""
...
claim.Annotations[asmetrics.ObservabilityAnnotation] = timestamp.Format(time.RFC3339Nano)
```

When the claim later transitions to Ready, the reconciler reads those annotations, parses them as `time.RFC3339Nano`, guards against parse failures and negative durations, and calls the matching helper. The `RecordSandboxCreationLatency` helper also keys off `Sandbox.CreationTimestamp` and the Ready condition's `LastTransitionTime`, so it can be computed even for warm launches where the pod predates the claim. See `recordClaimStartupLatency`, `recordControllerStartupLatency`, and `recordSandboxCreationLatency` in `extensions/controllers/sandboxclaim_controller.go:1345-1395`.

`RecordSandboxClaimCreation` is invoked from two different places in the reconciler — once when warm adoption succeeds and once when a cold Sandbox is created — and uses the `pod_condition` label (`ready` vs. `not_ready`) to distinguish whether the adopted warm pod was already Ready at handoff:

```go
// extensions/controllers/sandboxclaim_controller.go:710
asmetrics.RecordSandboxClaimCreation(claim.Namespace, claim.Spec.TemplateRef.Name,
    asmetrics.LaunchTypeWarm, poolName, podCondition)
// extensions/controllers/sandboxclaim_controller.go:1065
asmetrics.RecordSandboxClaimCreation(claim.Namespace, claim.Spec.TemplateRef.Name,
    asmetrics.LaunchTypeCold, "none", "not_ready")
```

Sources: [internal/metrics/metrics.go:26-36](internal/metrics/metrics.go), [extensions/controllers/sandboxclaim_controller.go:285-300](extensions/controllers/sandboxclaim_controller.go), [extensions/controllers/sandboxclaim_controller.go:706-710](extensions/controllers/sandboxclaim_controller.go), [extensions/controllers/sandboxclaim_controller.go:1334-1395](extensions/controllers/sandboxclaim_controller.go)

### Latency Recording Sequence

```mermaid
sequenceDiagram
    autonumber
    participant W as Admission Webhook
    participant API as kube-apiserver
    participant R as SandboxClaim Reconciler
    participant M as metrics.Record*
    participant P as Prometheus /metrics

    W->>API: stamp WebhookAnnotation on create
    API-->>R: watch event (claim)
    R->>API: initializeAnnotations<br/>stamp ObservabilityAnnotation
    R->>R: adopt warm Sandbox<br/>or create cold Sandbox
    R->>M: RecordSandboxClaimCreation<br/>(namespace, template, launch_type,<br/>warmpool_name, pod_condition)
    Note over R,API: ...Sandbox transitions to Ready...
    R->>M: RecordClaimStartupLatency<br/>(now - WebhookAnnotation)
    R->>M: RecordClaimControllerStartupLatency<br/>(now - ObservabilityAnnotation)
    R->>M: RecordSandboxCreationLatency<br/>(SandboxReady - Sandbox.CreationTimestamp)
    P-->>M: scrape
```

Sources: [extensions/controllers/sandboxclaim_controller.go:285-300](extensions/controllers/sandboxclaim_controller.go), [extensions/controllers/sandboxclaim_controller.go:1345-1395](extensions/controllers/sandboxclaim_controller.go), [internal/metrics/metrics.go:141-161](internal/metrics/metrics.go)

### Build Info

`BuildInfo` is a `prometheus.NewGaugeFunc` that always returns `1` and carries `git_version`, `git_commit`, `build_date`, `go_version`, `compiler`, and `platform` as constant labels resolved from `sigs.k8s.io/agent-sandbox/internal/version`. `TestBuildInfo` pins the exact `# HELP`/`# TYPE`/sample line that `/metrics` must produce, so any change to the label set or help string fails the build.

Sources: [internal/metrics/metrics.go:112-129](internal/metrics/metrics.go), [internal/metrics/metrics_test.go:101-111](internal/metrics/metrics_test.go)

## The Custom Sandbox Collector

`SandboxCollector` implements the `prometheus.Collector` interface (`Describe` + `Collect`) and is the only path that produces the `agent_sandboxes` gauge. Unlike the reconciler-driven vectors, this metric is *not* maintained incrementally; each Prometheus scrape recomputes the full set of label combinations by listing live `Sandbox` objects.

### Descriptor and Aggregation Key

The descriptor and its companion key struct establish the label contract:

```go
// internal/metrics/metrics.go:105
AgentSandboxesDesc = prometheus.NewDesc(
    "agent_sandboxes",
    "Monitor the point-in-time number of sandboxes in the cluster.",
    []string{"namespace", "ready_condition", "expired", "launch_type", "sandbox_template", "owned_by"},
    nil,
)
```

```go
// internal/metrics/sandbox_collector.go:37
type AgentSandboxesMetricKey struct {
    Namespace      string
    ReadyCondition string
    Expired        string
    LaunchType     string
    Template       string
    OwnedBy        string
}
```

`NewAgentSandboxesConstMetric` projects a `(count, key)` pair into a `prometheus.MustNewConstMetric` of type `GaugeValue`. Using `ConstMetric` is what lets the collector emit a series only when it observes one — there is no risk of stale label combinations lingering after the underlying Sandboxes disappear, which is the failure mode a long-lived `GaugeVec` would have here.

Sources: [internal/metrics/metrics.go:97-110](internal/metrics/metrics.go), [internal/metrics/sandbox_collector.go:37-59](internal/metrics/sandbox_collector.go)

### Collect Pipeline

`Collect` is a four-stage pipeline executed under a 5-second per-scrape deadline (`metricsCollectTimeout`). The collector applies `client.UnsafeDisableDeepCopy` to the `List` call to skip the per-object deep copy controller-runtime normally performs when serving cached reads; the source comments document why this is safe and why a `GaugeVec` was deliberately rejected:

```go
// internal/metrics/sandbox_collector.go:94
// Collect fetches sandboxes, calculates labels, and sends metrics to the channel.
// UnsafeDisableDeepCopy avoids O(N) deep-copy overhead on every scrape; safe here because
// Collect only reads fields for label aggregation and never mutates or retains the objects.
// A GaugeVec updated in the Reconcile loop would be more performant (O(1) per scrape),
// but this is a known trade-off to keep the Reconcile loop simpler.
```

```text
            ┌───────────────────────────────────────────────────────────┐
            │ Collect(ch)                                              │
            ├───────────────────────────────────────────────────────────┤
            │ 1. ctx with 5s timeout                                    │
            │ 2. client.List(&SandboxList, UnsafeDisableDeepCopy)       │
            │      │                                                     │
            │      └─▶ on error: logger.Error + return (no series)       │
            │ 3. for each sandbox:                                      │
            │      derive (namespace, ready_condition, expired,         │
            │              launch_type, sandbox_template, owned_by)     │
            │      counts[key]++                                        │
            │ 4. for key,count := range counts:                         │
            │      ch <- NewAgentSandboxesConstMetric(count, key)       │
            └───────────────────────────────────────────────────────────┘
```

If `List` fails, the collector logs and returns without emitting any series for that scrape — Prometheus simply observes the absence and continues. There is no fall-back to a cached snapshot.

Sources: [internal/metrics/sandbox_collector.go:32-34](internal/metrics/sandbox_collector.go), [internal/metrics/sandbox_collector.go:89-162](internal/metrics/sandbox_collector.go)

### Label Derivation Rules

Each Sandbox is mapped to exactly one `AgentSandboxesMetricKey`, then the keys are aggregated by count:

| Label | Default | Becomes non-default when… | Source |
|---|---|---|---|
| `namespace` | `sandbox.Namespace` | — | `sandbox_collector.go:149` |
| `ready_condition` | `"false"` | `Ready` condition has `Status == ConditionTrue` | `sandbox_collector.go:111-117` |
| `expired` | `"false"` | `Ready` condition has `Reason == SandboxReasonExpired` (`"SandboxExpired"`) | `sandbox_collector.go:111-121`, `api/v1beta1/sandbox_types.go:53-54` |
| `launch_type` | `"cold"` | `SandboxPodNameAnnotation` (`agents.x-k8s.io/pod-name`) is set and non-empty (warm-pool adoption) | `sandbox_collector.go:123-126`, `api/v1beta1/sandbox_types.go:56-57` |
| `sandbox_template` | `"unknown"` | `SandboxTemplateRefAnnotation` (`agents.x-k8s.io/sandbox-template-ref`) is set and non-empty | `sandbox_collector.go:128-133`, `api/v1beta1/sandbox_types.go:58-59` |
| `owned_by` | `"None"` | Controller `OwnerReference.APIVersion` matches `extensions/v1beta1.GroupVersion` and `Kind` is `"SandboxClaim"` or `"SandboxWarmPool"` | `sandbox_collector.go:135-146` |

Two subtleties are explicit in code: an `expired` reading depends on the `Ready` condition being present at all (a missing condition leaves both `ready_condition` and `expired` at `"false"`), and `owned_by` only recognizes controllers from the `agent-sandbox` extensions group — third-party controllers wrapping a Sandbox would surface as `"None"`. A user-created Sandbox without the template annotation defaults to `sandbox_template="unknown"`, called out in a comment at `sandbox_collector.go:129-130`.

Sources: [internal/metrics/sandbox_collector.go:109-156](internal/metrics/sandbox_collector.go), [api/v1beta1/sandbox_types.go:35-59](api/v1beta1/sandbox_types.go)

### Verified Cases from the Tests

`TestSandboxCollector` builds a fake client with hand-crafted Sandboxes and asserts the exact set of `agent_sandboxes` series produced. The matrix below summarizes what each case proves about the derivation rules above:

| Test case | Inputs | Asserted series |
|---|---|---|
| `single ready cold unknown sandbox` | `Ready=True`, no annotations, no owner | `namespace=default ready=true expired=false launch=cold template=unknown owned_by=None → 1` |
| `missing ready condition` | `Conditions: nil` | Both `ready_condition` and `expired` default to `"false"` |
| `mixed sandboxes` | 4 Sandboxes, one warm + expired in `test-ns`, two duplicate cold not-ready in `default` | 3 distinct series; the two duplicates collapse to the same key and count `2` |
| `claimed sandbox` | controller OwnerReference `Kind=SandboxClaim` | `owned_by=SandboxClaim` |
| `warmpool sandbox` | controller OwnerReference `Kind=SandboxWarmPool` | `owned_by=SandboxWarmPool` |

The `mixed sandboxes` case in particular pins the aggregation contract — Sandboxes with identical label tuples must collapse into a single gauge sample whose value is the count.

Sources: [internal/metrics/sandbox_collector_test.go:39-256](internal/metrics/sandbox_collector_test.go)

## Operational Notes

- **Metrics endpoint.** The controller manager runs the Prometheus HTTP server on the address passed to `--metrics-bind-address`. `metricsserver.Options{BindAddress: metricsAddr}` is the only wiring, so the standard controller-runtime flags govern auth, TLS, and `ExtraHandlers` (`pprof` is mounted on the same listener when `--enable-pprof` is on). See `cmd/agent-sandbox-controller/main.go:179-214`.
- **Scrape cost.** Every scrape lists *all* Sandboxes through the cache. The `UnsafeDisableDeepCopy` opt-in keeps that cost to a pointer walk over already-cached objects, but very large clusters still pay an O(N) iteration per scrape; the source explicitly chose this over the O(1) `GaugeVec` alternative to keep the Reconcile loop simpler.
- **Per-scrape timeout.** A 5-second deadline (`metricsCollectTimeout`) bounds the List call. Exceeding it produces a logged error and an empty scrape, not a partial series set.
- **Idempotent registration.** `RegisterSandboxCollector` tolerates `prometheus.AlreadyRegisteredError`, which matters when tests or restart paths re-invoke registration against the global registry.
- **Leak discipline.** `internal/metrics/testmain_test.go` wraps the package's tests with `goleak.VerifyTestMain`, so any goroutine leaked by the collector or the OTel instrumenter would fail the suite.

Sources: [internal/metrics/sandbox_collector.go:32-107](internal/metrics/sandbox_collector.go), [cmd/agent-sandbox-controller/main.go:179-234](cmd/agent-sandbox-controller/main.go), [internal/metrics/testmain_test.go:1-25](internal/metrics/testmain_test.go)

## Summary

The metrics package is intentionally split along a push/pull seam. The reconciler pushes timing and counting events into pre-registered histograms and counters via thin `Record*` helpers, with claim lifecycle bracketed by two well-defined RFC3339Nano annotations stamped by the webhook and the controller's first reconcile. In parallel, `SandboxCollector` pulls a fresh snapshot of all Sandboxes on every Prometheus scrape, aggregates them by `(namespace, ready_condition, expired, launch_type, sandbox_template, owned_by)`, and emits `agent_sandboxes` as a set of `ConstMetric` gauge samples — accepting an O(N) per-scrape List, with `UnsafeDisableDeepCopy` softening the cost, in exchange for a Reconcile loop that never needs to bookkeep cardinality.