# SandboxWarmPool Reconciler

> Pool maintenance loop: parallel batch creation/deletion bounded by max-batch-size, rollout on template changes, and watcher coordination with SandboxClaim.

- Repository: kubernetes-sigs/agent-sandbox
- GitHub: https://github.com/kubernetes-sigs/agent-sandbox
- Human wiki: https://grok-wiki.com/public/wiki/kubernetes-sigs-agent-sandbox-c3f2597a654a
- Complete Markdown: https://grok-wiki.com/public/wiki/kubernetes-sigs-agent-sandbox-c3f2597a654a/llms-full.txt

## Source Files

- `extensions/controllers/sandboxwarmpool_controller.go`
- `extensions/controllers/sandboxwarmpool_controller_test.go`

---

<details>
<summary>Relevant source files</summary>
The following files were used as context for generating this wiki page:
- [extensions/controllers/sandboxwarmpool_controller.go](extensions/controllers/sandboxwarmpool_controller.go)
- [extensions/controllers/sandboxwarmpool_controller_test.go](extensions/controllers/sandboxwarmpool_controller_test.go)
- [extensions/api/v1beta1/sandboxwarmpool_types.go](extensions/api/v1beta1/sandboxwarmpool_types.go)
- [extensions/controllers/sandboxclaim_controller.go](extensions/controllers/sandboxclaim_controller.go)
- [extensions/controllers/utils.go](extensions/controllers/utils.go)
</details>

# SandboxWarmPool Reconciler

The `SandboxWarmPoolReconciler` maintains a pool of pre-allocated `Sandbox` CRs so that a `SandboxClaim` can adopt a warm sandbox instead of paying a full cold-start. Each pool is bound to a `SandboxTemplate`; the reconciler keeps the actual replica count converging on `Spec.Replicas`, garbage-collects sandboxes that fail to become Ready, and rolls forward stale sandboxes when the underlying template drifts. Coordination with `SandboxClaimReconciler` happens through shared labels and a controller-side queue rather than a direct API.

This page covers the reconcile loop, the parallel slow-start batching used for create/delete, the staleness model and rollout strategies, and how the controller hands off sandboxes to claims through ownership transfer.

## Controller Shape and Watches

`SandboxWarmPoolReconciler` owns `SandboxWarmPool` and the `Sandbox` CRs it creates. It also watches `SandboxTemplate` so that any template mutation re-queues every pool that references it. A field indexer on `.spec.sandboxTemplateRef.name` powers the reverse lookup.

```go
// extensions/controllers/sandboxwarmpool_controller.go:548-557
return ctrl.NewControllerManagedBy(mgr).
    For(&extensionsv1beta1.SandboxWarmPool{}).
    Owns(&sandboxv1beta1.Sandbox{}).
    WithOptions(controller.Options{MaxConcurrentReconciles: concurrentWorkers}).
    Watches(
        &extensionsv1beta1.SandboxTemplate{},
        handler.EnqueueRequestsFromMapFunc(r.findWarmPoolsForTemplate),
    ).
    Complete(r)
```

`findWarmPoolsForTemplate` lists pools by the `TemplateRefField` index and returns one `reconcile.Request` per match, so a template change fans out to every dependent pool exactly once.

```mermaid
flowchart LR
    subgraph API["Kubernetes API"]
        WP["SandboxWarmPool"]
        T["SandboxTemplate"]
        S["Sandbox (owned)"]
        SC["SandboxClaim"]
    end
    subgraph WPC["SandboxWarmPoolReconciler"]
        REC["reconcilePool()"]
        FW["findWarmPoolsForTemplate()"]
    end
    subgraph SCC["SandboxClaimReconciler"]
        EH["sandboxEventHandler"]
        Q["WarmSandboxQueue"]
        ADOPT["adoptSandboxFromCandidates"]
    end
    WP -- Reconcile --> REC
    T -- Watch --> FW --> REC
    REC -- Create/Delete/Adopt --> S
    S -- Owns --> WP
    S -- Watch --> EH
    EH -- Add/Remove key --> Q
    SC -- Reconcile --> ADOPT
    ADOPT -- Pop key, Patch ownership --> S
```

Sources: [extensions/controllers/sandboxwarmpool_controller.go:533-583](), [extensions/controllers/sandboxclaim_controller.go:1446-1540]()

## API Surface

`SandboxWarmPoolSpec` is intentionally small: a desired replica count, a template reference, and an optional update strategy. The CRD is `Scale`-subresource enabled, which lets an HPA drive `replicas`.

| Field | Type | Notes |
|---|---|---|
| `spec.replicas` | int32, required, min 0 | Desired pool size; targeted by the `scale` subresource. |
| `spec.sandboxTemplateRef.name` | string | Indexed as `.spec.sandboxTemplateRef.name`; powers the template watch. |
| `spec.updateStrategy.type` | enum `Recreate`/`OnReplenish`, default `OnReplenish` | Controls how stale sandboxes are reconciled. |
| `status.replicas` | int32 | Count of active (non-deleting, owned/adoptable) sandboxes. |
| `status.readyReplicas` | int32 | Count whose `Ready` condition is `True`. |
| `status.selector` | string | Label-selector string used by the `scale` subresource. |

Sources: [extensions/api/v1beta1/sandboxwarmpool_types.go:24-119]()

## Reconcile Pipeline

`Reconcile` itself is thin: fetch the CR, exit on deletion, snapshot status, run `reconcilePool`, then patch status via Server-Side Apply with field owner `warmpool-controller`. The work happens inside `reconcilePool`.

```
reconcilePool steps                                File reference
─────────────────────────────────────────────────  ───────────────────────────
1. NameHash(pool.Name)                  → label    controller.go:108-115
2. List Sandboxes by warmPoolSandboxLabel          controller.go:111-123
3. fetchTemplateAndHash                            controller.go:125-126,312-325
4. filterActiveSandboxes (delete stale + adopt)    controller.go:128-129,239-301
5. Garbage-collect sandboxes stuck non-Ready > 5m  controller.go:131-148
6. Write Replicas / ReadyReplicas / Selector       controller.go:159-169
7. Create (replicas < desired)                     controller.go:174-192
8. Delete (replicas > desired)                     controller.go:195-222
9. Join template error if not NotFound             controller.go:224-228
```

Two non-obvious properties: the template is fetched only once per reconcile (its hash is reused for staleness checks and label propagation), and a missing template (`IsNotFound`) is *swallowed* — the pool stops creating but does not report the missing template as a hard error, while every other template error is surfaced.

Sources: [extensions/controllers/sandboxwarmpool_controller.go:67-229](), [extensions/controllers/sandboxwarmpool_controller.go:312-325]()

### Status Patching

Status writes are skipped when `equality.Semantic.DeepEqual(oldStatus, &warmPool.Status)` holds, then applied as a typed SSA patch with `client.ForceOwnership`. The `nolint:staticcheck` annotation reflects that `client.Apply` is used without generated apply configurations.

Sources: [extensions/controllers/sandboxwarmpool_controller.go:415-443]()

## Parallel Batch Creation and Deletion

Both create and delete loops are bounded by `MaxBatchSize`. The constructor defaults `MaxBatchSize` to `sandboxCreateDeleteMaxBatchSize = 300` if the caller passes zero or negative.

```go
// extensions/controllers/sandboxwarmpool_controller.go:534-537
if r.MaxBatchSize <= 0 {
    r.MaxBatchSize = sandboxCreateDeleteMaxBatchSize
}
```

`min(desiredDelta, maxBatchSize)` caps the per-reconcile delta; the remainder is left for subsequent reconciles. Inside the cap, work runs through `slowStartBatch`, which doubles the parallelism each successful round, starting at one:

```
batch sizes for count=14, initialBatchSize=1:
1 → 2 → 4 → 7  (the last batch is trimmed to `remaining`)
```

This is verified by `TestSlowStartBatch` which checks "all succeed with batch trimming (count=14)" produces 14 successful calls in those four rounds, and "early exit on failure" stops after `1+2+4 = 7` calls when an injected error fires.

Failure semantics:

- An `errgroup.WithContext` collects the first error per batch; on failure the loop returns immediately without launching the next round.
- The returned `successes` includes the partial successes of the failed batch (via `batchSuccesses atomic.Int64`).
- Context cancellation is checked at the top of every round; mid-batch cancellations short-circuit subsequent rounds, as `TestSlowStartBatch` "context canceled in middle of batch" demonstrates.

Sources: [extensions/controllers/sandboxwarmpool_controller.go:48-52](), [extensions/controllers/sandboxwarmpool_controller.go:171-222](), [extensions/controllers/sandboxwarmpool_controller.go:585-621](), [extensions/controllers/sandboxwarmpool_controller_test.go:1461-1537]()

### Deletion Priority

When the pool is over-provisioned, `slices.SortFunc` sorts candidates so unready sandboxes are deleted before ready ones; within each group, newest first. This biases the pool toward keeping ready, settled capacity:

```go
// extensions/controllers/sandboxwarmpool_controller.go:201-211
slices.SortFunc(activeSandboxes, func(a, b sandboxv1beta1.Sandbox) int {
    aReady := isSandboxReady(&a)
    bReady := isSandboxReady(&b)
    if aReady != bReady {
        if aReady { return 1 }
        return -1
    }
    return b.CreationTimestamp.Compare(a.CreationTimestamp.Time)
})
```

`deletePoolSandbox` ignores `NotFound` so racing deletions inside a batch do not abort the rest.

Sources: [extensions/controllers/sandboxwarmpool_controller.go:195-222](), [extensions/controllers/sandboxwarmpool_controller.go:405-413]()

## Sandbox Construction

`buildSandboxCR` builds one blueprint per reconcile and the create loop deep-copies it for each replica. The blueprint:

- Carries three pool-identity labels: `warmPoolSandboxLabel` (hash of pool name), `sandboxTemplateRefHash` (hash of template name), `SandboxPodTemplateHashLabel` (hash of pod template JSON).
- Annotates the sandbox with `SandboxTemplateRefAnnotation` for metrics linkage.
- Propagates the same three labels into the pod template labels so platform informers can target the pods.
- Copies `VolumeClaimTemplates` from the template (verified by `TestCreatePoolSandboxPropagatesVolumeClaimTemplates`).
- Calls `ApplySandboxSecureDefaults` to set `AutomountServiceAccountToken: false` when unset and, under "Secure By Default" (`NetworkPolicyManagement` empty or `Managed` and `NetworkPolicy == nil`), pins `DNSPolicy: None` with public resolvers `8.8.8.8` and `1.1.1.1`.
- Sets the `SandboxWarmPool` as `OwnerReference` (controller=true) so deletions cascade.

Names are server-generated via `GenerateName: "<poolName>-"`, so the same blueprint can be created N times concurrently.

Sources: [extensions/controllers/sandboxwarmpool_controller.go:327-403](), [extensions/controllers/utils.go:23-48](), [extensions/controllers/sandboxwarmpool_controller_test.go:459-648]()

## Staleness, Adoption, and Rollout

`filterActiveSandboxes` walks the listed sandboxes and partitions them into "kept" vs. "must delete or skip":

```mermaid
flowchart TD
    A[Sandbox candidate] --> B{DeletionTimestamp set?}
    B -- yes --> SKIP[skip]
    B -- no --> C{controllerRef?}
    C -- foreign --> SKIP2[ignore – belongs to other controller]
    C -- nil (orphan) --> D
    C -- this pool --> E
    D[Vet staleness via comparePodSpecs] --> F{stale?}
    E{strategy == Recreate?} -- yes --> G[Vet staleness]
    E -- no, OnReplenish --> KEEP
    G --> F
    F -- yes --> DEL[Delete sandbox]
    F -- no, orphan --> ADOPT[SetControllerReference + Update]
    F -- no, owned --> KEEP[append to activeSandboxes]
    ADOPT --> KEEP
```

Sources: [extensions/controllers/sandboxwarmpool_controller.go:239-301](), [extensions/controllers/sandboxwarmpool_controller.go:231-237]()

### isSandboxStale

`isSandboxStale` is layered for cost and security:

1. If `sandboxTemplateRefHash` label does not match `SandboxTemplateRefHash(template.Name)`, the sandbox is stale immediately — covers the "template ref rename" case proven by `TestReconcilePool_TemplateRefUpdate_SameSpec` (sandboxes are recreated even if the pod spec is identical).
2. Orphans always run the full semantic comparison (`comparePodSpecs`) regardless of the hash label, because an unowned sandbox could be carrying a spoofed hash with a mutated PodSpec. `TestIsSandboxStale_OrphanedSandboxVetting` exercises both the spoofed and genuine paths.
3. Otherwise, if `SandboxPodTemplateHashLabel` equals the freshly computed template hash, the sandbox is fresh.
4. If hash computation failed (`currentPodTemplateHash == ""`), the function returns *not stale* and logs, to avoid mass-deleting pods because of a transient marshal failure.
5. As a final fallback, `comparePodSpecs` runs and the verdict is memoized in `vettedHashes` so two sandboxes with the same drift hash do not pay the cost twice.

`comparePodSpecs` defends against false positives by re-applying `ApplySandboxSecureDefaults` to a copy of the template's pod spec before the `equality.Semantic.DeepEqual` — without that normalization, every fresh sandbox would appear stale because the controller injected `AutomountServiceAccountToken` or DNS fields the template did not set. `TestComparePodSpecsNormalization` pins these cases.

Sources: [extensions/controllers/sandboxwarmpool_controller.go:462-531](), [extensions/controllers/sandboxwarmpool_controller_test.go:1194-1296](), [extensions/controllers/sandboxwarmpool_controller_test.go:1384-1459](), [extensions/controllers/sandboxwarmpool_controller_test.go:1027-1137]()

### Update Strategies

| Strategy | Behavior on template drift | Behavior on orphans |
|---|---|---|
| `Recreate` | Stale owned sandboxes are deleted every reconcile; pool replenishes at the next cycle. | Always vetted; stale orphans deleted. |
| `OnReplenish` (default, also fallback for unknown values) | Owned sandboxes keep their old spec until they are deleted (manually or by claim adoption); only then is the replacement created from the current template. | Orphans are still vetted and may be deleted. |

`TestReconcilePool_TemplateUpdateRollout` verifies both directions: after `image-v1 → image-v2`, `Recreate` flips every replica to `image-v2`, while `OnReplenish` keeps `image-v1` and only the replenishment sandbox observes `image-v2`. The same test asserts an empty `Spec.UpdateStrategy.Type` is treated as `OnReplenish`; the `switch` at controller.go:253-262 also defaults *unknown* values to `OnReplenish` with a log line.

Sources: [extensions/controllers/sandboxwarmpool_controller.go:247-262](), [extensions/api/v1beta1/sandboxwarmpool_types.go:49-69](), [extensions/controllers/sandboxwarmpool_controller_test.go:858-1025]()

### Stuck-Sandbox Garbage Collection

Independent of staleness, any non-Ready sandbox older than the constant `warmPoolReadinessGracePeriod = 5 * time.Minute` is deleted in-line. This prevents image-pull or scheduling failures from monopolizing pool slots; `TestReconcilePoolGCStuckSandboxes` covers both the "delete after grace period" and "keep within grace period" cases.

Sources: [extensions/controllers/sandboxwarmpool_controller.go:131-148](), [extensions/controllers/sandboxwarmpool_controller_test.go:755-856]()

## Coordination with SandboxClaim

The warm pool reconciler does not actively hand sandboxes to claims. Instead, `SandboxClaimReconciler` runs a `sandboxEventHandler` against the same `Sandbox` watch and maintains an in-memory `WarmSandboxQueue` keyed by `sandboxTemplateRefHash`. The contract is enforced through `isAdoptable` and three pool-managed labels:

```go
// extensions/controllers/sandboxclaim_controller.go:1503-1519
if !candidate.DeletionTimestamp.IsZero() { ... }
if _, ok := candidate.Labels[warmPoolSandboxLabel]; !ok { ... }
if _, ok := candidate.Labels[sandboxTemplateRefHash]; !ok { ... }
if controllerRef != nil && controllerRef.Kind != "SandboxWarmPool" { ... }
```

When a sandbox transitions from non-adoptable to adoptable (or its template-ref hash changes), the handler adds the key to the queue; on Delete the key is purged. The claim reconciler pops keys from the queue, verifies the sandbox is still adoptable, and then `completeAdoption` strips the three pool labels and re-parents the sandbox to the claim:

```go
// extensions/controllers/sandboxclaim_controller.go:732-741
delete(adopted.Labels, warmPoolSandboxLabel)
delete(adopted.Labels, sandboxTemplateRefHash)
delete(adopted.Labels, v1beta1.SandboxPodTemplateHashLabel)
adopted.OwnerReferences = nil
if err := controllerutil.SetControllerReference(claim, adopted, r.Scheme); err != nil { ... }
```

Because the pool's `List` is gated on `warmPoolSandboxLabel`, removing that label is exactly what causes the next `reconcilePool` to treat the adoption as a missing replica and create a replacement — that is the entire "replenish" signal. For `WarmPoolPolicyNone` claims, the claim reconciler skips this path and creates a cold sandbox directly.

```mermaid
sequenceDiagram
    participant WPR as SandboxWarmPoolReconciler
    participant API as Kube API
    participant CEH as sandboxEventHandler
    participant Q as WarmSandboxQueue
    participant SCR as SandboxClaimReconciler

    WPR->>API: Create Sandbox (labels: warmPool, refHash, podHash; owner=WarmPool)
    API-->>CEH: CreateEvent
    CEH->>Q: Add(refHash, key) if adoptable
    SCR->>Q: Get(refHash) → key
    SCR->>API: Patch Sandbox (drop labels, owner=Claim)
    API-->>WPR: Watch event (Sandbox no longer matches selector)
    WPR->>API: List → currentReplicas < desired → Create replacement
```

Sources: [extensions/controllers/sandboxclaim_controller.go:591-746](), [extensions/controllers/sandboxclaim_controller.go:1446-1540](), [extensions/controllers/sandboxclaim_controller.go:1155-1180](), [extensions/controllers/sandboxwarmpool_controller.go:111-123]()

## Failure Modes and Error Aggregation

Errors are accumulated with `errors.Join` rather than short-circuiting. Stale-delete failures, adoption failures, stuck-GC failures, batch-create failures, and batch-delete failures all surface together at the end of `reconcilePool`, while `Reconcile` returns the joined error so controller-runtime requeues with backoff.

The one swallowed error is `IsNotFound` on the template fetch: the controller logs, refuses to *create* (`tmplErr == nil` gates `buildSandboxCR`), but still updates status and may continue to *delete* excess sandboxes. Other template errors are joined into the returned error.

Sources: [extensions/controllers/sandboxwarmpool_controller.go:128-229](), [extensions/controllers/sandboxwarmpool_controller.go:445-460]()

## Summary

The `SandboxWarmPool` reconciler is a small set-reconciliation loop with three notable properties: bounded-parallel batching via `slowStartBatch` that doubles concurrency on success and stops on first error, a layered staleness check that combines cheap label-hash comparisons with a memoized semantic-equal fallback (normalized through `ApplySandboxSecureDefaults` to suppress drift caused by the controller's own injected fields), and a coordination protocol with `SandboxClaim` that relies entirely on label/ownership transitions — the warm pool produces labelled sandboxes, the claim reconciler watches the same objects and strips the labels on adoption, and the resulting "missing replica" reconcile is what drives replenishment.