# Warm Sandbox Queue > The in-memory queue shared between the warm-pool and claim reconcilers that hands off warm sandboxes to incoming claims. - Repository: kubernetes-sigs/agent-sandbox - GitHub: https://github.com/kubernetes-sigs/agent-sandbox - Human wiki: https://grok-wiki.com/public/wiki/kubernetes-sigs-agent-sandbox-c3f2597a654a - Complete Markdown: https://grok-wiki.com/public/wiki/kubernetes-sigs-agent-sandbox-c3f2597a654a/llms-full.txt ## Source Files - `extensions/controllers/queue/simple_sandbox_queue.go` - `extensions/controllers/queue/simple_sandbox_queue_test.go` ---
Relevant source files The following files were used as context for generating this wiki page: - [extensions/controllers/queue/simple_sandbox_queue.go](extensions/controllers/queue/simple_sandbox_queue.go) - [extensions/controllers/queue/simple_sandbox_queue_test.go](extensions/controllers/queue/simple_sandbox_queue_test.go) - [extensions/controllers/sandboxclaim_controller.go](extensions/controllers/sandboxclaim_controller.go) - [cmd/agent-sandbox-controller/main.go](cmd/agent-sandbox-controller/main.go) - [extensions/controllers/utils.go](extensions/controllers/utils.go)
# Warm Sandbox Queue The Warm Sandbox Queue is the in-process handoff structure that lets the `SandboxClaim` controller skip cold-starting a `Sandbox` whenever a `SandboxWarmPool` has already pre-provisioned an adoptable one for the same template. It is a single in-memory data structure, owned by the controller manager process, partitioned by template hash, and shared between the controller's `Sandbox` watch (producer) and the `SandboxClaim` reconciler (consumer). This page covers the public interface, the underlying `sync.Map`-of-FIFOs implementation, how items enter and leave the queue, the concurrency and "Ghost Pod" semantics that drive the design, and how the queue is wired into the controller manager. The queue lives entirely in memory; nothing is persisted, so it is rebuilt from `Watches` on each controller restart. ## Role in the warm-pool handoff There is no Kubernetes work-queue or message bus between the warm pool and claim sides — the queue is just a Go object held by `SandboxClaimReconciler.WarmSandboxQueue` and populated by event handlers registered on the same controller. The producer side is a `Watches(&v1beta1.Sandbox{}, &sandboxEventHandler{...})` registration that observes every `Sandbox` create/update/delete in the cluster; the consumer side is `getCandidate`, which is called from the claim reconcile path before any cold-start logic runs. ```text SandboxWarmPool reconciler SandboxClaim reconciler creates reconciles | | v v +-----------------+ +----------------------+ | Sandbox object | -- watch --> | sandboxEventHandler | | (adoptable) | | .Update -> Add() | +-----------------+ +----------+-----------+ | v +-------------------------------+ | WarmSandboxQueue (in-mem) | | templateHash -> FIFO+dedup | +-------------------------------+ ^ | Get() / RemoveItem() | +------------+--------------+ | getCandidate / | | templateEventHandler | +----------------------------+ ``` Sources: [extensions/controllers/sandboxclaim_controller.go:1286-1298](), [extensions/controllers/sandboxclaim_controller.go:591-646]() ## The `SandboxQueue` interface The queue is defined as a four-method interface so the reconciler depends on behavior, not the concrete map type. `SandboxKey` is a thin alias around `types.NamespacedName`, so a key uniquely identifies a `Sandbox` resource. | Method | Signature | Purpose | |--------|-----------|---------| | `Add` | `Add(templateHash string, item SandboxKey)` | Push an adoptable sandbox onto the per-template FIFO; idempotent on duplicate keys. | | `Get` | `Get(templateHash string) (SandboxKey, bool)` | Pop the front item; `ok=false` when the queue is empty or absent. | | `RemoveItem` | `RemoveItem(templateHash string, item SandboxKey)` | Eagerly drop a specific entry; used when a sandbox is deleted from the cluster. | | `RemoveQueue` | `RemoveQueue(templateHash string)` | Drop an entire per-template queue; used when the owning `SandboxTemplate` is deleted. | ```go // extensions/controllers/queue/simple_sandbox_queue.go type SandboxKey types.NamespacedName type SandboxQueue interface { Add(templateHash string, item SandboxKey) Get(templateHash string) (SandboxKey, bool) RemoveQueue(templateHash string) RemoveItem(templateHash string, item SandboxKey) } ``` Sources: [extensions/controllers/queue/simple_sandbox_queue.go:23-33]() ## `SimpleSandboxQueue` implementation `SimpleSandboxQueue` is the only implementation. It composes two layers of concurrency primitives: - An outer `sync.Map` named `queues`, keyed by `templateHash` (string), valued by `*synchronizedQueue`. This map handles the high-churn read path of "find or create the per-template queue" without a global lock. - An inner `synchronizedQueue` per template, which holds an ordered `[]SandboxKey` slice and a `map[SandboxKey]struct{}` set guarded by a single `sync.Mutex`. The slice gives FIFO ordering for fair adoption; the set gives O(1) deduplication so the same sandbox cannot be queued twice. ```go // extensions/controllers/queue/simple_sandbox_queue.go type SimpleSandboxQueue struct { queues sync.Map // templateHash -> *synchronizedQueue } type synchronizedQueue struct { mu sync.Mutex items []SandboxKey set map[SandboxKey]struct{} // Used for O(1) deduplication } ``` The hash key itself is computed by `SandboxTemplateRefHash`, which delegates to `sandboxcontrollers.NameHash(templateRefName)`. So all consumers and producers agree on partitioning by template name hash, not by template UID. Sources: [extensions/controllers/queue/simple_sandbox_queue.go:35-107](), [extensions/controllers/utils.go:50-58]() ### Push and Pop semantics `Push` is gated on the dedup set: if the key already exists, the call is a no-op. Otherwise it is appended to the tail and inserted into the set. `Pop` removes the head element, clears the slot to release the underlying string allocations to the garbage collector, and reslices. ```go // extensions/controllers/queue/simple_sandbox_queue.go func (q *synchronizedQueue) Push(key SandboxKey) { q.mu.Lock() defer q.mu.Unlock() if _, exists := q.set[key]; !exists { q.set[key] = struct{}{} q.items = append(q.items, key) } } func (q *synchronizedQueue) Pop() (SandboxKey, bool) { q.mu.Lock() defer q.mu.Unlock() if len(q.items) == 0 { return SandboxKey{}, false } item := q.items[0] q.items[0] = SandboxKey{} // release references to the GC q.items = q.items[1:] delete(q.set, item) return item, true } ``` `Add` and `Get` on `SimpleSandboxQueue` wrap these calls with a `LoadOrStore` / `Load` against the outer `sync.Map`. `Add` lazily creates a per-template queue; `Get` returns `false` cleanly if no queue has ever been created for the hash. The basic FIFO contract is verified by `TestSimpleSandboxQueue_BasicOperations` and dedup by `TestSynchronizedQueue_Deduplication`. Sources: [extensions/controllers/queue/simple_sandbox_queue.go:47-58](), [extensions/controllers/queue/simple_sandbox_queue.go:109-140](), [extensions/controllers/queue/simple_sandbox_queue_test.go:20-47](), [extensions/controllers/queue/simple_sandbox_queue_test.go:98-116]() ### Ghost Pod removal `RemoveItem` exists specifically to keep the in-memory queue consistent with cluster state when a `Sandbox` is deleted out from under it (the controller calls these "Ghost Pods"). It scans the slice for the key, shifts subsequent entries left, and explicitly zeroes the tail slot so the removed `SandboxKey` cannot linger in the backing array. The dedup set is updated in lockstep, so a future `Add` for the same key after removal will succeed. ```go // extensions/controllers/queue/simple_sandbox_queue.go func (q *synchronizedQueue) Remove(key SandboxKey) { q.mu.Lock() defer q.mu.Unlock() if _, exists := q.set[key]; !exists { return } delete(q.set, key) for i, k := range q.items { if k == key { last := len(q.items) - 1 copy(q.items[i:], q.items[i+1:]) q.items[last] = SandboxKey{} q.items = q.items[:last] break } } } ``` `TestSimpleSandboxQueue_RemoveItem_GhostPodFix` asserts both the post-removal ordering and that every unused slot in the backing array down to `cap(items)` is the zero-value, preventing the slice from retaining references to deleted sandboxes. Sources: [extensions/controllers/queue/simple_sandbox_queue.go:62-91](), [extensions/controllers/queue/simple_sandbox_queue_test.go:49-96]() ### Queue lifecycle and memory leaks `RemoveQueue` deletes the entire per-template entry from the outer `sync.Map`. It is invoked when a `SandboxTemplate` is deleted, after which no claim will ever again ask for that hash. The comment on `synchronizedQueue` flags an open TODO to also remove queues when their `SandboxWarmPool` is deleted; today, the deletion is template-driven only. ```go // extensions/controllers/queue/simple_sandbox_queue.go // TODO(vicentefb): Implement queue cleanup mechanism. // We should remove the queue from the sync.Map when the corresponding // SandboxWarmPool for a given template is deleted to prevent memory leaks. func (s *SimpleSandboxQueue) RemoveQueue(templateHash string) { s.queues.Delete(templateHash) } ``` The `TestSimpleSandboxQueue_RemoveQueue_MemoryLeakFix` test pins the contract: after `RemoveQueue`, `Get` for the same hash returns `false`. Sources: [extensions/controllers/queue/simple_sandbox_queue.go:93-100](), [extensions/controllers/queue/simple_sandbox_queue.go:142-146](), [extensions/controllers/queue/simple_sandbox_queue_test.go:118-133]() ## Producers: who calls `Add` and `RemoveItem` The queue does not poll the API server. It is filled and pruned by two event handlers attached to `SandboxClaimReconciler` via `Watches`: | Source event | Handler | Action | |--------------|---------|--------| | `Sandbox` Create / Update (transitions to adoptable, or template-hash label change while adoptable) | `sandboxEventHandler.Update` | `Add(templateHash, key)` | | `Sandbox` Delete | `sandboxEventHandler.Delete` | `RemoveItem(templateHash, key)` | | `SandboxTemplate` Delete | `templateEventHandler.Delete` | `RemoveQueue(templateHash)` | `isAdoptable` defines what "adoptable" means: not being deleted, owned by a `SandboxWarmPool` (or unowned), and labeled with both `warmPoolSandboxLabel` and the `sandboxTemplateRefHash` label. The hash used as the queue key is `newSandbox.Labels[sandboxTemplateRefHash]`, i.e. it comes from the live label set rather than being recomputed. ```go // extensions/controllers/sandboxclaim_controller.go if (!oldAdoptable && newAdoptable) || (newAdoptable && hashChanged) { key := queue.SandboxKey{Namespace: newSandbox.Namespace, Name: newSandbox.Name} h.sandboxQueue.Add(newSandbox.Labels[sandboxTemplateRefHash], key) } ``` Sources: [extensions/controllers/sandboxclaim_controller.go:1446-1481](), [extensions/controllers/sandboxclaim_controller.go:1503-1541](), [extensions/controllers/sandboxclaim_controller.go:1543-1566]() ## Consumer: `getCandidate` and the skip-list pattern The claim reconciler pulls from the queue inside `getCandidate`. The shape of the loop is the most important behavior to understand because it explains why the queue tolerates stale entries: 1. Pop a key. If empty, return `(nil, nil)` — the warm pool is exhausted and the caller falls back to cold-start. 2. `Get` the corresponding `Sandbox` from the informer cache. If it is `NotFound`, the entry was a Ghost Pod and is simply skipped (no re-queue). On any other error, the key is `Add`ed back and the error is returned. 3. Run `verifySandboxCandidate`: rejects wrong-namespace, deleted, mislabeled, or wrong-template-hash sandboxes. Cross-namespace matches are added to a local `skipped` list (so they go back into the queue via the `defer`); other rejections are dropped. 4. If the claim asks for a specific warm pool, sandboxes from a different pool are also pushed onto the local `skipped` list and the loop continues. 5. On success, returns the chosen `Sandbox` and its key. A `defer` re-queues anything on `skipped` so other claims can still adopt those entries. ```go // extensions/controllers/sandboxclaim_controller.go var skipped []queue.SandboxKey defer func() { for _, key := range skipped { r.WarmSandboxQueue.Add(templateHash, key) } }() for { adoptedKey, ok := r.WarmSandboxQueue.Get(templateHash) if !ok { return nil, queue.SandboxKey{}, nil } ... if k8errors.IsNotFound(err) { continue // Ghost Pod: silently drop } ... } ``` The adoption finalization path in `adoptSandboxFromCandidates` also re-queues the candidate on `Update` or `completeAdoption` failures (`r.WarmSandboxQueue.Add(templateHash, adoptedKey)`), so a transient API conflict does not permanently remove a usable warm sandbox. Sources: [extensions/controllers/sandboxclaim_controller.go:591-646](), [extensions/controllers/sandboxclaim_controller.go:648-699](), [extensions/controllers/sandboxclaim_controller.go:1487-1519]() ## Wiring in the controller manager A single `SimpleSandboxQueue` is constructed at program start and shared by reference. Because both the producer event handlers and the consumer reconciler hold the same `queue.SandboxQueue` value, there is no need for serialization or fan-out — the in-memory pointer is the integration point. ```go // cmd/agent-sandbox-controller/main.go if extensions { warmSandboxQueue := queue.NewSimpleSandboxQueue() if err = (&extensionscontrollers.SandboxClaimReconciler{ Client: mgr.GetClient(), Scheme: mgr.GetScheme(), WarmSandboxQueue: warmSandboxQueue, ... }).SetupWithManager(mgr, sandboxClaimConcurrentWorkers); err != nil { ... } } ``` Inside `SetupWithManager`, the same `r.WarmSandboxQueue` is passed into both `&sandboxEventHandler{sandboxQueue: r.WarmSandboxQueue}` and `&templateEventHandler{sandboxQueue: r.WarmSandboxQueue}`. The queue is therefore one Go value shared across N reconciler workers (`MaxConcurrentReconciles`), which is exactly why the inner mutex and outer `sync.Map` matter. Because the queue is in-memory only, a controller restart starts with an empty queue. The watch-driven producer reconciles by replaying `Create` events for every existing `Sandbox`, which re-populates the queue for any already-adoptable sandboxes the next time their state is observed. Sources: [cmd/agent-sandbox-controller/main.go:246-257](), [extensions/controllers/sandboxclaim_controller.go:1286-1298]() ## Failure modes the queue is designed for | Failure mode | How the queue copes | |--------------|--------------------| | Same `Sandbox` observed multiple times by the informer | `Push` is a no-op on dedup set hits, so the queue holds at most one entry per key. | | `Sandbox` deleted while still in the queue ("Ghost Pod") | `RemoveItem` from the delete handler eagerly prunes; if missed, `getCandidate` silently drops `NotFound` entries on pop. | | Adoption fails mid-flight (API conflict, transient error) | The reconciler `Add`s the key back, preserving warm-pool capacity. | | Cross-namespace or wrong-pool candidate found | Held in a local `skipped` slice and re-queued via `defer` so other claims can still adopt it. | | `SandboxTemplate` deleted | `templateEventHandler.Delete` calls `RemoveQueue`, wiping the entire per-template queue from the `sync.Map`. | | `SandboxWarmPool` deleted (no template deletion) | Not handled today — see the `TODO(vicentefb)` on `synchronizedQueue`. The per-template queue would persist with stale or never-popped entries until the template itself is deleted. | | Controller restart | Queue is empty; producer event handlers refill it as the informer replays `Sandbox` state. | Sources: [extensions/controllers/queue/simple_sandbox_queue.go:93-99](), [extensions/controllers/sandboxclaim_controller.go:603-645](), [extensions/controllers/sandboxclaim_controller.go:1521-1566]() ## Summary The Warm Sandbox Queue is a deliberately small piece of infrastructure: a `sync.Map` of per-template FIFOs with O(1) deduplication, four methods on its `SandboxQueue` interface, and two short event handlers that keep it in step with the cluster's `Sandbox` and `SandboxTemplate` state. The cleverness lives in the consumer protocol — Ghost Pod tolerance, the `skipped` defer-requeue pattern, and explicit re-`Add` on adoption errors — which together let the queue stay non-authoritative: it is an opportunistic cache of hints about who is adoptable right now, and any inconsistency is reconciled the next time `getCandidate` runs.