.svc.cluster.local: ── no router headers
or http://: if use_pod_ip ── no router headers
LocalTunnel rejected at __init__ (ValueError)
```
The `_inject_router_headers` flag is `False` only for `SandboxInClusterConnectionConfig`. In all other modes, every request carries `X-Sandbox-ID`, `X-Sandbox-Namespace`, `X-Sandbox-Port`, and (when available) `X-Sandbox-Pod-IP`, which lets the router route to the correct backend.
Sources: [clients/python/agentic-sandbox-client/k8s_agent_sandbox/async_connector.py:38-91](), [clients/python/agentic-sandbox-client/k8s_agent_sandbox/async_connector.py:125-149]()
### Pod-IP resolution and the auth latch
The connector caches a pod IP for routing speed, but it must defend against permission errors:
- First time around it calls the `get_pod_ip` callback (provided by `AsyncSandbox.get_pod_ip`).
- If the call raises and the underlying response is `401`/`403`, `_pod_ip_auth_failed` is **permanently** set on this client instance — pod-IP routing is disabled for its lifetime to avoid hammering the K8s API with a token that cannot read sandboxes.
- Transient errors are logged at debug and retried on later requests.
For `SandboxInClusterConnectionConfig`, `_resolve_base_url` upgrades the URL once pod-IP resolution succeeds, otherwise it falls back to cluster DNS.
Sources: [clients/python/agentic-sandbox-client/k8s_agent_sandbox/async_connector.py:69-149]()
### Retry and cache invalidation
`send_request` retries up to `MAX_RETRIES = 5` times on `{500, 502, 503, 504}` with exponential backoff (`BACKOFF_FACTOR * 2**attempt`, i.e. 0.5 s, 1 s, 2 s, …). On any HTTP error it raises `SandboxRequestError` and clears caches that may have gone stale:
- For `SandboxGatewayConnectionConfig`, the `_base_url` is dropped so the next request re-queries the gateway IP.
- Pod-IP caching state (`_pod_ip_resolved`, `_cached_pod_ip_url`, `_pod_ip`) is reset.
`close()` calls `httpx.AsyncClient.aclose()` and clears the same caches.
```mermaid
sequenceDiagram
participant Caller as AsyncCommandExecutor/AsyncFilesystem
participant Conn as AsyncSandboxConnector
participant K8s as AsyncK8sHelper
participant Router as Sandbox router / pod
Caller->>Conn: send_request(METHOD, endpoint, ...)
Conn->>Conn: _resolve_base_url()
alt Gateway mode, no cached IP
Conn->>K8s: wait_for_gateway_ip(...)
K8s-->>Conn: ip_address
else InCluster + use_pod_ip
Conn->>K8s: get_pod_ip via callback
K8s-->>Conn: podIPs[0] or None
end
Conn->>Router: httpx.AsyncClient.request(...)
alt 5xx and attempt < 5
Router-->>Conn: retryable status
Conn->>Conn: asyncio.sleep(0.5 * 2^attempt)
Conn->>Router: retry
end
Router-->>Caller: httpx.Response (or SandboxRequestError)
```
Sources: [clients/python/agentic-sandbox-client/k8s_agent_sandbox/async_connector.py:33-208]()
## `AsyncK8sHelper`
`AsyncK8sHelper` wraps `kubernetes_asyncio` and is shared by the client and all of its sandboxes through `AsyncSandbox.k8s_helper`. Initialization is lazy and once-only: an `asyncio.Lock` guards `_ensure_initialized`, which tries in-cluster config first and falls back to `load_kube_config()`. `close()` shuts down the shared `ApiClient` and resets the latch so a later call would re-initialize cleanly.
### CRD operations
| Method | Resource | Behavior |
| --- | --- | --- |
| `create_sandbox_claim(name, template, namespace, annotations, labels, lifecycle, warmpool)` | `SandboxClaim` | Builds a manifest with `spec.sandboxTemplateRef`, optional `spec.lifecycle` and `spec.warmpool`; calls `create_namespaced_custom_object`. |
| `resolve_sandbox_name(claim_name, namespace, timeout)` | `SandboxClaim` (watch) | Watches the claim until `status.sandbox.name` is populated (warm-pool adoption may produce a different name from the claim). Surfaces `Ready=False` + `TemplateNotFound` as `SandboxTemplateNotFoundError`. Supports both `name` (post-rename) and legacy `Name` keys. |
| `wait_for_sandbox_ready(name, namespace, timeout)` | `Sandbox` (watch) | Watches until a condition with `type=Ready, status=True`. Returns the first `status.podIPs` entry (or `None` on older controllers). |
| `delete_sandbox_claim(name, namespace)` | `SandboxClaim` | `404` swallowed; other API errors raise `SandboxNotFoundError`. |
| `get_sandbox`, `get_sandbox_claim` | both | Return the raw object dict or `None` on `404`. |
| `list_sandbox_claims(namespace, label_selector)` | `SandboxClaim` | Returns `metadata.name` for each item, optionally filtered by selector. |
| `wait_for_gateway_ip(gateway_name, namespace, timeout)` | `Gateway` (watch) | Watches the Gateway custom resource for `status.addresses[0].value`. |
All watch-based methods (`resolve_sandbox_name`, `wait_for_sandbox_ready`, `wait_for_gateway_ip`) follow the same pattern: compute a remaining-time budget, open a `watch.Watch()`, iterate `w.stream(...)` with `timeout_seconds=remaining`, and always `await w.close()` in a `finally` block to release the streaming HTTP connection — important because `kubernetes_asyncio` watches hold a live connection until closed.
Sources: [clients/python/agentic-sandbox-client/k8s_agent_sandbox/async_k8s_helper.py:37-331]()
## `AsyncCommandExecutor`
The executor is a thin wrapper over `connector.send_request("POST", "execute", json={"command": ...}, timeout=...)`. The response is parsed into the shared `ExecutionResult` Pydantic model (`stdout`, `stderr`, `exit_code`); decode or schema failures are raised as `RuntimeError` so callers can distinguish them from `SandboxRequestError`. The whole `run` method is wrapped in `@async_trace_span("run")`, with `sandbox.command` and `sandbox.exit_code` recorded as span attributes when tracing is recording.
```python
# clients/python/agentic-sandbox-client/k8s_agent_sandbox/commands/async_command_executor.py:30-56
@async_trace_span("run")
async def run(self, command: str, timeout: int = 60) -> ExecutionResult:
...
response = await self.connector.send_request("POST", "execute", json={"command": command}, timeout=timeout)
...
return ExecutionResult(**response.json())
```
Sources: [clients/python/agentic-sandbox-client/k8s_agent_sandbox/commands/async_command_executor.py:20-57](), [clients/python/agentic-sandbox-client/k8s_agent_sandbox/models.py:18-22]()
## `AsyncFilesystem`
`AsyncFilesystem` mirrors the four core sync filesystem methods, all decorated with `@async_trace_span(...)`:
| Method | HTTP | Notes |
| --- | --- | --- |
| `write(path, content, timeout, allow_unsafe_paths)` | `POST /upload` | Encodes `str` content as UTF-8 bytes; routes through `Filesystem._safe_upload_path` unless `allow_unsafe_paths=True`. |
| `read(path, timeout, allow_unsafe_paths)` | `GET /download/{path}` | Same path sanitization gate. Returns `response.content` bytes. |
| `list(path, timeout)` | `GET /list/{path}` | Parses into `list[FileEntry]`. |
| `exists(path, timeout)` | `GET /exists/{path}` | Returns the `exists` boolean from the JSON body. |
The path-safety contract is shared with the sync class via direct call into `Filesystem._safe_upload_path` (re-implementing it asynchronously would risk drift). The accompanying comment explains why `os.path.basename` is not sufficient — `basename("foo\x00../etc/passwd")` returns the string unchanged because of the embedded NUL, which would then truncate at the C layer when handed to the OS. The hardened sanitizer rejects empty / bare-`.`, embedded NUL and ASCII control characters, and any `..` segment after normalisation. URL-path interpolation goes through `urllib.parse.quote(path, safe="")` on every read/list/exists call.
Sources: [clients/python/agentic-sandbox-client/k8s_agent_sandbox/files/async_filesystem.py:24-139]()
## Optional dependency boundary
The async stack is opt-in. The top-level `__init__.py` imports `AsyncSandboxClient` from `async_sandbox_client`; on `ImportError` it replaces the symbol with a stub class whose `__init__` raises a guided `ImportError` telling the caller to install the `async` extra. This keeps the sync surface usable on minimal installations while still letting users `from k8s_agent_sandbox import AsyncSandboxClient` and discover the missing-extras error at instantiation time rather than at import time.
```python
# clients/python/agentic-sandbox-client/k8s_agent_sandbox/__init__.py:26-35
try:
from .async_sandbox_client import AsyncSandboxClient
except ImportError:
class AsyncSandboxClient: # type: ignore[no-redef]
def __init__(self, *args, **kwargs):
raise ImportError(
"AsyncSandboxClient requires the 'async' extras. "
"Install with: pip install k8s-agent-sandbox[async]"
)
```
Sources: [clients/python/agentic-sandbox-client/k8s_agent_sandbox/__init__.py:26-35]()
## Sync vs. Async at a glance
The two stacks share models, label-validation rules, path sanitization, and tracing decorators. The differences worth knowing before choosing:
| Aspect | Sync | Async |
| --- | --- | --- |
| Local development via `kubectl port-forward` | Supported (`SandboxLocalTunnelConnectionConfig`) | **Not supported** — connector rejects it; client refuses `connection_config=None` |
| Process-exit safety net | `atexit` cleanup | None; must use `async with` or explicit `delete_all` + `close` |
| HTTP transport | `requests` | `httpx.AsyncClient` (transport-level retries=3) |
| K8s API | `kubernetes` | `kubernetes_asyncio` (watches must be closed in `finally`) |
| Mutual exclusion | `threading.Lock` | `asyncio.Lock` |
| Cancellation handling on create | Exceptions | `Exception | CancelledError`, claim cleanup under `asyncio.shield` |
Sources: [clients/python/agentic-sandbox-client/k8s_agent_sandbox/async_sandbox_client.py:50-173](), [clients/python/agentic-sandbox-client/k8s_agent_sandbox/async_connector.py:55-91](), [clients/python/agentic-sandbox-client/k8s_agent_sandbox/async_k8s_helper.py:107-207]()
## Putting it together
A minimal end-to-end async session — provision a sandbox, run a command, upload and download a file, and let the context manager clean up — exercises every layer described above:
```python
from k8s_agent_sandbox import AsyncSandboxClient
from k8s_agent_sandbox.models import SandboxDirectConnectionConfig
async def main():
config = SandboxDirectConnectionConfig(api_url="http://router.example")
async with AsyncSandboxClient(connection_config=config) as client:
sandbox = await client.create_sandbox(
template="python-sandbox-template",
shutdown_after_seconds=600,
)
result = await sandbox.commands.run("echo hello")
await sandbox.files.write("/tmp/note.txt", "hi")
data = await sandbox.files.read("/tmp/note.txt")
# __aexit__ -> delete_all() -> client.close() releases all claims
```
The `async with` block is what makes this safe: it guarantees `delete_all()` runs even if the body raises, which is the only reliable cleanup path given the deliberate absence of an `atexit` fallback.
Sources: [clients/python/agentic-sandbox-client/k8s_agent_sandbox/async_sandbox_client.py:41-105](), [clients/python/agentic-sandbox-client/k8s_agent_sandbox/async_sandbox_client.py:107-173]()
---
## 23. Python Extensions, Gateway & Sandbox Router
> Optional Python add-ons: computer-use extension, GKE pod-snapshot extensions, the sandbox-router service, and the kind-based gateway harness.
- Page Markdown: https://grok-wiki.com/public/wiki/kubernetes-sigs-agent-sandbox-c3f2597a654a/pages/23-python-extensions-gateway-sandbox-router.md
- Generated: 2026-05-25T22:49:37.367Z
### Source Files
- `clients/python/agentic-sandbox-client/k8s_agent_sandbox/extensions/computer_use.py`
- `clients/python/agentic-sandbox-client/k8s_agent_sandbox/gke_extensions`
- `clients/python/agentic-sandbox-client/sandbox-router/sandbox_router.py`
- `clients/python/agentic-sandbox-client/sandbox-router/README.md`
- `clients/python/agentic-sandbox-client/gateway-kind/README.md`
Relevant source files
The following files were used as context for generating this wiki page:
- [clients/python/agentic-sandbox-client/k8s_agent_sandbox/extensions/computer_use.py](clients/python/agentic-sandbox-client/k8s_agent_sandbox/extensions/computer_use.py)
- [clients/python/agentic-sandbox-client/k8s_agent_sandbox/extensions/README.md](clients/python/agentic-sandbox-client/k8s_agent_sandbox/extensions/README.md)
- [clients/python/agentic-sandbox-client/k8s_agent_sandbox/gke_extensions/snapshots/podsnapshot_client.py](clients/python/agentic-sandbox-client/k8s_agent_sandbox/gke_extensions/snapshots/podsnapshot_client.py)
- [clients/python/agentic-sandbox-client/k8s_agent_sandbox/gke_extensions/snapshots/sandbox_with_snapshot_support.py](clients/python/agentic-sandbox-client/k8s_agent_sandbox/gke_extensions/snapshots/sandbox_with_snapshot_support.py)
- [clients/python/agentic-sandbox-client/k8s_agent_sandbox/gke_extensions/snapshots/snapshot_engine.py](clients/python/agentic-sandbox-client/k8s_agent_sandbox/gke_extensions/snapshots/snapshot_engine.py)
- [clients/python/agentic-sandbox-client/k8s_agent_sandbox/gke_extensions/snapshots/README.md](clients/python/agentic-sandbox-client/k8s_agent_sandbox/gke_extensions/snapshots/README.md)
- [clients/python/agentic-sandbox-client/sandbox-router/sandbox_router.py](clients/python/agentic-sandbox-client/sandbox-router/sandbox_router.py)
- [clients/python/agentic-sandbox-client/sandbox-router/sandbox_router.yaml](clients/python/agentic-sandbox-client/sandbox-router/sandbox_router.yaml)
- [clients/python/agentic-sandbox-client/sandbox-router/gateway.yaml](clients/python/agentic-sandbox-client/sandbox-router/gateway.yaml)
- [clients/python/agentic-sandbox-client/sandbox-router/README.md](clients/python/agentic-sandbox-client/sandbox-router/README.md)
- [clients/python/agentic-sandbox-client/sandbox-router/Dockerfile](clients/python/agentic-sandbox-client/sandbox-router/Dockerfile)
- [clients/python/agentic-sandbox-client/gateway-kind/README.md](clients/python/agentic-sandbox-client/gateway-kind/README.md)
- [clients/python/agentic-sandbox-client/gateway-kind/gateway-kind.yaml](clients/python/agentic-sandbox-client/gateway-kind/gateway-kind.yaml)
- [clients/python/agentic-sandbox-client/gateway-kind/run-test-kind.sh](clients/python/agentic-sandbox-client/gateway-kind/run-test-kind.sh)
# Python Extensions, Gateway & Sandbox Router
The Python client ships a small but layered set of optional add‑ons that live outside the core `SandboxClient`/`Sandbox` pair. Two of them are pure client subclasses — the `computer_use` extension and the GKE Pod Snapshot extension — that bolt extra HTTP verbs or Kubernetes operations onto a sandbox handle without changing core code. The other two are deployment surfaces: a FastAPI reverse proxy (`sandbox-router`) that fronts the cluster on a single Gateway IP, and a `gateway-kind` harness that wires the same routing into a local KinD cluster via `cloud-provider-kind`. Together they form the optional perimeter that turns the in‑cluster CRDs from the controller into something a Python program can reach from outside the cluster, and that lets sandboxes do more than execute shell commands.
This page covers what each piece is, how they compose, and the contracts they expose. The router is the linchpin: it is required by the `computer_use` extension, by Gateway Mode in the SDK, and by the KinD harness.
## High-level Topology
The four pieces fit into one request path and one in‑cluster control path, both of which the SDK can exercise from a developer laptop.
```mermaid
flowchart LR
subgraph Client["Python SDK process"]
CU["ComputerUseSandboxClient
(extensions.computer_use)"]
PS["PodSnapshotSandboxClient
(gke_extensions.snapshots)"]
Base["SandboxClient / Sandbox"]
end
subgraph Gateway["Edge"]
GKE["GKE Gateway
(gateway.yaml)"]
KIND["KinD Gateway
(gateway-kind.yaml +
cloud-provider-kind)"]
end
subgraph Router["sandbox-router (FastAPI)"]
Svc["sandbox-router-svc
ClusterIP :8080"]
Pods["router pods
uvicorn sandbox_router:app"]
end
subgraph K8s["agent-sandbox in-cluster"]
SBox["Sandbox Pod
id.ns.svc.cluster.local:port"]
Snap["PodSnapshot /
PodSnapshotManualTrigger CRs"]
end
CU -- "HTTP POST /agent + X-Sandbox-* headers" --> GKE
CU -. "alternate path" .-> KIND
GKE --> Svc --> Pods --> SBox
KIND --> Svc
Base -- "kube-apiserver" --> SBox
PS -- "Custom Objects API" --> Snap
Snap -- "manages" --> SBox
```
Sources: [clients/python/agentic-sandbox-client/sandbox-router/sandbox_router.py:74-128](), [clients/python/agentic-sandbox-client/sandbox-router/gateway.yaml:1-52](), [clients/python/agentic-sandbox-client/gateway-kind/gateway-kind.yaml:1-32](), [clients/python/agentic-sandbox-client/k8s_agent_sandbox/extensions/computer_use.py:20-39]()
## Computer-Use Extension (`extensions/computer_use.py`)
The computer-use extension is the smallest of the four components: a pair of subclasses that wire a single new HTTP verb (`POST /agent`) onto the sandbox handle. It is meant for sandboxes that run a virtual-desktop / browser agent runtime inside the pod, where the SDK does not want to exec shells but instead post a natural-language query.
`SandboxWithComputerUseSupport` extends `Sandbox` and adds `agent(query: str, timeout: int = 60) -> ExecutionResult`. The method checks `self.is_active`, builds a `{"query": query}` payload, sends it through `self.connector` as an HTTP `POST` to the `agent` path, and parses the JSON body back into an `ExecutionResult` (so callers get the same `stdout`/`stderr`/`exit_code` shape as `execute()`). The whole method is decorated with `@trace_span("agent_query")` so that traces emitted by the SDK include a span for each agent call.
`ComputerUseSandboxClient` is a typed `SandboxClient[SandboxWithComputerUseSupport]` that simply overrides `sandbox_class`. Because the base client constructs and re‑hydrates handles through this class attribute, any sandbox returned by `create_sandbox(...)` or `get_sandbox(...)` is already a `SandboxWithComputerUseSupport` instance with `.agent()` available.
```python
# clients/python/agentic-sandbox-client/k8s_agent_sandbox/extensions/computer_use.py
class SandboxWithComputerUseSupport(Sandbox):
@trace_span("agent_query")
def agent(self, query: str, timeout: int = 60) -> ExecutionResult:
if not self.is_active:
raise ConnectionError("Sandbox is not active. Cannot execute agent queries.")
payload = {"query": query}
response = self.connector.send_request("POST", "agent", json=payload, timeout=timeout)
response_data = response.json()
return ExecutionResult(**(response_data or {}))
```
The extension does not provision the runtime itself — the README points users at `examples/gemini-cu-sandbox/sandbox-gemini-computer-use.yaml` for the `SandboxTemplate` and at a `gemini-api-key` `Secret` for credentials.
Sources: [clients/python/agentic-sandbox-client/k8s_agent_sandbox/extensions/computer_use.py:15-39](), [clients/python/agentic-sandbox-client/k8s_agent_sandbox/extensions/README.md:6-43]()
## GKE Pod Snapshot Extension (`gke_extensions/snapshots/`)
The GKE Pod Snapshot extension wraps Google's experimental `podsnapshot.gke.io` CRDs into a sandbox‑centric API. It assumes a GKE Standard cluster running gVisor with the `PodSnapshot` controller installed; if the CRDs are missing the client refuses to construct.
### Class layout
```mermaid
classDiagram
class SandboxClient~T~
class Sandbox
class PodSnapshotSandboxClient {
+sandbox_class = SandboxWithSnapshotSupport
-_check_snapshot_crd_installed() bool
}
class SandboxWithSnapshotSupport {
-_snapshots: SnapshotEngine
+snapshots: SnapshotEngine
+suspend(snapshot_before_suspend, wait_timeout) SuspendResponse
+resume(wait_timeout) ResumeResponse
+is_suspended() bool
+is_restored_from_snapshot(uid) RestoreCheckResult
+terminate()
}
class SnapshotEngine {
+create(trigger_name, timeout) SnapshotResponse
+list(filter_by) ListSnapshotResult
+delete(uid, timeout) DeleteSnapshotResult
+delete_all(delete_by, label_value, timeout) DeleteSnapshotResult
+delete_manual_triggers(max_retries)
}
SandboxClient <|-- PodSnapshotSandboxClient
Sandbox <|-- SandboxWithSnapshotSupport
SandboxWithSnapshotSupport o--> SnapshotEngine
PodSnapshotSandboxClient ..> SandboxWithSnapshotSupport : sandbox_class
```
`PodSnapshotSandboxClient.__init__` calls `_check_snapshot_crd_installed()` which lists API resources under `PODSNAPSHOT_API_GROUP`/`PODSNAPSHOT_API_VERSION` and looks for `PODSNAPSHOT_API_KIND`. It treats `403` and `404` from the API server as "not installed" and raises `RuntimeError("Pod Snapshot Controller is not ready. ...")` otherwise.
Sources: [clients/python/agentic-sandbox-client/k8s_agent_sandbox/gke_extensions/snapshots/podsnapshot_client.py:24-63]()
### `SnapshotEngine` — CRD wrapper
`SnapshotEngine` is the only piece that touches `podsnapshot.gke.io` CRDs directly. `create()` builds a `PodSnapshotManualTrigger` manifest, sanitizes the trigger name into K8s-safe characters (`lower`, `_`→`-`, truncated to 38 characters before a `-{timestamp}-{suffix}` tail to stay under the 63‑char limit, defaulted to `snap` if empty), creates it via the Custom Objects API, and then blocks on `wait_for_snapshot_to_be_completed` starting from the freshly returned `resourceVersion` to avoid losing a watch event. Created trigger names are appended to `self.created_manual_triggers` so that `delete_manual_triggers(max_retries=3)` can later sweep them up on `terminate()`.
`list()` always scopes by `SANDBOX_NAME_HASH_LABEL={hash}`, optionally ANDs grouping labels, and sorts by `creationTimestamp` descending. `_execute_deletion()` is shared by `delete(uid)` and `delete_all(delete_by="all"|"labels")`; it calls `delete_namespaced_custom_object` followed by `wait_for_snapshot_deletion` to confirm removal.
Sources: [clients/python/agentic-sandbox-client/k8s_agent_sandbox/gke_extensions/snapshots/snapshot_engine.py:87-261](), [clients/python/agentic-sandbox-client/k8s_agent_sandbox/gke_extensions/snapshots/snapshot_engine.py:397-547]()
### Suspend / Resume lifecycle
`SandboxWithSnapshotSupport` maps suspend/resume onto `spec.replicas` of the `Sandbox` CR.
```mermaid
stateDiagram-v2
[*] --> Running
Running --> Snapshotting : suspend(snapshot_before_suspend=True)
Snapshotting --> Suspending : SnapshotResponse.success
Snapshotting --> Failed : snapshot failed
Running --> Suspending : suspend(snapshot_before_suspend=False)
Suspending --> Suspended : pod terminated\n(wait_for_pod_termination)
Suspending --> Failed : timeout
Suspended --> Resuming : resume()
Resuming --> Restored : pod ready &&\nis_restored_from_snapshot(uid).success
Resuming --> NotRestored : pod ready, no prior snapshot
Resuming --> Failed : timeout / verification fail
Restored --> Running
NotRestored --> Running
```
`suspend()` first cheaply checks `is_suspended()` (defined as `spec.replicas == 0` on the `Sandbox` CR, with informational logs when `podIPs` lag the spec). It then forces a resolution of `get_sandbox_name_hash()` *before* scaling — once `replicas` hits zero the pod is gone and that label can no longer be read — and only then optionally calls `snapshots.create("suspend-{sandbox_id}")`, patches `spec.replicas=0`, and waits for the pod to terminate by UID. `resume()` is symmetric: it captures `_get_latest_snapshot_uid()` *before* scaling back to 1, then after `wait_for_pod_ready` it calls `is_restored_from_snapshot(uid)` to verify the restore actually happened. If no prior snapshot existed it returns success with `restored_from_snapshot=False` rather than treating it as an error. `terminate()` always calls `_snapshots.delete_manual_triggers()` in a `try/finally` so cleanup runs even if termination throws.
Sources: [clients/python/agentic-sandbox-client/k8s_agent_sandbox/gke_extensions/snapshots/sandbox_with_snapshot_support.py:48-133](), [clients/python/agentic-sandbox-client/k8s_agent_sandbox/gke_extensions/snapshots/sandbox_with_snapshot_support.py:144-329]()
### Result types
| Type | Defined in | Fields |
|---|---|---|
| `SuspendResponse` | `sandbox_with_snapshot_support.py` | `success`, `snapshot_response`, `error_reason`, `error_code` |
| `ResumeResponse` | `sandbox_with_snapshot_support.py` | `success`, `restored_from_snapshot`, `snapshot_uid`, `error_reason`, `error_code` |
| `SnapshotResponse` | `snapshot_engine.py` | `success`, `trigger_name`, `snapshot_uid`, `snapshot_timestamp`, `error_reason`, `error_code` |
| `ListSnapshotResult` | `snapshot_engine.py` | `success`, `snapshots: list[SnapshotDetail]`, `error_reason`, `error_code` |
| `DeleteSnapshotResult` | `snapshot_engine.py` | `success`, `deleted_snapshots: list[str]`, `error_reason`, `error_code` |
| `SnapshotFilter` | `snapshot_engine.py` | `ready_only=True`, `grouping_labels`, `extra="forbid"` |
All responses use a `SUCCESS_CODE = 0` / `ERROR_CODE = 1` convention shared across the module so callers can branch on `error_code` or on the boolean `success`.
Sources: [clients/python/agentic-sandbox-client/k8s_agent_sandbox/gke_extensions/snapshots/snapshot_engine.py:30-86](), [clients/python/agentic-sandbox-client/k8s_agent_sandbox/gke_extensions/snapshots/sandbox_with_snapshot_support.py:28-47]()
## `sandbox-router` Service
`sandbox-router` is a FastAPI + httpx + uvicorn process whose only job is "look at headers, build an in‑cluster URL, stream the request through". It exists because creating an `HTTPRoute` per ephemeral sandbox would not scale; a single static route on the Gateway hands every request to this service instead.
### Request handling
```mermaid
sequenceDiagram
participant Client
participant Gateway as Gateway (gateway.yaml / gateway-kind.yaml)
participant Router as sandbox-router pod
participant Sandbox as Sandbox pod
Client->>Gateway: HTTP request
X-Sandbox-ID, X-Sandbox-Namespace,
X-Sandbox-Port, X-Sandbox-Pod-IP?
Gateway->>Router: HTTPRoute -> sandbox-router-svc:8080
Router->>Router: validate headers
sanitize namespace
parse port
alt X-Sandbox-Pod-IP present
Router->>Sandbox: http://{pod_ip}:{port}/{path}
else fallback
Router->>Sandbox: http://{id}.{ns}.svc.{cluster_domain}:{port}/{path}
end
Sandbox-->>Router: response (streamed)
Router-->>Gateway: StreamingResponse
Gateway-->>Client: response
```
The single catch‑all route at `@app.api_route("/{full_path:path}", methods=['GET','POST','PUT','DELETE','PATCH'])` enforces the contract:
- `X-Sandbox-ID` is required; missing returns `HTTP 400 "X-Sandbox-ID header is required."`.
- `X-Sandbox-Namespace` defaults to `default`; the router rejects anything where `namespace.replace("-", "").isalnum()` is false, explicitly to "prevent DNS injection".
- `X-Sandbox-Port` defaults to `8888` and must parse as `int`.
- `X-Sandbox-Pod-IP`, when present, short-circuits DNS and is used as the target host directly; otherwise the router constructs `f"{sandbox_id}.{namespace}.svc.{cluster_domain}"`.
- The original `Host` header is filtered out before forwarding; all other headers are preserved.
- Upstream connection failures map to `HTTP 502` ("Could not connect to the backend sandbox"), other exceptions to `HTTP 500`.
The response is returned as a `StreamingResponse` over `resp.aiter_bytes()` so large or long-lived bodies do not have to be buffered in router memory. A separate `GET /healthz` always returns `{"status":"ok"}` and is what the GKE `HealthCheckPolicy` and the pod's readiness/liveness probes hit.
Sources: [clients/python/agentic-sandbox-client/sandbox-router/sandbox_router.py:68-137]()
### Configuration
The router reads two environment variables at import time and logs the effective values.
| Variable | Default | Behavior |
|---|---|---|
| `PROXY_TIMEOUT_SECONDS` | `180.0` | Passed to the shared `httpx.AsyncClient(timeout=...)`. Non‑numeric or non‑positive values log a warning and fall back to the default. |
| `CLUSTER_DOMAIN` | `cluster.local` | Used when building the in‑cluster DNS target. Empty strings log a warning and fall back to the default. |
```python
# clients/python/agentic-sandbox-client/sandbox-router/sandbox_router.py
cluster_domain = _get_cluster_domain()
proxy_timeout = _get_proxy_timeout()
client = httpx.AsyncClient(timeout=proxy_timeout)
```
Sources: [clients/python/agentic-sandbox-client/sandbox-router/sandbox_router.py:26-65](), [clients/python/agentic-sandbox-client/sandbox-router/README.md:46-66]()
### Deployment manifest
`sandbox_router.yaml` defines the `sandbox-router-svc` `ClusterIP` (port `8080` → targetPort `8080`) and a `sandbox-router-deployment` with `replicas: 2` for HA, a `topologySpreadConstraints` block on `topology.kubernetes.io/zone`, readiness/liveness probes against `/healthz`, and `securityContext: { runAsUser: 1000, runAsGroup: 1000 }`. The image reference is templated as `${ROUTER_IMAGE}` and resolved by the deployment scripts via `sed`. The container itself comes from the `Dockerfile`, which installs hashed dependencies (`pip install --no-cache-dir --require-hashes -r requirements.txt`), drops to UID `1000`, exposes `8080`, and runs `uvicorn sandbox_router:app --host 0.0.0.0 --port 8080`.
Sources: [clients/python/agentic-sandbox-client/sandbox-router/sandbox_router.yaml:1-70](), [clients/python/agentic-sandbox-client/sandbox-router/Dockerfile:1-22]()
### Front-door Gateway (GKE)
`gateway.yaml` provisions:
- A `gateway.networking.k8s.io/v1` `Gateway` named `external-http-gateway` using `gatewayClassName: gke-l7-global-external-managed`.
- A single `HTTPRoute` (`sandbox-router-route`) with `path.type=PathPrefix, value=/` that points every request at `sandbox-router-svc:8080`.
- A GKE‑specific `networking.gke.io/v1 HealthCheckPolicy` that overrides the GKE Gateway's default `/` health check to use `/healthz` against the same service.
The README is explicit that the manifest is GKE-specific and that other implementations require swapping the `gatewayClassName` and dropping `HealthCheckPolicy`.
Sources: [clients/python/agentic-sandbox-client/sandbox-router/gateway.yaml:1-52](), [clients/python/agentic-sandbox-client/sandbox-router/README.md:68-78]()
## `gateway-kind` Harness
`gateway-kind/` is the local-developer equivalent of `gateway.yaml`. KinD has no native load balancer, so this directory pairs `cloud-provider-kind` (installed via `make deploy-cloud-provider-kind`) with a `Gateway` whose `gatewayClassName: cloud-provider-kind` lets that controller assign a real IP.
`gateway-kind.yaml` is intentionally minimal: a `v1 Gateway` named `kind-gateway` listening on port `80/HTTP` with `allowedRoutes.namespaces.from: All`, plus a `v1beta1 HTTPRoute` (`sandbox-router-route`) routing `PathPrefix: /` to `sandbox-router-svc:8080`. There is no GKE‑specific health-check policy here; KinD's load balancer does not need one.
The README walks operators through the manual flow (`make build`, `make deploy-kind EXTENSIONS=true`, `make deploy-cloud-provider-kind`, applying the router and template manifests with the image tag substituted in, then `kubectl apply -f gateway-kind.yaml`) and confirms success by polling `kubectl get gateway` until `ADDRESS` is populated and finally running `python ../test_client.py --gateway-name="kind-gateway"`.
`run-test-kind.sh` automates the same sequence: it resolves an `IMAGE_TAG` via the `dev/tools/shared/utils.get_image_tag` helper, exports `SANDBOX_ROUTER_IMG=kind.local/sandbox-router:${IMAGE_TAG}` and `SANDBOX_PYTHON_RUNTIME_IMG=kind.local/python-runtime-sandbox:${IMAGE_TAG}`, applies the templated manifests, `kubectl rollout status` waits for the router deployment, `kubectl wait --for=condition=Programmed=True gateway/kind-gateway` waits for the gateway, then spins up a `.venv`, installs the SDK in editable mode, and runs `python3 ./test_client.py --gateway-name kind-gateway`. A `trap cleanup EXIT` tears down the venv, `cloud-provider-kind`, and the KinD cluster.
Sources: [clients/python/agentic-sandbox-client/gateway-kind/gateway-kind.yaml:1-32](), [clients/python/agentic-sandbox-client/gateway-kind/README.md:1-86](), [clients/python/agentic-sandbox-client/gateway-kind/run-test-kind.sh:17-99]()
## How the Pieces Compose
```text
+-------------------------+ +------------------------------+
| ComputerUseSandboxClient| | PodSnapshotSandboxClient |
| (subclass of | | (subclass of SandboxClient, |
| SandboxClient) | | asserts PodSnapshot CRDs) |
+-----------+-------------+ +--------------+---------------+
| sandbox_class | sandbox_class
v v
+-------------------------+ +------------------------------+
| SandboxWithComputerUse | | SandboxWithSnapshotSupport |
| .agent(query) -> POST | | .snapshots: SnapshotEngine |
| /agent | | .suspend()/.resume() |
+-----------+-------------+ +--------------+---------------+
| HTTP via SDK connector | kube-apiserver
v v
+------------------------------+ +------------------------------+
| sandbox-router (FastAPI) | | podsnapshot.gke.io CRDs |
| Service: sandbox-router-svc | | (PodSnapshot, |
| Route: /{full_path:path} | | PodSnapshotManualTrigger) |
+--------------+---------------+ +------------------------------+
^
| HTTPRoute parentRef
|
+----------------------------------+
| Gateway (gke-l7-global-external- |
| managed) | (cloud-provider-kind) |
+----------------------------------+
```
The two SDK extensions are independent — they subclass different things and never talk to each other — but they share infrastructure assumptions. The computer-use path needs the router and a Gateway because the SDK reaches the in‑pod agent over HTTP; the snapshot path does not need the router for snapshotting itself (it talks to the API server), but a real workflow normally pairs it with the same Gateway+router so the resumed sandbox is still reachable. `gateway-kind` exists exclusively to let a developer reproduce the GKE Gateway path locally on KinD, so a `python test_client.py --gateway-name=kind-gateway` exercises the same router code that production does.
In short: the SDK extensions teach the `Sandbox` handle new tricks (a new HTTP verb, a CRD-driven lifecycle), the router gives the cluster a single front door that scales to thousands of ephemeral sandboxes by routing on a header, and `gateway-kind` makes that same front door work without a cloud load balancer.
Sources: [clients/python/agentic-sandbox-client/sandbox-router/README.md:1-25](), [clients/python/agentic-sandbox-client/k8s_agent_sandbox/extensions/computer_use.py:35-39](), [clients/python/agentic-sandbox-client/k8s_agent_sandbox/gke_extensions/snapshots/podsnapshot_client.py:24-44]()
---
## 24. Helm Chart Layout
> Structure of the Helm chart: CRD shipping, deployment template, controller-args helper, RBAC bindings, and values knobs that map to controller flags.
- Page Markdown: https://grok-wiki.com/public/wiki/kubernetes-sigs-agent-sandbox-c3f2597a654a/pages/24-helm-chart-layout.md
- Generated: 2026-05-25T22:46:58.183Z
### Source Files
- `helm/Chart.yaml`
- `helm/values.yaml`
- `helm/templates/deployment.yaml`
- `helm/templates/_controller-args.tpl`
- `helm/README.md`
- `helm/crds/agents.x-k8s.io_sandboxes.yaml`
Relevant source files
The following files were used as context for generating this wiki page:
- [helm/Chart.yaml](helm/Chart.yaml)
- [helm/values.yaml](helm/values.yaml)
- [helm/README.md](helm/README.md)
- [helm/templates/deployment.yaml](helm/templates/deployment.yaml)
- [helm/templates/_controller-args.tpl](helm/templates/_controller-args.tpl)
- [helm/templates/_helpers.tpl](helm/templates/_helpers.tpl)
- [helm/templates/serviceaccount.yaml](helm/templates/serviceaccount.yaml)
- [helm/templates/service.yaml](helm/templates/service.yaml)
- [helm/templates/namespace.yaml](helm/templates/namespace.yaml)
- [helm/templates/rbac.generated.yaml](helm/templates/rbac.generated.yaml)
- [helm/templates/extensions-rbac.generated.yaml](helm/templates/extensions-rbac.generated.yaml)
- [helm/templates/clusterrolebinding.yaml](helm/templates/clusterrolebinding.yaml)
- [helm/templates/clusterrolebinding-extensions.yaml](helm/templates/clusterrolebinding-extensions.yaml)
- [helm/crds/agents.x-k8s.io_sandboxes.yaml](helm/crds/agents.x-k8s.io_sandboxes.yaml)
- [helm/crds/extensions.agents.x-k8s.io_sandboxclaims.yaml](helm/crds/extensions.agents.x-k8s.io_sandboxclaims.yaml)
- [helm/crds/extensions.agents.x-k8s.io_sandboxtemplates.yaml](helm/crds/extensions.agents.x-k8s.io_sandboxtemplates.yaml)
- [helm/crds/extensions.agents.x-k8s.io_sandboxwarmpools.yaml](helm/crds/extensions.agents.x-k8s.io_sandboxwarmpools.yaml)
# Helm Chart Layout
The `helm/` directory ships the agent-sandbox controller as a self-contained Helm chart named `agent-sandbox` (`type: application`, `version: 0.1.0`). The chart bundles the four custom resource definitions used by the controller, a single `Deployment` for the controller pod, a `Service` exposing the metrics endpoint, RBAC primitives (a `ServiceAccount`, two generated `ClusterRole`s, and the matching `ClusterRoleBinding`s), an optional `Namespace`, and a controller-args helper template that translates `values.yaml` knobs into command-line flags. This page maps every file in the chart to the runtime object it produces and shows how `controller.*` values become `args:` on the controller container.
The chart is intentionally small: there are no subcharts, no hooks, and no shared configmaps. Templating is concentrated in two named templates (`agent-sandbox.controllerArgs` and the helpers in `_helpers.tpl`); every other file is a thin wrapper around a single Kubernetes object.
Sources: [helm/Chart.yaml:1-15](), [helm/README.md:1-67]()
## Directory Layout
```text
helm/
├── Chart.yaml # chart metadata: name, version, sources
├── values.yaml # default knobs (image, controller flags, pod overrides)
├── README.md # install/upgrade docs + parameter reference
├── crds/ # auto-installed by Helm before templates
│ ├── agents.x-k8s.io_sandboxes.yaml
│ ├── extensions.agents.x-k8s.io_sandboxclaims.yaml
│ ├── extensions.agents.x-k8s.io_sandboxtemplates.yaml
│ └── extensions.agents.x-k8s.io_sandboxwarmpools.yaml
└── templates/
├── _helpers.tpl # name/namespace/labels/image helpers
├── _controller-args.tpl # controller flag generator
├── namespace.yaml # gated on namespace.create
├── serviceaccount.yaml
├── deployment.yaml # the controller pod
├── service.yaml # metrics service (8080)
├── rbac.generated.yaml # core ClusterRole (Sandbox)
├── extensions-rbac.generated.yaml # extensions ClusterRole
├── clusterrolebinding.yaml
└── clusterrolebinding-extensions.yaml # gated on controller.extensions
```
The `.generated.yaml` suffix on the two RBAC role files signals that the manifests are produced by upstream tooling (`controller-gen`) and copied into the chart; do not edit them by hand. The `clusterrolebinding*.yaml` files are hand-written because they reference Helm release context (`{{ include "agent-sandbox.namespace" . }}`).
Sources: [helm/templates/rbac.generated.yaml:1-60](), [helm/templates/extensions-rbac.generated.yaml:1-88](), [helm/templates/clusterrolebinding.yaml:1-15]()
## CRD Shipping
CRDs live under `helm/crds/`, which is a Helm-special directory: every file there is applied **before** any template renders, exactly once, on first `helm install`. Helm intentionally does **not** upgrade or delete CRDs across releases, so the chart's `README.md` documents the manual `kubectl apply -f helm/crds/` step for upgrades and the manual `kubectl delete -f helm/crds/` step for full removal.
Four CRDs ship with the chart:
| File | API group | Kind | Scope |
|------|-----------|------|-------|
| `agents.x-k8s.io_sandboxes.yaml` | `agents.x-k8s.io` | `Sandbox` | Namespaced |
| `extensions.agents.x-k8s.io_sandboxclaims.yaml` | `extensions.agents.x-k8s.io` | `SandboxClaim` | Namespaced |
| `extensions.agents.x-k8s.io_sandboxtemplates.yaml` | `extensions.agents.x-k8s.io` | `SandboxTemplate` | Namespaced |
| `extensions.agents.x-k8s.io_sandboxwarmpools.yaml` | `extensions.agents.x-k8s.io` | `SandboxWarmPool` | Namespaced |
The CRDs always install — there is no value gate around `crds/` (Helm does not template that directory). The three extension CRDs are installed even when `controller.extensions=false`; what the flag actually gates is the *controller* logic that reconciles them and the extension RBAC binding (see [Conditional Templates](#conditional-templates)).
Each CRD carries the `controller-gen.kubebuilder.io/version: v0.19.0` annotation, confirming they are generated artifacts mirrored into the chart rather than maintained inline.
Sources: [helm/README.md:48-66](), [helm/crds/agents.x-k8s.io_sandboxes.yaml:1-18](), [helm/crds/extensions.agents.x-k8s.io_sandboxclaims.yaml:1-17]()
## Rendered Object Graph
```mermaid
flowchart LR
subgraph Values["values.yaml"]
VImg["image.{repository,tag,pullPolicy}"]
VNs["namespace.{create,name}"]
VCtrl["controller.*"]
VPod["replicaCount / podAnnotations
podLabels / resources
nodeSelector / tolerations / affinity"]
end
subgraph Helpers["templates/_*.tpl"]
HArgs["agent-sandbox.controllerArgs"]
HHelp["agent-sandbox.{name,namespace,
labels,selectorLabels,image}"]
end
subgraph Workload["Controller workload"]
Ns["Namespace
(if namespace.create)"]
SA["ServiceAccount
agent-sandbox-controller"]
Dep["Deployment
agent-sandbox-controller"]
Svc["Service :8080 metrics"]
end
subgraph RBAC["Cluster-scoped RBAC"]
CRcore["ClusterRole
agent-sandbox-controller"]
CRext["ClusterRole
agent-sandbox-controller-extensions"]
CRBcore["ClusterRoleBinding
core"]
CRBext["ClusterRoleBinding
extensions (gated)"]
end
subgraph CRDs["helm/crds/ (pre-install, once)"]
CRDsb["Sandbox CRD"]
CRDsc["SandboxClaim CRD"]
CRDst["SandboxTemplate CRD"]
CRDsw["SandboxWarmPool CRD"]
end
VCtrl --> HArgs --> Dep
VImg --> HHelp --> Dep
VPod --> Dep
VNs --> Ns
VNs --> HHelp
SA --> Dep
SA --> CRBcore
SA --> CRBext
CRcore --> CRBcore
CRext --> CRBext
Svc -. selects .-> Dep
```
The diagram traces three independent value→object pathways: `controller.*` knobs flow through the `agent-sandbox.controllerArgs` helper into the `Deployment`'s `args:`; `image.*` and naming knobs flow through `_helpers.tpl` into image and label fields; and `namespace.*` controls both the optional `Namespace` object and the namespace placement for every other object via `agent-sandbox.namespace`.
Sources: [helm/templates/deployment.yaml:1-65](), [helm/templates/_helpers.tpl:1-43](), [helm/values.yaml:1-63]()
## The Controller Deployment
`helm/templates/deployment.yaml` renders a single `Deployment` named `agent-sandbox-controller`. Notable wiring:
- `replicas: {{ .Values.replicaCount }}` — defaults to 1; leader election (on by default) makes >1 safe.
- `serviceAccountName: agent-sandbox-controller` — hard-coded, matches `serviceaccount.yaml` and both `ClusterRoleBinding`s.
- `image: {{ include "agent-sandbox.image" . }}` — uses the `_helpers.tpl` builder, which `require`s a non-empty `image.tag` and produces `repository:tag`.
- `args:` is rendered entirely by `{{- include "agent-sandbox.controllerArgs" . | nindent 8 }}`.
- Two named container ports: `metrics` on 8080 and `healthz` on 8081. The `Service` only exposes the metrics port.
- `livenessProbe` hits `GET /healthz` on the `healthz` port (15s initial delay, 20s period); `readinessProbe` hits `GET /readyz` on the same port (5s/10s).
- Pod-level overrides — `resources`, `nodeSelector`, `affinity`, `tolerations` — are all rendered behind `{{- with }}` guards, so an empty value renders nothing rather than an empty field.
Pod template labels merge the chart's `selectorLabels` with `Values.podLabels` so user labels coexist with the immutable selector:
```yaml
{{- $selectorLabels := include "agent-sandbox.selectorLabels" . | fromYaml }}
labels:
{{- toYaml (merge (dict) $selectorLabels .Values.podLabels) | nindent 8 }}
```
The order of arguments to `merge` matters: in Helm's `merge`, earlier maps win on conflict, so selector labels take precedence over user-supplied `podLabels` — preventing a stray override from breaking selector matching.
Sources: [helm/templates/deployment.yaml:1-65](), [helm/templates/_helpers.tpl:31-42]()
## The controller-args Helper
`templates/_controller-args.tpl` defines the named template `agent-sandbox.controllerArgs`, which is the single source of truth for converting `controller.*` values into CLI flags. Each block follows the same pattern: `{{- if .Values.controller.X }}` guards a line that emits `- --flag-name={{ value }}`.
```text
controller. --> --=
─────────────────────────────────────────────────────────────────
leaderElect --> --leader-elect
clusterDomain --> --cluster-domain
leaderElectionNamespace --> --leader-election-namespace
extensions --> --extensions
enableTracing --> --enable-tracing
enablePprof --> --enable-pprof
enablePprofDebug --> --enable-pprof-debug
pprofBlockProfileRate --> --pprof-block-profile-rate
pprofMutexProfileFraction --> --pprof-mutex-profile-fraction
kubeApiQps --> --kube-api-qps
kubeApiBurst --> --kube-api-burst
sandboxConcurrentWorkers --> --sandbox-concurrent-workers
sandboxClaimConcurrentWorkers --> --sandbox-claim-concurrent-workers
sandboxWarmPoolConcurrentWorkers --> --sandbox-warm-pool-concurrent-workers
sandboxTemplateConcurrentWorkers --> --sandbox-template-concurrent-workers
extraArgs[] --> verbatim, one per element (quoted)
```
Two consequences of the `{{- if }}` guards:
1. **Unset values produce no flag.** The controller's own defaults apply for any key the user does not set. `values.yaml` deliberately leaves most knobs commented out (`# clusterDomain: "cluster.local"`, `# kubeApiQps: -1.0`, …) so the rendered `args:` list stays minimal.
2. **Falsy values suppress the flag.** Because the guard is `if`, not `not (kindIs "invalid" ...)`, a setting like `controller.leaderElect: false` will *omit* `--leader-elect=false` rather than emit it. This is intentional for the boolean toggles but worth noting when overriding defaults.
`extraArgs` is the escape hatch for any flag the helper does not cover (the chart's example use case is `zap` logging flags). Each element is emitted verbatim and quoted:
```yaml
{{- range .Values.controller.extraArgs }}
- {{ . | quote }}
{{- end }}
```
Sources: [helm/templates/_controller-args.tpl:1-50](), [helm/values.yaml:29-49]()
## Values Knobs at a Glance
The full reference lives in `helm/README.md`; the table below highlights the knobs that the helper translates into flags, plus the ones that shape the pod itself.
| Value | Default | Effect |
|-------|---------|--------|
| `image.tag` | `""` | **Required.** Enforced by `required "image.tag is required"` in `agent-sandbox.image`. |
| `image.repository` | `registry.k8s.io/agent-sandbox/agent-sandbox-controller` | Image repository half of the ref. |
| `replicaCount` | `1` | `Deployment.spec.replicas`. |
| `namespace.create` | `true` | Gates rendering of `namespace.yaml`. |
| `namespace.name` | `agent-sandbox-system` | Overrides `Release.Namespace` for every object. |
| `controller.leaderElect` | `true` | Emits `--leader-elect=true`. |
| `controller.extensions` | `false` | Emits `--extensions=true` **and** renders the extensions `ClusterRoleBinding`. |
| `controller.enableTracing` | unset | Emits `--enable-tracing=true` when truthy. |
| `controller.enablePprof` / `enablePprofDebug` | unset | Toggle pprof endpoint on the metrics port. |
| `controller.kubeApiQps` / `kubeApiBurst` | unset | Tune client-go rate limits. |
| `controller.sandbox*ConcurrentWorkers` | unset | Reconciler concurrency per controller. |
| `controller.extraArgs` | `[]` | Free-form passthrough flags. |
| `podAnnotations` / `podLabels` | `{}` | Added to the pod template; selector labels win on label conflict. |
| `resources` / `nodeSelector` / `tolerations` / `affinity` | `{}` / `[]` | Standard pod overrides, all `{{- with }}`-guarded. |
Sources: [helm/values.yaml:1-63](), [helm/README.md:72-101](), [helm/templates/_helpers.tpl:39-42]()
## Naming and Label Helpers
`_helpers.tpl` defines five named templates that the rest of the chart depends on:
| Helper | Purpose |
|--------|---------|
| `agent-sandbox.name` | Truncates chart name (or `nameOverride`) to 63 chars. Used inside the label helpers. |
| `agent-sandbox.namespace` | `default .Release.Namespace .Values.namespace.name` — every object uses this to resolve its namespace. |
| `agent-sandbox.labels` | Common labels: `helm.sh/chart`, `app.kubernetes.io/{name,instance,managed-by}`, plus `app.kubernetes.io/version` when `image.tag` is set. |
| `agent-sandbox.selectorLabels` | Stable subset (`name` + `instance`) used by both `Deployment.selector` and `Service.selector`. |
| `agent-sandbox.image` | `repository:tag`; calls `required` on `image.tag` so installs fail fast when the user forgets to set it. |
The selector helper is deliberately narrower than the labels helper: `Deployment.spec.selector` is immutable after creation, so it must not include chart-version-derived labels.
Sources: [helm/templates/_helpers.tpl:1-43]()
## Service, ServiceAccount, and Namespace
These three templates are minimal:
- `serviceaccount.yaml` creates `agent-sandbox-controller` in the chart's namespace. No annotations are templated, so workload-identity wiring is not built in.
- `service.yaml` is a ClusterIP `Service` named `agent-sandbox-controller` exposing only the `metrics` port (8080 → named target port `metrics`). There is no service for the `healthz` port — probes go straight to the pod.
- `namespace.yaml` is wrapped entirely in `{{- if .Values.namespace.create }}`. When `false`, no namespace is rendered and the user is expected to point the release at a pre-existing namespace (the README's "Install into an existing namespace" path).
Sources: [helm/templates/service.yaml:1-16](), [helm/templates/serviceaccount.yaml:1-8](), [helm/templates/namespace.yaml:1-9]()
## RBAC Bindings
Cluster-scoped RBAC ships as two independent role/binding pairs:
```text
ServiceAccount: agent-sandbox-controller (in namespace)
│
├── ClusterRoleBinding: agent-sandbox-controller
│ └── ClusterRole: agent-sandbox-controller (always)
│ ├── core: pods, services, persistentvolumeclaims (CRUD+watch)
│ ├── agents.x-k8s.io: sandboxes (CRUD+watch),
│ │ sandboxes/{finalizers,status} (get/patch/update)
│ ├── coordination.k8s.io: leases (CRU+watch)
│ └── events.k8s.io: events (create/patch)
│
└── ClusterRoleBinding: agent-sandbox-controller-extensions (if controller.extensions)
└── ClusterRole: agent-sandbox-controller-extensions (always present in chart)
├── core: pods (read+patch+update+watch), events (create/patch/update)
├── coordination.k8s.io: leases (CRUD+watch)
├── extensions.agents.x-k8s.io: sandboxclaims, sandboxtemplates,
│ sandboxwarmpools (CRUD+watch) and
│ their finalizers/status subresources
├── agents.x-k8s.io: sandboxes (CRUD+watch)
└── networking.k8s.io: networkpolicies (CRUD+watch)
```
A subtlety: `extensions-rbac.generated.yaml` (the `ClusterRole`) is **not** gated on `controller.extensions`, but `clusterrolebinding-extensions.yaml` **is**. The role is therefore present even when extensions are disabled — it simply has no subjects bound to it until the user opts in. Both binding files reference `agent-sandbox-controller` as the only subject and the matching `ClusterRole` name as `roleRef`.
Sources: [helm/templates/rbac.generated.yaml:1-60](), [helm/templates/extensions-rbac.generated.yaml:1-88](), [helm/templates/clusterrolebinding.yaml:1-15](), [helm/templates/clusterrolebinding-extensions.yaml:1-16]()
## Conditional Templates
Only a handful of templates have install-time gates; the rest always render.
| Template | Gate | Behavior when gate is false |
|----------|------|-----------------------------|
| `namespace.yaml` | `Values.namespace.create` | Nothing rendered; existing namespace is reused via `agent-sandbox.namespace`. |
| `clusterrolebinding-extensions.yaml` | `Values.controller.extensions` | The extensions `ClusterRole` exists but is unbound; the controller flag is also omitted. |
| `_controller-args.tpl` entries | per-key `{{- if .Values.controller.X }}` | Flag is omitted; controller falls back to its own default. |
| `Deployment` pod fields | `{{- with }}` on `resources` / `nodeSelector` / `affinity` / `tolerations` / `podAnnotations` | Field is omitted entirely from the rendered manifest. |
Both `namespace.yaml` and `clusterrolebinding-extensions.yaml` use a top-level `{{- if … }} … {{- end }}` wrapper, so disabling them yields zero output (no empty document separator) and Helm produces a clean manifest.
Sources: [helm/templates/namespace.yaml:1-9](), [helm/templates/clusterrolebinding-extensions.yaml:1-16](), [helm/templates/deployment.yaml:49-64]()
## End-to-End: From values to args
The flow below shows what the user sets, how it threads through the chart, and what lands inside the running pod's argv. This is the path a maintainer follows when adding a new controller flag.
```mermaid
sequenceDiagram
autonumber
actor User
participant Vals as values.yaml
participant Tpl as templates/_controller-args.tpl
participant Dep as templates/deployment.yaml
participant K8s as kube-apiserver
participant Pod as agent-sandbox-controller pod
User->>Vals: --set controller.extensions=true
--set controller.kubeApiQps=50
Vals->>Tpl: render agent-sandbox.controllerArgs
Note over Tpl: emits --extensions=true and
--kube-api-qps=50,
skips guarded-out flags
Tpl-->>Dep: include via {{ include "agent-sandbox.controllerArgs" . }}
Dep->>K8s: apply Deployment with args[]
K8s->>Pod: start container with rendered argv
Pod->>Pod: parse flags and start reconcilers
```
To add a new flag, append a guarded block to `_controller-args.tpl`, add the default (or commented-out placeholder) to `values.yaml`, and document it in `helm/README.md`'s configuration table — no other template needs to change.
Sources: [helm/templates/_controller-args.tpl:1-50](), [helm/templates/deployment.yaml:28-29](), [helm/values.yaml:29-49]()
## Summary
The agent-sandbox Helm chart is a deliberately thin packaging of one controller `Deployment`, its `Service`, a `ServiceAccount`, two generated `ClusterRole`s with their bindings, an optional `Namespace`, and four CRDs shipped under the auto-installed `crds/` directory. All controller-tuning lives in `controller.*` values, and the `agent-sandbox.controllerArgs` named template is the single junction that turns those values into command-line flags — keeping the deployment template itself almost free of templating logic. The `controller.extensions` toggle is the one knob with multi-object impact: it both injects `--extensions=true` and renders the extensions `ClusterRoleBinding`, while the extension CRDs and `ClusterRole` always ship.
---
## 25. Static Manifests & Generated RBAC
> The kubectl-apply-ready manifests in k8s/ plus the generated ClusterRole/Binding files used by both core and extensions controllers.
- Page Markdown: https://grok-wiki.com/public/wiki/kubernetes-sigs-agent-sandbox-c3f2597a654a/pages/25-static-manifests-generated-rbac.md
- Generated: 2026-05-25T22:48:51.322Z
### Source Files
- `k8s/controller.yaml`
- `k8s/extensions.controller.yaml`
- `k8s/extensions.yaml`
- `k8s/rbac.generated.yaml`
- `k8s/extensions-rbac.generated.yaml`
Relevant source files
The following files were used as context for generating this wiki page:
- [k8s/controller.yaml](k8s/controller.yaml)
- [k8s/extensions.controller.yaml](k8s/extensions.controller.yaml)
- [k8s/extensions.yaml](k8s/extensions.yaml)
- [k8s/rbac.generated.yaml](k8s/rbac.generated.yaml)
- [k8s/extensions-rbac.generated.yaml](k8s/extensions-rbac.generated.yaml)
- [codegen.go](codegen.go)
- [dev/tools/deploy-to-kube](dev/tools/deploy-to-kube)
- [controllers/sandbox_controller.go](controllers/sandbox_controller.go)
- [extensions/controllers/sandboxclaim_controller.go](extensions/controllers/sandboxclaim_controller.go)
- [extensions/controllers/sandboxtemplate_controller.go](extensions/controllers/sandboxtemplate_controller.go)
- [extensions/controllers/sandboxwarmpool_controller.go](extensions/controllers/sandboxwarmpool_controller.go)
- [cmd/agent-sandbox-controller/main.go](cmd/agent-sandbox-controller/main.go)
# Static Manifests & Generated RBAC
The `k8s/` directory is the canonical, `kubectl apply`-ready home for the agent-sandbox controller's cluster install: it bundles the `Namespace`, `ServiceAccount`, `Service`, `Deployment`, and `ClusterRoleBinding` for the controller, together with machine-generated `CustomResourceDefinition` and `ClusterRole` files. The split into hand-written and `controller-gen`–produced manifests matters because RBAC and CRD changes must follow code changes to the kubebuilder markers — never the other way around.
This page documents what each file in `k8s/` provides, how the generated RBAC files are produced from `//+kubebuilder:rbac` markers, how the core and extensions controllers compose their permissions, and how the deploy tooling assembles these manifests for both the core-only and `--extensions` deployment modes.
## Layout of `k8s/`
| File | Type | Generated? | Role |
| --- | --- | --- | --- |
| `controller.yaml` | Multi-doc YAML | hand-written | Core install: `Namespace`, `ServiceAccount`, `ClusterRoleBinding`, `Service`, `Deployment` |
| `extensions.controller.yaml` | YAML | hand-written | `Deployment` override that adds `--extensions` to the controller args |
| `extensions.yaml` | YAML | hand-written | `ClusterRoleBinding` for the extensions `ClusterRole` |
| `rbac.generated.yaml` | YAML | `controller-gen` | `ClusterRole agent-sandbox-controller` (core permissions) |
| `extensions-rbac.generated.yaml` | YAML | `controller-gen` | `ClusterRole agent-sandbox-controller-extensions` (extension permissions) |
| `crds/` | YAML files | `controller-gen` | One CRD per Go API type (`sandboxes`, `sandboxclaims`, `sandboxtemplates`, `sandboxwarmpools`) |
A parallel set of generated files lives under `helm/templates/` (`rbac.generated.yaml`, `extensions-rbac.generated.yaml`, `clusterrolebinding*.yaml`, `deployment.yaml`, etc.) for the Helm install path, produced by the same `controller-gen` invocations.
Sources: [k8s/controller.yaml:1-82](), [k8s/extensions.controller.yaml:1-33](), [k8s/extensions.yaml:1-15](), [k8s/rbac.generated.yaml:1-60](), [k8s/extensions-rbac.generated.yaml:1-88]()
## Core controller install (`controller.yaml`)
The hand-written `controller.yaml` is a single multi-document YAML that creates the namespace, identity, in-cluster service, and workload for the core controller. The five documents are emitted in install order:
1. `Namespace agent-sandbox-system` — every other object lives here.
2. `ServiceAccount agent-sandbox-controller` — the workload identity.
3. `ClusterRoleBinding agent-sandbox-controller` — binds the generated core `ClusterRole` (same name) to the service account.
4. `Service agent-sandbox-controller` — exposes the `metrics` port (`8080/TCP`) so a scraper can reach the controller pod.
5. `Deployment agent-sandbox-controller` — one replica, runs the controller with `--leader-elect=true`, exposes container ports `metrics` (`8080`) and `healthz` (`8081`).
The image is encoded as a `ko://` reference (`ko://sigs.k8s.io/agent-sandbox/cmd/agent-sandbox-controller`) and is rewritten by the deploy tooling before apply; the manifest is not directly applicable from a clean checkout without that rewrite.
```yaml
# k8s/controller.yaml (Deployment excerpt)
containers:
- name: agent-sandbox-controller
image: ko://sigs.k8s.io/agent-sandbox/cmd/agent-sandbox-controller # placeholder value, replaced by deployment scripts
args:
- --leader-elect=true
ports:
- name: metrics
containerPort: 8080
- name: healthz
containerPort: 8081
```
Sources: [k8s/controller.yaml:1-82]()
## Extensions overlay (`extensions.controller.yaml`, `extensions.yaml`)
The extensions install is intentionally minimal: rather than duplicating the whole core stack, two small documents add the extension surface on top.
- `k8s/extensions.controller.yaml` is a `Deployment` with the **same name and namespace** as the core controller. Its only material difference is an additional `--extensions` arg on the container:
```yaml
args:
- "--leader-elect=true"
- "--extensions"
```
The deploy tool treats this as a replacement, not an addition (see "Deploy ordering" below).
- `k8s/extensions.yaml` is a single `ClusterRoleBinding` named `agent-sandbox-controller-extensions` that binds the generated extensions `ClusterRole` to the same `ServiceAccount agent-sandbox-controller`. There is no separate service account for extensions — the controller process is one binary that opts into reconcilers based on the `--extensions` flag.
The `--extensions` flag is read by the controller entrypoint and gates registration of the extensions scheme and the three reconcilers (`SandboxClaim`, `SandboxTemplate`, `SandboxWarmPool`):
```go
// cmd/agent-sandbox-controller/main.go (excerpt)
flag.BoolVar(&extensions, "extensions", false, "Enable extensions controllers.")
...
if extensions {
utilruntime.Must(extensionsv1beta1.AddToScheme(scheme))
}
...
if extensions {
if err = (&extensionscontrollers.SandboxClaimReconciler{ ... }).SetupWithManager(mgr); err != nil { ... }
if err = (&extensionscontrollers.SandboxTemplateReconciler{ ... }).SetupWithManager(mgr); err != nil { ... }
if err = (&extensionscontrollers.SandboxWarmPoolReconciler{ ... }).SetupWithManager(mgr); err != nil { ... }
}
```
Sources: [k8s/extensions.controller.yaml:1-33](), [k8s/extensions.yaml:1-15](), [cmd/agent-sandbox-controller/main.go:55-78](), [cmd/agent-sandbox-controller/main.go:175-271]()
## Generated RBAC pipeline
The two `*.generated.yaml` files in `k8s/` are produced by `sigs.k8s.io/controller-tools/cmd/controller-gen` and **must not be hand-edited**. The pipeline is declared as `go:generate` directives in [codegen.go](codegen.go):
```go
// codegen.go (excerpt)
//go:generate go tool -modfile=tools.mod sigs.k8s.io/controller-tools/cmd/controller-gen \
// paths=./controllers/... output:rbac:dir=k8s \
// rbac:roleName=agent-sandbox-controller,fileName=rbac.generated.yaml
//go:generate go tool -modfile=tools.mod sigs.k8s.io/controller-tools/cmd/controller-gen \
// paths=./extensions/controllers/... output:rbac:dir=k8s \
// rbac:roleName=agent-sandbox-controller-extensions,fileName=extensions-rbac.generated.yaml
```
Each invocation:
1. Walks the Go packages in `paths=…`.
2. Collects every `//+kubebuilder:rbac:…` marker found on reconciler types.
3. Aggregates the markers into a single `ClusterRole` whose name comes from `rbac:roleName=…` and writes it to `k8s/`.
Identical directives target `helm/templates/` to keep the Helm chart in sync. Regeneration is wired through `make all` (which runs `dev/tools/fix-go-generate`), so editing markers without rerunning the generator will fail CI.
```mermaid
flowchart LR
subgraph Source["Go sources (//+kubebuilder:rbac markers)"]
Core["controllers/sandbox_controller.go"]
Claim["extensions/controllers/sandboxclaim_controller.go"]
Tmpl["extensions/controllers/sandboxtemplate_controller.go"]
Warm["extensions/controllers/sandboxwarmpool_controller.go"]
end
subgraph Gen["codegen.go go:generate"]
CG1["controller-gen
roleName=agent-sandbox-controller"]
CG2["controller-gen
roleName=agent-sandbox-controller-extensions"]
end
subgraph Out["k8s/ generated outputs"]
R1["rbac.generated.yaml
ClusterRole agent-sandbox-controller"]
R2["extensions-rbac.generated.yaml
ClusterRole agent-sandbox-controller-extensions"]
end
subgraph Bind["Hand-written bindings"]
B1["controller.yaml
ClusterRoleBinding agent-sandbox-controller"]
B2["extensions.yaml
ClusterRoleBinding ...-extensions"]
SA["ServiceAccount
agent-sandbox-controller"]
end
Core --> CG1 --> R1
Claim --> CG2
Tmpl --> CG2
Warm --> CG2
CG2 --> R2
R1 -. roleRef .-> B1
R2 -. roleRef .-> B2
B1 --> SA
B2 --> SA
```
Sources: [codegen.go:20-27](), [controllers/sandbox_controller.go:130-137](), [extensions/controllers/sandboxclaim_controller.go:127-136](), [extensions/controllers/sandboxtemplate_controller.go:46-50](), [extensions/controllers/sandboxwarmpool_controller.go:61-64]()
## Core `ClusterRole` (`rbac.generated.yaml`)
The core role grants the controller exactly what the `Sandbox` reconciler needs: full lifecycle on pods, services, and PVCs in the user namespaces it manages; full lifecycle plus subresources on `agents.x-k8s.io/sandboxes`; leases for leader election; and event emission.
| API group | Resources | Verbs |
| --- | --- | --- |
| `""` (core) | `pods`, `services`, `persistentvolumeclaims` | `create, delete, get, list, patch, update, watch` |
| `agents.x-k8s.io` | `sandboxes` | `create, delete, get, list, patch, update, watch` |
| `agents.x-k8s.io` | `sandboxes/finalizers`, `sandboxes/status` | `get, patch, update` |
| `coordination.k8s.io` | `leases` | `create, get, list, patch, update, watch` (no `delete`) |
| `events.k8s.io` | `events` | `create, patch` |
These rules are the exact union of the markers above `SandboxReconciler`:
```go
// controllers/sandbox_controller.go
//+kubebuilder:rbac:groups=agents.x-k8s.io,resources=sandboxes,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups=agents.x-k8s.io,resources=sandboxes/finalizers,verbs=get;update;patch
//+kubebuilder:rbac:groups=agents.x-k8s.io,resources=sandboxes/status,verbs=get;update;patch
//+kubebuilder:rbac:groups=core,resources=pods,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups=core,resources=services,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups=core,resources=persistentvolumeclaims,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups=events.k8s.io,resources=events,verbs=create;patch
//+kubebuilder:rbac:groups=coordination.k8s.io,resources=leases,verbs=get;list;watch;create;update;patch
```
Sources: [k8s/rbac.generated.yaml:1-60](), [controllers/sandbox_controller.go:130-137]()
## Extensions `ClusterRole` (`extensions-rbac.generated.yaml`)
The extensions role is the union of markers from three reconcilers and is broader than the core role in two notable ways: it can manage `extensions.agents.x-k8s.io/*` resources (claims, templates, warm pools), and it can `create/delete/get/list/patch/update/watch` `networking.k8s.io/networkpolicies` — needed because `SandboxTemplate` and `SandboxClaim` materialize network policy isolation around the sandboxes they own.
| API group | Resources | Verbs | Contributed by |
| --- | --- | --- | --- |
| `""` | `pods` | `get, list, patch, update, watch` | `SandboxClaim` |
| `""`, `events.k8s.io` | `events` | `create, patch, update` | all three |
| `agents.x-k8s.io` | `sandboxes` | `create, delete, get, list, patch, update, watch` | `SandboxClaim`, `SandboxWarmPool` |
| `coordination.k8s.io` | `leases` | `create, delete, get, list, patch, update, watch` | `SandboxClaim` |
| `extensions.agents.x-k8s.io` | `sandboxclaims`, `sandboxtemplates`, `sandboxwarmpools` | `create, delete, get, list, patch, update, watch` | all three |
| `extensions.agents.x-k8s.io` | `sandboxclaims/finalizers`, `sandboxclaims/status`, `sandboxtemplates/finalizers`, `sandboxwarmpools/finalizers`, `sandboxwarmpools/status` | `get, patch, update` | per-reconciler |
| `networking.k8s.io` | `networkpolicies` | `create, delete, get, list, patch, update, watch` | `SandboxTemplate` (full), `SandboxClaim` (`get, list, watch, delete`) |
Two quirks worth noting:
- The role contains rules for both API groups `""` and `events.k8s.io` on `events`. That's because some markers declare `groups=core` and others declare `groups=events.k8s.io` — `controller-gen` collapses them into a single rule that lists both groups.
- The extensions role permits `delete` on `leases`, while the core role does not. This comes from `SandboxClaim`'s marker (`verbs=get;list;watch;create;update;patch;delete`) and exists because the extensions controllers manage their own per-claim leases rather than only leader-election ones.
Sources: [k8s/extensions-rbac.generated.yaml:1-88](), [extensions/controllers/sandboxclaim_controller.go:127-136](), [extensions/controllers/sandboxtemplate_controller.go:46-50](), [extensions/controllers/sandboxwarmpool_controller.go:61-64]()
## How bindings tie roles to the service account
Both `ClusterRoleBinding`s point at the same `ServiceAccount agent-sandbox-controller` in `agent-sandbox-system` but reference different `ClusterRole`s by name:
```text
ServiceAccount: agent-sandbox-controller (ns: agent-sandbox-system)
^ ^
| |
ClusterRoleBinding ClusterRoleBinding
agent-sandbox-controller agent-sandbox-controller-extensions
| |
ClusterRole ClusterRole
agent-sandbox-controller agent-sandbox-controller-extensions
(rbac.generated.yaml) (extensions-rbac.generated.yaml)
```
The names of the two `ClusterRole`s are not coincidental — they are passed directly to `controller-gen` via `rbac:roleName=…` in [codegen.go](codegen.go), so the bindings in `controller.yaml` and `extensions.yaml` are coupled by string to the generator invocation.
Sources: [k8s/controller.yaml:19-30](), [k8s/extensions.yaml:1-15](), [codegen.go:22-23]()
## Deploy ordering and the core/extensions composition
The `k8s/` directory is consumed by `dev/tools/deploy-to-kube` (and indirectly by `make deploy-kind`). The script walks `k8s/` recursively, then filters and sorts documents before `kubectl apply`:
1. **Extensions filter.** Files whose names start with `extensions` are skipped unless `--extensions` was passed.
2. **Core controller replacement.** When `--extensions` is set, the core `Deployment` from `controller.yaml` is filtered out so the `extensions.controller.yaml` `Deployment` (with `--extensions` in `args`) takes its place — they share the same name and namespace.
3. **Image rewrite.** Any `ko://…` image or `:latest` tag is rewritten using the configured `image_prefix` and `image_tag`.
4. **Flag append.** A `CONTROLLER_ARGS` string is appended to the controller container's `args`.
5. **Three-phase apply, in order:**
- prerequisites — `Namespace` and `CustomResourceDefinition` objects (from `controller.yaml` and `k8s/crds/`)
- other documents — `ServiceAccount`, `Service`, both `ClusterRole`s, the core `ClusterRoleBinding`, and the chosen `Deployment`
- extension documents (only if `--extensions`) — the extensions `ClusterRoleBinding` from `extensions.yaml`
```mermaid
flowchart TD
Start["dev/tools/deploy-to-kube"] --> Walk["gather_manifests(k8s/)"]
Walk --> Filter{extensions flag?}
Filter -- no --> SkipExt["drop files starting with 'extensions'"]
Filter -- yes --> KeepExt["keep extensions files;
drop core Deployment"]
SkipExt --> Process["find_and_replace_images +
append_controller_flags"]
KeepExt --> Process
Process --> Sort{kind?}
Sort -- CRD/Namespace --> Pre["prereq_docs"]
Sort -- extensions file --> Ext["extensions_docs"]
Sort -- otherwise --> Other["other_docs"]
Pre --> Apply1["kubectl apply (prereqs)"]
Apply1 --> Apply2["kubectl apply (others)"]
Apply2 --> ApplyExt{extensions?}
ApplyExt -- yes --> Apply3["kubectl apply (extensions)"]
ApplyExt -- no --> Done["done"]
Apply3 --> Done
```
Because both `ClusterRole`s and both bindings are accepted regardless of whether `--extensions` is set (only the binding from `extensions.yaml` is gated by the `extensions` filename prefix), an extensions-enabled deploy ends up with both roles, both bindings, the extensions-enabled `Deployment`, and all CRDs applied to the cluster.
Sources: [dev/tools/deploy-to-kube:155-230](), [codegen.go:20-27]()
## Modifying RBAC: the source-of-truth rule
The static manifests in `k8s/` are deliberately the **install artifact**, not the **specification**. To change what the controller can do, edit the `//+kubebuilder:rbac` markers on the reconciler type in `controllers/` or `extensions/controllers/`, then regenerate:
```sh
make all # runs fix-go-generate, which invokes controller-gen
# or, narrowly:
go generate ./...
```
This regenerates the matching files under both `k8s/` and `helm/templates/`. Hand-editing `rbac.generated.yaml` or `extensions-rbac.generated.yaml` is unsafe — the next generator run will overwrite the change and CI's `fix-go-generate` check will reject the PR.
Conversely, the hand-written files (`controller.yaml`, `extensions.controller.yaml`, `extensions.yaml`) are where bindings, namespaces, services, deployment shape, and controller flags live. New permissions belong in markers; new wiring belongs in the hand-written manifests.
Sources: [codegen.go:20-27](), [Makefile:1-10]()
## Summary
`k8s/` is a small, deliberately layered install surface: hand-written cluster identity and workload definitions (`controller.yaml`, `extensions.controller.yaml`, `extensions.yaml`) bind to `ClusterRole`s that are mechanically reproduced from `//+kubebuilder:rbac` markers on the reconcilers (`rbac.generated.yaml`, `extensions-rbac.generated.yaml`). The extensions install is an overlay rather than a parallel stack — same `ServiceAccount`, same `Deployment` name, with `--extensions` and an extra binding turning on additional reconcilers. The `dev/tools/deploy-to-kube` script orchestrates the apply, ensuring CRDs and namespaces land before any objects that depend on them and that the right controller `Deployment` wins when extensions are enabled.
---
## 26. Examples Library Map
> Tour of the examples/ tree: AIO sandbox, Chrome/VSCode/JupyterLab, agent frameworks (Hermes, LangChain, ADK, Ray, Kueue), policy and scaling scenarios.
- Page Markdown: https://grok-wiki.com/public/wiki/kubernetes-sigs-agent-sandbox-c3f2597a654a/pages/26-examples-library-map.md
- Generated: 2026-05-25T22:50:45.765Z
### Source Files
- `examples/README.md`
- `examples/chrome-sandbox`
- `examples/vscode-sandbox`
- `examples/jupyterlab`
- `examples/policy`
- `examples/hpa-swp-scaling`
- `extensions/examples/README.md`
Relevant source files
The following files were used as context for generating this wiki page:
- [examples/README.md](examples/README.md)
- [examples/aio-sandbox/README.md](examples/aio-sandbox/README.md)
- [examples/aio-sandbox/aio-sandbox.yaml](examples/aio-sandbox/aio-sandbox.yaml)
- [examples/chrome-sandbox/README.md](examples/chrome-sandbox/README.md)
- [examples/vscode-sandbox/README.md](examples/vscode-sandbox/README.md)
- [examples/jupyterlab/README.md](examples/jupyterlab/README.md)
- [examples/hermes-agent/README.md](examples/hermes-agent/README.md)
- [examples/langchain/README.md](examples/langchain/README.md)
- [examples/code-interpreter-agent-on-adk/README.md](examples/code-interpreter-agent-on-adk/README.md)
- [examples/ray-integration/README.md](examples/ray-integration/README.md)
- [examples/kueue-agent-sandbox/README.md](examples/kueue-agent-sandbox/README.md)
- [examples/hpa-swp-scaling/README.md](examples/hpa-swp-scaling/README.md)
- [examples/hpa-swp-scaling/hpa.yaml](examples/hpa-swp-scaling/hpa.yaml)
- [examples/policy/sandbox-automount-token-policy.yaml](examples/policy/sandbox-automount-token-policy.yaml)
- [examples/policy/sandbox-automount-token-binding.yaml](examples/policy/sandbox-automount-token-binding.yaml)
- [examples/policy/kyverno/README.md](examples/policy/kyverno/README.md)
- [examples/policy/network-policy-management/README.md](examples/policy/network-policy-management/README.md)
- [examples/policy/vap/README.md](examples/policy/vap/README.md)
- [examples/policy/opa-gatekeeper/README.md](examples/policy/opa-gatekeeper/README.md)
- [examples/policy/policy-controller/README.md](examples/policy/policy-controller/README.md)
- [examples/composing-sandbox-nw-policies/README.md](examples/composing-sandbox-nw-policies/README.md)
- [examples/quickstart/README.md](examples/quickstart/README.md)
- [examples/python-sdk-quickstart/README.md](examples/python-sdk-quickstart/README.md)
- [extensions/examples/README.md](extensions/examples/README.md)
- [extensions/examples/sandboxtemplate.yaml](extensions/examples/sandboxtemplate.yaml)
# Examples Library Map
The `examples/` tree in `kubernetes-sigs/agent-sandbox` is the canonical place where the project demonstrates how its core `Sandbox` CRD (and the `extensions.agents.x-k8s.io` types `SandboxTemplate`, `SandboxClaim`, and `SandboxWarmPool`) compose with real workloads, agent frameworks, isolation runtimes, admission policies, and autoscalers. The README at the root of the directory enumerates roughly twenty self-contained subdirectories, each tuned to a different reader: a platform operator standing up a quickstart cluster, an agent author wiring an SDK into LangGraph or ADK, or a security administrator validating Kyverno/VAP/Gatekeeper guardrails.
This page maps that tree by audience and purpose, links each subdirectory to the representative manifest or script that anchors it, and clarifies the boundary between the core `Sandbox` API (in this repo) and the higher-level extension CRDs that the more sophisticated examples consume.
Sources: [examples/README.md:1-20](), [extensions/examples/README.md:1-3](), [extensions/examples/sandboxtemplate.yaml:1-44]()
## Top-Level Layout
The two example roots in the repository serve different layers of the API:
| Path | Purpose | API surface |
| --- | --- | --- |
| `examples/` | End-to-end scenarios that consume the controller. Each subdirectory pairs a `README.md` with `.yaml` manifests and (often) Python/Go test drivers. | Mixed: core `Sandbox` plus, where useful, the extension CRDs and Python SDK. |
| `extensions/examples/` | Minimal reference manifests for the extension CRDs — one `SandboxTemplate`, one `SandboxClaim`, one `SandboxWarmPool`, plus a `SecurityContext`-locked-down variant. | `extensions.agents.x-k8s.io/v1alpha1` only. |
The split is intentional: `extensions/examples/` is a starter pack of canonical CRD shapes, while `examples/` shows how a sandbox is wired into a real agent or platform.
```text
examples/
├── README.md # index of all examples
├── quickstart/ # end-to-end walkthrough (KIND default)
├── python-sdk-quickstart/ # SDK-first variant
├── hello-world-sandbox/ # smallest possible Sandbox
├── aio-sandbox/ # All-in-One runtime (VNC/VSCode/Jupyter)
├── chrome-sandbox/ # headless Chrome, used by e2e tests
├── vscode-sandbox/ # code-server + kustomize overlays
├── jupyterlab/ # JupyterLab + PVC + HF models
├── analytics-tool/ # ADK analytics tool on GKE
├── hermes-agent/ # Hermes agent with custom skills
├── langchain/ # LangGraph coding agent + local model
├── code-interpreter-agent-on-adk/ # ADK + Sandbox SDK as a tool
├── ray-integration/ # Ray + warm pool + gVisor proxy exec
├── kueue-agent-sandbox/ # Admission control via Kueue
├── hpa-swp-scaling/ # HPA scaling a SandboxWarmPool
├── manual-pdb/ # PodDisruptionBudget recipe
├── composing-sandbox-nw-policies/ # KRO composition: Sandbox+Service+NetPol
├── policy/ # VAP, Kyverno, OPA, Policy Controller
├── python-runtime-sandbox/ # FastAPI runtime image used by SDK
├── gemini-cu-sandbox/ # Gemini Computer Use sandboxed runtime
├── openclaw-sandbox/ # OpenClaw inside Sandbox
├── kata-gke-sandbox/ # Kata Containers on GKE node pools
└── sandbox-ksa/ # Sandbox bound to a custom ServiceAccount
extensions/examples/ # canonical CRD shapes
├── sandboxtemplate.yaml # SandboxTemplate
├── sandbox-claim.yaml / sandboxclaim.yaml
├── sandboxwarmpool.yaml
├── secure-sandboxtemplate.yaml
└── llm.yaml
```
Sources: [examples/README.md:1-20](), [extensions/examples/README.md:1-3]()
## Onboarding Path: Quickstarts and Hello World
These examples are entry points: they require no agent framework and validate the controller, runtime, and SDK are working before moving on.
- **`quickstart/`** is the full walkthrough. It enumerates the resource set a new user will encounter — `Sandbox`, `SandboxTemplate`, `SandboxClaim`, `SandboxWarmPool`, the Python SDK, and the Router Service — and branches off to `gvisor.md` or `kata-containers.md` if the reader wants hardened isolation rather than a vanilla KIND cluster. It is the only example that explicitly enumerates all the public types up front.
- **`python-sdk-quickstart/`** flips the same flow around the SDK. It documents the three connection modes the client supports (`SandboxLocalTunnelConnectionConfig` for tunneled local development, `SandboxGatewayConnectionConfig` for a public Kubernetes Gateway, `SandboxDirectConnectionConfig` for in-cluster agents) and requires that a `python-sandbox-template` already exists from the Python Runtime example.
- **`hello-world-sandbox/`** is the smallest example: a Dockerfile, a Sandbox manifest, and instructions to push to Artifact Registry. Useful for sanity checks against a fresh cluster.
Sources: [examples/quickstart/README.md:1-60](), [examples/python-sdk-quickstart/README.md:1-30](), [examples/hello-world-sandbox/README.md:1-30]()
## Interactive Workspaces
This cluster of examples shows the `Sandbox` CRD wrapping general-purpose interactive runtimes — what an end user might think of as "a remote dev environment for an agent."
### AIO Sandbox
`aio-sandbox/` deploys the All-in-One image from `agent-infra/sandbox`, which bundles VNC, VSCode, Jupyter, and a terminal behind a single port. The manifest is minimal and demonstrates the bare-minimum hardened `securityContext`:
```yaml
# examples/aio-sandbox/aio-sandbox.yaml
spec:
podTemplate:
spec:
containers:
- name: aio-sandbox
image: ghcr.io/agent-infra/sandbox:1.0.0.152
securityContext:
allowPrivilegeEscalation: false
ports:
- containerPort: 8080
resources:
limits:
memory: "4Gi"
cpu: "2000m"
```
The README distinguishes the in-pod `agent-sandbox` Python SDK (which drives tools inside the running sandbox — browser, shell, files) from the `agentic-sandbox-client` SDK that provisions sandboxes via the CRDs. They are designed to compose.
### Chrome Sandbox
`chrome-sandbox/` runs a headless Chrome inside a `Sandbox`, exposing the DevTools port `9222`. It doubles as the foundation for the project's e2e tests; the README points at `test/e2e/chromesandbox_test.go` as the authoritative consumer.
### VSCode Sandbox
`vscode-sandbox/` is the most layered of the workspace examples. It ships a base kustomization plus three overlays — `overlays/gvisor`, `overlays/kata`, `overlays/kata-mshv` — that inject the corresponding `runtimeClassName`. The README explains the access pattern carefully: when a hardened runtime is used, direct `kubectl port-forward` to the Pod is incompatible, so traffic must be routed through the Sandbox Router using the `X-Sandbox-ID` / `X-Sandbox-Port` headers (or a Gateway in production).
### JupyterLab
`jupyterlab/` offers two deployment shapes — a single-file `jupyterlab-full.yaml` or the modular `jupyterlab.yaml` plus a `kubectl create configmap` step from `files/` — and demonstrates init-container patterns: the `setup-environment` init container downloads Python deps and HuggingFace models into a `volumeClaimTemplates` PVC before the JupyterLab container starts. The README also covers force-re-init by removing `/home/jovyan/.initialized`.
Sources: [examples/aio-sandbox/README.md:1-92](), [examples/aio-sandbox/aio-sandbox.yaml:1-22](), [examples/chrome-sandbox/README.md:1-95](), [examples/vscode-sandbox/README.md:1-258](), [examples/jupyterlab/README.md:1-90]()
## Agent Framework Integrations
These examples show how to plug the sandbox primitives into common agent frameworks. They are the cleanest reference for "how do I make my agent run untrusted code in a sandbox?"
```mermaid
flowchart LR
subgraph Local["Local / Trusted process"]
Agent[Agent code
LangGraph / ADK / Ray Actor]
SDK[k8s-agent-sandbox SDK
SandboxClient]
end
subgraph Cluster["Kubernetes cluster"]
Router[sandbox-router
Service]
subgraph WarmPool["SandboxWarmPool"]
Pod1[Pod
runtime image]
Pod2[Pod
runtime image]
end
Template[SandboxTemplate]
end
Agent --> SDK
SDK -->|Tunnel / Gateway / Direct| Router
Router -->|X-Sandbox-ID / X-Sandbox-Port| Pod1
WarmPool -.references.-> Template
```
- **`langchain/`** wires a LangGraph coding agent to a sandboxed Python runtime. The init container caches a HuggingFace model (`Salesforce/codegen-350M-mono` by default) on a PVC; the agent's execution loop generates code, sends it into the sandbox, captures errors, and retries up to three times. The README documents memory tuning and how to swap models.
- **`code-interpreter-agent-on-adk/`** is the lightest framework integration: a six-line ADK `Agent` whose only tool wraps `SandboxClient` to create a sandbox, write `run.py`, execute `python3 run.py`, and tear the sandbox down. It is the recommended pattern for "ADK + execute_python."
- **`hermes-agent/`** runs the Hermes agent inside a sandbox with a ConfigMap-mounted custom "Kubernetes Developer" skill (`hermes-skills` ConfigMap from `k8s-developer.md`). It includes `test_hermes.py` for automated verification and `chat_hermes.py` for interactive sessions on port 8642.
- **`ray-integration/`** is the most architecturally explicit example. It documents the **Proxy Execution Model**: a Ray rollout actor stays inside the trusted cluster, and every untrusted code execution is proxied via `SandboxClient` into a gVisor-isolated pod claimed from a `SandboxWarmPool`. The README provides `rl_poc_local.py` (using `SandboxLocalTunnelConnectionConfig`) and `rl_poc_prod.py` (using `SandboxGatewayConnectionConfig` with `gateway_name: "external-http-gateway"`).
- **`kueue-agent-sandbox/`** is admission control rather than a framework. It shows the `kueue.x-k8s.io/queue-name` label being attached to the `podTemplate.metadata.labels` of a `Sandbox`, so Kueue queues pod admission until cluster-queue quota is available. The README is explicit about scope: Kueue controls *when* a sandbox starts, not how long it runs.
Sources: [examples/langchain/README.md:1-90](), [examples/code-interpreter-agent-on-adk/README.md:1-70](), [examples/hermes-agent/README.md:1-105](), [examples/ray-integration/README.md:1-220](), [examples/kueue-agent-sandbox/README.md:1-109]()
## Scaling and Disruption
Two examples target the operational lifecycle of sandbox pools rather than the workloads inside them.
`hpa-swp-scaling/` is GKE-specific: it scales a `SandboxWarmPool` with a stock Kubernetes `HorizontalPodAutoscaler` that consumes a Prometheus counter (`agent_sandbox_claim_creation_total`) exposed through the GKE Managed Service for Prometheus and the Custom Metrics Adapter. The HPA spec scales the warm pool itself rather than a Deployment:
```yaml
# examples/hpa-swp-scaling/hpa.yaml
scaleTargetRef:
apiVersion: extensions.agents.x-k8s.io/v1alpha1
kind: SandboxWarmPool
name: python-sdk-warmpool
minReplicas: 10
maxReplicas: 100
metrics:
- type: External
external:
metric:
name: "prometheus.googleapis.com|agent_sandbox_claim_creation_total|counter"
target:
type: Value
value: "0.5"
```
`create-claim.py` is the load generator and `pod-monitoring.yaml` exposes the metric to Managed Prometheus. The README contains a real `kubectl get hpa -w` trace showing the warm pool scaling from 10 to 100 and back down as claim rate rises and falls.
`manual-pdb/` shows the opt-in `PodDisruptionBudget` pattern: a shared PDB selects pods carrying `sandbox-disruption-policy: "manual-protection"`, set on the `SandboxTemplate`. Sandboxes without that label are unprotected from voluntary disruptions. The README warns specifically against `minAvailable: 1` on a shared PDB.
Sources: [examples/hpa-swp-scaling/README.md:1-97](), [examples/hpa-swp-scaling/hpa.yaml:1-23](), [examples/manual-pdb/README.md:1-40]()
## Policy and Admission
`examples/policy/` is the largest single sub-tree and demonstrates how the project layers cluster admission controllers on top of the `Sandbox` CRD. Each subdirectory targets a different policy engine but solves the same family of problems: prevent privilege escalation, enforce isolation defaults, and lock down the ServiceAccounts a `Sandbox` references.
| Subdirectory | Engine | Headline rule |
| --- | --- | --- |
| `vap/` | Kubernetes `ValidatingAdmissionPolicy` (built-in, v1.30+) | Enforces a "Secure by Default" posture across containers/initContainers/ephemeralContainers via CEL — `runtimeClassName: gvisor`, `hostNetwork/hostPID/hostIPC: false`, `automountServiceAccountToken: false`, `privileged: false`, `capabilities.drop: ["ALL"]`, resource limits, and GKE-specific `nodeSelector`/`tolerations`. |
| `kyverno/` | Kyverno `ValidatingPolicy` | Denies `RoleBinding`/`ClusterRoleBinding` creates or updates that grant permissions to a ServiceAccount used by a `Sandbox`-owned Pod, including indirect references via `system:serviceaccounts[:ns]` groups. Includes Chainsaw tests in `.chainsaw-tests/`. |
| `opa-gatekeeper/` | OPA Gatekeeper | Same goal, Rego implementation. |
| `policy-controller/` | Google Cloud Policy Controller (Anthos) | Same goal, fleet-managed on GKE. |
| (top-level) `sandbox-automount-token-{policy,binding}.yaml` | Plain VAP | A single rule: every `Sandbox` must explicitly set `spec.podTemplate.spec.automountServiceAccountToken == false`. |
| `network-policy-management/` | Controller-managed `NetworkPolicy` | Documents the Template-Level Shared Network Policy model and the Secure-by-Default ingress/egress posture (allow only the Sandbox Router; block RFC1918, link-local metadata, cluster DNS). |
The standalone `sandbox-automount-token-policy.yaml` is a useful read because it is small and concrete:
```yaml
# examples/policy/sandbox-automount-token-policy.yaml
spec:
failurePolicy: Fail
matchConstraints:
resourceRules:
- apiGroups: ["agents.x-k8s.io"]
apiVersions: ["v1alpha1"]
operations: ["CREATE", "UPDATE"]
resources: ["sandboxes"]
validations:
- expression: "object.spec.podTemplate.spec.automountServiceAccountToken == false"
```
The binding scopes the policy to every namespace except `kube-system`. Together these two files are the minimal hardening pattern the repo recommends.
`composing-sandbox-nw-policies/` complements the policy directory by demonstrating composition with **KRO** (`ResourceGraphDefinition`). It bundles a `Sandbox`, a `Service`, and a `NetworkPolicy` into a single higher-level `AgenticSandbox` CRD, illustrating one of three composition strategies the README discusses (custom controller, KRO, Helm).
Sources: [examples/policy/vap/README.md:1-46](), [examples/policy/kyverno/README.md:1-90](), [examples/policy/opa-gatekeeper/README.md:1-60](), [examples/policy/policy-controller/README.md:1-60](), [examples/policy/sandbox-automount-token-policy.yaml:1-15](), [examples/policy/sandbox-automount-token-binding.yaml:1-15](), [examples/policy/network-policy-management/README.md:1-90](), [examples/composing-sandbox-nw-policies/README.md:1-40]()
## Runtime Hardening Scenarios
A subset of examples is dedicated to providing a stronger isolation boundary under the `Sandbox` CRD by changing the container runtime. They all converge on `spec.podTemplate.spec.runtimeClassName`.
| Example | Runtime | Cluster target |
| --- | --- | --- |
| `vscode-sandbox/overlays/gvisor` | gVisor | Any cluster with the `gvisor` RuntimeClass |
| `vscode-sandbox/overlays/kata` | Kata Containers (QEMU) | Local minikube with nested virtualization |
| `vscode-sandbox/overlays/kata-mshv` | Kata + Microsoft Hypervisor | Azure AKS Pod Sandboxing |
| `kata-gke-sandbox/` | Kata Containers on GKE | Requires N2 Intel + Ubuntu node pool, via `setup.sh` |
| `quickstart/gvisor.md`, `quickstart/kata-containers.md` | Either | KIND/minikube introductory paths |
Each of these examples emphasizes that direct pod port-forward stops working under non-default runtimes; access must go through the Sandbox Router service or a Kubernetes Gateway.
Sources: [examples/vscode-sandbox/README.md:20-150](), [examples/kata-gke-sandbox/README.md:1-50](), [examples/quickstart/README.md:1-60]()
## Runtime Images and Specialized Workloads
The remaining examples package container images and surrounding glue for specific workloads.
- **`python-runtime-sandbox/`** is the FastAPI image with `/execute` that the Python SDK and the Ray integration consume. It defines two Pydantic models, `ExecuteRequest` and `ExecuteResponse`, and ships `run-test-docker.sh` and `run-test-kind.sh` for local validation.
- **`gemini-cu-sandbox/`** wraps the `computer-use-preview` agent (Gemini Computer Use) in a sandboxed FastAPI server, requiring `GEMINI_API_KEY`.
- **`openclaw-sandbox/`** runs OpenClaw (formerly Moltbot) inside a `Sandbox`; the README documents the gVisor + port-forward incompatibility and points at the Router as the workaround.
- **`analytics-tool/`** packages an ADK analytics tool image for GKE and shows the Service/JupyterLab integration on top.
- **`sandbox-ksa/`** is the canonical pattern for binding a `Sandbox` to a custom Kubernetes ServiceAccount — useful for per-sandbox identity and RBAC scoping.
Sources: [examples/python-runtime-sandbox/README.md:1-40](), [examples/gemini-cu-sandbox/README.md:1-40](), [examples/openclaw-sandbox/README.md:1-45](), [examples/analytics-tool/README.md:1-40](), [examples/sandbox-ksa/README.md:1-40]()
## Choosing An Example
The decision tree most readers follow is roughly:
1. **Just trying it out?** Start with `quickstart/` (or `python-sdk-quickstart/` for the SDK-centric path). Use `hello-world-sandbox/` as a smoke test.
2. **Wiring an agent framework?** Pick the framework-aligned example: `code-interpreter-agent-on-adk/` for ADK, `langchain/` for LangGraph, `hermes-agent/` for Hermes, `ray-integration/` for Ray RL.
3. **Need an interactive environment for a human or model to inhabit?** `aio-sandbox/`, `vscode-sandbox/`, `jupyterlab/`, or `chrome-sandbox/` depending on the tool surface.
4. **Operating at scale?** `hpa-swp-scaling/` for autoscaling warm pools, `manual-pdb/` for disruption control, `kueue-agent-sandbox/` for admission quotas.
5. **Securing the platform?** Start with `examples/policy/vap/` for the headline VAP, then layer `examples/policy/network-policy-management/` and one of `kyverno/`, `opa-gatekeeper/`, or `policy-controller/` depending on your engine.
6. **Hardening the runtime?** Pick a `vscode-sandbox/overlays/*` overlay or `kata-gke-sandbox/` matching your cluster.
The `extensions/examples/` directory is the place to look when you only need the raw CRD shapes for `SandboxTemplate`, `SandboxClaim`, and `SandboxWarmPool` without surrounding application code — it is referenced by `manifest.yaml`/`extensions.yaml` installation flows discussed in the higher-level examples.
Sources: [examples/README.md:1-20](), [extensions/examples/README.md:1-3](), [extensions/examples/sandboxtemplate.yaml:1-44]()
---
## 27. Build, Codegen & Repository Tools
> Make targets, the codegen.go shim, deepcopy/CRD generation, lint configuration, and the dev/tools scripts that power local development.
- Page Markdown: https://grok-wiki.com/public/wiki/kubernetes-sigs-agent-sandbox-c3f2597a654a/pages/27-build-codegen-repository-tools.md
- Generated: 2026-05-25T22:52:34.631Z
### Source Files
- `Makefile`
- `codegen.go`
- `tools.mod`
- `dev/tools/client-gen-go.sh`
- `dev/tools/lint-api`
- `dev/tools/test-unit`
- `docs/development.md`
Relevant source files
The following files were used as context for generating this wiki page:
- [Makefile](Makefile)
- [codegen.go](codegen.go)
- [tools.mod](tools.mod)
- [dev/tools/client-gen-go.sh](dev/tools/client-gen-go.sh)
- [dev/tools/lint-api](dev/tools/lint-api)
- [dev/tools/test-unit](dev/tools/test-unit)
- [dev/tools/lint-go](dev/tools/lint-go)
- [dev/tools/fix-go-generate](dev/tools/fix-go-generate)
- [dev/tools/build-kal](dev/tools/build-kal)
- [dev/tools/.golangci.yaml](dev/tools/.golangci.yaml)
- [dev/tools/.golangci-kal.yml](dev/tools/.golangci-kal.yml)
- [dev/tools/client-gen-go.sh](dev/tools/client-gen-go.sh)
- [dev/tools/go.mod](dev/tools/go.mod)
- [dev/tools/shared/utils.py](dev/tools/shared/utils.py)
- [dev/tools/fix-boilerplate](dev/tools/fix-boilerplate)
- [dev/tools/update-toc](dev/tools/update-toc)
- [dev/tools/verify-toc](dev/tools/verify-toc)
- [docs/development.md](docs/development.md)
# Build, Codegen & Repository Tools
This page documents how `kubernetes-sigs/agent-sandbox` is built, how its generated code (CRDs, RBAC, deepcopy methods, typed clientsets/listers/informers) is produced, how Go and Kubernetes-API lint checks are configured, and how the `dev/tools/` Python helpers tie those steps together. The repository keeps developer tooling deliberately co-located: a single top-level `Makefile` exposes named targets that shell out to per-script entry points under `dev/tools/`, while a separate `tools.mod` / `dev/tools/go.mod` keeps developer-tool dependencies out of the production `go.mod`.
The flow matters because the controller's CRDs, RBAC manifests (under `k8s/` and `helm/`), `zz_generated.deepcopy.go`, and the typed clients under `clients/k8s/` are all derived from the Go API packages — they must be regenerated whenever `api/` or `extensions/` change, and the same generators are wired into both `make all` and CI.
## Make targets and the `make all` pipeline
The `Makefile` declares a single canonical orchestration target `all` that runs the full local-validation pipeline:
```make
all: fix-go-generate build lint-go lint-api test-unit toc-verify
```
Sources: [Makefile:1-2](Makefile)
Each phase is a thin shell-out to a script under `dev/tools/`, so the Makefile acts mostly as a dispatcher rather than encoding logic itself.
| Target | Action | Underlying tool |
|---|---|---|
| `fix-go-generate` | Runs all `//go:generate` directives | `dev/tools/fix-go-generate` ([Makefile:4-6](Makefile)) |
| `build` | Builds the controller binary to `bin/manager` with version ldflags | `go build` ([Makefile:27-29](Makefile)) |
| `lint-go` / `fix-go` | Runs `golangci-lint` (with or without `--fix`) | `dev/tools/lint-go` ([Makefile:70-76](Makefile)) |
| `lint-api` / `fix-api` | Runs Kube-API Linter (KAL) on `api/` and `extensions/api/` | `dev/tools/lint-api` ([Makefile:78-84](Makefile)) |
| `test-unit` | Runs Go and Python unit suites | `dev/tools/test-unit` ([Makefile:54-56](Makefile)) |
| `test-e2e` / `test-e2e-race` / `test-e2e-benchmarks` | Drives `dev/ci/presubmits/test-e2e` | `RACE` env toggles `-race` ([Makefile:58-68](Makefile)) |
| `deploy-kind` | Creates a kind cluster, pushes images, deploys controller | `dev/tools/create-kind-cluster`, `push-images`, `deploy-to-kube` ([Makefile:33-40](Makefile)) |
| `toc-update` / `toc-verify` | Updates/verifies KEP table-of-contents using `mdtoc` | `dev/tools/update-toc`, `dev/tools/verify-toc` ([Makefile:128-134](Makefile)) |
| `generate-api-docs` | Generates `docs/api.md` from CRD markers | `crd-ref-docs` ([Makefile:10-15](Makefile)) |
| `release-promote` / `release-publish` / `release-manifests` / `release-python-sdk` | Tag/publish flows; require `TAG=` | `dev/tools/tag-promote-images`, `dev/tools/release`, `dev/tools/release-python` ([Makefile:98-126](Makefile)) |
| `clean` | Removes `dev/tools/tmp` and `bin/manager` | (none) ([Makefile:136-139](Makefile)) |
### Version ldflags
The `build` target injects build metadata into `internal/version` via linker flags. `GIT_VERSION`, `GIT_SHA`, and `BUILD_DATE` are computed from `git describe`/`git rev-parse`/`date -u`, and stamped through `-X` flags into `sigs.k8s.io/agent-sandbox/internal/version`:
```make
VERSION_PKG := sigs.k8s.io/agent-sandbox/internal/version
LD_FLAGS := -s -w -X $(VERSION_PKG).gitVersion=$(GIT_VERSION) \
-X $(VERSION_PKG).gitSHA=$(GIT_SHA) \
-X $(VERSION_PKG).buildDate=$(BUILD_DATE)
```
Sources: [Makefile:17-29](Makefile)
## The `codegen.go` shim and `tools.mod`
`codegen.go` is a tiny, otherwise-empty file at the repo root whose sole purpose is to host project-wide `//go:generate` directives. The package compiles as part of the module but contributes no runtime code:
```go
// This file just exists as a place to put //go:generate directives that should apply to the entire project
package agentsandbox
```
Sources: [codegen.go:15-17](codegen.go)
It declares eight `controller-gen` invocations plus the clientset script:
```go
//go:generate go tool -modfile=tools.mod sigs.k8s.io/controller-tools/cmd/controller-gen object crd:maxDescLen=0 paths=./api/... output:crd:dir=k8s/crds
//go:generate go tool -modfile=tools.mod sigs.k8s.io/controller-tools/cmd/controller-gen object crd:maxDescLen=0 paths=./extensions/... output:crd:dir=k8s/crds
//go:generate go tool -modfile=tools.mod sigs.k8s.io/controller-tools/cmd/controller-gen paths=./controllers/... output:rbac:dir=k8s rbac:roleName=agent-sandbox-controller,fileName=rbac.generated.yaml
//go:generate go tool -modfile=tools.mod sigs.k8s.io/controller-tools/cmd/controller-gen paths=./extensions/controllers/... output:rbac:dir=k8s rbac:roleName=agent-sandbox-controller-extensions,fileName=extensions-rbac.generated.yaml
//go:generate go tool -modfile=tools.mod sigs.k8s.io/controller-tools/cmd/controller-gen object crd:maxDescLen=0 paths=./api/... output:crd:dir=helm/crds
//go:generate go tool -modfile=tools.mod sigs.k8s.io/controller-tools/cmd/controller-gen object crd:maxDescLen=0 paths=./extensions/... output:crd:dir=helm/crds
//go:generate go tool -modfile=tools.mod sigs.k8s.io/controller-tools/cmd/controller-gen paths=./controllers/... output:rbac:dir=helm/templates rbac:roleName=agent-sandbox-controller,fileName=rbac.generated.yaml
//go:generate go tool -modfile=tools.mod sigs.k8s.io/controller-tools/cmd/controller-gen paths=./extensions/controllers/... output:rbac:dir=helm/templates rbac:roleName=agent-sandbox-controller-extensions,fileName=extensions-rbac.generated.yaml
//go:generate ./dev/tools/client-gen-go.sh
```
Sources: [codegen.go:20-28](codegen.go)
Two design choices are worth calling out:
1. **`go tool -modfile=tools.mod`** invokes `controller-gen` from a sibling module file, not from `go.mod`. `tools.mod` declares it as a Go tool: `tool sigs.k8s.io/controller-tools/cmd/controller-gen` ([tools.mod:97](tools.mod)). That keeps `controller-tools`, `code-generator`, and `gengo` out of the production module while still pinning their versions reproducibly.
2. **Each `paths=...` invocation emits to two output trees** (`k8s/...` and `helm/...`). The repository deliberately duplicates the generated CRDs and RBAC into both a raw manifest tree (`k8s/crds`, `k8s/rbac.generated.yaml`) and the Helm chart tree (`helm/crds`, `helm/templates/...`). The matching files under those trees (e.g., `k8s/crds/agents.x-k8s.io_sandboxes.yaml`) are written by the `object` + `crd` generators; deepcopy methods land alongside the API types as `zz_generated.deepcopy.go` ([api/v1beta1/zz_generated.deepcopy.go:1-5](api/v1beta1/zz_generated.deepcopy.go)).
### `controller-gen` generators in use
| Generator | Inputs | Outputs |
|---|---|---|
| `object` | `./api/...`, `./extensions/...` | `zz_generated.deepcopy.go` next to API types |
| `crd:maxDescLen=0` | same | `k8s/crds/*.yaml`, `helm/crds/*.yaml` |
| `rbac:roleName=agent-sandbox-controller` | `./controllers/...` | `k8s/rbac.generated.yaml`, `helm/templates/rbac.generated.yaml` |
| `rbac:roleName=agent-sandbox-controller-extensions` | `./extensions/controllers/...` | `k8s/extensions-rbac.generated.yaml`, `helm/templates/extensions-rbac.generated.yaml` |
`maxDescLen=0` disables the default truncation of CRD description text, so the on-cluster schemas keep full kubebuilder-doc descriptions.
Sources: [codegen.go:20-28](codegen.go), [k8s/crds/](k8s/crds), [helm/crds/](helm/crds)
### `fix-go-generate`
`make fix-go-generate` delegates to `dev/tools/fix-go-generate`, which simply runs `go generate -v ./...` from the repo root so every package's `//go:generate` lines (in practice just `codegen.go`) fire:
```python
subprocess.check_call(["go", "generate", "-v", "./..."], cwd=repo_root)
```
Sources: [dev/tools/fix-go-generate:30-31](dev/tools/fix-go-generate)
## Typed-client generation: `client-gen-go.sh`
The last `//go:generate` directive in `codegen.go` is a shell script that produces Kubernetes typed clients, listers, and informers for both the core and extensions API groups. It bootstraps the `k8s.io/code-generator` binaries through `go run -modfile=tools.mod`, so they too come from the dev-only tool module rather than `go.mod`.
```bash
CMD="go run -modfile=tools.mod k8s.io/code-generator"
API_PKG="sigs.k8s.io/agent-sandbox/api/v1beta1"
CLIENT_PKG="sigs.k8s.io/agent-sandbox/clients/k8s"
```
Sources: [dev/tools/client-gen-go.sh:24-26](dev/tools/client-gen-go.sh)
It then invokes three generators in sequence per API group:
```text
client-gen --output-dir clients/k8s/clientset --clientset-name versioned --input
lister-gen --output-dir clients/k8s/listers
informer-gen --output-dir clients/k8s/informers
--versioned-clientset-package /clientset/versioned
--listers-package /listers
```
The same three calls run again for `extensions/api/v1beta1`, writing into `clients/k8s/extensions/{clientset,listers,informers}`. Finally, the script applies repo-standard Apache license headers to the freshly generated files:
```bash
echo "Fixing license headers..."
"${SCRIPT_ROOT}"/dev/tools/fix-boilerplate
```
Sources: [dev/tools/client-gen-go.sh:24-79](dev/tools/client-gen-go.sh)
`fix-boilerplate` walks the tree and prepends the Apache 2.0 header (skipping `**/zz_generated.*` and `site/**`) using the `dev/tools/shared/headers.py` helper:
```python
excludes = [
"**/zz_generated.*",
"**/site/**",
]
headers.apply_headers_to_tree(repo_root, excludes=excludes)
```
Sources: [dev/tools/fix-boilerplate:31-38](dev/tools/fix-boilerplate), [dev/tools/shared/headers.py:40-72](dev/tools/shared/headers.py)
### End-to-end codegen view
```mermaid
flowchart LR
subgraph Sources["Source packages"]
APIv["api/v1beta1/*.go"]
EXTAPI["extensions/api/v1beta1/*.go"]
CTL["controllers/**.go"]
EXTCTL["extensions/controllers/**.go"]
end
subgraph Driver["go generate driver"]
CODEGEN["codegen.go
//go:generate"]
FIXGEN["dev/tools/fix-go-generate"]
end
subgraph Tools["tools.mod (dev-only deps)"]
CG["controller-gen
(object / crd / rbac)"]
CODEGEN_K["k8s.io/code-generator
(client/lister/informer)"]
end
subgraph Outputs["Generated artifacts"]
DC["api/**/zz_generated.deepcopy.go"]
CRDK["k8s/crds/*.yaml"]
CRDH["helm/crds/*.yaml"]
RBACK["k8s/*rbac.generated.yaml"]
RBACH["helm/templates/*rbac.generated.yaml"]
CLS["clients/k8s/{clientset,listers,informers}"]
EXTCLS["clients/k8s/extensions/{clientset,listers,informers}"]
end
FIXGEN -->|go generate ./...| CODEGEN
CODEGEN --> CG
CODEGEN --> CODEGEN_K
APIv --> CG
EXTAPI --> CG
CTL --> CG
EXTCTL --> CG
CG --> DC
CG --> CRDK
CG --> CRDH
CG --> RBACK
CG --> RBACH
APIv --> CODEGEN_K
EXTAPI --> CODEGEN_K
CODEGEN_K --> CLS
CODEGEN_K --> EXTCLS
CLS -->|fix-boilerplate| CLS
EXTCLS -->|fix-boilerplate| EXTCLS
```
Sources: [codegen.go:20-28](codegen.go), [dev/tools/client-gen-go.sh:24-79](dev/tools/client-gen-go.sh), [dev/tools/fix-go-generate:30-31](dev/tools/fix-go-generate), [tools.mod:88-97](tools.mod)
## Lint configuration
The repository runs two distinct Go linters: a general-purpose `golangci-lint` and a Kubernetes-API-specific KAL (Kube-API Linter). Both pull their binaries from the dev tool module via `go tool -modfile=dev/tools/go.mod`.
### General Go lint (`lint-go`)
`dev/tools/lint-go` shells out to `golangci-lint` with the repo's config:
```python
args = ["golangci-lint", "run", f"--config={repo_root}/dev/tools/.golangci.yaml"]
if "--fix" in sys.argv:
args.append("--fix")
result = subprocess.run(utils.go_tool_args(*args), cwd=repo_root)
```
Sources: [dev/tools/lint-go:22-28](dev/tools/lint-go)
The helper `utils.go_tool_args` prepends `go tool -modfile=/dev/tools/go.mod`:
```python
def go_tool_args(*args):
repo_root = get_repo_root()
return ["go", "tool", f"-modfile={repo_root}/dev/tools/go.mod", *args]
```
Sources: [dev/tools/shared/utils.py:64-67](dev/tools/shared/utils.py)
`.golangci.yaml` enables a curated set of linters (`depguard`, `revive`, `staticcheck`, `govet`, `errcheck`, `ineffassign`, `unparam`, `testifylint`, `sloglint`, `misspell`, modernize, …) and bans `import "sort"` via `depguard`, steering callers to the `slices` package. Formatters (`gofmt`, `goimports`) are configured alongside, with `third_party`, `builtin`, and `examples` excluded:
```yaml
depguard:
rules:
forbid-pkg-errors:
deny:
- pkg: sort
desc: Should be replaced with slices package
```
Sources: [dev/tools/.golangci.yaml:1-55](dev/tools/.golangci.yaml)
### Kube-API Linter (`lint-api`)
KAL is shipped as a `golangci-lint` plugin and must be compiled into a custom golangci-lint binary. `dev/tools/lint-api` builds it on first use, then runs it against the API trees only:
```python
kal_binary = os.path.join(tools_dir, "tmp", "bin", "golangci-kube-api-linter")
if not os.path.exists(kal_binary):
build_script = os.path.join(tools_dir, "build-kal")
build_result = subprocess.run([build_script], cwd=repo_root)
...
args = [kal_binary, "run", "--config", kal_config]
if "--fix" in sys.argv:
args.append("--fix")
args.extend(["./api/...", "./extensions/api/..."])
```
Sources: [dev/tools/lint-api:31-53](dev/tools/lint-api)
`build-kal` compiles the custom binary using `golangci-lint custom`:
```bash
(cd "${SCRIPT_DIR}" && go tool -modfile "${SCRIPT_DIR}/go.mod" golangci-lint custom)
```
Sources: [dev/tools/build-kal:19-26](dev/tools/build-kal)
The output lives at `dev/tools/tmp/bin/golangci-kube-api-linter` — note that `make clean` removes `dev/tools/tmp` to force a rebuild ([Makefile:136-139](Makefile)).
`.golangci-kal.yml` disables all default linters (`default: none`) and enables only `kubeapilinter` with an explicit whitelist of KAL checks for CRD authoring discipline. The list includes:
| KAL linter | Enforces |
|---|---|
| `nobools` / `nofloats` | No `bool`/`float` fields in API types |
| `commentstart` / `jsontags` | godoc starts with field name, all fields have JSON tags |
| `optionalorrequired` | Every field is marked `+optional` or `+required` |
| `statussubresource` / `statusoptional` | `status` is a subresource with optional fields |
| `nophase` / `nomaps` / `nonullable` | No `phase` field, no maps, no `nullable` markers |
| `conflictingmarkers` | `default` cannot coexist with `required` |
| `duplicatemarkers` / `uniquemarkers` | No duplicate or non-unique markers on a field |
| `forbiddenmarkers` | Bans `PreserveUnknownFields`, `XPreserveUnknownFields`, `EmbeddedResource`, etc. |
Sources: [dev/tools/.golangci-kal.yml:11-57](dev/tools/.golangci-kal.yml)
Path rules restrict KAL to `api/.*` paths, and explicitly exclude `_test.go` and `zz_generated.*\.go$` so generated deepcopy code is not flagged:
```yaml
exclusions:
generated: strict
paths:
- _test\.go
- zz_generated.*\.go$
rules:
- path-except: "api/.*"
linters:
- kubeapilinter
```
Sources: [dev/tools/.golangci-kal.yml:58-68](dev/tools/.golangci-kal.yml)
## Unit tests: `test-unit`
`dev/tools/test-unit` is the single entry point used by both `make test-unit` and CI. It runs Go and Python suites and emits JUnit XML files under `bin/`.
### Go tests
It enumerates all packages with `go list ./...`, filters out `test/e2e`, then runs them under `gotestsum` (a Go test wrapper that produces JUnit), with `-race` enabled:
```python
go_list_output = subprocess.check_output(["go", "list", "./..."], cwd=repo_root, text=True)
filtered_packages = [pkg for pkg in packages if "test/e2e" not in pkg]
result = subprocess.run(utils.go_tool_args(
"gotestsum",
f"--junitfile={repo_root}/bin/unit-junit.xml",
"--",
"-race",
*filtered_packages
), cwd=repo_root)
```
Sources: [dev/tools/test-unit:37-53](dev/tools/test-unit)
`gotestsum` is provided by `tool gotest.tools/gotestsum` in `dev/tools/go.mod` ([dev/tools/go.mod:5-10](dev/tools/go.mod)), so it runs via the same `go tool -modfile=…` mechanism as the other dev tools.
### Python tests
Two Python suites under `clients/python/agentic-sandbox-client/` are exercised inside a clean per-suite venv:
```python
PYTHON_TEST_SUITES = [
{"name": "sandbox-router", "dir": "clients/python/.../sandbox-router", "requirements": "requirements.txt"},
{"name": "k8s-agent-sandbox", "dir": "clients/python/.../k8s_agent_sandbox", "requirements": "requirements.txt"},
]
```
For each suite the runner:
1. Wipes any existing venv at `bin/python-venv-` (verifying `pyvenv.cfg` exists before removal, to avoid trashing an unrelated directory).
2. Creates a fresh venv with `python -m venv`.
3. Installs `requirements.txt`; if absent and the suite is `k8s-agent-sandbox`, installs the package itself with `pip install -e .[test]`.
4. Installs `pytest` separately.
5. Runs `pytest` with `--junitxml=bin/python--junit.xml -v`.
Sources: [dev/tools/test-unit:55-108](dev/tools/test-unit)
The runner returns the Go exit code first if non-zero, otherwise the Python exit code — both must pass for `make test-unit` to succeed.
## Shared helpers and tool module layout
The Python scripts under `dev/tools/` share a small library:
| Module | Purpose |
|---|---|
| `shared/utils.py` | `get_repo_root()`, `go_tool_args()`, image-tag/git helpers used by deploy/release scripts |
| `shared/headers.py` | Apache-header insertion engine used by `fix-boilerplate` |
| `shared/golang.py` | Walks the repo's Go modules (used by `fix-gomod`, `fix-go-format`) |
| `shared/git_ops.py` | Git helpers used by release/tagging flows |
Sources: [dev/tools/shared/utils.py:21-67](dev/tools/shared/utils.py), [dev/tools/shared/headers.py:23-72](dev/tools/shared/headers.py), [dev/tools/fix-boilerplate:31-38](dev/tools/fix-boilerplate), [dev/tools/fix-gomod:24-39](dev/tools/fix-gomod)
A compact map of the repository's module/tool surfaces:
```text
repo/
├── go.mod -> production module (controller, controllers, api)
├── tools.mod -> codegen tools: controller-gen, code-generator
│ (used by //go:generate via `go tool -modfile=tools.mod`)
├── codegen.go -> hosts all //go:generate directives
├── api/ -> kubebuilder-marked source for CRDs / deepcopy
├── extensions/api/ -> same, for extensions group
├── clients/k8s/ -> code-generator output (clientset/listers/informers)
├── k8s/crds/, k8s/*rbac.generated.yaml -> raw manifest output
├── helm/crds/, helm/templates/*rbac.generated.yaml -> Helm chart output
└── dev/tools/
├── go.mod -> lint/test tools: golangci-lint, gotestsum,
│ kube-api-linter, benchstat
├── .golangci.yaml -> general Go lint config
├── .golangci-kal.yml -> KAL config (CRD conventions)
├── client-gen-go.sh -> drives client/lister/informer-gen
├── fix-go-generate, fix-boilerplate, fix-go-format, fix-gomod
├── lint-go, lint-api, build-kal
├── test-unit, test-e2e
├── update-toc, verify-toc
└── shared/ -> Python helpers
```
Sources: [tools.mod:1-97](tools.mod), [dev/tools/go.mod:1-12](dev/tools/go.mod), [codegen.go:20-28](codegen.go)
The three-module split is the key invariant: production code depends only on `go.mod`; CRD/client codegen tools come from `tools.mod`; lint/test tools come from `dev/tools/go.mod`. Each has its own `tool` block so `go tool -modfile=` is the universal invocation pattern.
## Documentation utilities
Two small Bash scripts maintain table-of-contents stability on KEP docs:
- `update-toc` (`make toc-update`) installs `sigs.k8s.io/mdtoc@v1.1.0` into a `mktemp -d` directory (to avoid mutating the project go.mod/go.sum), then runs it across `docs/keps/**.md` with `--max-depth=5`, filtering out the names listed in `dev/tools/.notableofcontents`. Sources: [dev/tools/update-toc:22-52](dev/tools/update-toc)
- `verify-toc` (`make toc-verify`) does the same install, then runs `mdtoc --dryrun` to fail when the TOCs are out of date. The two scripts share a `TOOL_VERSION=v1.1.0` constant with a "keep in sync" comment. Sources: [dev/tools/verify-toc:21-53](dev/tools/verify-toc)
`generate-api-docs` follows the same temp-install pattern for `github.com/elastic/crd-ref-docs`, rendering `docs/api.md` from the CRD markers ([Makefile:10-15](Makefile)).
## Putting it together
The combined effect is that a maintainer editing an API type only needs to run `make all` ([Makefile:1-2](Makefile)) to:
1. Re-run all `//go:generate` directives → updated deepcopy, CRDs (in both `k8s/` and `helm/`), RBAC, and typed clients with license headers fixed up.
2. Rebuild the controller binary with embedded version metadata.
3. Run general Go lint (`golangci-lint`) and the CRD-convention lint (KAL) against `api/`/`extensions/api/`.
4. Run Go + Python unit tests with race detection and JUnit output.
5. Verify KEP TOCs are current.
The same scripts run under Prow CI (presubmits and postsubmits) so local and CI behavior remain identical, anchored on `dev/tools/` and the two side-car module files (`tools.mod`, `dev/tools/go.mod`) that isolate developer tooling from production dependencies.
Sources: [Makefile:1-2](Makefile), [codegen.go:20-28](codegen.go), [dev/tools/client-gen-go.sh:24-79](dev/tools/client-gen-go.sh), [dev/tools/test-unit:37-108](dev/tools/test-unit), [docs/development.md:34-49](docs/development.md)
---
## 28. E2E Test Framework
> Layout of the Go e2e suite, the framework client/predicates/watchset helpers, and the parallel/replica/shutdown scenario coverage.
- Page Markdown: https://grok-wiki.com/public/wiki/kubernetes-sigs-agent-sandbox-c3f2597a654a/pages/28-e2e-test-framework.md
- Generated: 2026-05-25T22:53:04.074Z
### Source Files
- `test/e2e/README.md`
- `test/e2e/framework/client.go`
- `test/e2e/framework/watchset.go`
- `test/e2e/basic_test.go`
- `test/e2e/parallelism_test.go`
- `test/e2e/extensions/warmpool_rollout_test.go`
- `docs/testing.md`
Relevant source files
The following files were used as context for generating this wiki page:
- [test/e2e/README.md](test/e2e/README.md)
- [docs/testing.md](docs/testing.md)
- [test/e2e/framework/client.go](test/e2e/framework/client.go)
- [test/e2e/framework/watchset.go](test/e2e/framework/watchset.go)
- [test/e2e/framework/context.go](test/e2e/framework/context.go)
- [test/e2e/framework/testlogs.go](test/e2e/framework/testlogs.go)
- [test/e2e/framework/componentlogs.go](test/e2e/framework/componentlogs.go)
- [test/e2e/framework/predicates/predicates.go](test/e2e/framework/predicates/predicates.go)
- [test/e2e/framework/predicates/conditions.go](test/e2e/framework/predicates/conditions.go)
- [test/e2e/framework/predicates/metadata.go](test/e2e/framework/predicates/metadata.go)
- [test/e2e/framework/predicates/sandbox.go](test/e2e/framework/predicates/sandbox.go)
- [test/e2e/basic_test.go](test/e2e/basic_test.go)
- [test/e2e/replicas_test.go](test/e2e/replicas_test.go)
- [test/e2e/shutdown_test.go](test/e2e/shutdown_test.go)
- [test/e2e/parallelism_test.go](test/e2e/parallelism_test.go)
- [test/e2e/extensions/warmpool_rollout_test.go](test/e2e/extensions/warmpool_rollout_test.go)
- [test/e2e/utils.go](test/e2e/utils.go)
- [Makefile](Makefile)
# E2E Test Framework
The agent-sandbox end-to-end suite is a Go `testing` package that runs against a real Kubernetes cluster (typically a kind cluster created by `make deploy-kind`), driving the `Sandbox`, `SandboxClaim`, `SandboxTemplate`, and `SandboxWarmPool` CRDs through the agent-sandbox controller. It is structured around a thin `framework` package that owns cluster credentials, object lifecycle, predicate evaluation, and a shared watch fan-out, and a set of `*_test.go` scenarios that exercise basic reconciliation, replica scaling, shutdown lifecycle, parallel reconciliation under load, and warm-pool rollout strategies.
This page maps the framework's public surface, the wiring between `TestContext`, `ClusterClient`, predicates, and `WatchSet`, and then walks through the parallel, replica, shutdown, and warm-pool scenarios to show how the helpers are composed.
## How tests are launched
Tests assume that a cluster is reachable through `bin/KUBECONFIG` (or `$KUBECONFIG`) and that the agent-sandbox controller is already installed. The README documents the canonical invocation:
```shell
make deploy-kind
go test ./test/e2e/... --parallel=1
```
`--parallel=1` serializes the top-level tests because they share a single cluster and patch the controller `Deployment` in some scenarios.
Sources: [test/e2e/README.md:1-30](), [Makefile:33-68](), [test/e2e/framework/context.go:40-67]()
The `Makefile` exposes higher-level entry points used by CI:
| Target | Behavior |
| --- | --- |
| `make test-e2e` | Runs `./dev/ci/presubmits/test-e2e` (race detector toggled via `RACE`). |
| `make test-e2e-race` | Same script with `RACE=1`. |
| `make test-e2e-benchmarks` | Same script with `--suite benchmarks`. |
| `make deploy-kind` / `make delete-kind` | Provision/teardown the kind cluster used by the suite. |
Sources: [Makefile:33-68](), [docs/testing.md:12-35]()
## Repository layout
```text
test/e2e/
├── README.md # run instructions
├── basic_test.go # TestSimpleSandbox
├── replicas_test.go # TestSandboxReplicas
├── shutdown_test.go # TestSandboxShutdownTime, retain-expiry
├── parallelism_test.go # parallel Sandboxes / SandboxClaims
├── volumeclaimtemplate_test.go
├── chromesandbox_test.go, chromesandbox_claim_test.go
├── utils.go # AtomicTimeDuration helper
├── framework/ # shared client + watch + predicates
│ ├── client.go # ClusterClient (Get/Create/WaitForObject/Watch)
│ ├── context.go # TestContext, kubeconfig, before/afterEach
│ ├── watchset.go # WatchSet/ResourceWatch/Subscription fan-out
│ ├── testlogs.go # per-test artifacts log file
│ ├── componentlogs.go # kubelet/containerd log capture from kind nodes
│ └── predicates/ # ObjectPredicate library
└── extensions/ # CRD-specific scenarios
├── warmpool_rollout_test.go
├── warmpool_sandbox_watcher_test.go
├── pythonruntime_test.go
├── sandboxclaim_metric_test.go
└── shutdown_policy_test.go
```
Sources: [test/e2e/framework/client.go:15-45](), [test/e2e/framework/context.go:14-37](), [test/e2e/extensions/warmpool_rollout_test.go:15-34]()
## Framework architecture
The framework wraps `*testing.T` (or `*testing.B`) into a `TestContext` that exposes a `ClusterClient` and a per-test `WatchSet`. The `ClusterClient` delegates CRUD to a controller-runtime `client.Client`, while waits and watches go through a `dynamic.Interface` so the framework can drive arbitrary GVRs without registering Go types ahead of time.
```mermaid
flowchart LR
subgraph TestCase["*_test.go scenario"]
TC["framework.NewTestContext(t)"]
end
subgraph Framework["test/e2e/framework"]
Ctx["TestContext\n(context.go)"]
CC["ClusterClient\n(client.go)"]
WS["WatchSet\n(watchset.go)"]
Logs["logCapturingT\n(testlogs.go)"]
Comp["dumpControllerLogs\nMustGetKubeletLogs"]
end
subgraph Predicates["framework/predicates"]
IF["ObjectPredicate interface"]
Status["StatusPredicate\nConditionReasonPredicate\nReadyReplicasPredicate"]
Meta["Annotation/Label/Owner\nNotDeleted/HasDeletionTimestamp"]
Sandbox["SandboxHasStatus\n(go-cmp diff)"]
end
subgraph K8s["Kubernetes cluster (kind)"]
API["kube-apiserver"]
Ctrl["agent-sandbox-controller\n(agent-sandbox-system)"]
CRDs["Sandbox / SandboxClaim\nSandboxTemplate / SandboxWarmPool"]
end
TC --> Ctx
Ctx --> CC
Ctx --> Logs
Ctx --> Comp
CC --> WS
CC -->|"controller-runtime client"| API
WS -->|"dynamic.Interface\nWatch loop"| API
CC --> Predicates
Predicates --> Status
Predicates --> Meta
Predicates --> Sandbox
Ctrl --> CRDs
API --> Ctrl
```
Sources: [test/e2e/framework/context.go:91-161](), [test/e2e/framework/client.go:51-78](), [test/e2e/framework/watchset.go:59-148](), [test/e2e/framework/predicates/predicates.go:26-33]()
### `TestContext` lifecycle
`NewTestContext` is the single entry point used by every scenario. It loads the kubeconfig, builds both a typed `controller-runtime` client and a `dynamic.Interface`, creates a `WatchSet`, attaches log capture, and registers cleanup hooks. `beforeEach` validates that the CRDs, the `agent-sandbox-system` namespace, and the `agent-sandbox-controller` Deployment exist before the test body runs; `afterEach` dumps controller logs into the test's artifacts directory if the test failed.
Key responsibilities, in order:
1. Resolve `KUBECONFIG` (env var, otherwise `/bin/KUBECONFIG`).
2. Build `*rest.Config`, shared HTTP client, controller-runtime client, and dynamic client against the cluster.
3. Create a `WatchSet(dynamicClient)` and register `watchSet.Close()` as test cleanup.
4. Wrap `T` with `logCapturingT` so every `Logf` / `Errorf` is mirrored to `//test.log` with elapsed-time prefixes.
5. Run `validateAgentSandboxInstallation()` — fails fast if the controller or CRDs are missing.
6. Register `afterEach()` to call `dumpControllerLogs` when `t.Failed()`.
Sources: [test/e2e/framework/context.go:91-178](), [test/e2e/framework/context.go:180-240](), [test/e2e/framework/client.go:521-549](), [test/e2e/framework/testlogs.go:34-77]()
### `ClusterClient` API
`ClusterClient` is the workhorse for all scenarios. Every method is `Helper()`-marked so failures point at the calling test line, and create operations register cleanup that deletes the object and then blocks on `WaitForObjectNotFound` so the next test starts with a clean namespace.
| Method | Purpose |
| --- | --- |
| `Get` / `List` / `Update` / `Delete` | Thin error-wrapping facades over the controller-runtime client. |
| `CreateWithCleanup` / `MustCreateWithCleanup` | Create the object and schedule deletion + wait-not-found in `t.Cleanup`. Uses `context.Background()` during cleanup since the test context is already done. |
| `MustUpdateObject[T]` | Re-fetch the latest version, apply `updateFunc`, then `Update`. Avoids stale-version conflicts. |
| `MatchesPredicates` / `MustMatchPredicates` / `MustExist` | One-shot predicate evaluation against the current API state. |
| `ValidateObjectNotFound` / `WaitForObjectNotFound` | Confirm deletion; namespaces get a 3-minute timeout because of cascading cleanup, everything else is 60s. |
| `PollUntilObjectMatches` | 1-second polling fallback used when watches aren't a good fit (for example, the shutdown-time scenario). |
| `WaitForObject` / `MustWaitForObject` | Watch-driven wait with predicate evaluation; the preferred timing primitive. |
| `Watch` / `Watch[T]` / `MustWatch[T]` | Generic event callback against a `GVR + WatchFilter`. |
| `WaitForSandboxReady` / `WaitForWarmPoolReady` / `GetSandbox` | Convenience wrappers for the specific CRDs. |
| `PortForward` | Spawns `kubectl port-forward` for tests that need direct pod access (e.g., Chrome debug port). |
| `ExecuteOnNode` / `IsKindCluster` | `docker exec` into the kind node container; tests can opt into kind-only behavior. |
`DefaultTimeout` is 60 seconds; a shorter deadline can be supplied via the caller's context, and `WaitForObject` clamps to whichever is smaller.
Sources: [test/e2e/framework/client.go:46-265](), [test/e2e/framework/client.go:485-549](), [test/e2e/framework/client.go:551-718]()
### `WatchForObject` and the typed `Watch[T]` helper
`WaitForObject` is the single timing primitive that the suite leans on. It first checks if the object already satisfies the predicates (fast path), looks up the GVR from a hard-coded GVK table (`gvrForGVK` covers Pod, Deployment, Namespace, Sandbox, SandboxWarmPool, SandboxClaim), subscribes through the shared `WatchSet`, and runs the predicate callback on each event. On timeout or non-match it logs the last observed object as YAML, which is invaluable when debugging asymmetric status diffs.
```go
// test/e2e/framework/client.go (excerpt)
done, err := Watch(ctx, cl, gvr, watchFilter, func(event watch.Event, obj *unstructured.Unstructured) (bool, error) {
if event.Type == watch.Deleted {
return false, fmt.Errorf("object was deleted while waiting for predicates to be satisfied")
}
var notMatching []predicates.ObjectPredicate
for _, predicate := range p {
if match, err := predicate.Matches(obj); err != nil {
return false, err
} else if !match {
notMatching = append(notMatching, predicate)
}
}
lastNotMatching = notMatching
lastObject = obj.DeepCopy()
return len(notMatching) == 0, nil
})
```
The generic `Watch[T]` decodes events to either `*unstructured.Unstructured` (passthrough) or any concrete `client.Object` by running `DefaultUnstructuredConverter.FromUnstructured`. Bookmark events are dropped, and `Error` events become callback errors. `MustWatch` swallows `context.Canceled` so tests that intentionally cancel the watch context don't fail.
Sources: [test/e2e/framework/client.go:267-349](), [test/e2e/framework/client.go:351-428](), [test/e2e/framework/client.go:430-473]()
### `WatchSet`: shared dynamic watches
A naïve "one Watch per `WaitForObject` call" would create one HTTP watch per assertion and incur listing + setup latency on every wait. The framework instead keeps **one** persistent watch per `(GVR, namespace)` and fans out events to per-call subscriptions that apply a `WatchFilter{Namespace, Name}`.
```mermaid
classDiagram
class WatchSet {
+watches: map[watchKey]*ResourceWatch
+dynamicClient: dynamic.Interface
+Subscribe(gvr, filter) *Subscription
+Close()
-getOrCreateWatch(gvr, namespace) *ResourceWatch
-removeWatchIfIdle(rw)
}
class ResourceWatch {
+gvr: GroupVersionResource
+namespace: string
+subscriptions: map[uint64]*Subscription
+cancelWatchLoop: CancelFunc
-watchLoop(ctx)
-broadcast(event)
-subscribe(filter) *Subscription
-unsubscribe(sub)
}
class Subscription {
+Events: chan watch.Event
+filter: WatchFilter
+Close()
}
class WatchFilter {
+Namespace: string
+Name: string
}
WatchSet "1" --> "*" ResourceWatch : owns
ResourceWatch "1" --> "*" Subscription : broadcasts to
Subscription --> WatchFilter : per-call filter
```
Notable design details:
- The watch loop runs under `context.Background()` so it survives the caller's per-call context and only stops when `WatchSet.Close()` runs (registered in `NewTestContext`) or when the last subscription unsubscribes (`removeWatchIfIdle`).
- On a `watch.Error` event the resource version is reset to `""` and the loop restarts from scratch; transient errors back off for 100 ms before retry.
- `broadcast` always forwards `watch.Error` and `watch.Bookmark` events to all subscribers, but applies name/namespace filtering for `Added/Modified/Deleted`.
- Each subscription's `Events` channel is buffered at 100 to absorb bursts without blocking the watch loop.
- Cluster-scoped resources use `namespace == ""` and a single cluster-wide watch.
Sources: [test/e2e/framework/watchset.go:29-148](), [test/e2e/framework/watchset.go:150-211](), [test/e2e/framework/watchset.go:230-328]()
### Predicates
`predicates.ObjectPredicate` is a two-method interface: `Matches(obj) (bool, error)` and `fmt.Stringer`. The stringer requirement means `WaitForObject` can log the unmatched predicates as `lastNotMatching` for fast debugging.
| Predicate | File | Behavior |
| --- | --- | --- |
| `ReadyConditionIsTrue` (`StatusPredicate`) | `conditions.go` | Looks for `status.conditions[Type=Ready, Status=True]` via unstructured conversion. |
| `ConditionReasonEquals(type, reason)` | `conditions.go` | Same shape but matches on `Reason`; used by the retain-expiry test. |
| `ReadyReplicasConditionIsTrue` | `conditions.go` | `status.readyReplicas == spec.replicas`; used to wait for the controller Deployment after patching. |
| `ObservedGenerationMatchesGeneration` | `conditions.go` | `status.observedGeneration == metadata.generation`; pairs with the replicas check on controller restarts. |
| `SandboxHasStatus(want)` | `sandbox.go` | Full `go-cmp` diff against `SandboxStatus`, ignoring `LastTransitionTime` and `PodIPs`. |
| `HasAnnotation` / `HasLabel` / `HasOwnerReferences` | `metadata.go` | Map and slice comparisons; owner refs are sorted by UID before diffing. |
| `NotDeleted` / `HasDeletionTimestamp` | `metadata.go` | DeletionTimestamp checks for cascading-delete and rollout assertions. |
All status predicates funnel through `asUnstructured` and `runtime.DefaultUnstructuredConverter.FromUnstructured`, which lets the same predicate evaluate any CRD without registering its Go type.
Sources: [test/e2e/framework/predicates/predicates.go:26-33](), [test/e2e/framework/predicates/conditions.go:38-189](), [test/e2e/framework/predicates/sandbox.go:38-66](), [test/e2e/framework/predicates/metadata.go:26-148]()
### Logs and artifacts
Two log sinks are wired up per test:
- `logCapturingT` mirrors `Log/Logf/Error/Errorf/Fatal/Fatalf` to `//test.log` with `[ 12.345s]` elapsed-time prefixes. `ARTIFACTS` defaults to `./artifacts`.
- On failure, `dumpControllerLogs` lists pods in `agent-sandbox-system` labelled `app=agent-sandbox-controller`, writes the full log of each to `/controller-.log`, and inlines the last 42 lines into the test output ("following k8s e2e convention").
For kind-specific diagnostics, `componentlogs.go` exposes `MustGetKubeletLogs` and `MustGetContainerdLogs`, both of which shell out via `ClusterClient.ExecuteOnNode` (`docker exec journalctl -u kubelet ...`) and persist results under the same artifacts directory.
Sources: [test/e2e/framework/testlogs.go:26-102](), [test/e2e/framework/context.go:180-240](), [test/e2e/framework/componentlogs.go:25-52]()
## Scenario coverage
The scenario tests are deliberately small and composable. Each one creates a uniquely-named namespace with `time.Now().UnixNano()` to avoid collisions, then registers it with `CreateWithCleanup` so the deferred delete tears down every child resource.
### Basic reconciliation — `TestSimpleSandbox`
`basic_test.go` exercises the happy path: create a `Sandbox` from a tiny `simpleSandbox` fixture (a single `registry.k8s.io/pause:3.10` container with a test annotation and label), wait for the controller to populate the full `SandboxStatus` (Service name, FQDN, replicas, label selector built from `NameHash` of the sandbox name, and a `Ready=True` condition with `SandboxReasonDependenciesReady`), and then assert that the generated Pod and Service exist with the expected owner references and propagated metadata. `NameHash` is an FNV-1a hex digest used to compute the `agents.x-k8s.io/sandbox-name-hash=` selector that the controller stamps onto the Service.
Sources: [test/e2e/basic_test.go:31-132]()
### Replica scaling — `TestSandboxReplicas`
`replicas_test.go` first creates a Sandbox with `Spec.Replicas=1` and waits for the same `Ready=True` status. It then uses `MustUpdateObject` to flip replicas to 0; `MustUpdateObject` re-reads the latest version before mutating so the test does not race the controller. The expected status flips to `Ready=False` with reason `SandboxReasonSuspended`, plus a second `Suspended=True` condition with reason `SandboxReasonSuspendedPodTerminated`. The test finally asserts the Pod is gone (`WaitForObjectNotFound`) while the Service still exists (`NotDeleted` predicate) — verifying that scaling to zero is non-destructive at the Service level.
Sources: [test/e2e/replicas_test.go:30-107](), [test/e2e/framework/client.go:113-129]()
### Shutdown lifecycle — `shutdown_test.go`
Two scenarios cover the `Spec.ShutdownTime` and `Spec.Lifecycle.ShutdownPolicy` surfaces:
- `TestSandboxShutdownTime` creates a Sandbox, then patches `ShutdownTime` to ~10 seconds in the future (truncated to RFC3339 second precision to match Kubernetes storage). It uses `PollUntilObjectMatches` (rather than the watch-based `WaitForObject`) to observe the transitioned status — `Service`/`ServiceFQDN` cleared, `Replicas: 0`, and `Ready=False` with reason `SandboxReasonExpired`. It then asserts the wall-clock time has passed the shutdown moment and that both the Pod and Service are deleted via `WaitForObjectNotFound`.
- `TestSandboxRetainedExpiryPreservesFinishedCondition` builds a Sandbox whose container exits cleanly (`busybox sh -c exit 0`) with `ShutdownPolicyRetain`. It first uses `ConditionReasonEquals(SandboxConditionFinished, SandboxReasonPodSucceeded)` to confirm the Finished condition is set, then `require.Eventually` waits until both `Ready=Expired` and `Finished=PodSucceeded` are simultaneously present alongside cleared service fields — proving that the retain policy preserves the Finished condition through expiry.
Sources: [test/e2e/shutdown_test.go:31-103](), [test/e2e/shutdown_test.go:105-160]()
### Parallel reconciliation — `parallelism_test.go`
`parallelism_test.go` stresses the controller's worker pool. The helper `patchControllerConcurrency` is the most interesting piece: it snapshots the controller Deployment, rewrites the container args to bump `--sandbox-concurrent-workers`, `--sandbox-claim-concurrent-workers`, `--sandbox-warm-pool-concurrent-workers` to 10, raises `--kube-api-qps=50` / `--kube-api-burst=100`, ensures `--extensions` is present, and waits for the new pod to roll out (using `ReadyReplicasConditionIsTrue` + `ObservedGenerationMatchesGeneration`). Cleanup restores the original spec under `retry.RetryOnConflict` and sleeps 5 s for leader election to settle.
Three scenarios then drive load through the patched controller:
| Test | Setup | What it stresses |
| --- | --- | --- |
| `TestParallelSandboxes` | 20 Sandboxes created concurrently via goroutines, each waiting on `ReadyConditionIsTrue`. | Pure Sandbox reconciliation under concurrency. |
| `TestParallelSandboxClaimsWithSufficientWarmPool` | `SandboxWarmPool` of 25 replicas, 20 parallel claims. | Claim binding when pre-warmed pods are available. |
| `TestParallelSandboxClaimsWithInsufficientWarmPool` | Pool of 5, 20 parallel claims. | Forces on-demand pod creation when the pool is exhausted. |
Errors flow through a buffered `errCh`; the test fails after `wg.Wait()` so each goroutine completes before reporting.
Sources: [test/e2e/parallelism_test.go:36-108](), [test/e2e/parallelism_test.go:110-145](), [test/e2e/parallelism_test.go:147-218]()
### Warm-pool rollout — `extensions/warmpool_rollout_test.go`
The extensions suite owns the `SandboxWarmPool` rollout semantics. Four scenarios share the helpers `createSandboxTemplate`, `createSandboxWarmPool`, `updateSandboxTemplateSpec` (appends `TEST_ENV=updated` to the pod), `verifySandboxStaysSame`, `verifySandboxRecreated`, and `verifySandboxHasUpdatedSpec` (which lists Sandboxes in the namespace, filters by `metav1.IsControlledBy(warmPool)`, and uses `require.Eventually` to wait for a fresh one).
```mermaid
stateDiagram-v2
[*] --> Created : create SandboxTemplate + SandboxWarmPool
Created --> Ready : WaitForWarmPoolReady\n(ReadyReplicasConditionIsTrue)
Ready --> TemplateUpdated : Update SandboxTemplate
state Strategy <>
TemplateUpdated --> Strategy
Strategy --> StaysSame : default / OnReplenish\n(verifySandboxStaysSame)
StaysSame --> Replenished : Delete existing Sandbox\n(verifyOnReplenishLifecycle)
Replenished --> Ready : new Sandbox has\nTEST_ENV=updated
Strategy --> Recreated : Recreate\n(verifySandboxRecreated)
Recreated --> Ready : new Sandbox observed,\nold marked for deletion
Strategy --> NoRollout : metadata-only update\n(TestWarmPoolRolloutMetadataUpdate)
NoRollout --> Ready : sandbox unchanged
```
Concrete scenarios:
- `TestWarmPoolRollout` is a table test over `default`, `onreplenish`, and `recreate` strategies, asserting the expected lifecycle (`verifyOnReplenishLifecycle` for the first two, `verifySandboxRecreated` for the third).
- `TestWarmPoolRolloutMultiTemplateIsolation` verifies that updating template A under the `Recreate` strategy rolls only warm-pool A's sandbox while warm-pool B's sandbox is untouched (no deletion timestamp).
- `TestWarmPoolRolloutSwitchTemplate` changes `Spec.TemplateRef.Name` from A to B (specs identical) and asserts that the new sandbox is annotated with the new template name via `sandboxv1beta1.SandboxTemplateRefAnnotation`.
- `TestWarmPoolRolloutMetadataUpdate` updates only labels in the template's pod metadata and asserts no rollout occurs (the existing sandbox keeps an empty deletion timestamp after a 5 s settling sleep).
Sources: [test/e2e/extensions/warmpool_rollout_test.go:33-136](), [test/e2e/extensions/warmpool_rollout_test.go:138-240](), [test/e2e/extensions/warmpool_rollout_test.go:242-442]()
## How the pieces compose in a typical test
A representative wait sits on top of three independent abstractions cooperating:
```text
test code framework.ClusterClient framework.WatchSet dynamic.Interface
───────── ──────────────────────── ────────────────── ─────────────────
WaitForObject(...)
│ fast-path MatchesPredicates
│ │ ↓ miss
│ gvkForObject / gvrForGVK
│ │
│ Subscribe(gvr, {ns,name}) ──────► getOrCreateWatch(gvr,ns)
│ │ first time?
│ └─► go watchLoop ──── Watch(ctx, ListOpts{Watch:true})
│ ◄─────── sub.Events (buffered 100) broadcast(event) ◄─── ResultChan()
│ │ run predicates per event
│ └─► last YAML logged on miss
│ └─► return on first all-match or timeout
│ sub.Close() ─► removeWatchIfIdle (keeps watch if other subs alive)
```
The result is that ten parallel `WaitForObject` calls in a `TestParallelSandboxes` goroutine pool reuse a single Sandbox watch per namespace, which keeps the API server load low while still giving each test a precise, predicate-driven completion signal.
Sources: [test/e2e/framework/client.go:267-349](), [test/e2e/framework/watchset.go:80-211](), [test/e2e/parallelism_test.go:118-145]()
## Summary
The e2e framework is intentionally small: `TestContext` boots a kubeconfig-driven client pair per test, `ClusterClient` wraps controller-runtime CRUD with cleanup-aware helpers, `predicates.ObjectPredicate` provides composable assertions over arbitrary CRDs via unstructured conversion, and `WatchSet` keeps a single dynamic watch per `(GVR, namespace)` so dozens of concurrent waits remain cheap. The scenario files build on these primitives to cover the Sandbox happy path, replica scaling, time- and policy-driven shutdown, controller behavior under parallel Sandbox and SandboxClaim load, and the full matrix of `SandboxWarmPool` rollout strategies.
---
## 29. Load Testing & CI Pipelines
> cluster-loader-based load-test recipes plus the prowjob presubmit/periodic configuration that runs them in CI.
- Page Markdown: https://grok-wiki.com/public/wiki/kubernetes-sigs-agent-sandbox-c3f2597a654a/pages/29-load-testing-ci-pipelines.md
- Generated: 2026-05-25T23:21:59.475Z
### Source Files
- `dev/load-test/README.md`
- `dev/load-test/cluster-loader-sandbox.yaml`
- `dev/ci/presubmits`
- `dev/ci/periodics`
- `docs/prowjob_manual_run.md`
> ⚠️ The agent returned an invalid wiki page. This page needs recovery.
>
> First failure: the page did not include the required "# Load Testing & CI Pipelines" heading near the top
> Retry failure: the page did not include the required "# Load Testing & CI Pipelines" heading near the top
Relevant source files
The following files were used as context for generating this wiki page:
- [dev/load-test/README.md](dev/load-test/README.md)
- [dev/load-test/agent-sandbox-load-test.yaml](dev/load-test/agent-sandbox-load-test.yaml)
- [dev/load-test/cluster-loader-sandbox.yaml](dev/load-test/cluster-loader-sandbox.yaml)
- [dev/load-test/test-recipes/README-rapid-burst.md](dev/load-test/test-recipes/README-rapid-burst.md)
- [dev/load-test/test-recipes/rapid-burst-test.yaml](dev/load-test/test-recipes/rapid-burst-test.yaml)
- [dev/load-test/test-recipes/run_rapid_burst.sh](dev/load-test/test-recipes/run_rapid_burst.sh)
- [dev/load-test/test-recipes/high-volume-test.yaml](dev/load-test/test-recipes/high-volume-test.yaml)
- [dev/load-test/test-recipes/medium-scale-concurrent-load-test.yaml](dev/load-test/test-recipes/medium-scale-concurrent-load-test.yaml)
- [dev/load-test/test-recipes/throughput-test.yaml](dev/load-test/test-recipes/throughput-test.yaml)
- [dev/load-test/test-recipes/warmpool-burst-test.yaml](dev/load-test/test-recipes/warmpool-burst-test.yaml)
- [dev/load-test/test-recipes/monitor/agent-sandbox-controller-monitor.yaml](dev/load-test/test-recipes/monitor/agent-sandbox-controller-monitor.yaml)
- [dev/load-test/test-recipes/templates/cluster-loader-sandbox-template.yaml](dev/load-test/test-recipes/templates/cluster-loader-sandbox-template.yaml)
- [dev/load-test/test-recipes/templates/cluster-loader-warmpool.yaml](dev/load-test/test-recipes/templates/cluster-loader-warmpool.yaml)
- [dev/load-test/test-recipes/templates/cluster-loader-sandbox-claim.yaml](dev/load-test/test-recipes/templates/cluster-loader-sandbox-claim.yaml)
- [dev/load-test/test-recipes/templates/cluster-loader-hpa.yaml](dev/load-test/test-recipes/templates/cluster-loader-hpa.yaml)
- [dev/load-test/test-recipes/templates/cluster-loader-capacity-buffer.yaml](dev/load-test/test-recipes/templates/cluster-loader-capacity-buffer.yaml)
- [dev/ci/periodics/test-load-test](dev/ci/periodics/test-load-test)
- [dev/ci/presubmits/test-e2e](dev/ci/presubmits/test-e2e)
- [dev/ci/presubmits/test-unit](dev/ci/presubmits/test-unit)
- [dev/ci/presubmits/lint-go](dev/ci/presubmits/lint-go)
- [dev/ci/presubmits/lint-api](dev/ci/presubmits/lint-api)
- [dev/ci/presubmits/test-autogen-up-to-date](dev/ci/presubmits/test-autogen-up-to-date)
- [dev/ci/presubmits/shared/utils.py](dev/ci/presubmits/shared/utils.py)
- [dev/ci/shared/runner.py](dev/ci/shared/runner.py)
- [docs/prowjob_manual_run.md](docs/prowjob_manual_run.md)
# Load Testing & CI Pipelines
This page documents how `kubernetes-sigs/agent-sandbox` exercises the controller under load and how those tests are wired into Prow as presubmit and periodic jobs. Two surfaces collaborate: the `dev/load-test/` recipes built on top of [ClusterLoader2](https://github.com/kubernetes/perf-tests/tree/master/clusterloader2) (CL2), and the `dev/ci/` Python entrypoints that Prow invokes. The recipes describe *what* to measure — `Sandbox`/`SandboxClaim` startup latency, throughput, churn, warm-pool burst behaviour — while the CI entrypoints describe *how* to bring up an isolated KinD cluster, push the controller image, run a recipe, and emit a JUnit artifact that Prow can surface in Spyglass.
The two layers are deliberately separable. Any recipe can be run by hand against any cluster (the rapid-burst recipe is targeted at GKE; the entrypoint recipe at KinD), and CI just composes a thin wrapper around the entrypoint recipe with smaller defaults so it fits inside a Prow pod.
## High-Level Architecture
```mermaid
flowchart LR
subgraph Prow["kubernetes/test-infra (Prow)"]
PJ["pj-on-kind.sh /
periodic job spec"]
end
subgraph CIEntry["dev/ci/ (Python wrappers)"]
Periodic["periodics/test-load-test
(LoadTestRunner)"]
Pre_e2e["presubmits/test-e2e"]
Pre_unit["presubmits/test-unit"]
Pre_lintgo["presubmits/lint-go"]
Pre_lintapi["presubmits/lint-api"]
Pre_autogen["presubmits/test-autogen-up-to-date"]
Runner["shared/runner.py
TestRunner"]
end
subgraph DevTools["dev/tools/"]
Kind["create-kind-cluster"]
Push["push-images"]
Deploy["deploy-to-kube
+ deploy-cloud-provider"]
Fix["fix-* scripts"]
Lint["lint-go / lint-api / test-unit / test-e2e"]
end
subgraph LoadTest["dev/load-test/"]
EntryRecipe["agent-sandbox-load-test.yaml
(entrypoint recipe)"]
SandboxTpl["cluster-loader-sandbox.yaml
(Sandbox object template)"]
Recipes["test-recipes/*.yaml
(rapid-burst, throughput,
high-volume, medium-scale, warmpool-burst)"]
Templates["test-recipes/templates/*.yaml
(SandboxTemplate, WarmPool, Claim,
HPA, CapacityBuffer)"]
Monitor["test-recipes/monitor/
ServiceMonitor"]
end
PJ --> Periodic
PJ --> Pre_e2e
PJ --> Pre_unit
PJ --> Pre_lintgo
PJ --> Pre_lintapi
PJ --> Pre_autogen
Periodic --> Runner
Pre_e2e --> Runner
Periodic --> EntryRecipe
EntryRecipe --> SandboxTpl
Recipes --> Templates
Recipes --> Monitor
Runner --> Kind
Runner --> Push
Runner --> Deploy
Pre_unit --> Lint
Pre_lintgo --> Lint
Pre_lintapi --> Lint
Pre_autogen --> Fix
```
Sources: [dev/ci/shared/runner.py:23-80](), [dev/ci/periodics/test-load-test:48-110](), [dev/load-test/agent-sandbox-load-test.yaml:1-56](), [dev/load-test/test-recipes/rapid-burst-test.yaml:1-46]()
## The CI Entrypoint Layer (`dev/ci/`)
Every Prow job in this repo is a small Python script that Prow shells out to. There are two flavours:
1. **Lint/unit wrappers** that just delegate to a script in `dev/tools/`.
2. **Cluster-backed runners** that subclass `TestRunner` and provision a KinD cluster before running a binary.
### `TestRunner` base class
`dev/ci/shared/runner.py` is the spine shared by every cluster-backed CI job. It exposes a four-method lifecycle — `setup_cluster`, `run_tests`, `copy_artifacts`, `main` — and threads the repository root and an `--image-prefix` argument through every invocation:
```python
# dev/ci/shared/runner.py
class TestRunner:
def __init__(self, name, description):
...
self.parser.add_argument(
"--image-prefix",
...
default="kind.local/",
)
def setup_cluster(self, args, extra_push_images_args=None):
image_tag = tools_utils.get_image_tag()
subprocess.run([f"{self.repo_root}/dev/tools/create-kind-cluster",
self.name, "--recreate",
"--kubeconfig", f"{self.repo_root}/bin/KUBECONFIG"])
...
subprocess.run([f"{self.repo_root}/dev/tools/push-images", ...])
subprocess.run([f"{self.repo_root}/dev/tools/deploy-to-kube",
"--image-prefix", args.image_prefix,
"--image-tag", image_tag, "--extensions"])
subprocess.run([f"{self.repo_root}/dev/tools/deploy-cloud-provider"])
```
`setup_cluster` is the same for every job: recreate a KinD cluster named after the job, build and load images into the cluster, deploy the controller plus extension CRDs, and stand up the cloud-provider mock. `run_tests` is `NotImplementedError` — subclasses must override it. `copy_artifacts` is a no-op by default; subclasses override it to publish JUnit files into `$ARTIFACTS`, the Prow-set output directory.
Sources: [dev/ci/shared/runner.py:23-80]()
### Presubmits
| Entrypoint | Subclass / pattern | Underlying script | JUnit output |
|---|---|---|---|
| `dev/ci/presubmits/test-e2e` | `E2ETestRunner(TestRunner)` | `dev/tools/test-e2e --suite {all,tests,benchmarks}` | `bin/e2e-go-junit.xml` → `$ARTIFACTS/junit_go.xml`, `bin/e2e-python-sdk-junit.xml` → `$ARTIFACTS/junit_python_sdk.xml` |
| `dev/ci/presubmits/test-unit` | thin wrapper via `shared/utils.py` | `dev/tools/test-unit` | `bin/unit-junit.xml` → `$ARTIFACTS/junit.xml` |
| `dev/ci/presubmits/lint-go` | thin wrapper | `dev/tools/lint-go` | none |
| `dev/ci/presubmits/lint-api` | thin wrapper | `dev/tools/lint-api` | none |
| `dev/ci/presubmits/test-autogen-up-to-date` | inline | runs every `dev/tools/fix-*` and fails if `git diff --name-only` is non-empty | none |
The lint/unit wrappers reuse `dev/ci/presubmits/shared/utils.py`, which walks up four parents from `__file__` to find the repo root and runs the named tool from `dev/tools/`:
```python
# dev/ci/presubmits/shared/utils.py
def get_repo_root():
presubmit_dir = os.path.dirname(os.path.dirname(os.path.realpath(__file__)))
return os.path.dirname(os.path.dirname(os.path.dirname(presubmit_dir)))
def run_dev_tool(tool_name):
repo_root = get_repo_root()
result = subprocess.run([f"{repo_root}/dev/tools/{tool_name}"])
return result.returncode
```
`test-autogen-up-to-date` is the odd one out — it does not invoke a single tool but iterates over `sorted(glob.glob("dev/tools/fix-*"))`, runs each one, and exits non-zero if any of them produces a diff. It prints the full `git diff` and a remediation hint to re-run the script locally.
Sources: [dev/ci/presubmits/test-e2e:28-47](), [dev/ci/presubmits/test-unit:23-35](), [dev/ci/presubmits/lint-go:21-23](), [dev/ci/presubmits/lint-api:21-23](), [dev/ci/presubmits/test-autogen-up-to-date:22-48](), [dev/ci/presubmits/shared/utils.py:20-29]()
### The periodic load-test entrypoint
`dev/ci/periodics/test-load-test` is the only entry under `dev/ci/periodics/`. It subclasses `TestRunner` as `LoadTestRunner`, takes four CL2 parameters as CLI arguments, and reduces the defaults so the test fits inside a Prow pod backed by KinD:
| Flag | Default | Forwarded as |
|---|---|---|
| `--replicas` | 5 | `CL2_REPLICAS` |
| `--namespaces` | 1 | `CL2_NAMESPACES` |
| `--qps` | 10 | `CL2_QPS` |
| `--namespace-prefix` | `agent-sandbox` | `CL2_NAMESPACE_PREFIX` |
`setup_cluster` is overridden to pass `extra_push_images_args=["--controller-only"]`, so the load-test job only loads the controller image into KinD — none of the auxiliary images needed for full e2e are pushed:
```python
# dev/ci/periodics/test-load-test
def setup_cluster(self, args):
return super().setup_cluster(args, extra_push_images_args=["--controller-only"])
```
`run_tests` (a) installs ClusterLoader2 on the fly by `git clone --depth 1` of `kubernetes/perf-tests` and `go build -o bin/clusterloader2 ./cmd/clusterloader.go`, (b) writes the four CLI flags into a `--testoverrides` file, then (c) invokes `clusterloader2` from `dev/load-test/` (because the entrypoint recipe references `cluster-loader-sandbox.yaml` via a relative path):
```python
cmd = [cl2_path,
f"--testconfig={test_config}",
f"--kubeconfig={kubeconfig}",
f"--testoverrides={overrides_path}",
"--provider=kind",
"--v=2",
f"--report-dir={report_dir}"]
subprocess.run(cmd, cwd=os.path.join(self.repo_root, "dev/load-test"))
```
`copy_artifacts` then copies `bin/junit.xml` to `$ARTIFACTS/junit_load_test.xml` so the Prow UI can render it. The clusterloader2 binary and the temporary kubeconfig are removed at the end of each run so subsequent invocations rebuild fresh.
Sources: [dev/ci/periodics/test-load-test:29-116]()
## The Load-Test Recipe Layer (`dev/load-test/`)
There are two cohorts of CL2 recipes here. The top-level `agent-sandbox-load-test.yaml` is the one driven by CI; the recipes under `test-recipes/` are tuned for larger GKE clusters and are run by humans.
### The CI-driven entrypoint recipe
`dev/load-test/agent-sandbox-load-test.yaml` is intentionally minimal. It accepts four parameters (`CL2_REPLICAS`, `CL2_NAMESPACES`, `CL2_QPS`, `CL2_NAMESPACE_PREFIX`) and executes five CL2 steps:
1. `Start Startup Latency Measurement` — registers a `PodStartupLatency` measurement.
2. `Create Sandboxes` — instantiates `cluster-loader-sandbox.yaml` (a `Sandbox` CR with an alpine container that sleeps 3600s) under the `BurstCreate` tuning set.
3. `Wait for Sandboxes to be Ready` — uses `WaitForRunningPods` against the label selector `group=agent-sandbox-load-test`.
4. `Gather Results` — collects the `SandboxStartupLatency` measurement.
5. `Delete Sandboxes` — drops the replica count to 0.
A TODO at the top of the file notes that the recipe currently measures only the downstream Pod latency, not the end-to-end `Sandbox` lifecycle, and that it should switch to `GenericPrometheusQuery` once `agent_sandbox_creation_latency_ms` metrics are exposed by the controller:
```yaml
# dev/load-test/agent-sandbox-load-test.yaml
# TODO: Replace PodStartupLatency with GenericPrometheusQuery
# once agent_sandbox_creation_latency_ms metrics are available to measure
# end-to-end controller overhead.
- Identifier: SandboxStartupLatency
Method: PodStartupLatency # Warning: this only measures the downstream Pod latency, not the end-to-end Sandbox lifecycle
```
Sources: [dev/load-test/agent-sandbox-load-test.yaml:1-56](), [dev/load-test/cluster-loader-sandbox.yaml:1-17](), [dev/load-test/README.md:33-86]()
### Test recipes under `dev/load-test/test-recipes/`
The five `*.yaml` recipes under `test-recipes/` cover different scenarios. They are designed to be driven by hand (or by a shell wrapper such as `run_rapid_burst.sh`) against a real GKE cluster running the agent-sandbox controller.
| Recipe | Tuning set(s) | Defaults | What it stresses |
|---|---|---|---|
| `rapid-burst-test.yaml` | `BurstCreate` (QPS 100) | `BURST_SIZE=50`, `TOTAL_BURSTS=100`, `WARMPOOL_SIZE=200` | Repeated bursts of `SandboxClaim` creation against a populated `SandboxWarmPool`, with optional HPA + GKE `CapacityBuffer` scaling |
| `warmpool-burst-test.yaml` | `RampUp` / `RampDown` (100 / 50 QPS) | `REPLICAS=100`, `NAMESPACES=1` | Warm-pool churn under ramp-up/down |
| `high-volume-test.yaml` | `RampUp` / `RampDown` (10 / 50 QPS) | `REPLICAS=100` | Capacity ceiling — "keep increasing this to find the max" |
| `throughput-test.yaml` | `ConstantRate` / `QuickFinalDeletion` (200 / 100 QPS) | `REPLICAS=12000` | Sustained creation throughput |
| `medium-scale-concurrent-load-test.yaml` | `ConstantChurn` / `QuickFinalDeletion` (2 / 100 QPS) | `REPLICAS=1200` | Long-running steady-state churn |
The recipes share a common set of object templates under `test-recipes/templates/`:
| Template | Kind | API |
|---|---|---|
| `cluster-loader-sandbox-template.yaml` | `SandboxTemplate` | `extensions.agents.x-k8s.io/v1beta1` |
| `cluster-loader-warmpool.yaml` | `SandboxWarmPool` | `extensions.agents.x-k8s.io/v1beta1` |
| `cluster-loader-sandbox-claim.yaml` | `SandboxClaim` | `extensions.agents.x-k8s.io/v1beta1` |
| `cluster-loader-hpa.yaml` | `HorizontalPodAutoscaler` (External metric, targets `SandboxWarmPool`) | `autoscaling/v2` |
| `cluster-loader-capacity-buffer.yaml` | `CapacityBuffer` | `autoscaling.x-k8s.io/v1beta1` |
A single `ServiceMonitor` under `test-recipes/monitor/` scrapes the controller in `agent-sandbox-system` every 10s on the `metrics` port; it is loaded into CL2 via `--prometheus-additional-monitors-path`.
Sources: [dev/load-test/test-recipes/rapid-burst-test.yaml:1-46](), [dev/load-test/test-recipes/high-volume-test.yaml:1-15](), [dev/load-test/test-recipes/medium-scale-concurrent-load-test.yaml:1-15](), [dev/load-test/test-recipes/throughput-test.yaml:1-17](), [dev/load-test/test-recipes/warmpool-burst-test.yaml:1-16](), [dev/load-test/test-recipes/templates/cluster-loader-sandbox-template.yaml:1-20](), [dev/load-test/test-recipes/templates/cluster-loader-warmpool.yaml:1-10](), [dev/load-test/test-recipes/templates/cluster-loader-sandbox-claim.yaml:1-8](), [dev/load-test/test-recipes/templates/cluster-loader-hpa.yaml:1-24](), [dev/load-test/test-recipes/templates/cluster-loader-capacity-buffer.yaml:1-12](), [dev/load-test/test-recipes/monitor/agent-sandbox-controller-monitor.yaml:1-19]()
### Anatomy of the rapid-burst recipe
`rapid-burst-test.yaml` is the most complete recipe and the one most worth understanding, because it exercises every template and uses both `PodStartupLatency` and `GenericPrometheusQuery` measurements.
```mermaid
stateDiagram-v2
[*] --> StartMeasurements
StartMeasurements --> SetupTemplate: Setup Sandbox Template
SetupTemplate --> SetupWarmPool: Setup Sandbox Warm Pool
SetupWarmPool --> WaitWarmReady: Wait for Warm Pool Sandboxes\n(WaitForGenericK8sObjects,\nlabel: agents.x-k8s.io/warm-pool-sandbox)
WaitWarmReady --> SetupHPA: if CL2_ENABLE_HPA=true
SetupHPA --> SetupCapacityBuffer
WaitWarmReady --> SetupCapacityBuffer: if CL2_ENABLE_HPA=false
SetupCapacityBuffer --> PauseForNodes: if CL2_ENABLE_CAPACITY_BUFFER=true\nSleep CL2_CAPACITY_BUFFER_PAUSE
SetupCapacityBuffer --> BurstLoop: if CL2_ENABLE_CAPACITY_BUFFER=false
PauseForNodes --> BurstLoop
BurstLoop --> BurstLoop: range 1..CL2_TOTAL_BURSTS\nCreate, WaitFor (Ready=True), Sleep 20s
BurstLoop --> PauseScrape: Pause 1m for Prometheus scrape
PauseScrape --> Gather: Gather Results\n(PodStartupLatency +\nGenericPrometheusQuery x2)
Gather --> TeardownClaims
TeardownClaims --> TeardownHPA: if enabled
TeardownClaims --> TeardownBuffer: if enabled
TeardownHPA --> TeardownBuffer
TeardownBuffer --> TeardownWarmPool
TeardownClaims --> TeardownWarmPool: otherwise
TeardownWarmPool --> TeardownTemplate
TeardownTemplate --> [*]
```
Two things are notable.
First, the recipe couples directly to the controller's CRDs through `WaitForGenericK8sObjects`. It waits on `agents.x-k8s.io/v1beta1` `sandboxes` (label `agents.x-k8s.io/warm-pool-sandbox`) for warm-pool readiness, and on `extensions.agents.x-k8s.io/v1beta1` `sandboxclaims` for burst readiness — both with `successfulConditions: ["Ready=True"]` and `maxFailedObjectCount: 0`:
```yaml
# dev/load-test/test-recipes/rapid-burst-test.yaml
- Identifier: WaitForBurst{{$burstIteration}}SandboxClaims
Method: WaitForGenericK8sObjects
Params:
objectGroup: extensions.agents.x-k8s.io
objectVersion: v1beta1
objectResource: sandboxclaims
successfulConditions: ["Ready=True"]
failedConditions: []
minDesiredObjectCount: {{MultiplyInt $replicaCount $namespaces}}
maxFailedObjectCount: 0
timeout: 60m
refreshInterval: 20ms
```
Second, the recipe uses three Prometheus histograms exposed by the controller — `agent_sandbox_claim_startup_latency_ms_bucket` and `agent_sandbox_claim_controller_startup_latency_ms_bucket` — for `p50`, `p90`, `p99` violations, with thresholds (1s, 1s, 5s) declared inline. This is the pattern the entrypoint recipe's TODO is moving toward.
The recipe is driven by `run_rapid_burst.sh`. The shell wrapper hard-codes the GKE flow (`--provider=gke`, `$HOME/perf-tests`, `$HOME/agent-sandbox`), enables the CL2 Prometheus server, plugs in the `ServiceMonitor`, and writes a JSON test-overrides file under `dev/load-test/test-recipes/tmp//`:
```bash
# dev/load-test/test-recipes/run_rapid_burst.sh
cd "$CL2_DIR"
go run cmd/clusterloader.go \
--enable-prometheus-server=true \
--kubeconfig=$HOME/.kube/config \
--prometheus-additional-monitors-path="${TEST_DIR}/monitor" \
--provider=gke \
--report-dir="${LOGS_DIR}" \
--testconfig="${TEST_CONFIG}" \
--testoverrides="${LOGS_DIR}/testoverrides.json" \
--v=2
```
The wrapper also performs pre-flight validation — it pings the cluster, refuses to enable `CapacityBuffer` if `autoscaling.x-k8s.io` is missing from `kubectl api-resources`, and rejects `ENABLE_HPA=true` if `WARMPOOL_SIZE` is outside `[HPA_MIN_REPLICAS, HPA_MAX_REPLICAS]`. The README documents that `ENABLE_CAPACITY_BUFFER=true` triggers a 5-minute sleep after the buffer is created so GKE node auto-provisioning has time to spin up standby nodes before the burst loop starts.
Sources: [dev/load-test/test-recipes/rapid-burst-test.yaml:44-238](), [dev/load-test/test-recipes/run_rapid_burst.sh:20-113](), [dev/load-test/test-recipes/README-rapid-burst.md:66-126]()
## Test-time Object Graph
The five recipe templates and the controller's CRDs form a tight graph at test time:
```mermaid
classDiagram
class SandboxTemplate {
+metadata.name
+spec.podTemplate
}
class SandboxWarmPool {
+spec.replicas
+spec.sandboxTemplateRef
}
class SandboxClaim {
+spec.sandboxTemplateRef
+status.conditions[Ready]
}
class Sandbox {
+metadata.labels[agents.x-k8s.io/warm-pool-sandbox]
+status.conditions[Ready]
}
class HorizontalPodAutoscaler {
+spec.scaleTargetRef = SandboxWarmPool
+spec.metrics[External]
}
class CapacityBuffer {
+spec.percentage
+spec.scalableRef = SandboxWarmPool
+spec.provisioningStrategy
}
SandboxWarmPool --> SandboxTemplate : sandboxTemplateRef
SandboxClaim --> SandboxTemplate : sandboxTemplateRef
SandboxWarmPool ..> Sandbox : pre-creates pool members
SandboxClaim ..> Sandbox : binds a warm Sandbox
HorizontalPodAutoscaler --> SandboxWarmPool : scaleTargetRef
CapacityBuffer --> SandboxWarmPool : scalableRef
```
The HPA template uses an `External` metric with selectors on `metric.labels.warmpool_name` and `metric.labels.exported_namespace`, allowing the same HPA shape to be reused across namespaces created by CL2.
Sources: [dev/load-test/test-recipes/templates/cluster-loader-sandbox-template.yaml:1-20](), [dev/load-test/test-recipes/templates/cluster-loader-warmpool.yaml:1-10](), [dev/load-test/test-recipes/templates/cluster-loader-sandbox-claim.yaml:1-8](), [dev/load-test/test-recipes/templates/cluster-loader-hpa.yaml:1-24](), [dev/load-test/test-recipes/templates/cluster-loader-capacity-buffer.yaml:1-12](), [dev/load-test/test-recipes/rapid-burst-test.yaml:117-132]()
## End-to-End CI Flow for the Periodic
Putting the two layers together, the periodic load-test job follows this sequence inside a Prow pod:
```mermaid
sequenceDiagram
participant Prow
participant Periodic as periodics/test-load-test
participant Runner as TestRunner (shared/runner.py)
participant DevTools as dev/tools/*
participant KinD
participant CL2 as clusterloader2 (cloned & built)
participant Recipe as agent-sandbox-load-test.yaml
participant Out as $ARTIFACTS
Prow->>Periodic: exec entrypoint
Periodic->>Runner: LoadTestRunner.main()
Runner->>DevTools: create-kind-cluster --recreate
DevTools->>KinD: bootstrap cluster
Runner->>DevTools: push-images --controller-only
Runner->>DevTools: deploy-to-kube --extensions
Runner->>DevTools: deploy-cloud-provider
Periodic->>CL2: git clone perf-tests + go build
Periodic->>CL2: clusterloader2 --testconfig=agent-sandbox-load-test.yaml --testoverrides=...
CL2->>Recipe: execute steps 1..5
Recipe->>KinD: create Sandboxes, WaitForRunningPods
CL2-->>Periodic: bin/junit.xml
Periodic->>Out: copy junit.xml -> junit_load_test.xml
```
`WaitForGenericK8sObjects` is what couples the rapid-burst recipe to the controller's CRDs, while the entrypoint recipe waits on raw `Pod` readiness via `WaitForRunningPods`. The shape of the JUnit output (one testcase per CL2 step) is illustrated in the load-test README:
```xml
```
Sources: [dev/load-test/README.md:73-86](), [dev/load-test/test-recipes/rapid-burst-test.yaml:189-204](), [dev/load-test/agent-sandbox-load-test.yaml:34-48](), [dev/ci/periodics/test-load-test:80-116]()
## Running a Prow Job Locally
`docs/prowjob_manual_run.md` documents how an engineer reproduces the periodic outside of CI. The job runs as KinD-in-Docker (nested virtualisation), so the doc raises host inotify limits before invocation:
```bash
sudo sysctl -w fs.inotify.max_user_watches=524288
sudo sysctl -w fs.inotify.max_user_instances=512
```
From a checkout of `kubernetes/test-infra`:
```bash
./config/pj-on-kind.sh periodic-agent-sandbox-perf-load-test
kubectl get pods
kubectl logs -f -c test
# After completion, artifacts live under the the script prints at start
cd /periodic-agent-sandbox-perf-load-test/
cat finished.json
```
The job referenced is the same `LoadTestRunner` entrypoint; its Prow spec is maintained in `config/jobs/kubernetes-sigs/agent-sandbox/agent-sandbox-periodics-main.yaml` in the `kubernetes/test-infra` repository, not in this one. The doc also notes minimum host resources of 8 CPU and 16 GB RAM for the nested cluster to come up cleanly.
Sources: [docs/prowjob_manual_run.md:1-84]()
## Extending the Pipelines
Three patterns recur when adding new coverage:
1. **A new presubmit shell wrapper** — copy `dev/ci/presubmits/lint-go` and change `run_dev_tool("lint-go")` to a target script in `dev/tools/`. The `shared/utils.py` helper handles repo-root resolution.
2. **A new cluster-backed test** — subclass `TestRunner` from `dev/ci/shared/runner.py`, override `run_tests` to invoke a binary, and override `copy_artifacts` to deposit a `junit_*.xml` into `$ARTIFACTS`. `setup_cluster` is inherited and handles KinD bring-up, image push, and CRD install; pass `extra_push_images_args=["--controller-only"]` when you only need the controller image.
3. **A new load-test recipe** — add a YAML under `dev/load-test/test-recipes/` (parameterised with `DefaultParam .CL2_*`), reuse the existing template manifests for `SandboxTemplate` / `SandboxWarmPool` / `SandboxClaim` / `HorizontalPodAutoscaler` / `CapacityBuffer`, and drive it from a local shell wrapper or a new CL2 invocation block in `LoadTestRunner.run_tests`. Prefer `GenericPrometheusQuery` over `PodStartupLatency` whenever the controller already exports a relevant histogram (e.g., `agent_sandbox_claim_startup_latency_ms_bucket`, used by the rapid-burst recipe).
Together these two surfaces give the project a stable signal (the small periodic in KinD) and a manual proving ground (the GKE-targeted recipes) without duplicating the wiring that brings the controller online.
Sources: [dev/ci/shared/runner.py:23-80](), [dev/ci/periodics/test-load-test:48-110](), [dev/load-test/test-recipes/rapid-burst-test.yaml:1-92](), [dev/load-test/test-recipes/rapid-burst-test.yaml:44-92]()
---
## 30. KEPs & Roadmap
> In-flight design proposals tracked under docs/keps (suspended state, metadata propagation, Python SDK refactor) and the published roadmap.
- Page Markdown: https://grok-wiki.com/public/wiki/kubernetes-sigs-agent-sandbox-c3f2597a654a/pages/30-keps-roadmap.md
- Generated: 2026-05-25T22:53:57.404Z
### Source Files
- `roadmap.md`
- `docs/keps/README.md`
- `docs/keps/119-sandbox-suspended-state/README.md`
- `docs/keps/174-metadata-propagation/README.md`
- `docs/keps/359-refactor-python-sdk/README.md`
Relevant source files
The following files were used as context for generating this wiki page:
- [roadmap.md](roadmap.md)
- [docs/keps/README.md](docs/keps/README.md)
- [docs/keps/NNNN-template/README.md](docs/keps/NNNN-template/README.md)
- [docs/keps/NNNN-template/kep.yaml](docs/keps/NNNN-template/kep.yaml)
- [docs/keps/119-sandbox-suspended-state/README.md](docs/keps/119-sandbox-suspended-state/README.md)
- [docs/keps/119-sandbox-suspended-state/kep.yaml](docs/keps/119-sandbox-suspended-state/kep.yaml)
- [docs/keps/174-metadata-propagation/README.md](docs/keps/174-metadata-propagation/README.md)
- [docs/keps/174-metadata-propagation/kep.yaml](docs/keps/174-metadata-propagation/kep.yaml)
- [docs/keps/359-refactor-python-sdk/README.md](docs/keps/359-refactor-python-sdk/README.md)
- [docs/keps/359-refactor-python-sdk/kep.yaml](docs/keps/359-refactor-python-sdk/kep.yaml)
# KEPs & Roadmap
This page documents the forward-looking design surface of `kubernetes-sigs/agent-sandbox`: the Agent Sandbox Enhancement Proposal (KEP) process, the three in-flight KEPs currently checked into `docs/keps/`, and the published `roadmap.md`. KEPs are how non-trivial changes are proposed, debated, and coordinated; the roadmap states the high-level strategic priorities for the calendar year. Together they describe what shape the project is likely to take next, independent of what is already implemented.
The body below pulls each proposal apart into its motivation, the API or behavior it changes, and what state of maturity it is in (per the `status` field in each `kep.yaml`). The roadmap section maps the roadmap bullets back to the KEPs and tracking issues they correspond to, so a reader can connect a strategic priority to the concrete design document that backs it.
## The KEP Process
The KEP process is modeled directly on the upstream [Kubernetes Enhancement Proposal](https://github.com/kubernetes/enhancements/tree/master/keps) workflow. Writing one is optional and reserved for "non-trivial changes" — controversial proposals, new features, major changes to existing features, and anything with wide project impact. Lightweight changes go straight to PRs. Discussion happens on the `#sig-apps` Kubernetes Slack channel and the SIG-Apps mailing list before a KEP is drafted.
Each KEP lives in a folder under `docs/keps/-/` and consists of two files:
| File | Purpose |
| :--- | :--- |
| `README.md` | The proposal itself: summary, motivation, proposal, API changes, alternatives. Sections are generated from `docs/keps/NNNN-template/README.md`. |
| `kep.yaml` | Structured metadata — title, `kep-number`, authors, reviewers, approvers, `status`, `creation-date`, and optionally `stage` and `last-updated`. |
The `NNN` prefix is the tracking issue number on the `agent-sandbox` GitHub repo, which gives every proposal a stable identifier and a single place to discuss live state. The template embeds an auto-generated `` block maintained by `make toc-update`.
Sources: [docs/keps/README.md:1-30](), [docs/keps/NNNN-template/README.md:1-94](), [docs/keps/NNNN-template/kep.yaml:1-18]()
### KEP Lifecycle and Status Vocabulary
`status` in `kep.yaml` is the canonical truth for where a proposal sits. The template offers `implementable|implemented` as the example vocabulary; existing KEPs use `provisional` and `implementable`. The optional `stage` field (e.g., `alpha`) tracks rollout maturity once an implementation lands.
```text
draft ─► provisional ─► implementable ─► implemented
│
└─► stage: alpha → beta → ga
```
Sources: [docs/keps/NNNN-template/kep.yaml:5](), [docs/keps/119-sandbox-suspended-state/kep.yaml:1-10](), [docs/keps/174-metadata-propagation/kep.yaml:1-9](), [docs/keps/359-refactor-python-sdk/kep.yaml:1-10]()
## In-Flight KEPs
Three KEPs are currently checked in. The table below is a quick index; full detail follows in dedicated sections.
| KEP | Title | Status | Stage | Authors | Created |
| :--- | :--- | :--- | :--- | :--- | :--- |
| 119 | Agent Sandbox Suspended Condition Status | `implementable` | — | @SHRUTI6991 | 2026-03-16 |
| 174 | Label and Annotation Propagation to Sandbox Pods | `provisional` | `alpha` | @chenyiwang | 2026-03-18 |
| 359 | Python SDK Refactor From Context Manager to Resource Handle | `implementable` | — | @SHRUTI6991 | 2026-03-03 |
Note that the `kep-number` field inside `docs/keps/359-refactor-python-sdk/kep.yaml` is currently `174` — the folder name (`359-...`) reflects the tracking issue, while the YAML field has not been updated to match.
Sources: [docs/keps/119-sandbox-suspended-state/kep.yaml:1-10](), [docs/keps/174-metadata-propagation/kep.yaml:1-9](), [docs/keps/359-refactor-python-sdk/kep.yaml:1-10]()
### KEP-119: Sandbox `Suspended` Condition
KEP-119 adds an explicit `Suspended` status condition to the `Sandbox` CRD so that observers can distinguish "scaling down" from "scaled down" without inspecting child Pods, PVCs, or Services. Today there is only the aggregate `Ready` condition; once a suspension is requested, that single signal cannot say whether the termination is in progress or already complete, and it cannot model future "soft pause" modes such as freezing container processes or hibernating to disk.
The proposal defines two conditions and a Reason-driven hierarchy that the controller evaluates top-down:
| Scenario | `Suspended` | Suspended Reason | Pod Phase | `Ready` (Root) | Ready Reason |
| :--- | :--- | :--- | :--- | :--- | :--- |
| Provisioning | None | None | None | `False` | `DependenciesNotReady` |
| Pod Starting | None | None | Pending | `False` | `DependenciesNotReady` |
| Operational | None | None | Running & Ready | `True` | `DependenciesReady` |
| Suspending | `False` | `PodNotTerminated` | Running / Terminating | `False` | `SandboxSuspended` |
| Suspended | `True` | `PodTerminated` | None | `False` | `SandboxSuspended` |
The moment a suspension is requested (e.g., `replicas: 0`), `Ready` immediately flips to `False` with reason `SandboxSuspended`, and the `Suspended` condition tracks the actual progress. This lets standard tooling drive automation directly:
```bash
kubectl wait --for=condition=Ready sandbox/my-env
kubectl wait --for=condition=Suspended=True sandbox/my-env
```
The KEP rejects two alternatives: retaining the legacy `status.phase` string (deprecated by Kubernetes API conventions and prone to combinatorial explosion as new suspension modes are added) and overloading `Ready` with reason codes (no headroom to distinguish "freeze" vs "hibernate" vs "scale-to-zero" in the future).
```mermaid
stateDiagram-v2
[*] --> Provisioning
Provisioning --> PodStarting: dependencies created
PodStarting --> Operational: Pod Ready
Operational --> Suspending: replicas=0 / suspend request
Suspending --> Suspended: Pod terminated
Suspended --> Provisioning: scale back up
Provisioning: Ready=False
DependenciesNotReady
PodStarting: Ready=False
DependenciesNotReady
Operational: Ready=True
DependenciesReady
Suspending: Ready=False / SandboxSuspended
Suspended=False / PodNotTerminated
Suspended: Ready=False / SandboxSuspended
Suspended=True / PodTerminated
```
Sources: [docs/keps/119-sandbox-suspended-state/README.md:1-82]()
### KEP-174: Metadata Propagation to Sandbox Pods
KEP-174 standardizes how labels and annotations flow from a top-level `SandboxClaim` through the `Sandbox` and finally onto the backing `Pod`. The motivation is twofold: enabling per-claim "personalization" (cost attribution, observability, stateful session identifiers) on otherwise homogeneous warm-pool pods, and avoiding "template explosion" caused by creating a new `SandboxTemplate` for every unique session.
The design adds a single new field, `additionalPodMetadata`, on `SandboxClaim`. `SandboxTemplate` and `SandboxWarmPool` are left untouched so that pool resources remain interchangeable.
```go
// sandbox_types.go (existing)
type PodMetadata struct {
Labels map[string]string `json:"labels,omitempty" protobuf:"bytes,1,rep,name=labels"`
Annotations map[string]string `json:"annotations,omitempty" protobuf:"bytes,2,rep,name=annotations"`
}
// sandboxclaim_types.go (new)
type SandboxClaimSpec struct {
// ...
AdditionalPodMetadata sandboxv1alpha1.PodMetadata `json:"additionalPodMetadata,omitempty"`
}
```
The data flow and safety contract:
```text
SandboxClaim.spec.additionalPodMetadata
│ (SandboxClaim controller)
▼
Sandbox.spec.podTemplate.metadata ◄── merged with SandboxTemplate
│ (Sandbox controller)
▼
Pod.metadata.{labels,annotations}
│
├── agents.x-k8s.io/propagated-labels = "k1,k2,..."
└── agents.x-k8s.io/propagated-annotations = "a1,a2,..."
```
Two annotations on the resulting Pod — `agents.x-k8s.io/propagated-labels` and `agents.x-k8s.io/propagated-annotations` — record which keys the controller put there, so it can later prune removed keys without disturbing labels added by mutating webhooks or other actors. The controller refuses requests where a key collides between the template and the claim with conflicting values ("Safety Principle: No Overrides").
Two propagation scenarios are spelled out:
| Scenario | Trigger | Controller behavior |
| :--- | :--- | :--- |
| Cold Start (new Pod) | First creation, no warmpool | Sandbox controller merges template + claim metadata before Pod creation. |
| Cold Start (Pod exists) | `SandboxClaim` metadata updated after creation | In-place patch of Pod `metadata`, no restart. |
| Warmpool adoption | Claim binds to a pre-warmed Sandbox | `SandboxClaim` controller patches the Sandbox's `spec.podTemplate.metadata` for sub-millisecond dispatch; no resource re-creation. |
| Warmpool post-adoption update | Claim metadata changes later | `SandboxClaim` controller re-syncs the metadata into the Sandbox. |
The scope is intentionally narrow: only label/annotation propagation that does not require a Pod restart. Anything that imposes functional control needing a restart is deferred to [Issue 208](https://github.com/kubernetes-sigs/agent-sandbox/issues/208). For metadata that must apply uniformly across an entire pool, users still modify `SandboxTemplate` directly per [Issue 347](https://github.com/kubernetes-sigs/agent-sandbox/issues/347).
Sources: [docs/keps/174-metadata-propagation/README.md:1-108]()
### KEP-359: Python SDK Refactor — Context Manager → Resource Handle
KEP-359 reshapes the Python client from a transient `with Sandbox() as sbx:` context manager into a persistent **Resource Handle** keyed by `sandbox_id`. The context-manager pattern fits short scripts but is a poor match for long-lived agent workflows: an agent may suspend a sandbox for hours, hand ownership across processes, juggle multiple sandboxes concurrently, or reattach to one created by a different component. None of those are expressible inside a single `with` block.
The new SDK is layered into three tiers:
```mermaid
classDiagram
class SandboxClient {
+router_dns: str
+create_sandbox(template, namespace) Sandbox
+get_sandbox(sandbox_id) Sandbox
}
class Sandbox {
+id: str
+commands: CoreExecution
+files: Filesystem
+status()
+suspend()
+resume()
+terminate()
}
class CoreExecution {
+run_code()
+run_cmd()
}
class Filesystem {
+read()
+write()
+list()
}
class ProcessSystem {
+create_process()
+kill_process()
}
SandboxClient ..> Sandbox : factory / reattach
Sandbox *-- CoreExecution
Sandbox *-- Filesystem
Sandbox ..> ProcessSystem : sbx.process
```
`SandboxClient` is the entry point and factory; it owns global configuration such as `router_dns`. `Sandbox` is a stable handle around a `sandbox_id` and exposes lifecycle verbs (`status`, `suspend`, `resume`, `terminate`) plus dot-namespaced engines (`sbx.files`, `sbx.commands`, `sbx.process`). Engines talk to the Sandbox Router over a stable DNS name and pass an `X-Sandbox-Id` header for session affinity.
The motivating properties the design calls out:
1. **Identity stability** — the same `Sandbox` object remains valid whether the underlying Pod is running, suspended, or resuming.
2. **Orchestration vs. execution split** — management calls go through the control plane; execution calls go through the router.
3. **Capability discovery** — new functionality slots into a new engine namespace rather than the root object.
4. **Distributed ownership** — a `sandbox_id` can be handed to another process which calls `get_sandbox(id)` and continues from there.
5. **Non-linear logic** — multiple long-lived sandboxes coexist as plain objects, no nested `with` blocks.
Worked example from the KEP:
```python
client = SandboxClient(router_dns="router.sandbox.svc")
sbx = client.create_sandbox(template="python-ml")
sbx.files.write("data.py", "x = 42")
sbx.core.run_code("import data; print(data.x)")
sbx.suspend()
# Re-attach later, possibly in a different process
old_sbx = client.get_sandbox("sbx_123")
old_sbx.resume()
```
A proof-of-concept implementation lives at [PR #365](https://github.com/kubernetes-sigs/agent-sandbox/pull/365). The PoC initiates the router session at `Sandbox` construction time and updates the test client to use the handle-based flow. The "Scalability" section notes a future hook for a pluggable Sandbox Manager.
Sources: [docs/keps/359-refactor-python-sdk/README.md:1-129]()
## Roadmap
`roadmap.md` lists the project's "main strategic priorities for 2026." The bullets span documentation, distribution, SDK functionality, API shape, observability, isolation technology, and ecosystem integrations. Many bullets carry a GitHub issue number that links the strategic intent to the live tracking issue.
### Roadmap → KEP / Issue Map
The roadmap items split into ones that already have a KEP in flight versus ones that are tracked only by an issue and a sentence of intent.
| Roadmap theme | Tracking issue | KEP |
| :--- | :--- | :--- |
| Status Updates | [#119](https://github.com/kubernetes-sigs/agent-sandbox/issues/119) | KEP-119 (Suspended condition) |
| Metadata Propagation | [#174](https://github.com/kubernetes-sigs/agent-sandbox/issues/174) | KEP-174 |
| Expand SDK functionality (`read`, `write`, `run_code`, …) | — | KEP-359 (resource handle + engines) |
| Website Refresh | [#166](https://github.com/kubernetes-sigs/agent-sandbox/issues/166) | — |
| PyPI Distribution of `agent-sandbox-client` | [#146](https://github.com/kubernetes-sigs/agent-sandbox/issues/146) | — |
| Strict Sandbox-to-Pod Mapping | [#127](https://github.com/kubernetes-sigs/agent-sandbox/issues/127) | — |
| Go Client | [#227](https://github.com/kubernetes-sigs/agent-sandbox/issues/227) | — |
| Startup Actions | [#58](https://github.com/kubernetes-sigs/agent-sandbox/issues/58) | — |
| Creation Latency Metrics | [#123](https://github.com/kubernetes-sigs/agent-sandbox/issues/123) | — |
| Headless Service Port Handling | [#154](https://github.com/kubernetes-sigs/agent-sandbox/issues/154) | — |
| OpenEnv Support | [#132](https://github.com/kubernetes-sigs/agent-sandbox/issues/132) | — |
### Themes Without Linked Issues
Several priorities currently exist only as roadmap prose:
- **Documentation overhaul** — restructure and expand current documentation to lower the barrier to entry.
- **Benchmarking Guide** — published guidance for measuring Agent Sandbox performance.
- **Expand Sandbox use cases** — Computer Use, browser use, additional base images.
- **Decouple API from Runtime** — let users fully customize the runtime without breaking the API.
- **Scale-down / Resume PVC based** — preserve PVC when replicas drop to 0; restore on scale-up.
- **Complete CR, SDK, and template support** — feature-parity across surfaces.
- **API Support for Multi-Sandbox per Pod** — multiple sandboxes inside a single Pod.
- **Auto-deletion of bursty sandboxes** — typical RL training pattern.
- **Runtime API OTEL/Tracing instrumentation** — instrument and document.
- **Detailed Falco config extension** — surface gVisor logging configuration through the Agent Sandbox API.
- **Other isolation technologies** — QEMU, Firecracker, process isolation (pydantic).
- **Agent & RL framework integration** — CrewAI, Ray RLlib.
- **Integration with kAgent** and other sandbox offerings.
- **Deliver Beta/GA versions.**
Sources: [roadmap.md:1-29]()
## Reading and Contributing to a KEP
A practical workflow for a contributor proposing a non-trivial change:
1. Socialize the idea on `#sig-apps` Slack or the SIG-Apps mailing list.
2. Open a tracking issue on `kubernetes-sigs/agent-sandbox`; the issue number becomes the KEP's `NNN` prefix and folder name.
3. Copy `docs/keps/NNNN-template/` to `docs/keps/-/` and fill in the `README.md` sections (Summary, Motivation, Proposal, API Changes, Implementation Guidance, Scalability, Alternatives).
4. Populate `kep.yaml` — `title`, `kep-number`, `authors`, `reviewers`, `approvers`, `status`, `creation-date`, optionally `stage`, `last-updated`, `see-also`, and `replaces`.
5. Regenerate the table of contents with `make toc-update`.
6. Land the KEP through normal review; update `status` (and `stage` once implementation begins) as the proposal moves through `provisional` → `implementable` → `implemented`.
Sources: [docs/keps/README.md:1-30](), [docs/keps/NNNN-template/README.md:1-94](), [docs/keps/NNNN-template/kep.yaml:1-18]()
## Summary
The `docs/keps/` tree is the project's design archive: three live proposals (119 surfacing a `Suspended` status condition on the `Sandbox` CRD, 174 wiring `SandboxClaim` labels and annotations down to the backing Pod without breaking warm-pool homogeneity, and 359 rebuilding the Python SDK around a persistent `Sandbox` handle with namespaced engines) plus a template and process document. `roadmap.md` sits alongside as the 2026 strategic agenda; many of its bullets reference the same GitHub issues used to number the KEPs, making the roadmap → KEP → tracking-issue chain the natural path for anyone trying to understand where Agent Sandbox is heading next.
---