# E2E Test Framework > Layout of the Go e2e suite, the framework client/predicates/watchset helpers, and the parallel/replica/shutdown scenario coverage. - Repository: kubernetes-sigs/agent-sandbox - GitHub: https://github.com/kubernetes-sigs/agent-sandbox - Human wiki: https://grok-wiki.com/public/wiki/kubernetes-sigs-agent-sandbox-c3f2597a654a - Complete Markdown: https://grok-wiki.com/public/wiki/kubernetes-sigs-agent-sandbox-c3f2597a654a/llms-full.txt ## Source Files - `test/e2e/README.md` - `test/e2e/framework/client.go` - `test/e2e/framework/watchset.go` - `test/e2e/basic_test.go` - `test/e2e/parallelism_test.go` - `test/e2e/extensions/warmpool_rollout_test.go` - `docs/testing.md` ---
Relevant source files The following files were used as context for generating this wiki page: - [test/e2e/README.md](test/e2e/README.md) - [docs/testing.md](docs/testing.md) - [test/e2e/framework/client.go](test/e2e/framework/client.go) - [test/e2e/framework/watchset.go](test/e2e/framework/watchset.go) - [test/e2e/framework/context.go](test/e2e/framework/context.go) - [test/e2e/framework/testlogs.go](test/e2e/framework/testlogs.go) - [test/e2e/framework/componentlogs.go](test/e2e/framework/componentlogs.go) - [test/e2e/framework/predicates/predicates.go](test/e2e/framework/predicates/predicates.go) - [test/e2e/framework/predicates/conditions.go](test/e2e/framework/predicates/conditions.go) - [test/e2e/framework/predicates/metadata.go](test/e2e/framework/predicates/metadata.go) - [test/e2e/framework/predicates/sandbox.go](test/e2e/framework/predicates/sandbox.go) - [test/e2e/basic_test.go](test/e2e/basic_test.go) - [test/e2e/replicas_test.go](test/e2e/replicas_test.go) - [test/e2e/shutdown_test.go](test/e2e/shutdown_test.go) - [test/e2e/parallelism_test.go](test/e2e/parallelism_test.go) - [test/e2e/extensions/warmpool_rollout_test.go](test/e2e/extensions/warmpool_rollout_test.go) - [test/e2e/utils.go](test/e2e/utils.go) - [Makefile](Makefile)
# E2E Test Framework The agent-sandbox end-to-end suite is a Go `testing` package that runs against a real Kubernetes cluster (typically a kind cluster created by `make deploy-kind`), driving the `Sandbox`, `SandboxClaim`, `SandboxTemplate`, and `SandboxWarmPool` CRDs through the agent-sandbox controller. It is structured around a thin `framework` package that owns cluster credentials, object lifecycle, predicate evaluation, and a shared watch fan-out, and a set of `*_test.go` scenarios that exercise basic reconciliation, replica scaling, shutdown lifecycle, parallel reconciliation under load, and warm-pool rollout strategies. This page maps the framework's public surface, the wiring between `TestContext`, `ClusterClient`, predicates, and `WatchSet`, and then walks through the parallel, replica, shutdown, and warm-pool scenarios to show how the helpers are composed. ## How tests are launched Tests assume that a cluster is reachable through `bin/KUBECONFIG` (or `$KUBECONFIG`) and that the agent-sandbox controller is already installed. The README documents the canonical invocation: ```shell make deploy-kind go test ./test/e2e/... --parallel=1 ``` `--parallel=1` serializes the top-level tests because they share a single cluster and patch the controller `Deployment` in some scenarios. Sources: [test/e2e/README.md:1-30](), [Makefile:33-68](), [test/e2e/framework/context.go:40-67]() The `Makefile` exposes higher-level entry points used by CI: | Target | Behavior | | --- | --- | | `make test-e2e` | Runs `./dev/ci/presubmits/test-e2e` (race detector toggled via `RACE`). | | `make test-e2e-race` | Same script with `RACE=1`. | | `make test-e2e-benchmarks` | Same script with `--suite benchmarks`. | | `make deploy-kind` / `make delete-kind` | Provision/teardown the kind cluster used by the suite. | Sources: [Makefile:33-68](), [docs/testing.md:12-35]() ## Repository layout ```text test/e2e/ ├── README.md # run instructions ├── basic_test.go # TestSimpleSandbox ├── replicas_test.go # TestSandboxReplicas ├── shutdown_test.go # TestSandboxShutdownTime, retain-expiry ├── parallelism_test.go # parallel Sandboxes / SandboxClaims ├── volumeclaimtemplate_test.go ├── chromesandbox_test.go, chromesandbox_claim_test.go ├── utils.go # AtomicTimeDuration helper ├── framework/ # shared client + watch + predicates │ ├── client.go # ClusterClient (Get/Create/WaitForObject/Watch) │ ├── context.go # TestContext, kubeconfig, before/afterEach │ ├── watchset.go # WatchSet/ResourceWatch/Subscription fan-out │ ├── testlogs.go # per-test artifacts log file │ ├── componentlogs.go # kubelet/containerd log capture from kind nodes │ └── predicates/ # ObjectPredicate library └── extensions/ # CRD-specific scenarios ├── warmpool_rollout_test.go ├── warmpool_sandbox_watcher_test.go ├── pythonruntime_test.go ├── sandboxclaim_metric_test.go └── shutdown_policy_test.go ``` Sources: [test/e2e/framework/client.go:15-45](), [test/e2e/framework/context.go:14-37](), [test/e2e/extensions/warmpool_rollout_test.go:15-34]() ## Framework architecture The framework wraps `*testing.T` (or `*testing.B`) into a `TestContext` that exposes a `ClusterClient` and a per-test `WatchSet`. The `ClusterClient` delegates CRUD to a controller-runtime `client.Client`, while waits and watches go through a `dynamic.Interface` so the framework can drive arbitrary GVRs without registering Go types ahead of time. ```mermaid flowchart LR subgraph TestCase["*_test.go scenario"] TC["framework.NewTestContext(t)"] end subgraph Framework["test/e2e/framework"] Ctx["TestContext\n(context.go)"] CC["ClusterClient\n(client.go)"] WS["WatchSet\n(watchset.go)"] Logs["logCapturingT\n(testlogs.go)"] Comp["dumpControllerLogs\nMustGetKubeletLogs"] end subgraph Predicates["framework/predicates"] IF["ObjectPredicate interface"] Status["StatusPredicate\nConditionReasonPredicate\nReadyReplicasPredicate"] Meta["Annotation/Label/Owner\nNotDeleted/HasDeletionTimestamp"] Sandbox["SandboxHasStatus\n(go-cmp diff)"] end subgraph K8s["Kubernetes cluster (kind)"] API["kube-apiserver"] Ctrl["agent-sandbox-controller\n(agent-sandbox-system)"] CRDs["Sandbox / SandboxClaim\nSandboxTemplate / SandboxWarmPool"] end TC --> Ctx Ctx --> CC Ctx --> Logs Ctx --> Comp CC --> WS CC -->|"controller-runtime client"| API WS -->|"dynamic.Interface\nWatch loop"| API CC --> Predicates Predicates --> Status Predicates --> Meta Predicates --> Sandbox Ctrl --> CRDs API --> Ctrl ``` Sources: [test/e2e/framework/context.go:91-161](), [test/e2e/framework/client.go:51-78](), [test/e2e/framework/watchset.go:59-148](), [test/e2e/framework/predicates/predicates.go:26-33]() ### `TestContext` lifecycle `NewTestContext` is the single entry point used by every scenario. It loads the kubeconfig, builds both a typed `controller-runtime` client and a `dynamic.Interface`, creates a `WatchSet`, attaches log capture, and registers cleanup hooks. `beforeEach` validates that the CRDs, the `agent-sandbox-system` namespace, and the `agent-sandbox-controller` Deployment exist before the test body runs; `afterEach` dumps controller logs into the test's artifacts directory if the test failed. Key responsibilities, in order: 1. Resolve `KUBECONFIG` (env var, otherwise `/bin/KUBECONFIG`). 2. Build `*rest.Config`, shared HTTP client, controller-runtime client, and dynamic client against the cluster. 3. Create a `WatchSet(dynamicClient)` and register `watchSet.Close()` as test cleanup. 4. Wrap `T` with `logCapturingT` so every `Logf` / `Errorf` is mirrored to `//test.log` with elapsed-time prefixes. 5. Run `validateAgentSandboxInstallation()` — fails fast if the controller or CRDs are missing. 6. Register `afterEach()` to call `dumpControllerLogs` when `t.Failed()`. Sources: [test/e2e/framework/context.go:91-178](), [test/e2e/framework/context.go:180-240](), [test/e2e/framework/client.go:521-549](), [test/e2e/framework/testlogs.go:34-77]() ### `ClusterClient` API `ClusterClient` is the workhorse for all scenarios. Every method is `Helper()`-marked so failures point at the calling test line, and create operations register cleanup that deletes the object and then blocks on `WaitForObjectNotFound` so the next test starts with a clean namespace. | Method | Purpose | | --- | --- | | `Get` / `List` / `Update` / `Delete` | Thin error-wrapping facades over the controller-runtime client. | | `CreateWithCleanup` / `MustCreateWithCleanup` | Create the object and schedule deletion + wait-not-found in `t.Cleanup`. Uses `context.Background()` during cleanup since the test context is already done. | | `MustUpdateObject[T]` | Re-fetch the latest version, apply `updateFunc`, then `Update`. Avoids stale-version conflicts. | | `MatchesPredicates` / `MustMatchPredicates` / `MustExist` | One-shot predicate evaluation against the current API state. | | `ValidateObjectNotFound` / `WaitForObjectNotFound` | Confirm deletion; namespaces get a 3-minute timeout because of cascading cleanup, everything else is 60s. | | `PollUntilObjectMatches` | 1-second polling fallback used when watches aren't a good fit (for example, the shutdown-time scenario). | | `WaitForObject` / `MustWaitForObject` | Watch-driven wait with predicate evaluation; the preferred timing primitive. | | `Watch` / `Watch[T]` / `MustWatch[T]` | Generic event callback against a `GVR + WatchFilter`. | | `WaitForSandboxReady` / `WaitForWarmPoolReady` / `GetSandbox` | Convenience wrappers for the specific CRDs. | | `PortForward` | Spawns `kubectl port-forward` for tests that need direct pod access (e.g., Chrome debug port). | | `ExecuteOnNode` / `IsKindCluster` | `docker exec` into the kind node container; tests can opt into kind-only behavior. | `DefaultTimeout` is 60 seconds; a shorter deadline can be supplied via the caller's context, and `WaitForObject` clamps to whichever is smaller. Sources: [test/e2e/framework/client.go:46-265](), [test/e2e/framework/client.go:485-549](), [test/e2e/framework/client.go:551-718]() ### `WatchForObject` and the typed `Watch[T]` helper `WaitForObject` is the single timing primitive that the suite leans on. It first checks if the object already satisfies the predicates (fast path), looks up the GVR from a hard-coded GVK table (`gvrForGVK` covers Pod, Deployment, Namespace, Sandbox, SandboxWarmPool, SandboxClaim), subscribes through the shared `WatchSet`, and runs the predicate callback on each event. On timeout or non-match it logs the last observed object as YAML, which is invaluable when debugging asymmetric status diffs. ```go // test/e2e/framework/client.go (excerpt) done, err := Watch(ctx, cl, gvr, watchFilter, func(event watch.Event, obj *unstructured.Unstructured) (bool, error) { if event.Type == watch.Deleted { return false, fmt.Errorf("object was deleted while waiting for predicates to be satisfied") } var notMatching []predicates.ObjectPredicate for _, predicate := range p { if match, err := predicate.Matches(obj); err != nil { return false, err } else if !match { notMatching = append(notMatching, predicate) } } lastNotMatching = notMatching lastObject = obj.DeepCopy() return len(notMatching) == 0, nil }) ``` The generic `Watch[T]` decodes events to either `*unstructured.Unstructured` (passthrough) or any concrete `client.Object` by running `DefaultUnstructuredConverter.FromUnstructured`. Bookmark events are dropped, and `Error` events become callback errors. `MustWatch` swallows `context.Canceled` so tests that intentionally cancel the watch context don't fail. Sources: [test/e2e/framework/client.go:267-349](), [test/e2e/framework/client.go:351-428](), [test/e2e/framework/client.go:430-473]() ### `WatchSet`: shared dynamic watches A naïve "one Watch per `WaitForObject` call" would create one HTTP watch per assertion and incur listing + setup latency on every wait. The framework instead keeps **one** persistent watch per `(GVR, namespace)` and fans out events to per-call subscriptions that apply a `WatchFilter{Namespace, Name}`. ```mermaid classDiagram class WatchSet { +watches: map[watchKey]*ResourceWatch +dynamicClient: dynamic.Interface +Subscribe(gvr, filter) *Subscription +Close() -getOrCreateWatch(gvr, namespace) *ResourceWatch -removeWatchIfIdle(rw) } class ResourceWatch { +gvr: GroupVersionResource +namespace: string +subscriptions: map[uint64]*Subscription +cancelWatchLoop: CancelFunc -watchLoop(ctx) -broadcast(event) -subscribe(filter) *Subscription -unsubscribe(sub) } class Subscription { +Events: chan watch.Event +filter: WatchFilter +Close() } class WatchFilter { +Namespace: string +Name: string } WatchSet "1" --> "*" ResourceWatch : owns ResourceWatch "1" --> "*" Subscription : broadcasts to Subscription --> WatchFilter : per-call filter ``` Notable design details: - The watch loop runs under `context.Background()` so it survives the caller's per-call context and only stops when `WatchSet.Close()` runs (registered in `NewTestContext`) or when the last subscription unsubscribes (`removeWatchIfIdle`). - On a `watch.Error` event the resource version is reset to `""` and the loop restarts from scratch; transient errors back off for 100 ms before retry. - `broadcast` always forwards `watch.Error` and `watch.Bookmark` events to all subscribers, but applies name/namespace filtering for `Added/Modified/Deleted`. - Each subscription's `Events` channel is buffered at 100 to absorb bursts without blocking the watch loop. - Cluster-scoped resources use `namespace == ""` and a single cluster-wide watch. Sources: [test/e2e/framework/watchset.go:29-148](), [test/e2e/framework/watchset.go:150-211](), [test/e2e/framework/watchset.go:230-328]() ### Predicates `predicates.ObjectPredicate` is a two-method interface: `Matches(obj) (bool, error)` and `fmt.Stringer`. The stringer requirement means `WaitForObject` can log the unmatched predicates as `lastNotMatching` for fast debugging. | Predicate | File | Behavior | | --- | --- | --- | | `ReadyConditionIsTrue` (`StatusPredicate`) | `conditions.go` | Looks for `status.conditions[Type=Ready, Status=True]` via unstructured conversion. | | `ConditionReasonEquals(type, reason)` | `conditions.go` | Same shape but matches on `Reason`; used by the retain-expiry test. | | `ReadyReplicasConditionIsTrue` | `conditions.go` | `status.readyReplicas == spec.replicas`; used to wait for the controller Deployment after patching. | | `ObservedGenerationMatchesGeneration` | `conditions.go` | `status.observedGeneration == metadata.generation`; pairs with the replicas check on controller restarts. | | `SandboxHasStatus(want)` | `sandbox.go` | Full `go-cmp` diff against `SandboxStatus`, ignoring `LastTransitionTime` and `PodIPs`. | | `HasAnnotation` / `HasLabel` / `HasOwnerReferences` | `metadata.go` | Map and slice comparisons; owner refs are sorted by UID before diffing. | | `NotDeleted` / `HasDeletionTimestamp` | `metadata.go` | DeletionTimestamp checks for cascading-delete and rollout assertions. | All status predicates funnel through `asUnstructured` and `runtime.DefaultUnstructuredConverter.FromUnstructured`, which lets the same predicate evaluate any CRD without registering its Go type. Sources: [test/e2e/framework/predicates/predicates.go:26-33](), [test/e2e/framework/predicates/conditions.go:38-189](), [test/e2e/framework/predicates/sandbox.go:38-66](), [test/e2e/framework/predicates/metadata.go:26-148]() ### Logs and artifacts Two log sinks are wired up per test: - `logCapturingT` mirrors `Log/Logf/Error/Errorf/Fatal/Fatalf` to `//test.log` with `[ 12.345s]` elapsed-time prefixes. `ARTIFACTS` defaults to `./artifacts`. - On failure, `dumpControllerLogs` lists pods in `agent-sandbox-system` labelled `app=agent-sandbox-controller`, writes the full log of each to `/controller-.log`, and inlines the last 42 lines into the test output ("following k8s e2e convention"). For kind-specific diagnostics, `componentlogs.go` exposes `MustGetKubeletLogs` and `MustGetContainerdLogs`, both of which shell out via `ClusterClient.ExecuteOnNode` (`docker exec journalctl -u kubelet ...`) and persist results under the same artifacts directory. Sources: [test/e2e/framework/testlogs.go:26-102](), [test/e2e/framework/context.go:180-240](), [test/e2e/framework/componentlogs.go:25-52]() ## Scenario coverage The scenario tests are deliberately small and composable. Each one creates a uniquely-named namespace with `time.Now().UnixNano()` to avoid collisions, then registers it with `CreateWithCleanup` so the deferred delete tears down every child resource. ### Basic reconciliation — `TestSimpleSandbox` `basic_test.go` exercises the happy path: create a `Sandbox` from a tiny `simpleSandbox` fixture (a single `registry.k8s.io/pause:3.10` container with a test annotation and label), wait for the controller to populate the full `SandboxStatus` (Service name, FQDN, replicas, label selector built from `NameHash` of the sandbox name, and a `Ready=True` condition with `SandboxReasonDependenciesReady`), and then assert that the generated Pod and Service exist with the expected owner references and propagated metadata. `NameHash` is an FNV-1a hex digest used to compute the `agents.x-k8s.io/sandbox-name-hash=` selector that the controller stamps onto the Service. Sources: [test/e2e/basic_test.go:31-132]() ### Replica scaling — `TestSandboxReplicas` `replicas_test.go` first creates a Sandbox with `Spec.Replicas=1` and waits for the same `Ready=True` status. It then uses `MustUpdateObject` to flip replicas to 0; `MustUpdateObject` re-reads the latest version before mutating so the test does not race the controller. The expected status flips to `Ready=False` with reason `SandboxReasonSuspended`, plus a second `Suspended=True` condition with reason `SandboxReasonSuspendedPodTerminated`. The test finally asserts the Pod is gone (`WaitForObjectNotFound`) while the Service still exists (`NotDeleted` predicate) — verifying that scaling to zero is non-destructive at the Service level. Sources: [test/e2e/replicas_test.go:30-107](), [test/e2e/framework/client.go:113-129]() ### Shutdown lifecycle — `shutdown_test.go` Two scenarios cover the `Spec.ShutdownTime` and `Spec.Lifecycle.ShutdownPolicy` surfaces: - `TestSandboxShutdownTime` creates a Sandbox, then patches `ShutdownTime` to ~10 seconds in the future (truncated to RFC3339 second precision to match Kubernetes storage). It uses `PollUntilObjectMatches` (rather than the watch-based `WaitForObject`) to observe the transitioned status — `Service`/`ServiceFQDN` cleared, `Replicas: 0`, and `Ready=False` with reason `SandboxReasonExpired`. It then asserts the wall-clock time has passed the shutdown moment and that both the Pod and Service are deleted via `WaitForObjectNotFound`. - `TestSandboxRetainedExpiryPreservesFinishedCondition` builds a Sandbox whose container exits cleanly (`busybox sh -c exit 0`) with `ShutdownPolicyRetain`. It first uses `ConditionReasonEquals(SandboxConditionFinished, SandboxReasonPodSucceeded)` to confirm the Finished condition is set, then `require.Eventually` waits until both `Ready=Expired` and `Finished=PodSucceeded` are simultaneously present alongside cleared service fields — proving that the retain policy preserves the Finished condition through expiry. Sources: [test/e2e/shutdown_test.go:31-103](), [test/e2e/shutdown_test.go:105-160]() ### Parallel reconciliation — `parallelism_test.go` `parallelism_test.go` stresses the controller's worker pool. The helper `patchControllerConcurrency` is the most interesting piece: it snapshots the controller Deployment, rewrites the container args to bump `--sandbox-concurrent-workers`, `--sandbox-claim-concurrent-workers`, `--sandbox-warm-pool-concurrent-workers` to 10, raises `--kube-api-qps=50` / `--kube-api-burst=100`, ensures `--extensions` is present, and waits for the new pod to roll out (using `ReadyReplicasConditionIsTrue` + `ObservedGenerationMatchesGeneration`). Cleanup restores the original spec under `retry.RetryOnConflict` and sleeps 5 s for leader election to settle. Three scenarios then drive load through the patched controller: | Test | Setup | What it stresses | | --- | --- | --- | | `TestParallelSandboxes` | 20 Sandboxes created concurrently via goroutines, each waiting on `ReadyConditionIsTrue`. | Pure Sandbox reconciliation under concurrency. | | `TestParallelSandboxClaimsWithSufficientWarmPool` | `SandboxWarmPool` of 25 replicas, 20 parallel claims. | Claim binding when pre-warmed pods are available. | | `TestParallelSandboxClaimsWithInsufficientWarmPool` | Pool of 5, 20 parallel claims. | Forces on-demand pod creation when the pool is exhausted. | Errors flow through a buffered `errCh`; the test fails after `wg.Wait()` so each goroutine completes before reporting. Sources: [test/e2e/parallelism_test.go:36-108](), [test/e2e/parallelism_test.go:110-145](), [test/e2e/parallelism_test.go:147-218]() ### Warm-pool rollout — `extensions/warmpool_rollout_test.go` The extensions suite owns the `SandboxWarmPool` rollout semantics. Four scenarios share the helpers `createSandboxTemplate`, `createSandboxWarmPool`, `updateSandboxTemplateSpec` (appends `TEST_ENV=updated` to the pod), `verifySandboxStaysSame`, `verifySandboxRecreated`, and `verifySandboxHasUpdatedSpec` (which lists Sandboxes in the namespace, filters by `metav1.IsControlledBy(warmPool)`, and uses `require.Eventually` to wait for a fresh one). ```mermaid stateDiagram-v2 [*] --> Created : create SandboxTemplate + SandboxWarmPool Created --> Ready : WaitForWarmPoolReady\n(ReadyReplicasConditionIsTrue) Ready --> TemplateUpdated : Update SandboxTemplate state Strategy <> TemplateUpdated --> Strategy Strategy --> StaysSame : default / OnReplenish\n(verifySandboxStaysSame) StaysSame --> Replenished : Delete existing Sandbox\n(verifyOnReplenishLifecycle) Replenished --> Ready : new Sandbox has\nTEST_ENV=updated Strategy --> Recreated : Recreate\n(verifySandboxRecreated) Recreated --> Ready : new Sandbox observed,\nold marked for deletion Strategy --> NoRollout : metadata-only update\n(TestWarmPoolRolloutMetadataUpdate) NoRollout --> Ready : sandbox unchanged ``` Concrete scenarios: - `TestWarmPoolRollout` is a table test over `default`, `onreplenish`, and `recreate` strategies, asserting the expected lifecycle (`verifyOnReplenishLifecycle` for the first two, `verifySandboxRecreated` for the third). - `TestWarmPoolRolloutMultiTemplateIsolation` verifies that updating template A under the `Recreate` strategy rolls only warm-pool A's sandbox while warm-pool B's sandbox is untouched (no deletion timestamp). - `TestWarmPoolRolloutSwitchTemplate` changes `Spec.TemplateRef.Name` from A to B (specs identical) and asserts that the new sandbox is annotated with the new template name via `sandboxv1beta1.SandboxTemplateRefAnnotation`. - `TestWarmPoolRolloutMetadataUpdate` updates only labels in the template's pod metadata and asserts no rollout occurs (the existing sandbox keeps an empty deletion timestamp after a 5 s settling sleep). Sources: [test/e2e/extensions/warmpool_rollout_test.go:33-136](), [test/e2e/extensions/warmpool_rollout_test.go:138-240](), [test/e2e/extensions/warmpool_rollout_test.go:242-442]() ## How the pieces compose in a typical test A representative wait sits on top of three independent abstractions cooperating: ```text test code framework.ClusterClient framework.WatchSet dynamic.Interface ───────── ──────────────────────── ────────────────── ───────────────── WaitForObject(...) │ fast-path MatchesPredicates │ │ ↓ miss │ gvkForObject / gvrForGVK │ │ │ Subscribe(gvr, {ns,name}) ──────► getOrCreateWatch(gvr,ns) │ │ first time? │ └─► go watchLoop ──── Watch(ctx, ListOpts{Watch:true}) │ ◄─────── sub.Events (buffered 100) broadcast(event) ◄─── ResultChan() │ │ run predicates per event │ └─► last YAML logged on miss │ └─► return on first all-match or timeout │ sub.Close() ─► removeWatchIfIdle (keeps watch if other subs alive) ``` The result is that ten parallel `WaitForObject` calls in a `TestParallelSandboxes` goroutine pool reuse a single Sandbox watch per namespace, which keeps the API server load low while still giving each test a precise, predicate-driven completion signal. Sources: [test/e2e/framework/client.go:267-349](), [test/e2e/framework/watchset.go:80-211](), [test/e2e/parallelism_test.go:118-145]() ## Summary The e2e framework is intentionally small: `TestContext` boots a kubeconfig-driven client pair per test, `ClusterClient` wraps controller-runtime CRUD with cleanup-aware helpers, `predicates.ObjectPredicate` provides composable assertions over arbitrary CRDs via unstructured conversion, and `WatchSet` keeps a single dynamic watch per `(GVR, namespace)` so dozens of concurrent waits remain cheap. The scenario files build on these primitives to cover the Sandbox happy path, replica scaling, time- and policy-driven shutdown, controller behavior under parallel Sandbox and SandboxClaim load, and the full matrix of `SandboxWarmPool` rollout strategies.