# Load Testing & CI Pipelines > cluster-loader-based load-test recipes plus the prowjob presubmit/periodic configuration that runs them in CI. - Repository: kubernetes-sigs/agent-sandbox - GitHub: https://github.com/kubernetes-sigs/agent-sandbox - Human wiki: https://grok-wiki.com/public/wiki/kubernetes-sigs-agent-sandbox-c3f2597a654a - Complete Markdown: https://grok-wiki.com/public/wiki/kubernetes-sigs-agent-sandbox-c3f2597a654a/llms-full.txt ## Source Files - `dev/load-test/README.md` - `dev/load-test/cluster-loader-sandbox.yaml` - `dev/ci/presubmits` - `dev/ci/periodics` - `docs/prowjob_manual_run.md` --- > ⚠️ The agent returned an invalid wiki page. This page needs recovery. > > First failure: the page did not include the required "# Load Testing & CI Pipelines" heading near the top > Retry failure: the page did not include the required "# Load Testing & CI Pipelines" heading near the top
Relevant source files The following files were used as context for generating this wiki page: - [dev/load-test/README.md](dev/load-test/README.md) - [dev/load-test/agent-sandbox-load-test.yaml](dev/load-test/agent-sandbox-load-test.yaml) - [dev/load-test/cluster-loader-sandbox.yaml](dev/load-test/cluster-loader-sandbox.yaml) - [dev/load-test/test-recipes/README-rapid-burst.md](dev/load-test/test-recipes/README-rapid-burst.md) - [dev/load-test/test-recipes/rapid-burst-test.yaml](dev/load-test/test-recipes/rapid-burst-test.yaml) - [dev/load-test/test-recipes/run_rapid_burst.sh](dev/load-test/test-recipes/run_rapid_burst.sh) - [dev/load-test/test-recipes/high-volume-test.yaml](dev/load-test/test-recipes/high-volume-test.yaml) - [dev/load-test/test-recipes/medium-scale-concurrent-load-test.yaml](dev/load-test/test-recipes/medium-scale-concurrent-load-test.yaml) - [dev/load-test/test-recipes/throughput-test.yaml](dev/load-test/test-recipes/throughput-test.yaml) - [dev/load-test/test-recipes/warmpool-burst-test.yaml](dev/load-test/test-recipes/warmpool-burst-test.yaml) - [dev/load-test/test-recipes/monitor/agent-sandbox-controller-monitor.yaml](dev/load-test/test-recipes/monitor/agent-sandbox-controller-monitor.yaml) - [dev/load-test/test-recipes/templates/cluster-loader-sandbox-template.yaml](dev/load-test/test-recipes/templates/cluster-loader-sandbox-template.yaml) - [dev/load-test/test-recipes/templates/cluster-loader-warmpool.yaml](dev/load-test/test-recipes/templates/cluster-loader-warmpool.yaml) - [dev/load-test/test-recipes/templates/cluster-loader-sandbox-claim.yaml](dev/load-test/test-recipes/templates/cluster-loader-sandbox-claim.yaml) - [dev/load-test/test-recipes/templates/cluster-loader-hpa.yaml](dev/load-test/test-recipes/templates/cluster-loader-hpa.yaml) - [dev/load-test/test-recipes/templates/cluster-loader-capacity-buffer.yaml](dev/load-test/test-recipes/templates/cluster-loader-capacity-buffer.yaml) - [dev/ci/periodics/test-load-test](dev/ci/periodics/test-load-test) - [dev/ci/presubmits/test-e2e](dev/ci/presubmits/test-e2e) - [dev/ci/presubmits/test-unit](dev/ci/presubmits/test-unit) - [dev/ci/presubmits/lint-go](dev/ci/presubmits/lint-go) - [dev/ci/presubmits/lint-api](dev/ci/presubmits/lint-api) - [dev/ci/presubmits/test-autogen-up-to-date](dev/ci/presubmits/test-autogen-up-to-date) - [dev/ci/presubmits/shared/utils.py](dev/ci/presubmits/shared/utils.py) - [dev/ci/shared/runner.py](dev/ci/shared/runner.py) - [docs/prowjob_manual_run.md](docs/prowjob_manual_run.md)
# Load Testing & CI Pipelines This page documents how `kubernetes-sigs/agent-sandbox` exercises the controller under load and how those tests are wired into Prow as presubmit and periodic jobs. Two surfaces collaborate: the `dev/load-test/` recipes built on top of [ClusterLoader2](https://github.com/kubernetes/perf-tests/tree/master/clusterloader2) (CL2), and the `dev/ci/` Python entrypoints that Prow invokes. The recipes describe *what* to measure — `Sandbox`/`SandboxClaim` startup latency, throughput, churn, warm-pool burst behaviour — while the CI entrypoints describe *how* to bring up an isolated KinD cluster, push the controller image, run a recipe, and emit a JUnit artifact that Prow can surface in Spyglass. The two layers are deliberately separable. Any recipe can be run by hand against any cluster (the rapid-burst recipe is targeted at GKE; the entrypoint recipe at KinD), and CI just composes a thin wrapper around the entrypoint recipe with smaller defaults so it fits inside a Prow pod. ## High-Level Architecture ```mermaid flowchart LR subgraph Prow["kubernetes/test-infra (Prow)"] PJ["pj-on-kind.sh /
periodic job spec"] end subgraph CIEntry["dev/ci/ (Python wrappers)"] Periodic["periodics/test-load-test
(LoadTestRunner)"] Pre_e2e["presubmits/test-e2e"] Pre_unit["presubmits/test-unit"] Pre_lintgo["presubmits/lint-go"] Pre_lintapi["presubmits/lint-api"] Pre_autogen["presubmits/test-autogen-up-to-date"] Runner["shared/runner.py
TestRunner"] end subgraph DevTools["dev/tools/"] Kind["create-kind-cluster"] Push["push-images"] Deploy["deploy-to-kube
+ deploy-cloud-provider"] Fix["fix-* scripts"] Lint["lint-go / lint-api / test-unit / test-e2e"] end subgraph LoadTest["dev/load-test/"] EntryRecipe["agent-sandbox-load-test.yaml
(entrypoint recipe)"] SandboxTpl["cluster-loader-sandbox.yaml
(Sandbox object template)"] Recipes["test-recipes/*.yaml
(rapid-burst, throughput,
high-volume, medium-scale, warmpool-burst)"] Templates["test-recipes/templates/*.yaml
(SandboxTemplate, WarmPool, Claim,
HPA, CapacityBuffer)"] Monitor["test-recipes/monitor/
ServiceMonitor"] end PJ --> Periodic PJ --> Pre_e2e PJ --> Pre_unit PJ --> Pre_lintgo PJ --> Pre_lintapi PJ --> Pre_autogen Periodic --> Runner Pre_e2e --> Runner Periodic --> EntryRecipe EntryRecipe --> SandboxTpl Recipes --> Templates Recipes --> Monitor Runner --> Kind Runner --> Push Runner --> Deploy Pre_unit --> Lint Pre_lintgo --> Lint Pre_lintapi --> Lint Pre_autogen --> Fix ``` Sources: [dev/ci/shared/runner.py:23-80](), [dev/ci/periodics/test-load-test:48-110](), [dev/load-test/agent-sandbox-load-test.yaml:1-56](), [dev/load-test/test-recipes/rapid-burst-test.yaml:1-46]() ## The CI Entrypoint Layer (`dev/ci/`) Every Prow job in this repo is a small Python script that Prow shells out to. There are two flavours: 1. **Lint/unit wrappers** that just delegate to a script in `dev/tools/`. 2. **Cluster-backed runners** that subclass `TestRunner` and provision a KinD cluster before running a binary. ### `TestRunner` base class `dev/ci/shared/runner.py` is the spine shared by every cluster-backed CI job. It exposes a four-method lifecycle — `setup_cluster`, `run_tests`, `copy_artifacts`, `main` — and threads the repository root and an `--image-prefix` argument through every invocation: ```python # dev/ci/shared/runner.py class TestRunner: def __init__(self, name, description): ... self.parser.add_argument( "--image-prefix", ... default="kind.local/", ) def setup_cluster(self, args, extra_push_images_args=None): image_tag = tools_utils.get_image_tag() subprocess.run([f"{self.repo_root}/dev/tools/create-kind-cluster", self.name, "--recreate", "--kubeconfig", f"{self.repo_root}/bin/KUBECONFIG"]) ... subprocess.run([f"{self.repo_root}/dev/tools/push-images", ...]) subprocess.run([f"{self.repo_root}/dev/tools/deploy-to-kube", "--image-prefix", args.image_prefix, "--image-tag", image_tag, "--extensions"]) subprocess.run([f"{self.repo_root}/dev/tools/deploy-cloud-provider"]) ``` `setup_cluster` is the same for every job: recreate a KinD cluster named after the job, build and load images into the cluster, deploy the controller plus extension CRDs, and stand up the cloud-provider mock. `run_tests` is `NotImplementedError` — subclasses must override it. `copy_artifacts` is a no-op by default; subclasses override it to publish JUnit files into `$ARTIFACTS`, the Prow-set output directory. Sources: [dev/ci/shared/runner.py:23-80]() ### Presubmits | Entrypoint | Subclass / pattern | Underlying script | JUnit output | |---|---|---|---| | `dev/ci/presubmits/test-e2e` | `E2ETestRunner(TestRunner)` | `dev/tools/test-e2e --suite {all,tests,benchmarks}` | `bin/e2e-go-junit.xml` → `$ARTIFACTS/junit_go.xml`, `bin/e2e-python-sdk-junit.xml` → `$ARTIFACTS/junit_python_sdk.xml` | | `dev/ci/presubmits/test-unit` | thin wrapper via `shared/utils.py` | `dev/tools/test-unit` | `bin/unit-junit.xml` → `$ARTIFACTS/junit.xml` | | `dev/ci/presubmits/lint-go` | thin wrapper | `dev/tools/lint-go` | none | | `dev/ci/presubmits/lint-api` | thin wrapper | `dev/tools/lint-api` | none | | `dev/ci/presubmits/test-autogen-up-to-date` | inline | runs every `dev/tools/fix-*` and fails if `git diff --name-only` is non-empty | none | The lint/unit wrappers reuse `dev/ci/presubmits/shared/utils.py`, which walks up four parents from `__file__` to find the repo root and runs the named tool from `dev/tools/`: ```python # dev/ci/presubmits/shared/utils.py def get_repo_root(): presubmit_dir = os.path.dirname(os.path.dirname(os.path.realpath(__file__))) return os.path.dirname(os.path.dirname(os.path.dirname(presubmit_dir))) def run_dev_tool(tool_name): repo_root = get_repo_root() result = subprocess.run([f"{repo_root}/dev/tools/{tool_name}"]) return result.returncode ``` `test-autogen-up-to-date` is the odd one out — it does not invoke a single tool but iterates over `sorted(glob.glob("dev/tools/fix-*"))`, runs each one, and exits non-zero if any of them produces a diff. It prints the full `git diff` and a remediation hint to re-run the script locally. Sources: [dev/ci/presubmits/test-e2e:28-47](), [dev/ci/presubmits/test-unit:23-35](), [dev/ci/presubmits/lint-go:21-23](), [dev/ci/presubmits/lint-api:21-23](), [dev/ci/presubmits/test-autogen-up-to-date:22-48](), [dev/ci/presubmits/shared/utils.py:20-29]() ### The periodic load-test entrypoint `dev/ci/periodics/test-load-test` is the only entry under `dev/ci/periodics/`. It subclasses `TestRunner` as `LoadTestRunner`, takes four CL2 parameters as CLI arguments, and reduces the defaults so the test fits inside a Prow pod backed by KinD: | Flag | Default | Forwarded as | |---|---|---| | `--replicas` | 5 | `CL2_REPLICAS` | | `--namespaces` | 1 | `CL2_NAMESPACES` | | `--qps` | 10 | `CL2_QPS` | | `--namespace-prefix` | `agent-sandbox` | `CL2_NAMESPACE_PREFIX` | `setup_cluster` is overridden to pass `extra_push_images_args=["--controller-only"]`, so the load-test job only loads the controller image into KinD — none of the auxiliary images needed for full e2e are pushed: ```python # dev/ci/periodics/test-load-test def setup_cluster(self, args): return super().setup_cluster(args, extra_push_images_args=["--controller-only"]) ``` `run_tests` (a) installs ClusterLoader2 on the fly by `git clone --depth 1` of `kubernetes/perf-tests` and `go build -o bin/clusterloader2 ./cmd/clusterloader.go`, (b) writes the four CLI flags into a `--testoverrides` file, then (c) invokes `clusterloader2` from `dev/load-test/` (because the entrypoint recipe references `cluster-loader-sandbox.yaml` via a relative path): ```python cmd = [cl2_path, f"--testconfig={test_config}", f"--kubeconfig={kubeconfig}", f"--testoverrides={overrides_path}", "--provider=kind", "--v=2", f"--report-dir={report_dir}"] subprocess.run(cmd, cwd=os.path.join(self.repo_root, "dev/load-test")) ``` `copy_artifacts` then copies `bin/junit.xml` to `$ARTIFACTS/junit_load_test.xml` so the Prow UI can render it. The clusterloader2 binary and the temporary kubeconfig are removed at the end of each run so subsequent invocations rebuild fresh. Sources: [dev/ci/periodics/test-load-test:29-116]() ## The Load-Test Recipe Layer (`dev/load-test/`) There are two cohorts of CL2 recipes here. The top-level `agent-sandbox-load-test.yaml` is the one driven by CI; the recipes under `test-recipes/` are tuned for larger GKE clusters and are run by humans. ### The CI-driven entrypoint recipe `dev/load-test/agent-sandbox-load-test.yaml` is intentionally minimal. It accepts four parameters (`CL2_REPLICAS`, `CL2_NAMESPACES`, `CL2_QPS`, `CL2_NAMESPACE_PREFIX`) and executes five CL2 steps: 1. `Start Startup Latency Measurement` — registers a `PodStartupLatency` measurement. 2. `Create Sandboxes` — instantiates `cluster-loader-sandbox.yaml` (a `Sandbox` CR with an alpine container that sleeps 3600s) under the `BurstCreate` tuning set. 3. `Wait for Sandboxes to be Ready` — uses `WaitForRunningPods` against the label selector `group=agent-sandbox-load-test`. 4. `Gather Results` — collects the `SandboxStartupLatency` measurement. 5. `Delete Sandboxes` — drops the replica count to 0. A TODO at the top of the file notes that the recipe currently measures only the downstream Pod latency, not the end-to-end `Sandbox` lifecycle, and that it should switch to `GenericPrometheusQuery` once `agent_sandbox_creation_latency_ms` metrics are exposed by the controller: ```yaml # dev/load-test/agent-sandbox-load-test.yaml # TODO: Replace PodStartupLatency with GenericPrometheusQuery # once agent_sandbox_creation_latency_ms metrics are available to measure # end-to-end controller overhead. - Identifier: SandboxStartupLatency Method: PodStartupLatency # Warning: this only measures the downstream Pod latency, not the end-to-end Sandbox lifecycle ``` Sources: [dev/load-test/agent-sandbox-load-test.yaml:1-56](), [dev/load-test/cluster-loader-sandbox.yaml:1-17](), [dev/load-test/README.md:33-86]() ### Test recipes under `dev/load-test/test-recipes/` The five `*.yaml` recipes under `test-recipes/` cover different scenarios. They are designed to be driven by hand (or by a shell wrapper such as `run_rapid_burst.sh`) against a real GKE cluster running the agent-sandbox controller. | Recipe | Tuning set(s) | Defaults | What it stresses | |---|---|---|---| | `rapid-burst-test.yaml` | `BurstCreate` (QPS 100) | `BURST_SIZE=50`, `TOTAL_BURSTS=100`, `WARMPOOL_SIZE=200` | Repeated bursts of `SandboxClaim` creation against a populated `SandboxWarmPool`, with optional HPA + GKE `CapacityBuffer` scaling | | `warmpool-burst-test.yaml` | `RampUp` / `RampDown` (100 / 50 QPS) | `REPLICAS=100`, `NAMESPACES=1` | Warm-pool churn under ramp-up/down | | `high-volume-test.yaml` | `RampUp` / `RampDown` (10 / 50 QPS) | `REPLICAS=100` | Capacity ceiling — "keep increasing this to find the max" | | `throughput-test.yaml` | `ConstantRate` / `QuickFinalDeletion` (200 / 100 QPS) | `REPLICAS=12000` | Sustained creation throughput | | `medium-scale-concurrent-load-test.yaml` | `ConstantChurn` / `QuickFinalDeletion` (2 / 100 QPS) | `REPLICAS=1200` | Long-running steady-state churn | The recipes share a common set of object templates under `test-recipes/templates/`: | Template | Kind | API | |---|---|---| | `cluster-loader-sandbox-template.yaml` | `SandboxTemplate` | `extensions.agents.x-k8s.io/v1beta1` | | `cluster-loader-warmpool.yaml` | `SandboxWarmPool` | `extensions.agents.x-k8s.io/v1beta1` | | `cluster-loader-sandbox-claim.yaml` | `SandboxClaim` | `extensions.agents.x-k8s.io/v1beta1` | | `cluster-loader-hpa.yaml` | `HorizontalPodAutoscaler` (External metric, targets `SandboxWarmPool`) | `autoscaling/v2` | | `cluster-loader-capacity-buffer.yaml` | `CapacityBuffer` | `autoscaling.x-k8s.io/v1beta1` | A single `ServiceMonitor` under `test-recipes/monitor/` scrapes the controller in `agent-sandbox-system` every 10s on the `metrics` port; it is loaded into CL2 via `--prometheus-additional-monitors-path`. Sources: [dev/load-test/test-recipes/rapid-burst-test.yaml:1-46](), [dev/load-test/test-recipes/high-volume-test.yaml:1-15](), [dev/load-test/test-recipes/medium-scale-concurrent-load-test.yaml:1-15](), [dev/load-test/test-recipes/throughput-test.yaml:1-17](), [dev/load-test/test-recipes/warmpool-burst-test.yaml:1-16](), [dev/load-test/test-recipes/templates/cluster-loader-sandbox-template.yaml:1-20](), [dev/load-test/test-recipes/templates/cluster-loader-warmpool.yaml:1-10](), [dev/load-test/test-recipes/templates/cluster-loader-sandbox-claim.yaml:1-8](), [dev/load-test/test-recipes/templates/cluster-loader-hpa.yaml:1-24](), [dev/load-test/test-recipes/templates/cluster-loader-capacity-buffer.yaml:1-12](), [dev/load-test/test-recipes/monitor/agent-sandbox-controller-monitor.yaml:1-19]() ### Anatomy of the rapid-burst recipe `rapid-burst-test.yaml` is the most complete recipe and the one most worth understanding, because it exercises every template and uses both `PodStartupLatency` and `GenericPrometheusQuery` measurements. ```mermaid stateDiagram-v2 [*] --> StartMeasurements StartMeasurements --> SetupTemplate: Setup Sandbox Template SetupTemplate --> SetupWarmPool: Setup Sandbox Warm Pool SetupWarmPool --> WaitWarmReady: Wait for Warm Pool Sandboxes\n(WaitForGenericK8sObjects,\nlabel: agents.x-k8s.io/warm-pool-sandbox) WaitWarmReady --> SetupHPA: if CL2_ENABLE_HPA=true SetupHPA --> SetupCapacityBuffer WaitWarmReady --> SetupCapacityBuffer: if CL2_ENABLE_HPA=false SetupCapacityBuffer --> PauseForNodes: if CL2_ENABLE_CAPACITY_BUFFER=true\nSleep CL2_CAPACITY_BUFFER_PAUSE SetupCapacityBuffer --> BurstLoop: if CL2_ENABLE_CAPACITY_BUFFER=false PauseForNodes --> BurstLoop BurstLoop --> BurstLoop: range 1..CL2_TOTAL_BURSTS\nCreate, WaitFor (Ready=True), Sleep 20s BurstLoop --> PauseScrape: Pause 1m for Prometheus scrape PauseScrape --> Gather: Gather Results\n(PodStartupLatency +\nGenericPrometheusQuery x2) Gather --> TeardownClaims TeardownClaims --> TeardownHPA: if enabled TeardownClaims --> TeardownBuffer: if enabled TeardownHPA --> TeardownBuffer TeardownBuffer --> TeardownWarmPool TeardownClaims --> TeardownWarmPool: otherwise TeardownWarmPool --> TeardownTemplate TeardownTemplate --> [*] ``` Two things are notable. First, the recipe couples directly to the controller's CRDs through `WaitForGenericK8sObjects`. It waits on `agents.x-k8s.io/v1beta1` `sandboxes` (label `agents.x-k8s.io/warm-pool-sandbox`) for warm-pool readiness, and on `extensions.agents.x-k8s.io/v1beta1` `sandboxclaims` for burst readiness — both with `successfulConditions: ["Ready=True"]` and `maxFailedObjectCount: 0`: ```yaml # dev/load-test/test-recipes/rapid-burst-test.yaml - Identifier: WaitForBurst{{$burstIteration}}SandboxClaims Method: WaitForGenericK8sObjects Params: objectGroup: extensions.agents.x-k8s.io objectVersion: v1beta1 objectResource: sandboxclaims successfulConditions: ["Ready=True"] failedConditions: [] minDesiredObjectCount: {{MultiplyInt $replicaCount $namespaces}} maxFailedObjectCount: 0 timeout: 60m refreshInterval: 20ms ``` Second, the recipe uses three Prometheus histograms exposed by the controller — `agent_sandbox_claim_startup_latency_ms_bucket` and `agent_sandbox_claim_controller_startup_latency_ms_bucket` — for `p50`, `p90`, `p99` violations, with thresholds (1s, 1s, 5s) declared inline. This is the pattern the entrypoint recipe's TODO is moving toward. The recipe is driven by `run_rapid_burst.sh`. The shell wrapper hard-codes the GKE flow (`--provider=gke`, `$HOME/perf-tests`, `$HOME/agent-sandbox`), enables the CL2 Prometheus server, plugs in the `ServiceMonitor`, and writes a JSON test-overrides file under `dev/load-test/test-recipes/tmp//`: ```bash # dev/load-test/test-recipes/run_rapid_burst.sh cd "$CL2_DIR" go run cmd/clusterloader.go \ --enable-prometheus-server=true \ --kubeconfig=$HOME/.kube/config \ --prometheus-additional-monitors-path="${TEST_DIR}/monitor" \ --provider=gke \ --report-dir="${LOGS_DIR}" \ --testconfig="${TEST_CONFIG}" \ --testoverrides="${LOGS_DIR}/testoverrides.json" \ --v=2 ``` The wrapper also performs pre-flight validation — it pings the cluster, refuses to enable `CapacityBuffer` if `autoscaling.x-k8s.io` is missing from `kubectl api-resources`, and rejects `ENABLE_HPA=true` if `WARMPOOL_SIZE` is outside `[HPA_MIN_REPLICAS, HPA_MAX_REPLICAS]`. The README documents that `ENABLE_CAPACITY_BUFFER=true` triggers a 5-minute sleep after the buffer is created so GKE node auto-provisioning has time to spin up standby nodes before the burst loop starts. Sources: [dev/load-test/test-recipes/rapid-burst-test.yaml:44-238](), [dev/load-test/test-recipes/run_rapid_burst.sh:20-113](), [dev/load-test/test-recipes/README-rapid-burst.md:66-126]() ## Test-time Object Graph The five recipe templates and the controller's CRDs form a tight graph at test time: ```mermaid classDiagram class SandboxTemplate { +metadata.name +spec.podTemplate } class SandboxWarmPool { +spec.replicas +spec.sandboxTemplateRef } class SandboxClaim { +spec.sandboxTemplateRef +status.conditions[Ready] } class Sandbox { +metadata.labels[agents.x-k8s.io/warm-pool-sandbox] +status.conditions[Ready] } class HorizontalPodAutoscaler { +spec.scaleTargetRef = SandboxWarmPool +spec.metrics[External] } class CapacityBuffer { +spec.percentage +spec.scalableRef = SandboxWarmPool +spec.provisioningStrategy } SandboxWarmPool --> SandboxTemplate : sandboxTemplateRef SandboxClaim --> SandboxTemplate : sandboxTemplateRef SandboxWarmPool ..> Sandbox : pre-creates pool members SandboxClaim ..> Sandbox : binds a warm Sandbox HorizontalPodAutoscaler --> SandboxWarmPool : scaleTargetRef CapacityBuffer --> SandboxWarmPool : scalableRef ``` The HPA template uses an `External` metric with selectors on `metric.labels.warmpool_name` and `metric.labels.exported_namespace`, allowing the same HPA shape to be reused across namespaces created by CL2. Sources: [dev/load-test/test-recipes/templates/cluster-loader-sandbox-template.yaml:1-20](), [dev/load-test/test-recipes/templates/cluster-loader-warmpool.yaml:1-10](), [dev/load-test/test-recipes/templates/cluster-loader-sandbox-claim.yaml:1-8](), [dev/load-test/test-recipes/templates/cluster-loader-hpa.yaml:1-24](), [dev/load-test/test-recipes/templates/cluster-loader-capacity-buffer.yaml:1-12](), [dev/load-test/test-recipes/rapid-burst-test.yaml:117-132]() ## End-to-End CI Flow for the Periodic Putting the two layers together, the periodic load-test job follows this sequence inside a Prow pod: ```mermaid sequenceDiagram participant Prow participant Periodic as periodics/test-load-test participant Runner as TestRunner (shared/runner.py) participant DevTools as dev/tools/* participant KinD participant CL2 as clusterloader2 (cloned & built) participant Recipe as agent-sandbox-load-test.yaml participant Out as $ARTIFACTS Prow->>Periodic: exec entrypoint Periodic->>Runner: LoadTestRunner.main() Runner->>DevTools: create-kind-cluster --recreate DevTools->>KinD: bootstrap cluster Runner->>DevTools: push-images --controller-only Runner->>DevTools: deploy-to-kube --extensions Runner->>DevTools: deploy-cloud-provider Periodic->>CL2: git clone perf-tests + go build Periodic->>CL2: clusterloader2 --testconfig=agent-sandbox-load-test.yaml --testoverrides=... CL2->>Recipe: execute steps 1..5 Recipe->>KinD: create Sandboxes, WaitForRunningPods CL2-->>Periodic: bin/junit.xml Periodic->>Out: copy junit.xml -> junit_load_test.xml ``` `WaitForGenericK8sObjects` is what couples the rapid-burst recipe to the controller's CRDs, while the entrypoint recipe waits on raw `Pod` readiness via `WaitForRunningPods`. The shape of the JUnit output (one testcase per CL2 step) is illustrated in the load-test README: ```xml ``` Sources: [dev/load-test/README.md:73-86](), [dev/load-test/test-recipes/rapid-burst-test.yaml:189-204](), [dev/load-test/agent-sandbox-load-test.yaml:34-48](), [dev/ci/periodics/test-load-test:80-116]() ## Running a Prow Job Locally `docs/prowjob_manual_run.md` documents how an engineer reproduces the periodic outside of CI. The job runs as KinD-in-Docker (nested virtualisation), so the doc raises host inotify limits before invocation: ```bash sudo sysctl -w fs.inotify.max_user_watches=524288 sudo sysctl -w fs.inotify.max_user_instances=512 ``` From a checkout of `kubernetes/test-infra`: ```bash ./config/pj-on-kind.sh periodic-agent-sandbox-perf-load-test kubectl get pods kubectl logs -f -c test # After completion, artifacts live under the the script prints at start cd /periodic-agent-sandbox-perf-load-test/ cat finished.json ``` The job referenced is the same `LoadTestRunner` entrypoint; its Prow spec is maintained in `config/jobs/kubernetes-sigs/agent-sandbox/agent-sandbox-periodics-main.yaml` in the `kubernetes/test-infra` repository, not in this one. The doc also notes minimum host resources of 8 CPU and 16 GB RAM for the nested cluster to come up cleanly. Sources: [docs/prowjob_manual_run.md:1-84]() ## Extending the Pipelines Three patterns recur when adding new coverage: 1. **A new presubmit shell wrapper** — copy `dev/ci/presubmits/lint-go` and change `run_dev_tool("lint-go")` to a target script in `dev/tools/`. The `shared/utils.py` helper handles repo-root resolution. 2. **A new cluster-backed test** — subclass `TestRunner` from `dev/ci/shared/runner.py`, override `run_tests` to invoke a binary, and override `copy_artifacts` to deposit a `junit_*.xml` into `$ARTIFACTS`. `setup_cluster` is inherited and handles KinD bring-up, image push, and CRD install; pass `extra_push_images_args=["--controller-only"]` when you only need the controller image. 3. **A new load-test recipe** — add a YAML under `dev/load-test/test-recipes/` (parameterised with `DefaultParam .CL2_*`), reuse the existing template manifests for `SandboxTemplate` / `SandboxWarmPool` / `SandboxClaim` / `HorizontalPodAutoscaler` / `CapacityBuffer`, and drive it from a local shell wrapper or a new CL2 invocation block in `LoadTestRunner.run_tests`. Prefer `GenericPrometheusQuery` over `PodStartupLatency` whenever the controller already exports a relevant histogram (e.g., `agent_sandbox_claim_startup_latency_ms_bucket`, used by the rapid-burst recipe). Together these two surfaces give the project a stable signal (the small periodic in KinD) and a manual proving ground (the GKE-targeted recipes) without duplicating the wiring that brings the controller online. Sources: [dev/ci/shared/runner.py:23-80](), [dev/ci/periodics/test-load-test:48-110](), [dev/load-test/test-recipes/rapid-burst-test.yaml:1-92](), [dev/load-test/test-recipes/rapid-burst-test.yaml:44-92]()