# Go High-Level SDK (clients/go/sandbox) > The high-level Go client: Sandbox lifecycle, command execution, file transfer, port tunnels, gateway, connector strategies, and tracing helpers. - Repository: kubernetes-sigs/agent-sandbox - GitHub: https://github.com/kubernetes-sigs/agent-sandbox - Human wiki: https://grok-wiki.com/public/wiki/kubernetes-sigs-agent-sandbox-c3f2597a654a - Complete Markdown: https://grok-wiki.com/public/wiki/kubernetes-sigs-agent-sandbox-c3f2597a654a/llms-full.txt ## Source Files - `clients/go/sandbox/sandbox.go` - `clients/go/sandbox/client.go` - `clients/go/sandbox/connector.go` - `clients/go/sandbox/commands.go` - `clients/go/sandbox/files.go` - `clients/go/sandbox/tunnel.go` - `clients/go/sandbox/gateway.go` ---
Relevant source files The following files were used as context for generating this wiki page: - [clients/go/sandbox/sandbox.go](clients/go/sandbox/sandbox.go) - [clients/go/sandbox/client.go](clients/go/sandbox/client.go) - [clients/go/sandbox/connector.go](clients/go/sandbox/connector.go) - [clients/go/sandbox/commands.go](clients/go/sandbox/commands.go) - [clients/go/sandbox/files.go](clients/go/sandbox/files.go) - [clients/go/sandbox/tunnel.go](clients/go/sandbox/tunnel.go) - [clients/go/sandbox/gateway.go](clients/go/sandbox/gateway.go) - [clients/go/sandbox/strategy.go](clients/go/sandbox/strategy.go) - [clients/go/sandbox/options.go](clients/go/sandbox/options.go) - [clients/go/sandbox/types.go](clients/go/sandbox/types.go) - [clients/go/sandbox/tracing.go](clients/go/sandbox/tracing.go) - [clients/go/sandbox/k8s.go](clients/go/sandbox/k8s.go)
# Go High-Level SDK (clients/go/sandbox) The `sigs.k8s.io/agent-sandbox/clients/go/sandbox` package is the high-level Go client for the agent-sandbox project. It hides the Kubernetes API plumbing required to provision a sandbox (creating a `SandboxClaim`, waiting for a `Sandbox` to become ready), and exposes a small surface for running commands and moving files inside the sandbox over HTTP. It also encapsulates how the SDK reaches the in-cluster `sandbox-router`: directly, through a `Gateway` resource, or via an SPDY port-forward tunnel. This page maps the package by responsibility: the `Sandbox` lifecycle object and the `Client` registry, the `connector` retry/transport layer, the three `ConnectionStrategy` implementations, the `Commands` and `Files` sub-objects, and the OpenTelemetry tracing helpers that thread spans through every operation. ## Package architecture The package is layered: user code holds a `Sandbox` (or a `Client` that owns many). The `Sandbox` exposes `Commands` and `Files` sub-objects; both delegate to a shared `connector` that owns the HTTP client, retry policy, and request-ID instrumentation. The `connector` does not know how to find the sandbox; it asks a `ConnectionStrategy` for the base URL on `Connect`. Three strategies are wired by `New` based on `Options`, and a `K8sHelper` provides the Kubernetes clientsets used for claim creation, name resolution, readiness watches, and pod discovery. ```mermaid flowchart LR subgraph user["User code"] App[Application] end subgraph sdk["clients/go/sandbox"] Client[Client
registry + signals] Sandbox[Sandbox
lifecycle + lock] Commands[Commands] Files[Files] Connector[connector
HTTP + retry] subgraph strategies["ConnectionStrategy"] Direct[DirectStrategy] Gateway[gatewayStrategy] Tunnel[tunnelStrategy] end K8s[K8sHelper
clientsets] Tracing[tracing.go
spans + OTLP] end subgraph cluster["Kubernetes / sandbox-router"] Claim[SandboxClaim] SBox[Sandbox CR] GW[Gateway] Router[sandbox-router Pod] end App --> Client --> Sandbox App --> Sandbox Sandbox --> Commands --> Connector Sandbox --> Files --> Connector Sandbox --> K8s Connector --> Direct Connector --> Gateway Connector --> Tunnel K8s --> Claim K8s --> SBox Gateway --> GW Tunnel --> Router Direct --> Router Commands -. spans .-> Tracing Files -. spans .-> Tracing Sandbox -. lifecycle span .-> Tracing ``` Sources: [clients/go/sandbox/sandbox.go:32-169](clients/go/sandbox/sandbox.go), [clients/go/sandbox/client.go:32-76](clients/go/sandbox/client.go), [clients/go/sandbox/connector.go:54-119](clients/go/sandbox/connector.go), [clients/go/sandbox/strategy.go:20-32](clients/go/sandbox/strategy.go) ## Sandbox lifecycle `Sandbox` is the central object. Construction (`New`) only configures dependencies; the cluster is not touched until `Open(ctx)` is called. `Open` serializes through a single-slot `lifecycleSem` channel so concurrent `Open`/`Close`/`Disconnect` cannot interleave, registers an `openCancel` so a parallel `Close` or `Disconnect` can preempt a stuck `Open`, and starts a long-lived lifecycle span that becomes the parent of all subsequent operation spans. The first `Open` follows the create path: `createClaim` → `resolveSandboxName` (warm-pool aware: the resolved sandbox name may differ from the generated claim name) → `waitForSandboxReady` → `connector.Connect`. The remaining ready-timeout budget is reduced by the time spent resolving the name so the wait cannot overrun. If any step after claim creation fails, `rollbackOpen` deletes the claim on a detached `CleanupTimeout` context and either clears or preserves `claimName` depending on whether deletion succeeded. A preserved `claimName` is what causes the next `Open` to take the reconnect path via `ErrOrphanedClaim` retry semantics. `Close` first cancels any in-progress `Open`, then tears down the transport before flipping `draining = true` and swapping the in-flight `WaitGroup`. The ordering is deliberate: closing the transport first makes new operations fail fast with `ErrNotReady`, so no operation can slip past `trackOp` and still reach a live connection. The drain budget is half of `CleanupTimeout`; the other half is reserved for claim deletion. `Disconnect` is the suspend variant — it releases the transport but does not delete the claim, and notably does not call `connector.Close()` on a timed-out semaphore acquisition because doing so would race with a just-succeeding `Open`. ```mermaid stateDiagram-v2 [*] --> New: sandbox.New(opts) New --> Opening: Open(ctx) Opening --> Ready: createClaim
resolveSandboxName
waitForSandboxReady
connector.Connect Opening --> Orphaned: failure after claim create
(rollbackOpen could not delete) Opening --> Failed: failure before claim
or rollback succeeded Orphaned --> Opening: Open() retries via reconnect path Orphaned --> Closed: Close() succeeds Ready --> Disconnected: Disconnect(ctx) Disconnected --> Opening: Open(ctx) → reconnect() Ready --> Draining: Close(ctx) Draining --> Closed: drain + deleteClaim Failed --> [*] Closed --> [*] ``` Sources: [clients/go/sandbox/sandbox.go:175-275](clients/go/sandbox/sandbox.go), [clients/go/sandbox/sandbox.go:277-336](clients/go/sandbox/sandbox.go), [clients/go/sandbox/sandbox.go:338-364](clients/go/sandbox/sandbox.go), [clients/go/sandbox/sandbox.go:367-503](clients/go/sandbox/sandbox.go), [clients/go/sandbox/k8s.go:123-269](clients/go/sandbox/k8s.go) ### In-flight operation tracking Per-call methods on `Commands` and `Files` call `defer trackOp()()` on entry. `trackOp` reads `draining` and the current `inflightOps` under `s.mu`: if draining, it returns a no-op; otherwise it adds to the WaitGroup. `Close` swaps `inflightOps` for a fresh `WaitGroup` while marking `draining = true` under the same lock, so the two states are observed atomically. This is the only mechanism the SDK uses to guarantee that operations cannot slip into a sandbox that is about to delete its claim. Sources: [clients/go/sandbox/sandbox.go:404-414](clients/go/sandbox/sandbox.go), [clients/go/sandbox/sandbox.go:574-589](clients/go/sandbox/sandbox.go) ## Client: multi-sandbox registry `Client` is a thin manager over many `Sandbox` instances. It keeps a `map[Key]*Sandbox` keyed by `{Namespace, ClaimName}` and shares a single `K8sHelper` (so the underlying clientsets and REST config are not rebuilt per sandbox). `CreateSandbox` provisions and opens a new one, registering it on success. `GetSandbox` returns a cached, ready handle, evicts a stale one, or constructs a new `Sandbox` and forces the reconnect path by pre-populating `claimName` and `sandboxName` before `Open`. `EnableAutoCleanup` installs a SIGINT/SIGTERM handler that calls `DeleteAll` and then re-raises the signal so the default handler can terminate the process. The returned `stop` function detaches the handler. | Method | Effect | |---|---| | `CreateSandbox(ctx, template, ns)` | New claim + open + register | | `GetSandbox(ctx, claim, ns)` | Cached if ready; else verify + reconnect | | `ListActiveSandboxes()` | Tracked handles, prunes unready entries | | `ListAllSandboxes(ctx, ns)` | Lists all `SandboxClaim` names in namespace | | `DeleteSandbox(ctx, claim, ns)` | Close if tracked, else delete claim directly | | `DeleteAll(ctx)` | Best-effort close of every tracked handle | | `EnableAutoCleanup()` | SIGINT/SIGTERM → `DeleteAll` | Sources: [clients/go/sandbox/client.go:38-231](clients/go/sandbox/client.go), [clients/go/sandbox/client.go:235-265](clients/go/sandbox/client.go) ## Connection strategies `ConnectionStrategy` is the seam between the SDK and the cluster topology. Each strategy returns a `baseURL`; the connector then composes requests by appending an endpoint and setting the routing headers. ```text +---------------------+ Options.APIURL -> | DirectStrategy | returns Options.APIURL verbatim +---------------------+ Options.GatewayName -> | gatewayStrategy | watches Gateway.status.addresses[0].value +-----------------+ builds http(s):// after validation default -> | tunnelStrategy | SPDY port-forward to a router endpoint pod +---------------------+ base URL = http://127.0.0.1: ``` `New` picks one in `clients/go/sandbox/sandbox.go:83-109`: `APIURL` wins (advanced/test mode), then `GatewayName` (production routing), otherwise port-forward (developer mode). After `Connect`, the connector logs the discovered URL with a mode label so logs unambiguously identify which path was taken. Sources: [clients/go/sandbox/sandbox.go:83-126](clients/go/sandbox/sandbox.go), [clients/go/sandbox/strategy.go:20-32](clients/go/sandbox/strategy.go), [clients/go/sandbox/connector.go:128-147](clients/go/sandbox/connector.go) ### DirectStrategy The simplest case: a pre-configured URL passed in `Options.APIURL`. `Options.validate` enforces an `http`/`https` scheme and a non-empty host. Useful for tests against `httptest` servers and for environments that already expose a reachable router URL. Sources: [clients/go/sandbox/strategy.go:26-31](clients/go/sandbox/strategy.go), [clients/go/sandbox/options.go:242-253](clients/go/sandbox/options.go) ### gatewayStrategy `gatewayStrategy` lists then watches a Gateway resource (`gateway.networking.k8s.io/v1`, plural `gateways`) by name within `GatewayNamespace`. It loops list→watch with exponential backoff capped at 5 s and re-lists with a cleared `ResourceVersion` after a watch closes. `extractGatewayAddress` reads `status.addresses[0].value` and validates it as either an IP or a hostname matching `[a-zA-Z0-9.-]` (no empty labels, no leading/trailing dot or dash). Addresses containing `/`, `?`, `#`, or `@` are rejected to prevent SSRF via a compromised Gateway. IPv6 addresses are wrapped in brackets in `formatURL`. Watch events of type `Deleted` return `ErrGatewayDeleted`. Sources: [clients/go/sandbox/gateway.go:33-128](clients/go/sandbox/gateway.go), [clients/go/sandbox/gateway.go:168-232](clients/go/sandbox/gateway.go) ### tunnelStrategy The developer-mode path. `tunnelStrategy.Connect` resolves a ready `sandbox-router-svc` endpoint pod through `EndpointSlices` (label selector `kubernetes.io/service-name=sandbox-router-svc`), opens an SPDY round-tripper using `spdy.RoundTripperFor`, wraps it in a `trackingDialer` (so a stop-during-dial can force-close the connection), and asks `client-go`'s `portforward.New` to forward `0:8080`. The local port is read back via `fw.GetPorts()` and turned into `http://127.0.0.1:`. A background `monitorPortForward` goroutine waits on either the port-forward error channel or the stop channel. When the port-forward dies unexpectedly, it calls `connector.SetLastError` with `ErrPortForwardDied` plus up to 256 bytes of stderr; the connector then surfaces that error on the next `SendRequest` instead of a bare `ErrNotReady`. The stderr buffer is a `syncBuffer` capped at 64 KB to bound memory under chatty failures. ```mermaid sequenceDiagram participant U as Sandbox.Open participant T as tunnelStrategy participant K as Kubernetes API participant PF as portforward.ForwardPorts participant C as connector U->>T: Connect(ctx) T->>K: List EndpointSlices for sandbox-router-svc K-->>T: pod = first ready endpoint T->>K: POST .../pods//portforward (SPDY upgrade) T->>PF: New(trackingDialer, [0:8080], stop, ready) PF->>K: Stream port-forward PF-->>T: readyChan closed T->>T: fw.GetPorts() → local= T-->>U: "http://127.0.0.1:" T->>T: go monitorPortForward(...) Note over PF,C: tunnel later dies PF-->>T: errChan ← err T->>C: SetLastError(ErrPortForwardDied: err) ``` Sources: [clients/go/sandbox/tunnel.go:109-256](clients/go/sandbox/tunnel.go), [clients/go/sandbox/tunnel.go:261-335](clients/go/sandbox/tunnel.go), [clients/go/sandbox/tunnel.go:38-70](clients/go/sandbox/tunnel.go) ## connector: HTTP transport and retries `connector` owns the `*http.Client`, the discovered base URL, retry state, and the routing headers attached to every request. It is constructed once per `Sandbox` and shared by both `Commands` and `Files`. The default transport sets `DialContext` and `ResponseHeaderTimeout` to `PerAttemptTimeout`, 100 idle conns total, 10 per host, a 90 s idle timeout, and a 10 s TLS handshake timeout. A caller-supplied `HTTPTransport` bypasses this; in that case `ownsTransport` is false and `Close` will not call `CloseIdleConnections` on it. `SendRequest` is the single retry engine: 1. If the caller's ctx has no deadline, it wraps it with `RequestTimeout`. 2. A request ID is generated and stamped onto the current span as `sandbox.request_id`. 3. Up to `maxAttempts` (default 6), it: reads `baseURL`/`sandboxID`/`namespace`/`serverPort` under `c.mu`, builds the URL by trimming slashes, attaches headers (`X-Sandbox-ID`, `X-Sandbox-Namespace`, `X-Sandbox-Port`, `X-Request-ID`, `Content-Type`, W3C trace context), and runs `c.httpClient.Do` under a per-attempt `time.AfterFunc(PerAttemptTimeout, attemptCancel)`. 4. Retries on transport error and on `500/502/503/504`. Non-retryable status codes are returned as `HTTPError`. Retries with non-`io.Seeker` bodies after the first attempt return early with a clear error. 5. Backoff is `baseBackoff * 2^(attempt-1)` (500 ms → 8 s cap) plus uniform jitter in ±25 %. `backoffScale` is a test hook that compresses sleeps to milliseconds in unit tests. 6. The successful response body is wrapped in `cancelOnClose` so closing the body cancels both the per-attempt context and the request-wide timeout, preventing leaked goroutines on long-lived bodies. Two reliability invariants are encoded inline: - The "per-attempt timer fires between `Do` returning and `Stop`" race is detected by re-checking `attemptCtx.Err()`. If it fires after a successful `Do`, the response body is unusable, so the body is drained, closed, and the attempt is retried. - Tunnel death is surfaced via `c.lastError` (set by `tunnelStrategy.monitorPortForward`). When `baseURL == ""` and `lastError != nil`, requests fail with `ErrNotReady: ` instead of a generic message. Both the success and failure paths drain up to `maxDrainBytes` (4 KiB) before closing — this is what allows `net/http`'s transport to reuse the underlying TCP connection. | Constant | Value | Where | |---|---|---| | `maxAttempts` | 6 | retry cap | | `baseBackoff` | 500 ms | first non-zero sleep | | `maxBackoff` | 8 s | per-attempt sleep cap | | `maxDrainBytes` | 4 KiB | body drain for keepalive | | Retryable codes | 500, 502, 503, 504 | `retryableStatusCodes` | Sources: [clients/go/sandbox/connector.go:37-119](clients/go/sandbox/connector.go), [clients/go/sandbox/connector.go:129-184](clients/go/sandbox/connector.go), [clients/go/sandbox/connector.go:201-377](clients/go/sandbox/connector.go), [clients/go/sandbox/types.go:33-37](clients/go/sandbox/types.go) ## Commands sub-object `Commands.Run` POSTs `{"command": ""}` to `execute`, decodes the JSON response into `ExecutionResult{Stdout, Stderr, ExitCode}`, and bounds the decode at `maxExecutionResponseSize = 16 MB`. If decoding hits the limit (`lr.N <= 0`), the error is wrapped as `ErrResponseTooLarge` rather than a generic JSON error. Because execution is non-idempotent, the default is a single attempt. Callers that know their command is safe to retry opt in via `WithMaxAttempts(n)`: ```go // clients/go/sandbox/commands.go:51 result, err := s.Run(ctx, "cat /etc/hostname", sandbox.WithMaxAttempts(6)) ``` Per-call options are applied through `applyCallOpts`, which derives an optional `context.WithTimeout` from `WithTimeout(d)` and forwards `maxAttempts`. A `WithTimeout` of zero leaves the caller's deadline (or the connector's `RequestTimeout`) in charge. Sources: [clients/go/sandbox/commands.go:29-97](clients/go/sandbox/commands.go), [clients/go/sandbox/types.go:68-95](clients/go/sandbox/types.go), [clients/go/sandbox/files.go:59-69](clients/go/sandbox/files.go) ## Files sub-object `Files` exposes `Write`, `Read`, `List`, and `Exists`. The defaults differ from `Commands` because file operations on the server are idempotent: `maxAttempts` falls through to the connector default of 6. | Method | HTTP | Endpoint | Notable behavior | |---|---|---|---| | `Write` | POST | `upload` (multipart) | Rejects content over `MaxUploadSize` *before* I/O; rejects non-plain filenames (no separators, no `.`/`..`) | | `Read` | GET | `download/` | Limits body to `MaxDownloadSize+1`; returns oversize error if exceeded | | `List` | GET | `list/` | 8 MB JSON cap; filters out entries that are neither `file` nor `directory` | | `Exists` | GET | `exists/` | 8 MB JSON cap; returns `{exists: bool}` | Path encoding goes through `percentEncode`, which encodes everything outside RFC 3986 unreserved (`A-Za-z0-9-_.~`) — including `/`. The server therefore receives the full path as a single opaque path segment, eliminating any router-level path traversal. `Write` buffers the entire upload in memory as a `multipart` body. This is by design: a `*bytes.Reader` is an `io.Seeker`, so the connector's retry loop can `Seek(0, io.SeekStart)` before re-sending. A streaming body would have made the first failure terminal. Sources: [clients/go/sandbox/files.go:32-69](clients/go/sandbox/files.go), [clients/go/sandbox/files.go:90-277](clients/go/sandbox/files.go), [clients/go/sandbox/connector.go:241-253](clients/go/sandbox/connector.go), [clients/go/sandbox/types.go:122-143](clients/go/sandbox/types.go) ## Options and validation `Options` is the single configuration struct. `setDefaults` fills in safe defaults — including a `funcr.New` stderr logger unless `Quiet` is set — and `validate` enforces them. `TemplateName` is required and validated as a DNS subdomain. `Namespace` and `GatewayNamespace` are validated as DNS labels (≤63, no dots). All durations and size limits must be strictly positive. | Field | Default | Purpose | |---|---|---| | `TemplateName` | required | `SandboxTemplate` to clone | | `Namespace` | `default` | where to create the `SandboxClaim` | | `GatewayName` | — | enables `gatewayStrategy` | | `GatewayScheme` | `http` | scheme used to build URL from address | | `APIURL` | — | enables `DirectStrategy`; wins over Gateway | | `ServerPort` | 8888 | value sent as `X-Sandbox-Port` | | `SandboxReadyTimeout` | 180 s | budget for resolve + ready | | `GatewayReadyTimeout` | 180 s | per-`Gateway` discovery | | `PortForwardReadyTimeout` | 30 s | `readyChan` wait + SPDY HTTP client timeout | | `CleanupTimeout` | 30 s | detached drain + claim delete (Close uses 2× for sem) | | `RequestTimeout` | 180 s | wraps caller ctx when deadline-less | | `PerAttemptTimeout` | 60 s | per HTTP attempt header timeout | | `MaxDownloadSize` | 256 MB | cap for `Read` | | `MaxUploadSize` | 256 MB | cap for `Write` | | `HTTPTransport` | — | override transport (skips idle-conn close on `Close`) | | `TraceServiceName` | `sandbox-client` | OTel instrumentation scope + resource attribute | | `TracerProvider` | `otel.GetTracerProvider()` | source of spans | Sources: [clients/go/sandbox/options.go:30-194](clients/go/sandbox/options.go), [clients/go/sandbox/options.go:196-300](clients/go/sandbox/options.go) ## Tracing Tracing is wired through every layer. `newTracer` picks `Options.TracerProvider` (or the global), turns `TraceServiceName` into an OTel scope name (with `-` → `_`), and the same `tracer` is shared by the sandbox, commands, files, and strategies. A long-lived lifecycle span (`.lifecycle`) is started by `Open` and ended by `Close`/`Disconnect`/failed `Open`. Each operation calls `withLifecycleSpan` so that operation spans (`.run`, `.read`, `.upload` and so on) are children of the lifecycle span even if the caller's context has changed. W3C trace context is propagated outward in two places: `connector.SendRequest` injects it into HTTP headers per attempt, and `K8sHelper.createClaim` injects it into the `opentelemetry.io/trace-context` annotation on the new `SandboxClaim` so the controller can continue the trace server-side. `recordError` is the consistent shape for failures: it both records the error and sets span status to `codes.Error`. The package also exposes attribute keys in the `sandbox.*` namespace (`sandbox.claim.name`, `sandbox.command`, `sandbox.exit_code`, `sandbox.file.*`, `sandbox.gateway.*`, `sandbox.request_id`) and a convenience `NewTracerProvider(ctx, serviceName)` that returns an OTLP/gRPC batched provider reading `OTEL_EXPORTER_OTLP_ENDPOINT`. Sources: [clients/go/sandbox/tracing.go:34-120](clients/go/sandbox/tracing.go), [clients/go/sandbox/sandbox.go:210-233](clients/go/sandbox/sandbox.go), [clients/go/sandbox/k8s.go:128-157](clients/go/sandbox/k8s.go), [clients/go/sandbox/connector.go:217-272](clients/go/sandbox/connector.go) ## Error model Errors are sentinel values exported from `types.go`, optionally wrapped in `HTTPError` for non-OK server responses. `HTTPError.Error` truncates server bodies at 256 bytes before printing, and `maxErrorBodySize` (512) bounds how much body is read into the error chain. Sandbox-scoped errors prefix `sandbox[/]:` so a multi-sandbox log is greppable by claim. | Sentinel | Raised by | |---|---| | `ErrNotReady` | `connector.SendRequest` when no base URL | | `ErrTimeout` | `gatewayStrategy`, `K8sHelper` watches | | `ErrClaimFailed` | `K8sHelper.createClaim` | | `ErrPortForwardDied` | `tunnelStrategy.monitorPortForward` | | `ErrAlreadyOpen` | `Sandbox.Open` when already connected | | `ErrOrphanedClaim` | `Sandbox.Open`/`reconnect` when claim exists but verification fails | | `ErrRetriesExhausted` | `connector.SendRequest` after `maxAttempts` | | `ErrSandboxDeleted` | claim watch sees `Deleted` | | `ErrGatewayDeleted` | gateway watch sees `Deleted` | | `ErrResponseTooLarge` | `Commands.Run` decode bounded by 16 MB | Sources: [clients/go/sandbox/types.go:39-66](clients/go/sandbox/types.go), [clients/go/sandbox/connector.go:222-355](clients/go/sandbox/connector.go), [clients/go/sandbox/commands.go:83-92](clients/go/sandbox/commands.go) ## Summary `clients/go/sandbox` is intentionally narrow: `Sandbox` owns lifecycle and locking, `Client` owns multi-sandbox bookkeeping, the `connector` owns transport and retries, and three pluggable `ConnectionStrategy` implementations isolate how the router URL is discovered. The same `tracer` flows from `Options` through every layer, so a single OTel span tree covers claim creation, gateway/tunnel discovery, the lifecycle window, and every individual command or file operation.