# OpenTelemetry Tracing Setup > Provider-neutral OTLP tracing wiring used by both the controller binary and the SDKs; instrumenter interface and no-op fallback. - Repository: kubernetes-sigs/agent-sandbox - GitHub: https://github.com/kubernetes-sigs/agent-sandbox - Human wiki: https://grok-wiki.com/public/wiki/kubernetes-sigs-agent-sandbox-c3f2597a654a - Complete Markdown: https://grok-wiki.com/public/wiki/kubernetes-sigs-agent-sandbox-c3f2597a654a/llms-full.txt ## Source Files - `internal/metrics/tracing.go` - `clients/go/sandbox/tracing.go` - `clients/python/agentic-sandbox-client/k8s_agent_sandbox/trace_manager.py` - `clients/python/agentic-sandbox-client/otel-collector-config.yaml.example` ---
Relevant source files The following files were used as context for generating this wiki page: - [internal/metrics/tracing.go](internal/metrics/tracing.go) - [clients/go/sandbox/tracing.go](clients/go/sandbox/tracing.go) - [clients/python/agentic-sandbox-client/k8s_agent_sandbox/trace_manager.py](clients/python/agentic-sandbox-client/k8s_agent_sandbox/trace_manager.py) - [clients/python/agentic-sandbox-client/otel-collector-config.yaml.example](clients/python/agentic-sandbox-client/otel-collector-config.yaml.example) - [cmd/agent-sandbox-controller/main.go](cmd/agent-sandbox-controller/main.go) - [clients/go/sandbox/options.go](clients/go/sandbox/options.go) - [controllers/sandbox_controller.go](controllers/sandbox_controller.go) - [clients/go/sandbox/sandbox.go](clients/go/sandbox/sandbox.go) - [clients/go/sandbox/tracing_test.go](clients/go/sandbox/tracing_test.go) - [clients/python/agentic-sandbox-client/GCP.md](clients/python/agentic-sandbox-client/GCP.md)
# OpenTelemetry Tracing Setup The repository ships provider-neutral OpenTelemetry (OTel) tracing for three runtime surfaces: the `agent-sandbox-controller` binary, the Go SDK (`clients/go/sandbox`), and the Python SDK (`clients/python/agentic-sandbox-client`). Each surface configures its own `TracerProvider` with the OTLP/gRPC exporter, then emits spans whose lineage is stitched back across the controller/SDK boundary using a JSON-encoded W3C Trace Context written to the Kubernetes object annotation `opentelemetry.io/trace-context`. The wiring is deliberately minimal: no vendor SDKs are pulled in, the exporter endpoint is taken from the standard `OTEL_EXPORTER_OTLP_ENDPOINT` environment variable, and every surface degrades safely to a no-op when tracing is disabled or the optional dependency is missing. A user-run OTel Collector (an `otel-collector-config.yaml.example` is supplied) is the integration point for any backend (Google Cloud, Jaeger, Tempo, etc.). ## Architecture Overview The controller and SDKs participate in one logical trace per sandbox. The controller starts the root span during reconcile and persists the carrier on the `Sandbox` CR; the SDKs start a per-instance "lifecycle" span that becomes the parent for all sandbox operations (`run`, `read`, `write`, `list`, `exists`, `create_claim`, `wait_for_sandbox_ready`). ```mermaid flowchart LR subgraph Controller["agent-sandbox-controller (Go)"] Main["cmd/.../main.go
--enable-tracing"] Setup["internal/metrics
SetupOTel / NewNoOp"] Inst["Instrumenter interface
StartSpan / GetTraceContext
AddEvent / IsRecording"] Recon["controllers/sandbox_controller.go
Reconcile()"] end subgraph GoSDK["Go SDK (clients/go/sandbox)"] GoOpts["Options.TracerProvider
Options.TraceServiceName"] GoTrace["tracing.go
NewTracerProvider / newTracer
startSpan / recordError"] GoLife["sandbox.go
{svc}.lifecycle span"] end subgraph PySDK["Python SDK (k8s_agent_sandbox)"] PyCfg["SandboxTracerConfig
enable_tracing / trace_service_name"] PyTM["trace_manager.py
initialize_tracer
TracerManager
@trace_span / @async_trace_span"] end subgraph K8s["Kubernetes API"] CR["Sandbox / SandboxClaim
annotations[opentelemetry.io/trace-context]
= JSON {traceparent, tracestate}"] end Collector["OTel Collector
OTLP :4317 gRPC / :4318 HTTP
processors: batch
exporters: googlecloud / ..."] Main --> Setup --> Inst --> Recon Recon -- Inject carrier --> CR CR -- Extract carrier --> Inst GoOpts --> GoTrace --> GoLife --> CR PyCfg --> PyTM --> CR Setup -- "OTLP/gRPC
OTEL_EXPORTER_OTLP_ENDPOINT" --> Collector GoTrace -- "OTLP/gRPC" --> Collector PyTM -- "OTLP/gRPC" --> Collector ``` Sources: [internal/metrics/tracing.go:35-147](internal/metrics/tracing.go), [clients/go/sandbox/tracing.go:47-72](clients/go/sandbox/tracing.go), [clients/python/agentic-sandbox-client/k8s_agent_sandbox/trace_manager.py:114-161](clients/python/agentic-sandbox-client/k8s_agent_sandbox/trace_manager.py), [controllers/sandbox_controller.go:160-186](controllers/sandbox_controller.go) ## Controller Tracing (`internal/metrics`) ### Instrumenter Interface and No-Op Fallback `internal/metrics/tracing.go` defines a small abstraction the controller uses everywhere instead of calling OTel APIs directly. This keeps the controller compilable and runnable without an OTel backend and lets reconcilers be unit-tested without a real `TracerProvider`. ```go // internal/metrics/tracing.go type Instrumenter interface { StartSpan(ctx context.Context, obj metav1.Object, spanName string, attrs map[string]string) (context.Context, func()) GetTraceContext(ctx context.Context) string AddEvent(ctx context.Context, name string, attrs map[string]string) IsRecording(ctx context.Context) bool } ``` Two implementations satisfy it: | Type | Constructor | Behavior | | --- | --- | --- | | `noopInstrumenter` | `metrics.NewNoOp()` | Returns the input context, a no-op closer, empty trace context, and `IsRecording=false`. Used when `--enable-tracing` is unset. | | `otelInstrumenter` | Returned by `metrics.SetupOTel(ctx, serviceName)` | Wraps a real `trace.Tracer`, a `propagation.TextMapPropagator`, and a `logr.Logger`. | Sources: [internal/metrics/tracing.go:41-57](internal/metrics/tracing.go) ### `SetupOTel`: Bootstrapping the SDK `SetupOTel` is the controller's single entry point for initializing the OTel SDK. It constructs an OTLP/gRPC exporter (which honors `OTEL_EXPORTER_OTLP_ENDPOINT` and `OTEL_EXPORTER_OTLP_INSECURE`), wires it into a batching `TracerProvider`, installs that provider and a W3C `TraceContext` propagator as the OTel globals, and returns the `Instrumenter` together with a shutdown closer. ```go // internal/metrics/tracing.go exporter, err := otlptracegrpc.New(ctx) ... tp := sdktrace.NewTracerProvider( sdktrace.WithBatcher(exporter), sdktrace.WithResource(resource.NewWithAttributes( semconv.SchemaURL, semconv.ServiceNameKey.String(serviceName), )), ) otel.SetTracerProvider(tp) otel.SetTextMapPropagator(propagation.TraceContext{}) ``` Notes from the implementation: - Only the W3C `TraceContext` propagator is installed; baggage is intentionally excluded. - The resource is built with `semconv/v1.38.0` and a single `service.name` attribute. - The instrumentation scope name is `agent-sandbox-controller`. Sources: [internal/metrics/tracing.go:123-147](internal/metrics/tracing.go) ### Controller Binary Wiring `cmd/agent-sandbox-controller/main.go` registers a `--enable-tracing` flag. When unset, the controller runs with `NewNoOp()`; when set, `SetupOTel` is invoked under a 10-second initialization timeout, and the shutdown closer is deferred for graceful flush. ```go // cmd/agent-sandbox-controller/main.go flag.BoolVar(&enableTracing, "enable-tracing", false, "Enable OpenTelemetry tracing via OTLP.") ... var instrumenter = asmetrics.NewNoOp() if enableTracing { initCtx, cancel := context.WithTimeout(ctx, 10*time.Second) defer cancel() instrumenter, cleanup, err = asmetrics.SetupOTel(initCtx, "agent-sandbox-controller") if err != nil { setupLog.Error(err, "unable to initialize tracing") os.Exit(1) } defer cleanup() } ``` The `Instrumenter` is then handed to reconcilers (for example, `controllers.SandboxReconciler.Tracer`). Sources: [cmd/agent-sandbox-controller/main.go:57-168](cmd/agent-sandbox-controller/main.go), [controllers/sandbox_controller.go:126-186](controllers/sandbox_controller.go) ### Trace-Context Propagation Across Reconcile Boundaries `otelInstrumenter.StartSpan` does two things that make cross-binary tracing work: 1. If `obj` has the `opentelemetry.io/trace-context` annotation, it JSON-decodes the carrier and extracts a parent context using the global W3C propagator before starting the span. 2. It collapses callers' `map[string]string` attrs into `attribute.KeyValue` slices via `trace.WithAttributes(...)`. `GetTraceContext` does the inverse: it injects the active span into an empty `propagation.MapCarrier` and JSON-marshals it. `SandboxReconciler.Reconcile` calls this immediately after starting `ReconcileSandbox` and patches the resulting JSON onto the `Sandbox`'s annotations when missing — that single write becomes the carrier that SDK clients later extract. ```go // internal/metrics/tracing.go const TraceContextAnnotation = "opentelemetry.io/trace-context" // Example: {"traceparent": "00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01"} ``` Sources: [internal/metrics/tracing.go:36-105](internal/metrics/tracing.go), [controllers/sandbox_controller.go:160-186](controllers/sandbox_controller.go) ## Go SDK Tracing (`clients/go/sandbox`) The Go SDK does not call `SetupOTel`. It is library code, so it never installs OTel globals; instead, it asks the caller for a `TracerProvider` and exposes a convenience constructor (`NewTracerProvider`) for callers that just want OTLP/gRPC defaults. ### Options Surface ```go // clients/go/sandbox/options.go // TraceServiceName is the OpenTelemetry service name used for the tracer's // instrumentation scope and the resource's service.name attribute. // Default: "sandbox-client". TraceServiceName string // TracerProvider sets the OpenTelemetry TracerProvider for span creation. // If nil, falls back to otel.GetTracerProvider (noop by default). TracerProvider trace.TracerProvider ``` `setDefaults` fills `TraceServiceName` with `"sandbox-client"` when unset. `newTracer(opts)` (in `tracing.go`) selects `opts.TracerProvider` if non-nil, falls back to `otel.GetTracerProvider()` (a no-op unless the application installed one), and converts the service name into the instrumentation scope by replacing dashes with underscores. | Option | Default | Effect | | --- | --- | --- | | `TraceServiceName` | `"sandbox-client"` | Span name prefix (e.g. `sandbox-client.run`), resource `service.name`, and underscored scope. | | `TracerProvider` | `nil` → `otel.GetTracerProvider()` | If both are noop, all span machinery becomes free. | Sources: [clients/go/sandbox/options.go:131-180](clients/go/sandbox/options.go), [clients/go/sandbox/tracing.go:107-120](clients/go/sandbox/tracing.go) ### `NewTracerProvider`: Optional OTLP Helper For applications that don't already have a global `TracerProvider`, the SDK ships `NewTracerProvider(ctx, serviceName)` that constructs an OTLP/gRPC exporter, merges a default resource with a `service.name` attribute, and returns a batching `*sdktrace.TracerProvider`. Ownership of `Shutdown` is left to the caller. ```go // clients/go/sandbox/tracing.go // NewTracerProvider creates a TracerProvider with an OTLP/gRPC exporter. // The endpoint is read from OTEL_EXPORTER_OTLP_ENDPOINT (default: localhost:4317). // serviceName becomes the service.name resource attribute. // The caller owns the returned provider and must call Shutdown when done. ``` Sources: [clients/go/sandbox/tracing.go:47-72](clients/go/sandbox/tracing.go) ### Lifecycle Spans, Error Recording, and Carrier Injection `Sandbox.Open` starts a `{svcName}.lifecycle` span as the per-instance root. Each operation (`Run`, `Read`, `Write`, `List`, `Exists`, `reconnect`, …) is parented to it via `startSpan(ctx, tracer, svcName, "", attrs...)`, which formats span names as `{svcName}.{operation}`. When an operation fails, `recordError(span, err)` calls `span.RecordError(err)` and `span.SetStatus(codes.Error, err.Error())`. When the SDK needs to hand the lineage to the controller (e.g., during claim creation), `traceContextJSON(ctx)` injects the active span into a `propagation.MapCarrier` and JSON-marshals it for the `opentelemetry.io/trace-context` annotation. The function returns `""` when no active span is present, so callers won't write a meaningless annotation when tracing is off. The semantic attribute keys are centralized at the top of `tracing.go`: | Constant | String key | Used by | | --- | --- | --- | | `AttrClaimName` | `sandbox.claim.name` | `create_claim` | | `AttrCommand` | `sandbox.command` | `run` | | `AttrExitCode` | `sandbox.exit_code` | `run` | | `AttrFilePath` | `sandbox.file.path` | `read`, `write`, `list`, `exists` | | `AttrFileSize` | `sandbox.file.size` | `read`, `write` | | `AttrFileCount` | `sandbox.file.count` | `list` | | `AttrFileExists` | `sandbox.file.exists` | `exists` | | `AttrGatewayName` / `AttrGatewayNamespace` | `sandbox.gateway.{name,namespace}` | Gateway connection strategy | | `AttrRequestID` | `sandbox.request_id` | HTTP requests | These keys (and the lifecycle-as-parent invariant) are asserted in `clients/go/sandbox/tracing_test.go`, which uses `tracetest.NewInMemoryExporter` to verify that all `test-svc.*` spans share the lifecycle span's `SpanID` as parent and that the captured `SandboxClaim` carries a `traceparent` annotation. Sources: [clients/go/sandbox/tracing.go:33-105](clients/go/sandbox/tracing.go), [clients/go/sandbox/sandbox.go:80-216](clients/go/sandbox/sandbox.go), [clients/go/sandbox/tracing_test.go:155-302](clients/go/sandbox/tracing_test.go) ## Python SDK Tracing (`k8s_agent_sandbox.trace_manager`) The Python SDK mirrors the Go SDK's shape but adds two Python-specific concerns: OpenTelemetry is an *optional* dependency (graceful import failure must keep the SDK usable), and a process-wide `TracerProvider` is needed so that multiple `SandboxClient` instances share one exporter. ### Optional-Dependency Mock Layer The module attempts the OTel imports inside a `try`; on `ImportError` it sets `OPENTELEMETRY_AVAILABLE = False` and defines `MockSpan`, `MockTracer`, `TraceStub`, `ContextStub`, and a mock `TraceContextTextMapPropagator`. The mock `Tracer.start_as_current_span` returns `contextlib.nullcontext()`, so existing `with tracer.start_as_current_span(...)` callsites in the SDK keep working without modification. ```python # clients/python/agentic-sandbox-client/k8s_agent_sandbox/trace_manager.py try: from opentelemetry import trace, context from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter ... OPENTELEMETRY_AVAILABLE = True except ImportError: OPENTELEMETRY_AVAILABLE = False logging.debug("OpenTelemetry not installed; using MockTracer.") class MockSpan: ... class MockTracer: def start_as_current_span(self, *a, **k): return nullcontext() ... ``` Sources: [clients/python/agentic-sandbox-client/k8s_agent_sandbox/trace_manager.py:32-107](clients/python/agentic-sandbox-client/k8s_agent_sandbox/trace_manager.py) ### `initialize_tracer`: Process-Wide Singleton `initialize_tracer(service_name)` uses double-checked locking around a module-level `_TRACER_PROVIDER`. On first call (when OTel is installed and no provider is set), it builds a `Resource(attributes={"service.name": service_name})`, a `TracerProvider`, and a `BatchSpanProcessor(OTLPSpanExporter())`, installs it as the global provider, and registers `atexit` shutdown. On subsequent calls, if the requested `service_name` differs from the one already installed, a warning is logged and the existing provider is kept — the **first** client to initialize wins for the whole process. ```python # clients/python/agentic-sandbox-client/k8s_agent_sandbox/trace_manager.py with _TRACER_PROVIDER_LOCK: if _TRACER_PROVIDER is None: resource = Resource(attributes={"service.name": service_name}) _TRACER_PROVIDER = TracerProvider(resource=resource) _TRACER_PROVIDER.add_span_processor( BatchSpanProcessor(OTLPSpanExporter()) ) trace.set_tracer_provider(_TRACER_PROVIDER) atexit.register(_TRACER_PROVIDER.shutdown) ``` Sources: [clients/python/agentic-sandbox-client/k8s_agent_sandbox/trace_manager.py:109-161](clients/python/agentic-sandbox-client/k8s_agent_sandbox/trace_manager.py) ### `TracerManager`: Per-Client Lifecycle Context Each `SandboxClient` instance owns a `TracerManager(service_name)` that: - Builds a tracer with an instrumentation scope derived from `service_name` (dashes → underscores), matching the Go SDK convention. - Holds a single `parent_span` named `{service_name}.lifecycle` and a `context_token` returned from `context.attach(...)`. - Exposes `get_trace_context_json()` which uses `TraceContextTextMapPropagator` to inject the current context into a dict carrier and `json.dumps` it for the `opentelemetry.io/trace-context` annotation (returns `""` if the carrier is empty). `create_tracer_manager(config)` is the factory used by `SandboxClient`. It short-circuits to `(None, None)` either when `config.enable_tracing` is false or when OTel is not installed; the calling code then treats `self.tracer is None` as "tracing disabled" and skips the decorators. Sources: [clients/python/agentic-sandbox-client/k8s_agent_sandbox/trace_manager.py:219-271](clients/python/agentic-sandbox-client/k8s_agent_sandbox/trace_manager.py) ### `@trace_span` / `@async_trace_span` Decorators These decorators wrap sync/async methods on objects that expose `self.tracer` and `self.trace_service_name`. The span name is composed at call time as `f"{self.trace_service_name}.{span_suffix}"`. When `self.tracer` is `None`, the wrapped function is invoked without any span — the safe path when tracing is disabled. ```python # clients/python/agentic-sandbox-client/k8s_agent_sandbox/trace_manager.py def trace_span(span_suffix): def decorator(func): @functools.wraps(func) def wrapper(self, *args, **kwargs): tracer = getattr(self, 'tracer', None) if not tracer: return func(self, *args, **kwargs) service_name = getattr(self, 'trace_service_name', 'sandbox-client') with tracer.start_as_current_span(f"{service_name}.{span_suffix}"): return func(self, *args, **kwargs) return wrapper return decorator ``` The async variant has the same shape with `async def wrapper` and `await func(...)`. Sources: [clients/python/agentic-sandbox-client/k8s_agent_sandbox/trace_manager.py:164-216](clients/python/agentic-sandbox-client/k8s_agent_sandbox/trace_manager.py) ### Client Wiring (`SandboxTracerConfig`) The Python SDK enables tracing through a Pydantic config: ```python # clients/python/agentic-sandbox-client/k8s_agent_sandbox/models.py class SandboxTracerConfig(BaseModel): enable_tracing: bool = False trace_service_name: str = "sandbox-client" ``` `SandboxClient.__init__` calls `initialize_tracer(self.tracer_config.trace_service_name)` if enabled, then `create_tracer_manager(self.tracer_config)` to obtain `(tracing_manager, tracer)`. The pair is passed through to executors, file helpers, and other subordinates, which read `self.tracer` and `self.trace_service_name` for the decorators. Sources: [clients/python/agentic-sandbox-client/k8s_agent_sandbox/sandbox_client.py:29-82](clients/python/agentic-sandbox-client/k8s_agent_sandbox/sandbox_client.py) ## Comparison Across Surfaces | Concern | Controller (`internal/metrics`) | Go SDK (`clients/go/sandbox`) | Python SDK (`k8s_agent_sandbox`) | | --- | --- | --- | --- | | Owns global `TracerProvider`? | Yes — installs via `otel.SetTracerProvider`. | No — uses caller-provided or `otel.GetTracerProvider()`. | Yes — module-level singleton with `atexit` shutdown. | | Enable switch | `--enable-tracing` CLI flag | `Options.TracerProvider != nil` | `SandboxTracerConfig.enable_tracing` | | Fallback when disabled | `metrics.NewNoOp()` Instrumenter | Global noop `TracerProvider` | `None` tracer; decorators short-circuit | | Fallback when OTel missing | N/A (build-time dependency) | N/A (build-time dependency) | `MockSpan` / `MockTracer` / stubs | | Exporter | `otlptracegrpc.New(ctx)` | `otlptracegrpc.New(ctx)` (via helper) | `OTLPSpanExporter()` with `BatchSpanProcessor` | | Endpoint source | `OTEL_EXPORTER_OTLP_ENDPOINT` env | `OTEL_EXPORTER_OTLP_ENDPOINT` env (default `localhost:4317`) | Environment, as resolved by the OTel Python SDK | | Propagator | W3C `TraceContext` only | W3C `TraceContext` only | `TraceContextTextMapPropagator` | | Annotation key | `opentelemetry.io/trace-context` | `opentelemetry.io/trace-context` | `opentelemetry.io/trace-context` | | Span name shape | `ReconcileSandbox`, `reconcilePod`, `reconcilePVCs` | `{TraceServiceName}.{op}`, e.g. `sandbox-client.run` | `{trace_service_name}.{op}`, e.g. `sandbox-client.run` | | Instrumentation scope | `"agent-sandbox-controller"` | `strings.ReplaceAll(svcName, "-", "_")` | `service_name.replace('-', '_')` | | Lifecycle root span | Per `Reconcile` invocation | `{svc}.lifecycle` per `Sandbox` instance | `{svc}.lifecycle` per `SandboxClient` | | Shutdown | `defer cleanup()` in `main` | Caller-owned `tp.Shutdown(...)` | `atexit.register(_TRACER_PROVIDER.shutdown)` | Sources: [internal/metrics/tracing.go:123-147](internal/metrics/tracing.go), [clients/go/sandbox/tracing.go:47-120](clients/go/sandbox/tracing.go), [clients/python/agentic-sandbox-client/k8s_agent_sandbox/trace_manager.py:114-271](clients/python/agentic-sandbox-client/k8s_agent_sandbox/trace_manager.py) ## End-to-End Span Flow The sequence below shows how a single trace travels from a Python or Go client through the Kubernetes API to the controller and out to a collector, using the annotation as the only out-of-band carrier. ```mermaid sequenceDiagram participant App as Client App participant SDK as SDK (Go / Python) participant K8s as Kubernetes API participant Ctrl as Controller (SandboxReconciler) participant Col as OTel Collector (OTLP/gRPC) App->>SDK: New / open SandboxClient SDK->>SDK: start "{svc}.lifecycle" span SDK->>SDK: traceContextJSON / get_trace_context_json SDK->>K8s: create SandboxClaim
annotations[opentelemetry.io/trace-context] = JSON K8s-->>Ctrl: Reconcile(Sandbox) Ctrl->>Ctrl: StartSpan(obj, "ReconcileSandbox", attrs)
extract carrier from annotation alt annotation missing Ctrl->>K8s: Patch Sandbox.annotations[trace-context] = GetTraceContext(ctx) end Ctrl->>Ctrl: AddEvent("NewPodStatusObserved", ...) Ctrl-->>Col: batched OTLP spans (controller scope) SDK->>SDK: child spans: run / read / write / list / exists SDK-->>Col: batched OTLP spans (SDK scope) App->>SDK: Close / end_lifecycle_span SDK->>SDK: span.End() + recordError (if any) ``` Sources: [controllers/sandbox_controller.go:160-186](controllers/sandbox_controller.go), [clients/go/sandbox/sandbox.go:210-233](clients/go/sandbox/sandbox.go), [clients/python/agentic-sandbox-client/k8s_agent_sandbox/trace_manager.py:237-257](clients/python/agentic-sandbox-client/k8s_agent_sandbox/trace_manager.py), [internal/metrics/tracing.go:66-105](internal/metrics/tracing.go) ## Collector Configuration The Python client ships a reference collector config that documents the expected OTLP receivers on `0.0.0.0:4317` (gRPC) and `0.0.0.0:4318` (HTTP), a batch processor, and a single `googlecloud` exporter. It is a starting point — any OTLP-compatible exporter (Jaeger, Tempo, OTLP HTTP, debug) can replace it without code changes in this repository. ```yaml # clients/python/agentic-sandbox-client/otel-collector-config.yaml.example receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 http: endpoint: 0.0.0.0:4318 processors: batch: exporters: googlecloud: project: "YOUR-GCP-PROJECT-ID" service: pipelines: traces: receivers: [otlp] processors: [batch] exporters: [googlecloud] ``` Pointing clients at a non-local collector is purely an environment-variable concern; for example, `OTEL_EXPORTER_OTLP_ENDPOINT="http://otel-collector.default.svc.cluster.local:4317"` is the in-cluster pattern called out by `clients/python/agentic-sandbox-client/GCP.md`. The provider-neutrality of this design comes from a single fact: none of the three Go/Python surfaces import a vendor SDK — they only speak OTLP, and the collector handles fan-out. Sources: [clients/python/agentic-sandbox-client/otel-collector-config.yaml.example:1-22](clients/python/agentic-sandbox-client/otel-collector-config.yaml.example), [clients/python/agentic-sandbox-client/GCP.md:33-112](clients/python/agentic-sandbox-client/GCP.md) ## Summary The repository implements OpenTelemetry tracing as three small, structurally-similar modules that all speak OTLP/gRPC to an external collector and propagate context across the controller/SDK boundary through one JSON annotation (`opentelemetry.io/trace-context`). The controller hides span emission behind a four-method `Instrumenter` interface so it can run with a no-op when tracing is disabled; the Go SDK exposes `TracerProvider`/`TraceServiceName` options and a convenience `NewTracerProvider` helper but never touches OTel globals; the Python SDK guards optional imports with a complete mock layer and protects the process-wide `TracerProvider` with a double-checked-locked singleton. The shape — provider-neutral exporter, W3C-only propagation, lifecycle-rooted spans, and annotation-based context handoff — keeps the integration portable across any OTLP-compatible backend.