# Observability and debugging

> Operations: Backend tracing initialization, Datadog ECS sidecars, browser RUM and logs, sourcemaps, local debug signals, and iOS WebView freeze diagnosis.

- Repository: macro-inc/macro
- GitHub: https://github.com/macro-inc/macro
- Human docs: https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e
- Complete Markdown: https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/llms-full.txt

## Source Files

- `rust/cloud-storage/macro_entrypoint/src/lib.rs`
- `infra/packages/resources/src/resources/datadog.ts`
- `js/app/packages/observability/src/index.ts`
- `js/app/packages/app/index.tsx`
- `.github/workflows/deploy-web-app.yml`
- `js/app/AGENTS.md`

---

---
title: "Observability and debugging"
description: "Operations: Backend tracing initialization, Datadog ECS sidecars, browser RUM and logs, sourcemaps, local debug signals, and iOS WebView freeze diagnosis."
---

Macro’s observability surface is split between Rust entrypoint initialization, Pulumi-managed Datadog sidecars for ECS/Fargate services, the `@observability` browser package, and deployment-time sourcemap upload for the web app.

## Runtime map

```text
Rust service
  MacroEntrypoint::init()
    ├─ local: pretty logs or hierarchical tracing tree
    └─ dev/prod: JSON logs + OpenTelemetry OTLP -> 127.0.0.1:4317

ECS task
  service container
    ├─ FireLens log driver -> log_router -> Datadog logs intake
    └─ OTel spans -> datadog-agent sidecar -> Datadog APM

Web app
  app/index.tsx idle import("@observability")
    ├─ Datadog RUM
    ├─ Datadog browser logs
    └─ sourcemaps uploaded by GitHub Actions after deploy
```

## Backend tracing initialization

Rust binaries should initialize tracing through:

```rust
macro_entrypoint::MacroEntrypoint::default().init();
```

`MacroEntrypoint::default()` loads `.env` if present and resolves `macro_env::Environment` from `ENVIRONMENT`, falling back to production when the variable is missing or invalid. Valid values are:

| `ENVIRONMENT` | Runtime enum |
| --- | --- |
| `prod` | `Environment::Production` |
| `dev` | `Environment::Develop` |
| `local` | `Environment::Local` |

### Local behavior

For `Environment::Local`, the default subscriber uses `tracing_subscriber::fmt()` with:

| Setting | Value |
| --- | --- |
| ANSI | enabled |
| filter | `EnvFilter::from_default_env()` |
| file and line | enabled |
| formatter | pretty |
| OpenTelemetry exporter | disabled |

Set `RUST_LOG` to control local verbosity, for example:

```sh
RUST_LOG=document_storage_service=trace,tower_http=debug
```

For tree-shaped local output, build the entrypoint with `tree_tracing`:

```rust
macro_entrypoint::MacroEntrypoint::new(macro_env::Environment::Local)
  .local()
  .tree_tracing(Some(4))
  .build()
  .init();
```

### Develop and production behavior

For `Environment::Develop` and `Environment::Production`, `MacroEntrypoint::init()`:

- installs `tracing_panic::panic_hook`;
- creates an OpenTelemetry OTLP gRPC span exporter pointed at `http://127.0.0.1:4317`;
- sets OpenTelemetry resource attributes from:
  - `DD_SERVICE`, defaulting to `unknown-service`;
  - `DD_ENV`, defaulting to `unknown`;
- attaches a `tracing_opentelemetry` layer;
- emits JSON logs with current span, flattened event fields, file, and line number;
- injects Datadog correlation fields `dd.trace_id` and `dd.span_id` into each valid span-backed JSON log event.

<Note>
`InitializedEntrypoint::shutdown()` exists to flush the OpenTelemetry tracer provider before process exit. Retain the returned value and call `shutdown()` for short-lived jobs where trace flushing matters.
</Note>

## Datadog ECS sidecars

The shared Datadog containers live in Pulumi resources and are added to ECS tasks by the service components unless a component explicitly opts out with `noDatadog`.

| Container | Image | Purpose |
| --- | --- | --- |
| `log_router` | `amazon/aws-for-fluent-bit:latest` | FireLens log routing to Datadog logs intake |
| `datadog-agent` | `public.ecr.aws/datadog/agent:latest` | APM/trace collection and OTLP receiver |

### `log_router`

The FireLens sidecar is essential, reserves 50 MiB, enables ECS metadata, and uses Fluent Bit’s bundled `/fluent-bit/configs/parse-json.conf`.

Environment:

| Variable | Value |
| --- | --- |
| `ECS_FARGATE` | `true` |
| `DD_API_KEY` | AWS Secrets Manager secret `datadog-api-key` |
| `DD_ENV` | Pulumi stack name |

Service containers route logs with `awsfirelens` to `http-intake.logs.us5.datadoghq.com` and set Datadog options such as `dd_service`, `dd_source=fargate`, `dd_tags`, and `provider=ecs`.

### `datadog-agent`

The Datadog agent sidecar listens for OTLP gRPC spans on port `4317`.

| Variable | Value |
| --- | --- |
| `DD_SITE` | `us5.datadoghq.com` |
| `DD_APM_ENABLED` | `true` |
| `DD_OTLP_CONFIG_RECEIVER_PROTOCOLS_GRPC_ENDPOINT` | `0.0.0.0:4317` |
| `DD_APM_SAMPLE_RATE` | `0.1` in `prod`, `1.0` otherwise |
| `DD_APM_MAX_TPS` | `100` |
| `DD_APM_RECEIVER_SOCKET` | empty string |

The empty `DD_APM_RECEIVER_SOCKET` disables socket buffering in the agent configuration used by this repo.

## Browser RUM and logs

The web app initializes browser observability lazily from `packages/app/index.tsx`. During non-HMR runs, it schedules an idle import of `@observability` and calls:

```ts
Observability.init(import.meta.env.__APP_VERSION__);
```

During `vite dev`, `import.meta.hot` is present, so Datadog browser observability is not injected.

### Datadog browser configuration

`@observability` configures both RUM and browser logs with:

| Field | Value |
| --- | --- |
| `service` | `web-app` |
| `site` | `us5.datadoghq.com` |
| `env` | `prod` when `import.meta.env.MODE === "production"`, otherwise `dev` |
| `version` | `import.meta.env.__APP_VERSION__` by default |
| RUM `sessionSampleRate` | `100` |
| RUM `sessionReplaySampleRate` | `0` |
| RUM `trackViewsManually` | `true` |
| RUM `defaultPrivacyLevel` | `mask` |
| logs `telemetrySampleRate` | `0` |

RUM tracing uses `tracecontext` propagation. In production, tracing is allowed only for selected Macro service hosts: auth, cognition, document storage, email, and notification. In non-production modes, all configured `SERVER_HOSTS` are allowed.

### Event filtering

The browser package drops known noisy events before they leave the client:

| Source | Filter |
| --- | --- |
| RUM resource events | non-200 `unfurl-service` resources |
| RUM errors | `ResizeObserver loop completed with undelivered notifications` |
| browser logs | messages containing `unfurl-service` |
| browser logs | messages containing the same ResizeObserver notification |

### Logging API

Use `@observability` or `@observability/logger` for client-side application logs:

```ts
import { logger } from '@observability';

logger.log('Uploaded file', { documentId });
logger.warn('Retrying request', { attempt });
logger.error('Failed to save draft', { cause: error, draftId });
```

When Datadog is unavailable, not initialized, or the app is running under HMR, the logger falls back to `console.log`, `console.warn`, or `console.error`.

### Routing and actions

`useObserveRouting()` starts manual RUM views from the current route. It derives the view name from the first path segment after `/app`; for `/app/component/:name`, it uses the component name segment instead.

It also records a RUM action named `split changed` when the joined path segments change.

For explicit user actions:

```ts
import { startAction } from '@observability';

startAction('opened command palette', { source: 'hotkey' });
```

## Sourcemaps and release versions

The Vite app build always enables sourcemaps. The app version is computed from `packages/app/package.json` plus the current short Git SHA:

```text
<package-version>+<short-sha>
```

After `vite build`, `postbuild:semver` writes the same value to:

```text
js/app/packages/app/dist/semver.txt
```

Deployment workflows pass Datadog browser credentials into the build, deploy the web app with Pulumi, then upload sourcemaps from `js/app/packages/app`:

```sh
bun run ddupload:dev
bun run ddupload:prod
```

The upload scripts call `datadog-ci sourcemaps upload ./dist` with:

| Environment | `--service` | `--release-version` | `--minified-path-prefix` |
| --- | --- | --- | --- |
| dev | `web-app` | `$(cat ./dist/semver.txt)` | `https://dev.macro.com/app` |
| prod | `web-app` | `$(cat ./dist/semver.txt)` | `https://macro.com/app` |

Production web-app infrastructure excludes `*.js.map` files from the S3 sync, so production sourcemaps are uploaded to Datadog rather than served with the app assets.

## Local debug signals

### Web app startup signals

On startup, the app logs the resolved version:

```text
App Version <version>
```

The root document element also receives runtime attributes useful for local debugging and CSS/state inspection:

| Attribute | Value source |
| --- | --- |
| `data-platform` | `getPlatform()` |
| `data-touch-device` | `isTouchDevice()` |
| `data-modality` | last captured `keydown`, `mousedown`, or `touchstart` |

In development mode, rendering is wrapped in a Solid `ErrorBoundary` that shows `FatalError`. In non-development builds, the root renders directly.

### Dynamic import deployment mismatch

The app registers a `vite:preloadError` listener outside HMR. If a dynamic module load fails after a new version is deployed, the browser shows:

```text
Please refresh page to update app to new version
```

Use this as the expected signal for stale chunks after deployment.

## iOS WebView freeze diagnosis

On iOS WKWebView, eagerly constructing an ES module Web Worker from a `tauri://` custom URL can deadlock the WebContent process. The rule in this repository is:

<Warning>
Do not call `new Worker(...)` at module-load time in code that runs on iOS. Construct workers lazily on first use. The `import Worker from "./worker?worker"` import is safe; instantiating it is the dangerous operation.
</Warning>

A lazy singleton can use a proxy:

```ts
export const svc = new Proxy({} as Service, {
  get: (_, p, r) => Reflect.get(Service.getInstance(), p, r),
});
```

### Symptom signature

The app loads HTML, starts initial JavaScript, then JavaScript silently stops. Safari Web Inspector may attach but show no useful page state.

### Diagnosis procedure

<Steps>
  <Step title="Reproduce on the iOS Simulator">
    Use the simulator so logs are available through macOS unified logging.

    ```sh
    cargo tauri ios dev "iPhone 15"
    ```
  </Step>

  <Step title="Stream Macro process logs">
    Use the full `/usr/bin/log` path because `zsh` has a `log` builtin.

    ```sh
    /usr/bin/log stream --predicate 'process == "macro"' --info --debug --style compact
    ```
  </Step>

  <Step title="Find the last meaningful URL request">
    If protocol-handler request logging is enabled, the last `tauri://` request before silence is usually the file being loaded when the freeze occurred.

    ```sh
    tail -200 <logfile> | grep -vE 'tauri:// request|tauri_protocol.rs|^\s*\\134'
    ```
  </Step>

  <Step title="Check whether WebContent is parked">
    Get the WebContent PID from a `[com.apple.WebKit:...] [...PID=N...]` log line, then inspect CPU.

    ```sh
    ps -o pid,pcpu,comm -p <pid>
    ```

    `0.0%` CPU with a still-alive process points to a deadlock or parked thread, not a hot loop.
  </Step>

  <Step title="Sample the WebContent process">
    ```sh
    sample <webcontent_pid> 3 -file /tmp/sample.txt
    ```

    Look for `WebCore: Worker` stacks containing:

    ```text
    WorkerOrWorkletScriptController::loadModuleSynchronously
      → WorkerDedicatedRunLoop::runInMode
        → Condition::waitUntilUnchecked
    ```
  </Step>
</Steps>

If the last request was a `*-worker.js?worker_file&type=module` URL and a worker thread is parked in `loadModuleSynchronously`, trace the corresponding `new Worker(...)` call and move construction behind a lazy path.

Do not treat these as primary causes unless the stack confirms them: `NSKeyedArchiver` main-thread warnings, IPC throttling warnings, or failed bundle-updater requests to `localhost:3001`.

## Related pages

- Web app deployment and sourcemap upload
- Backend service deployment on ECS/Fargate
- iOS/Tauri local debugging