Agent-readable docs
Macro Engineering Documentation
Technical documentation for the Macro monorepo: SolidJS app, Rust services, sync worker, AI tooling, infrastructure, local development, APIs, configuration, and operations.
Pages
- OverviewOverview: Public entry points, repository surfaces, runtime assumptions, and the shortest successful path through the docs.
- InstallationInstall: Required tools, Nix option, encrypted environment setup, Docker resources, LocalStack, FusionAuth, and first setup command.
- Local development quickstartQuickstart: Bring up local services with just recipes, shared Compose resources, database initialization, and expected health signals.
- Frontend quickstartQuickstart: Run the SolidJS app, choose local or remote services, understand app routing, and start Tauri development.
- Local E2E smoke testsGuide: Run deterministic Playwright and ignored Rust smoke tests against the local-only stack and shared seed fixtures.
- Monorepo mapConcept: Workspace boundaries, package managers, major runtime folders, generated documentation, and where source-of-truth manifests live.
- Runtime environments and service URLsConcept: Environment values, local versus dev versus production URL resolution, CORS rules, and frontend service selection.
- Service topologyConcept: Rust services, workers, queues, storage dependencies, ports, and deployment boundaries used by the local and cloud stacks.
- Entities and blocksConcept: Item types, document-backed and non-document blocks, aliases, load sources, nesting rules, and file type resolution.
- Split layout and navigationConcept: Route encoding, component registry entries, split history, navigation causes, popovers, and desktop versus mobile split behavior.
- Real-time document syncConcept: Sync-service worker routes, Durable Object sessions, permission tokens, Bebop messages, snapshot lifecycle, and client reconnect behavior.
- Run backend services locallyGuide: Start databases, LocalStack, FusionAuth, Rust services, optional processor profiles, and local health checks.
Complete Markdown
# Macro Engineering Documentation
> Technical documentation for the Macro monorepo: SolidJS app, Rust services, sync worker, AI tooling, infrastructure, local development, APIs, configuration, and operations.
## Context Links
- [Agent index](https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/llms.txt)
- [Human interactive docs](https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e)
- [GitHub repository](https://github.com/macro-inc/macro)
## Repository Metadata
- Repository: macro-inc/macro
- Generated: 2026-06-01T01:07:41.847Z
- Updated: 2026-06-01T02:14:57.236Z
- Runtime: Pi · Codex · gpt-5.5
- Format: Documentation
- Pages: 29
## Page Index
- 01. [Overview](https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/01-overview.md) - Overview: Public entry points, repository surfaces, runtime assumptions, and the shortest successful path through the docs.
- 02. [Installation](https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/02-installation.md) - Install: Required tools, Nix option, encrypted environment setup, Docker resources, LocalStack, FusionAuth, and first setup command.
- 03. [Local development quickstart](https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/03-local-development-quickstart.md) - Quickstart: Bring up local services with just recipes, shared Compose resources, database initialization, and expected health signals.
- 04. [Frontend quickstart](https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/04-frontend-quickstart.md) - Quickstart: Run the SolidJS app, choose local or remote services, understand app routing, and start Tauri development.
- 05. [Local E2E smoke tests](https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/05-local-e2e-smoke-tests.md) - Guide: Run deterministic Playwright and ignored Rust smoke tests against the local-only stack and shared seed fixtures.
- 06. [Monorepo map](https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/06-monorepo-map.md) - Concept: Workspace boundaries, package managers, major runtime folders, generated documentation, and where source-of-truth manifests live.
- 07. [Runtime environments and service URLs](https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/07-runtime-environments-and-service-urls.md) - Concept: Environment values, local versus dev versus production URL resolution, CORS rules, and frontend service selection.
- 08. [Service topology](https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/08-service-topology.md) - Concept: Rust services, workers, queues, storage dependencies, ports, and deployment boundaries used by the local and cloud stacks.
- 09. [Entities and blocks](https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/09-entities-and-blocks.md) - Concept: Item types, document-backed and non-document blocks, aliases, load sources, nesting rules, and file type resolution.
- 10. [Split layout and navigation](https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/10-split-layout-and-navigation.md) - Concept: Route encoding, component registry entries, split history, navigation causes, popovers, and desktop versus mobile split behavior.
- 11. [Real-time document sync](https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/11-real-time-document-sync.md) - Concept: Sync-service worker routes, Durable Object sessions, permission tokens, Bebop messages, snapshot lifecycle, and client reconnect behavior.
- 12. [Run backend services locally](https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/12-run-backend-services-locally.md) - Guide: Start databases, LocalStack, FusionAuth, Rust services, optional processor profiles, and local health checks.
- 13. [Develop SolidJS and Tauri apps](https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/13-develop-solidjs-and-tauri-apps.md) - Guide: Use Bun and Vite, run local service overrides, build app bundles, start Tauri targets, and avoid iOS worker deadlocks.
- 14. [Change a Rust service API](https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/14-change-a-rust-service-api.md) - Guide: Update Axum handlers, OpenAPI emitters, generated client specs, Orval output, and CI checks for service API changes.
- 15. [Generate service clients and tool schemas](https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/15-generate-service-clients-and-tool-schemas.md) - Guide: Regenerate OpenAPI JSON, TypeScript clients, DCS model types, AI tool schemas, and MCP documentation pages from Rust sources.
- 16. [Work with local seed data](https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/16-work-with-local-seed-data.md) - Guide: Use seed_cli, local_e2e fixtures, manifest aliases, reset SQL, and shared Playwright and Rust fixture loaders.
- 17. [Storage and workspace APIs](https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/17-storage-and-workspace-apis.md) - Reference: Document, channel, project, call, CRM, soup, pin, history, permissions, properties, and search endpoints.
- 18. [Auth and team APIs](https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/18-auth-and-team-apis.md) - Reference: Login, OAuth callbacks, JWT refresh, account links, team membership, invites, permissions, billing, and user endpoints.
- 19. [Email and notification APIs](https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/19-email-and-notification-apis.md) - Reference: Gmail init and sync, drafts, threads, labels, attachments, notification preferences, unread state, and unsubscribe routes.
- 20. [AI chat streaming API](https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/20-ai-chat-streaming-api.md) - Reference: DCS chat endpoints, stream payloads, toolset selection, model identifiers, extraction statuses, structured completion, and stop semantics.
- 21. [MCP server and tool registry](https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/21-mcp-server-and-tool-registry.md) - Reference: Remote MCP endpoint, OAuth-backed server, toolset composition, SendEmail boundary, generated schemas, and docs tool pages.
- 22. [Environment variables reference](https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/22-environment-variables-reference.md) - Reference: Required and optional service environment variables, defaults, secrets, queue names, ports, and provider-specific keys.
- 23. [Database migrations and SQLx cache](https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/23-database-migrations-and-sqlx-cache.md) - Operations: MacroDB migrations, local database just recipes, SQLx offline cache, fixture data, and migration validation workflow.
- 24. [Feature flags and server selection](https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/24-feature-flags-and-server-selection.md) - Reference: Vite environment overrides, feature flag defaults, local service selection syntax, sync-service host selection, and runtime host helpers.
- 25. [Infrastructure stacks reference](https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/25-infrastructure-stacks-reference.md) - Reference: Pulumi stack layout, reusable resources, service stacks, buckets, queues, Datadog sidecars, FusionAuth, and deployment inputs.
- 26. [Build, test, and quality gates](https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/26-build-test-and-quality-gates.md) - Operations: Rust checks, clippy, SQLx preparation, TypeScript checks, Biome, Tailwind hygiene, Vitest, Playwright, and CI path filters.
- 27. [Deploy services and web app](https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/27-deploy-services-and-web-app.md) - Operations: Generic service deployment, web app deployment, production release workflow, database migrations, Pulumi stacks, and deployment concurrency.
- 28. [Observability and debugging](https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/28-observability-and-debugging.md) - Operations: Backend tracing initialization, Datadog ECS sidecars, browser RUM and logs, sourcemaps, local debug signals, and iOS WebView freeze diagnosis.
- 29. [Documentation site maintenance](https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/29-documentation-site-maintenance.md) - Contribution: Maintain the Mintlify docs site, navigation manifest, generated MCP pages, local preview, broken-link checks, and writing conventions.
## Source File Index
- `.github/workflows/code-check-cloud-storage.yml`
- `.github/workflows/deploy-service-generic.yml`
- `.github/workflows/deploy-web-app.yml`
- `.github/workflows/migrate-macro-db.yml`
- `.github/workflows/release-production.yml`
- `.github/workflows/reusable-deploy-service.yml`
- `.github/workflows/web-app-check-main.yml`
- `docker-compose-databases.yml`
- `docker-compose.local-e2e.yml`
- `docker-compose.yml`
- `docs/AGENTS.md`
- `docs/AI/mcp/overview.mdx`
- `docs/AI/mcp/tools/index.mdx`
- `docs/config/tool-pages.json`
- `docs/CONTRIBUTING.md`
- `docs/docs.json`
- `docs/package.json`
- `docs/README.md`
- `docs/scripts/generate-mcp-tool-pages.ts`
- `flake.nix`
- `infra/packages/lambda/src/index.ts`
- `infra/packages/resources/src/index.ts`
- `infra/packages/resources/src/resources/datadog.ts`
- `infra/packages/service/src/index.ts`
- `infra/packages/shared/src/ai_tools.ts`
- `infra/README.md`
- `infra/stacks/document-storage/index.ts`
- `infra/stacks/fusionauth-instance/README.md`
- `infra/stacks/mcp-server/index.ts`
- `js/app/AGENTS.md`
- `js/app/justfile`
- `js/app/package.json`
- `js/app/packages/app/component/Root.tsx`
- `js/app/packages/app/component/split-layout/componentRegistry.tsx`
- `js/app/packages/app/component/split-layout/layout.ts`
- `js/app/packages/app/component/split-layout/layoutManager.ts`
- `js/app/packages/app/component/split-layout/SplitLayoutRoute.tsx`
- `js/app/packages/app/component/split-layout/tests/layoutManager.test.ts`
- `js/app/packages/app/index.tsx`
- `js/app/packages/app/vite.config.ts`
- `js/app/packages/block-md/definition.ts`
- `js/app/packages/block-project/definition.ts`
- `js/app/packages/core/auth/logout.ts`
- `js/app/packages/core/auth/sso.ts`
- `js/app/packages/core/block.ts`
- `js/app/packages/core/component/AI/types/index.ts`
- `js/app/packages/core/constant/allBlocks.ts`
- `js/app/packages/core/constant/featureFlags.ts`
- `js/app/packages/core/constant/servers.ts`
- `js/app/packages/core/tests/featureFlags.test.ts`
- `js/app/packages/observability/src/index.ts`
- `js/app/packages/service-clients/orval.config.ts`
- `js/app/packages/service-clients/service-auth/client.ts`
- `js/app/packages/service-clients/service-auth/openapi.json`
- `js/app/packages/service-clients/service-cognition/client.ts`
- `js/app/packages/service-clients/service-cognition/openapi.json`
- `js/app/packages/service-clients/service-email/client.ts`
- `js/app/packages/service-clients/service-email/openapi.json`
- `js/app/packages/service-clients/service-notification/client.ts`
- `js/app/packages/service-clients/service-notification/openapi.json`
- `js/app/packages/service-clients/service-properties/client.ts`
- `js/app/packages/service-clients/service-properties/openapi.json`
- `js/app/packages/service-clients/service-search/client.ts`
- `js/app/packages/service-clients/service-search/openapi.json`
- `js/app/packages/service-clients/service-storage/client.ts`
- `js/app/packages/service-clients/service-storage/openapi.json`
- `js/app/packages/service-clients/service-sync/client.ts`
- `js/app/packages/service-clients/service-sync/source.ts`
- `js/app/README.md`
- `js/app/scripts/generate-api-schema.ts`
- `js/app/scripts/generate-dcs-tools.ts`
- `js/app/scripts/services.ts`
- `js/app/tauri/src-tauri/README.md`
- `js/app/tests/e2e/fixtures/local-e2e-seed.ts`
- `js/package.json`
- `justfile`
- `local_stack.just`
- `README.md`
- `RUNNING_LOCALLY.md`
- `rust/cloud-storage/agent/src/model.rs`
- `rust/cloud-storage/AGENTS.md`
- `rust/cloud-storage/ai_tools/src/bin/gen_tool_schemas.rs`
- `rust/cloud-storage/ai_tools/src/lib.rs`
- `rust/cloud-storage/authentication_service/src/config.rs`
- `rust/cloud-storage/authentication_service/src/openapi.rs`
- `rust/cloud-storage/Cargo.toml`
- `rust/cloud-storage/chat/.sqlx/query-57fc603adf83fd7c07968425c98f9a57924573692cf0529ad021b6eef00ce99e.json`
- `rust/cloud-storage/database.just`
- `rust/cloud-storage/document_cognition_service/src/api/stream/chat_message/mod.rs`
- `rust/cloud-storage/document_cognition_service/src/api/stream/stop.rs`
- `rust/cloud-storage/document_cognition_service/src/config.rs`
- `rust/cloud-storage/document_cognition_service/src/model/stream.rs`
- `rust/cloud-storage/document_cognition_service/src/openapi.rs`
- `rust/cloud-storage/document_storage_service/src/config.rs`
- `rust/cloud-storage/document_storage_service/src/openapi.rs`
- `rust/cloud-storage/email_service/src/config.rs`
- `rust/cloud-storage/email_service/src/openapi.rs`
- `rust/cloud-storage/integration_tests/local_e2e/README.md`
- `rust/cloud-storage/justfile`
- `rust/cloud-storage/local_e2e_test_support/README.md`
- `rust/cloud-storage/macro_cors/src/lib.rs`
- `rust/cloud-storage/macro_db_client/migrations/0001_baseline.sql`
- `rust/cloud-storage/macro_db_client/README.md`
- `rust/cloud-storage/macro_entrypoint/src/lib.rs`
- `rust/cloud-storage/macro_env/src/lib.rs`
- `rust/cloud-storage/macro_service_urls/src/lib.rs`
- `rust/cloud-storage/mcp_service/src/main.rs`
- `rust/cloud-storage/mcp_service/src/tool_service.rs`
- `rust/cloud-storage/models_soup/src/lib.rs`
- `rust/cloud-storage/notification_service/src/config.rs`
- `rust/cloud-storage/notification_service/src/openapi.rs`
- `rust/cloud-storage/README.md`
- `rust/cloud-storage/search_service/src/config.rs`
- `rust/cloud-storage/seed_cli/README.md`
- `rust/cloud-storage/seed_cli/seed/local_e2e/manifest.json`
- `rust/cloud-storage/seed_cli/seed/local_e2e/reset.sql`
- `rust/cloud-storage/seed_cli/seed/local_e2e/users.json`
- `rust/cloud-storage/seed_cli/src/main.rs`
- `rust/cloud-storage/sqlx.just`
- `rust/sync-service/Cargo.toml`
- `rust/sync-service/src/cf_worker.rs`
- `rust/sync-service/src/durable_object.rs`
- `rust/sync-service/src/generated/schema.rs`
- `rust/sync-service/src/websocket.rs`
---
## 01. Overview
> Overview: Public entry points, repository surfaces, runtime assumptions, and the shortest successful path through the docs.
- Page Markdown: https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/01-overview.md
- Generated: 2026-06-01T00:46:58.329Z
### Source Files
- `README.md`
- `RUNNING_LOCALLY.md`
- `js/app/README.md`
- `rust/cloud-storage/AGENTS.md`
- `docker-compose.yml`
- `docs/docs.json`
---
title: "Overview"
description: "Overview: Public entry points, repository surfaces, runtime assumptions, and the shortest successful path through the docs."
---
Macro is a SolidJS and Rust monorepo for the Macro workspace: a static SPA frontend, Rust backend services, generated MCP tool documentation, Pulumi infrastructure, Docker Compose local services, and deterministic local smoke-test fixtures.
## Public entry points
| Surface | Entry point | Backing repository area |
| --- | --- | --- |
| Web application | `https://macro.com/app` | `js/app`, `infra/stacks/web-app` |
| Product docs | `docs.macro.com` site config | `docs/docs.json`, `docs/index.mdx`, `docs/product/*` |
| MCP server | `https://mcp-server.macro.com/mcp` | `rust/cloud-storage/mcp_service`, `rust/cloud-storage/ai_tools`, `docs/AI/mcp/*` |
| Local backend stack | `just run_local` | `justfile`, `docker-compose.yml`, `docker-compose-databases.yml` |
| Local E2E smoke stack | `just local-e2e` | `docker-compose.local-e2e.yml`, `rust/cloud-storage/seed_cli`, `js/app/tests/e2e` |
<Note>
The local setup is explicitly single-instance. The root `justfile` exports `COMPOSE_PROJECT_NAME=macro`, and the Compose files use the fixed project name `macro`, so multiple checkouts share the same local containers, volumes, networks, LocalStack, and FusionAuth resources.
</Note>
## Repository surfaces
:::files
```text
.
├── README.md # product, security, license, community
├── RUNNING_LOCALLY.md # supported local setup and E2E path
├── justfile # root setup, local stack, E2E orchestration
├── docker-compose*.yml # local services, databases, E2E overrides
├── local_stack.just # LocalStack SQS, DynamoDB, and S3 setup
├── docs/ # public docs site and generated MCP docs
├── js/
│ ├── app/ # SolidJS/Vite frontend workspace
│ ├── lexical-service/ # Lexical conversion service
│ └── websocket-service/ # Bun websocket service
├── rust/
│ ├── cloud-storage/ # Rust service workspace
│ └── sync-service/ # sync service container
├── infra/ # Pulumi stacks and shared service package
└── agents/transcription/ # transcription agent service
```
:::
### Frontend
The frontend lives in `js/app` and uses Bun, Vite, SolidJS, Tailwind, TypeScript, and workspace packages under `js/app/packages/*`. The app package builds a static SPA with base path `/app`; development serves from `/` on port `3000` by default.
Common commands:
```bash
cd js/app
bun i
just local
bun run check
bun run local:e2e
```
The Vite config exposes build-time values such as `import.meta.env.__APP_VERSION__`, `ASSETS_PATH`, `__LOCAL_DOCKER__`, `__LOCAL_JWT__`, and `__GIT_BRANCH__`.
### Rust services
The main backend surface is the Cargo workspace in `rust/cloud-storage`. It contains service binaries, worker binaries, Lambda handlers, database clients, domain crates, generated schemas, and integration tests.
Representative service binaries include:
| Binary | Local Compose service | Port mapping |
| --- | --- | --- |
| `authentication_service` | `authentication-service` | `8080:8080` |
| `connection_gateway_service` | `connection_gateway` | `8082:8080` |
| `contacts_service` | `contacts_service` | `8083:8080` |
| `document_storage_service` | `document_storage_service` | `8086:8080` |
| `email_service` | `email_service` | `8087:8080` |
| `notification_service` | `notification_service` | `8089:8080` |
| `search_processing_service` | `search_processing_service` | `8092:8080` |
| `static_file_service` | `static_file_service` | `8094:8080` |
| `unfurl_service` | `unfurl_service` | `8095:8080` |
| `image_proxy_service` | `image_proxy_service` | `8097:8080` |
`rust/cloud-storage/Dockerfile.dev` builds the standard service bundle into the `macro-local-rust-services:dev` image. `search_processing_service` uses a separate Dockerfile and is behind the `processors` Compose profile.
### Docs and MCP reference
The docs site is configured by `docs/docs.json`. Current navigation contains:
- Getting Started: `docs/index.mdx`
- Product pages: AI Chat, Email, Channels, Docs, Canvas, Tasks, Search, Files
- MCP setup: `docs/AI/mcp/overview.mdx`
- Generated MCP tool pages: `docs/AI/mcp/tools/*`
- Changelog: `docs/changelog/introduction`
MCP tool pages are generated from the Rust tool registry:
```bash
cd docs
bun install
bun run generate:tools
bun run dev
```
The generator builds `gen_tool_schemas` from `rust/cloud-storage`, reads `rust/cloud-storage/ai_tools/schemas/tools.json`, writes MDX pages under `docs/AI/mcp/tools`, and updates `docs/config/tool-pages.json`.
<Info>
The MCP docs and transport are portable at the repository boundary: the public setup page points clients at the hosted HTTP MCP endpoint and authenticates through Macro OAuth. Tool references are generated from repository schemas rather than from a specific client, editor, or model provider.
</Info>
### Infrastructure
Infrastructure lives under `infra/stacks/*` and shared Pulumi packages under `infra/packages/*`.
Important stacks include:
| Stack area | Purpose |
| --- | --- |
| `infra/stacks/web-app` | Publishes the built SPA to S3 and wires route/encoding Lambda functions |
| `infra/stacks/document-storage` | Creates document storage buckets, replication, and the shared ECS cluster |
| `infra/stacks/mcp-server` | Deploys the MCP service and configures OAuth, Redis, queues, buckets, and secrets |
| `infra/stacks/fusionauth-instance` | Local and deployed FusionAuth configuration |
| `infra/packages/service` | Shared ECS Fargate service component with load balancer, IAM role, Datadog sidecars, and environment variables |
## Runtime assumptions
### Required local tools
`RUNNING_LOCALLY.md` lists these local prerequisites:
- `sops`
- Docker
- AWS CLI/configuration
- SQLx / `sqlx-cli`
- Pulumi
- Bun
- Node
- `just`
If Nix is available, the repo supports `nix develop` for toolchain management.
### Local data plane
The default local stack uses:
| Dependency | Local implementation |
| --- | --- |
| PostgreSQL | `pgvector/pgvector:pg16` on `localhost:5432` |
| Redis | `redis/redis-stack` on `localhost:6379` and UI/tools on `8001` |
| OpenSearch | `opensearchproject/opensearch` on `9200` |
| AWS SQS/DynamoDB/S3 | LocalStack v4 on `4566` |
| FusionAuth | `infra/stacks/fusionauth-instance/docker-compose.yml` |
| Static CDN emulation | nginx `static_file_cdn` on `8100` |
Local E2E overrides force services onto local Postgres, Redis, LocalStack queues, LocalStack buckets, and deterministic seed data.
### Environment and secrets
The root setup path decrypts `.env-local*.enc` into `.env` with `sops`, patches local FusionAuth values when available, creates shared Docker networks and volumes, initializes local AWS resources, initializes databases, and builds the Rust dev service image.
```bash
just setup
```
Use `just get_environment` when only the `.env` file is needed.
## Shortest successful path
<Steps>
<Step title="Read the product and license boundary">
Start with `README.md` for the public product surface, security contact, AGPLv3 license, and self-hosting note.
</Step>
<Step title="Initialize the local environment">
Run the repository setup from the root:
```bash
just setup
```
This prepares `.env`, Docker networks and volumes, LocalStack resources, local databases, FusionAuth, and Rust service images.
</Step>
<Step title="Run backend services">
```bash
just run_local
```
If service images changed, rebuild during startup:
```bash
just run_local --build
```
Add the `processors` profile only when you need processor services such as `search_processing_service`.
</Step>
<Step title="Run the frontend">
```bash
cd js/app
bun i
just local
```
The Vite dev server defaults to port `3000`.
</Step>
<Step title="Run deterministic smoke tests">
```bash
just local-e2e
```
For Rust ignored integration tests against the same local stack:
```bash
just local-e2e-rust
```
For both Rust and Playwright smoke coverage:
```bash
just local-e2e-all
```
</Step>
</Steps>
## Verification signals
| Command | Expected signal |
| --- | --- |
| `just run_local` | Compose services start and healthchecks pass |
| `just local-e2e` | LocalStack setup completes, seed data loads, Playwright smoke tests run |
| `cd rust/cloud-storage && just check` | Rust workspace type-checks with `SQLX_OFFLINE=true` |
| `cd rust/cloud-storage && just clippy` | Workspace clippy passes with warnings denied |
| `cd js/app && bun run check` | TypeScript and Biome checks pass |
| `cd docs && bun run generate:tools` | MCP tool pages regenerate from Rust schemas |
| `cd docs && bun run lint` | Docs broken-link check runs through Mintlify |
## Failure modes to check first
| Symptom | Likely cause | Check |
| --- | --- | --- |
| Local services share unexpected state | Compose project is intentionally frozen to `macro` | Do not run two local stacks at the same time |
| `just run_local` fails before services start | Missing `.env` | Run `just get_environment` or `just setup` |
| FusionAuth values are missing | Local Pulumi/FusionAuth setup not complete | Run `just setup` or inspect `infra/stacks/fusionauth-instance` |
| Local E2E seed refuses to run | Safety guard rejected the target database | Ensure `LOCAL_E2E_SEED=true` and `DATABASE_URL` targets local `macrodb` |
| SQLx reports missing cached query data | Rust SQL changed without refreshing offline cache | Run `just prepare_db` from `rust/cloud-storage` |
| Docs tool pages drift from code | Generated pages were not rebuilt | Run `cd docs && bun run generate:tools` |
## Related pages
<CardGroup cols={2}>
<Card title="MCP setup" href="/AI/mcp/overview">
Connect clients to Macro's hosted MCP endpoint.
</Card>
<Card title="MCP tool reference" href="/AI/mcp/tools">
Inspect generated tool schemas from the Rust registry.
</Card>
<Card title="Product introduction" href="/">
Navigate the public product docs surface.
</Card>
</CardGroup>
---
## 02. Installation
> Install: Required tools, Nix option, encrypted environment setup, Docker resources, LocalStack, FusionAuth, and first setup command.
- Page Markdown: https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/02-installation.md
- Generated: 2026-06-01T00:46:44.840Z
### Source Files
- `RUNNING_LOCALLY.md`
- `flake.nix`
- `justfile`
- `local_stack.just`
- `docker-compose-databases.yml`
- `rust/cloud-storage/README.md`
---
title: "Installation"
description: "Install: Required tools, Nix option, encrypted environment setup, Docker resources, LocalStack, FusionAuth, and first setup command."
---
Macro local installation is driven by `just setup` at the repository root. The setup path decrypts `.env`, creates shared Docker networks and volumes, provisions LocalStack AWS resources, initializes the local Postgres database, configures a local FusionAuth stack, and builds local Rust service images.
## Prerequisites
| Tool | Required for |
|---|---|
| `just` | Repository task runner |
| Docker with Compose v2 | Databases, services, FusionAuth, LocalStack |
| `sops` | Decrypting `.env-local*.enc` files |
| AWS CLI and AWS credentials | LocalStack provisioning and encrypted environment access |
| `pulumi` | Local FusionAuth stack configuration |
| `bun` | Frontend and FusionAuth stack dependencies |
| Node.js | JavaScript tooling and Pulumi Node runtime |
| `sqlx-cli` | Database create, migrate, prepare, and reset commands |
| Rust toolchain | Backend builds and local Rust tests |
<Note>
The repository currently documents local development as work in progress and primarily supports running services against dev assets or the provided local E2E stack.
</Note>
## Nix option
If Nix is available, use the repository flake instead of installing most tools manually:
```bash
nix develop
```
The default shell provides the Rust toolchain, `just`, `bun`, `pulumi`, `sops`, `sqlx-cli`, Node.js 24, Docker Compose tooling, `jq`, and related backend/frontend utilities. It also sets `SOPS_KMS_ARN` for encrypted environment decryption.
For Tauri/frontend platform work, use the JavaScript app shell:
```bash
nix develop .#js-app
```
<Warning>
Nix supplies CLI tools, not the Docker daemon. Docker must still be running on the host.
</Warning>
## Encrypted environment setup
The root `justfile` decrypts encrypted dotenv bundles into a plain root `.env` file.
| Command | Input | Output | Notes |
|---|---|---|---|
| `just get_environment` | `.env-local.enc` | `.env` | Fully local backing services |
| `just get_environment dev` | `.env-localdev.enc` | `.env` | Dev backing services, used for ad-hoc dev-service work |
| `just edit_environment` | `.env-local.enc` | encrypted file edited through `sops` | Maintainer workflow |
| `just edit_environment dev` | `.env-localdev.enc` | encrypted file edited through `sops` | Maintainer workflow |
| `just fix_environment [dev]` | encrypted dotenv | re-encrypted dotenv | Decrypts with `--ignore-mac`, re-encrypts, removes temporary plaintext |
If you are not inside `nix develop`, export the KMS ARNs before decrypting:
```bash
export SOPS_KMS_ARN="arn:aws:kms:us-east-1:569036502058:key/mrk-cab29bf948044eb79005a81f48d40e93,arn:aws:kms:us-west-1:569036502058:key/mrk-cab29bf948044eb79005a81f48d40e93"
```
When `just get_environment dev` is used and `~/.aws/credentials` exists, the recipe replaces `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` in `.env` from the `[default]` profile.
<Warning>
`just setup` always runs `just get_environment` with no suffix, so it regenerates `.env` from `.env-local.enc`.
</Warning>
## First setup command
Run from the repository root:
```bash
just setup
```
`just setup` performs this sequence:
<Steps>
<Step title="Decrypt the root environment">
Generates `.env` from `.env-local.enc`.
</Step>
<Step title="Create shared Docker resources">
Creates the `databases` and `auth` networks plus the shared local volumes.
</Step>
<Step title="Provision LocalStack">
Starts the `localstack` container on port `4566`, then creates local SQS queues, DynamoDB tables, S3 buckets, CORS rules, and the document upload finalizer notification.
</Step>
<Step title="Initialize local databases">
Starts Postgres and Redis, creates `macrodb`, runs migrations through SQLx, then stops the database Compose file.
</Step>
<Step title="Configure FusionAuth">
Starts the local FusionAuth stack, waits for health, initializes the Pulumi `local` stack, patches root `.env` with local FusionAuth values, and stops FusionAuth.
</Step>
<Step title="Build service images">
Builds `macro-local-rust-services:dev` and the local search processing image.
</Step>
</Steps>
Expected final output:
```text
Setup complete.
```
## Docker resources
Local Docker resources are intentionally frozen to the `macro` Compose project. Multiple checkouts and worktrees share the same containers, volumes, networks, LocalStack instance, and FusionAuth instance.
<Warning>
Do not run two local stacks at the same time from different checkouts.
</Warning>
### Shared networks and volumes
`just create_networks` creates:
| Resource | Name |
|---|---|
| Network | `databases` |
| Network | `auth` |
| Volume | `macro_postgres_data` |
| Volume | `macro_redis_data` |
| Volume | `macro_opensearch_data` |
| Volume | `fusionauth_db_data` |
| Volume | `fusionauth_config` |
### Database Compose services
`docker-compose-databases.yml` defines:
| Service | Image | Ports | Purpose |
|---|---|---:|---|
| `postgres` | `pgvector/pgvector:pg16` | `5432` | Local `macrodb` database |
| `redis` | `redis/redis-stack:latest` | `6379`, `8001` | Redis plus Redis Stack UI/tools |
| `search` | `opensearchproject/opensearch:latest` | `9200`, `9600` | Local OpenSearch |
The local database URL used by cloud-storage justfiles is:
```text
postgres://user:password@localhost:5432/macrodb
```
## LocalStack
Local AWS emulation is handled by `local_stack.just`.
```bash
just setup_localstack
```
The setup starts `localstack/localstack:4` with:
| Setting | Value |
|---|---|
| Container name | `localstack` |
| Network | `databases` |
| Port | `4566` |
| Services | `sqs,dynamodb,s3` |
| Health endpoint | `http://localhost:4566/_localstack/health` |
The recipe provisions SQS queues used by notification, email, contacts, conversion, document deletion, document upload finalization, document text extraction, search events, and static-file events. It also creates these DynamoDB tables:
| Table | Key shape |
|---|---|
| `bulk-upload` | `PK` + `SK`, with `DocumentPkIndex` |
| `connection-gateway-table` | `PK` + `SK`, with `ConnectionPkIndex` |
| `static-file-metadata` | `file_id` hash key |
S3 buckets created locally:
| Bucket |
|---|
| `macro-email-attachments` |
| `doc-storage` |
| `docx-upload` |
| `static-file-storage` |
| `bulk-upload-staging` |
All buckets receive CORS rules for `http://localhost:3000` through `http://localhost:3009`. The `doc-storage` bucket is also wired so `s3:ObjectCreated:*` events send messages to `document-upload-finalizer-queue`.
Backend AWS clients switch to LocalStack when `LOCAL_AWS_URL` is set. In that mode, the Rust AWS config uses `us-east-1`, test credentials, the configured endpoint URL, and path-style S3 behavior.
## FusionAuth
The local FusionAuth stack lives under `infra/stacks/fusionauth-instance`.
```bash
just setup_fusionauth
```
The root setup command calls the same stack setup through:
```bash
just infra/stacks/fusionauth-instance/setup
```
Local FusionAuth uses:
| Setting | Value |
|---|---|
| App image | `fusionauth/fusionauth-app:1.62.1` |
| Database image | `postgres:16.0-bookworm` |
| Port | `9011` |
| Runtime mode | `development` |
| App URL inside Docker | `http://fusionauth:9011` |
| Local issuer | `local.macro.com` |
| Pulumi stack | `local` |
Local admin credentials documented by the stack:
```text
username: admin@macro.com
password: macroIsGreat!
api-key: bf69486b-4733-4954-a44e-2e1b5f2c8a91
```
<Warning>
Create the local Pulumi stack as `local`; do not use a `macro-inc/` stack prefix for the local FusionAuth instance.
</Warning>
During setup, the FusionAuth recipe patches root `.env` with local values including:
| Key |
|---|
| `AUDIENCE` |
| `FUSIONAUTH_TENANT_ID` |
| `FUSIONAUTH_CLIENT_ID` |
| `FUSIONAUTH_CLIENT_SECRET_KEY` |
| `JWT_SECRET_KEY` |
| `ISSUER` |
| `FUSIONAUTH_BASE_URL` |
| `FUSIONAUTH_API_KEY_SECRET_KEY` |
| `FUSIONAUTH_OAUTH_REDIRECT_URI` |
`just run_local` also calls `just patch_local_fusionauth_env`, which patches `.env` again if the local Pulumi stack exists. If FusionAuth is not running, that recipe starts it temporarily to read the OAuth client secret, then stops it.
## Running after installation
Start backend services:
```bash
just run_local
```
If service images changed, rebuild selected images while starting:
```bash
just run_local --build
```
By default, `convert_service` and `search_processing_service` are not required for the frontend dev path. `search_processing_service` is behind the `processors` Compose profile and is pinned to `linux/amd64` because its local PDFium library is amd64-only.
Start the frontend against local services:
```bash
cd js/app
bun i
just local
```
## Local E2E smoke setup
After installation:
```bash
just local-e2e
```
This uses `docker-compose.local-e2e.yml` overrides so services use local Postgres and LocalStack instead of shared dev assets, seeds deterministic fixture data, launches the frontend with local bearer-token auth, and runs Playwright.
Additional E2E commands:
```bash
just local-e2e-rust
just local-e2e-all
just local-e2e-ui
```
The local E2E seed path is guarded: it requires `LOCAL_E2E_SEED=true` and rejects database URLs that are not the local Docker database shape `postgres://user:...@(localhost|127.0.0.1|postgres):5432/macrodb`.
## Cleanup and reset
| Command | Effect |
|---|---|
| `just stop-local` | `docker compose down` for local services |
| `just stop-databases` | Stops the database Compose file |
| `just stop_fusionauth` | Stops the FusionAuth Compose file |
| `just destroy` | Destroys the local FusionAuth stack and runs `docker compose down -v` |
| `just docker_cache_clear` | Clears all BuildKit cache |
| `just docker_cache_clear_targets` | Clears Rust target cache mounts only |
| `just docker_cache_usage` | Shows BuildKit cache usage |
## Troubleshooting
| Symptom | Cause | Fix |
|---|---|---|
| `.env not found` during local run | Environment was not decrypted | Run `just get_environment` or rerun `just setup` |
| `Pulumi local stack not found` | FusionAuth local stack has not been initialized | Run `just setup` |
| LocalStack AWS commands fail | AWS CLI is missing or unavailable | Install/configure AWS CLI; LocalStack recipes call `aws --endpoint-url=http://localhost:4566 ...` |
| Browser cannot resolve LocalStack bucket hostnames | Presigned URLs generated inside Docker use `localstack` hostnames | Use local mode with `LOCAL_AWS_URL`; the Rust AWS helper rewrites local URLs to `localhost` |
| Different checkout changes local containers | Docker resources are shared under Compose project `macro` | Stop the other checkout before starting this one |
| FusionAuth client secret cannot be read | FusionAuth is not running or local stack is incomplete | Run `just setup_fusionauth` or rerun `just setup` |
## Related pages
<CardGroup>
<Card title="Running locally" href="/running-locally">
Backend, frontend, and local E2E commands after installation.
</Card>
<Card title="Cloud storage" href="/cloud-storage">
Rust backend services, database setup, and deployment notes.
</Card>
<Card title="FusionAuth instance" href="/fusionauth-instance">
Local FusionAuth stack behavior, Pulumi configuration, and credentials.
</Card>
</CardGroup>
---
## 03. Local development quickstart
> Quickstart: Bring up local services with just recipes, shared Compose resources, database initialization, and expected health signals.
- Page Markdown: https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/03-local-development-quickstart.md
- Generated: 2026-06-01T00:46:27.751Z
### Source Files
- `RUNNING_LOCALLY.md`
- `justfile`
- `docker-compose.yml`
- `docker-compose-databases.yml`
- `local_stack.just`
- `infra/stacks/fusionauth-instance/README.md`
---
title: "Local development quickstart"
description: "Quickstart: Bring up local services with just recipes, shared Compose resources, database initialization, and expected health signals."
---
Local development is orchestrated from the root `justfile`: `just setup` prepares `.env`, shared Docker resources, LocalStack, Postgres migrations, FusionAuth, and cached Rust service images; `just run_local` starts the Docker Compose service stack under the fixed `macro` Compose project.
## Prerequisites
Install the tools expected by `RUNNING_LOCALLY.md` and the recipes:
| Tool | Used for |
| --- | --- |
| Docker / Docker Compose | Databases, LocalStack, FusionAuth, backend services |
| `just` | Repository task runner |
| `sops` | Decrypting `.env-local*.enc` into `.env` |
| AWS CLI | Creating LocalStack SQS, DynamoDB, and S3 resources |
| SQLx CLI | Creating and migrating `macrodb` |
| Pulumi | Creating the local FusionAuth stack |
| Bun and Node | Frontend, FusionAuth stack dependencies, Playwright |
If not using the repository Nix shell, export the configured SOPS KMS ARN before decrypting local environment files.
```bash
export SOPS_KMS_ARN="arn:aws:kms:us-east-1:569036502058:key/mrk-cab29bf948044eb79005a81f48d40e93,arn:aws:kms:us-west-1:569036502058:key/mrk-cab29bf948044eb79005a81f48d40e93"
```
<Warning>
The local Docker resources are intentionally frozen to the `macro` Compose project. Multiple checkouts and worktrees share the same containers, networks, and volumes. Do not run two local stacks at the same time.
</Warning>
## Quickstart
<Steps>
<Step title="Initialize local dependencies">
Run the repository setup recipe from the repository root.
```bash
just setup
```
This decrypts `.env`, creates shared Docker networks and volumes, initializes LocalStack, creates and migrates the local database, configures FusionAuth, and prebuilds development service images.
</Step>
<Step title="Start backend services">
Start the Compose stack.
```bash
just run_local
```
To rebuild service containers after local service changes, pass the build flag.
```bash
just run_local --build
```
</Step>
<Step title="Start the frontend">
In a second terminal, run the frontend against local services.
```bash
cd js/app
bun i
just local
```
`just local` runs the app with `VITE_LOCAL_SERVERS=ALL` and defaults to `PORT=3000`.
</Step>
</Steps>
## Shared Docker resources
`create_networks` creates the shared resources used across the root Compose file, the database Compose file, and the FusionAuth Compose file.
| Resource | Name |
| --- | --- |
| Compose project | `macro` |
| External networks | `databases`, `auth` |
| Compose service network | `services` |
| Postgres volume | `macro_postgres_data` |
| Redis volume | `macro_redis_data` |
| OpenSearch volume | `macro_opensearch_data` |
| FusionAuth database volume | `fusionauth_db_data` |
| FusionAuth config volume | `fusionauth_config` |
The root `docker-compose.yml` includes `docker-compose-databases.yml` and `infra/stacks/fusionauth-instance/docker-compose.yml`, so `just run_local` can start application services together with Postgres, Redis, OpenSearch, and FusionAuth.
## What `just setup` initializes
| Stage | Recipe | Result |
| --- | --- | --- |
| Environment | `just get_environment` | Decrypts `.env-local.enc` into root `.env`; optionally patches AWS credentials from the default local AWS profile when an environment suffix is used |
| Docker resources | `just create_networks` | Creates shared networks and volumes |
| Local AWS | `just setup_localstack` | Starts `localstack/localstack:4` on port `4566`; creates SQS queues, DynamoDB tables, S3 buckets, bucket CORS, and the document upload finalizer notification |
| Local database | `just setup_local_dbs` | Starts Postgres and Redis, creates `macrodb`, runs migrations, then stops the database Compose services |
| FusionAuth | `infra/stacks/fusionauth-instance/setup` | Starts FusionAuth, waits for `/api/status`, applies the Pulumi `local` stack, patches root `.env`, then stops FusionAuth |
| Service image cache | `rust/cloud-storage/build_dev_service_images` | Builds `macro-local-rust-services:dev` and the search-processing image |
The local Macro database URL used by the Rust database recipes is:
```text
postgres://user:password@localhost:5432/macrodb
```
## LocalStack resources
`local_stack.just` starts LocalStack as a standalone Docker container named `localstack` on the shared `databases` network.
| Type | Local resources |
| --- | --- |
| Services | `sqs`, `dynamodb`, `s3` |
| Endpoint | `http://localhost:4566` |
| Buckets | `macro-email-attachments`, `doc-storage`, `docx-upload`, `static-file-storage`, `bulk-upload-staging` |
| DynamoDB tables | `bulk-upload`, `connection-gateway-table`, `static-file-metadata` |
| Document upload event | S3 `ObjectCreated` events from `doc-storage` are wired to `document-upload-finalizer-queue` |
All buckets receive a local CORS policy for `http://localhost:3000` through `http://localhost:3009`.
## Service ports and health signals
Most Rust services expose container port `8080` and map it to a distinct host port.
| Service | Host port | Health check |
| --- | ---: | --- |
| `authentication-service` | `8080` | `GET /health` |
| `connection_gateway` | `8082` | `GET /health` |
| `contacts_service` | `8083` | `GET /health` |
| `document_cognition_service` | `8085` | `GET /health` |
| `document_storage_service` | `8086` | `GET /health` |
| `email_service` | `8087` | `GET /health` |
| `notification_service` | `8089` | `GET /health` |
| `search_processing_service` | `8092` | `GET /health`; behind the `processors` profile |
| `static_file_service` | `8094` | `GET /api/health` |
| `static_file_cdn` | `8100` | Nginx proxy to S3/static file service |
| `unfurl_service` | `8095` | `GET /health` |
| `image_proxy_service` | `8097` | `GET /health` |
| `lexical_service` | `8096` | `GET /health` |
| `sync_service` | `8787` | `GET /health` |
| `websocket_service` | `6969` | WebSocket endpoint; no Compose health check |
| FusionAuth | `9011` | `GET /api/status` returns `{"status":"Ok"}` |
| LocalStack | `4566` | `GET /_localstack/health` |
| Postgres | `5432` | `pg_isready -U user` |
| Redis | `6379`, UI on `8001` | Container availability |
| OpenSearch | `9200`, analyzer on `9600` | `GET /` |
Use Docker Compose waiting when starting a subset or detached stack:
```bash
just run_local -d --wait
```
## Frontend local service selection
`js/app/justfile` provides local frontend shortcuts. The default local command selects all local service hosts.
```bash
cd js/app
just local
```
The underlying app config reads `VITE_LOCAL_SERVERS`:
| Value | Behavior |
| --- | --- |
| unset | Uses remote development service hosts in development mode |
| `ALL` | Uses all local service hosts |
| comma-separated service names | Uses local hosts only for the selected services |
| `service-name:port` | Uses a local host with a port override for that service |
List recognized local service names with:
```bash
cd js/app
just local-services
```
## Local E2E smoke path
The local smoke harness runs the backend with local-only service overrides, seeds deterministic fixtures, starts the frontend with local bearer-token auth, and runs Playwright.
```bash
just local-e2e
```
Related harnesses:
```bash
just local-e2e-rust
just local-e2e-all
just local-e2e-ui
```
The E2E path uses `docker-compose.local-e2e.yml` to override database, Redis, LocalStack, bucket, table, and queue environment variables so the smoke suite does not mutate shared dev assets. The seed command is guarded by `LOCAL_E2E_SEED=true` and rejects any `DATABASE_URL` outside the local Docker database shape:
```text
postgres://user:...@(localhost|127.0.0.1|postgres):5432/macrodb
```
## Stopping and resetting
| Command | Effect |
| --- | --- |
| `just stop-local` | Runs `docker compose down` for the root stack |
| `just stop-databases` | Stops the database Compose file |
| `just stop_fusionauth` | Stops the FusionAuth Compose file |
| `just destroy` | Destroys the local FusionAuth Pulumi stack and runs Compose down with volumes for the root stack |
| `just docker_cache_clear` | Clears all BuildKit build cache |
| `just docker_cache_clear_targets` | Clears Rust target cache mounts only |
## Troubleshooting
### `.env not found`
`just run_local` patches local FusionAuth values into `.env`. If `.env` is missing, run:
```bash
just get_environment
```
For a fresh checkout, prefer the full setup:
```bash
just setup
```
### FusionAuth local stack not found
If `patch_local_fusionauth_env` reports that the Pulumi local stack is missing, run:
```bash
just setup
```
FusionAuth local admin defaults are:
```text
username: admin@macro.com
password: macroIsGreat!
api-key: bf69486b-4733-4954-a44e-2e1b5f2c8a91
```
### Local E2E token generation fails
Prefer the repository-level harness:
```bash
just local-e2e
```
If running Playwright directly, seed first and ensure `.env` exists:
```bash
just local-e2e-seed
cd js/app
LOCAL_E2E=true bunx playwright test
```
You can bypass token generation by exporting `LOCAL_JWT`.
### Search processing on Apple Silicon
`search_processing_service` is pinned to `linux/amd64` because its local `pdfium` library is AMD64-only. On Apple Silicon, enabling the processor profile uses QEMU emulation for build and runtime.
## Related pages
<CardGroup>
<Card title="Frontend local service selection" href="/frontend-local-services">
How `VITE_LOCAL_SERVERS` maps frontend clients to local or remote service hosts.
</Card>
<Card title="Local E2E smoke testing" href="/local-e2e-smoke-testing">
Deterministic seed data, Playwright auth, and Rust local E2E integration tests.
</Card>
</CardGroup>
---
## 04. Frontend quickstart
> Quickstart: Run the SolidJS app, choose local or remote services, understand app routing, and start Tauri development.
- Page Markdown: https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/04-frontend-quickstart.md
- Generated: 2026-06-01T00:46:48.071Z
### Source Files
- `js/app/README.md`
- `js/app/package.json`
- `js/app/justfile`
- `js/app/packages/app/index.tsx`
- `js/app/packages/app/component/Root.tsx`
- `js/app/AGENTS.md`
---
title: "Frontend quickstart"
description: "Quickstart: Run the SolidJS app, choose local or remote services, understand app routing, and start Tauri development."
---
The Macro frontend is a Bun-managed Vite/SolidJS SPA in `js/app`; the web app serves under `/app`, while Tauri loads the same frontend bundle through native desktop and mobile shells.
## Prerequisites
- Bun
- `just`
- Optional: `nix develop` from the repository root to enter a shell with the expected toolchain
- For Tauri: Rust, Tauri CLI, and platform SDKs for Android or iOS
## Run the web app
<Steps>
<Step title="Install frontend dependencies">
```bash
cd js/app
bun i
```
</Step>
<Step title="Start Vite">
```bash
bun run dev
```
</Step>
<Step title="Open the app">
```text
http://localhost:3000/app
```
</Step>
</Steps>
`bun run dev` runs `cd packages/app && bun run --bun dev`, which starts Vite with `packages/app/vite.config.ts`. The dev server defaults to port `3000`, binds to `0.0.0.0`, uses strict port mode, and enables HMR over WebSocket.
## Choose remote or local services
In development mode, service endpoints come from `packages/core/constant/servers.ts`.
| Command or environment | Service behavior |
| --- | --- |
| `bun run dev` | Uses remote development Macro services. |
| `VITE_LOCAL_SERVERS=ALL bun run dev` | Uses all registered localhost service URLs. |
| `VITE_LOCAL_SERVERS=document-storage-service,email-service bun run dev` | Uses local URLs for selected services and remote development URLs for the rest. |
| `VITE_LOCAL_SERVERS=document-storage-service:9090 bun run dev` | Uses the selected local service with a port override. |
| `just local` | Starts the app with `VITE_LOCAL_SERVERS=ALL` and `PORT=3000`. |
Useful shortcuts from `js/app/justfile`:
```bash
cd js/app
just local-services
just local
just local-dss
just local-search
just local-email
```
<Warning>
Local backend services require the repository-level local setup. The local stack is documented by `RUNNING_LOCALLY.md`; the frontend command there is `cd js/app && bun i && just local`.
</Warning>
## App routing model
The runtime selects the router by platform:
| Runtime | Router | Base |
| --- | --- | --- |
| Web | `Router` | `/app` |
| Tauri | `HashRouter` | `/` |
The canonical authenticated landing route is:
```text
/component/inbox
```
The root route `/` preloads user info and history, stores known campaign query parameters as short-lived cookies, normalizes short IDs in the URL, then redirects:
| State | Result |
| --- | --- |
| Authenticated | Navigate to `/component/inbox`, preserving query parameters. |
| Unauthenticated | Navigate to `/welcome`, which redirects to `/login`. |
| Native mobile with login cookie and failed user-info query | Show an offline retry fallback. |
The split-layout route decodes URL segments as type/id pairs:
```text
/component/inbox
/<block-or-alias>/<id>
/component/inbox/<block-or-alias>/<id>
```
If no valid pair is present, the layout falls back to the inbox component.
Top-level auth and utility routes include:
| Path | Behavior |
| --- | --- |
| `/login` | Login UI |
| `/signup` | Signup UI |
| `/email-signup-callback` | Email signup callback |
| `/inbox-link-callback` | Email link callback |
| `/login/popup/success` | Broadcasts `login-success` and closes the popup |
| `/team-invite` | Team invite acceptance |
| `*404` | Redirects native mobile to the default route; web navigates to the origin |
## Runtime initialization
`packages/app/index.tsx` performs the browser entrypoint work:
- Imports global CSS and font packages.
- Initializes Lexical markdown support.
- Initializes monochrome icon behavior.
- Sets `document.documentElement.dataset.platform` to `web`, `desktop`, `ios`, or `android`.
- Tracks current input modality as `keyboard`, `mouse`, or `touch`.
- In Tauri, proxies non-localhost `fetch` calls through `@tauri-apps/plugin-http` for native compatibility.
- In development mode, wraps `Root` in a Solid `ErrorBoundary`.
- Outside Vite HMR sessions, lazy-loads observability and listens for `vite:preloadError` to prompt a refresh after deployments.
`Root` mounts global providers for analytics, PostHog, entities, user context, query sync, undo, channels, calls, quick access, search, chat attachments, notifications, split layout routing, onboarding, and toasts.
## Build and preview
```bash
cd js/app
bun run build
bun run preview
```
Build output is emitted from `packages/app` into `packages/app/dist`. Production-style builds use `/app` as the Vite base.
Additional build recipes:
```bash
cd js/app
just build-dev
just build-staging
just build-prod
```
## Local smoke tests
Prefer the repository-level harness for deterministic local E2E:
```bash
just local-e2e
```
That command starts the local service subset with compose overrides, seeds deterministic data, and runs Playwright with `LOCAL_E2E=true`.
For Playwright UI mode:
```bash
just local-e2e-ui
```
## Start Tauri development
The Tauri shell lives under `js/app/tauri` and packages the same `packages/app` frontend.
```bash
cd js/app/tauri
cargo tauri dev
cargo tauri android dev
cargo tauri ios dev
```
Tauri config uses:
| Config key | Value |
| --- | --- |
| `build.devUrl` | `http://localhost:3000` |
| `build.frontendDist` | `../../packages/app/dist` |
| `beforeDevCommand` | `just dev-tauri` from `js/app` |
| `beforeBuildCommand` | `just build-tauri` from `js/app` |
Set `TAURI_DEV_HOST` when a device or emulator needs HMR to connect to a host other than `localhost`.
```bash
TAURI_DEV_HOST=192.168.1.20 cargo tauri android dev
```
## Contributor checks
Common frontend checks from `js/app/AGENTS.md`:
```bash
cd js/app
bun run test
bun run check
bun run lint
bun run format
bun run knip
```
## Next
- Use `RUNNING_LOCALLY.md` for backend stack setup.
- Use `js/app/AGENTS.md` for frontend contribution conventions.
- Use `js/app/tauri/src-tauri/README.md` for native shell development notes.
---
## 05. Local E2E smoke tests
> Guide: Run deterministic Playwright and ignored Rust smoke tests against the local-only stack and shared seed fixtures.
- Page Markdown: https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/05-local-e2e-smoke-tests.md
- Generated: 2026-06-01T00:49:10.463Z
### Source Files
- `RUNNING_LOCALLY.md`
- `justfile`
- `docker-compose.local-e2e.yml`
- `rust/cloud-storage/seed_cli/README.md`
- `js/app/tests/e2e/fixtures/local-e2e-seed.ts`
- `rust/cloud-storage/integration_tests/local_e2e/README.md`
- `rust/cloud-storage/local_e2e_test_support/README.md`
---
title: "Local E2E smoke tests"
description: "Guide: Run deterministic Playwright and ignored Rust smoke tests against the local-only stack and shared seed fixtures."
---
`just local-e2e`, `just local-e2e-rust`, and `just local-e2e-all` start the Macro local stack with `docker-compose.local-e2e.yml` overrides, reset and seed deterministic fixtures from `rust/cloud-storage/seed_cli/seed`, then run Playwright or ignored Rust integration smoke suites against localhost services.
## Prerequisites
Run the normal local setup first:
```bash
just setup
```
The harness expects the same local toolchain used by the repository: Docker, Bun, Rust/Cargo, AWS CLI, SQLx tooling, Pulumi, and a decrypted `.env`. Local Docker resources are intentionally frozen to the `macro` Compose project, so do not run two local stacks from different checkouts at the same time.
<Warning>
The local E2E seed path is destructive for its deterministic fixture ranges. The seed command is guarded by `LOCAL_E2E_SEED=true` and rejects database URLs outside the local Docker database shape `postgres://user:...@(localhost|127.0.0.1|postgres):5432/macrodb`.
</Warning>
## Command reference
| Command | Runs | Notes |
| --- | --- | --- |
| `just local-e2e` | Playwright local smoke tests | Starts LocalStack, starts the local service subset, seeds fixtures, then runs `LOCAL_E2E=true bunx playwright test` from `js/app`. |
| `just local-e2e-ui` | Playwright UI mode | Same setup and seed pass, then opens Playwright UI mode. |
| `just local-e2e-rust` | Ignored Rust integration tests | Runs `SQLX_OFFLINE=true cargo test -p local_e2e_integration_tests -- --ignored --nocapture`. |
| `just local-e2e-all` | Rust tests, then Playwright | Starts and seeds once, runs the ignored Rust suite, then runs Playwright. |
| `just local-e2e-seed` | Seed only | Starts local Postgres/Redis, drops and recreates Macro DB state, initializes DBs, then runs the local E2E seed scenario. |
Forward Playwright file filters or flags through the `just` target:
```bash
just local-e2e tests/e2e/local-smoke.spec.ts
just local-e2e --project=local-chromium
```
For Rust filters, pass the test name after the target:
```bash
just local-e2e-rust channel_message_posts_to_http_and_delivers_to_websocket
```
## Runtime shape
```text
just local-e2e*
├─ setup_localstack
│ ├─ LocalStack on localhost:4566
│ ├─ SQS queues
│ ├─ DynamoDB tables
│ └─ S3 buckets + doc-storage notification wiring
├─ COMPOSE_FILE=docker-compose.yml:docker-compose.local-e2e.yml just run_local -d --wait ...
│ ├─ Postgres localhost:5432
│ ├─ Redis localhost:6379
│ └─ Macro service subset on local ports
├─ just local-e2e-seed
└─ Playwright and/or ignored Rust tests
```
The compose override pins application services to local Postgres, Redis, and LocalStack instead of shared dev assets. It also supplies deterministic bucket, queue, table, AWS credential, and region values for the local smoke stack.
The startup service subset is defined in the root `justfile` as:
```text
authentication-service
connection_gateway
contacts_service
document_storage_service
email_service
notification_service
static_file_service
static_file_cdn
sync_service
websocket_service
```
Common local ports used by the tests include:
| Surface | Default local endpoint |
| --- | --- |
| Postgres | `postgres://user:password@localhost:5432/macrodb` |
| Redis | `redis://localhost:6379` |
| LocalStack | `http://localhost:4566` |
| Authentication service | `http://localhost:8080` |
| Connection gateway WebSocket | `ws://localhost:8082/` |
| Document storage service | `http://localhost:8086` |
| Notification service | `http://localhost:8089` |
| Frontend dev server | `http://localhost:${PORT:-3000}/app` |
## Seed fixtures
The shared seed contract lives under `rust/cloud-storage/seed_cli/seed`.
| File | Purpose |
| --- | --- |
| `local_e2e/manifest.json` | Stable smoke aliases: primary user email, project roadmap document, general channel, and canonical welcome message. |
| `local_e2e/users.json` | Local users including `e2e@macro.local`, `bob@example.com`, `charlie@example.com`, `dana@example.com`, and `eve@example.com`. |
| `local_e2e/reset.sql` | Deletes deterministic local E2E channels, channel access rows, mentions, activity, and documents before reseeding. |
| `documents/documents.json` | Seeded document metadata. |
| `channels.json` | Seeded channels. |
| `channel_messages.json` | Seeded channel messages. |
The seed CLI scenario resets fixture ranges, inserts users and verification rows, then seeds documents, channels, and channel messages. A successful seed prints:
```text
Local e2e smoke seed data ready for macro|e2e@macro.local
```
## Playwright local smoke suite
When `LOCAL_E2E=true`, `js/app/playwright.config.ts` switches to a local-only browser project named `local-chromium`. The Playwright web server command sets:
```bash
PORT=${PORT:-3000}
VITE_LOCAL_SERVERS=ALL
VITE_ENABLE_BEARER_TOKEN_AUTH=true
LOCAL_JWT=<generated token>
bun run dev
```
If `LOCAL_JWT` is not already exported, Playwright runs `js/app/scripts/generate-local-e2e-token.ts`. That script calls the Rust `local_e2e_test_support` binary `generate_local_e2e_token`, which loads the shared seed user and signs a Macro API token using `.env` or process environment values. The default local token lifetime is eight hours.
The local Playwright specs skip unless `LOCAL_E2E=true`:
| Spec | Main verification |
| --- | --- |
| `local-smoke.spec.ts` | Documents list shows the seeded Project Roadmap; channels list and channel page show the seeded general channel and welcome message. |
| `local-sidebar.spec.ts` | Sidebar routes open expected list views and show seeded entities where applicable. |
| `local-channel-actions.spec.ts` | Sends a message, reacts with `👍`, and opens the reply composer in the seeded channel. |
Tests import `localE2ESeed` from `js/app/tests/e2e/fixtures/local-e2e-seed.ts`. That fixture loader reads the Rust seed JSON files directly and exposes raw arrays, lookup maps, and `smoke` aliases.
## Rust local E2E suite
The Rust suite lives in `rust/cloud-storage/integration_tests/local_e2e` and is intentionally marked `#[ignore]` so normal workspace test runs do not require Docker services. Run it through:
```bash
just local-e2e-rust
```
The crate uses `local_e2e_test_support` to load the same seed files, generate local JWTs, and resolve local service endpoints. The support crate rejects non-local service URLs for mutating tests.
Supported local overrides include:
| Environment variable | Default | Constraint |
| --- | --- | --- |
| `LOCAL_E2E_DOCUMENT_STORAGE_URL` | `http://localhost:8086` | Must use `http` or `https` and a localhost host. |
| `LOCAL_E2E_CONNECTION_GATEWAY_WS_URL` | `ws://localhost:8082/` | Must use `ws` or `wss` and a localhost host. |
| `LOCAL_E2E_NOTIFICATION_URL` | `http://localhost:8089` | Must use `http` or `https` and a localhost host. |
| `LOCAL_E2E_DATABASE_URL` | `postgres://user:password@localhost:5432/macrodb` | Used by Rust verification queries. |
| `LOCAL_E2E_CHANNELS_BASE_URL` | `${document_storage_url}/channels` | Channel mutation API under test. |
| `LOCAL_E2E_CHANNELS_READ_BASE_URL` | `${document_storage_url}/channels` | Channel read API under test. |
The ignored Rust tests cover channel mutations, persistence checks, notifications/contacts side effects where workers are available, and connection gateway WebSocket delivery.
## Running Playwright directly
Prefer the repository-level harness. If you need a manual loop, start and seed the stack first:
```bash
AWS_ACCESS_KEY_ID=test AWS_SECRET_ACCESS_KEY=test AWS_DEFAULT_REGION=us-east-1 just setup_localstack
COMPOSE_FILE=docker-compose.yml:docker-compose.local-e2e.yml just run_local -d --wait authentication-service connection_gateway contacts_service document_storage_service email_service notification_service static_file_service static_file_cdn sync_service websocket_service
just local-e2e-seed
cd js/app
LOCAL_E2E=true bunx playwright test
```
You can bypass token generation by exporting `LOCAL_JWT`, but the generated token path is the default and keeps Playwright aligned with the Rust helper.
## Troubleshooting
### `Failed to generate LOCAL_JWT for LOCAL_E2E Playwright`
Ensure `.env` exists from `just get_environment` or `just setup`, then rerun the repo-level harness:
```bash
just local-e2e
```
If running Playwright directly, seed first with `just local-e2e-seed`. As a temporary bypass, export `LOCAL_JWT`.
### Seed refuses to run
Use `just local-e2e-seed` rather than invoking the seed CLI manually. The recipe sets `LOCAL_E2E_SEED=true`, local AWS values, and the expected local database URL.
### Tests refuse a service URL
Rust helpers reject non-local service URLs. Check `LOCAL_E2E_DOCUMENT_STORAGE_URL`, `LOCAL_E2E_CONNECTION_GATEWAY_WS_URL`, and `LOCAL_E2E_NOTIFICATION_URL`; they must point at localhost or `127.0.0.1`.
### Local stack conflicts across checkouts
The Compose project name, networks, and volumes are shared as `macro`. Stop the current stack before starting another checkout:
```bash
just stop-local
just stop-databases
```
## Adding local E2E coverage
1. Put stable shared fixture rows in `rust/cloud-storage/seed_cli/seed`.
2. Add only aliases to `local_e2e/manifest.json`; keep the source data in the fixture files.
3. Use `localE2ESeed` in Playwright and `LocalE2eSeed` in Rust instead of duplicating IDs.
4. Keep Playwright specs gated by `LOCAL_E2E=true`.
5. Keep Rust integration tests marked `#[ignore]`.
6. Use generated unique text or IDs for mutating assertions so repeated smoke runs can coexist with the deterministic base seed.
## Related pages
- `RUNNING_LOCALLY.md`
- `rust/cloud-storage/seed_cli/README.md`
- `rust/cloud-storage/integration_tests/local_e2e/README.md`
- `rust/cloud-storage/local_e2e_test_support/README.md`
---
## 06. Monorepo map
> Concept: Workspace boundaries, package managers, major runtime folders, generated documentation, and where source-of-truth manifests live.
- Page Markdown: https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/06-monorepo-map.md
- Generated: 2026-06-01T00:50:29.936Z
### Source Files
- `rust/cloud-storage/Cargo.toml`
- `rust/sync-service/Cargo.toml`
- `js/package.json`
- `js/app/package.json`
- `infra/README.md`
- `docs/docs.json`
---
title: "Monorepo map"
description: "Concept: Workspace boundaries, package managers, major runtime folders, generated documentation, and where source-of-truth manifests live."
---
Macro is organized as several adjacent workspaces rather than one repository-wide package workspace: Rust backend services live under `rust/cloud-storage`, the Cloudflare Durable Object sync worker lives under `rust/sync-service`, the Solid/Tauri app and JS services live under `js`, Pulumi infrastructure lives under `infra`, and the Mintlify documentation site lives under `docs`.
## Repository boundary map
```text
.
├── justfile # Local orchestration entrypoint
├── docker-compose*.yml # Local service, database, and E2E topology
├── flake.nix # Nix shells and cached Rust/API build jobs
├── rust/
│ ├── rust-toolchain.toml # Rust toolchain source of truth
│ ├── cloud-storage/ # Main Rust Cargo workspace
│ └── sync-service/ # Cloudflare Worker/Durable Object sync runtime
├── js/
│ ├── package.json # Bun workspace for app + selected JS packages
│ ├── app/ # Solid app, package libraries, tests, Tauri frontend
│ ├── lexical-core/ # Shared Lexical package
│ ├── lexical-service/ # Wrangler service for Lexical conversion
│ ├── websocket-service/ # Standalone Bun WebSocket service
│ └── analytics-proxy/ # Standalone Wrangler proxy
├── infra/
│ ├── package.json # Bun workspace for Pulumi stacks/packages
│ ├── stacks/ # Deployable Pulumi projects
│ └── packages/ # Shared Pulumi resource libraries
└── docs/
├── docs.json # Rendered docs navigation/config
├── package.json # Mintlify scripts
└── AI/mcp/tools/ # Generated MCP tool reference pages
```
<Info>
The repository root is orchestration-oriented. Do not infer JavaScript workspace membership from the root `package.json`; the active JS workspaces are declared in `js/package.json`, `js/app/package.json` scripts, and `infra/package.json`.
</Info>
## Package managers and workspace manifests
| Boundary | Source-of-truth manifest | Package manager/tooling | Lock/config files | Scope |
| --- | --- | --- | --- | --- |
| Rust backend services | `rust/cloud-storage/Cargo.toml` | Cargo, `just`, SQLx | `rust/cloud-storage/Cargo.lock`, `.sqlx/`, `rust/rust-toolchain.toml` | Axum services, Lambda handlers, model crates, service clients, database migrations |
| Sync worker | `rust/sync-service/Cargo.toml`, `rust/sync-service/wrangler.toml` | Cargo, Wrangler, npm for Docker install | `rust/sync-service/Cargo.lock`, `package.json`, D1 migrations | Cloudflare Worker with Durable Objects, D1, R2, KV, Bebop-generated bindings |
| JS app workspace | `js/package.json`, `js/app/package.json` | Bun `1.3.5` at `js/` | `js/bun.lock`, `js/app/bun.lock`, `js/app/biome.jsonc`, `js/app/tsconfig.json` | Solid app, package libraries, generated service clients, Playwright/Vitest tests |
| Tauri app | `js/app/tauri/Cargo.toml`, `js/app/tauri/src-tauri/tauri.conf.json` | Cargo, Tauri CLI, Bun build hooks | `js/app/tauri/Cargo.lock` | Desktop/mobile shell around `js/app/packages/app/dist` |
| Infrastructure | `infra/package.json` | Bun `1.2.0`, Pulumi, TypeScript | `infra/bun.lock`, `infra/tsconfig.json`, per-stack `Pulumi*.yaml` | AWS infrastructure stacks and shared resource packages |
| Docs | `docs/docs.json`, `docs/package.json` | Bun, Mintlify CLI | generated `docs/config/tool-pages.json` | Product docs and generated MCP tool reference |
| Nix shells/builds | `flake.nix` | Nix, Fenix, Crane | `flake.lock` | Reproducible Rust, JS app, Pulumi, SQLx, Bun, pnpm, and Tauri tools |
### Nested package-manager exceptions
- `js/loro-mirror/package.json` has pnpm scripts and pnpm overrides. It is included by the `js/package.json` workspace list, but its own scripts call `pnpm`.
- `js/websocket-service` and `js/analytics-proxy` have package manifests but are not listed in the `js/package.json` workspace array.
- `infra/stacks/web-app` has a Pulumi project, but `infra/tsconfig.json` explicitly excludes `stacks/web-app` from the infra TypeScript check.
## Major runtime folders
| Folder | Runtime role | Primary manifests |
| --- | --- | --- |
| `rust/cloud-storage` | Main Rust backend workspace for services such as authentication, document storage, document cognition, email, notifications, MCP, search, static files, image proxy, unfurling, and Lambda-style workers | `Cargo.toml`, `justfile`, `database.just`, `sqlx.just`, crate-level `Cargo.toml` files |
| `rust/cloud-storage/macro_db_client` | MacroDB client and SQL migrations | `Cargo.toml`, `migrations/*.sql`, crate `justfile` |
| `rust/cloud-storage/ai_tools` | Rust AI/MCP tool registry used by app and docs generators | `Cargo.toml`, `src/`, generated `schemas/tools.json` after running schema generation |
| `rust/sync-service` | Cloudflare Worker sync service using Durable Objects, D1, R2, KV, and Bebop bindings | `Cargo.toml`, `wrangler.toml`, `Dockerfile`, `database/user-peer-mapping/migrations`, `bebop/schema.bop` |
| `js/app/packages/app` | Solid/Vite frontend app package | `package.json`, `vite.config.ts`, `tsconfig.json` |
| `js/app/packages/service-clients` | Generated and checked-in frontend API clients | `orval.config.ts`, per-service `openapi.json`, generated client/schema folders |
| `js/app/tauri` | Tauri Rust workspace and plugins for desktop/mobile packaging | workspace `Cargo.toml`, plugin crates, `src-tauri/tauri.conf.json` |
| `js/lexical-service` | Wrangler service for Lexical document conversion | `package.json`, `wrangler` scripts |
| `infra/stacks/*` | Deployable Pulumi projects; run Pulumi from inside a stack directory | `Pulumi.yaml`, `Pulumi.dev.yaml`, `Pulumi.prod.yaml`, `index.ts` |
| `infra/packages/*` | Shared Pulumi abstractions for services, VPC, Lambda, resources, and stack constants | package `src/index.ts` exports |
| `docs` | Mintlify documentation site | `docs.json`, MDX files, generator scripts |
<Warning>
There are two Rust packages named `sync_service` in different workspaces: `rust/sync-service` is the Cloudflare Worker runtime, while `rust/cloud-storage/sync_service` is a crate inside the backend workspace. Keep workspace-relative paths explicit when changing sync code.
</Warning>
## Backend Cargo workspace
`rust/cloud-storage/Cargo.toml` owns the backend Cargo workspace membership and shared dependency versions. Most backend crates set `publish = false` and depend on sibling crates by relative `path`.
Common commands run from `rust/cloud-storage`:
```bash
just check # SQLX_OFFLINE=true cargo check with warnings denied
just clippy # workspace clippy with all features
just format # cargo fmt
just prepare_db # refresh SQLx offline cache through sqlx.just
just build # SQLX_OFFLINE=true cargo build
```
Database lifecycle commands are routed through `rust/cloud-storage/macro_db_client/justfile`:
```bash
just macro_db_client/create_db
just macro_db_client/migrate_db
just macro_db_client/drop_db -y -f
```
MacroDB schema migrations live under `rust/cloud-storage/macro_db_client/migrations`. Several worker/test crates also carry local migrations for their own test schemas.
## Sync worker boundary
`rust/sync-service` is a separate Cargo project compiled to WebAssembly for Cloudflare Workers. Its `wrangler.toml` defines:
| Binding | Purpose |
| --- | --- |
| `DOCUMENT_SYNC_SESSION` | Durable Object class |
| `USER_PEER_MAPPING` | D1 database with migrations under `database/user-peer-mapping/migrations` |
| `DOCUMENT_SNAPSHOT_BUCKET` | R2 snapshot bucket |
| `DOCUMENT_VERSIONING_KV` | KV namespace |
| `SNAPSHOT_STORE_KV` | KV namespace |
| `INTERNAL_API_SECRET_KEY`, `SPS_URL`, `ENVIRONMENT` | Worker runtime variables per environment |
The Dockerfile installs Node 22, `worker-build`, Wrangler dependencies, builds Bebop TypeScript bindings from `bebop/schema.bop`, applies local D1 migrations, and starts Wrangler on port `8787`.
## Frontend and app packages
`js/package.json` declares the Bun workspace boundary:
```json
[
"app",
"app/packages/*",
"app/infra/*",
"loro-mirror",
"lexical-core",
"lexical-service"
]
```
`js/app/package.json` is the app-level command surface:
```bash
cd js/app
bun run dev # Vite app through packages/app
bun run build # Vite production-style build
bun run test # Vitest
bun run local:e2e # Playwright with LOCAL_E2E=true
bun run check # TypeScript + Biome
bun run gen-api # Rust OpenAPI binaries -> service-clients
bun run gen-tools # Rust AI tool schemas -> DCS tool types
```
`js/app/tsconfig.json` includes `packages/**/*`, `../loro-mirror/**/*`, and `../lexical-core/**/*`, and defines path aliases for generated service clients such as `@service-storage/*`, `@service-cognition/*`, `@service-email/*`, and `@service-sync/*`.
### Tauri packaging
The Tauri workspace lives in `js/app/tauri`. `src-tauri/tauri.conf.json` points `frontendDist` at `../../packages/app/dist`, uses `http://localhost:3000` for development, and delegates build hooks back to the app justfile:
```json
{
"beforeDevCommand": { "script": "just dev-tauri", "cwd": "../.." },
"beforeBuildCommand": { "script": "just build-tauri", "cwd": "../.." }
}
```
Use `js/app/justfile` for app-local run modes:
```bash
cd js/app
just local # Vite against local services
just local-dcs # Local cognition service only
just local-dss # Local document storage service only
just build-prod # Production-mode frontend bundle
just dist-archive # Zip dist for native_app_server tests
```
## Infrastructure workspace
Infrastructure code is TypeScript Pulumi under `infra`. The root `infra/package.json` declares Bun workspaces for `stacks/*` and `packages/*`, requires Node `>=21`, and enforces Bun through a `preinstall` script.
Run Pulumi from inside a stack directory:
```bash
cd infra/stacks/document-storage
pulumi up --stack dev
pulumi up --stack prod
```
Operational defaults from the infra README:
| Setting | Value |
| --- | --- |
| Cloud provider | AWS |
| Main AWS region | `us-east-1` |
| Logging provider | Datadog |
| Datadog region | `us-central-1` |
Shared stack constants and exported helpers live in `infra/packages/shared/src/index.ts`; reusable resource constructors live in `infra/packages/resources/src/index.ts`.
## Local orchestration
The root `justfile` is the local runtime entrypoint. It freezes Docker Compose resources under the `macro` project name so multiple checkouts share the same local containers, networks, and volumes.
Primary commands:
```bash
just setup # Environment, networks, LocalStack, DBs, FusionAuth, Rust images
just run_dbs -d # Postgres + Redis
just run_local # Docker Compose services
just run_local --build
just stop-local
just local-e2e
just local-e2e-rust
just local-e2e-all
```
Local service topology is defined by:
| File | Role |
| --- | --- |
| `docker-compose.yml` | Rust services, sync worker, lexical service, WebSocket service, FusionAuth include, service networks |
| `docker-compose-databases.yml` | Postgres/pgvector, Redis Stack, OpenSearch, external volumes |
| `docker-compose.local-e2e.yml` | Local-only overrides for deterministic E2E data and LocalStack resources |
| `local_stack.just` | LocalStack setup recipes imported by the root justfile |
The local E2E harness seeds deterministic data through `rust/cloud-storage/seed_cli` and runs Playwright from `js/app`.
## Generated code and documentation
| Generated surface | Command | Source of truth | Output |
| --- | --- | --- | --- |
| OpenAPI JSON and frontend clients | `cd js/app && bun scripts/generate-api-schema.ts` | Rust service OpenAPI binaries built from `rust/cloud-storage` | `js/app/packages/service-clients/service-*/openapi.json` and `generated/` folders |
| OpenAPI freshness check | `cd js/app && bun scripts/generate-api-schema.ts --check` | Same as above | Fails if generated files differ from Git |
| DCS tool TypeScript/Zod types | `cd js/app && bun run gen-tools` | `rust/cloud-storage/ai_tools` and `gen_tool_schemas` | `js/app/packages/service-clients/service-cognition/generated/tools` |
| MCP docs pages | `cd docs && bun run generate:tools` | Rust AI tool schemas from `rust/cloud-storage/ai_tools` | `docs/AI/mcp/tools/*.mdx`, `docs/config/tool-pages.json` |
| Docs navigation | edit `docs/docs.json` | Mintlify config | Rendered docs tabs/groups/pages |
<Note>
AI tool references are generated from local Rust schema code and checked-in files. The generation path does not require a specific model provider or hosted AI connector.
</Note>
## Source-of-truth checklist
When changing a boundary, update the owning manifest first:
| Change | Update first | Then verify |
| --- | --- | --- |
| Add a Rust backend crate | `rust/cloud-storage/Cargo.toml` | `cd rust/cloud-storage && just check` |
| Add a Rust service OpenAPI client | Rust service `*_openapi` binary and `js/app/scripts/services.ts` | `cd js/app && bun scripts/generate-api-schema.ts --check` |
| Add a database migration | `rust/cloud-storage/macro_db_client/migrations` | `just rust/cloud-storage/macro_db_client/migrate_db` or local setup recipe |
| Add a frontend package | `js/app/packages/<name>/package.json` | `cd js/app && bun run check` |
| Add a Tauri plugin | `js/app/tauri/Cargo.toml` workspace members | Tauri dev/build command |
| Add a Pulumi stack | `infra/stacks/<stack>/Pulumi.yaml` and `infra/package.json` workspace match | `cd infra && bun run check` unless intentionally excluded |
| Add generated MCP docs | Rust tool registry and docs generator | `cd docs && bun run generate:tools && bun run lint` |
| Change local runtime wiring | `docker-compose*.yml` and root `justfile` | `just run_local --build` or `just local-e2e` |
## Related pages
<CardGroup>
<Card title="Local development" href="/development">
Commands and environment notes for running Macro locally.
</Card>
<Card title="MCP overview" href="/AI/mcp/overview">
Product-facing documentation for Macro MCP surfaces.
</Card>
<Card title="MCP tool reference" href="/AI/mcp/tools">
Generated tool pages derived from the Rust tool registry.
</Card>
</CardGroup>
---
## 07. Runtime environments and service URLs
> Concept: Environment values, local versus dev versus production URL resolution, CORS rules, and frontend service selection.
- Page Markdown: https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/07-runtime-environments-and-service-urls.md
- Generated: 2026-06-01T00:49:31.354Z
### Source Files
- `rust/cloud-storage/macro_env/src/lib.rs`
- `rust/cloud-storage/macro_service_urls/src/lib.rs`
- `rust/cloud-storage/macro_cors/src/lib.rs`
- `js/app/packages/core/constant/servers.ts`
- `js/app/scripts/services.ts`
- `docker-compose.local-e2e.yml`
---
title: "Runtime environments and service URLs"
description: "Concept: Environment values, local versus dev versus production URL resolution, CORS rules, and frontend service selection."
---
Macro resolves runtime environments through separate backend, frontend, and tooling selectors: Rust services read `ENVIRONMENT`, the Vite app reads `import.meta.env.MODE` plus `VITE_LOCAL_SERVERS`, and generation scripts read Node process variables such as `MODE` and `LOCAL_BACKEND`.
## Runtime selectors
| Surface | Selector | Accepted values | Default behavior |
| --- | --- | --- | --- |
| Rust services | `ENVIRONMENT` | `prod`, `dev`, `local` | `Environment::new_or_prod()` falls back to `Production` if the value is missing or invalid. |
| Rust local app URL | `FRONTEND_PORT` | port string | Defaults to `3000` for local app redirects and local app URL construction. |
| Frontend runtime | `MODE` via Vite config | commonly `development` or `production` | `development` enables local-server selection logic; any other mode uses remote hosts. |
| Frontend local overrides | `VITE_LOCAL_SERVERS` | empty, `ALL`, or comma-separated service names | Empty in development still points at remote dev services. |
| Client/tool generation | `MODE`, `LOCAL_BACKEND` | `production`, `local`, or fallback | `MODE=production` selects prod schema URLs; `MODE=local` or `LOCAL_BACKEND=true` selects local; otherwise dev. |
<Warning>
For Rust services, missing `ENVIRONMENT` does not mean local. It resolves as production through `new_or_prod()`.
</Warning>
## Backend environment model
The backend environment enum has three variants:
| Variant | `ENVIRONMENT` string | Display string | Meaning in URL resolution |
| --- | --- | --- | --- |
| `Production` | `prod` | `prod` | Public production domains. |
| `Develop` | `dev` | `dev` | Shared dev domains, usually with `-dev` hostnames. |
| `Local` | `local` | `local` | Localhost ports and local WebSocket schemes. |
`macro_entrypoint` loads `.env` when present, reads the environment, and uses it for logging/tracing behavior. Local uses pretty tracing without the OpenTelemetry sidecar; dev and prod initialize OTLP tracing to the Datadog agent endpoint on `127.0.0.1:4317`.
## Backend service URL resolver
Rust code uses `EnvExtMacroServiceUrls` on `macro_env::Environment` for canonical backend URLs.
| Method | Production | Develop | Local |
| --- | --- | --- | --- |
| `app()` | `https://macro.com` | `https://dev.macro.com` | `http://localhost:${FRONTEND_PORT:-3000}` |
| `auth_service()` | `https://auth-service.macro.com` | `https://auth-service-dev.macro.com` | `http://localhost:8080` |
| `pdf_service()` | `https://pdf-service.macro.com` | `https://pdf-service-dev.macro.com` | `http://localhost:4567` |
| `document_storage_service()` | `https://cloud-storage.macro.com` | `https://cloud-storage-dev.macro.com` | `http://localhost:8086` |
| `websocket_service()` | `wss://services.macro.com` | `wss://services-dev.macro.com` | `ws://localhost:6969` |
| `cognition_service()` | `https://document-cognition.macro.com` | `https://document-cognition-dev.macro.com` | `http://localhost:8085` |
| `connection_gateway()` | `wss://connection-gateway.macro.com` | `wss://connection-gateway-dev.macro.com` | `ws://localhost:8082` |
| `notification_service()` | `https://notifications.macro.com` | `https://notifications-dev.macro.com` | `http://localhost:8089` |
| `static_file_service()` | `https://static-file-service.macro.com` | `https://static-file-service-dev.macro.com` | `http://localhost:8100` |
| `unfurl_service()` | `https://unfurl-service.macro.com` | `https://unfurl-service-dev.macro.com` | `http://localhost:8095` |
| `contacts_service()` | `https://contacts.macro.com` | `https://contacts-dev.macro.com` | `http://localhost:8083` |
| `email_service()` | `https://email-service.macro.com` | `https://email-service-dev.macro.com` | `http://localhost:8087` |
| `image_proxy_service()` | `https://image-proxy.macro.com` | `https://image-proxy-dev.macro.com` | `http://localhost:8097` |
The resolver is used by backend features such as auth, GitHub redirects, notification preferences, and email formatting. URL parse tests verify every resolver method returns a valid `Url` for all three environments.
## Frontend service selection
`js/app/packages/core/constant/servers.ts` owns browser-facing runtime hosts.
### Remote host selection
Remote hosts use a suffix based on Vite mode:
| `import.meta.env.MODE` | Remote suffix | Example |
| --- | --- | --- |
| `development` | `-dev` | `https://cloud-storage-dev.macro.com` |
| any other mode | none | `https://cloud-storage.macro.com` |
`auth-logout` is special: development uses a FusionAuth dev logout URL, while production uses `https://auth.macro.com/oauth2/logout...`.
### Local host selection
Local hosts are only selected when `MODE === 'development'`.
| `VITE_LOCAL_SERVERS` | Result |
| --- | --- |
| unset or empty | Use remote dev hosts. |
| `ALL` | Use every local frontend host. |
| `document-storage-service,email-service` | Use local hosts for those names and remote dev hosts for the rest. |
| `document-storage-service:9000` | Use the local URL for that service but replace its port with `9000`. |
| unknown service name | Throw `Error("unknown server name ...")`. |
Example:
```bash
MODE=development VITE_LOCAL_SERVERS=document-storage-service:9000,email-service bun run dev
```
Frontend local host keys include the backend services plus `auth-logout` and `scheduled-action`. Runtime code imports `SERVER_HOSTS` directly in generated service clients, auth flows, WebSocket clients, image proxying, observability, and static-file helpers.
<Note>
The frontend local `static-file` host is `http://localhost:8100`, which points at the local CDN/nginx layer. The static file service container itself is exposed on `8094` for API health and OpenAPI-related usage.
</Note>
## Sync service host selection
The sync service has a separate host map:
| Mode | Worker URL | WebSocket URL |
| --- | --- | --- |
| Local | `http://localhost:8787` | `ws://localhost:8787` |
| Development remote | `https://sync-service-dev3.macroverse.workers.dev` | `wss://sync-service-dev3.macroverse.workers.dev` |
| Production remote | `https://sync-service-prod2.macroverse.workers.dev` | `wss://sync-service-prod2.macroverse.workers.dev` |
Local sync is selected when the frontend is in development mode and `VITE_LOCAL_SERVERS` is `ALL` or contains `sync-service`.
`SYNC_PERMISSION_TOKEN_DSS_HOST` intentionally follows the sync service. If sync is remote, permission tokens come from remote document storage so the token is signed with the JWT secret that matches the remote sync service. If sync is local, it uses the selected document storage host from `SERVER_HOSTS`.
## Local Docker and local E2E
`docker-compose.yml` exposes most Rust services on stable localhost ports while containers listen internally on `8080`. Important browser-facing ports include:
| Service | Local port |
| --- | --- |
| `authentication-service` | `8080` |
| `connection_gateway` | `8082` |
| `contacts_service` | `8083` |
| `document_cognition_service` | `8085` |
| `document_storage_service` | `8086` |
| `email_service` | `8087` |
| `notification_service` | `8089` |
| `static_file_service` | `8094` |
| `static_file_cdn` | `8100` |
| `unfurl_service` | `8095` |
| `image_proxy_service` | `8097` |
| `websocket_service` | `6969` |
| `sync_service` | `8787` |
Local E2E uses `docker-compose.local-e2e.yml` to override shared dev infrastructure with local Docker dependencies such as Postgres, Redis, and LocalStack. The Playwright local E2E web server starts Vite with:
```bash
VITE_LOCAL_SERVERS=ALL VITE_ENABLE_BEARER_TOKEN_AUTH=true bun run dev
```
The repository-level harness is:
```bash
just local-e2e
```
## CORS rules
Backend Axum services generally install `macro_cors::cors_layer()`; authentication adds the refresh-token header through `cors_layer_with_headers(...)`.
Default CORS behavior:
| Rule | Behavior |
| --- | --- |
| Credentials | Enabled. |
| Methods | `GET`, `POST`, `PUT`, `PATCH`, `DELETE`, `OPTIONS`. |
| Base headers | `authorization`, `content-type`. |
| Extra headers | `x-permissions-token`, `traceparent`, `tracestate`. |
| Additional headers | Per-service callers can pass extra `HeaderName` values. |
| Static origins | Built-in Macro origins include localhost dev, dashboard, dev, staging, production, Tauri, and Apollo testing origins. |
| `ALLOWED_ORIGINS` | Comma-separated env var that replaces the static origin list. |
| Preview origins | HTTPS origins ending in `preview.macro.com` are allowed. |
| Localhost dev ports | `http://localhost:3000` through `http://localhost:3999` are allowed by predicate. |
The sync service has its own Worker-side CORS implementation. It allows a similar but not identical origin set and accepts preview origins shaped like `https://{subdomain}.preview.macro.com`.
## Service-client and schema generation URLs
`js/app/scripts/services.ts` defines service metadata for generated clients and tooling:
- `services[]` maps each service name to dev/prod/local OpenAPI URLs, output directories, and Orval project keys.
- `serviceUrl(service)` selects local when `MODE=local` or `LOCAL_BACKEND=true`, production when `MODE=production`, and dev otherwise.
- `generate-api-schema.ts` uses the service list for names, output directories, and Orval keys, but generates OpenAPI JSON by building and running local Rust OpenAPI binaries.
- `generate-dcs-models.ts` can fetch Document Cognition models from the selected service URL, or read from `MODELS_JSON` when supplied.
## Contribution checklist
When adding or changing a service URL:
<Steps>
<Step title="Update the backend resolver if Rust code needs the URL">
Add a method or branch in `macro_service_urls` and extend the parse tests.
</Step>
<Step title="Update frontend runtime hosts if browser code calls the service">
Add the key to both local and remote maps in `SERVER_HOSTS`; keep the `Servers` type satisfied.
</Step>
<Step title="Update Docker Compose if the service runs locally">
Expose a stable local port and align the frontend local map with the intended browser-facing endpoint.
</Step>
<Step title="Update generation metadata if clients depend on OpenAPI output">
Add or adjust the entry in `scripts/services.ts` and the Orval project in `packages/service-clients/orval.config.ts`.
</Step>
<Step title="Check CORS before exposing browser traffic">
Confirm the app origin is covered by default rules, `ALLOWED_ORIGINS`, the preview-origin predicate, or the localhost `3000-3999` rule.
</Step>
</Steps>
## Troubleshooting
| Symptom | Likely cause | Check |
| --- | --- | --- |
| Local Rust service redirects to `macro.com` | `ENVIRONMENT` is missing or invalid and fell back to production. | Set `ENVIRONMENT=local`. |
| Vite dev server calls shared dev services | `VITE_LOCAL_SERVERS` is empty. | Use `VITE_LOCAL_SERVERS=ALL` or a comma-separated service list. |
| Only one local service should be overridden | `ALL` is too broad. | Use `VITE_LOCAL_SERVERS=service-name` or `service-name:port`. |
| Browser CORS failure on a local frontend port | Port is outside the accepted `3000-3999` range or origin is not in `ALLOWED_ORIGINS`. | Use a `3000-3999` frontend port or configure `ALLOWED_ORIGINS`. |
| Sync permission token fails against remote sync | Token came from local DSS while sync is remote. | Use `SYNC_PERMISSION_TOKEN_DSS_HOST` behavior instead of hard-coding `SERVER_HOSTS['document-storage-service']`. |
---
## 08. Service topology
> Concept: Rust services, workers, queues, storage dependencies, ports, and deployment boundaries used by the local and cloud stacks.
- Page Markdown: https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/08-service-topology.md
- Generated: 2026-06-01T00:50:24.621Z
### Source Files
- `docker-compose.yml`
- `rust/cloud-storage/Cargo.toml`
- `rust/cloud-storage/AGENTS.md`
- `infra/stacks/document-storage/index.ts`
- `infra/packages/resources/src/index.ts`
- `infra/packages/shared/src/ai_tools.ts`
---
title: "Service topology"
description: "Concept: Rust services, workers, queues, storage dependencies, ports, and deployment boundaries used by the local and cloud stacks."
---
`macro-inc/macro` runs the cloud-storage surface as a Rust Cargo workspace deployed either as local Docker Compose containers or as Pulumi-managed AWS ECS/Fargate services, Lambda handlers, SQS queues, S3 buckets, DynamoDB tables, Redis caches, OpenSearch, and Postgres databases.
## Runtime boundaries
The repository has two primary topology descriptions:
| Boundary | Source of truth | Runtime shape |
| --- | --- | --- |
| Local stack | `docker-compose.yml`, `docker-compose-databases.yml`, `docker-compose.local-e2e.yml` | Docker services on bridge networks, one shared Rust service image, local Postgres/Redis/OpenSearch, FusionAuth compose include |
| Cloud stack | `infra/stacks/**`, `infra/packages/resources/**`, `infra/packages/shared/**` | Pulumi stacks that create ECS/Fargate services, ALBs, Route53 records, SQS queues, Lambda handlers, S3 buckets, DynamoDB tables, Redis, OpenSearch, and RDS |
| Rust service workspace | `rust/cloud-storage/Cargo.toml` | One workspace containing HTTP services, queue workers, Lambda handlers, clients, models, and shared infrastructure crates |
<Note>
Most Rust HTTP binaries default to `PORT=8080`. Local Compose maps each container port to a distinct host port; cloud ECS services keep container port `8080` behind HTTPS ALBs.
</Note>
## Local Docker topology
`docker-compose.yml` freezes the Compose project name to `macro`, includes the database and FusionAuth compose files, and builds `macro-local-rust-services:dev` from `rust/cloud-storage/Dockerfile.dev`.
The dev Dockerfile builds a bundle of standard Rust binaries into `/app/out/*` and each service chooses a command from that bundle. `search_processing_service` uses a separate Dockerfile and is pinned to `linux/amd64` because its Pdfium library is amd64-only.
| Compose service | Host port | Container command or runtime | Main dependencies |
| --- | ---: | --- | --- |
| `authentication-service` | `8080` | `/app/out/authentication_service` | Postgres, FusionAuth, Redis |
| `connection_gateway` | `8082` | `/app/out/connection_gateway_service` | Redis |
| `contacts_service` | `8083` | `/app/out/contacts_service` | Postgres/Redis via env |
| `document_cognition_service` | `8085` | `/app/out/document_cognition_service` | Document storage, email, static file, sync, lexical |
| `document_storage_service` | `8086` | `/app/out/document_storage_service` | Redis, connection gateway, auth |
| `document_upload_finalizer` | none | `/app/out/document_upload_finalizer_local_worker` | Postgres, sync, lexical; polls LocalStack SQS |
| `email_service` | `8087` | `/app/out/email_service` | Auth, document storage, connection gateway, static file, Redis |
| `notification_service` | `8089` | `/app/out/notification_service` | Redis, cognition, auth, connection gateway, document storage |
| `search_processing_service` | `8092` | `Dockerfile.search_processing_service.dev` | Email service; profile `processors` |
| `static_file_service` | `8094` | `/app/out/static_file_service` | DynamoDB/S3-style env |
| `static_file_cdn` | `8100` | `nginx:alpine` | Routes `/file/*` to LocalStack S3 and `/api/*` to `static_file_service` |
| `unfurl_service` | `8095` | `/app/out/unfurl_service` | No Compose dependency |
| `image_proxy_service` | `8097` | `/app/out/image_proxy_service` | No Compose dependency |
| `websocket_service` | `6969` | JS WebSocket service | Services network |
| `sync_service` | `8787` | Rust sync-service Dockerfile | Internal API/document permission secrets |
| `lexical_service` | `8096` | Bun/JS lexical-service Dockerfile | Sync service |
### Local infrastructure containers
`docker-compose-databases.yml` provides:
| Service | Ports | Notes |
| --- | ---: | --- |
| `postgres` | `5432` | `pgvector/pgvector:pg16`, external volume `macro_postgres_data` |
| `redis` | `6379`, `8001` | `redis/redis-stack:latest`, external volume `macro_redis_data` |
| `search` | `9200`, `9600` | OpenSearch single-node with security disabled |
| `fusionauth` | `9011` | Included from `infra/stacks/fusionauth-instance/docker-compose.yml`; uses its own Postgres container |
Local networks separate service traffic:
| Network | Purpose |
| --- | --- |
| `services` | Service-to-service HTTP/WebSocket traffic |
| `databases` | Postgres, Redis, OpenSearch, LocalStack-style AWS endpoints |
| `auth` | FusionAuth access from the auth service |
| `auth-internal` | FusionAuth internal database link |
<Warning>
The local document upload finalizer defaults `LOCAL_AWS_URL` to `http://localstack:4566`, but the base Compose files do not define a `localstack` service. Local E2E overrides assume that endpoint exists and provide deterministic bucket, table, and queue names.
</Warning>
## Cloud service boundary
Cloud services are provisioned with Pulumi stacks under `infra/stacks`. Public Rust HTTP services typically follow the same pattern:
1. Build a service-specific ECR image from `rust/cloud-storage/Dockerfile` with `SERVICE_NAME`.
2. Create an ECS/Fargate service in private subnets.
3. Put an Application Load Balancer in public subnets when `isPrivate: false`.
4. Terminate HTTPS on port `443`; redirect HTTP `80` to HTTPS.
5. Forward ALB traffic to container port `8080`.
6. Attach service-specific IAM policies for Secrets Manager, SQS, S3, DynamoDB, SNS, Lambda, or OpenSearch.
```mermaid
flowchart LR
subgraph Local["Local Compose"]
BrowserLocal["Host/browser"]
ComposePorts["Host port mappings"]
RustBundle["macro-local-rust-services:dev"]
LocalDB["Postgres + Redis + OpenSearch"]
Fusion["FusionAuth"]
LocalS3["LocalStack-compatible AWS endpoint"]
end
subgraph Cloud["Pulumi AWS stacks"]
Route53["Route53 + HTTPS ALB"]
ECS["ECS/Fargate Rust services"]
Workers["ECS workers + Lambda handlers"]
Queues["SQS queues + DLQs"]
Storage["S3 buckets + DynamoDB"]
Data["RDS MacroDB + Redis + OpenSearch"]
end
BrowserLocal --> ComposePorts --> RustBundle
RustBundle --> LocalDB
RustBundle --> Fusion
RustBundle --> LocalS3
Route53 --> ECS
ECS --> Data
ECS --> Storage
ECS --> Queues
Queues --> Workers
Storage --> Workers
```
## Core cloud stacks
| Stack | Main resources | Deployment boundary |
| --- | --- | --- |
| `document-storage` | Versioned document S3 bucket, lifecycle rules, replication bucket, ECS cluster `cloud-storage-cluster-${stack}` | Shared storage and cluster foundation for many Rust services |
| `cloud-storage-service` | `document_storage_service` ECS service, DOCX upload bucket, delete document/chat queues, docx unzip Lambda, document upload finalizer Lambda | Main document API plus document upload/conversion event handlers |
| `document-storage-bucket-integrations` | EventBridge S3 Object Created rules and DLQs | Connects document bucket events to search upload, text extraction, and upload finalization Lambdas |
| `document-cognition-service` | `document_cognition_service` ECS service | AI/document cognition API and tool host |
| `email-service` | `email_service` ECS API, `email-service-pubsub-workers` ECS worker service, Redis, attachment bucket, CloudFront, many SQS queues, refresh/scheduled Lambdas | Email API plus queue consumers |
| `connection-gateway` | `connection_gateway` ECS service, Redis, DynamoDB connection table | Realtime connection tracking and message gateway |
| `contacts-service` | `contacts_service` ECS service, contacts queue | Contact API plus SQS/outbox workers |
| `notification-service` | `notification_service` ECS service, notification queues, SNS platform applications, push event handler | Notification API and push/event workers |
| `static-file-service` | `static_file_service` ECS service, static-file S3 bucket, CloudFront, DynamoDB metadata table, S3 event queue, image optimizer Lambda | Static file API/CDN boundary |
| `search-processing-service` | `search_processing_service` ECS service, backfill job DynamoDB table | Search indexing workers plus backfill API |
| `convert-service` | `convert_service` ECS service, convert queue | LibreOffice-based conversion API/worker |
| `opensearch` | OpenSearch domain | Search index storage |
| `macrodb` | RDS Postgres primary and read replica | MacroDB storage boundary |
## Service URLs and stack routing
Cloud service URL values are centralized in `ServiceUrl` and resolved by stack (`dev` or `prod`). Pulumi stacks inject these values as environment variables instead of hard-coding peer endpoints inside Rust binaries.
| Env key | Dev URL | Prod URL |
| --- | --- | --- |
| `DOCUMENT_STORAGE_SERVICE_URL` | `https://cloud-storage-dev.macro.com` | `https://cloud-storage.macro.com` |
| `AUTHENTICATION_SERVICE_URL` | `https://auth-service-dev.macro.com` | `https://auth-service.macro.com` |
| `CONNECTION_GATEWAY_URL` | `https://connection-gateway-dev.macro.com` | `https://connection-gateway.macro.com` |
| `DOCUMENT_COGNITION_SERVICE_URL` | `https://document-cognition-dev.macro.com` | `https://document-cognition.macro.com` |
| `EMAIL_SERVICE_URL` | `https://email-service-dev.macro.com` | `https://email-service.macro.com` |
| `STATIC_FILE_SERVICE_URL` | `https://static-file-service-dev.macro.com` | `https://static-file-service.macro.com` |
| `NOTIFICATION_SERVICE_URL` | `https://notifications-dev.macro.com` | `https://notifications.macro.com` |
| `SYNC_SERVICE_URL` | `https://sync-service-dev3.macroverse.workers.dev` | `https://sync-service-prod2.macroverse.workers.dev` |
| `LEXICAL_SERVICE_URL` | `https://lexical-service-dev.macroverse.workers.dev` | `https://lexical-service.macroverse.workers.dev` |
## Storage dependencies
| Storage | Local implementation | Cloud implementation | Primary consumers |
| --- | --- | --- | --- |
| MacroDB | Postgres `pgvector/pgvector:pg16` on `5432` | RDS Postgres, plus read replica in `macrodb` stack | Document storage, auth, email, contacts, notification, cognition, search processing |
| Redis | Redis Stack on `6379` | ElastiCache Redis or secret-backed shared cache, depending on stack | Auth sessions/rate limits, connection gateway, email rate limits, contacts, notification |
| OpenSearch | Single-node OpenSearch on `9200` | OpenSearch domain with prod VPC placement and multi-AZ settings | Search processing, document storage search paths |
| Document files | LocalStack-compatible S3 endpoint when configured | Versioned S3 bucket from `document-storage` stack | Document storage, search/text extraction, upload finalizer |
| Static files | LocalStack bucket through `static_file_cdn` | `static-file-storage-${stack}` S3 bucket + CloudFront | Static file service, image optimizer |
| Connection state | Local DynamoDB-style table name from env overrides | DynamoDB connection table in `connection-gateway` | Connection gateway, cognition realtime integration |
| Metadata tables | Local DynamoDB-style env names | DynamoDB tables such as `static-file-metadata-${stack}` and search backfill jobs | Static file service, search processing |
## Queues, workers, and event flows
| Queue or event | Producer | Consumer | Notes |
| --- | --- | --- | --- |
| `search-event-queue-${stack}` | Document storage, email, cognition, auth flows | `search_processing_service` | Cloud queue has DLQ and production backlog/age alarms |
| Document text extractor queue | Document cognition | `document_text_extractor` Lambda | Lambda queue has DLQ and event source mapping with batch size `1` |
| Document bucket Object Created | S3 EventBridge | Search upload Lambda, text extractor Lambda, document upload finalizer Lambda | `document-storage-bucket-integrations` creates EventBridge rules and DLQs |
| `document-upload-finalizer-queue` local | LocalStack S3 notification | `document_upload_finalizer_local_worker` | Local worker long-polls SQS, processes up to 10 messages, deletes only after success |
| `convert-service-queue-${stack}` | DOCX unzip/document flows | `convert_service` | Convert service also exposes HTTP health/API and runs a queue worker unless built with `disable_worker` |
| Email queues | Gmail webhooks, scheduled sends, backfill, SFS mapping, link manager | `email_service`, `email-service-pubsub-workers`, refresh/scheduled Lambdas | Email stack owns multiple queues and exports names for other stacks |
| `contacts-queue-${stack}` | Document/email/channel flows | `contacts_service` worker | Contacts service also runs an outbox worker |
| Notification ingress/egress queues | Document/email/cognition/auth flows | `notification_service` | Notification service also integrates SNS platform applications |
| Static file S3 event queue | Static file bucket notifications | `static_file_service` | Used for static file metadata/event handling |
## Document upload and processing path
```mermaid
sequenceDiagram
participant Client
participant DSS as document_storage_service
participant S3 as document-storage S3 bucket
participant EB as EventBridge/SQS
participant Finalizer as document_upload_finalizer
participant Extractor as document_text_extractor
participant Search as search_processing_service
participant OS as OpenSearch
Client->>DSS: Request document upload/update
DSS->>S3: Write object or issue presigned upload path
S3-->>EB: Object Created
EB-->>Finalizer: Finalize versioned object
EB-->>Extractor: Extract document text
DSS-->>EB: Publish search event when applicable
EB-->>Search: Search event queue message
Search->>S3: Read source content
Search->>OS: Index searchable representation
```
Local development replaces the EventBridge Lambda path with the `document_upload_finalizer_local_worker` polling a LocalStack SQS queue. Cloud uses EventBridge rules and Lambda targets with DLQs.
## AI tool hosting topology
`getAiToolsInfra()` defines the shared infrastructure contract for services that host the Rust `ai_tools` crate. It returns:
| Field | Meaning |
| --- | --- |
| `envVars` | Service URLs, bucket names, queue names, CloudFront signer names, and internal auth values required by `ai_tools` context construction |
| `secretArns` | Secrets Manager ARNs for sync auth, CloudFront signing, and MCP credentials |
| `queueArns` | Queue access required by hosted tools, including email scheduled queue |
| `bucketArns` | Document storage and DOCX upload bucket permissions |
`getAiToolsServiceRoleArns()` centralizes the IAM role ARNs for tool-hosting services such as MCP server, document cognition, and agent schedule service. Resource-side policies can grant access to that set without coupling the topology to one model provider.
<Info>
The AI tool infrastructure contract is provider-neutral at the topology layer: services receive AWS resources, service URLs, secrets, queues, and buckets. Model-provider keys are service-specific environment or secret inputs, not a requirement of the shared tool-hosting boundary.
</Info>
## Health checks and verification signals
| Runtime | Health path |
| --- | --- |
| Most Rust HTTP services | `/health` |
| `static_file_service` | `/api/health` |
| FusionAuth local container | `/api/status` |
| `sync_service` | `/health` on port `8787` |
| `lexical_service` | `/health` on port `8096` |
Local Compose health checks use the container port, usually `8080`, even when the host port differs. Cloud ALB target groups use the configured `healthCheckPath` and forward to the ECS task container port.
## Operations notes
### Local startup constraints
- Create the external Docker networks and volumes expected by Compose before starting the full stack.
- Use the `processors` profile only when the local machine can run the amd64 search processing image or QEMU emulation.
- Use `COMPOSE_FILE=docker-compose.yml:docker-compose.local-e2e.yml` for deterministic local E2E environment overrides.
- Ensure a LocalStack-compatible endpoint exists when using S3/SQS-backed local flows.
### Cloud deployment constraints
- New service environment variables must be added to the matching Pulumi stack so ECS tasks or Lambda handlers receive them.
- Services that call AWS APIs need both runtime environment values and IAM policy coverage.
- Queue-backed workflows should define DLQs and alarms; the shared `Queue` component creates a queue, DLQ, and DLQ alarm by default.
- Rust services use `SQLX_OFFLINE=true` during Docker builds; SQLx query metadata must be prepared from the Rust workspace when database queries change.
## Related pages
<CardGroup>
<Card title="Document storage" href="/architecture/document-storage">
Document API, bucket layout, upload/finalization flow, and storage permissions.
</Card>
<Card title="Search indexing" href="/architecture/search-indexing">
Search event queues, extraction workers, OpenSearch indexes, and backfill operations.
</Card>
<Card title="Local development stack" href="/development/local-stack">
Compose startup, environment overrides, LocalStack setup, and health checks.
</Card>
</CardGroup>
---
## 09. Entities and blocks
> Concept: Item types, document-backed and non-document blocks, aliases, load sources, nesting rules, and file type resolution.
- Page Markdown: https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/09-entities-and-blocks.md
- Generated: 2026-06-01T00:51:49.790Z
### Source Files
- `js/app/packages/core/block.ts`
- `js/app/packages/core/constant/allBlocks.ts`
- `js/app/packages/block-md/definition.ts`
- `js/app/packages/block-project/definition.ts`
- `js/app/packages/service-clients/service-storage/client.ts`
- `rust/cloud-storage/models_soup/src/lib.rs`
---
title: "Entities and blocks"
description: "Concept: Item types, document-backed and non-document blocks, aliases, load sources, nesting rules, and file type resolution."
---
Macro maps storage-service items and repository-defined block definitions into Solid block instances. The frontend resolves an entity-like item to a `BlockName` or `BlockAlias`, creates a `Source`, runs the selected block definition’s `load(source, intent)`, and exposes the loaded data through block-scoped signals.
## Core identifiers
| Identifier | Purpose |
| --- | --- |
| `BlockName` | Concrete block type from `BlockRegistry`, such as `pdf`, `md`, `code`, `image`, `canvas`, `chat`, `project`, `channel`, `email`, `video`, `automation`, and `unknown`. |
| `BlockAlias` | Pseudo-block type that preserves UX semantics while reusing a base block implementation. Current aliases are `task` and `csv`. |
| `ItemType` | Storage/client item category: cloud storage item types (`document`, `chat`, `project`) plus `channel`, `email`, `channel_message`, `call`, and `automation`. |
| `Source` | Runtime loading input. Remote sources are `dss` and `sync-service`; local/intermediate sources include `blob`, `buffer`, `opfs`, `gen`, and `preload`. |
| `SoupItem` | Backend aggregate enum for list/search results: documents, chats, projects, email threads, channels, calls, CRM companies, and foreign entities. |
<Warning>
`BlockRegistry` contains `write` and `contact`, but this checkout does not include `block-write/definition.ts` or `block-contact/definition.ts`. Treat them as registry-level names, not loadable block implementations, until definition modules are added.
</Warning>
## Block definitions
Block definitions are collected with `import.meta.glob('../../block-*/definition.ts')`. Each definition provides:
```ts
defineBlock({
name,
description,
component,
accepted,
load,
defaultFilename?,
aliases?,
liveTrackingEnabled?,
syncServiceEnabled?,
editPermissionEnabled?,
})
```
| Block | Accepted file types | Primary load source | Notes |
| --- | --- | --- | --- |
| `pdf` | `pdf`, `docx` | `dss` | Fetches binary document data, downloads the blob, and initializes pdf.js. |
| `md` | `md` | `sync-service` | Uses backend-owned sync-service content and Loro state. Alias: `task`. |
| `code` | Code extensions from `FileTypeMap` where `app === 'code'` | `dss` | Loads text via `getTextDocument`. Alias: `csv`. |
| `image` | `png`, `jpg`, `jpeg`, `gif`, `svg`, `webp` | `dss` | Fetches binary data and builds a DSS file wrapper. |
| `canvas` | `canvas` | `dss` | Fetches binary canvas data and builds a DSS file wrapper. |
| `video` | Video extensions when `ENABLE_VIDEO_BLOCK` is enabled | `dss` | Fetches metadata and a presigned playback URL for supported formats. |
| `unknown` | none | `dss` | Metadata-only fallback for unsupported document file types. |
| `project` | none | `dss` | Special-cases `root` and `trash`; otherwise fetches project metadata. |
| `chat` | none | `dss` | Fetches chat data from cognition and synthesizes document-like metadata. |
| `channel` | none | `dss` | Loads by channel id only. |
| `email` | none | `dss` | Fetches and caches an email thread. |
| `call` | none | `dss` | Enabled only when `ENABLE_CALLS()` returns true. |
| `automation` | none | `dss` | Loads an automation schedule id. |
## Document-backed and non-document blocks
`NonDocumentBlockTypes` marks blocks that do not correspond to document file content:
```ts
[
'call',
'chat',
'channel',
'project',
'email',
'contact',
'automation',
]
```
Everything else normally behaves as document-backed content. Aliases are not listed as non-document blocks: `task` resolves to `md`, and `csv` resolves to `code`, so both remain document-backed.
`blockNameToItemType()` converts blocks back to item types for history and storage operations:
| Block or alias | Item type |
| --- | --- |
| `chat` | `chat` |
| `call` | `call` |
| `channel` | `channel` |
| `project` | `project` |
| `email` | `email` |
| `automation` | `automation` |
| all other blocks and aliases | `document` |
## Aliases
Aliases are declared inside block definitions, not in a separate routing table.
| Alias | Base block | Default filename | Meaning |
| --- | --- | --- | --- |
| `task` | `md` | `New Task` | A markdown document with task subtype metadata. |
| `csv` | `code` | `New CSV` | A text/code document presented as a CSV-specific entity. |
Alias handling has two layers:
1. `fileTypeToBlockName()` may return the alias when the input is an alias or when an item `subType.type` is an alias.
2. `resolveBlockAlias()` flattens aliases to the concrete block implementation before mounting or calling block methods.
Split-layout URLs preserve aliases with `aliasContext`, so a route can encode `/task/:id` while the mounted implementation is still the `md` block. Duplicate split checks compare resolved block names, preventing the same item from opening twice as both `/md/:id` and `/task/:id`.
## File type resolution
`allBlocks.ts` builds lookup tables from every block definition’s `accepted` map:
| Lookup | Output |
| --- | --- |
| `blockAcceptedMimetypeToFileExtension` | MIME type to first registered extension. |
| `blockAcceptedFileExtensionToMimeType` | Extension to MIME type. |
| `blockAcceptedFileExtensionSet` | All accepted extensions. |
| `blockNameToFileExtensions` | Block name to accepted extensions. |
| `blockNameToMimeTypes` | Block name to accepted MIME types. |
| `fileTypeToBlockName()` | Block name, alias, or `unknown`. |
| `fileTypeToResolvedBlockName()` | Concrete block name with aliases flattened. |
Resolution order for `fileTypeToBlockName(input, icon?)`:
1. Missing input returns `unknown`.
2. `channel_message` maps to `channel`.
3. With `ENABLE_DOCX_TO_PDF`, `docx` and `write` map to `pdf` for behavior, but `icon: true` returns `write`.
4. Alias names return the alias.
5. Registered block names return themselves.
6. Accepted file extensions map to their owning block.
7. Unrecognized input returns `unknown`.
`itemToBlockName(item, icon?)` gives document subtypes precedence over file types. If `item.subType.type` is a known alias, it returns that alias; otherwise it resolves `item.fileType`, then falls back to `item.type`.
## Load sources and lifecycle
The orchestrator’s default source resolver returns `sync-service` when a block definition has `syncServiceEnabled`; otherwise it returns a `dss` source with the id and optional upload metadata from router state.
```text
item or route
-> block/file-type resolution
-> createBlockInstance(type, id)
-> Source: sync-service or dss
-> definition.load(source, 'initial')
-> BlockLoader signals
```
`BlockLoader` rejects nested preload results during initial load, records load errors as `UNAUTHORIZED`, `MISSING`, `GONE`, or `INVALID`, and publishes common fields when present:
| Loaded field | Published signal behavior |
| --- | --- |
| `dssFile` | Sets block file signal. |
| `text` | Sets block text signal. |
| `userAccessLevel` | Sets user access; defaults to `view` when absent. |
| `documentMetadata` | Sets document metadata. |
| `projectMetadata` | Converted into a document-metadata-shaped value for shared UI. |
| `loroManager` | Sets collaborative document state manager. |
| `syncSource` | Registered for cleanup on unmount. |
Markdown is the main sync-service-backed block. It requires the storage location to be `syncServiceContent`; object-storage markdown content is treated as invalid because markdown initialization and persistence are backend-owned.
## Backend entity shape
The Rust `SoupItem` enum is the backend list/search aggregate. Each variant can produce a canonical `Entity` with an entity type and string id. Documents with `SoupDocumentSubType::Task` report property entity type `Task`; ordinary documents report `Document`.
Property references are only available for documents, projects, email threads, and chats. Channels, calls, CRM companies, and foreign entities intentionally return no property reference in `to_entity_reference()`.
## Nesting rules
Nested blocks are gated by `ValidNestingCombinations` and checked through `canNestBlock(name, parentName)`. The rule is keyed by child block and stores allowed parent blocks.
| Child block | Allowed parent blocks |
| --- | --- |
| `canvas` | `md` |
| `pdf` | `md` |
| `code` | `md` |
| all other registered children | none |
Alias-aware callers resolve aliases before checking nesting. For example, `task` resolves to `md`, and `csv` resolves to `code`; a CSV/code preview can be nested in markdown if the caller resolves it to `code`.
## Creation and upload paths
Document creation and upload use storage-service client methods:
| Method | Endpoint | Use |
| --- | --- | --- |
| `createDocument()` | `POST /documents` | Generic object-storage document upload. Supports `fileType`, `mimeType`, `isTask`, `teamId`, and related metadata. |
| `createMarkdownDocument()` | `POST /documents/create_markdown` | Backend-initialized markdown sync-service document. |
| `createTask()` | `POST /documents/create_task` | Backend-initialized task document with task properties. |
Upload file type inference first uses an explicit `options.fileType`, then MIME-to-extension lookup from block definitions, then the filename extension when the MIME type is empty. The backend owns object-created finalization and sync-service initialization after upload.
## Related pages
<CardGroup>
<Card title="Docs" href="/product/docs">
Real-time collaborative markdown documents.
</Card>
<Card title="Tasks" href="/product/tasks">
Task documents and task-specific properties.
</Card>
<Card title="Files" href="/product/files">
Stored files and upload behavior.
</Card>
</CardGroup>
---
## 10. Split layout and navigation
> Concept: Route encoding, component registry entries, split history, navigation causes, popovers, and desktop versus mobile split behavior.
- Page Markdown: https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/10-split-layout-and-navigation.md
- Generated: 2026-06-01T00:52:49.230Z
### Source Files
- `js/app/packages/app/component/Root.tsx`
- `js/app/packages/app/component/split-layout/SplitLayoutRoute.tsx`
- `js/app/packages/app/component/split-layout/componentRegistry.tsx`
- `js/app/packages/app/component/split-layout/layoutManager.ts`
- `js/app/packages/app/component/split-layout/layout.ts`
- `js/app/packages/app/component/split-layout/tests/layoutManager.test.ts`
---
title: "Split layout and navigation"
description: "Concept: Route encoding, component registry entries, split history, navigation causes, popovers, and desktop versus mobile split behavior."
---
The app renders workspace content through a Solid Router catch-all split route whose path segments encode the visible split contents as `type/id` pairs, while `createSplitLayout` owns mounted content, per-split history, focus events, popovers, URL synchronization, and desktop/mobile layout decisions.
## Route encoding
A split URL is parsed as alternating path segments:
```text
/component/inbox/md/doc_123
└─ type ─┘└ id ┘└type┘└ id ┘
split 1 split 2
```
| URL | Decoded content |
| --- | --- |
| `/component/inbox` | one registered component split: `{ type: 'component', id: 'inbox' }` |
| `/md/doc_123` | one block split for block or alias type `md` and id `doc_123` |
| `/md/doc_123/component/search` | two visible splits |
| `/` or an invalid empty pair list | default split `{ type: 'component', id: 'inbox' }` |
`decodePairs()` consumes pairs until a missing `type` or `id` is found. Non-component types are resolved through the block alias registry; aliases keep `aliasContext` so the URL can re-encode the alias instead of only the resolved base block type.
<Note>
Only the current visible split contents are URL encoded. Component `params`, block `params`, per-entry `state`, and split history are runtime state and are not restored from the path after a full reload.
</Note>
## Route mounting and URL sync
`LAYOUT_ROUTE` uses `path: '/*splits'` and passes `props.params.splits?.split('/') ?? []` into `SplitLayoutContainer`. The root route table also maps top-level app paths such as `/inbox`, `/agents`, `/mail`, `/documents`, `/tasks`, `/channels`, `/calls`, and `/files` to the same layout component, while `DEFAULT_ROUTE` is `/component/inbox`.
`SplitLayoutContainer` performs two-way synchronization:
| Direction | Behavior |
| --- | --- |
| Manager to URL | When `splitManager.getUrlSegments()` differs from the router segments, the container navigates to `/${segments.join('/')}`. |
| URL to manager | When router segments change externally, the container calls `splitManager.reconcile(decodedPairs())`. |
| Excluded splits | `getUrlSegments()` omits splits hidden by the current exclusion filter, used by mobile background panels. |
## Split content model
The layout manager treats all mounted content as `SplitContent`:
```ts title="js/app/packages/app/component/split-layout/layoutManager.ts"
type SplitContent =
| {
type: BlockName | BlockAlias;
id: string;
params?: BlockComponentProps[BlockName];
aliasContext?: BlockAliasContext;
state?: EntryState;
}
| {
type: 'component';
id: string;
params?: Record<string, unknown>;
state?: EntryState;
};
```
Block content is mounted through the global block orchestrator. Component content is resolved through the split component registry.
## Component registry entries
Component splits use `{ type: 'component', id }`. The `id` must be registered in `componentRegistry.tsx`; otherwise `resolveComponent()` throws `Component '<name>' not registered`.
Registered app-level component ids include:
| Component id | Runtime behavior |
| --- | --- |
| `unified-list` | Redirects in-place to `{ type: 'component', id: 'inbox' }`. |
| `inbox`, `agents`, `mail`, `documents`, `tasks`, `channels`, `calls`, `folders` | Auth-gated `SoupView` presets with page-view tracking. |
| `search` | Auth-gated `SoupView` that can receive `initialQuery`, `initialFilters`, and `initialClientFilters` when opened programmatically. |
| `loading` | Loading block placeholder. |
| `channel-compose`, `email-compose`, `task-compose` | Compose surfaces. |
| `settings` | Settings panel wrapper. |
| Local/dev-only ids | Debug and internal tools gated by `LOCAL_ONLY` or `DEV_MODE_ENV`. |
<Warning>
A component opened with runtime `params` works during programmatic navigation, but those params are not encoded into the URL. Reloading the same path recreates the component with no params.
</Warning>
## Opening and replacing splits
Most callers use `useSplitLayout()` rather than the manager directly.
| Helper | Behavior |
| --- | --- |
| `openWithSplit(content, options)` | General navigation entry point. On mobile, `preferNewSplit` is forced to `false` by the hook. |
| `replaceOrInsertSplit(content, referredFrom)` | Opens in the current panel when called inside a split panel, activates it, and records a referral cause. |
| `replaceSplit({ content, mergeHistory, referredFrom })` | Replaces the current panel and forces `preferNewSplit: false`. |
| `insertSplit(content, referredFrom)` | Prefers a new split and activates it. |
| `popoverSplit(content)` | Creates a temporary dialog-backed split instead of a URL-backed panel. |
| `resetSplit()` | Resets the current panel to the default split. |
`openWithSplit()` follows this order:
1. A registered navigation interceptor may consume the navigation. Mobile swipe layout uses this.
2. Unless `allowDuplicate` is true, an existing visible split with the same `type` and `id` is activated and returned.
3. If no explicit handle is provided, the active split becomes the target handle.
4. The target handle is replaced when `preferNewSplit` is false or the resize zone cannot fit another `400px` panel.
5. Otherwise a new split is appended.
`replaceWhenFull` defaults to enabled. Setting `replaceWhenFull: false` allows callers to request a new split even when `canAppendSplit()` reports insufficient space, but desktop rendering still depends on the resize zone.
## Split history and navigation causes
Each split owns an independent `History<SplitContent>` with `push`, `merge`, `back`, `forward`, `replaceCurrent`, `goToIndex`, and `remove`.
| Action | History effect | `NavigationCause` |
| --- | --- | --- |
| Initial split creation | Pushes initial content. Optional `initialHistory` is pushed first. | `fresh` |
| `replace()` with `mergeHistory: false` | Pushes a new entry, forking away any forward entries. | `fresh` |
| `replace()` with `mergeHistory: true` | Merges into the current entry. | `replace` |
| `goBack()` | Moves to the previous entry. | `history-back` |
| `goForward()` | Moves to the next entry. | `history-forward` |
| `goToEntry(predicate)` | Jumps to the closest matching prior entry, then closest forward entry. | `history-back` or `history-forward` |
| `removeFromHistory(predicate)` | Removes matching entries and reattaches the current surviving entry. | `replace` |
Components can read the latest cause with `useNavigationCause()`. This is intended for behavior that differs between fresh navigation and restoration, such as suppressing auto-focus after back/forward.
`useEntryState(key, { default })` stores component-owned state on the current history entry. Registered captors run before navigation away, then the captured state is mirrored back onto `split.content.state`.
## Referral metadata
`referredFrom` records where navigation originated. It is stored on `SplitState` and exposed through `SplitHandle.referredFrom()`.
Allowed values are:
```ts
'list-view' | 'kommand-menu' | 'mention' | 'attachment' | 'launcher' |
'sidebar' | 'dock' | 'entity-actions-menu' | 'hotkey' | 'quick-access' |
'file-upload' | 'search' | null
```
Use the closest existing value when adding a navigation caller. If the source is not meaningful to downstream behavior or analytics, pass `null`.
## Popover splits
Popover splits are temporary mounts managed by `createPopoverSplit()` and rendered by `PopoverSplitRenderer`.
A popover split:
- creates a `popover-...` id;
- acquires a focus lock before state updates;
- mounts the same block/component content model as normal splits;
- renders inside a `Dialog` and `Panel`;
- provides a stub `SplitHandle` with `isPopover() === true`;
- has no URL segments, no history navigation, and `lastNavigationCause() === 'fresh'`;
- closes on `escape` or dialog close;
- releases the focus lock and removes its map entry after a short animation delay.
Popover content still receives `SplitPanelContext`, so components that depend on panel context can render inside the dialog. Do not expect URL persistence, back/forward history, or split resizing inside a popover.
## Desktop layout behavior
Desktop split rendering uses a horizontal `Resize.Zone` with `Resize.Panel` children. Each panel has `minSize={400}` and renders a `SplitPanel` containing:
- a split-specific hotkey DOM scope;
- `SoupContextProvider`;
- split header and toolbar slots;
- the mounted block or registered component element;
- spotlight support;
- focus restoration and active split tracking.
Focus changes inside a panel activate that panel after a short debounce. Insert and remove events explicitly move focus to the inserted split or the nearest remaining split.
Relevant split hotkeys include:
| Hotkey | Behavior |
| --- | --- |
| `cmd+\` or `\` | Create a new inbox split when space allows. |
| `cmd+escape` / `opt+escape` | Go home or close split depending on current content and split count. |
| `opt+[` / `opt+]` | Per-split history back/forward. |
| `shift+escape` | Toggle spotlight when multiple splits exist. |
| `shift+h` / `shift+arrowleft` | Focus split left. |
| `shift+l` / `shift+arrowright` | Focus split right. |
## Mobile layout behavior
There are two mobile-related checks:
| Check | Effect |
| --- | --- |
| `isMobile()` in `useSplitLayout()` | Forces `preferNewSplit` to `false` for callers using the helper. |
| `isNativeMobilePlatform()` in `SplitLayoutContainer` | Switches rendering from desktop `Resize.Zone` to the native mobile two-slot swipe layout. |
The native mobile layout uses slot A and slot B. One slot is foreground and the other may hold a background split for swipe-back. The background split is excluded from URL encoding, duplicate detection, and content lookup.
Mobile navigation behavior differs from desktop:
- `createMobileSwipeLayout()` registers a navigation interceptor.
- Non-`mergeHistory` navigation is handled as forward navigation into the background slot, then promoted to foreground after animation.
- Swipe-back promotes the background slot, removes the old foreground split, and lazily mounts the promoted split’s previous history entry as the next background split.
- `mergeHistory` navigation bypasses the interceptor and uses normal replace behavior.
- `MobileDock` uses `mergeHistory` when switching between component/list views so dock tab switches do not create swipe-back entries.
## Reconciliation contracts
`reconcile(newSplits)` is used for URL-driven changes. It compares the visible split key sequence with the decoded URL sequence, preserves excluded splits unchanged, and rebuilds changed visible positions. When a visible split exists at the same index, the new split reuses that split id to keep panel identity stable at that position.
Run the layout manager tests when changing reconciliation, history, or URL behavior. The test suite covers URL-state reconciliation, block-to-component replacement, and browser-back-like ordering scenarios.
## Troubleshooting
| Symptom | Likely cause | Fix |
| --- | --- | --- |
| `No split manager found` | A split helper ran before `SplitLayoutContainer` registered the global manager. | Call split helpers only under the app layout route or guard for missing manager. |
| `Component '<id>' not registered` | A URL or caller opened `{ type: 'component', id }` without a registry entry. | Add `registerComponent(id, factory)` in the split component registry. |
| URL opens inbox instead of expected content | The path did not contain a complete `type/id` pair. | Use `/component/<id>` for registered components or `/<blockType>/<id>` for blocks. |
| Component state disappears after reload | Runtime `params`, entry state, and history are not URL encoded. | Persist durable state outside split runtime state or encode it in a supported route/content id. |
| New block does not open in another split | Duplicate non-component content is prevented unless allowed. | Pass `allowDuplicate: true` only when duplicate mounts are intentional. |
| Mobile background split is missing from URL | The mobile swipe layout excludes the background slot. | Treat this as expected; only foreground-visible content is URL encoded. |
---
## 11. Real-time document sync
> Concept: Sync-service worker routes, Durable Object sessions, permission tokens, Bebop messages, snapshot lifecycle, and client reconnect behavior.
- Page Markdown: https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/11-real-time-document-sync.md
- Generated: 2026-06-01T00:52:59.543Z
### Source Files
- `rust/sync-service/src/cf_worker.rs`
- `rust/sync-service/src/durable_object.rs`
- `rust/sync-service/src/websocket.rs`
- `rust/sync-service/src/generated/schema.rs`
- `js/app/packages/service-clients/service-sync/client.ts`
- `js/app/packages/service-clients/service-sync/source.ts`
- `js/app/packages/block-md/definition.ts`
---
title: "Real-time document sync"
description: "Concept: Sync-service worker routes, Durable Object sessions, permission tokens, Bebop messages, snapshot lifecycle, and client reconnect behavior."
---
The sync service is a Cloudflare Worker backed by one `DocumentSyncSession` Durable Object per `document_id`; it exposes HTTP document endpoints, upgrades `/document/{document_id}/connect` to a WebSocket session, serializes realtime messages with Bebop, and stores Loro snapshots plus pending operations for reconnect and recovery.
## Runtime shape
```mermaid
flowchart LR
subgraph Client
MD["block-md load()"]
Source["createSyncServiceSource()"]
Rest["syncServiceClient"]
end
subgraph Worker["sync-service Worker"]
Router["cf_worker::router"]
Schema["/schema"]
Copy["/document/{id}/copy"]
end
subgraph DO["DocumentSyncSession Durable Object"]
WS["/connect WebSocket"]
API["metadata/raw/snapshot/active_peers/initialize"]
State["DocumentState(LoroDoc)"]
Awareness["EphemeralStore"]
Alarm["alarm()"]
end
subgraph Storage
SQLKV["Durable Object SQL snapshot store"]
KVFallback["SNAPSHOT_STORE_KV fallback"]
DOKV["Durable Object KV pending ops"]
D1["USER_PEER_MAPPING D1"]
end
MD --> Source
Source --> WS
Rest --> Router
Router --> Schema
Router --> Copy
Router --> DO
WS --> State
WS --> Awareness
API --> State
State --> SQLKV
SQLKV --> KVFallback
DOKV --> State
WS --> D1
Alarm --> SQLKV
Alarm --> DOKV
```
## Worker routes
The top-level Worker routes health/schema traffic directly and forwards document traffic to the Durable Object namespace `DOCUMENT_SYNC_SESSION` using `id_from_name(document_id)`.
| Route | Handler | Authentication | Response |
|---|---|---:|---|
| `/` | Worker | None | `Hello Sync Service!` |
| `/health` | Worker | None | `healthy` |
| `/schema` | Worker | None | Raw `bebop/schema.bop` |
| `/document/{document_id}/copy` | Worker orchestration | Header token via forwarded Durable Object calls | Copies a snapshot into another document |
| `/document/{document_id}/{*rest}` | Durable Object pass-through | Depends on Durable Object route | Durable Object response or `408` on RPC timeout |
`pass_to_durable_object` wraps Durable Object fetches with the service default timeout and returns status `408` if the RPC times out.
## Durable Object routes
`DocumentSyncSession` owns the document session state. CORS validation runs before route handling. Allowed origins include local app ports, Macro production/dev/staging origins, Capacitor localhost, Apollo testing, and non-empty `https://{subdomain}.preview.macro.com` preview origins.
| Route | Purpose | Auth requirement |
|---|---|---|
| `/document/{document_id}/connect?token=...` | WebSocket upgrade and initial sync | JWT in query params |
| `/document/{document_id}/exists` | Existence check | None |
| `/document/{document_id}/wakeup` | Warm in-memory state and keep worker alive | None |
| `/document/{document_id}/peer/{peer_id}` | Resolve registered peer to user | None |
| `/document/{document_id}/metadata` | Return `{ id, peers, version_id }` | Bearer JWT or internal admin |
| `/document/{document_id}/raw` | Return Loro deep JSON | Bearer JWT or internal admin |
| `/document/{document_id}/snapshot` | Return binary Loro snapshot | Bearer JWT or internal admin |
| `/document/{document_id}/active_peers` | Return active peer IDs as strings | Bearer JWT or internal admin |
| `/document/{document_id}/initialize` | Store initial binary snapshot | Bearer JWT with at least `edit` |
| `/document/{document_id}/debug_dump_operations` | Return pending operations | Admin |
| `/document/{document_id}/debug_do_kv_get/{key}` | Inspect Durable Object KV key | Admin |
| `/document/{document_id}/debug_do_kv_list/{prefix}` | Inspect Durable Object KV prefix | Admin |
<Note>
Route matching is path-based. The implementation handles CORS `OPTIONS` separately, but most Durable Object handlers do not enforce a specific HTTP method after the path is matched.
</Note>
## Permission tokens
The service accepts two token sources:
| Surface | Token source | Decoder mode |
|---|---|---|
| WebSocket connect | `token` query param | `TokenFrom::QueryParams` |
| HTTP document APIs | `Authorization: Bearer {jwt}` | `TokenFrom::Headers` |
| Internal/admin APIs | `x-internal-auth-key` | `TokenFrom::Headers` admin shortcut |
JWTs are HS256 tokens signed with `DOCUMENT_PERMISSIONS_SECRET` and contain:
```ts
type AuthToken = {
user_id?: string;
document_id: string;
access_level: 'view' | 'comment' | 'edit' | 'owner' | 'admin';
};
```
Document-scoped requests must match `document_id` unless the token is internal admin. `initialize` requires `edit` or stronger. Debug routes require `admin`.
For WebSocket document updates, `PeerUpdate` is ignored when `Wsm::can_edit()` is false. The current runtime write gate treats `comment`, `edit`, `owner`, and `admin` as editable because `AccessLevel::can_edit()` checks for `AccessLevel::Comment` or stronger; `view` users cannot push document updates. Awareness updates are accepted for connected users independently of edit access.
## WebSocket session lifecycle
1. `connect_handler` decodes the query token.
2. The Durable Object stores the `document_id` if not already set.
3. A Cloudflare `WebSocketPair` is created.
4. The server socket is accepted with a generated WebSocket tag.
5. `WebSocketMetadata` is stored in Durable Object storage and memory:
- `user_id`
- `access_level`
- `peer_ids`
6. The current `DocumentState` is loaded or reused.
7. The server sends `RemoteInitialSync` with:
- a shallow Loro snapshot
- encoded awareness state
8. The client receives the paired WebSocket response.
String `"ping"` messages receive `"pong"`. Binary messages are decoded as `FromPeer` Bebop messages.
## Bebop message protocol
The schema lives in `rust/sync-service/bebop/schema.bop` and is generated into Rust and TypeScript clients.
### Client to service: `FromPeer`
| Message | Fields | Behavior |
|---|---|---|
| `PeerUpdate` | `update: byte[]` | Applies update to `DocumentState`, records pending op, broadcasts `RemoteUpdate` to other sockets, sends `RemoteUpdateAck` to sender |
| `PeerAwareness` | `awareness: byte[]` | Applies awareness update and broadcasts `RemoteAwareness` to other sockets |
| `PeerRequestSince` | `frontiers: byte[]` | Decodes Loro frontiers and returns `RemoteUpdateSince` |
| `PeerRequestSnapshot` | none | Returns `RemoteSnapshot` with a shallow snapshot |
| `PeerRegisterId` | `peerid: uint64` | Associates a Loro peer ID with the socket metadata and, when `user_id` exists, D1 |
### Service to client: `FromRemote`
| Message | Fields | Emitted when |
|---|---|---|
| `RemoteInitialSync` | `snapshot`, `awareness` | Immediately after WebSocket connect |
| `RemoteUpdate` | `update` | Another peer pushes an accepted update |
| `RemoteAwareness` | `awareness` | Another peer updates or clears awareness |
| `RemoteSnapshot` | `snapshot` | Alarm broadcast or explicit snapshot request |
| `RemoteUpdateAck` | `update` | Sender’s update was accepted and processed |
| `RemoteUpdateSince` | `update`, `frontiers` | Response to `PeerRequestSince` |
Messages larger than 1 MB log a warning before deserialization; they are not rejected solely because of that size check.
## Snapshot and operation lifecycle
`DocumentState` wraps a Loro document. It imports snapshots with the `from_service` tag, imports client updates with the `from_client` tag, and tracks whether a snapshot should be saved by comparing `last_update` and `last_export`.
### Storage layers
| Data | Storage |
|---|---|
| Current snapshot | Default feature: Durable Object SQL via `DurableSQLStorage`, with `SNAPSHOT_STORE_KV` fallback for reads |
| Pending operations | Durable Object KV keys under `o/` |
| All recent operations | Durable Object KV keys under `a/` |
| Last saved version vector | Durable Object KV key `LAST_VERSION_VECTOR` |
| User-peer mappings | D1 binding `USER_PEER_MAPPING`, table `peer_user_map` |
The default Cargo feature set enables `do-sqlite-snapshot-storage`, so snapshots are written to Durable Object SQL. The combined storage backend reads SQL first and falls back to KV when SQL has no snapshot.
### Save loop
After each binary WebSocket message, the Durable Object schedules an alarm roughly 5 seconds in the future if no later alarm is already scheduled.
When the alarm fires:
1. The Durable Object loads `DocumentState`.
2. If `state.should_save()` is true:
- exports a full Loro snapshot
- stores the snapshot
- stores the Loro operation version vector
- clears applied pending operation keys
- marks the state exported
3. If sockets are still connected:
- schedules the next alarm
- broadcasts `RemoteSnapshot` with a shallow snapshot
4. If no sockets remain, it logs that the Durable Object reached zero connections.
`wakeup` calls warm the session storage and document state when the document exists, then use a JavaScript timeout keepalive with the default 60 second TTL. The TypeScript client’s `safeWakeup` debounces wakeups by 200 ms and suppresses repeat wakeups for 55 seconds per document.
## Initialization and copy
`initialize` accepts a Bebop `InitializeFromSnapshotRequest`:
```ts
type InitializeFromSnapshotRequest = {
snapshot: Uint8Array;
};
```
It fails if snapshot storage already contains a snapshot for the document. Otherwise it stores the snapshot, records the `DOCUMENT_ID` in Durable Object storage, and prepares `SessionStorage`.
`copy` is handled at the Worker layer because it touches two Durable Object instances:
1. POST `/document/{source_id}/snapshot` with optional `version_id`.
2. Wrap the returned binary snapshot in `InitializeFromSnapshotRequest`.
3. POST `/document/{target_id}/initialize`.
The JSON copy request is:
```json
{
"target_document_id": "new-document-id",
"version_id": {
"peer": "123",
"counter": 10
}
}
```
`version_id` is optional. When present, the snapshot handler exports state at the requested Loro frontier.
## TypeScript client behavior
`syncServiceClient` exposes HTTP helpers against `SYNC_SERVICE_HOSTS.worker`:
| Method | Route |
|---|---|
| `wakeup({ documentId })` | `GET /document/{id}/wakeup` |
| `safeWakeup(id)` | Debounced `GET /document/{id}/wakeup` |
| `exists({ documentId })` | `HEAD /document/{id}/exists` |
| `initializeFromSnapshot({ documentId, snapshot })` | `POST /document/{id}/initialize` |
| `getDocumentMetadata({ documentId })` | `GET /document/{id}/metadata` |
| `getSnapshot({ documentId })` | `GET /document/{id}/snapshot` |
| `getRaw({ documentId })` | `GET /document/{id}/raw` |
Tauri requests add an explicit `Origin` header of `https://dev.macro.com` in development and `https://macro.com` otherwise.
`createSyncServiceSource(documentId, token)` builds the WebSocket URL:
```text
{SYNC_SERVICE_HOSTS.ws}/document/{documentId}/connect?token={token}
```
Connection settings:
| Setting | Value |
|---|---:|
| Reconnect backoff | Constant 500 ms |
| Max retries | 20 |
| Initial sync timeout | 10 seconds |
| Update ACK timeout | 3 seconds |
| Snapshot request timeout | 10 seconds |
| Updates-since timeout | 10 seconds |
| Heartbeat interval | 10 seconds |
| Heartbeat timeout | 5 seconds |
| Max missed heartbeats | 2 |
The first connection uses the provided token. Reconnects request a fresh permission token from `storageServiceClient.permissionsTokens.createPermissionToken({ document_id })`; if token refresh fails, the client falls back to the last known WebSocket URL.
Heartbeat is intentionally started only after the first `RemoteInitialSync` arrives. On reconnect, the client starts heartbeat, waits for another `RemoteInitialSync`, and emits a `reconnect` sync event containing the new snapshot and awareness. If reconnect sync times out, it logs the failure and lets heartbeat monitoring close/retry the socket.
## Markdown block integration
The Markdown block only loads realtime collaboration when the source is `sync-service`.
Load flow:
1. Fetch document metadata, document location, and permission token in parallel.
2. If the location is a pending presigned URL, wait for `syncServiceContent` readiness.
3. Reject the load when the final location is not `syncServiceContent`.
4. Create the sync-service source with the permission token.
5. Create a Loro manager using `MARKDOWN_LORO_SCHEMA`.
6. Initialize local Loro state from `initialSync.snapshot`.
7. Return the block data with `syncSource`, `loroManager`, metadata, and user access level.
Markdown initialization and lifecycle persistence are backend-owned; the frontend does not repair an object-storage-backed Markdown document into sync-service content during block load.
## Operational configuration
Cloudflare bindings in `wrangler.toml`:
| Binding | Purpose |
|---|---|
| `DOCUMENT_SYNC_SESSION` | Durable Object namespace |
| `USER_PEER_MAPPING` | D1 database for peer/user mappings |
| `DOCUMENT_SNAPSHOT_BUCKET` | R2 bucket when the R2 snapshot feature is enabled |
| `SNAPSHOT_STORE_KV` | KV snapshot fallback |
| `DOCUMENT_VERSIONING_KV` | Configured KV namespace |
| `INTERNAL_API_SECRET_KEY` | Variable naming the secret binding for internal API auth |
| `DOCUMENT_PERMISSIONS_SECRET` | JWT signing secret expected at runtime |
| `SPS_URL` | Search processing service URL when `search-service` is enabled |
Useful local commands:
```bash
cd rust/sync-service
# Build the Worker used by Miniflare tests
just worker-build
# Run e2e tests and Rust unit tests
just test
# Apply local D1 migrations and start Wrangler
just dev
# Regenerate TypeScript Bebop bindings used by tests
cd bebop && npx bebopc build
```
## Error and verification signals
| Symptom | Likely cause |
|---|---|
| `401` | Missing/malformed Bearer token, invalid JWT, wrong document token, or missing internal key |
| `403` | Request `Origin` is not allowed |
| `404` | Snapshot/document does not exist for routes that check existence |
| `408` | Worker-to-Durable-Object RPC timeout |
| Missing update ACK | Client did not receive `RemoteUpdateAck` within 3 seconds |
| View-only edits do not propagate | `PeerUpdate` was ignored by the WebSocket write gate |
| Initial sync timeout | Client did not receive `RemoteInitialSync` within 10 seconds |
| Snapshot already exists during initialize | Target document already has stored snapshot state |
## Related pages
<CardGroup>
<Card title="Sync service client" href="/service-clients/service-sync">
TypeScript source wrapper, reconnect behavior, and SyncSource integration.
</Card>
<Card title="Markdown block loading" href="/blocks/markdown">
Markdown-specific sync-service source loading and Loro initialization.
</Card>
</CardGroup>
---
## 12. Run backend services locally
> Guide: Start databases, LocalStack, FusionAuth, Rust services, optional processor profiles, and local health checks.
- Page Markdown: https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/12-run-backend-services-locally.md
- Generated: 2026-06-01T00:53:22.606Z
### Source Files
- `justfile`
- `docker-compose.yml`
- `docker-compose-databases.yml`
- `docker-compose.local-e2e.yml`
- `local_stack.just`
- `rust/cloud-storage/justfile`
---
title: "Run backend services locally"
description: "Guide: Start databases, LocalStack, FusionAuth, Rust services, optional processor profiles, and local health checks."
---
The local backend runtime is owned by `just` recipes and Docker Compose files at the repository root. `just setup` prepares `.env`, external Docker networks and volumes, LocalStack AWS resources, the local Postgres database, FusionAuth configuration, and prebuilt Rust service images; `just run_local` then starts the Compose stack.
<Warning>
Local Docker resources are intentionally single-instance. The Compose project is fixed to `macro`, so multiple checkouts or worktrees share the same containers, volumes, networks, LocalStack container, and FusionAuth instance.
</Warning>
## Prerequisites
Install the tools used directly by the local recipes:
| Tool | Used by |
| --- | --- |
| `just` | Repository command runner |
| Docker and Docker Compose | Databases, FusionAuth, Rust services, JS services |
| `sops` | Decrypting `.env-local*.enc` into `.env` |
| AWS CLI | Creating LocalStack SQS, DynamoDB, and S3 resources |
| Pulumi | Local FusionAuth stack outputs and configuration |
| SQLx CLI | Database create, migrate, setup, and reset commands |
| Bun and Node | FusionAuth setup dependencies and JS-backed services/tests |
If not using the repository shell environment, export `SOPS_KMS_ARN` before decrypting local environment files.
```bash
export SOPS_KMS_ARN="arn:aws:kms:us-east-1:569036502058:key/mrk-cab29bf948044eb79005a81f48d40e93,arn:aws:kms:us-west-1:569036502058:key/mrk-cab29bf948044eb79005a81f48d40e93"
```
## One-command setup
```bash
just setup
```
`just setup` performs this sequence:
1. Decrypts the root local environment into `.env`.
2. Creates external Docker networks: `databases` and `auth`.
3. Creates persistent local volumes for Postgres, Redis, OpenSearch, and FusionAuth.
4. Starts and provisions LocalStack.
5. Creates and migrates the local Macro Postgres database.
6. Starts FusionAuth, deploys the local Pulumi stack, patches root `.env`, then stops FusionAuth.
7. Builds the development Rust service images.
<Note>
`setup_local_dbs` starts Postgres and Redis to initialize the database, then stops the database Compose stack. `run_local` starts the services again when needed.
</Note>
## Start only infrastructure
Use these commands when you want to rebuild or inspect dependencies before starting application services.
<Steps>
<Step title="Create shared Docker resources">
```bash
just create_networks
```
</Step>
<Step title="Start Postgres and Redis for database work">
```bash
just run_dbs -d
```
This targets only `postgres` and `redis` from `docker-compose-databases.yml`.
</Step>
<Step title="Create or migrate the Macro database">
```bash
just rust/cloud-storage/macro_db_client/create_db
just rust/cloud-storage/macro_db_client/migrate_db
```
The local database URL is `postgres://user:password@localhost:5432/macrodb`.
</Step>
<Step title="Provision LocalStack">
```bash
AWS_ACCESS_KEY_ID=test AWS_SECRET_ACCESS_KEY=test AWS_DEFAULT_REGION=us-east-1 just setup_localstack
```
</Step>
<Step title="Set up FusionAuth">
```bash
just setup_fusionauth
```
</Step>
</Steps>
## Local infrastructure endpoints
| Component | Container/image | Host endpoint | Notes |
| --- | --- | --- | --- |
| Postgres | `pgvector/pgvector:pg16` | `localhost:5432` | User `user`, password `password`, database `macrodb` after setup |
| Redis Stack | `redis/redis-stack:latest` | `localhost:6379`, UI/tools on `localhost:8001` | Uses persistent `macro_redis_data` volume |
| OpenSearch | `opensearchproject/opensearch:latest` | `localhost:9200`, analyzer on `localhost:9600` | Defined in the database Compose file; not targeted by `just run_dbs` |
| LocalStack | `localstack/localstack:4` | `localhost:4566` | Runs SQS, DynamoDB, and S3 on the `databases` network |
| FusionAuth | `fusionauth/fusionauth-app:1.62.1` | `localhost:9011` | Exposed to backend services through the external `auth` network |
## LocalStack resources
`just setup_localstack` starts LocalStack with `SERVICES=sqs,dynamodb,s3`, waits for `/_localstack/health`, then creates queues, tables, buckets, and a document-upload S3 notification.
### S3 buckets
| Bucket | Used for |
| --- | --- |
| `macro-email-attachments` | Email attachments |
| `doc-storage` | Document storage and document-upload finalizer events |
| `docx-upload` | DOCX upload flow |
| `static-file-storage` | Static file objects |
| `bulk-upload-staging` | Bulk upload staging |
All buckets receive a local CORS configuration allowing `http://localhost:3000` through `http://localhost:3009`.
### DynamoDB tables
| Table | Key shape |
| --- | --- |
| `bulk-upload` | `PK` hash, `SK` range, `DocumentPkIndex` GSI |
| `connection-gateway-table` | `PK` hash, `SK` range, `ConnectionPkIndex` GSI |
| `static-file-metadata` | `file_id` hash |
### SQS queues
Local setup creates queues for notifications, email jobs, contacts, conversion, document deletion, document text extraction, search events, static-file events, and document-upload finalization. The document-upload finalizer wires S3 `ObjectCreated` events from `doc-storage` to `document-upload-finalizer-queue`.
Inside Docker, services use `http://localstack:4566`. Host-side commands and generated browser-facing LocalStack URLs use `http://localhost:4566`.
## FusionAuth local setup
FusionAuth local setup is split between Docker Compose and a Pulumi stack under `infra/stacks/fusionauth-instance`.
```bash
just setup_fusionauth
```
The setup recipe:
1. Downloads the FusionAuth container `.env` if missing.
2. Starts the FusionAuth Postgres database and app.
3. Waits for `http://localhost:9011/api/status` to report `{"status":"Ok"}`.
4. Initializes or updates the Pulumi `local` stack.
5. Writes local FusionAuth values into the root `.env`.
6. Stops FusionAuth after setup.
Useful local credentials:
| Field | Value |
| --- | --- |
| Admin username | `admin@macro.com` |
| Admin password | `macroIsGreat!` |
| API key | `bf69486b-4733-4954-a44e-2e1b5f2c8a91` |
`just run_local` calls `patch_local_fusionauth_env` before starting services. If FusionAuth is not already running, the patch recipe starts it temporarily, reads the client secret, updates `.env`, and stops the temporary container afterward.
## Start backend services
```bash
just run_local
```
By default, this runs `docker compose up` in the foreground. Pass Docker Compose arguments through `just` when you want detached startup, waiting, or a subset of services.
```bash
just run_local -d --wait
```
For a clean rebuild after changing service code:
```bash
just run_local --build
```
`run_local` always builds the shared Rust services image target `rust_services_image`. With `--build`, it also rebuilds `websocket_service`, `sync_service`, and `lexical_service`.
## Service ports and health endpoints
| Service | Host port | Health check | Runtime notes |
| --- | ---: | --- | --- |
| `authentication-service` | `8080` | `/health` | Depends on Postgres, FusionAuth, and Redis |
| `connection_gateway` | `8082` | `/health` | WebSocket gateway; service alias `connection-gateway` |
| `contacts_service` | `8083` | `/health` | Contact management service |
| `document_cognition_service` | `8085` | `/health` | Depends on document storage, email, static files, sync, and lexical |
| `document_storage_service` | `8086` | `/health` | Depends on Redis, connection gateway, and auth |
| `document_upload_finalizer` | none | none | Local SQS worker for `doc-storage` object-created events |
| `email_service` | `8087` | `/health` | Depends on auth, document storage, connection gateway, static files, and Redis |
| `notification_service` | `8089` | `/health` | Depends on Redis, cognition, auth, connection gateway, and document storage |
| `search_processing_service` | `8092` | `/health` | Optional `processors` profile |
| `static_file_service` | `8094` | `/api/health` | Uses DynamoDB/static-file metadata |
| `static_file_cdn` | `8100` | none | Nginx local CloudFront-style emulator |
| `unfurl_service` | `8095` | `/health` | URL unfurling service |
| `image_proxy_service` | `8097` | `/health` | External image proxy |
| `websocket_service` | `6969` | none | JS WebSocket service |
| `sync_service` | `8787` | `/health` | Cloudflare Worker/Durable Object sync runtime via Docker |
| `lexical_service` | `8096` | `/health` | Lexical conversion service |
Check the stack:
```bash
docker compose ps
curl -fsS http://localhost:8080/health
curl -fsS http://localhost:8086/health
curl -fsS http://localhost:8094/api/health
curl -fsS http://localhost:8787/health
curl -fsS http://localhost:9011/api/status
curl -fsS http://localhost:4566/_localstack/health
```
## Optional processor profile
`search_processing_service` is behind the Compose profile `processors`.
```bash
just run_local --profile processors
```
Rebuild it explicitly when changing processor code:
```bash
just run_local --build --profile processors
```
The processor image uses `Dockerfile.search_processing_service.dev` and is pinned to `linux/amd64` because the bundled `search_processing_service/pdfium-lib/linux/libpdfium.so` is amd64-only. Apple Silicon hosts run this service through emulation.
## Local E2E backend profile
The local E2E commands start LocalStack, start a narrowed backend service set with `docker-compose.local-e2e.yml` overrides, seed deterministic data, then run tests.
```bash
just local-e2e
```
Rust ignored integration tests use the same seeded stack:
```bash
just local-e2e-rust
```
Run Rust integration tests and Playwright after one stack startup and seed pass:
```bash
just local-e2e-all
```
Open Playwright UI mode:
```bash
just local-e2e-ui
```
The E2E override file forces services to use local dependencies instead of shared development infrastructure:
| Key | Local value |
| --- | --- |
| `DATABASE_URL` | `postgres://user:password@postgres:5432/macrodb` |
| `DATABASE_URL_READONLY` | `postgres://user:password@postgres:5432/macrodb` |
| `LOCAL_AWS_URL` | `http://localstack:4566` |
| `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` | `test` / `test` |
| `REDIS_URI` | `redis://redis:6379` |
| `DOCUMENT_STORAGE_SERVICE_REDIS_URI` | `redis://redis:6379` |
The deterministic seed command is guarded. It requires `LOCAL_E2E_SEED=true` and refuses database URLs outside the local Docker database shape `postgres://user:...@(localhost|127.0.0.1|postgres):5432/macrodb`.
## Stop and reset
| Command | Effect |
| --- | --- |
| `just stop-local` | Runs `docker compose down` for the main local stack |
| `just stop-databases` | Stops the database Compose stack |
| `just stop_fusionauth` | Stops the FusionAuth Compose stack |
| `docker rm -f localstack` | Removes the LocalStack container |
| `just rust/cloud-storage/macro_db_client/reset_db` | Drops and recreates the local Macro database |
| `just docker_cache_usage` | Shows BuildKit cache disk usage |
| `just docker_cache_clear_targets` | Clears Rust target cache mounts |
| `just docker_cache_clear` | Clears all BuildKit build cache |
## Troubleshooting
### `.env not found`
Run:
```bash
just get_environment
```
For a fresh checkout, prefer:
```bash
just setup
```
### FusionAuth local stack is missing
If `run_local` prints a warning that the Pulumi local stack was not found, run:
```bash
just setup_fusionauth
```
Then restart the backend stack.
### LocalStack resources are missing
Recreate LocalStack resources:
```bash
AWS_ACCESS_KEY_ID=test AWS_SECRET_ACCESS_KEY=test AWS_DEFAULT_REGION=us-east-1 just setup_localstack
```
The recipe uses `localstack/localstack:4`; it is intentionally not `latest`.
### Processor builds are slow on Apple Silicon
`search_processing_service` runs as `linux/amd64` because of the bundled PDFium library. Use the default stack unless you need the `processors` profile.
### Local E2E seed refuses to run
Check that the seed command has `LOCAL_E2E_SEED=true` and that `DATABASE_URL` points at the local Compose database, not a shared development database.
## Related pages
<CardGroup>
<Card title="Frontend local E2E" href="/js-app-local-e2e">
Run Playwright against the local backend stack and seeded fixtures.
</Card>
<Card title="Seed CLI" href="/seed-cli">
Populate local Macro data with deterministic scenarios.
</Card>
<Card title="FusionAuth instance stack" href="/fusionauth-instance-stack">
Maintain the local FusionAuth Pulumi stack and Docker runtime.
</Card>
</CardGroup>
---
## 13. Develop SolidJS and Tauri apps
> Guide: Use Bun and Vite, run local service overrides, build app bundles, start Tauri targets, and avoid iOS worker deadlocks.
- Page Markdown: https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/13-develop-solidjs-and-tauri-apps.md
- Generated: 2026-06-01T00:54:19.836Z
### Source Files
- `js/app/README.md`
- `js/app/package.json`
- `js/app/justfile`
- `js/app/packages/app/index.tsx`
- `js/app/packages/app/vite.config.ts`
- `js/app/tauri/src-tauri/README.md`
- `js/app/AGENTS.md`
---
title: "Develop SolidJS and Tauri apps"
description: "Guide: Use Bun and Vite, run local service overrides, build app bundles, start Tauri targets, and avoid iOS worker deadlocks."
---
The Macro frontend is a SolidJS single-page app in `js/app/packages/app` built by Vite and run with Bun; the same Vite output is used for web, desktop, iOS, and Android through the Tauri wrapper in `js/app/tauri`.
## Project layout
:::files
js/app/
├─ package.json # Bun scripts for dev, build, tests, codegen
├─ justfile # frontend build and local-service shortcuts
├─ packages/app/
│ ├─ index.tsx # Solid entry point and Tauri fetch override
│ ├─ vite.config.ts # app Vite config entry
│ └─ vite.base.ts # shared Vite server/build/env configuration
├─ packages/core/constant/
│ └─ servers.ts # remote/local service host selection
├─ packages/core/util/
│ ├─ platform.ts # web/desktop/iOS/Android runtime detection
│ └─ platformFetch.ts # Tauri HTTP plugin fetch adapter
└─ tauri/src-tauri/
├─ tauri.conf.json # Tauri dev/build hooks and bundle settings
└─ README.md # native target commands
:::
## Prerequisites
Use `nix develop` from the repository root when available. Otherwise install:
- Bun
- Node tooling used by Bun scripts
- Rust and Cargo
- Tauri CLI
- Xcode for iOS targets
- Android Studio and Android SDK tooling for Android targets
For local backend services, complete the repository-level local setup first:
```bash
just setup
```
The local stack uses the fixed Docker Compose project name `macro`; do not run two local stacks at the same time.
## Run the web app with Vite
Run commands from `js/app`.
```bash
bun i
bun run dev
```
`bun run dev` changes into `packages/app` and starts:
```bash
bun run --bun dev
```
The package-level app script runs:
```bash
vite -c vite.config.ts
```
Vite serves on `0.0.0.0:${PORT:-3000}` with `strictPort: true`, WebSocket HMR, CORS enabled, polling file watch, and filesystem access to the workspace root.
<Check>
The dev app is available at `http://localhost:3000/app` for normal browser routing. Tauri development uses the same Vite server through `http://localhost:3000`.
</Check>
## Vite build behavior
The shared config sets different paths for serve and build:
| Surface | Value |
| --- | --- |
| Dev server base | `/` |
| Built app base | `/app` |
| Build output | `js/app/packages/app/dist` |
| Build target | `esnext` |
| CSS transformer/minifier | `lightningcss` |
| Worker format | ES modules |
| Sourcemaps | enabled |
Build modes are selected with `MODE`:
```bash
just build-dev
just build-staging
just build-prod
```
Equivalent direct command shape:
```bash
cd js/app/packages/app
MODE=production NODE_ENV=production bun run --bun build
```
The app build also writes `dist/semver.txt` as:
```text
<package-version>+<git-short-sha>
```
Use `NO_MINIFY=true` when a readable bundle is needed for debugging. The Vite config switches output names to stable `assets/[name].js` and disables minification.
## Runtime environment values
`vite.base.ts` injects compile-time values through `define`.
| Identifier | Source or default | Notes |
| --- | --- | --- |
| `import.meta.env.__APP_VERSION__` | `packages/app/package.json` version plus Git short SHA | Logged at startup and passed to observability outside Vite HMR |
| `import.meta.env.ASSETS_PATH` | derived from `MODE` and Vite command | `/local` in dev server development mode, `/dev` for development builds, `/staging` for staging, `/` otherwise |
| `import.meta.env.__LOCAL_DOCKER__` | `LOCAL_DOCKER === "true"` | boolean define |
| `import.meta.env.__LOCAL_JWT__` | `LOCAL_JWT` | used by local bearer-token auth paths |
| `import.meta.env.__GIT_BRANCH__` | current branch during `vite serve` | updated over a custom HMR event when `.git/HEAD` changes |
## Run against local service overrides
In development mode, service hosts come from `packages/core/constant/servers.ts`.
If `VITE_LOCAL_SERVERS` is unset or empty, the app uses remote development hosts. If it is `ALL`, all configured services use localhost. Otherwise, provide a comma-separated list of service names to override individually.
```bash
# All local services
just local
# Cognition HTTP + websocket services
just local-dcs
# Document storage only
just local-dss
# Search only
just local-search
# Email only
just local-email
```
Direct examples:
```bash
cd js/app/packages/app
# Use all local hosts
VITE_LOCAL_SERVERS=ALL bun run --bun dev
# Override one service
VITE_LOCAL_SERVERS=document-storage-service bun run --bun dev
# Override multiple services
VITE_LOCAL_SERVERS=document-storage-service,email-service bun run --bun dev
# Override a local port for one service
VITE_LOCAL_SERVERS=document-storage-service:18086 bun run --bun dev
```
Available service keys include:
| Service key | Default local host |
| --- | --- |
| `auth-service` | `http://localhost:8080` |
| `pdf-service` | `http://localhost:4567` |
| `document-storage-service` | `http://localhost:8086` |
| `websocket-service` | `ws://localhost:6969` |
| `cognition-service` | `http://localhost:8085` |
| `connection-gateway` | `ws://localhost:8082` |
| `notification-service` | `http://localhost:8089` |
| `static-file` | `http://localhost:8100` |
| `unfurl-service` | `http://localhost:8095` |
| `contacts` | `http://localhost:8083` |
| `email-service` | `http://localhost:8087` |
| `image-proxy-service` | `http://localhost:8097` |
| `scheduled-action` | `http://localhost:8098` |
| `sync-service` | `http://localhost:8787` and `ws://localhost:8787` when selected |
<Note>
`sync-service` is handled separately: it switches to local hosts when `VITE_LOCAL_SERVERS=ALL` or the list contains `sync-service`. When sync remains remote, sync permission tokens use the remote document-storage host so secrets match the remote sync service.
</Note>
## Run the local smoke suite
From the repository root:
```bash
just local-e2e
```
The harness starts the local stack with `docker-compose.local-e2e.yml`, seeds deterministic smoke data, launches the frontend with:
```text
LOCAL_E2E=true
VITE_LOCAL_SERVERS=ALL
VITE_ENABLE_BEARER_TOKEN_AUTH=true
LOCAL_JWT=<generated-or-exported-token>
```
Then it runs Playwright from `js/app`.
Open Playwright UI mode with:
```bash
just local-e2e-ui
```
Run the frontend Playwright command directly only after the local stack and seed data are ready:
```bash
cd js/app
LOCAL_E2E=true bunx playwright test
```
## Start Tauri targets
Run native commands from the Tauri workspace under `js/app/tauri`.
<Tabs>
<Tab title="Desktop">
```bash
cd js/app/tauri
cargo tauri dev
```
</Tab>
<Tab title="iOS simulator">
```bash
cd js/app/tauri
cargo tauri ios dev
```
</Tab>
<Tab title="Android emulator">
```bash
cd js/app/tauri
cargo tauri android dev
```
</Tab>
</Tabs>
`tauri.conf.json` defines:
| Tauri build key | Value |
| --- | --- |
| `devUrl` | `http://localhost:3000` |
| `frontendDist` | `../../packages/app/dist` |
| `beforeDevCommand` | `just dev-tauri` with cwd `../..` |
| `beforeBuildCommand` | `just build-tauri` with cwd `../..` |
`just dev-tauri` starts the same Vite app. `just build-tauri` builds the production frontend bundle before Tauri packages native artifacts.
For devices or emulators that cannot resolve localhost HMR, set:
```bash
export TAURI_DEV_HOST=<host-reachable-from-device>
cargo tauri ios dev
```
Vite uses `TAURI_DEV_HOST` as the HMR host while still serving on port `3000`.
## Build native bundles
<Tabs>
<Tab title="Desktop">
```bash
cd js/app/tauri
cargo tauri build
```
</Tab>
<Tab title="iOS">
```bash
cd js/app/tauri
cargo tauri ios build
```
</Tab>
<Tab title="Android">
```bash
cd js/app/tauri
cargo tauri android build
```
</Tab>
</Tabs>
Tauri packages the static files emitted into `js/app/packages/app/dist`. The configured bundle target is `all`, with iOS minimum system version `14.0`.
For updater-server testing, create the frontend archive from `js/app`:
```bash
just dist-archive
```
That recipe builds production assets, zips `packages/app/dist`, and writes:
```text
rust/cloud-storage/tools/native_app_server/app-archive.zip
```
## Platform-specific runtime behavior
The app detects platform at runtime instead of producing separate web/mobile bundles.
| Runtime surface | Behavior |
| --- | --- |
| `getPlatform()` | returns `web`, `desktop`, `ios`, or `android` |
| `isTauri()` | checks for `window.__TAURI_INTERNALS__` |
| Router | browser uses `Router` with `/app`; Tauri uses `HashRouter` with `/` |
| Fetch | Tauri replaces global `window.fetch` with `platformFetch` except localhost requests |
| WebSocket | Tauri uses the Tauri WebSocket plugin wrapper; web uses the browser WebSocket |
| Native provider | `MaybeTauriProvider` mounts Tauri update, safe-area, share-target, push, and navigation wiring only in Tauri |
The fetch override intentionally skips localhost URLs so Vite HMR and dev-server requests continue to use normal browser fetch during Tauri development.
## iOS worker deadlock rule
<Warning>
Do not construct ES module Web Workers at module-load time in code that can run inside iOS WKWebView under Tauri.
</Warning>
The problematic pattern is eager construction:
```ts
import WorkerImpl from './worker?worker';
const worker = new WorkerImpl(); // avoid at module load on iOS
```
Use lazy construction instead. Importing a worker constructor is safe; calling `new WorkerImpl()` must happen on first use.
```ts
import HeicWorker from './heic-worker?worker';
class Service {
private static instance: Service | null = null;
private constructor() {
this.workerPool = WorkerPool.getInstance(HeicWorker);
}
static getInstance(): Service {
if (!Service.instance) Service.instance = new Service();
return Service.instance;
}
}
export const service = new Proxy({} as Service, {
get: (_target, prop, receiver) =>
Reflect.get(Service.getInstance(), prop, receiver),
});
```
Safe examples in this codebase instantiate workers inside runtime paths such as a block `load()` function or a service singleton that is reached only when a conversion/upload operation starts.
### Symptom signature
An iOS worker deadlock usually looks like:
- HTML loads.
- Initial JavaScript starts.
- JavaScript silently stops.
- Safari Web Inspector attaches but shows no useful app progress.
- The WebContent process stays alive with `0.0%` CPU.
### Diagnose an iOS worker freeze
<Steps>
<Step title="Reproduce on the simulator">
Use the iOS Simulator so logs flow through macOS unified logging.
```bash
cd js/app/tauri
cargo tauri ios dev "iPhone 15"
```
</Step>
<Step title="Stream app logs">
Use the full path because zsh has a `log` builtin.
```bash
/usr/bin/log stream --predicate 'process == "macro"' --info --debug --style compact
```
</Step>
<Step title="Filter noisy protocol logs">
```bash
tail -200 <logfile> | grep -vE 'tauri:// request|tauri_protocol.rs|^\s*\\134'
```
Look for the last meaningful URL request before silence, especially a `*-worker.js?worker_file&type=module` request.
</Step>
<Step title="Check whether WebContent is parked">
```bash
ps -o pid,pcpu,comm -p <webcontent_pid>
```
`0.0%` CPU with a live process indicates a parked deadlock rather than a hot loop.
</Step>
<Step title="Sample the WebContent process">
```bash
sample <webcontent_pid> 3 -file /tmp/sample.txt
```
Confirm worker threads are stuck in a stack containing:
```text
WorkerOrWorkletScriptController::loadModuleSynchronously
WorkerDedicatedRunLoop::runInMode
Condition::waitUntilUnchecked
```
</Step>
</Steps>
## Quality checks
Run these from `js/app` before handing off frontend changes:
```bash
bun run check
bun run lint
bun run test
```
Use the AGENTS guidance for SolidJS changes:
- Avoid `createEffect` unless syncing with an external imperative system.
- Use derived signals for derived state.
- Use `createMemo` only for referential stability or expensive derivations.
- Check `solid-primitives` before adding custom reactive utilities.
- Keep reusable components small and decoupled from query/mutation state.
- Use semantic color tokens instead of raw Tailwind color classes.
## Related pages
<CardGroup>
<Card title="Run the repository locally" href="/running-locally">
Set up Docker, LocalStack, FusionAuth, seeded data, and local services.
</Card>
<Card title="Test the app with Playwright" href="/frontend-playwright">
Use the local E2E harness and UI mode for smoke coverage.
</Card>
<Card title="Maintain Tauri runtime integrations" href="/tauri-runtime">
Native providers, bundle updates, share targets, mobile plugins, and platform adapters.
</Card>
</CardGroup>
---
## 14. Change a Rust service API
> Guide: Update Axum handlers, OpenAPI emitters, generated client specs, Orval output, and CI checks for service API changes.
- Page Markdown: https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/14-change-a-rust-service-api.md
- Generated: 2026-06-01T00:55:38.799Z
### Source Files
- `rust/cloud-storage/document_storage_service/src/openapi.rs`
- `rust/cloud-storage/document_cognition_service/src/openapi.rs`
- `js/app/scripts/generate-api-schema.ts`
- `js/app/scripts/services.ts`
- `js/app/packages/service-clients/orval.config.ts`
- `.github/workflows/web-app-check-main.yml`
- `rust/cloud-storage/AGENTS.md`
---
title: "Change a Rust service API"
description: "Guide: Update Axum handlers, OpenAPI emitters, generated client specs, Orval output, and CI checks for service API changes."
---
Rust service API changes flow from Axum handlers and `utoipa` annotations into local `*_openapi` Rust binaries, then through `js/app/scripts/generate-api-schema.ts` into committed `openapi.json` files and Orval-generated TypeScript under `js/app/packages/service-clients`.
## API generation path
```text
Rust handler + request/response models
└─ #[utoipa::path(...)] + ToSchema
└─ api/swagger.rs ApiDoc paths(...) and components.schemas(...)
└─ src/openapi.rs binary prints ApiDoc::openapi().to_pretty_json()
└─ js/app/scripts/generate-api-schema.ts writes openapi.json
└─ orval.config.ts regenerates generated client/schema files
└─ biome formats service-clients
└─ --check mode fails if git diff or untracked generated files remain
```
The live services also mount Swagger UI at `/docs` and serve the same OpenAPI document at `/api-doc/openapi.json`.
## Service registry
The generator accepts service names from `js/app/scripts/services.ts` and maps them to Rust crate names in `js/app/scripts/generate-api-schema.ts`.
| Service argument | Rust crate | OpenAPI binary | Generated package | Orval project |
| --- | --- | --- | --- | --- |
| `cloud-storage` | `document_storage_service` | `document_storage_service_openapi` | `service-storage` | `storageService` |
| `properties-service` | `properties_service` | `properties_service_openapi` | `service-properties` | `propertiesService` |
| `document-cognition` | `document_cognition_service` | `document_cognition_service_openapi` | `service-cognition` | `cognitionService` |
| `auth-service` | `authentication_service` | `authentication_service_openapi` | `service-auth` | `authService` |
| `notification-service` | `notification_service` | `notification_service_openapi` | `service-notification` | `notificationService` |
| `static-files` | `static_file_service` | `static_file_service_openapi` | `service-static-files` | `staticFileService` |
| `connection-gateway` | `connection_gateway` | `connection_gateway_openapi` | `service-connection` | `connectionGateway` |
| `contacts-service` | `contacts_service` | `contacts_service_openapi` | `service-contacts` | `contactService` |
| `unfurl-service` | `unfurl_service` | `unfurl_service_openapi` | `service-unfurl` | `unfurlService` |
| `email-service` | `email_service` | `email_service_openapi` | `service-email` | `emailService` |
| `search-service` | `search_service` | `search_service_openapi` | `service-search` | `searchService` |
| `scheduled-action` | `scheduled_action` | `scheduled_action_openapi` | `service-scheduled-action` | `scheduledActionService` |
## Change an existing endpoint
<Steps>
<Step title="Update the Axum handler contract">
Change the handler, request type, response type, status codes, and router registration in the Rust service. For request and response structs that must appear in OpenAPI, derive `utoipa::ToSchema` and keep `serde` casing attributes aligned with the wire format.
Example surfaces:
- Storage handlers live under `rust/cloud-storage/document_storage_service/src/api/**`.
- Cognition handlers live under `rust/cloud-storage/document_cognition_service/src/api/**`.
- Shared request/response models may live under service-local `model/**` modules or shared crates such as `model`, `models_*`, `chat`, or `memory`.
</Step>
<Step title="Update the `utoipa::path` annotation">
Keep the OpenAPI operation synchronized with the handler:
- HTTP method: `get`, `post`, `put`, `patch`, or `delete`
- `path = "..."` matching the mounted API path
- `tag = "..."`
- request body and params, when used
- `responses((status = ..., body = ...))`
If a route is mounted under both `/{version}` and the unversioned router, document the unversioned path in `#[utoipa::path]`, matching the existing service pattern.
</Step>
<Step title="Register the operation and schemas in `api/swagger.rs`">
Add new handlers to `paths(...)` and add request/response models to `components(schemas(...))`.
Storage uses `rust/cloud-storage/document_storage_service/src/api/swagger.rs`.
Cognition uses `rust/cloud-storage/document_cognition_service/src/api/swagger.rs`.
Missing `components.schemas` entries commonly produce incomplete generated TypeScript even when the Rust code compiles.
</Step>
<Step title="Regenerate API artifacts">
From the web app workspace, regenerate only the service you changed when possible.
```bash
cd js/app
bun run gen-api -- cloud-storage
bun run gen-api -- document-cognition
```
To regenerate every registered service:
```bash
cd js/app
bun run gen-api
```
The script builds OpenAPI binaries with `SQLX_OFFLINE=true cargo build`, runs each binary, writes `openapi.json`, removes the previous `generated/` directory, runs Orval, and then runs Biome on `packages/service-clients/`.
</Step>
<Step title="Commit generated files">
Commit the service `openapi.json` and all changed files under that service package’s `generated/` directory. Do not hand-edit generated files; fix the Rust schema source or Orval config and regenerate.
</Step>
</Steps>
## Document Cognition special generation
`document-cognition` has additional generated artifacts beyond Orval output.
When the target service includes `document-cognition`, `generate-api-schema.ts` also builds or runs:
- `document_cognition_service_models`
- `gen_tool_schemas`
It then runs `scripts/generate-dcs-types.ts`, which generates:
- `service-cognition/generated/schemas/model.ts`
- `service-cognition/generated/tools/schemas.ts`
- `service-cognition/generated/tools/types.ts`
- `service-cognition/generated/tools/tool.ts`
<Note>
The models generator can read `MODELS_JSON` from a local file. The main API generation script sets this automatically after running the Rust `document_cognition_service_models` binary.
</Note>
## Add a new service to generation
For a new Rust service client, add all registry points together:
1. Add a Rust OpenAPI binary, usually named `<crate>_openapi`, that prints `ApiDoc::openapi().to_pretty_json()`.
2. Add a `[[bin]]` entry in the service `Cargo.toml`.
3. Add the service entry in `js/app/scripts/services.ts` with `name`, `dev`, `prod`, `local`, `output`, and `orvalKey`.
4. Add the service-to-crate mapping in `js/app/scripts/generate-api-schema.ts`.
5. Add a matching Orval project in `js/app/packages/service-clients/orval.config.ts`.
6. Run targeted generation and commit `openapi.json` plus `generated/`.
## Orval output modes
`js/app/packages/service-clients/orval.config.ts` controls the generated TypeScript shape.
| Orval client mode | Services in this repo | Output pattern |
| --- | --- | --- |
| `fetch` | auth, cognition, connection, contacts, email, scheduled action, search, static files, unfurl | `generated/client.ts` plus schema files |
| `zod` with `mode: "split"` | notification, properties, storage | `generated/zod.ts` plus split schema files |
Storage, properties, and notification generate Zod validators. Most other services generate fetch clients and TypeScript schemas. Search writes schemas under `service-search/generated/models` instead of `generated/schemas`.
## Verification
Run the checks that match the change size.
```bash
cd js/app
bun run gen-api -- <service-name>
bun run gen-api -- --check
bun run --bun --silent tsc --project ./packages/app/tsconfig.json
```
For Rust changes:
```bash
cd rust/cloud-storage
cargo test -p <crate-name>
just check
just clippy
just format
```
If the API change includes SQLx query or schema changes, update the SQLx offline cache from the workspace root:
```bash
cd rust/cloud-storage
just prepare_db
```
## CI behavior
The web app PR workflow has two path filters:
- `should_run` for frontend package, app, lockfile, Biome, and workflow changes.
- `api_changed` for `rust/cloud-storage/**/*.rs`, Rust lock/config files, flake files, API generation scripts, setup actions, and the workflow.
When `api_changed` is true, the Typecheck job:
1. sets up web prerequisites,
2. restores Rust cache for `rust/cloud-storage`,
3. runs `bun run gen-api -- --check` in `js/app`,
4. runs TypeScript checking.
`--check` mode fails when generated service-client files differ from the committed tree or when untracked generated files exist.
## Troubleshooting
### `Generated types are out of sync with Rust API definitions`
Run generation locally and commit the resulting changes:
```bash
cd js/app
bun run gen-api
```
For a smaller diff, target the changed service:
```bash
bun run gen-api -- cloud-storage
```
### `No matching services found`
Use one of the `name` values from `js/app/scripts/services.ts`, such as `cloud-storage`, `document-cognition`, or `search-service`.
### `No crate mapping found, skipping`
The service exists in `services.ts` but is missing from `serviceToCrate` in `generate-api-schema.ts`. Add the crate mapping before regenerating.
### TypeScript type is missing or too broad
Check the Rust OpenAPI source:
- the model derives `ToSchema`,
- the model is included in `components(schemas(...))`,
- the handler is included in `paths(...)`,
- the `#[utoipa::path]` response body names the expected type.
### Orval writes to the wrong package
Keep these values aligned:
- `services.ts` `output`
- `services.ts` `orvalKey`
- `orval.config.ts` project key
- `orval.config.ts` `input.target`
- `orval.config.ts` `output.target` and `output.schemas`
### OpenAPI binary fails in generation
The generator runs binaries from `rust/cloud-storage/target/debug` unless `OPENAPI_BINS_DIR` is set. Each binary has a 120 second timeout and reports captured stderr on non-zero exit or timeout.
To skip the cargo build phase with prebuilt binaries:
```bash
cd js/app
OPENAPI_BINS_DIR=/path/to/bins bun run gen-api -- <service-name>
```
## Related pages
<CardGroup>
<Card title="Rust cloud-storage development" href="/rust-cloud-storage-development">
Service testing, SQLx offline cache, and workspace commands.
</Card>
<Card title="Generated service clients" href="/generated-service-clients">
TypeScript client package layout and generated artifact conventions.
</Card>
</CardGroup>
---
## 15. Generate service clients and tool schemas
> Guide: Regenerate OpenAPI JSON, TypeScript clients, DCS model types, AI tool schemas, and MCP documentation pages from Rust sources.
- Page Markdown: https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/15-generate-service-clients-and-tool-schemas.md
- Generated: 2026-06-01T00:55:20.399Z
### Source Files
- `js/app/scripts/generate-api-schema.ts`
- `js/app/scripts/generate-dcs-tools.ts`
- `js/app/scripts/services.ts`
- `rust/cloud-storage/ai_tools/src/lib.rs`
- `rust/cloud-storage/ai_tools/src/bin/gen_tool_schemas.rs`
- `docs/scripts/generate-mcp-tool-pages.ts`
- `docs/package.json`
---
title: "Generate service clients and tool schemas"
description: "Guide: Regenerate OpenAPI JSON, TypeScript clients, DCS model types, AI tool schemas, and MCP documentation pages from Rust sources."
---
Macro regenerates frontend service clients and AI tool artifacts from local Rust binaries under `rust/cloud-storage`; `js/app/scripts/generate-api-schema.ts` builds OpenAPI emitters, writes `openapi.json`, runs Orval, and applies DCS-specific model and tool generation when `document-cognition` is included.
## Generated artifact map
```text
Rust sources
├─ */src/openapi.rs -> service-*/openapi.json
├─ document_cognition_service/src/models_bin.rs -> service-cognition/generated/schemas/model.ts
└─ ai_tools/src/bin/gen_tool_schemas.rs -> ai_tools/schemas/tools.json
-> service-cognition/generated/tools/*
-> docs/AI/mcp/tools/*
```
| Artifact | Generator | Output |
| --- | --- | --- |
| OpenAPI JSON | `js/app/scripts/generate-api-schema.ts` | `js/app/packages/service-clients/service-*/openapi.json` |
| TypeScript service clients | Orval via `orval.config.ts` | `generated/client.ts`, `generated/zod.ts`, and generated schemas/models |
| DCS model enum | `document_cognition_service_models` + `generate-dcs-models.ts` | `service-cognition/generated/schemas/model.ts` |
| DCS AI tool validators and types | `gen_tool_schemas` + `generate-dcs-tools.ts` | `service-cognition/generated/tools/{schemas,types,tool}.ts` |
| MCP tool pages | `docs/scripts/generate-mcp-tool-pages.ts` | `docs/AI/mcp/tools/*.mdx`, `docs/config/tool-pages.json` |
<Note>
The main OpenAPI workflow runs local Rust binaries instead of fetching deployed `/api-doc/openapi.json` endpoints.
</Note>
## Prerequisites
- Bun installed for `js/app` and `docs` scripts.
- Rust toolchain available for `cargo build` in `rust/cloud-storage`.
- Run generation from the package directory shown in each command.
- SQLx is built in offline mode by the scripts with `SQLX_OFFLINE=true`.
## Regenerate service clients
<Steps>
<Step title="Generate all service clients">
```bash
cd js/app
bun run gen-api
```
This processes every service listed in `js/app/scripts/services.ts`.
</Step>
<Step title="Generate one service">
```bash
cd js/app
bun run gen-api -- document-cognition
```
Valid service names include `cloud-storage`, `properties-service`, `document-cognition`, `auth-service`, `notification-service`, `static-files`, `connection-gateway`, `contacts-service`, `unfurl-service`, `email-service`, `search-service`, and `scheduled-action`.
</Step>
<Step title="Verify generated clients are committed">
```bash
cd js/app
bun run gen-api -- --check
```
Check mode regenerates artifacts, compares `js/app/packages/service-clients`, reports changed and untracked files, and exits non-zero when generated files are out of sync.
</Step>
</Steps>
### Service and crate mapping
`generate-api-schema.ts` maps public service names to Rust crate binaries. Each mapped crate must expose an OpenAPI binary named `<crate>_openapi`.
| Service | Rust crate | Orval project | Output package |
| --- | --- | --- | --- |
| `cloud-storage` | `document_storage_service` | `storageService` | `service-storage` |
| `properties-service` | `properties_service` | `propertiesService` | `service-properties` |
| `document-cognition` | `document_cognition_service` | `cognitionService` | `service-cognition` |
| `auth-service` | `authentication_service` | `authService` | `service-auth` |
| `notification-service` | `notification_service` | `notificationService` | `service-notification` |
| `static-files` | `static_file_service` | `staticFileService` | `service-static-files` |
| `connection-gateway` | `connection_gateway` | `connectionGateway` | `service-connection` |
| `contacts-service` | `contacts_service` | `contactService` | `service-contacts` |
| `unfurl-service` | `unfurl_service` | `unfurlService` | `service-unfurl` |
| `email-service` | `email_service` | `emailService` | `service-email` |
| `search-service` | `search_service` | `searchService` | `service-search` |
| `scheduled-action` | `scheduled_action` | `scheduledActionService` | `service-scheduled-action` |
### Build behavior
The script builds all requested OpenAPI binaries in one Cargo invocation:
```bash
cd rust/cloud-storage
SQLX_OFFLINE=true cargo build --bin <crate>_openapi ...
```
Then it runs each binary from `target/debug`, captures stdout as OpenAPI JSON, writes the service package `openapi.json`, removes the old `generated` directory, and runs:
```bash
cd js/app/packages/service-clients
bun run orval --config orval.config.ts --project <orvalKey>
```
Set `OPENAPI_BINS_DIR` to reuse prebuilt binaries and skip the Cargo build phase:
```bash
cd js/app
OPENAPI_BINS_DIR=/absolute/path/to/bins bun run gen-api
```
## DCS model and tool generation
When `document-cognition` is part of the service set, `generate-api-schema.ts` performs extra work after Orval:
1. Runs `document_cognition_service_models`.
2. Writes temporary `.models.json`.
3. Runs `MODELS_JSON=.models.json bun scripts/generate-dcs-types.ts`.
4. Deletes `.models.json`.
`generate-dcs-types.ts` runs both DCS generators:
```bash
cd js/app
bun scripts/generate-dcs-types.ts
```
### Model enum output
`generate-dcs-models.ts` writes:
```text
js/app/packages/service-clients/service-cognition/generated/schemas/model.ts
```
When `MODELS_JSON` is set, it reads model metadata from the local file produced by the Rust model binary. Without `MODELS_JSON`, it fetches `${documentCognitionBase}/models`, where `MODE` and `LOCAL_BACKEND=true` control the selected base URL.
The generated file exports:
- `Model`
- `Model` constant map
- `AllModels`
- `ModelEnum`
### Tool schema output
Run the tool generator directly when only AI tool types changed:
```bash
cd js/app
bun run gen-tools
```
The generator builds and runs:
```bash
cd rust/cloud-storage
SQLX_OFFLINE=true cargo build --bin gen_tool_schemas
cd rust/cloud-storage/ai_tools
../target/debug/gen_tool_schemas
```
The Rust binary writes `rust/cloud-storage/ai_tools/schemas/tools.json` as a combined schema:
```json
{
"$defs": {},
"tools": [
{
"name": "ContentSearch",
"input": "ContentSearch",
"output": "SearchToolResponse"
}
]
}
```
`generate-dcs-tools.ts` validates that combined shape, dereferences shared definitions, and writes:
| File | Purpose |
| --- | --- |
| `generated/tools/schemas.ts` | Zod v3 validators generated from JSON Schema definitions |
| `generated/tools/types.ts` | TypeScript definition types generated from `$defs` |
| `generated/tools/tool.ts` | `ToolName`, `NamedTool`, `deserializeToolCall`, and `deserializeToolResponse` |
`deserializeToolCall` and `deserializeToolResponse` return `neverthrow` `Result` values. Unknown tool names return `not_found`; schema parse failures return `parse_error`.
## Regenerate MCP tool documentation pages
The docs package exposes a separate generator:
```bash
cd docs
bun install
bun run generate:tools
```
This script rebuilds `gen_tool_schemas`, removes the generated MCP tools directory, then writes:
```text
docs/AI/mcp/tools/index.mdx
docs/AI/mcp/tools/<slug>.mdx
docs/config/tool-pages.json
```
`docs/package.json` also runs this generator during `prepare`.
<Warning>
In this checkout, `gen_tool_schemas` writes the combined `$defs`/`tools` shape used by the app tool generator, while `docs/scripts/generate-mcp-tool-pages.ts` is typed to read a `schemas` array with inline `inputSchema` and `outputSchema`. If `bun run generate:tools` fails while sorting `toolSchemas.schemas`, align the docs generator with the combined schema or change the Rust schema emitter intentionally.
</Warning>
## Runtime and schema ownership
`rust/cloud-storage/ai_tools/src/lib.rs` owns toolset composition:
- `all_tools()` builds the DCS chat toolset and prompt.
- `all_tool_combined_schema()` merges `all_tools()` with the `ReadThread` phantom tool for schema generation.
- `mcp_tools()` builds the MCP runtime toolset separately.
Do not assume generated MCP documentation is automatically scoped to the MCP runtime toolset unless the schema generator is changed to use `mcp_tools()`.
## Verification
After regeneration, check the generated files that match the source change:
```bash
git diff -- js/app/packages/service-clients
git diff -- rust/cloud-storage/ai_tools/schemas/tools.json
git diff -- docs/AI/mcp/tools docs/config/tool-pages.json
```
For frontend type safety:
```bash
cd js/app
bun run type-check
```
For docs links:
```bash
cd docs
bun run lint
```
CI runs `bun run gen-api -- --check` for Rust/API changes in the web app workflow and fails when generated service clients drift from Rust sources.
## Troubleshooting
| Symptom | Cause | Fix |
| --- | --- | --- |
| `No matching services found` | Unknown service argument | Use a name from `services.ts` |
| Service skipped | Missing `serviceToCrate` entry | Add the service-to-crate mapping |
| Binary timeout after `120000ms` | OpenAPI/model binary hung or took too long | Run the specific Cargo binary locally and inspect stderr |
| Generated clients out of sync in CI | Rust API changed without committed generated files | Run `cd js/app && bun run gen-api`, then commit outputs |
| Biome binary fails on NixOS | npm-installed Biome dynamic linking issue | The scripts detect NixOS and use system `biome` when available |
| Docs tool generation fails on `schemas` | Docs generator expects the old inline schema shape | Update `generate-mcp-tool-pages.ts` for `$defs`/`tools` or restore the expected Rust output |
## Related pages
<CardGroup>
<Card title="MCP overview" href="/AI/mcp/overview">
Runtime context for Macro MCP integration.
</Card>
<Card title="MCP tool reference" href="/AI/mcp/tools">
Generated tool pages produced from Rust tool schemas.
</Card>
</CardGroup>
---
## 16. Work with local seed data
> Guide: Use seed_cli, local_e2e fixtures, manifest aliases, reset SQL, and shared Playwright and Rust fixture loaders.
- Page Markdown: https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/16-work-with-local-seed-data.md
- Generated: 2026-06-01T00:55:38.119Z
### Source Files
- `rust/cloud-storage/seed_cli/README.md`
- `rust/cloud-storage/seed_cli/src/main.rs`
- `rust/cloud-storage/seed_cli/seed/local_e2e/manifest.json`
- `rust/cloud-storage/seed_cli/seed/local_e2e/users.json`
- `rust/cloud-storage/seed_cli/seed/local_e2e/reset.sql`
- `js/app/tests/e2e/fixtures/local-e2e-seed.ts`
- `rust/cloud-storage/local_e2e_test_support/README.md`
---
title: "Work with local seed data"
description: "Guide: Use seed_cli, local_e2e fixtures, manifest aliases, reset SQL, and shared Playwright and Rust fixture loaders."
---
`seed_cli` owns the deterministic local fixture set in `rust/cloud-storage/seed_cli/seed`, and the repo-level local E2E harness applies it with `just local-e2e-seed` before Playwright or ignored Rust integration tests run against the local Docker stack.
## Fixture layout
:::files
```text
rust/cloud-storage/seed_cli/
README.md
justfile
seed/
channels.json
channel_messages.json
documents/
documents.json
files/
local_e2e/
manifest.json
reset.sql
users.json
src/
main.rs
entity/
scenario/mod.rs
document/mod.rs
channel/mod.rs
channel_message/mod.rs
```
:::
| File | Purpose |
| --- | --- |
| `seed/local_e2e/manifest.json` | Stable aliases for the smoke user, smoke document, general channel, and canonical welcome message. |
| `seed/local_e2e/users.json` | Local user fixtures shared by database seed code, Playwright fixture loading, and Rust token generation. |
| `seed/local_e2e/reset.sql` | Destructive cleanup for seeded channel, message, document, mention, and share-permission ranges. |
| `seed/documents/documents.json` | Document rows with stable UUIDs, display names, fixture file names, and public flags. |
| `seed/channels.json` | Channel rows with stable UUIDs, optional names, channel types, and participants. |
| `seed/channel_messages.json` | Message rows with stable UUIDs, senders, optional thread IDs, and optional entity mentions. |
## Run the local seed
<Steps>
<Step title="Start and seed the local database">
```bash
just local-e2e-seed
```
This starts local databases, drops and initializes `macrodb`, then runs the seed CLI local E2E scenario.
</Step>
<Step title="Run Playwright against the seeded stack">
```bash
just local-e2e
```
The harness starts the local E2E service subset, applies the seed, launches the frontend with local bearer-token auth, and runs `bunx playwright test` with `LOCAL_E2E=true`.
</Step>
<Step title="Run Rust integration tests against the same seed">
```bash
just local-e2e-rust
```
Rust local E2E tests are ignored by default and run with `SQLX_OFFLINE=true`.
</Step>
<Step title="Run both suites after one stack startup">
```bash
just local-e2e-all
```
</Step>
</Steps>
<Warning>
`scenario local-e2e-smoke` is destructive. It requires `LOCAL_E2E_SEED=true` and refuses database URLs that are not the local Docker database shape `postgres://user:...@(localhost|127.0.0.1|postgres):5432/macrodb`.
</Warning>
## Seed CLI behavior
The CLI entrypoint initializes the local Macro runtime, loads required environment variables, connects to Postgres, initializes FusionAuth and S3 clients, then dispatches an entity command.
The local smoke scenario is:
```bash
cd rust/cloud-storage/seed_cli
just local-e2e-smoke
```
Internally it runs:
```bash
cargo r -- scenario local-e2e-smoke
```
The `local-e2e-smoke` recipe supplies local-only defaults, including:
| Variable | Value used by recipe |
| --- | --- |
| `LOCAL_E2E_SEED` | `true` |
| `DATABASE_URL` | `postgres://user:password@localhost:5432/macrodb` |
| `LOCAL_AWS_URL` | `http://localhost:4566` |
| `DOCUMENT_STORAGE_BUCKET` | `doc-storage` |
| `FUSIONAUTH_BASE_URL` | `http://localhost:9011` |
| `SQLX_OFFLINE` | `true` |
| `ENVIRONMENT` | `local` |
### Local smoke scenario order
1. Load `local_e2e/manifest.json` and `local_e2e/users.json`.
2. Resolve the primary smoke user from `manifest.user.email`.
3. Delete local E2E contact backfill rows if `public.contacts_backfill_outbox` exists.
4. Execute `local_e2e/reset.sql`.
5. Delete and reinsert local user rows derived from `users.json`.
6. Seed documents from `seed/documents/documents.json`.
7. Seed channels from `seed/channels.json`.
8. Seed channel messages from `seed/channel_messages.json`.
9. Print `Local e2e smoke seed data ready for <user_id>`.
### Reset scope
`reset.sql` removes fixture data by deterministic UUID prefixes:
| Data | Reset condition |
| --- | --- |
| `entity_access` | Seed channel source IDs or seed document entity IDs. |
| `ChannelSharePermission` | Seed channel IDs. |
| `comms_entity_mentions` | Seed message IDs or seed document entity IDs. |
| `comms_activity` | Seed channel IDs. |
| `comms_channels` | Seed channel IDs. |
| `Document` | Seed document IDs. |
User cleanup is generated from `local_e2e/users.json` and deletes matching rows from `"User"` and `macro_user` by auth user ID, macro user ID, or email before reinsert.
## Manifest aliases
`local_e2e/manifest.json` is the stable contract for smoke tests. It does not duplicate full fixture rows; it names canonical fixture records that must exist in the seed files.
```json
{
"user": {
"email": "e2e@macro.local"
},
"documents": {
"projectRoadmap": {
"id": "00000000-0000-0000-0002-000000000001",
"name": "Project Roadmap"
}
},
"channels": {
"general": {
"id": "00000000-0000-0000-0000-000000000001",
"name": "general",
"message": "Welcome to the general channel everyone!"
}
}
}
```
When adding a new smoke alias, update the manifest and the underlying seed file in the same change. The shared Playwright and Rust loaders fail fast when an alias points at a missing user, document, channel, or message.
## Playwright fixture loader
`js/app/tests/e2e/fixtures/local-e2e-seed.ts` reads the seed JSON directly from the repository root. It locates the root by walking upward until `rust/cloud-storage/seed_cli/seed` exists.
The exported `localE2ESeed` object includes:
| Property | Contents |
| --- | --- |
| `user` | Primary smoke user resolved from `manifest.user.email`. |
| `users`, `documents`, `channels`, `channelMessages` | Full fixture arrays. |
| `usersById`, `usersByEmail`, `usersByMacroUserId` | User lookup maps. |
| `documentsById`, `documentsByName` | Document lookup maps. |
| `channelsById`, `channelsByName` | Channel lookup maps. |
| `channelMessagesById` | Message lookup map. |
| `channelMessagesByChannelId(channelId)` | Messages filtered by channel ID. |
| `smoke.projectRoadmap` | Document row for `manifest.documents.projectRoadmap.id`. |
| `smoke.generalChannel` | Channel row for `manifest.channels.general.id`. |
| `smoke.generalWelcomeMessage` | Message in the general channel matching `manifest.channels.general.message`. |
Local Playwright specs skip unless `LOCAL_E2E=true`. The Playwright config generates `LOCAL_JWT` automatically for local E2E mode unless `LOCAL_JWT` is already exported.
```bash
cd js/app
LOCAL_E2E=true bunx playwright test
```
Prefer the repo-level harness:
```bash
just local-e2e
```
It seeds first and starts the frontend with:
```bash
VITE_LOCAL_SERVERS=ALL
VITE_ENABLE_BEARER_TOKEN_AUTH=true
LOCAL_JWT=<generated token>
bun run dev
```
## Rust fixture loader
`local_e2e_test_support` is the Rust counterpart for ignored local integration tests. It reads the same seed files and exposes typed fixture accessors, service URL helpers, and local Macro API JWT generation.
```rust
use local_e2e_test_support::{
LocalE2eConfig, LocalE2eSeed, LocalE2eServices, LocalJwtOptions,
encode_local_jwt_with,
};
let config = LocalE2eConfig::load()?;
let seed = LocalE2eSeed::from_config(&config)?;
let services = LocalE2eServices::from_config(&config)?;
let user = seed.smoke_user()?;
let channel = seed.general_channel()?;
let token = encode_local_jwt_with(&config, LocalJwtOptions::new(user))?;
let ws_url = services.connection_gateway_ws_url_with_token(&token)?;
```
### Rust helper defaults
| Helper | Default |
| --- | --- |
| Seed directory | `<repo>/rust/cloud-storage/seed_cli/seed` |
| Document storage URL | `http://localhost:8086` |
| Connection gateway WebSocket URL | `ws://localhost:8082/` |
| Notification service URL | `http://localhost:8089` |
| JWT issuer | `MACRO_API_TOKEN_ISSUER`, or `local` |
| JWT expiry | `MACRO_API_TOKEN_EXPIRY_SECONDS`, or 8 hours |
Service URL overrides must remain local. The helper rejects non-local hosts and mismatched schemes for mutating local E2E tests.
| Override | Accepted schemes |
| --- | --- |
| `LOCAL_E2E_DOCUMENT_STORAGE_URL` | `http`, `https` |
| `LOCAL_E2E_CONNECTION_GATEWAY_WS_URL` | `ws`, `wss` |
| `LOCAL_E2E_NOTIFICATION_URL` | `http`, `https` |
## Generate a local E2E token
Playwright calls `js/app/scripts/generate-local-e2e-token.ts`, which shells out to the Rust binary in `local_e2e_test_support`.
```bash
cd js/app
bun scripts/generate-local-e2e-token.ts
```
Optional arguments are forwarded to the Rust generator:
```bash
bun scripts/generate-local-e2e-token.ts \
--email bob@example.com \
--expiry-seconds 3600 \
--organization-id 1
```
| Argument | Default |
| --- | --- |
| `--email` | `manifest.user.email` |
| `--macro-user-id` | Matching `users.json` `user_id`, or `macro|<email>` |
| `--fusion-user-id` | Matching `users.json` `fusion_user_id`, then `macro_user_id`, then the first local fixture UUID |
| `--expiry-seconds` | Argument, then `MACRO_API_TOKEN_EXPIRY_SECONDS`, then 8 hours |
| `--organization-id` | Omitted |
| `--issuer` | Argument, then `MACRO_API_TOKEN_ISSUER`, then `local` |
`MACRO_API_TOKEN_PRIVATE_SECRET_KEY` is required in `.env` or the process environment.
## Update fixture data safely
### Add or change a seeded user
1. Edit `rust/cloud-storage/seed_cli/seed/local_e2e/users.json`.
2. Keep `macro_user_id`, `fusion_user_id`, and `user_id` stable once tests depend on them.
3. If this is the primary smoke user, update `manifest.user.email`.
4. Run `just local-e2e-seed`.
5. Run at least one consumer:
```bash
just local-e2e
# or
just local-e2e-rust
```
### Add or change a seeded document
1. Add a row to `seed/documents/documents.json`.
2. Put the source file under `seed/documents/files/`.
3. Use a deterministic document ID in the `00000000-0000-0000-0002-*` range if it should be cleaned by `reset.sql`.
4. Update `manifest.documents` only for canonical smoke-test aliases.
5. Run `just local-e2e-seed`.
Document seeding creates database metadata and uploads the fixture file to the configured document storage bucket.
### Add or change a seeded channel
1. Edit `seed/channels.json`.
2. Use a deterministic channel ID in the `00000000-0000-0000-0000-00000000000*` range if it should be cleaned by `reset.sql`.
3. Set `channel_type` to a value accepted by the seed CLI model, such as `public`, `private`, or `direct_message`.
4. List participants excluding or including the owner; the seed command appends the scenario owner when absent.
5. Update `manifest.channels` only for canonical smoke-test aliases.
### Add or change a seeded message
1. Edit `seed/channel_messages.json`.
2. Use a deterministic message ID in the `00000000-0000-0000-0001-*` range if it should be cleaned by `reset.sql`.
3. Set `channel_id` to an existing seeded channel.
4. Set `sender_id` to a seeded auth user ID such as `macro|bob@example.com`.
5. Use `thread_id` only when the message is a reply.
6. Add `entity_mentions` when the message content references a document or user.
For non-user shareable mentions, the seed command attempts to create message mentions and update channel share permissions for the mentioned entity.
## Troubleshooting
| Symptom | Check |
| --- | --- |
| `refusing to run destructive local-e2e-smoke seed without LOCAL_E2E_SEED=true` | Use `just local-e2e-seed` or `just rust/cloud-storage/seed_cli/local-e2e-smoke` instead of invoking the scenario without the guard variable. |
| `refusing to run local-e2e-smoke seed against DATABASE_URL ...` | Point `DATABASE_URL` at the local Docker database with user `user`, database `macrodb`, and port `5432`. |
| Playwright cannot generate `LOCAL_JWT` | Run `just local-e2e-seed`, ensure `.env` exists from the local setup flow, or export `LOCAL_JWT` manually. |
| Missing fixture error in Playwright or Rust | Verify the alias in `local_e2e/manifest.json` matches an actual row in the corresponding JSON seed file. |
| Rust helper rejects a service URL | Use `localhost`, `127.0.0.1`, or `::1`; the helper refuses non-local service URLs. |
| Seed command reports an empty JSON file | Seed commands bail on empty document, channel, or message arrays. |
## Related pages
<CardGroup>
<Card title="Run the repository locally" href="/running-locally">
Local stack setup, local E2E commands, and environment prerequisites.
</Card>
<Card title="Local E2E integration tests" href="/rust-cloud-storage-integration-tests-local-e2e">
Rust test harness behavior for the deterministic local E2E stack.
</Card>
</CardGroup>
---
## 17. Storage and workspace APIs
> Reference: Document, channel, project, call, CRM, soup, pin, history, permissions, properties, and search endpoints.
- Page Markdown: https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/17-storage-and-workspace-apis.md
- Generated: 2026-06-01T01:00:28.495Z
### Source Files
- `js/app/packages/service-clients/service-storage/openapi.json`
- `js/app/packages/service-clients/service-storage/client.ts`
- `rust/cloud-storage/document_storage_service/src/openapi.rs`
- `js/app/packages/service-clients/service-properties/openapi.json`
- `js/app/packages/service-clients/service-properties/client.ts`
- `js/app/packages/service-clients/service-search/openapi.json`
- `js/app/packages/service-clients/service-search/client.ts`
---
title: "Storage and workspace APIs"
description: "Reference: Document, channel, project, call, CRM, soup, pin, history, permissions, properties, and search endpoints."
---
Macro’s workspace API surface is served primarily by Document Storage Service (DSS), with `/properties` and `/search` mounted on the same `document-storage-service` host and consumed from TypeScript clients that wrap `fetchWithToken` and return `neverthrow` `Result` values.
## Runtime and client boundaries
| Surface | TypeScript entry point | Host | Server mount |
|---|---|---:|---|
| Documents, channels, projects, soup, pins, history, permissions | `js/app/packages/service-clients/service-storage/client.ts` | `SERVER_HOSTS['document-storage-service']` | DSS root router |
| Calls | `js/app/packages/service-clients/service-call/client.ts` plus storage schemas | same DSS host | `/call` |
| Properties | `js/app/packages/service-clients/service-properties/client.ts` | same DSS host | `/properties` |
| Search | `js/app/packages/service-clients/service-search/client.ts` | same DSS host | `/search` |
The Rust OpenAPI JSON for DSS is produced from `rust/cloud-storage/document_storage_service/src/openapi.rs`, which prints the `utoipa` `ApiDoc`. The checked-in frontend OpenAPI snapshots under `js/app/packages/service-clients/*/openapi.json` and generated schema directories provide the wire types used by the clients.
<Note>
Client methods return `Result<T, ResultError[]>`; callers should branch on `isOk()` / `isErr()` or use the project’s query helpers rather than assuming failed HTTP responses throw.
</Note>
## Documents
### Core document operations
| Operation | Method and path | Client method | Notes |
|---|---|---|---|
| List recent user documents | `GET /documents?limit&offset` | `getUserDocuments` | Client maps `data.documents`, `total`, and `next_offset` to `nextOffset`. |
| Create upload-backed document | `POST /documents` | `createDocument` | Returns metadata plus S3 presigned upload URL, content type, and optional file type. |
| Create markdown document | `POST /documents/create_markdown` | `createMarkdownDocument` | Backend initializes sync-service content and returns `documentId`. |
| Create task document | `POST /documents/create_task` | `createTask` | Creates task document, initializes markdown content, and attaches task properties. |
| Get metadata | `GET /documents/{document_id}` or `/{version}` | `getDocumentMetadata` | Client retries with exponential delay up to five tries. |
| Edit metadata/share settings | `PATCH /documents/{document_id}` | `editDocument` | Used for name/project/share-permission edits. |
| Save PDF modification data | `PUT /documents/{document_id}` | `pdfSave` | Client validates modification data and response metadata with Zod. |
| Simple file/text save | `PUT /documents/{document_id}/simple_save` | `simpleSave`, `simpleSaveText` | Uses `FormData` upload body. |
| Soft delete | `DELETE /documents/{document_id}` | `deleteDocument` | Returns success data. |
| Permanent delete | `DELETE /documents/{document_id}/permanent` | `permanentlyDeleteDocument` | Removes a soft-deleted document permanently. |
| Restore soft delete | `PUT /documents/{document_id}/revert_delete` | `revertDocumentDelete` | Reverts document deletion. |
| Copy document | `POST /documents/{document_id}/copy?version_id=` | `copyDocument` | Can copy a specific document version. |
| Export | `GET /documents/{document_id}/export` | `exportDocument` | Returns export payload. |
| Batch previews | `POST /documents/preview` | `getBatchDocumentPreviews` | Body: `{ "document_ids": string[] }`. |
| List searchable docs | `GET /documents/list` | `listDocuments` | Used to populate document search surfaces. |
Creation and copy paths show the file-limit paywall when the returned error message includes `403`.
### Locations and document bytes
| Operation | Method and path | Client method | Cache behavior |
|---|---|---|---|
| Writer part URLs | `GET /documents/{uuid}/location?document_version_id=` | `getWriterPartUrls` | Cached for 14 minutes. |
| Document location v3 | `GET /documents/{document_id}/location_v3?get_converted_docx_url&document_version_id=` | `getDocumentLocation` | Cached for 14 minutes. |
| DOCX expanded file | metadata + writer parts | `getDocxFile` | Cached for 10 seconds; fails if file type is not `docx`. |
| Text document | metadata + location + presigned fetch | `getTextDocument` | Cached for 2 seconds; requires a `presignedUrl` location. |
| Binary document | metadata + location | `getBinaryDocument` | Returns `blobUrl` from the presigned URL. |
Presigned URL caching uses a 14-minute client lifetime because the server expiration is treated as 15 minutes.
### Processing and task helpers
| Operation | Path | Client method |
|---|---|---|
| Document processing result | `GET /documents/{document_id}/processing` | `getDocumentProcessingResult` |
| Job processing result | `GET /documents/{document_id}/processing/{job_id}` | `getJobProcessingResult` |
| Task short ID | `GET /documents/{document_id}/short_id` | `getDocumentShortId` |
| Task branch name | `GET /documents/{document_id}/branch_name` | `getDocumentBranchName` |
| GitHub pull requests | `GET /documents/{document_id}/github_prs` | `getDocumentGithubPullRequests` |
| Wake sync service | `POST /sync_service/wakeup` | `bulkWakeupSyncServiceDocuments` |
Processing helpers currently validate `PREPROCESS` results against `CoParseSchema`; missing or invalid JSON returns `INVALID_RESPONSE`.
## Annotations
The `storageServiceClient.annotations` namespace wraps document comments and anchors.
| Operation | Path |
|---|---|
| List comments | `GET /annotations/comments/document/{document_id}` |
| List anchors | `GET /annotations/anchors/document/{document_id}` |
| Create comment | `POST /annotations/comments/document/{document_id}` |
| Create unthreaded anchor | `POST /annotations/anchors/document/{document_id}` |
| Edit comment | `PATCH /annotations/comments/comment/{comment_id}` |
| Edit anchor | `PATCH /annotations/anchors` |
| Delete comment | `DELETE /annotations/comments/comment/{comment_id}` |
| Delete unthreaded anchor | `DELETE /annotations/anchors` |
## Channels
### Channel lifecycle
| Operation | Method and path | Client method |
|---|---|---|
| Create channel | `POST /channels` | `createChannel` |
| List channels | `GET /comms/channels` | `getChannels` |
| Get or create DM | `POST /channels/get_or_create_dm` | `getOrCreateDirectMessage` |
| Get or create private channel | `POST /channels/get_or_create_private` | `getOrCreatePrivateChannel` |
| Rename/update channel | `PATCH /channels/{channel_id}` | `patchChannel` |
| Delete channel | `DELETE /channels/{channel_id}` | `deleteChannel` |
| Batch previews | `POST /channels/preview` | `getBatchChannelPreviews` |
| Join / leave | `POST /channels/{channel_id}/join`, `/leave` | `joinChannel`, `leaveChannel` |
`getChannels` still targets `/comms/channels` on the DSS host; the source comment says it should move to `/channels` when the comms teardown finishes.
### Messages, replies, reactions, attachments
| Operation | Method and path | Client method |
|---|---|---|
| Send message | `POST /channels/{channel_id}/message` | `postMessage` |
| Edit message | `PATCH /channels/{channel_id}/message/{message_id}` | `patchMessage` |
| Delete message | `DELETE /channels/{channel_id}/message/{message_id}?nonce=` | `deleteMessage` |
| Add reaction | `POST /channels/{channel_id}/reaction` | `postReaction` |
| Typing update | `POST /channels/{channel_id}/typing` | `postTypingUpdate` |
| Page messages | `GET /channels/{channel_id}/messages?limit&cursor&previous_cursor&load_around_message_id` | `getChannelMessages` |
| Filter messages | `POST /channels/{channel_id}/messages?limit=` | `postChannelMessages` |
| Thread replies | `GET /channels/{channel_id}/messages/{message_id}/replies` | `getThreadReplies` |
| Message context | `GET /channels/{channel_id}/messages/{message_id}/context?before&after` | `getMessageWithContext` |
| Resolve message | `GET /channels/{channel_id}/messages/{message_id}/resolve` | `resolveChannelMessage` |
| Attachments page | `GET /channels/{channel_id}/attachments?limit&cursor&attachment_type` | `getChannelAttachments` |
| Participants | `GET/POST/DELETE /channels/{channel_id}/participants` | `getChannelParticipants`, `addParticipantsToChannel`, `removeParticipantsFromChannel` |
| Entity mentions | `POST /channels/mentions`, `DELETE /channels/mentions/{mention_id}` | `createEntityMention`, `deleteEntityMention` |
| Attachment references | `GET /channels/attachments/{entity_type}/{entity_id}/references` | `attachmentReferences` |
| Channel activity | `GET/POST /channels/activity` | `getActivity`, `postActivity` |
Message posting deduplicates `mentions` before sending. Several channel mutation methods accept a `nonce` and forward it in the JSON body or query string.
## Calls
The call client is a separate wrapper over DSS `/call` endpoints.
| Operation | Method and path | Client method |
|---|---|---|
| Get or create call token | `GET /call/{channel_id}` | `getOrCreateCall` |
| Leave or end call | `DELETE /call/{channel_id}` | `leaveCall` |
| Check active call | `GET /call/{channel_id}/active` | `checkActiveCall` |
| Get call record | `GET /call/record/{call_id}` | `getCallRecord` |
| Edit call record | `PATCH /call/record/{call_id}` | `editCallRecord` |
| Delete call record | `DELETE /call/record/{call_id}` | `deleteCallRecord` |
| Toggle team sharing | `POST /call/record/{call_id}/share-with-team/toggle` | `toggleShareWithTeam` |
| Batch previews | `POST /call/record/preview` | `storageServiceClient.getBatchCallPreviews` |
| Edit transcript | `PATCH /call/record/{call_id}/transcript` | no storage wrapper shown |
| Webhook ingest | `POST /call/webhook` | server route |
| Internal transcript ingest | `POST /call/{channel_id}/transcript` | internal agent route |
`checkActiveCall` treats an empty 204-style response as `null`. `toggleShareWithTeam` returns a primitive JSON boolean at runtime, so the client casts through an object type only to satisfy `fetchWithToken`’s generic constraint.
## Projects
| Operation | Method and path | Client method |
|---|---|---|
| List projects | `GET /projects` | `projects.getAll<ANSWER>---
title: "Storage and workspace APIs"
description: "Reference: Document, channel, project, call, CRM, soup, pin, history, permissions, properties, and search endpoints."
---
Macro’s workspace API surface is mounted on the document-storage-service host and consumed by TypeScript clients that wrap authenticated `fetchWithToken` calls in `neverthrow` `Result` values. Storage, properties, and search are separate client packages, but the browser-facing host is `SERVER_HOSTS['document-storage-service']`.
## Implementation surface
| Surface | Client entry point | Server mount | Spec artifact |
| --- | --- | --- | --- |
| Storage and workspace | `storageServiceClient` | `/documents`, `/channels`, `/projects`, `/items`, `/pins`, `/history`, `/crm`, `/call` | `service-storage/openapi.json` |
| Calls | `callServiceClient` | `/call` | storage OpenAPI schemas |
| Properties | `propertiesServiceClient` | `/properties` | `service-properties/openapi.json` |
| Search | `searchClient` | `/search` | `service-search/openapi.json` plus Rust router routes |
<Note>
The storage OpenAPI is generated from the Rust `utoipa` document. The `openapi.rs` binary prints `api::swagger::ApiDoc::openapi()` as JSON.
</Note>
## Transport and response conventions
- All handwritten clients use `fetchWithToken`, so callers receive `Result<T, ResultError[]>`.
- `storageServiceClient`, `propertiesServiceClient`, and `searchClient` all target the document-storage-service host.
- Most JSON bodies are serialized with `JSON.stringify`; file uploads use `FormData` or direct presigned URL `PUT`.
- Document permission token creation uses `SYNC_PERMISSION_TOKEN_DSS_HOST` instead of the normal DSS host so the token is signed by the DSS instance compatible with the sync service secret.
- Generated response wrappers commonly use `{ error, data }`; client methods often unwrap to the useful payload.
```ts
import { storageServiceClient } from '@service-storage/client';
import { propertiesServiceClient } from '@service-properties/client';
import { searchClient } from '@service-search/client';
const doc = await storageServiceClient.getDocumentMetadata({
documentId: 'doc-id',
});
const props = await propertiesServiceClient.getEntityProperties({
entity_type: 'DOCUMENT',
entity_id: 'doc-id',
query: { include_metadata: true },
});
const results = await searchClient.search({
params: { page_size: 10 },
request: {
query: 'roadmap',
match_type: 'partial',
search_on: 'name_content',
},
});
```
## Documents
| Operation | Endpoint | Client method | Notes |
| --- | --- | --- | --- |
| List recent documents | `GET /documents?limit=&offset=` | `getUserDocuments` | Returns documents, total, and next offset. |
| Create upload-backed document | `POST /documents` | `createDocument` | Returns metadata, `presignedUrl`, `contentType`, and optional `fileType`; client triggers file-limit paywall on 403. |
| Create markdown document | `POST /documents/create_markdown` | `createMarkdownDocument` | Backend initializes sync-service content. |
| Create task | `POST /documents/create_task` | `createTask` | Creates task document, properties, and initialized markdown content. |
| Get metadata | `GET /documents/{document_id}` or `/{version}` | `getDocumentMetadata` | Client retries metadata fetch up to 5 times with exponential delay. |
| Edit metadata/share | `PATCH /documents/{document_id}` | `editDocument` | Supports name/project/share-permission edits; moving requires access to target project. |
| Save PDF state | `PUT /documents/{document_id}` | `pdfSave` | Validates server modification-data schema before sending. |
| Save simple file/text | `PUT /documents/{document_id}/simple_save` | `simpleSave`, `simpleSaveText` | Uses `FormData` file upload. |
| Copy document | `POST /documents/{document_id}/copy?version_id=` | `copyDocument` | Creates a new document without re-uploading source content. |
| Soft delete | `DELETE /documents/{document_id}` | `deleteDocument` | Owner-only soft delete. |
| Permanent delete | `DELETE /documents/{document_id}/permanent` | `permanentlyDeleteDocument` | Hard-delete path. |
| Restore delete | `PUT /documents/{document_id}/revert_delete` | `revertDocumentDelete` | Reverts soft deletion. |
| Batch preview | `POST /documents/preview` | `getBatchDocumentPreviews` | Body: `{ document_ids: string[] }`. |
| Export | `GET /documents/{document_id}/export` | `exportDocument` | Returns export metadata/payload from DSS. |
| Processing result | `GET /documents/{document_id}/processing[/job_id]` | `getDocumentProcessingResult`, `getJobProcessingResult` | Client currently validates `PREPROCESS` results with `CoParseSchema`. |
| Location | `GET /documents/{document_id}/location_v3` | `getDocumentLocation` | Adds `get_converted_docx_url` and optional `document_version_id`. Cached for 14 minutes. |
| DOCX parts | `GET /documents/{id}/location?version_id=` | `getWriterPartUrls`, `getDocxFile` | Requires version ID; cached for 14 minutes. |
### Document content helpers
`getTextDocument`, `getBinaryDocument`, and `getDocxFile` compose metadata and location APIs. They return client-side errors such as `INVALID_DOCUMENT`, `INVALID_FILETYPE`, or `NOT_FOUND` when the location payload does not contain a usable presigned URL or the resource is gone.
## Channels and messages
| Operation | Endpoint | Client method |
| --- | --- | --- |
| Create channel | `POST /channels` | `createChannel` |
| List channels | `GET /comms/channels` | `getChannels` |
| Get or create DM | `POST /channels/get_or_create_dm` | `getOrCreateDirectMessage` |
| Get or create private channel | `POST /channels/get_or_create_private` | `getOrCreatePrivateChannel` |
| Rename channel | `PATCH /channels/{channel_id}` | `patchChannel` |
| Delete channel | `DELETE /channels/{channel_id}` | `deleteChannel` |
| Send message | `POST /channels/{channel_id}/message` | `postMessage` |
| Edit message | `PATCH /channels/{channel_id}/message/{message_id}` | `patchMessage` |
| Delete message | `DELETE /channels/{channel_id}/message/{message_id}?nonce=` | `deleteMessage` |
| React | `POST /channels/{channel_id}/reaction` | `postReaction` |
| Typing state | `POST /channels/{channel_id}/typing` | `postTypingUpdate` |
| Add/remove participants | `POST` / `DELETE /channels/{channel_id}/participants` | `addParticipantsToChannel`, `removeParticipantsFromChannel` |
| Join/leave | `POST /channels/{channel_id}/join`, `/leave` | `joinChannel`, `leaveChannel` |
| Messages page | `GET /channels/{channel_id}/messages` | `getChannelMessages` |
| Filtered messages | `POST /channels/{channel_id}/messages` | `postChannelMessages` |
| Thread replies | `GET /channels/{channel_id}/messages/{message_id}/replies` | `getThreadReplies` |
| Context around message | `GET /channels/{channel_id}/messages/{message_id}/context` | `getMessageWithContext` |
| Resolve message | `GET /channels/{channel_id}/messages/{message_id}/resolve` | `resolveChannelMessage` |
| Attachments page | `GET /channels/{channel_id}/attachments` | `getChannelAttachments` |
| Participants | `GET /channels/{channel_id}/participants` | `getChannelParticipants` |
| Entity mention | `POST /channels/mentions` | `createEntityMention` |
| Delete mention | `DELETE /channels/mentions/{mention_id}` | `deleteEntityMention` |
| Attachment references | `GET /channels/attachments/{entity_type}/{entity_id}/references` | `attachmentReferences` |
| Channel activity | `GET` / `POST /channels/activity` | `getActivity`, `postActivity` |
`postMessage` de-duplicates `mentions` before sending. Message mutations can pass a `nonce`; deletes serialize it as a query parameter while sends, edits, reactions, and typing include it in the JSON body.
## Projects
| Operation | Endpoint | Client method |
| --- | --- | --- |
| List projects | `GET /projects` | `projects.getAll` |
| Get project | `GET /projects/{id}` | `projects.getProject` |
| Get pending uploads | `GET /projects/pending` | `projects.getPending` |
| Create project | `POST /projects` | `projects.create` |
| Edit project | `PATCH /projects/{id}` | `projects.edit` |
| Soft delete project | `DELETE /projects/{id}` | `projects.delete` |
| Permanent delete | `DELETE /projects/{id}/permanent` | `projects.permanentlyDelete` |
| Restore | `PUT /projects/{id}/revert_delete` | `projects.revertDelete` |
| Project content | `GET /projects/{id}/content` | `projects.getContent` |
| Share permissions | `GET /projects/{id}/permissions` | `projects.getPermissions` |
| User access level | `GET /projects/{id}/access_level` | `projects.getUserAccessLevel` |
| Batch preview | `POST /projects/preview` | `projects.getPreview` |
| Zip upload request | `POST /projects/upload_extract` | `projects.createUploadZipRequest` |
## Calls
| Operation | Endpoint | Client method |
| --- | --- | --- |
| Join or create call | `GET /call/{channel_id}` | `callServiceClient.getOrCreateCall` |
| Leave or end call | `DELETE /call/{channel_id}` | `callServiceClient.leaveCall` |
| Active call check | `GET /call/{channel_id}/active` | `callServiceClient.checkActiveCall` |
| Get record | `GET /call/record/{call_id}` | `callServiceClient.getCallRecord` |
| Edit record | `PATCH /call/record/{call_id}` | `callServiceClient.editCallRecord` |
| Delete record | `DELETE /call/record/{call_id}` | `callServiceClient.deleteCallRecord` |
| Toggle team sharing | `POST /call/record/{call_id}/share-with-team/toggle` | `callServiceClient.toggleShareWithTeam` |
| Batch preview | `POST /call/record/preview` | `storageServiceClient.getBatchCallPreviews` |
| Edit transcript speakers | `PATCH /call/record/{call_id}/transcript` | OpenAPI route |
| Provider webhook | `POST /call/webhook` | Server route |
| Internal transcript ingest | `POST /call/{channel_id}/transcript` | Internal route guarded by `x-macro-internal-call` |
`checkActiveCall` maps an empty 204-style response to `null`. `toggleShareWithTeam` returns a primitive boolean at runtime even though the generic type is constrained to object-like values.
## CRM
| Operation | Endpoint | Request body | Behavior |
| --- | --- | --- | --- |
| List company contacts | `GET /crm/companies/{company_id}/contacts` | none | Returns non-hidden contacts for a team-owned company; owned companies with no visible contacts return `[]`. |
| Toggle email sync | `PUT /crm/companies/{company_id}/email-sync` | `{ email_sync: boolean }` | Setting `false` permanently removes existing CRM contacts and contact sources. Enabling on hidden companies returns 409. |
| Toggle company hidden | `PUT /crm/companies/{company_id}/hidden` | `{ hidden: boolean }` | Hiding disables `email_sync` and clears contacts/contact sources. Unhiding does not re-enable email sync. |
| Toggle contact hidden | `PUT /crm/contacts/{contact_id}/hidden` | `{ hidden: boolean }` | Display-only opt-out for contacts. |
| List comment threads | `GET /crm/comments/{entity_type}/{entity_id}` | none | `entity_type` is `crm_company` or `crm_contact`. |
| Create comment/reply | `POST /crm/comments/{entity_type}/{entity_id}` | `{ content, threadId? }` | Without `threadId`, creates a new thread; with `threadId`, appends a reply. |
| Edit comment | `PATCH /crm/comment/{comment_id}` | `{ content }` | Returns the updated comment. |
| Delete comment | `DELETE /crm/comment/{comment_id}` | none | Soft-deletes comment; also soft-deletes thread if it was the last live comment. |
## Soup workspace feed
Soup endpoints return workspace items with `next_cursor` and a `frecency_score`. The router supports regular filters, AST filters, and grouped AST queries.
| Operation | Endpoint | Client method | Key params |
| --- | --- | --- | --- |
| Default feed | `GET /items/soup` | not wrapped directly | `expand`, `limit`, `sort_method`, `cursor` |
| Filtered feed | `POST /items/soup?cursor=` | `getSoupItems` | Typed `EntityFilters`, `email_view`, `expand`, `limit`, `sort_method` |
| AST feed | `POST /items/soup/ast?cursor=` | `getSoupAstItems` | AST filters, `email_view`, `expand`, `limit`, `sort_method` |
| Grouped AST feed | `POST /items/soup/ast/grouped?cursor=` | `getGroupedSoupAstItems` | `group_by`, optional `group_key`, optional grouped sort |
Supported feed sort values are `viewed_at`, `created_at`, `updated_at`, `viewed_updated`, and `frecency`. Grouped queries use simple sorts only and default to `viewed_updated`.
Grouped responses include:
<ResponseField name="items" type="SoupApiItem[]">Flat item page ordered by group and sort.</ResponseField>
<ResponseField name="next_cursor" type="string | null">Global cursor for the next page.</ResponseField>
<ResponseField name="groups" type="ApiGroupMeta[]">Group metadata with `key`, `label`, `display_order`, `total_count`, `page_count`, `start_index`, and per-group `next_cursor`.</ResponseField>
<Warning>
CRM-scoped soup filters require team membership. Hidden CRM company filters require admin or owner team role.
</Warning>
## Pins and history
| Operation | Endpoint | Client method | Defaults |
| --- | --- | --- | --- |
| List pins | `GET /pins?limit=&offset=` | `getPins` | Client defaults to `limit=10`, `offset=0`. |
| Add pin | `POST /pins/{pinned_item_id}` | `pinItem` | Body is `AddPinRequest` without the path `id`. |
| Remove pin | `DELETE /pins/{pinned_item_id}` | `removePin` | Body is `PinRequest` without the path `id`. |
| Reorder pins | `PATCH /pins` | `reorderPins` | Body is an array of `ReorderPinRequest`. |
| List history | `GET /history` | `getUsersHistory` | Returns `{ data: Item[] }`. |
| Upsert history item | `POST /history/{item_type}/{item_id}` | `upsertItemToUserHistory` | Performs tracking side effects. |
| Remove history item | `DELETE /history/{item_type}/{item_id}` | `removeItemFromUserHistory` | Returns success response. |
Recognized client item types include `document`, `chat`, `project`, `channel`, `email`, `channel_message`, `call`, `automation`, and `thread`. `blockNameToItemType` maps unknown block names to `document`.
## Permissions and views
| Operation | Endpoint | Client method |
| --- | --- | --- |
| Entity permission | `GET /entity/{entity_type}/{entity_id}/permissions` | Use raw DSS fetch or generated schema |
| Document share permissions | `GET /documents/{document_id}/permissions` | `getDocumentPermissions` |
| Document permission token | `POST /documents/permissions_token/{document_id}` | `permissionsTokens.createPermissionToken` |
| Validate permission token | `POST /documents/permissions_token/validate` | `permissionsTokens.validatePermissionToken` |
| Project permissions | `GET /projects/{id}/permissions` | `projects.getPermissions` |
| Project access level | `GET /projects/{id}/access_level` | `projects.getUserAccessLevel` |
| Document viewers | `GET /documents/{document_id}/views` | `getDocumentViewers` |
| Save view location | `POST /user_document_view_location/{document_id}` | `upsertDocumentViewLocation` |
| Delete view location | `DELETE /user_document_view_location/{document_id}` | `deleteDocumentViewLocation` |
<Note>
The generated OpenAPI also advertises a v2 document permissions path for the response schema. The TypeScript storage client calls the runtime route at `/documents/{document_id}/permissions`.
</Note>
## Properties
`propertiesServiceClient` is a focused client for dynamic properties attached to entities. Entity types are uppercase wire values: `CHANNEL`, `CHAT`, `COMPANY`, `DOCUMENT`, `PROJECT`, `TASK`, `THREAD`, and `USER`.
| Operation | Endpoint | Client method |
| --- | --- | --- |
| List definitions | `GET /properties/definitions` | `listProperties` |
| Create definition | `POST /properties/definitions` | `createPropertyDefinition` |
| Delete definition | `DELETE /properties/definitions/{definition_id}` | `deletePropertyDefinition` |
| List options | `GET /properties/definitions/{definition_id}/options` | `getPropertyOptions` |
| Add option | `POST /properties/definitions/{definition_id}/options` | `addPropertyOption` |
| Delete option | `DELETE /properties/definitions/{definition_id}/options/{option_id}` | `deletePropertyOption` |
| Get entity properties | `GET /properties/entities/{entity_type}/{entity_id}` | `getEntityProperties` |
| Set entity property | `PUT /properties/entities/{entity_type}/{entity_id}/{property_id}` | `setEntityProperty` |
| Delete entity property | `DELETE /properties/entity_properties/{entity_property_id}` | `deleteEntityProperty` |
| Bulk get properties | `POST /properties/entities/bulk` | `getBulkEntityProperties` |
| Mark status complete | `PATCH /properties/entities/{entity_type}/{entity_id}/status/complete` | `setPropertyStatusComplete` |
### Definition filters
<ParamField body="scope" type="'user' | 'org' | 'system' | 'all'" required>
Scope filter for property definitions.
</ParamField>
<ParamField body="include_options" type="boolean">
When set, definitions include select options.
</ParamField>
<ParamField body="for_entity_type" type="EntityType">
Filters out definitions that cannot attach to the requested entity type.
</ParamField>
### Property values
Stored property values use capitalized response variants such as `Boolean`, `Number`, `String`, `Date`, `SelectOption`, `EntityReference`, and `Link`. Set requests use lower-case action variants such as `boolean`, `number`, `string`, `date`, `select_option`, `multi_select_option`, `entity_reference`, `multi_entity_reference`, `link`, and `multi_link`.
<RequestExample>
```json title="PUT /properties/entities/TASK/task-id/property-id"
{
"value": {
"type": "select_option",
"option_id": "00000000-0000-0000-0000-000000000000"
}
}
```
</RequestExample>
Bulk property requests accept `{ entities, property_ids? }`. The public endpoint returns only entities the user can view; inaccessible entities are omitted.
## Search
`searchClient` wraps unified and channel-specific search. Both endpoints accept `page_size` and optional base64 cursor query params. Server-side validation rejects page sizes outside `0..=100` and queries shorter than 3 characters.
| Operation | Endpoint | Client method | Response |
| --- | --- | --- | --- |
| Unified search | `POST /search` | `search` | `{ results, next_cursor }` |
| Simple unified search | `POST /search/simple` | generated `simpleUnifiedSearch` | Flat, non-grouped response |
| Channel content search | `POST /search/channel` | `searchChannel` | `{ results, next_cursor, total_count }` |
Unified request shape:
<ParamField body="query" type="string" required>
Search string; must be at least 3 characters after trimming.
</ParamField>
<ParamField body="match_type" type="'exact' | 'partial' | 'regexp' | 'query'" required>
Match mode passed to the search service.
</ParamField>
<ParamField body="search_on" type="'name' | 'content' | 'name_content'">
Defaults to content when omitted.
</ParamField>
<ParamField body="filters" type="EntityFilters">
Filter bundle for calls, channels, chats, CRM companies, documents, email, foreign entities, projects, and property filters.
</ParamField>
Channel search accepts `query` or `terms`, `match_type`, optional `sort`, and channel filters. The server requires at least one `channel_id` for `/search/channel`, even though the generated filter model describes empty arrays as broadly searchable.
<RequestExample>
```json title="POST /search/channel?page_size=20"
{
"query": "launch notes",
"match_type": "partial",
"sort": "message",
"channel_ids": ["00000000-0000-0000-0000-000000000000"]
}
```
</RequestExample>
## Operational checks for API changes
- Update Rust `utoipa` route annotations and rerun OpenAPI generation when adding or changing server routes.
- Regenerate TypeScript schemas before relying on new request or response types in clients.
- Verify handwritten clients when specs lag server routes; current examples include grouped soup and channel search wrappers.
- Keep client return mapping explicit when endpoints return primitives, 204 bodies, or wrapped `{ data }` payloads.
---
## 18. Auth and team APIs
> Reference: Login, OAuth callbacks, JWT refresh, account links, team membership, invites, permissions, billing, and user endpoints.
- Page Markdown: https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/18-auth-and-team-apis.md
- Generated: 2026-06-01T00:59:40.293Z
### Source Files
- `js/app/packages/service-clients/service-auth/openapi.json`
- `js/app/packages/service-clients/service-auth/client.ts`
- `rust/cloud-storage/authentication_service/src/openapi.rs`
- `rust/cloud-storage/authentication_service/src/config.rs`
- `js/app/packages/core/auth/sso.ts`
- `js/app/packages/core/auth/logout.ts`
---
title: "Auth and team APIs"
description: "Reference: Login, OAuth callbacks, JWT refresh, account links, team membership, invites, permissions, billing, and user endpoints."
---
The auth API is an Axum service exposed through `SERVER_HOSTS['auth-service']`; the frontend wraps it with `authServiceClient`, uses cookie credentials by default, and can switch to bearer-token requests with `ENABLE_BEARER_TOKEN_AUTH`.
## Runtime surface
| Surface | Runtime behavior |
| --- | --- |
| Base URL | `SERVER_HOSTS['auth-service']` (`http://localhost:8080` when `VITE_LOCAL_SERVERS` selects local auth) |
| Frontend client | `js/app/packages/service-clients/service-auth/client.ts` |
| OpenAPI | Generated by `rust/cloud-storage/authentication_service/src/openapi.rs` from `swagger::ApiDoc` |
| Swagger UI | Mounted at `/docs`, spec at `/api-doc/openapi.json` |
| Auth transport | HttpOnly cookies, or `Authorization: Bearer ...` for selected token flows |
| Standard JSON error | `{ "message": string }` |
| Client result shape | `neverthrow` `Result<T, ResultError[]>` from `safeFetch` / `fetchWithAuth` |
<Note>
Some OpenAPI query parameters are marked `required: true` even when server code treats them as optional. Use the Rust handler behavior and the hand-written frontend client as the integration source of truth.
</Note>
## Login and session lifecycle
### Password login
:::endpoint POST /login/password Password login
Accepts:
```json
{
"email": "user@example.com",
"password": "password"
}
```
Returns `UserTokensResponse`:
```json
{
"access_token": "jwt",
"refresh_token": "refresh"
}
```
The server lowercases the email, validates email format, authenticates through FusionAuth, sets access and refresh cookies, and returns the same tokens in JSON. If FusionAuth reports an unregistered user, the handler registers the user from email and retries login. Unverified users receive `401`.
:::
### Passwordless login
:::endpoint POST /login/passwordless Start passwordless login
Accepts:
```json
{
"email": "user@example.com",
"redirect_uri": "https://macro.com/app/login",
"referral_code": "optional"
}
```
Responses:
| Status | Meaning |
| --- | --- |
| `200` | Login code was generated, cached against the email, and sent. |
| `202` | The email belongs to an identity provider; response body is `{ "idp_id": string }`. |
| `400` | Invalid email or invalid request. |
| `403` | Blocked email. |
| `500` | FusionAuth, cache, or send failure. |
The callback endpoint is `GET /oauth/passwordless/{code}?email=...`. The frontend uses `disable_redirect=true` when it wants JSON tokens instead of a redirect.
:::
### SSO login
:::endpoint GET /login/sso Start SSO
Supported query parameters:
| Parameter | Required by server | Notes |
| --- | ---: | --- |
| `idp_name` | One of `idp_name` or `idp_id` | Identity provider display name, for example Google/Gmail. |
| `idp_id` | One of `idp_name` or `idp_id` | Direct FusionAuth identity provider ID. |
| `login_hint` | No | Pre-fills provider login when supported. |
| `original_url` | No | Web redirect target after successful login. |
| `is_mobile` | No | Causes callback to issue a session code instead of only cookies. |
| `referral_code` | No | Tracked after login when present. |
The handler builds FusionAuth authorization state and returns a temporary redirect to the provider.
:::
```mermaid
sequenceDiagram
participant Web as Web app
participant Auth as authentication_service
participant IdP as OAuth provider
participant Cache as Redis/session cache
Web->>Auth: GET /login/sso?idp_name=...&original_url=...
Auth-->>Web: Redirect to provider authorization URL
Web->>IdP: Provider login
IdP-->>Auth: GET /oauth/redirect?code=...&state=...
Auth->>Auth: Complete authorization code grant
alt mobile state
Auth->>Cache: Store refresh token by generated session code
Auth-->>Web: Redirect original_url?token=session_code
Web->>Auth: GET /session/login/{session_code}
Auth-->>Web: UserTokensResponse + cookies
else web state
Auth-->>Web: Set cookies + HTML redirect to original_url
end
```
### Native mobile SSO
`startSsoLogin()` uses the Tauri `plugin:auth|authenticate` command on iOS with:
```ts
{
authUrl,
callbackScheme: "macro",
ephemeralSession: true
}
```
The plugin returns a `token` session code. The client calls `authServiceClient.sessionLogin({ session_code })`, stores returned tokens in `macroAccessToken`, clears any pending token refresh promise, and invalidates auth queries after success.
## Token refresh and API tokens
:::endpoint POST /jwt/refresh Refresh access and refresh tokens
Tokens can be supplied by cookies or by headers:
```http
Authorization: Bearer <access_token>
x-macro-refresh-token: <refresh_token>
```
Behavior:
| Condition | Result |
| --- | --- |
| Access token is still valid | Returns original access and refresh tokens. |
| Access token is expired | Calls FusionAuth refresh, sets new cookies, returns new tokens. |
| Token cannot be decoded | `401 unauthorized`. |
| Refresh token is invalid | `400 invalid refresh token`. |
The frontend coalesces concurrent refreshes: `fetchWithToken()` keeps a single `tokenPromise`, and `authServiceClient.getAccessToken()` keeps a single `ongoingRefresh`.
:::
:::endpoint GET /jwt/macro_api_token Generate Macro API token
Uses the authenticated user context and optional `email` query parameter to issue a `macro_api_token` for the selected profile:
```json
{
"macro_api_token": "jwt"
}
```
`service-auth/fetch.ts` uses this token as the bearer credential for service calls when bearer-token auth is enabled.
:::
## Logout
`authServiceClient.logout()` clears persisted access-token state and posts to `/logout`. The server clears access and refresh cookies by setting expired cookies, then attempts FusionAuth logout for the current tenant.
`useLogout()` also clears the app's `login=false` cookie, resets cached user info to an unauthenticated shape, tracks `sign_out`, resets analytics, and then:
| Platform | Redirect behavior |
| --- | --- |
| Native mobile | Fetches `SERVER_HOSTS['auth-logout']` with `no-cors`, then navigates to `/login`. |
| Web | Sets `window.location.href` to `SERVER_HOSTS['auth-logout']`. |
## Account links and OAuth callbacks
Account linking requires an authenticated caller.
| Endpoint | Method | Client method | Purpose |
| --- | --- | --- | --- |
| `/link` | `POST` | none | Creates an in-progress link row and returns `{ link_id }`. Limited to 5 in-progress links per FusionAuth user. |
| `/link/github` | `POST` | `initGithubLink(originalUrl?)` | Creates a GitHub in-progress link and returns `{ authorization_url, link_id }`. |
| `/link/github` | `DELETE` | `deleteGithubLink()` | Deletes the user's GitHub link; it does not uninstall the GitHub app from team repositories. |
| `/link/gmail` | `POST` | `initGmailLink(originalUrl?)` | Creates a Gmail in-progress link and returns Google OAuth URL plus `link_id`. |
| `/user/link_exists` | `GET` | `checkLinkExists({ idp_name?, idp_id? })` | Returns `{ link_exists: boolean }`. One of `idp_name` or `idp_id` is expected. |
| `/oauth2/{provider}/callback` | `GET` | provider redirect | Completes custom Google/GitHub login or linking. |
Gmail linking requests Google scopes for OpenID profile/email, Gmail modify, contacts read-only, other contacts read-only, and Gmail settings basic. After Google consent, the callback redirects back to `original_url` with `link_id` appended so the frontend can finish inbox provisioning.
GitHub linking either redirects to `original_url` or returns a small HTML page that posts `{ type: "github-linked", success: true }` to `window.opener` and closes the popup.
<Warning>
The OpenAPI spec includes account merge paths (`/merge`, `/merge/verify/{code}`), but the auth service router shown in `api_router` does not mount `merge::router()`. Treat those paths as not available at runtime unless the router is updated.
</Warning>
## User endpoints
| Endpoint | Method | Client method | Response |
| --- | --- | --- | --- |
| `/user/me` | `GET` | `getUserInfo()` | `{ user_id, organization_id?, permissions }`, mapped to `{ authenticated, permissions, userId, organizationId }`. |
| `/user/me` | `DELETE` | `deleteUser()` | `{ success: boolean }`; client clears persisted token state first. |
| `/user/legacy_user_permissions` | `GET` | `getLegacyUserPermissions()` | Legacy UI aggregate: permissions, email, name, license status, tutorial flag, group, Chrome extension state, trial state, AI consent, referral code. |
| `/user/name` | `GET` | `getUserName()` | `{ id, first_name?, last_name? }`. |
| `/user/name` | `PUT` | `putUserName({ first_name?, last_name? })` | Empty response. |
| `/user/get_names` | `POST` | `getUserNames()` | `{ names: UserName[] }`. |
| `/user/get_names_with_email` | `POST` | `getUserNamesWithEmail()` | `{ names: UserName[] }`, with email-contact fallback. |
| `/user/profile_picture` | `PUT` | `putProfilePicture({ url })` | Empty response. |
| `/user/profile_pictures` | `POST` | `postProfilePictures({ user_id_list })` | `{ pictures: UserProfilePicture[] }`. |
| `/user/organization` | `GET` | `getOrganization()` | `{ organizationId, organizationName }`, or `204`; frontend maps errors/no-content to undefined organization fields. |
| `/user/quota` | `GET` | `userQuota()` | `{ documents, ai_chat_messages, max_documents, max_ai_chat_messages }`, or `204` for premium users with no quota. |
| `/user/tutorial` | `PATCH` | `patchUserTutorial({ tutorialComplete })` | Empty response. |
| `/user/onboarding` | `PATCH` | `completeOnboarding({ firstName, lastName, industry, title })` | Empty response. |
| `/user/group` | `PATCH` | `setGroup({ group })` | Empty response. |
| `/user/ai_consent` | `PATCH` | `patchAiConsent({ aiDataConsent })` | Empty response. |
## Permissions
| Endpoint | Auth | Response |
| --- | --- | --- |
| `GET /permissions` | Not wrapped in JWT middleware in the router | `Permission[]` with `{ id, description }`. |
| `GET /permissions/me` | JWT required | `string[]` permission IDs for the current user. |
| `GET /user/me` | JWT required | Includes current user permissions alongside user and organization IDs. |
## Teams, invites, and roles
All `/team` routes are protected by JWT middleware. Team access checks are enforced with typed extractors for member, admin, and owner roles.
| Role | Used for |
| --- | --- |
| `member` | Read team details. |
| `admin` | Patch team metadata/roles, list team invites, toggle CRM. |
| `owner` | Invite users, delete invites, remove users, delete team. |
### Team models
```ts
type TeamRole = "member" | "admin" | "owner";
interface Team {
id: string;
name: string;
owner_id: string;
slug: string;
}
interface TeamWithMembers {
team: Team;
members: {
team_id: string;
user_id: string;
role: TeamRole;
}[];
}
```
### Team endpoints
| Endpoint | Method | Client method | Access |
| --- | --- | --- | --- |
| `/team/user` | `GET` | `getUserTeams()` | Authenticated user. |
| `/team/user/invites` | `GET` | `getUserInvites()` | Authenticated user. |
| `/team` | `GET` | `getTeam()` | Team member. |
| `/team` | `POST` | `createTeam({ name })` | Authenticated user. |
| `/team` | `PATCH` | `patchTeam({ name?, slug?, user_role_updates? })` | Admin or owner. |
| `/team` | `DELETE` | `deleteTeam()` | Owner. Cancels team subscription and is irreversible. |
| `/team/invites` | `GET` | `getTeamInvites()` | Admin or owner. |
| `/team/invite` | `POST` | `inviteToTeam({ invites })` | Owner. Returns `201` when invites are created, `304` if nothing new was sent. |
| `/team/invite/{team_invite_id}` | `DELETE` | `deleteTeamInvite(id)` | Owner. |
| `/team/join/{team_invite_id}` | `GET` | `joinTeam(id)` | Invited authenticated user. |
| `/team/join/{team_invite_id}` | `DELETE` | `rejectInvitation(id)` | Invited authenticated user. |
| `/team/remove/{remove_user_id}` | `DELETE` | `removeUserFromTeam(userId)` | Owner. |
| `/team/crm` | `PATCH` | not wrapped in `authServiceClient` | Admin or owner. |
`InviteToTeamRequest` accepts:
```json
{
"invites": [{ "email": "person@example.com" }]
}
```
Emails are parsed and lowercased. Invalid emails, empty invite lists, and too many emails return `400`.
### Team CRM toggle
:::endpoint PATCH /team/crm Enable or disable team CRM
Request:
```json
{
"enabled": true
}
```
Response:
```json
{
"enabled": true,
"changed": true,
"backfill_enqueued": 4,
"backfill_failed": 0
}
```
Enabling CRM enqueues a best-effort `PopulateCrmForUser` message per team member. Disabling CRM purges the team's CRM data through the CRM table cascade.
:::
## Billing and subscription endpoints
Billing endpoints live under `/user/stripe/*` and require an authenticated user.
| Endpoint | Method | Client method | Notes |
| --- | --- | --- | --- |
| `/user/stripe/checkout` | `POST` | `createCheckoutSession()` | Legacy checkout. Accepts optional `tier`; defaults to `haiku`. |
| `/user/stripe/checkoutv2` | `POST` | `createCheckoutSessionV2()` | Current checkout. Backend uses configured price ID and can include team metadata when the caller owns a team. |
| `/user/stripe/portal` | `POST` | `createPortalSession({ returnUrl })` | Returns Stripe billing portal URL. |
| `/user/stripe/subscription` | `PATCH` | `patchSubscriptionTier({ newTier })` | Swaps both RBAC subscription role and Stripe subscription line item. |
### Checkout request shape
```json
{
"successUrl": "https://macro.com/app/?subscriptionSuccess=true",
"cancelUrl": "https://macro.com/app?subscriptionCancel=true",
"discount": "PROMO",
"metadata": {
"gaClientId": "optional",
"fbp": "optional",
"fbc": "optional"
}
}
```
The response is:
```json
{
"url": "https://checkout.stripe.com/..."
}
```
### Subscription tier changes
Supported tiers are:
```ts
type StripeProductTier = "haiku" | "sonnet" | "opus";
```
`PATCH /user/stripe/subscription` rejects:
| Status | Frontend semantic code | Meaning |
| --- | --- | --- |
| `400` | `TIER_UNCHANGED` | The user is already on the requested tier. |
| `403` | `USER_IN_TEAM` | Team members cannot manage their own tier. |
| `404` | `NO_SUBSCRIPTION` | No active subscription or subscription role exists. |
| `409` | `UPDATE_IN_PROGRESS` | Another tier update holds the per-user subscription lock. |
| `500` | `SERVER_ERROR` | Stripe, RBAC, DB, or inconsistent subscription state failure. |
The backend performs the RBAC role swap before the Stripe line-item update and rolls the role swap back if Stripe fails. Stripe requests use a stable idempotency key for the logical tier-swap operation.
## Configuration
The auth service loads configuration from environment variables in `Config::from_env()`.
| Key | Required | Purpose |
| --- | ---: | --- |
| `BASE_URL` | Yes | Public auth service base URL; used for OAuth2 callback URIs. |
| `DATABASE_URL` | Yes | Postgres connection. |
| `REDIS_URI` | Yes | Redis/cache connection for login codes, session codes, and rate limits. |
| `FUSIONAUTH_TENANT_ID` | Yes | FusionAuth tenant. |
| `FUSIONAUTH_API_KEY_SECRET_KEY` | Yes | Secret name, or local raw value in local environment. |
| `FUSIONAUTH_CLIENT_ID` | Yes | FusionAuth application client ID. |
| `FUSIONAUTH_CLIENT_SECRET_KEY` | Yes | Secret name, or local raw value in local environment. |
| `FUSIONAUTH_BASE_URL` | Yes | FusionAuth base URL. |
| `FUSIONAUTH_OAUTH_REDIRECT_URI` | Yes | FusionAuth OAuth redirect URI. |
| `GOOGLE_CLIENT_ID` | Yes | Google OAuth client ID. |
| `GOOGLE_CLIENT_SECRET_KEY` | Yes | Secret name, or local raw value in local environment. |
| `STRIPE_SECRET_KEY` | Yes | Secret name, or local raw value in local environment. |
| `SERVICE_INTERNAL_AUTH_KEY` | Yes | Internal service auth key. |
| `DOCUMENT_STORAGE_SERVICE_URL` | Yes | Document storage service URL. |
| `NOTIFICATION_QUEUE` | Yes | Notification SQS queue. |
| `SEARCH_EVENT_QUEUE` | Yes | Search-event SQS queue. |
| `LINK_MANAGER_QUEUE` | Yes | Email link manager queue. |
| `EMAIL_BACKFILL_QUEUE` | Yes | Team join / CRM backfill queue. |
| `GITHUB_CLIENT_ID` | Yes | GitHub OAuth client ID. |
| `GITHUB_CLIENT_SECRET` | Yes | GitHub OAuth secret. |
| `GITHUB_IDP_ID` | Yes | GitHub identity provider ID. |
| `PORT` | No | Defaults to `8080`. |
| `GA_MEASUREMENT_ID`, `GA_API_SECRET` | No | Google Analytics conversion tracking. |
| `META_PIXEL_ID`, `META_ACCESS_TOKEN`, `META_TEST_EVENT_CODE` | No | Meta conversion tracking. |
| `POSTHOG_API_KEY`, `POSTHOG_HOST` | No | PostHog analytics. |
Cookie names and attributes depend on environment:
| Environment | Cookie prefix | Domain | SameSite |
| --- | --- | --- | --- |
| `Production` | none | `macro.com` | `Strict` |
| `Develop` | `dev-` | `macro.com` | `None` |
| `Local` | `local-` | none | `None` |
Access and refresh cookies are `Secure`, `HttpOnly`, path `/`, and expire in 365 days.
## Client integration notes
- `authApiFetch()` always sends `credentials: "include"`.
- `fetchWithToken()` retries the original request after a `401` by calling `POST /jwt/refresh`.
- `service-auth/fetch.ts` obtains a Macro API token via `GET /jwt/macro_api_token` and sends it as `Authorization: Bearer ...`.
- `authServiceClient.sessionLogin()` and `passwordlessCallback()` persist `{ accessToken, refreshToken, expiresAt }` under `macroAccessToken`.
- User info queries use `getLegacyUserPermissions()` for the main UI cache and mark data stale after 15 seconds.
- Team mutations invalidate the relevant Solid Query team caches after create, patch, delete, invite, join, or reject operations.
## Related pages
<CardGroup>
<Card title="Service clients" href="/service-clients">
Frontend client wrappers, `safeFetch`, generated schemas, and mock client registration.
</Card>
<Card title="Teams and billing operations" href="/teams-billing">
Team ownership, Stripe subscriptions, CRM toggles, and operational recovery paths.
</Card>
</CardGroup>
---
## 19. Email and notification APIs
> Reference: Gmail init and sync, drafts, threads, labels, attachments, notification preferences, unread state, and unsubscribe routes.
- Page Markdown: https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/19-email-and-notification-apis.md
- Generated: 2026-06-01T00:58:55.627Z
### Source Files
- `js/app/packages/service-clients/service-email/openapi.json`
- `js/app/packages/service-clients/service-email/client.ts`
- `rust/cloud-storage/email_service/src/openapi.rs`
- `js/app/packages/service-clients/service-notification/openapi.json`
- `js/app/packages/service-clients/service-notification/client.ts`
- `rust/cloud-storage/notification_service/src/openapi.rs`
---
title: "Email and notification APIs"
description: "Reference: Gmail init and sync, drafts, threads, labels, attachments, notification preferences, unread state, and unsubscribe routes."
---
Macro exposes email and notification HTTP APIs through Rust `utoipa` OpenAPI specs, generated TypeScript schemas, and handwritten frontend clients: `emailClient` uses `SERVER_HOSTS['email-service']`, and `notificationServiceClient` uses `SERVER_HOSTS['notification-service']`.
## Runtime surface
| Area | Frontend client | OpenAPI artifact | Rust OpenAPI binary |
| --- | --- | --- | --- |
| Email | `js/app/packages/service-clients/service-email/client.ts` | `js/app/packages/service-clients/service-email/openapi.json` | `email_service_openapi` |
| Notifications | `js/app/packages/service-clients/service-notification/client.ts` | `js/app/packages/service-clients/service-notification/openapi.json` | `notification_service_openapi` |
Both clients call `fetchWithToken`, so callers receive `neverthrow` `Result` values and get token refresh behavior on unauthorized cookie-based requests. In development, `VITE_LOCAL_SERVERS=ALL` maps email to `http://localhost:8087` and notifications to `http://localhost:8089`; otherwise the remote service hosts are used.
<Note>
Generated schemas are refreshed from local Rust OpenAPI binaries with `cd js/app && bun run gen-api email-service` or `bun run gen-api notification-service`.
</Note>
## Email API
### Gmail initialization and backfill
`POST /email/init` initializes Gmail email for the authenticated user. It registers a Gmail watch, upserts an `email_links` row, stores the Gmail history id, creates a backfill job, and enqueues the initial backfill operation.
<ParamField body="link_id" type="uuid" optional>
Optional query parameter from a `/link/gmail` flow. When supplied, init uses the linked email recorded on the in-progress link instead of the JWT email.
</ParamField>
<ResponseField name="link_id" type="uuid">
The accessible email link id.
</ResponseField>
<ResponseField name="backfill_job_id" type="uuid | undefined">
Present when init enqueues a new backfill job. It can be absent when init delegates to an existing child inbox.
</ResponseField>
Common backfill routes:
| Method | Path | Purpose |
| --- | --- | --- |
| `GET` | `/email/backfill/gmail/active` | Return the active backfill job or `204` when none exists. |
| `GET` | `/email/backfill/gmail/{id}` | Return a backfill job by id for the current email link. |
| `DELETE` | `/email/backfill/gmail` | Cancel a job by JSON body `{ "job_id": "..." }`. |
`/email/init` limits non-`@macro.com` users to three recent jobs in the last 24 hours and rejects users already initialized for the same Gmail link.
### Sync lifecycle
| Client method | Route | Status |
| --- | --- | --- |
| `emailClient.init({ linkId })` | `POST /email/init?link_id=...` | Supported. |
| `emailClient.stopSync()` | `DELETE /email/sync` | Supported; enqueues asynchronous link deletion with reason `ManuallyDisabled`. |
| `emailClient.startSync()` | `POST /email/sync` | Client method exists, but the Rust router and OpenAPI currently expose only `DELETE /email/sync`. Use init to enable sync unless the server route is added. |
## Threads and unread state
Thread listing uses cursor pagination:
```ts
await emailClient.getPreviews({
view: 'inbox',
limit: 20,
sort_method: 'viewed_updated',
cursor,
});
```
| Route | Inputs | Response |
| --- | --- | --- |
| `GET /email/threads/previews/cursor/{view}` | `view`, optional `limit`, `sort_method`, `cursor` | `{ items, next_cursor? }` |
| `GET /email/threads/{thread_id}` | optional `offset` default `0`, `limit` default `5`, max `100` | `{ thread }` |
| `GET /email/threads/{id}/messages` | optional `since`, `limit` | `ParsedMessage[]` |
| `POST /email/threads/{id}/seen` | thread id | `EmptyResponse` |
| `PATCH /email/threads/{id}/archived` | `{ value: boolean }` | `EmptyResponse` |
| `PATCH /email/threads/{id}/labels` | `{ label_id, value }` | updated thread labels |
| `PATCH /email/threads/{thread_id}/project` | `{ projectId: string | null }` | previous project id |
Email unread state is persisted on the thread and message records and mirrored to Gmail labels. `POST /email/threads/{id}/seen` upserts user history, marks the thread and unread messages as read, removes the `UNREAD` label locally, then enqueues Gmail operations to remove the provider `UNREAD` label.
Supported preview `view` values include `inbox`, `sent`, `drafts`, `starred`, `all`, `important`, `other`, and `user:<label>`.
## Drafts, sending, and scheduling
Draft creation and sending use `ApiDraftInput`.
```ts
await emailClient.createDraft({
draft: {
subject: 'Review',
to: [{ email_address: 'person@example.com' }],
body_text: 'Please review this.',
},
});
await emailClient.sendMessage({
message: {
subject: 'Review',
to: [{ email_address: 'person@example.com' }],
body_text: 'Please review this.',
},
});
```
| Operation | Route | Body |
| --- | --- | --- |
| Create draft | `POST /email/drafts` | `{ draft: ApiDraftInput, send_time? }` |
| Delete draft | `DELETE /email/drafts/{id}` | none |
| Send message | `POST /email/messages` | `{ message: ApiDraftInput }` |
| Schedule draft | `PUT /email/drafts/scheduled/{id}` | scheduled send request |
| Unschedule draft | `DELETE /email/drafts/scheduled/{message_id}` | none |
| List scheduled drafts | `GET /email/drafts/scheduled?offset=&limit=` | pagination params |
`ApiDraftInput` requires `subject` and supports `to`, `cc`, `bcc`, `body_html`, `body_macro`, `body_text`, `provider_id`, `provider_thread_id`, `thread_db_id`, `replying_to_id`, `headers_json`, and `db_id`.
## Attachments
| Route | Purpose |
| --- | --- |
| `GET /email/attachments/{id}` | Return an attachment wrapper. |
| `GET /email/attachments/{id}/document_id` | Return or create the Macro document id for an email attachment. |
| `POST /email/drafts/{id}/attachments` | Create a draft attachment record and return a presigned upload URL. |
| `DELETE /email/drafts/{id}/attachments/{attachment_id}` | Remove a draft attachment. |
| `POST /email/drafts/{id}/forwarded-attachments` | Attach an existing email attachment to a draft. |
| `DELETE /email/drafts/{id}/forwarded-attachments/{attachment_id}` | Remove a forwarded attachment. |
`AddDraftAttachmentRequest` requires:
<ParamField body="file_name" type="string" required />
<ParamField body="sha" type="string" required>
SHA-256 hex string. The server validates exactly 64 ASCII hex characters.
</ParamField>
<ParamField body="size" type="number" required>
Raw byte size. The server rejects non-positive sizes and enforces an 18 MB safe raw attachment limit.
</ParamField>
The response contains `attachment_id`, `upload_url`, and `content_type`.
## Labels, filters, contacts, and settings
| Feature | Routes |
| --- | --- |
| Labels | `GET /email/labels`, `POST /email/labels`, `DELETE /email/labels/{id}` |
| Message labels | `PATCH /email/messages/labels` with `{ message_ids, label_id, value }` |
| Thread labels | `PATCH /email/threads/{id}/labels` with `{ label_id, value }` |
| Filters | `GET /email/filters`, `PUT /email/filters`, `DELETE /email/filters/{id}` |
| Contacts | `GET /email/contacts`, `POST /email/contacts/block`, `POST /email/contacts/unblock` |
| Links | `GET /email/links` |
| Settings | `PATCH /email/settings` with `{ settings }` |
The generated `Settings` schema currently exposes `signature_on_replies_forwards`.
## Notification API
### User notifications
The notification service mounts the internal router both under `/{version}` and unversioned paths. The OpenAPI annotations describe read routes under `/v1/user_notifications` and bulk mutation/delete routes under `/v2/user_notifications`, while the frontend client currently calls unversioned `/user_notifications` paths.
| Client method | Route used by client | Shape |
| --- | --- | --- |
| `userNotifications({ limit, cursor })` | `GET /user_notifications` | `{ items, next_cursor? }` |
| `getUserNotificationById(id)` | `GET /user_notifications/{id}` | `ApiUserNotification` |
| `bulkGetUserNotificationsByEventItemId(...)` | `POST /user_notifications/item/bulk` | body `{ eventItemIds }` |
| `markNotificationAsSeen(...)` | `PATCH /user_notifications/bulk/seen` | body `{ notificationIds }` |
| `markNotificationAsDone(...)` | `PATCH /user_notifications/bulk/done` | body `{ notificationIds }` |
| `bulkMarkNotificationAsUndone(...)` | `PATCH /user_notifications/bulk/undone` | body `{ notificationIds }` |
| `markNotificationEntityAsSeen(...)` | `PATCH /user_notifications/item/{event_item_id}/seen` | no body |
| `markNotificationEntityAsDone(...)` | `PATCH /user_notifications/item/{event_item_id}/done` | no body |
`ApiUserNotification` includes `id`, `owner_id`, `entity_id`, `entity_type`, `notification_event_type`, `notification_metadata`, `sent`, `done`, `viewed_at`, `created_at`, `updated_at`, optional `deleted_at`, and optional `sender_id`.
Notification unread state is represented by `viewed_at`; task/action completion is represented separately by `done`.
### Notification preferences
OpenAPI exposes notification-type preferences:
| Method | Path | Purpose |
| --- | --- | --- |
| `GET` | `/v1/user_notifications/preferences` | Return `{ disabled_types: string[] }`. |
| `PUT` | `/v1/user_notifications/preferences/{notification_event_type}/disable` | Disable a notification event type. |
| `PUT` | `/v1/user_notifications/preferences/{notification_event_type}/enable` | Re-enable a notification event type. |
The handwritten `notificationServiceClient` does not currently wrap these preference endpoints.
### Device registration
The notification client exposes push device registration helpers:
```ts
await notificationServiceClient.registerDevice({
deviceType: 'ios',
token: '<push-token>',
});
await notificationServiceClient.unregisterDevice({
deviceType: 'ios',
token: '<push-token>',
});
```
The local API router nests the device router at `/device`; the OpenAPI artifact includes the `DeviceRequest` schema but does not list device paths.
## Unsubscribe routes
| Route | Purpose |
| --- | --- |
| `GET /unsubscribe` | Return `UserUnsubscribe[]`, each with `item_id` and `item_type`. |
| `POST /unsubscribe/item/{item_type}/{item_id}` | Unsubscribe the current user from one notification item. |
| `DELETE /unsubscribe/item/{item_type}/{item_id}` | Remove one item-level unsubscribe. |
| `POST /unsubscribe/mute` | Mute all notifications for the current user. |
| `DELETE /unsubscribe/mute` | Unmute global notifications; manually muted items remain muted. |
| `POST /unsubscribe/email` | OpenAPI handler for email unsubscribe. |
<Warning>
The notification Rust router currently registers `/unsubscribe/email` with `GET`, while the handler annotation and generated OpenAPI declare `POST /unsubscribe/email`. The frontend client does not expose an `unsubscribeEmail` wrapper. Verify the runtime method before adding callers.
</Warning>
## Client return pattern
Both handwritten clients return `Result` objects:
```ts
const result = await emailClient.getThread({
thread_id,
offset: 0,
limit: 20,
});
if (result.isErr()) {
// ResultError<FetchWithTokenErrorCode>[]
return;
}
const { thread } = result.value;
```
Use generated schema types from `./generated/schemas` for request and response bodies, but prefer the handwritten `emailClient` and `notificationServiceClient` for application code because they apply service host selection and authenticated fetch behavior.
## Next
- Update OpenAPI/client drift before adding new sync, device, or unsubscribe-email callers.
- Regenerate schemas after Rust handler or `utoipa` annotation changes with `bun run gen-api <service-name>`.
---
## 20. AI chat streaming API
> Reference: DCS chat endpoints, stream payloads, toolset selection, model identifiers, extraction statuses, structured completion, and stop semantics.
- Page Markdown: https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/20-ai-chat-streaming-api.md
- Generated: 2026-06-01T00:59:46.491Z
### Source Files
- `js/app/packages/service-clients/service-cognition/openapi.json`
- `js/app/packages/service-clients/service-cognition/client.ts`
- `rust/cloud-storage/document_cognition_service/src/model/stream.rs`
- `rust/cloud-storage/document_cognition_service/src/api/stream/chat_message/mod.rs`
- `rust/cloud-storage/document_cognition_service/src/api/stream/stop.rs`
- `rust/cloud-storage/agent/src/model.rs`
- `js/app/packages/core/component/AI/types/index.ts`
---
title: "AI chat streaming API"
description: "Reference: DCS chat endpoints, stream payloads, toolset selection, model identifiers, extraction statuses, structured completion, and stop semantics."
---
DCS exposes chat generation as an HTTP initiation API plus durable stream delivery through `connection_gateway`; `POST /stream/chat/message` returns a `stream_id` and `chat_id`, then clients subscribe to the chat stream to receive `ChatStream` payloads until `stream_end`.
## Runtime shape
```text
UI / service-cognition client
POST /stream/chat/message
│ returns { stream_id, message_id, chat_id }
▼
DCS stream repo: id = { entity_type: "chat", entity_id: chat_id, stream_id }
│ publishes payloads
▼
connection_gateway websocket
│ message.type = "stream"
▼
@service-connection/stream.subscribe("chat", chat_id, stream_id)
```
<Note>
The chat endpoint is not an SSE response. The HTTP response only acknowledges stream creation. Chunks arrive later as `connection_gateway` websocket stream events.
</Note>
## Endpoints
:::endpoint POST /stream/chat/message Initiate a streamed chat response
Creates or reuses a chat, stores the incoming user message, starts the agent loop, and publishes stream payloads under the returned `stream_id`.
### Request body
| Field | Type | Required | Behavior |
| --- | --- | --- | --- |
| `content` | `string` | Yes | User message text. |
| `model` | `AgentModel` | Yes | Required by schema. In the current chat message handler, the effective response model is resolved from `ChatModelAccess` rather than read directly from this field. |
| `chat_id` | `string \| null` | No | Existing chat ID. If omitted, empty, or not found, DCS creates a new persistent chat named with the default chat name. |
| `attachments` | `Entity[] \| null` | No | Entities attached to the message. Resolved attachment parts from the current and prior message chain are included in the prompt. |
| `additional_instructions` | `string \| null` | No | Appended after the selected tool prompt. Frontend chat input also appends a model instruction string. |
| `toolset` | `ToolSet` | No | Defaults to `{ "type": "all" }`. See [Toolset selection](#toolset-selection). |
<RequestExample>
```http
POST /stream/chat/message
Content-Type: application/json
{
"content": "Summarize the attached document.",
"model": "smart",
"chat_id": "chat_123",
"attachments": [
{
"entity_type": "document",
"entity_id": "doc_456"
}
],
"toolset": { "type": "all" },
"additional_instructions": "Prefer concise bullets."
}
```
</RequestExample>
### Response body
| Field | Type | Behavior |
| --- | --- | --- |
| `stream_id` | `string` | Stream identifier for `connection_gateway` subscription. Generated server-side. |
| `message_id` | `string` | Assistant message ID. The implementation sets this to the same value as `stream_id`. |
| `chat_id` | `string` | Actual chat ID. May differ from the request if DCS created a new chat. |
<ResponseExample>
```json
{
"stream_id": "0de0fb91-3f54-4680-97a0-f3e97b9a2a69",
"message_id": "0de0fb91-3f54-4680-97a0-f3e97b9a2a69",
"chat_id": "chat_123"
}
```
</ResponseExample>
### Client helper
`cognitionApiServiceClient.sendStreamChatMessage(args)` posts to `/stream/chat/message`. The chat input flow then calls:
```ts
subscribe('chat', chat_id, stream_id)
```
and treats `stream_end` as the terminal marker.
:::endpoint POST /stream/chat/message/stop Stop an in-flight chat stream
Publishes a cancellation signal for an in-flight stream. The caller must have more than `View` or `Comment` access to the chat.
### Request body
| Field | Type | Required | Behavior |
| --- | --- | --- | --- |
| `chat_id` | `string` | Yes | Chat used for permission checks. |
| `stream_id` | `string` | Yes | Stream/message ID returned by `/stream/chat/message`. |
<ResponseExample>
```json
{
"stopped": true
}
```
</ResponseExample>
`stopped: false` means no matching live subscriber received the cancel signal, usually because the stream already finished or the serving DCS instance is gone.
:::endpoint POST /structured-completion Run a two-phase structured completion
Runs an agent loop to gather context, then asks the model to return JSON matching `output_schema`.
### Request body
| Field | Type | Required | Behavior |
| --- | --- | --- | --- |
| `prompt` | `string` | Yes | User prompt for the information-gathering phase. |
| `model` | `AgentModel` | Yes | Model passed to the agent loop and structured output call. |
| `output_schema` | `DynamicSchema` | Yes | `{ name, schema, description? }`; `schema` is the JSON schema to validate against. |
| `additional_instructions` | `string \| null` | No | Appended to the selected tool prompt. |
| `toolset` | `ToolSet` | No | Defaults to all-tool prompt behavior. |
<ResponseExample>
```json
{
"result": {
"summary": "The document describes renewal terms.",
"risk_level": "medium"
}
}
```
</ResponseExample>
## Stream event envelope
`connection_gateway` websocket messages use `type: "stream"` and carry a serialized stream item:
```json
{
"id": {
"entity_type": "chat",
"entity_id": "chat_123",
"stream_id": "0de0fb91-3f54-4680-97a0-f3e97b9a2a69"
},
"payload": {
"type": "chat_message_response",
"stream_id": "0de0fb91-3f54-4680-97a0-f3e97b9a2a69",
"chat_id": "chat_123",
"message_id": "0de0fb91-3f54-4680-97a0-f3e97b9a2a69",
"content": {
"type": "text",
"text": "Here is the summary..."
}
}
}
```
The frontend stream store appends payloads to the stream unless `payload.type === "stream_end"`, in which case it marks the stream done.
## ChatStream payloads
| `type` | Fields | Notes |
| --- | --- | --- |
| `chat_user_message` | `stream_id`, `chat_id`, `message_id`, `content`, `attachments` | First item emitted by the HTTP chat message handler so other clients can display the user message. |
| `chat_message_response` | `stream_id`, `message_id`, `chat_id`, `content` | Repeated assistant chunks. `content` is an `AssistantMessagePart`. |
| `stream_end` | `stream_id` | Terminal event. Stop and normal completion both end with this marker. |
| `error` | `stream_error` payload | Emitted for stream creation errors, AI stream errors, and idle timeout. |
| `chat_message_ack` | `message_id`, `chat_id` | Defined in the shared stream schema. |
| `chat_message_finished` | `message_id`, `chat_id`, `user_message_id` | Defined in the shared stream schema. |
| `chat_renamed` | `stream_id`, `chat_id`, `name` | Defined in the shared stream schema. |
| `model_selection_changed` | `chat_id`, `available_models`, `new_model?` | Defined in the shared stream schema. |
| `token_count_changed` | `chat_id`, `token_count` | Defined in the shared stream schema. |
| `chat_message_status_update` | `chat_id`, `message_id`, `message` | Defined in the shared stream schema. |
| `extraction_status_ack` | `attachment_id`, `status` | Initial extraction status response. |
| `extraction_status_update` | `attachment_id`, `status` | Later extraction status update. |
| `completion_response` | `completion_id`, `content`, `done` | PDF completion payload. |
| `completion_stream_chunk` | `completion_id`, `content`, `done` | PDF completion chunk payload. |
### Assistant message parts
`chat_message_response.content` uses camelCase part tags:
| `content.type` | Fields |
| --- | --- |
| `text` | `text` |
| `thinking` | `thinking` |
| `toolCall` | `name`, `json`, `id` |
| `mcpToolCall` | `name`, `service`, `display_name`, `json`, `id` |
| `toolCallResponseJson` | `name`, `json`, `id` |
| `toolCallErr` | `name`, `description`, `id` |
Frontend message assembly ignores non-`chat_message_response` payloads and merges adjacent `text` chunks and adjacent `thinking` chunks into single parts.
## Toolset selection
`ToolSet` is a tagged object:
```json
{ "type": "all" }
```
```json
{ "type": "none" }
```
| Toolset | Prompt selected |
| --- | --- |
| `all` | DCS `all_tools_prompt` |
| `none` | `ai_tools::prompts::BASE_PROMPT` |
<Warning>
In the current chat and structured-completion implementations, `toolset` selects the system prompt. The agent session is still constructed with the combined static and MCP toolset from enabled MCP records.
</Warning>
## Model identifiers
Use `AgentModel` values in API payloads and frontend state, not provider API IDs.
| API value | UI label | Provider route in agent model |
| --- | --- | --- |
| `smart` | Smart | Anthropic Opus 4.7 |
| `fast` | Fast | Anthropic Haiku 4.5 |
| `opus4_7` | Opus 4.7 | Anthropic Opus 4.7 |
| `sonnet4_6` | Sonnet 4.6 | Anthropic Sonnet 4.6 |
| `haiku4_5` | Haiku 4.5 | Anthropic Haiku 4.5 |
| `retired` | Retired | Falls back to the Smart/Opus route |
Additional model behavior:
| Model group | Context window | Thinking parameters |
| --- | ---:| --- |
| `smart`, `opus4_7`, `sonnet4_6`, `retired` | `1_000_000` tokens | Opus uses adaptive thinking; Sonnet uses enabled thinking with `budget_tokens: 10000`. |
| `fast`, `haiku4_5` | `200_000` tokens | Enabled thinking with `budget_tokens: 10000`. |
Unrecognized model strings deserialize to `retired`, which routes to the default Opus-backed path.
## Extraction statuses
Extraction statuses are tagged objects:
```json
{ "type": "incomplete" }
```
| Status | Meaning |
| --- | --- |
| `incomplete` | No extracted document text row exists yet. |
| `empty` | Extraction completed, but extracted text is empty or has no content length. |
| `insufficient` | Extracted text has fewer than `1000` non-whitespace characters. |
| `complete` | Extracted text has at least `1000` non-whitespace characters. |
`extraction_status_ack` tells the client the current status. If it is `incomplete`, clients should wait for later `extraction_status_update` events for the same `attachment_id`.
## Stop and persistence semantics
Stopping a stream publishes to Redis on a per-`stream_id` cancellation channel. The DCS instance running the stream subscribes when generation starts; any instance can publish the stop signal.
When cancellation is observed:
1. The streaming loop breaks.
2. DCS emits `stream_end`.
3. DCS persists the assistant parts already yielded.
4. If a yielded tool call has no matching response, DCS inserts a persisted `toolCallErr` with `description: "cancelled"` so later conversation turns remain well-formed.
5. DCS skips the notification summarization path for cancelled streams.
The chat stream also has a `3 minute` idle timeout between AI stream items. Timeout emits an `error` payload with `internal_error`, then the stream ends.
## Error responses
| Surface | Statuses in API schema | Body |
| --- | --- | --- |
| `POST /stream/chat/message` | `400`, `401`, `402`, `403` | `400` returns `{ "error": string, "stream_id"?: string }`. |
| `POST /stream/chat/message/stop` | `200`, `403` | `403` returns `{ "error": string }`. |
| `POST /structured-completion` | `400`, `401`, `402`, `500` | Error responses return `{ "error": string }`. |
Common failure points include invalid user IDs, missing chat permissions, message storage failure, request building failure, agent stream creation failure, AI stream errors, and structured-output validation or generation failure.
## Related pages
- Chat state and stream rendering
- Connection gateway stream subscription
- MCP tool registration and tool call rendering
---
## 21. MCP server and tool registry
> Reference: Remote MCP endpoint, OAuth-backed server, toolset composition, SendEmail boundary, generated schemas, and docs tool pages.
- Page Markdown: https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/21-mcp-server-and-tool-registry.md
- Generated: 2026-06-01T01:02:24.313Z
### Source Files
- `docs/AI/mcp/overview.mdx`
- `docs/AI/mcp/tools/index.mdx`
- `rust/cloud-storage/mcp_service/src/main.rs`
- `rust/cloud-storage/mcp_service/src/tool_service.rs`
- `rust/cloud-storage/ai_tools/src/lib.rs`
- `docs/scripts/generate-mcp-tool-pages.ts`
---
title: "MCP server and tool registry"
description: "Reference: Remote MCP endpoint, OAuth-backed server, toolset composition, SendEmail boundary, generated schemas, and docs tool pages."
---
Macro exposes a remote Streamable HTTP MCP endpoint at `https://mcp-server.macro.com/mcp`; the Rust `mcp_service` binary mounts the protected MCP service at `/mcp`, brokers OAuth through FusionAuth-backed routes, builds the runtime registry from `ai_tools::mcp_tools()`, and executes tool calls with the authenticated Macro user id.
## Remote endpoint
```text
https://mcp-server.macro.com/mcp
```
The public docs configure clients as an HTTP MCP server; no local Macro process is required.
<CodeGroup>
```bash title="Claude Code"
claude mcp add --transport http macro https://mcp-server.macro.com/mcp
```
```bash title="Codex CLI"
codex mcp add macro --url https://mcp-server.macro.com/mcp
```
```json title="Generic MCP JSON config"
{
"mcpServers": {
"macro": {
"type": "http",
"url": "https://mcp-server.macro.com/mcp"
}
}
}
```
</CodeGroup>
At runtime, `mcp_service` binds `0.0.0.0:${PORT}` with `PORT=8090` as the default. The server config enables JSON responses, disables stateful mode, and allows the host from `MCP_PUBLIC_URL` plus `localhost` and `127.0.0.1`.
```text
Client
├─ OAuth discovery/register/authorize/token ──> mcp_auth_proxy routes
└─ Bearer-authenticated MCP calls /mcp ───────> AuthenticatedToolService
└─ ai_tools::mcp_tools()
└─ Macro domain services
```
## HTTP routes
| Route | Auth | Purpose |
| --- | --- | --- |
| `GET /health` | No bearer token | ALB health check returning `ok`. |
| `GET /.well-known/oauth-protected-resource` | No bearer token | OAuth protected-resource metadata. |
| `GET /.well-known/oauth-protected-resource/mcp` | No bearer token | Protected-resource metadata for MCP clients. |
| `GET /mcp/.well-known/oauth-protected-resource` | No bearer token | Alternate protected-resource discovery path. |
| `GET /.well-known/oauth-authorization-server` | No bearer token | OAuth authorization-server metadata. |
| `GET /.well-known/oauth-authorization-server/mcp` | No bearer token | Authorization-server metadata for MCP clients. |
| `GET /mcp/.well-known/oauth-authorization-server` | No bearer token | Alternate authorization-server discovery path. |
| `POST /register` | No bearer token | Dynamic public-client registration. |
| `GET /authorize` | No bearer token | Starts an OAuth authorization-code + PKCE flow. |
| `GET /oauth/callback` | No bearer token | Receives the upstream FusionAuth callback. |
| `POST /token` | No bearer token | Exchanges broker-issued codes or refresh tokens. |
| `/mcp` | Bearer token required | Streamable HTTP MCP service. |
CORS is applied outside the bearer middleware so browser MCP clients can complete OAuth preflights and still receive `WWW-Authenticate` challenges. Allowed methods are `GET`, `POST`, `DELETE`, and `OPTIONS`; allowed headers include `Authorization`, `Content-Type`, `mcp-protocol-version`, and `mcp-session-id`.
## OAuth and access control
The MCP auth proxy presents itself as the OAuth authorization server to MCP clients while delegating the upstream sign-in flow to FusionAuth.
### OAuth behavior
| Behavior | Value |
| --- | --- |
| Supported `response_type` | `code` |
| Supported grant types | `authorization_code`, `refresh_token` |
| Supported PKCE method | `S256` |
| Dynamic client auth method | `none` |
| Pending authorization TTL | 10 minutes |
| Broker-issued authorization code TTL | 5 minutes |
| Redirect URI policy | `https` or loopback `http` for `localhost`, `127.0.0.1`, or `[::1]` |
| Upstream provider | FusionAuth OAuth provider using the `google_gmail` identity provider |
| In-flight state store | Redis keys under `mcp_auth_proxy:pending:*` and `mcp_auth_proxy:issued:*` |
The `/mcp` route validates `Authorization: Bearer <token>` with Macro JWT validation. Both Macro access tokens and Macro API tokens can supply the `macro_user_id`; the middleware stores that value as `MacroUserIdStr` in request extensions for the MCP service.
### Subscription gate
`list_tools` and `call_tool` both require the authenticated user to have at least one paid AI permission:
- `write:opus`
- `write:sonnet`
- `write:haiku`
If none are present, the MCP service returns an MCP invalid-request error with:
```text
MCP access requires a paid subscription
```
## Tool service contract
`AuthenticatedToolService` implements `rmcp::handler::server::ServerHandler`.
| MCP method | Runtime behavior |
| --- | --- |
| `get_info` | Advertises `macro-tools`, title `Macro`, package version, tool capability, and instructions for searching, reading, creating documents, and listing entities. |
| `list_tools` | Extracts the authenticated user, checks paid access, and returns every tool in `toolset.tools` with its name, description, and input schema. |
| `call_tool` | Extracts the authenticated user, checks paid access, requires object arguments, then invokes `toolset.try_tool_call(...)` with `RequestContext { user_id }`. |
### Call results and errors
| Case | MCP response |
| --- | --- |
| Missing `arguments` | Invalid params: `No params provided`. |
| Tool input deserialization fails | Parse error with the deserialization message. |
| Tool name is not registered | Resource-not-found error. |
| Tool returns `Ok(value)` | Structured MCP tool result. |
| Tool returns a domain `ToolCallError` | MCP tool-result error containing the user-facing error description as text. |
| Missing user id in request extensions | Internal error: `missing user identity — is auth configured?`. |
## Runtime toolset composition
The MCP runtime registry is constructed in `ai_tools::mcp_tools()`, not from the checked-in MDX files.
| Function | Composition |
| --- | --- |
| `subagent_toolset()` | Search tools, `ListEntities`, document tools, property tools, call tools, chat tools, channel tools, team tools, and Anthropic server-tool wrappers. It intentionally excludes email and the `Subagent` tool itself. |
| `all_tools()` | `subagent_toolset()` plus notification tools, the full email toolset, and `Subagent`. This is the broader AI-provider-facing registry. |
| `mcp_tools()` | `subagent_toolset()` plus notification tools, the MCP email toolset, and `Subagent`. This is what `mcp_service` exposes. |
| `no_tools()` | Empty toolset with the base prompt. |
<Note>
The registry boundary is provider-neutral at the MCP layer: tools are composed as Rust `AsyncToolCollection` entries. The current registry includes a subtoolset for Anthropic server-tool wrappers, but the MCP server contract is the tool registry and Streamable HTTP transport, not a requirement that every deployment use one model provider.
</Note>
## SendEmail boundary
`SendEmail` is deliberately outside the MCP email toolset.
| Email registry | Includes |
| --- | --- |
| `email::inbound::toolset::mcp_toolset()` | `UpdateThreadLabels`, `GetThread`, `ListLabels` |
| `email::inbound::toolset::email_toolset()` | Everything in `mcp_toolset()` plus `SendEmail` as a user tool |
| `ai_tools::mcp_tools()` | Uses `email_mcp_toolset()`; therefore excludes direct `SendEmail` execution |
| `ai_tools::all_tools()` | Uses the full `email_toolset()` |
`SendEmail` composes and sends an email by resolving the user’s linked email account, creating message input from `subject`, Markdown/HTML body, recipients, and optional `replyingToId`, then calling the email service. In the full AI tool registry it is registered through `add_user_tool`, whose wrapper returns `PendingUserExecution` when called by the automatic tool loop. That user-action boundary is not exposed through the MCP runtime registry.
<Warning>
The checked-in generated MCP tool pages currently include a `SendEmail` page. Treat the live MCP `list_tools` response and `ai_tools::mcp_tools()` as the runtime source of truth.
</Warning>
## Generated schemas and docs tool pages
The docs generator lives in `docs/scripts/generate-mcp-tool-pages.ts` and is wired through the docs package scripts.
```json title="docs/package.json"
{
"scripts": {
"generate:tools": "bun ./scripts/generate-mcp-tool-pages.ts",
"prepare": "bun run generate:tools"
}
}
```
The generator workflow is:
<Steps>
<Step title="Build the Rust schema binary">
It runs `SQLX_OFFLINE=true cargo build --bin gen_tool_schemas` from `rust/cloud-storage`.
</Step>
<Step title="Regenerate the schema JSON">
It removes `rust/cloud-storage/ai_tools/schemas`, then runs the `gen_tool_schemas` binary from the `ai_tools` directory. The binary writes `schemas/tools.json`.
</Step>
<Step title="Write MDX pages">
It rewrites `docs/AI/mcp/tools/*.mdx`, including `index.mdx`.
</Step>
<Step title="Update navigation">
It writes `docs/config/tool-pages.json`, which is referenced by `docs/config/navigation.json` under the MCP Tool Reference group.
</Step>
</Steps>
The Rust schema binary currently serializes `ai_tools::all_tool_combined_schema()`, which merges the broad `all_tools()` registry plus a schema-only `ReadThread` phantom tool into a combined schema with shared `$defs` and `tools` entries.
<Warning>
The TypeScript docs script expects a `schemas` array with `name`, `description`, `inputSchema`, and `outputSchema`, while the Rust binary emits the combined schema shape `{ "$defs": ..., "tools": [...] }`. Keep the Rust schema shape and docs script contract aligned before relying on `bun run generate:tools` in CI or release automation.
</Warning>
## Configuration
`McpEnvVars` loads these required environment variables during context construction. `PORT` is read separately and defaults to `8090`.
| Environment variable | Used for |
| --- | --- |
| `DATABASE_URL` | Postgres connection pool for tools, permissions, and domain services. |
| `EMAIL_SCHEDULED_QUEUE` | SQS email scheduling queue configuration. |
| `DOCUMENT_STORAGE_SERVICE_URL` | Search/document client base URL wiring. |
| `SYNC_SERVICE_URL` | Sync service client base URL. |
| `SYNC_SERVICE_AUTH_KEY` | Secret-manager key for sync service authentication. |
| `LEXICAL_SERVICE_URL` | Lexical client base URL. |
| `EMAIL_SERVICE_URL` | Email service client base URL. |
| `STATIC_FILE_SERVICE_URL` | Loaded as part of MCP environment configuration. |
| `DOCUMENT_STORAGE_BUCKET` | Document S3 upload adapter bucket. |
| `DOCX_DOCUMENT_UPLOAD_BUCKET` | DOCX upload bucket. |
| `DOCUMENT_STORAGE_SERVICE_CLOUDFRONT_DISTRIBUTION_URL` | CloudFront document distribution URL. |
| `DOCUMENT_STORAGE_SERVICE_CLOUDFRONT_SIGNER_PUBLIC_KEY_ID` | CloudFront signer public key id. |
| `DOCUMENT_STORAGE_SERVICE_CLOUDFRONT_SIGNER_PRIVATE_KEY_SECRET_NAME` | Secret-manager key for the CloudFront private key. |
| `MCP_PUBLIC_URL` | Public base URL for OAuth metadata, callback URL construction, and allowed host derivation. |
| `FUSIONAUTH_BASE_URL` | FusionAuth base URL. |
| `FUSIONAUTH_CLIENT_ID` | FusionAuth OAuth client id. |
| `FUSIONAUTH_TENANT_ID` | FusionAuth tenant id. |
| `FUSIONAUTH_API_KEY_SECRET_KEY` | Secret-manager key for FusionAuth API access. |
| `FUSIONAUTH_CLIENT_SECRET_KEY` | Secret-manager key for the FusionAuth client secret. |
| `GOOGLE_CLIENT_ID` | Google OAuth client id passed to FusionAuth client setup. |
| `GOOGLE_CLIENT_SECRET_KEY` | Secret-manager key for the Google client secret. |
| `REDIS_URL` | Redis connection string for in-flight OAuth state. |
| `PORT` | Optional listen port; defaults to `8090`. |
## Adding or changing MCP tools
1. Implement the tool as an `AsyncTool` with `Deserialize` input and `Serialize + JsonSchema` output.
2. Register it in the narrow domain toolset first.
3. Wire that domain toolset into `ai_tools::mcp_tools()` only if it should be exposed over remote MCP.
4. Add or update `FromRef<ToolServiceContext>` wiring and `build_context()` dependencies when the tool needs a new service.
5. Keep `email::mcp_toolset()` separate from `email_toolset()` if the operation requires user review or should not be remotely executable.
6. Regenerate docs with `cd docs && bun run generate:tools` after fixing or confirming the schema contract.
7. Verify the live runtime with an authenticated MCP `list_tools` call; do not use checked-in MDX pages as the only availability signal.
## Troubleshooting
| Symptom | Likely cause | Verification |
| --- | --- | --- |
| `401` with `WWW-Authenticate` and no `error` | Missing bearer token on `/mcp`. | Check the client completed OAuth and sends `Authorization: Bearer ...`. |
| `401` with `error="invalid_token"` | JWT validation failed or token user id was invalid. | Re-run the OAuth flow or inspect token issuer/configuration. |
| `redirect_uri must be https or a loopback address` | OAuth client used a non-HTTPS, non-loopback redirect URI. | Use HTTPS or `http://localhost`, `http://127.0.0.1`, or `http://[::1]`. |
| `unsupported code_challenge_method` | Client did not use PKCE `S256`. | Confirm MCP client OAuth settings. |
| `invalid or expired code` | Broker-issued code was already used or older than 5 minutes. | Restart the OAuth flow. |
| `MCP access requires a paid subscription` | User lacks `write:opus`, `write:sonnet`, and `write:haiku`. | Check user permissions in Macro. |
| `No params provided` | `call_tool` request omitted arguments. | Send an object in MCP tool arguments, even when the tool has few parameters. |
| Tool page exists but tool is absent from MCP | Generated docs are broader or stale relative to `mcp_tools()`. | Compare against authenticated `list_tools`. |
## Related pages
<CardGroup>
<Card title="MCP setup" href="/AI/mcp/overview">
Client connection commands and the public remote endpoint.
</Card>
<Card title="MCP tool reference" href="/AI/mcp/tools">
Generated tool pages and navigation for documented tool schemas.
</Card>
</CardGroup>
---
## 22. Environment variables reference
> Reference: Required and optional service environment variables, defaults, secrets, queue names, ports, and provider-specific keys.
- Page Markdown: https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/22-environment-variables-reference.md
- Generated: 2026-06-01T01:04:54.159Z
### Source Files
- `RUNNING_LOCALLY.md`
- `rust/cloud-storage/document_storage_service/src/config.rs`
- `rust/cloud-storage/document_cognition_service/src/config.rs`
- `rust/cloud-storage/authentication_service/src/config.rs`
- `rust/cloud-storage/email_service/src/config.rs`
- `rust/cloud-storage/notification_service/src/config.rs`
- `rust/cloud-storage/search_service/src/config.rs`
- `rust/cloud-storage/macro_env/src/lib.rs`
---
title: "Environment variables reference"
description: "Reference: Required and optional service environment variables, defaults, secrets, queue names, ports, and provider-specific keys."
---
Macro’s local and deployed service binaries read configuration directly from environment variables at startup. Most Rust services default `PORT` to `8080`, derive `ENVIRONMENT` with `prod` fallback, and fail fast when required variables are missing or cannot be parsed.
## Runtime conventions
| Convention | Behavior |
| --- | --- |
| `ENVIRONMENT` | Accepted values are `prod`, `dev`, and `local`. Missing or invalid values fall back to production behavior. |
| `env_var!` names | Rust `CamelCase` config structs map to `UPPER_SNAKE_CASE`; for example `DocumentStorageBucket` reads `DOCUMENT_STORAGE_BUCKET`. |
| Required variables | `env_var!` and direct `std::env::var(...).context(...)` reads fail service startup when absent. |
| Optional variables | `maybe_env_var!` and `.ok()` reads return `None` when absent. |
| Local secrets | In `ENVIRONMENT=local`, secret variables are normally used as literal secret values. |
| Deployed secrets | In `dev` and `prod`, variables passed through `LocalOrRemoteSecret` or explicit secret-manager lookups are treated as AWS Secrets Manager secret IDs. |
| AWS local mode | `LOCAL_AWS_URL` switches shared AWS clients to LocalStack-style endpoint and test credentials. |
| CORS override | `ALLOWED_ORIGINS` can replace the default comma-separated origin allowlist used by the shared CORS layer. |
<Warning>
Some numeric variables use `.unwrap()` after parsing. Invalid values can panic instead of returning a structured startup error.
</Warning>
## Local environment sources
Local setup expects an encrypted dotenv source and writes a runtime `.env` file.
```bash
just setup
just run_local
```
`just setup` decrypts `.env-local*.enc`, creates shared Docker networks and volumes, starts LocalStack, initializes databases, sets up FusionAuth, and builds local service images. `just run_local` patches local FusionAuth values into `.env` before starting Docker Compose.
Local E2E uses explicit overrides:
```bash
COMPOSE_FILE=docker-compose.yml:docker-compose.local-e2e.yml just run_local -d --wait
```
The local E2E override forces local Postgres, Redis, LocalStack, S3 buckets, DynamoDB tables, and queue names instead of shared dev assets.
## Ports
### Local service host ports
| Service | Host port | Container port / default |
| --- | ---: | ---: |
| `authentication-service` | `8080` | `8080` |
| `connection_gateway` | `8082` | `8080` |
| `contacts_service` | `8083` | `8080` |
| `document_cognition_service` | `8085` | `8080` |
| `document_storage_service` | `8086` | `8080` |
| `email_service` | `8087` | `8080` |
| `notification_service` | `8089` | `8080` |
| `search_processing_service` | `8092` | `8080` |
| `static_file_service` | `8094` | `8080` |
| `unfurl_service` | `8095` | `8080` |
| `lexical_service` | `8096` | `8096` |
| `image_proxy_service` | `8097` | `8080` |
| `static_file_cdn` | `8100` | `80` |
| `websocket_service` | `6969` | `6969` |
| `sync_service` | `8787` | `8787` |
| FusionAuth | `9011` | `9011` |
| Postgres | `5432` | `5432` |
| Redis | `6379` | `6379` |
| Redis Stack UI | `8001` | `8001` |
| OpenSearch REST | `9200` | `9200` |
| OpenSearch performance analyzer | `9600` | `9600` |
## Shared variables
### Environment and infrastructure
| Variable | Required by | Default | Notes |
| --- | --- | --- | --- |
| `ENVIRONMENT` | Most services | `prod` fallback | Values: `prod`, `dev`, `local`. |
| `PORT` | HTTP services | Usually `8080` | `lexical_service` defaults to `8096`; `websocket_service` uses `6969`. |
| `DATABASE_URL` | Most database-backed services | None | Primary MacroDB connection URL or secret ID where explicitly resolved. |
| `DATABASE_URL_READONLY` | `document_storage_service`, `search_processing_service` | Optional | DSS falls back to primary if readonly connection fails; search backfills use primary when absent or unreachable. |
| `MACRO_DB_URL` | `email_service`, `connection_gateway` | None | Primary MacroDB URL for services that use this name instead of `DATABASE_URL`. |
| `REDIS_URI` | Auth, DSS, email, notification, contacts, workers | None | Redis URL. |
| `REDIS_HOST` | `connection_gateway`, DCS | None | Redis URL-like host used by typed config. |
| `REDIS_URL` | MCP service and some stream/test paths | None | Separate name used by MCP auth proxy wiring. |
| `LOCAL_AWS_URL` | AWS client factory | Optional | Enables LocalStack endpoint and test credentials. |
| `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_REGION`, `AWS_DEFAULT_REGION` | LocalStack / AWS SDK | Local E2E sets test credentials | Needed by local AWS CLI setup and SDK calls when not using ambient credentials. |
| `DD_SERVICE`, `DD_ENV` | Deployed tracing | `unknown-service`, `unknown` | Used by OpenTelemetry/Datadog entrypoint in `dev` and `prod`. |
### Shared auth and JWT
| Variable | Required by | Default | Notes |
| --- | --- | --- | --- |
| `INTERNAL_API_SECRET_KEY` | Internal-auth protected services | None | Validated against `x-internal-auth-key`. Local value is literal; deployed uses secret-manager resolution where wired through `LocalOrRemoteSecret`. |
| `AUDIENCE` | Services constructing JWT validation | None | JWT validation config. |
| `ISSUER` | Services constructing JWT validation | None | JWT validation config. |
| `JWT_SECRET_KEY` | Services constructing JWT validation | None | Local value or deployed secret ID. |
| `MACRO_API_TOKEN_ISSUER` | Auth and JWT validation | None | Used for Macro API tokens. |
| `MACRO_API_TOKEN_PUBLIC_KEY` | JWT validation | None | Local public key or deployed secret ID. |
| `MACRO_API_TOKEN_PRIVATE_SECRET_KEY` | `authentication_service` | None | Local private key or deployed secret ID for Macro API token signing. |
| `MACRO_API_TOKEN_EXPIRY_SECONDS` | `authentication_service` | None | Parsed as `usize`. |
| `STRIPE_WEBHOOK_SECRET_KEY` | `authentication_service` | None | Local value or deployed secret ID. |
## Service reference
### `authentication_service`
Required:
| Variable group | Variables |
| --- | --- |
| Base URLs and stores | `BASE_URL`, `DATABASE_URL`, `REDIS_URI`, `DOCUMENT_STORAGE_SERVICE_URL` |
| FusionAuth | `FUSIONAUTH_TENANT_ID`, `FUSIONAUTH_API_KEY_SECRET_KEY`, `FUSIONAUTH_CLIENT_ID`, `FUSIONAUTH_CLIENT_SECRET_KEY`, `FUSIONAUTH_BASE_URL`, `FUSIONAUTH_OAUTH_REDIRECT_URI` |
| Google OAuth | `GOOGLE_CLIENT_ID`, `GOOGLE_CLIENT_SECRET_KEY` |
| Stripe | `STRIPE_SECRET_KEY`, `STRIPE_PRICE_ID_HAIKU`, `STRIPE_PRICE_ID_SONNET`, `STRIPE_PRICE_ID_OPUS` |
| GitHub OAuth | `GITHUB_CLIENT_ID`, `GITHUB_CLIENT_SECRET`, `GITHUB_IDP_ID` |
| Queues | `NOTIFICATION_QUEUE`, `SEARCH_EVENT_QUEUE`, `LINK_MANAGER_QUEUE`, `EMAIL_BACKFILL_QUEUE` |
| Internal auth | `SERVICE_INTERNAL_AUTH_KEY`, plus shared `INTERNAL_API_SECRET_KEY` in service startup |
Optional:
| Variable | Default | Notes |
| --- | --- | --- |
| `PORT` | `8080` | HTTP listener. |
| `GA_MEASUREMENT_ID`, `GA_API_SECRET` | None | Google Analytics Measurement Protocol. |
| `META_PIXEL_ID`, `META_ACCESS_TOKEN`, `META_TEST_EVENT_CODE` | None | Meta conversions tracking. |
| `POSTHOG_API_KEY`, `POSTHOG_HOST` | None | PostHog analytics. |
The active Stripe price ID is selected in code by `ENVIRONMENT`: production uses the production price ID, while `dev` and `local` use the development price ID.
### `document_storage_service`
Required:
| Variable group | Variables |
| --- | --- |
| Stores | `DATABASE_URL`, `DATABASE_URL_READONLY`, `REDIS_URI`, `DOCUMENT_STORAGE_BUCKET`, `DOCX_DOCUMENT_UPLOAD_BUCKET`, `UPLOAD_STAGING_BUCKET`, `BULK_UPLOAD_REQUESTS_TABLE` |
| Queues | `DOCUMENT_DELETE_QUEUE`, `NOTIFICATION_QUEUE`, `SEARCH_EVENT_QUEUE`, `CONTACTS_QUEUE` |
| Service URLs | `CONNECTION_GATEWAY_URL`, `SYNC_SERVICE_URL`, `LEXICAL_SERVICE_URL`, `GITHUB_SYNC_APP_URL` |
| Sync and search | `SYNC_SERVICE_AUTH_KEY`, `OPENSEARCH_URL`, `OPENSEARCH_USERNAME`, `OPENSEARCH_PASSWORD` |
| CloudFront | `DOCUMENT_STORAGE_SERVICE_CLOUDFRONT_DISTRIBUTION_URL`, `DOCUMENT_STORAGE_SERVICE_CLOUDFRONT_SIGNER_PUBLIC_KEY_ID`, `DOCUMENT_STORAGE_SERVICE_CLOUDFRONT_SIGNER_PRIVATE_KEY_SECRET_NAME` |
| Secrets and integrations | `DOCUMENT_PERMISSION_JWT_SECRET_KEY`, `GITHUB_WEBHOOK_SECRET_KEY`, `GITHUB_SYNC_APP_PEM_SECRET_KEY`, `GITHUB_SYNC_APP_CLIENT_ID`, `CAL_WEBHOOK_SECRET_KEY`, `CAL_EVENT_TYPE_CONTENT_NAMES_KEY`, `META_PIXEL_ID`, `META_ACCESS_TOKEN` |
| LiveKit calls | `LIVEKIT_SERVER_URL`, `LIVEKIT_API_KEY`, `LIVEKIT_API_SECRET` |
Optional and tunable:
| Variable | Default | Notes |
| --- | --- | --- |
| `PORT` | `8080` | HTTP listener. |
| `DOCUMENT_LIMIT` | `20` | Free-user document limit. |
| `DOCUMENT_STORAGE_SERVICE_PRESIGNED_URL_EXPIRY_SECONDS` | `900` | Signed document URL TTL. |
| `DOCUMENT_STORAGE_SERVICE_PRESIGNED_URL_BROWSER_CACHE_EXPIRY_SECONDS` | `840` | Browser cache suggestion for signed URLs. |
| `QUEUE_MAX_MESSAGES` | `10` | Delete-document worker poll batch size. |
| `QUEUE_WAIT_TIME_SECONDS` | `4` | Delete-document worker long-poll wait. |
| `META_TEST_EVENT_CODE` | None | Meta test events. |
| `LIVEKIT_TRANSCRIPTION_AGENT_NAME` | None | Requires `INTERNAL_CALL_SECRET` when set. |
| `INTERNAL_CALL_SECRET` | None | Shared secret for internal call endpoints. |
| `CALL_RECORDING_S3_BUCKET`, `CALL_RECORDING_S3_REGION`, `CALL_RECORDING_S3_ACCESS_KEY`, `CALL_RECORDING_S3_SECRET` | None | Call recording egress is enabled only when all four are present. |
| `APPLE_BUNDLE_ID`, `SNS_APNS_VOIP_PLATFORM_ARN` | None | Optional VoIP push sender. Empty VoIP ARN disables VoIP push. |
### `document_cognition_service`
Required:
| Variable group | Variables |
| --- | --- |
| Stores | `DATABASE_URL`, `DOCUMENT_STORAGE_BUCKET`, `DOCX_DOCUMENT_UPLOAD_BUCKET`, `REDIS_HOST` |
| Service URLs | `DOCUMENT_STORAGE_SERVICE_URL`, `DOCUMENT_COGNITION_SERVICE_URL`, `SYNC_SERVICE_URL`, `LEXICAL_SERVICE_URL`, `EMAIL_SERVICE_URL`, `STATIC_FILE_SERVICE_URL`, `AUTHENTICATION_SERVICE_URL` |
| Queues | `DOCUMENT_TEXT_EXTRACTOR_QUEUE`, `CHAT_DELETE_QUEUE`, `EMAIL_SCHEDULED_QUEUE`, `NOTIFICATION_QUEUE`, `SEARCH_EVENT_QUEUE` |
| Secrets | `SYNC_SERVICE_AUTH_KEY`, `AUTHENTICATION_SERVICE_SECRET_KEY`, `DOCUMENT_STORAGE_SERVICE_CLOUDFRONT_SIGNER_PRIVATE_KEY_SECRET_NAME`, `MCP_CREDENTIALS_KEY_SECRET_NAME` |
| CloudFront | `DOCUMENT_STORAGE_SERVICE_CLOUDFRONT_DISTRIBUTION_URL`, `DOCUMENT_STORAGE_SERVICE_CLOUDFRONT_SIGNER_PUBLIC_KEY_ID` |
Optional:
| Variable | Default | Notes |
| --- | --- | --- |
| `PORT` | `8080` | HTTP listener. |
| `DOCUMENT_BATCH_LIMIT` | `1000` | Maximum document query batch size. |
| `OPENAI_API_KEY` | Empty string | Used by the non-streaming OpenAI chat completions proxy. Missing key still sends an upstream request with an empty bearer token. |
### `email_service`
Required:
| Variable group | Variables |
| --- | --- |
| Stores | `MACRO_DB_URL`, `REDIS_URI`, `ATTACHMENT_BUCKET` |
| Queues | `LINK_MANAGER_QUEUE`, `EMAIL_SCHEDULED_QUEUE`, `GMAIL_INBOX_SYNC_QUEUE`, `GMAIL_INBOX_SYNC_RETRY_QUEUE`, `GMAIL_OPS_QUEUE`, `GMAIL_OPS_RETRY_QUEUE`, `SEARCH_EVENT_QUEUE`, `GMAIL_GCP_QUEUE`, `NOTIFICATION_QUEUE`, `BACKFILL_QUEUE`, `CONTACTS_QUEUE`, `SFS_UPLOADER_QUEUE`, `SFS_DELETE_QUEUE` |
| Service URLs | `AUTHENTICATION_SERVICE_URL`, `STATIC_FILE_SERVICE_URL`, `DOCUMENT_STORAGE_SERVICE_URL`, `CONNECTION_GATEWAY_URL` |
| Secrets | `AUTHENTICATION_SERVICE_SECRET_KEY`, `EMAIL_SERVICE_CLOUDFRONT_SIGNER_PRIVATE_KEY` |
| CloudFront | `EMAIL_SERVICE_CLOUDFRONT_DISTRIBUTION_URL`, `EMAIL_SERVICE_CLOUDFRONT_SIGNER_PUBLIC_KEY_ID` |
| Feature gates | `NOTIFICATIONS_ENABLED` |
Optional and tunable:
| Variable | Default | Notes |
| --- | --- | --- |
| `PORT` | `8080` | HTTP listener. |
| `SENT_UNDO_DELAY_SECS` | `10` | Delay before processing sent mail. |
| `USE_APOLLO_CRM_ENRICHMENT` | `false` | Enables Apollo.io CRM enrichment. |
| `APOLLO_API_KEY` | Empty string | Literal key locally; deployed comments indicate secret name/value resolution by startup wiring. |
| `QUEUE_MAX_MESSAGES` | `10` | Generic queue poll batch size. |
| `QUEUE_WAIT_TIME_SECONDS` | `20` | Generic long-poll wait. |
| `BACKFILL_QUEUE_WORKERS` | `25` | Backfill worker count. |
| `BACKFILL_QUEUE_MAX_MESSAGES` | `1` | Backfill batch size. |
| `INBOX_SYNC_QUEUE_WORKERS` | `10` | Gmail inbox sync worker count. |
| `INBOX_SYNC_QUEUE_MAX_MESSAGES` | `1` | Gmail inbox sync batch size. |
| `INBOX_SYNC_RETRY_QUEUE_WORKERS` | `10` | Gmail retry worker count. |
| `INBOX_SYNC_RETRY_QUEUE_MAX_MESSAGES` | `1` | Gmail retry batch size. |
| `GMAIL_OPS_QUEUE_WORKERS` | `5` | Gmail ops worker count. |
| `GMAIL_OPS_QUEUE_MAX_MESSAGES` | `10` | Gmail ops batch size. |
| `GMAIL_OPS_RETRY_QUEUE_WORKERS` | `2` | Gmail ops retry worker count. |
| `GMAIL_OPS_RETRY_QUEUE_MAX_MESSAGES` | `10` | Gmail ops retry batch size. |
| `SFS_UPLOADER_WORKERS` | `3` | Static-file upload mapper worker count. |
| `REDIS_RATE_LIMIT_REQS` | `14000` | Sliding-window request limit. |
| `REDIS_RATE_LIMIT_REQS_BACKFILL` | `13000` | Backfill-specific rate limit. |
| `REDIS_RATE_LIMIT_WINDOW_SECS` | `60` | Rate-limit window. |
| `EMAIL_SERVICE_PRESIGNED_URL_TTL_SECS` | `3600` | Attachment signed URL TTL. |
### `notification_service`
Required:
| Variable group | Variables |
| --- | --- |
| Base and stores | `BASE_URL`, `DATABASE_URL`, `REDIS_URI` |
| Internal auth | `INTERNAL_API_SECRET_KEY`, `URL_SIGNING_HMAC` |
| Queues | `NOTIFICATION_QUEUE`, `NOTIFICATION_INGRESS_QUEUE`, `PUSH_NOTIFICATION_EVENT_HANDLER_QUEUE` |
| Push providers | `SNS_APNS_PLATFORM_ARN`, `SNS_FCM_PLATFORM_ARN`, `APPLE_BUNDLE_ID` |
| Service URLs | `CONNECTION_GATEWAY_URL` |
| Email sender | `SENDER_BASE_ADDRESS` |
Optional and tunable:
| Variable | Default | Notes |
| --- | --- | --- |
| `PORT` | `8080` | HTTP listener. |
| `NOTIFICATION_QUEUE_MAX_MESSAGES` | `9` | Notification worker poll batch size. |
| `NOTIFICATION_QUEUE_WAIT_TIME_SECONDS` | `4` | Notification worker long-poll wait. |
| `SNS_APNS_VOIP_PLATFORM_ARN` | Required except local | Local can omit it; service uses an empty string locally to keep VoIP disabled. |
`SENDER_BASE_ADDRESS` is transformed into `no-reply@...`, `no-reply-dev@...`, or `no-reply-local@...` according to `ENVIRONMENT`.
### Search services
#### `search_service`
| Variable | Required | Default | Notes |
| --- | --- | --- | --- |
| `DATABASE_URL` | Yes | None | MacroDB connection. |
| `OPENSEARCH_URL` | Yes | None | OpenSearch endpoint. |
| `OPENSEARCH_USERNAME` | Yes | None | OpenSearch username. |
| `OPENSEARCH_PASSWORD` | Yes | None | Local password or deployed secret where caller resolves it. |
| `INTERNAL_API_SECRET_KEY` | Yes | None | Internal API auth. |
| `PORT` | No | `8080` | HTTP listener. |
#### `search_processing_service`
| Variable | Required | Default | Notes |
| --- | --- | --- | --- |
| `DATABASE_URL` | Yes | None | Local URL or deployed secret ID. |
| `DATABASE_URL_READONLY` | No | None | Optional read-replica URL or deployed secret ID for backfills. |
| `SEARCH_EVENT_QUEUE` | Yes | None | Queue consumed for indexing work. |
| `OPENSEARCH_URL` | Yes | None | OpenSearch endpoint. |
| `OPENSEARCH_USERNAME` | Yes | None | OpenSearch username. |
| `OPENSEARCH_PASSWORD` | Yes | None | Local password or deployed secret ID. |
| `DOCUMENT_STORAGE_BUCKET` | Yes | None | Source document bucket. |
| `LEXICAL_SERVICE_URL` | Yes | None | Lexical conversion service. |
| `BACKFILL_JOBS_TABLE` | Yes | None | DynamoDB job registry table. |
| `BACKFILL_JOB_TTL_SECONDS` | No | `86400` | TTL for completed job records. |
| `WORKER_COUNT` | No | `10` | Search worker count. |
| `QUEUE_MAX_MESSAGES` | No | `10` | Queue poll batch size. |
| `QUEUE_WAIT_TIME_SECONDS` | No | `20` | Queue long-poll wait. |
| `BACKFILL_CALLS_PAGE_SIZE` | No | `2000` | Must be `> 0`. |
| `BACKFILL_CHATS_PAGE_SIZE` | No | `5000` | Must be `> 0`. |
| `BACKFILL_CHANNELS_PAGE_SIZE` | No | `5000` | Must be `> 0`. |
| `BACKFILL_DOCUMENTS_PAGE_SIZE` | No | `1000` | Must be `> 0`. |
| `BACKFILL_EMAILS_PAGE_SIZE` | No | `1000` | Must be `> 0`. |
| `PORT` | No | `8080` | HTTP listener. |
### Other HTTP services and workers
| Service | Required variables | Optional/defaults |
| --- | --- | --- |
| `connection_gateway` | `REDIS_HOST`, `MACRO_DB_URL`, `CONNECTION_GATEWAY_TABLE`, shared JWT vars, `INTERNAL_API_SECRET_KEY` | `PORT=8080` |
| `contacts_service` | `DATABASE_URL`, `REDIS_URI`, `CONTACTS_QUEUE`, shared JWT vars | `PORT=8080`, `CONTACTS_QUEUE_MAX_MESSAGES=10`, `CONTACTS_QUEUE_WAIT_TIME_SECONDS=5`, optional `CONNECTION_GATEWAY_URL` |
| `static_file_service` | `STATIC_FILE_SERVICE_DYNAMODB_TABLE_NAME`, `STATIC_STORAGE_BUCKET`, `STATIC_FILE_SERVICE_URL`, `STATIC_FILE_SERVICE_S3_EVENT_QUEUE_URL`, `INTERNAL_API_SECRET_KEY`, shared JWT vars | `PORT=8080` |
| `convert_service` | `CONVERT_QUEUE`, `LOK_PATH`, `DATABASE_URL`, `DOCUMENT_STORAGE_BUCKET`, `WEB_SOCKET_RESPONSE_LAMBDA` | `PORT=8080`, `QUEUE_MAX_MESSAGES=5`, `QUEUE_WAIT_TIME_SECONDS=5` |
| `deleted_item_poller` | `DATABASE_URL`, `DOCUMENT_DELETE_QUEUE`, `CHAT_DELETE_QUEUE`, `SEARCH_EVENT_QUEUE` | None |
| `docx_unzip_handler` | `DATABASE_URL`, `REDIS_URI`, `DOCUMENT_STORAGE_BUCKET`, `WEB_SOCKET_RESPONSE_LAMBDA`, `CONVERT_QUEUE` | None |
| `document_text_extractor` | `DATABASE_URL` | `PDFIUM_LIB_PATH` is embedded at build time. |
| `document_upload_finalizer` local worker | `DATABASE_URL`, `INTERNAL_API_SECRET_KEY`, `SYNC_SERVICE_AUTH_KEY`, `LEXICAL_SERVICE_URL`, `SYNC_SERVICE_URL`, `DOCUMENT_UPLOAD_FINALIZER_QUEUE_URL` | `LOCAL_AWS_URL=http://localstack:4566` in Compose. |
| `email_refresh_handler` | `DATABASE_URL`, `LINK_MANAGER_QUEUE`, `DELETE_UNUSED_AFTER_DAYS`, `DELETE_INACTIVE_AFTER_DAYS` | None |
| `email_scheduled_handler` | `DATABASE_URL`, `EMAIL_SCHEDULED_QUEUE` | None |
| `email_sfs_delete_handler` | `DATABASE_URL`, `SFS_DELETE_QUEUE` | None |
| `sha_cleanup_worker` | `REDIS_URI`, `DATABASE_URL`, `DOCUMENT_STORAGE_BUCKET` | None |
| `unfurl_service` | None beyond shared runtime | `PORT=8080` |
| `image_proxy_service` | None beyond shared runtime | `PORT=8080` |
## Provider-specific variables
| Provider / integration | Variables | Required by |
| --- | --- | --- |
| FusionAuth | `FUSIONAUTH_TENANT_ID`, `FUSIONAUTH_API_KEY_SECRET_KEY`, `FUSIONAUTH_CLIENT_ID`, `FUSIONAUTH_CLIENT_SECRET_KEY`, `FUSIONAUTH_BASE_URL`, `FUSIONAUTH_OAUTH_REDIRECT_URI` | Auth service; MCP service uses the same base/client/tenant secret-key pattern. |
| Google OAuth | `GOOGLE_CLIENT_ID`, `GOOGLE_CLIENT_SECRET_KEY` | Auth service and MCP auth proxy. |
| Stripe | `STRIPE_SECRET_KEY`, `STRIPE_WEBHOOK_SECRET_KEY`, legacy `STRIPE_PRICE_ID_*` vars | Auth service. |
| GitHub | `GITHUB_CLIENT_ID`, `GITHUB_CLIENT_SECRET`, `GITHUB_IDP_ID`, `GITHUB_WEBHOOK_SECRET_KEY`, `GITHUB_SYNC_APP_PEM_SECRET_KEY`, `GITHUB_SYNC_APP_CLIENT_ID`, `GITHUB_SYNC_APP_URL` | Auth and document storage. |
| OpenSearch | `OPENSEARCH_URL`, `OPENSEARCH_USERNAME`, `OPENSEARCH_PASSWORD` | Search and document services. |
| OpenAI | `OPENAI_API_KEY` | DCS completions proxy. |
| Anthropic | `ANTHROPIC_API_KEY` | Anthropic client / AI tools. |
| Slack MCP | `SLACK_MCP_CLIENT_ID`, `SLACK_MCP_CLIENT_SECRET` | Pre-registered MCP provider registry; optional because registry creation tolerates absence. |
| Apollo.io | `USE_APOLLO_CRM_ENRICHMENT`, `APOLLO_API_KEY` | Email CRM enrichment. |
| Meta | `META_PIXEL_ID`, `META_ACCESS_TOKEN`, `META_TEST_EVENT_CODE` | Auth analytics and DSS Cal-to-Meta tracking. |
| Google Analytics | `GA_MEASUREMENT_ID`, `GA_API_SECRET` | Auth analytics. |
| PostHog | `POSTHOG_API_KEY`, `POSTHOG_HOST`, frontend `VITE_POSTHOG_API_KEY` | Auth analytics and web app builds. |
| LiveKit | `LIVEKIT_SERVER_URL`, `LIVEKIT_API_KEY`, `LIVEKIT_API_SECRET`, `LIVEKIT_TRANSCRIPTION_AGENT_NAME` | DSS call service. |
| AWS SNS push | `SNS_APNS_PLATFORM_ARN`, `SNS_FCM_PLATFORM_ARN`, `SNS_APNS_VOIP_PLATFORM_ARN`, `APPLE_BUNDLE_ID` | Notification service and optional DSS VoIP push. |
| Cal.com | `CAL_WEBHOOK_SECRET_KEY`, `CAL_EVENT_TYPE_CONTENT_NAMES_KEY` | DSS Cal webhook routing. |
## Local queues, tables, and buckets
LocalStack setup creates these queue names:
| Variable | Local name |
| --- | --- |
| `NOTIFICATION_QUEUE` | `notification-queue` |
| `NOTIFICATION_INGRESS_QUEUE` | `notification-ingress-queue` |
| `PUSH_NOTIFICATION_EVENT_HANDLER_QUEUE` | `push-delivery-queue` |
| `BACKFILL_QUEUE`, `EMAIL_BACKFILL_QUEUE` | `email-service-backfill-queue` |
| `CHAT_DELETE_QUEUE` | `delete-chat-handler-queue` |
| `CONTACTS_QUEUE` | `contacts-queue` |
| `CONVERT_QUEUE` | `convert-service-queue` |
| `DOCUMENT_DELETE_QUEUE` | `delete-document-handler-queue` |
| `DOCUMENT_UPLOAD_FINALIZER_QUEUE_URL` | `document-upload-finalizer-queue` URL |
| `DOCUMENT_TEXT_EXTRACTOR_QUEUE` | `document-text-extractor-lambda-queue` |
| `EMAIL_SCHEDULED_QUEUE` | `email-service-scheduled-queue` |
| `GMAIL_INBOX_SYNC_QUEUE` | `email-service-gmail-inbox-sync-queue` |
| `GMAIL_INBOX_SYNC_RETRY_QUEUE` | `email-service-gmail-inbox-retry-queue` |
| `GMAIL_OPS_QUEUE` | `email-service-gmail-ops-queue` |
| `GMAIL_OPS_RETRY_QUEUE` | `email-service-gmail-ops-retry-queue` |
| `LINK_MANAGER_QUEUE` | `email-service-refresh-queue` |
| `SEARCH_EVENT_QUEUE` | `search-event-queue` |
| `SFS_DELETE_QUEUE` | `email-sfs-delete-queue` |
| `SFS_UPLOADER_QUEUE` | `email-service-sfs-mapper-queue` |
| `STATIC_FILE_SERVICE_S3_EVENT_QUEUE_URL` | `static-file-s3-event-notification-queue` URL |
LocalStack setup creates these tables and buckets:
| Type | Names |
| --- | --- |
| DynamoDB tables | `bulk-upload`, `connection-gateway-table`, `static-file-metadata` |
| S3 buckets | `macro-email-attachments`, `doc-storage`, `docx-upload`, `static-file-storage`, `bulk-upload-staging` |
## Frontend and JS runtime variables
| Surface | Variables | Defaults / notes |
| --- | --- | --- |
| Vite app dev server | `PORT`, `MODE`, `LOCAL_DOCKER`, `LOCAL_JWT`, `TAURI_DEV_HOST` | App Vite config defaults `PORT` to `3000`; `LOCAL_JWT` is injected as `import.meta.env.__LOCAL_JWT__`. |
| Local backend selection | `VITE_LOCAL_SERVERS`, `VITE_ENABLE_BEARER_TOKEN_AUTH` | Playwright local E2E sets `VITE_LOCAL_SERVERS=ALL` and bearer-token auth. |
| Observability | `VITE_DD_WEB_APP_ID`, `VITE_DD_WEB_APP_TOKEN`, `VITE_POSTHOG_API_KEY` | Used by web observability and analytics packages. |
| Feature flags | `VITE_<FLAG_NAME>` | Feature flag helper reads Vite env keys by flag name. |
| Playwright local E2E | `LOCAL_E2E`, `LOCAL_JWT`, `PORT`, `CI` | `LOCAL_E2E=true` switches tests to the local stack. |
| `lexical_service` | `PORT`, `INTERNAL_AUTH_KEY` or `INTERNAL_API_SECRET_KEY`, `SYNC_SERVICE_AUTH_KEY`, `SYNC_SERVICE_URL` | Compose sets `PORT=8096`, `INTERNAL_AUTH_KEY=${INTERNAL_API_SECRET_KEY}`, and `SYNC_SERVICE_URL=http://sync-service:8787`. |
## Troubleshooting startup failures
| Symptom | Check |
| --- | --- |
| Service exits with `... must be provided` | The named required variable is missing from `.env`, Compose overrides, or deployment config. |
| Service panics while parsing a number | Verify numeric variables such as `PORT`, queue worker counts, TTLs, and page sizes contain only valid positive integers where required. |
| Local service attempts deployed AWS | Set `LOCAL_AWS_URL=http://localstack:4566` and local AWS test credentials. |
| Deployed service treats a secret value as a name | For variables resolved through Secrets Manager in `dev`/`prod`, set the env var to the secret ID, not the plaintext secret. |
| JWT-protected routes fail | Verify `AUDIENCE`, `ISSUER`, `JWT_SECRET_KEY`, `MACRO_API_TOKEN_ISSUER`, and `MACRO_API_TOKEN_PUBLIC_KEY` are aligned across auth-producing and auth-consuming services. |
| Internal service calls return `401` | Verify caller and callee share `INTERNAL_API_SECRET_KEY`; requests must send `x-internal-auth-key`. |
| Local E2E refuses to seed | The seed path requires `LOCAL_E2E_SEED=true` and a local Docker database URL. |
| VoIP push is unexpectedly disabled | DSS requires both `APPLE_BUNDLE_ID` and non-empty `SNS_APNS_VOIP_PLATFORM_ARN`; notification service allows missing VoIP ARN only in local mode. |
## Related pages
- Running locally
- Service architecture
- Local E2E smoke tests
---
## 23. Database migrations and SQLx cache
> Operations: MacroDB migrations, local database just recipes, SQLx offline cache, fixture data, and migration validation workflow.
- Page Markdown: https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/23-database-migrations-and-sqlx-cache.md
- Generated: 2026-06-01T01:02:18.636Z
### Source Files
- `rust/cloud-storage/macro_db_client/README.md`
- `rust/cloud-storage/macro_db_client/migrations/0001_baseline.sql`
- `rust/cloud-storage/database.just`
- `rust/cloud-storage/sqlx.just`
- `rust/cloud-storage/justfile`
- `rust/cloud-storage/chat/.sqlx/query-57fc603adf83fd7c07968425c98f9a57924573692cf0529ad021b6eef00ce99e.json`
- `.github/workflows/migrate-macro-db.yml`
---
title: "Database migrations and SQLx cache"
description: "Operations: MacroDB migrations, local database just recipes, SQLx offline cache, fixture data, and migration validation workflow."
---
MacroDB migrations live in `rust/cloud-storage/macro_db_client/migrations`, are executed with SQLx CLI recipes, and feed both local PostgreSQL setup and compile-time SQLx query metadata used by the Rust cloud-storage workspace.
## Migration ownership
| Surface | Path | Purpose |
| --- | --- | --- |
| Migration files | `rust/cloud-storage/macro_db_client/migrations/` | SQLx migration history for MacroDB. |
| Migration runner recipes | `rust/cloud-storage/macro_db_client/justfile` | Database create, migrate, reset, check, and remote dev/prod helpers. |
| Shared SQLx recipes | `rust/cloud-storage/sqlx.just` | Low-level `sqlx database`, `sqlx migrate`, and `cargo sqlx prepare` commands. |
| Local database URL | `rust/cloud-storage/database.just` | Default local MacroDB URL: `postgres://user:password@localhost:5432/macrodb`. |
| Workspace recipes | `rust/cloud-storage/justfile` | `setup_macrodb`, `initialize_dbs`, `prepare_db`, checks, and test `.env` setup. |
| Test migrator crate | `rust/cloud-storage/macro_db_migrator` | Lightweight static import of MacroDB migrations for crates that need migrations in tests without depending on `macro_db_client`. |
| GitHub migration action | `.github/actions/migrate-cloud-storage-db/action.yml` | CI/CD migration runner for dev and prod databases. |
## Baseline migration behavior
`0001_baseline.sql` is a guarded baseline from the Prisma-to-SQLx migration. It counts existing public tables except `_sqlx_migrations` and only creates the baseline schema when the database is empty.
<Warning>
Do not treat the baseline as a normal production bootstrap script. Its header states that dev and prod already had the declared objects when the project moved to SQLx. The guard prevents it from applying over an existing schema, but new migrations should still be additive SQLx migrations after the baseline.
</Warning>
Typical later migrations are timestamped files such as:
```text
20260511131821_enable_pgvector.sql
20260416135258_scheduled_agent.up.sql
20260416135258_scheduled_agent.down.sql
20260529164720_crm_domain_directory_apollo_fields.up.sql
20260529164720_crm_domain_directory_apollo_fields.down.sql
```
The migrations directory contains both single-file migrations and paired `.up.sql` / `.down.sql` migrations where rollback SQL is maintained.
## Local database recipes
The local MacroDB URL is imported from `database.just`:
```bash
postgres://user:password@localhost:5432/macrodb
```
Start local database services from the repository root:
```bash
just run_dbs -d
```
Initialize MacroDB from the repository root:
```bash
just rust/cloud-storage/initialize_dbs
```
Equivalent cloud-storage workflow:
```bash
cd rust/cloud-storage
just setup_macrodb
```
`setup_macrodb` runs:
```text
just macro_db_client/create_db
just macro_db_client/migrate_db
```
Root `setup_local_dbs` starts the database containers, creates and migrates MacroDB, prints `Local databases initialized`, then stops the database compose file.
### MacroDB just commands
Run these from `rust/cloud-storage/macro_db_client` unless invoking through the root justfile path.
| Command | Effect |
| --- | --- |
| `just create_db` | Runs `sqlx database create` against the local `DATABASE_URL`. |
| `just migrate_db` | Runs `sqlx migrate run` against the local `DATABASE_URL`. |
| `just check_db` through `sqlx.just` | Runs `sqlx migrate info`. |
| `just setup` | Runs `sqlx database setup`. |
| `just reset_db` | Drops and sets up the database. |
| `just force_drop_db` | Pipes `y` into the drop command for local force drops. |
| `just prepare_db` | Runs SQLx workspace prepare for offline query metadata. |
| `just prepare_db_dev` | Prepares query metadata against the dev RDS URL returned by `../scripts/rds_database_url.sh`. |
| `just check_db_dev` / `just check_db_prod` | Checks pending migrations against remote dev/prod. |
| `just migrate_db_dev` | Runs migrations against remote dev. |
| `just migrate_db_prod` | Prompts before running migrations against remote prod. |
## SQLx offline cache
The workspace uses SQLx compile-time query checking. Prepared query metadata is committed as JSON files under `.sqlx` directories, for example:
```json
{
"db_name": "PostgreSQL",
"query": "INSERT INTO \"ChatMessage\" ... RETURNING id",
"describe": {
"columns": [{ "ordinal": 0, "name": "id", "type_info": "Text" }],
"parameters": { "Left": ["Text", "Text", "Jsonb", "Text", "Text", "Timestamp", "Timestamp"] },
"nullable": [false]
},
"hash": "57fc603adf83fd7c07968425c98f9a57924573692cf0529ad021b6eef00ce99e"
}
```
Refresh the cache after adding or changing SQLx queries:
```bash
cd rust/cloud-storage
just prepare_db
```
The shared recipe expands to:
```bash
cargo sqlx prepare --workspace -- --all-features
```
Additional flags can be passed through the shared `sqlx.just` recipe. The cloud-storage `check`, `clippy`, `format`, and `build` recipes set `SQLX_OFFLINE=true`, so stale or missing `.sqlx` metadata can surface as compile-time failures before runtime.
<Note>
`macro_db_client/README.md` documents the expected test workflow: run local MacroDB in Docker, run `just setup_macrodb` from `cloud-storage`, then run `just prepare_db` before `cargo test` for code that depends on prepared SQLx metadata.
</Note>
## Test migrations and fixtures
Most database tests use `#[sqlx::test]` and fixture SQL files. Fixtures are stored near the crate that owns the behavior, for example:
```text
rust/cloud-storage/macro_db_client/fixtures/basic_user_with_document.sql
rust/cloud-storage/name_search/fixtures/chat.sql
rust/cloud-storage/channels/fixtures/channels_repo.sql
```
`macro_db_client` tests can use the crate-local migration directory directly. Other crates import migrations through `macro_db_migrator`:
```rust
pub static MACRO_DB_MIGRATIONS: sqlx::migrate::Migrator =
sqlx::migrate!("../macro_db_client/migrations");
```
A typical cross-crate test shape is:
```rust
#[sqlx::test(
migrator = "MACRO_DB_MIGRATIONS",
fixtures(path = "../../fixtures", scripts("chat"))
)]
async fn test_search_chat_names_empty_term(pool: Pool<Postgres>) -> anyhow::Result<()> {
// test body
}
```
The cloud-storage CI test job prepares test `.env` files and initializes MacroDB before running `cargo nextest`:
```bash
just rust/cloud-storage/setup_test_envs
just rust/cloud-storage/initialize_dbs
cd rust/cloud-storage
cargo nextest run --all-features --lib --bins --tests
```
`setup_test_envs` appends the same `DATABASE_URL` to many crate-local `.env` files so SQLx tests and service tests resolve MacroDB consistently.
## Fixture and seed data
There are two fixture paths with different purposes:
| Data type | Location | Used by |
| --- | --- | --- |
| SQL test fixtures | Per-crate `fixtures/*.sql` directories | `#[sqlx::test(fixtures(...))]` database tests. |
| Local E2E seed data | `rust/cloud-storage/seed_cli/seed/` | Deterministic local Playwright and Rust local E2E smoke data. |
The root `local-e2e-seed` recipe resets MacroDB and applies deterministic seed data:
```bash
just local-e2e-seed
```
It runs local databases, drops MacroDB, initializes migrations, then calls:
```bash
just rust/cloud-storage/seed_cli/local-e2e-smoke
```
The `local-e2e-smoke` seed command sets `LOCAL_E2E_SEED=true`, the local MacroDB URL, local AWS/FusionAuth settings, and `SQLX_OFFLINE=true`, then executes:
```bash
cargo r -- scenario local-e2e-smoke
```
The seed scenario validates that it is running against a local compose-style MacroDB URL before applying destructive reset SQL. Accepted hosts are `localhost`, `127.0.0.1`, `::1`, or `postgres`, with user `user`, database `macrodb`, and port `5432`.
## Remote migration workflow
Manual migration dispatch is defined in `.github/workflows/migrate-macro-db.yml` with an `environment` input of `dev` or `prod`.
The composite action:
1. Checks out `.github/` and `rust/cloud-storage/macro_db_client`.
2. Reads either `macro-db-dev` or `macro-db-prod` from AWS Secrets Manager.
3. Adds `sslmode=require` to the database URL when missing.
4. Masks the database URL in GitHub logs.
5. Runs SQLx migrations from `rust/cloud-storage/macro_db_client`.
Dev migrations run with:
```bash
cargo sqlx migrate run --ignore-missing
```
Prod migrations run with:
```bash
cargo sqlx migrate run
```
The main-branch cloud-storage deploy workflow runs the same migration action for `dev` before service deployment. The production release workflow runs the same action for `prod` before deploying cloud-storage services.
## Migration validation checklist
<Steps>
<Step title="Start or select the target database">
For local work, start Docker databases with `just run_dbs -d`. For remote checks, use the `check_db_dev` or `check_db_prod` recipes from `rust/cloud-storage/macro_db_client`.
</Step>
<Step title="Apply migrations locally">
Run `cd rust/cloud-storage && just setup_macrodb` for a fresh local database, or `cd rust/cloud-storage/macro_db_client && just migrate_db` when the database already exists.
</Step>
<Step title="Check migration status">
Run `just sqlx::check_db postgres://user:password@localhost:5432/macrodb` from `rust/cloud-storage`, or the crate-specific remote check recipes for dev/prod.
</Step>
<Step title="Refresh SQLx metadata">
Run `cd rust/cloud-storage && just prepare_db` after schema or query changes. Commit the changed `.sqlx/query-*.json` files with the migration and Rust query changes.
</Step>
<Step title="Run compile and test checks">
Use `just check`, `just clippy`, and the relevant `cargo test` or `cargo nextest` command. Database-backed tests require local MacroDB and the crate `.env` setup when they read `DATABASE_URL`.
</Step>
</Steps>
## Troubleshooting
| Symptom | Likely cause | Fix |
| --- | --- | --- |
| SQLx macro compile error for a changed query | Offline cache is missing or stale. | Run `cd rust/cloud-storage && just prepare_db` against an up-to-date MacroDB. |
| Migration appears pending locally after pulling changes | Local database has not applied new files in `macro_db_client/migrations`. | Run `cd rust/cloud-storage/macro_db_client && just migrate_db`. |
| Test crate cannot find `DATABASE_URL` | Crate-local `.env` was not populated. | Run `just rust/cloud-storage/setup_test_envs` from the repository root. |
| Local E2E seed refuses to run | Safety guard did not detect `LOCAL_E2E_SEED=true` or the URL is not the local compose MacroDB. | Use `just local-e2e-seed` or match `postgres://user:password@localhost:5432/macrodb`. |
| Remote prod migration needs confirmation locally | `migrate_db_prod` is intentionally interactive. | Use it from a terminal and confirm only after reviewing pending migrations with `check_db_prod`. |
| GitHub dev migration ignores missing migrations but prod does not | The composite action passes `--ignore-missing` only for `dev`. | Keep prod migration history complete and avoid deleting applied prod migration files. |
## Related pages
<CardGroup>
<Card title="Cloud-storage Rust workspace" href="/rust-cloud-storage">
Build, check, clippy, test, and service-level Rust workflows.
</Card>
<Card title="Local development stack" href="/local-development">
Docker compose databases, local services, and seeded E2E workflows.
</Card>
</CardGroup>
---
## 24. Feature flags and server selection
> Reference: Vite environment overrides, feature flag defaults, local service selection syntax, sync-service host selection, and runtime host helpers.
- Page Markdown: https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/24-feature-flags-and-server-selection.md
- Generated: 2026-06-01T01:03:44.927Z
### Source Files
- `js/app/packages/core/constant/featureFlags.ts`
- `js/app/packages/core/tests/featureFlags.test.ts`
- `js/app/packages/core/constant/servers.ts`
- `js/app/justfile`
- `js/app/packages/app/vite.config.ts`
- `js/app/package.json`
---
title: "Feature flags and server selection"
description: "Reference: Vite environment overrides, feature flag defaults, local service selection syntax, sync-service host selection, and runtime host helpers."
---
The frontend resolves feature gates and backend hosts at module load from `import.meta.env`, with Vite exposing `VITE_*` variables and `packages/app/vite.base.ts` injecting app metadata such as `__APP_VERSION__`, `ASSETS_PATH`, `__LOCAL_JWT__`, and `__GIT_BRANCH__`.
## Vite mode and environment inputs
`packages/app/vite.config.ts` delegates to `createAppViteConfig()`. The base config sets Vite `mode` from `process.env.MODE ?? mode`, so the shell `MODE` value wins over Vite's default mode.
| Input | Used by | Behavior |
| --- | --- | --- |
| `MODE=development` | feature flags, remote host suffixes, sync-service suffixes, assets path | Enables `DEV_MODE_ENV`, uses `-dev` service hosts, uses sync-service `-dev3`, and allows `VITE_LOCAL_SERVERS` selection. |
| `MODE=production` | feature flags, remote host suffixes, sync-service suffixes | Enables `PROD_MODE_ENV`, uses production service hosts, and uses sync-service `-prod2`. |
| `MODE=staging` | Vite assets path | Uses `/staging` assets, but `servers.ts` treats it as non-development for host selection. |
| `VITE_*` | Vite runtime env | Automatically exposed by Vite and read by feature flag and server selectors. |
| `LOCAL_JWT` | `defineEnv()` | Injected as `import.meta.env.__LOCAL_JWT__` for local bearer-token auth paths. |
| `PORT` | Vite dev/preview server | Defaults to `3000`; used by `just local` and preview commands. |
Common commands:
```bash
cd js/app
bun run dev
```
```bash
cd js/app
MODE=production NODE_ENV=production bun run build
```
```bash
cd js/app
just build-dev
just build-staging
just build-prod
```
## Feature flag override rules
Feature flags are centralized in `packages/core/constant/featureFlags.ts`.
```ts
resolveFeatureFlag(flagName: string, defaultValue: boolean): boolean
```
For a flag named `ENABLE_EMAIL`, the runtime override key is `VITE_ENABLE_EMAIL`.
| Override value | Result |
| --- | --- |
| `true` | Forces the flag on. |
| `false` | Forces the flag off. |
| unset | Uses the code default. |
| any other value, such as `1` or `yes` | Ignored; uses the code default. |
<Note>
Feature flag values are string comparisons. Use `VITE_ENABLE_EMAIL=true`, not `VITE_ENABLE_EMAIL=1`.
</Note>
Example:
```bash
cd js/app/packages/app
VITE_ENABLE_BEARER_TOKEN_AUTH=true bun run --bun dev
```
The resolver behavior is covered by `packages/core/tests/featureFlags.test.ts`, including default fallback, `true`, `false`, and invalid values.
## Feature flag defaults
### Runtime mode flags
| Export | Value |
| --- | --- |
| `LOCAL_ONLY` | `true` when `import.meta.hot` is present, normally Vite dev with hot reload. |
| `DEV_MODE_ENV` | `true` when `import.meta.env.MODE === 'development'`. |
| `PROD_MODE_ENV` | `true` when `import.meta.env.MODE === 'production'`. |
### Boolean flags resolved through `VITE_<flag>`
| Export | Override key | Default |
| --- | --- | --- |
| `ENABLE_PDF_MODIFICATION_DATA_AUTOSAVE` | `VITE_ENABLE_PDF_MODIFICATION_DATA_AUTOSAVE` | `true` |
| `ENABLE_PDF_LOCATION_AUTOSAVE` | `VITE_ENABLE_PDF_LOCATION_AUTOSAVE` | `true` |
| `ENABLE_PDF_TABS` | `VITE_ENABLE_PDF_TABS` | `true` |
| `ENABLE_PDF_MARKUP` | `VITE_ENABLE_PDF_MARKUP` | `true` |
| `ENABLE_SCRIPTING` | `VITE_ENABLE_SCRIPTING` | `false` |
| `ENABLE_PDF_MULTISPLIT` | `VITE_ENABLE_PDF_MULTISPLIT` | `true` |
| `ENABLE_PROJECT_SHARING` | `VITE_ENABLE_PROJECT_SHARING` | `true` |
| `ENABLE_CANVAS_IMAGES` | `VITE_ENABLE_CANVAS_IMAGES` | `true` |
| `ENABLE_CANVAS_FILES` | `VITE_ENABLE_CANVAS_FILES` | `true` |
| `ENABLE_CANVAS_TEXT` | `VITE_ENABLE_CANVAS_TEXT` | `true` |
| `ENABLE_LIVE_INDICATORS` | `VITE_ENABLE_LIVE_INDICATORS` | `true` |
| `ENABLE_PROFILE_PICTURES` | `VITE_ENABLE_PROFILE_PICTURES` | `true` |
| `ENABLE_VIDEO_BLOCK` | `VITE_ENABLE_VIDEO_BLOCK` | `true` |
| `ENABLE_DOCX_TO_PDF` | `VITE_ENABLE_DOCX_TO_PDF` | `true` |
| `ENABLE_MARKDOWN_LIVE_COLLABORATION` | `VITE_ENABLE_MARKDOWN_LIVE_COLLABORATION` | `true` |
| `ENABLE_EMAIL` | `VITE_ENABLE_EMAIL` | `true` |
| `ENABLE_BLOCK_IN_BLOCK` | `VITE_ENABLE_BLOCK_IN_BLOCK` | `true` |
| `ENABLE_SEARCH_SERVICE` | `VITE_ENABLE_SEARCH_SERVICE` | `true` |
| `ENABLE_MARKDOWN_DIFF` | `VITE_ENABLE_MARKDOWN_DIFF` | `true` |
| `ENABLE_HISTORY_COMPONENT` | `VITE_ENABLE_HISTORY_COMPONENT` | `false` |
| `ENABLE_BEARER_TOKEN_AUTH` | `VITE_ENABLE_BEARER_TOKEN_AUTH` | `false` |
| `ENABLE_MARKDOWN_SEARCH_TEXT` | `VITE_ENABLE_MARKDOWN_SEARCH_TEXT` | `DEV_MODE_ENV` |
| `CANVAS_SVG_IMPORT` | `VITE_CANVAS_SVG_IMPORT` | `true` |
| `ENABLE_CANVAS_VIDEO` | `VITE_ENABLE_CANVAS_VIDEO` | `true` |
| `ENABLE_CANVAS_HEIC` | `VITE_ENABLE_CANVAS_HEIC` | `false` |
| `ENABLE_MARKDOWN_COMMENTS` | `VITE_ENABLE_MARKDOWN_COMMENTS` | `true` |
| `ENABLE_REFERENCES_MODAL` | `VITE_ENABLE_REFERENCES_MODAL` | `true` |
| `ENABLE_MENTION_TRACKING` | `VITE_ENABLE_MENTION_TRACKING` | `true` |
| `ENABLE_CHAT_CHANNEL_ATTACHMENT` | `VITE_ENABLE_CHAT_CHANNEL_ATTACHMENT` | `true` |
| `ENABLE_SVG_PREVIEW` | `VITE_ENABLE_SVG_PREVIEW` | `true` |
| `USE_WIDE_ICONS` | `VITE_USE_WIDE_ICONS` | `true` |
| `ENABLE_ANIMATED_ICONS` | `VITE_ENABLE_ANIMATED_ICONS` | `true` |
| `ENABLE_PREVIEW` | `VITE_ENABLE_PREVIEW` | `true` |
| `ENABLE_PROJECT_VIEW_PREVIEW` | `VITE_ENABLE_PROJECT_VIEW_PREVIEW` | `true` |
| `ENABLE_TTFT` | `VITE_ENABLE_TTFT` | `DEV_MODE_ENV` |
| `ENABLE_MULTI_INBOX` | `VITE_ENABLE_MULTI_INBOX` | `DEV_MODE_ENV` |
| `ENABLE_EMAIL_SHARING` | `VITE_ENABLE_EMAIL_SHARING` | `true` |
| `ENABLE_DOCUMENT_MENTION_NOTIFICATIONS` | `VITE_ENABLE_DOCUMENT_MENTION_NOTIFICATIONS` | `DEV_MODE_ENV` |
| `ENABLE_STATIC_DOCUMENT_CARDS` | `VITE_ENABLE_STATIC_DOCUMENT_CARDS` | `false` |
| `ENABLE_MARKDOWN_AI_GENERATE` | `VITE_ENABLE_MARKDOWN_AI_GENERATE` | `false` |
| `ENABLE_UNIFIED_LIST_AI_INPUT` | `VITE_ENABLE_UNIFIED_LIST_AI_INPUT` | `true` |
| `ENABLE_EMAIL_SCHEDULED_SEND` | `VITE_ENABLE_EMAIL_SCHEDULED_SEND` | `true` |
| `ENABLE_FEATURED_SEARCH_RESULTS` | `VITE_ENABLE_FEATURED_SEARCH_RESULTS` | `true` |
| `ENABLE_PROXY_EMAIL_IMAGES` | `VITE_ENABLE_PROXY_EMAIL_IMAGES` | `true` |
| `ENABLE_CLIENT_EMAIL_SIGNAL_FILTER` | `VITE_ENABLE_CLIENT_EMAIL_SIGNAL_FILTER` | `false` |
| `ENABLE_APP_STORE_QR_CODE` | `VITE_ENABLE_APP_STORE_QR_CODE` | `true` |
| `ENABLE_RAIL_CHAT_TASK_COMMENTS` | `VITE_RAIL_CHAT_TASK_COMMENTS` | `true` |
| `ENABLE_AUTO_UPDATE_UI` | `VITE_ENABLE_AUTO_UPDATE_UI` | `true` |
| `ENABLE_CALLKIT` | `VITE_ENABLE_CALLKIT` | `false` |
| `ENABLE_MARKDOWN_SIDE_PANEL` | `VITE_ENABLE_MARKDOWN_SIDE_PANEL` | `true` |
| `ENABLE_REFOCUS_HIGHLIGHT` | `VITE_ENABLE_REFOCUS_HIGHLIGHT` | `true` |
| `ENABLE_CREATE_PROPERTY` | `VITE_ENABLE_CREATE_PROPERTY` | `false` |
### PostHog and override-style flags
| Export | Behavior |
| --- | --- |
| `ENABLE_TEAMS_OVERRIDE` | `true` in development or when `VITE_ENABLE_TEAMS=true`; otherwise `undefined`. A `false` override becomes `undefined`, not exported `false`. |
| `ENABLE_CALLS()` | Returns `true` in development; otherwise falls back to PostHog flag `enable-calls`. |
| `ENABLE_NEW_ONBOARDING_OVERRIDE` | `true` in development; otherwise `undefined`. |
| `ENABLE_NEW_LOGIN_OVERRIDE` | `true` in development; otherwise `undefined`. |
| `ENABLE_INVITE_TEAM_ONBOARDING_OVERRIDE` | `true` in development; otherwise `undefined`. |
| `ENABLE_TEAM_INVITE_TIERS_OVERRIDE` | `true` in development; otherwise `undefined`. |
| `ENABLE_SOUP_GROUP_BY_OVERRIDE` | `true` in development; otherwise `undefined`. |
| `ENABLE_NEW_PRICING_OVERRIDE` | `true` in development or when `VITE_ENABLE_NEW_PRICING=true`; otherwise `undefined`. |
## Server host selection
`packages/core/constant/servers.ts` exports:
```ts
export const SERVER_HOSTS: Servers
export const SYNC_SERVICE_HOSTS
export const SYNC_PERMISSION_TOKEN_DSS_HOST
```
`SERVER_HOSTS` is remote by default. Local host selection is only evaluated when `import.meta.env.MODE === 'development'`.
| Mode | `VITE_LOCAL_SERVERS` | `SERVER_HOSTS` result |
| --- | --- | --- |
| non-development | any value | Remote production-style hosts. |
| development | unset or empty | Remote dev hosts. |
| development | `ALL` | All regular services use local hosts. |
| development | comma-separated service names | Only listed services use local hosts; all others remain remote dev hosts. |
| development | `service-name:port` | Listed service uses its local URL with only the port replaced. |
Examples:
```bash
cd js/app
just local
```
```bash
cd js/app
just servers='document-storage-service' local
```
```bash
cd js/app
just servers='document-storage-service:9001,email-service' port='3001' local
```
```bash
cd js/app/packages/app
VITE_LOCAL_SERVERS='document-storage-service:9001' bun run --bun dev
```
When a partial local selection is used, the selector logs entries like:
```text
Using local server document-storage-service: http://localhost:8086
```
<Warning>
Unknown service names throw at startup with `unknown server name <name>`. The current `just local-dcs` and `just local-search` recipes include names that are not present in `SERVER_HOSTS`; use `just local-services` or the table below before adding a partial selector.
</Warning>
## Regular service host reference
In development remote mode, most remote hosts use a `-dev` suffix. In non-development modes, the suffix is empty.
| Key | Local host | Remote host pattern |
| --- | --- | --- |
| `auth-service` | `http://localhost:8080` | `https://auth-service[-dev].macro.com` |
| `auth-logout` | `http://localhost:3000` | FusionAuth logout URL for dev or prod auth tenant |
| `pdf-service` | `http://localhost:4567` | `https://pdf-service[-dev].macro.com` |
| `document-storage-service` | `http://localhost:8086` | `https://cloud-storage[-dev].macro.com` |
| `websocket-service` | `ws://localhost:6969` | `wss://services[-dev].macro.com` |
| `cognition-service` | `http://localhost:8085` | `https://document-cognition[-dev].macro.com` |
| `connection-gateway` | `ws://localhost:8082` | `wss://connection-gateway[-dev].macro.com` |
| `notification-service` | `http://localhost:8089` | `https://notifications[-dev].macro.com` |
| `static-file` | `http://localhost:8100` | `https://static-file-service[-dev].macro.com` |
| `unfurl-service` | `http://localhost:8095` | `https://unfurl-service[-dev].macro.com` |
| `contacts` | `http://localhost:8083` | `https://contacts[-dev].macro.com` |
| `email-service` | `http://localhost:8087` | `https://email-service[-dev].macro.com` |
| `image-proxy-service` | `http://localhost:8097` | `https://image-proxy[-dev].macro.com` |
| `scheduled-action` | `http://localhost:8098` | `https://agent-schedule[-dev].macro.com` |
## Sync-service host selection
`SYNC_SERVICE_HOSTS` is separate from `SERVER_HOSTS`.
| Mode and selection | Worker URL | WebSocket URL |
| --- | --- | --- |
| non-development | `https://sync-service-prod2.macroverse.workers.dev` | `wss://sync-service-prod2.macroverse.workers.dev` |
| development, remote sync | `https://sync-service-dev3.macroverse.workers.dev` | `wss://sync-service-dev3.macroverse.workers.dev` |
| development, local sync | `http://localhost:8787` | `ws://localhost:8787` |
The sync selector chooses local sync when `VITE_LOCAL_SERVERS === 'ALL'` or when the string contains `sync-service`.
<Warning>
`sync-service` is not a regular `SERVER_HOSTS` key. Because `SERVER_HOSTS` validates every comma-separated entry before sync host selection completes, `VITE_LOCAL_SERVERS=sync-service` currently throws `unknown server name sync-service`. Use `VITE_LOCAL_SERVERS=ALL` for local sync-service in the current implementation, or update the regular server selector before relying on partial sync-only selection.
</Warning>
`SYNC_PERMISSION_TOKEN_DSS_HOST` follows the sync-service host:
| Sync-service selected | Permission token DSS host |
| --- | --- |
| Remote sync-service | Remote `document-storage-service`, so JWT secrets match the remote sync-service. |
| Local sync-service | Current `SERVER_HOSTS['document-storage-service']`, which may be local when `ALL` is selected. |
## Runtime host helpers
`servers.ts` also exposes static-file URL helpers:
```ts
staticFileIdEndpoint(id: string): string
staticFileSizedEndpoint(id: string, size: 'small' | 'medium'): string
staticFileSizedUrl(url: string, size: 'small' | 'medium'): string
```
| Helper | Output shape |
| --- | --- |
| `staticFileIdEndpoint('abc')` | `${SERVER_HOSTS['static-file']}/file/abc` |
| `staticFileSizedEndpoint('abc', 'small')` | `${SERVER_HOSTS['static-file']}/file/abc?size=320` |
| `staticFileSizedEndpoint('abc', 'medium')` | `${SERVER_HOSTS['static-file']}/file/abc?size=1080` |
| `staticFileSizedUrl(url, 'small')` | `${url}?size=320` |
`staticFileSizedUrl()` appends `?size=` directly. Pass a base URL without an existing query string.
## Verification
```bash
cd js/app
bunx --bun vitest packages/core/tests/featureFlags.test.ts
```
```bash
cd js/app
just local-services
```
```bash
cd js/app
VITE_LOCAL_SERVERS=ALL VITE_ENABLE_BEARER_TOKEN_AUTH=true LOCAL_JWT='<token>' bun run dev
```
Local E2E startup uses the same pattern: `VITE_LOCAL_SERVERS=ALL`, `VITE_ENABLE_BEARER_TOKEN_AUTH=true`, and `LOCAL_JWT` injected into Vite as `__LOCAL_JWT__`.
## Next
<CardGroup>
<Card title="Running locally" href="/running-locally">
Local backend stack setup, seed data, and local E2E smoke-test workflow.
</Card>
<Card title="Service clients" href="/service-clients">
How frontend packages consume `SERVER_HOSTS` through typed service clients.
</Card>
</CardGroup>
---
## 25. Infrastructure stacks reference
> Reference: Pulumi stack layout, reusable resources, service stacks, buckets, queues, Datadog sidecars, FusionAuth, and deployment inputs.
- Page Markdown: https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/25-infrastructure-stacks-reference.md
- Generated: 2026-06-01T01:05:52.608Z
### Source Files
- `infra/README.md`
- `infra/packages/resources/src/index.ts`
- `infra/packages/lambda/src/index.ts`
- `infra/packages/service/src/index.ts`
- `infra/stacks/document-storage/index.ts`
- `infra/stacks/mcp-server/index.ts`
- `infra/stacks/fusionauth-instance/README.md`
---
title: "Infrastructure stacks reference"
description: "Reference: Pulumi stack layout, reusable resources, service stacks, buckets, queues, Datadog sidecars, FusionAuth, and deployment inputs."
---
Macro infrastructure is a Bun/TypeScript Pulumi monorepo under `infra/`, with deployable projects in `infra/stacks/*` and reusable components in `infra/packages/*`. AWS resources target `us-east-1`, and each stack is deployed from its own stack directory with Pulumi.
## Stack layout
```text
infra/
package.json # Bun workspaces: stacks/* and packages/*
packages/
resources/ # Shared AWS components: buckets, queues, ALB, RDS, Redis, Datadog
lambda/ # Rust Lambda packaging and worker trigger helpers
service/ # ECR image builder component
shared/ # stack/config constants, service URLs, shared StackReference helpers
vpc/ # legacy Coparse API VPC/subnet IDs
stacks/
document-storage/ # document S3 bucket, replication bucket, shared ECS cluster
cloud-storage-service/ # application service and docx upload bucket
mcp-server/ # public ECS/Fargate MCP server
fusion-auth/ # FusionAuth runtime service and RDS database
fusionauth-instance/ # FusionAuth tenant/application configuration
...
```
Run Pulumi from a stack directory:
```bash
cd infra/stacks/mcp-server
pulumi up --stack dev
pulumi up --stack prod
```
<Info>
The root `infra/package.json` enforces Bun as the package manager and exposes `bun run check`, `bun run lint`, and `bun run format` for TypeScript validation.
</Info>
## Shared packages
| Package | Main exports | Purpose |
| --- | --- | --- |
| `packages/resources` | `createBucket`, `createBucketV2`, `Queue`, `QueueAlarms`, `serviceLoadBalancer`, `DynamoDBTable`, `Database`, `Redis`, Datadog helpers | Reusable AWS building blocks for stacks |
| `packages/lambda` | `Lambda`, `WorkerTrigger`, `generateContentHash`, `SourceCodeHash` | Rust Lambda deployment and ECS worker trigger helpers |
| `packages/service` | `EcrImage` | ECR repository plus Docker image build/push |
| `packages/shared` | `stack`, `config`, constants, service URL helpers, stack-reference helpers | Shared Pulumi config, hard-coded account constants, cross-stack outputs |
| `packages/vpc` | `get_coparse_api_vpc`, `COPARSE_API_VPC` | Existing VPC and subnet IDs used by services |
## Cross-stack dependency model
```mermaid
flowchart LR
subgraph Foundation
DS["document-storage stack\nS3 bucket + ECS cluster"]
VPC["packages/vpc\nget_coparse_api_vpc()"]
end
subgraph SharedOutputs
CSS["cloud-storage-service\nDOCX bucket + queues"]
Email["email-service\nscheduled queue"]
Link["link-sharing\nCloudFront signer outputs"]
end
subgraph ServiceStacks
MCP["mcp-server\nMcpServer component"]
Cog["document-cognition-service\nDocumentCognitionService component"]
Contacts["contacts-service\nContactsService component"]
end
DS --> MCP
DS --> Cog
DS --> Contacts
VPC --> MCP
VPC --> Cog
VPC --> Contacts
CSS --> MCP
Email --> MCP
Link --> MCP
CSS --> Cog
Email --> Cog
Link --> Cog
```
`document-storage` is both a storage stack and a foundational service stack: it exports the document bucket and the shared ECS cluster outputs consumed by ECS service stacks.
## Buckets
### `createBucket`
`createBucket` creates an `aws.s3.Bucket` with common Macro defaults:
| Input | Behavior |
| --- | --- |
| `id` | Pulumi resource ID |
| `bucketName` | Physical S3 bucket name |
| `transferAcceleration` | Enables S3 acceleration when true |
| `enableVersioning` | Enables bucket versioning with MFA delete disabled |
| `lifecycleRules` | Passed through to S3 bucket lifecycle rules |
| `exposeHeaders` | Added to default CORS exposed headers |
| `tags` | Applied to the bucket |
Default behavior:
- `forceDestroy` is enabled outside `prod`.
- Production buckets log to `macro-logging-bucket` with prefix `${bucketName}/`.
- CORS allows `GET`, `PUT`, `POST`, `DELETE`, and `HEAD`.
- CORS allows all headers and exposes `ETag` plus any caller-provided headers.
- Allowed origins come from `packages/resources/src/resources/cors.ts`.
### `createBucketV2`
`createBucketV2` uses `aws.s3.BucketV2` and creates separate resources for:
- CORS configuration
- lifecycle configuration
- versioning
- transfer acceleration
- production logging
Use this helper when a stack needs the newer S3 V2 resource model or finer-grained Pulumi dependencies.
### `document-storage` bucket
The `document-storage` stack creates the primary document bucket:
| Stack | Bucket name |
| --- | --- |
| `prod` | `macro-document-storage-prod` |
| non-`prod` | `doc-storage-${stack}` |
Configured behavior:
- Versioning enabled.
- Transfer acceleration enabled only in `prod`.
- `temp_files/` objects expire after 1 day.
- expired delete markers are removed.
- noncurrent versions expire after 30 days.
- EventBridge bucket notifications are enabled.
- CORS exposes `Content-Length` and `Content-Range`.
The stack exports:
| Output | Value |
| --- | --- |
| `documentStorageBucketId` | bucket ID/name |
| `documentStorageBucketArn` | bucket ARN |
| `documentStorageBucketName` | same value as bucket ID |
| `documentStorageBucketReplicationRoleArn` | replication IAM role ARN |
| `cloudStorageClusterName` | shared ECS cluster name |
| `cloudStorageClusterArn` | shared ECS cluster ARN |
### Replication bucket
`document-storage/replication-bucket.ts` creates a cross-region replication target in `us-west-1`.
| Resource | Naming |
| --- | --- |
| bucket | `macro-doc-storage-replication` in `prod`, otherwise `macro-doc-storage-replication-${stack}` |
| IAM role | `replication-role-${stack}` |
| admin group | `document-store-admin-${stack}` |
The replication bucket is versioned. In production it logs to `macro-access-log-bucket-uswest2`. Its bucket policy denies non-admin access and allows the replication role to read replication-related object versions.
## Queues
### `Queue`
`Queue` is a Pulumi component that creates:
- an SQS queue
- a DLQ
- a DLQ visible-message alarm
- a default queue age alarm through `QueueAlarms`
Naming:
| Resource | Name pattern |
| --- | --- |
| main queue | `${name}-queue-${stack}` |
| DLQ | `${name}-dlq-${stack}` |
| FIFO queue | appends `.fifo` to both names |
Defaults:
| Option | Default |
| --- | --- |
| `maxReceiveCount` | `5` |
| `visibilityTimeoutSeconds` | `30` |
| DLQ retention | `1209600` seconds |
| `fifoQueue` | `false` |
The DLQ alarm fires when `ApproximateNumberOfMessagesVisible > 0` and sends actions to `CloudTrailSNS`.
### `QueueAlarms`
`QueueAlarms` creates an `ApproximateAgeOfOldestMessage` CloudWatch alarm for a queue.
| Option | Default |
| --- | --- |
| `approximateAgeOfOldestMessageEvaluationPeriods` | used as the alarm period, default `60` seconds |
| `approximateAgeOfOldestMessageThreshold` | `120` seconds |
| metric namespace | `AWS/SQS` |
| alarm action | `CloudTrailSNS` |
### Custom queue stacks
Some stacks define queues directly instead of using `Queue`. For example, `search-event-queue` creates:
- `search-event-queue-${stack}`
- `search-event-queue-dlq-${stack}`
- an SNS topic named `search-event-queue-alerts-${stack}`
- production-only alarms for DLQ messages, old messages, and visible message count
## Service stacks
Service stacks usually follow this pattern:
1. Read Pulumi config with `config.require(...)`.
2. Resolve AWS Secrets Manager values with `aws.secretsmanager.getSecretVersionOutput(...)`.
3. Import shared VPC values with `get_coparse_api_vpc()`.
4. Read other stack outputs with `pulumi.StackReference`.
5. Create an ECS/Fargate component for the service.
6. Export service URLs, role ARNs, queue names, bucket names, or Lambda names needed by downstream stacks.
### Shared VPC
`get_coparse_api_vpc()` returns a fixed VPC ID and fixed public/private subnet IDs. Service tasks run in private subnets. ALBs use public subnets unless `isPrivate` is true.
### Load balancers
`serviceLoadBalancer` creates:
| Resource | Behavior |
| --- | --- |
| target group | HTTP target group with `targetType: "ip"` and caller-provided health check path |
| ALB | public or internal application load balancer |
| HTTPS listener | port `443`, TLS policy `ELBSecurityPolicy-TLS13-1-2-2021-06`, shared Macro ACM certificate |
| HTTP listener | port `80`, redirects to HTTPS with `HTTP_301` |
| access logs | enabled in `prod` to `macro-alb-logging` |
`isPrivate` controls subnet selection:
| `isPrivate` | ALB subnets |
| --- | --- |
| `true` | VPC private subnets |
| false/undefined | VPC public subnets |
### ECR images
`EcrImage` creates:
- mutable ECR repository
- force-delete repository behavior
- skipped default AWSX lifecycle policy
- explicit lifecycle rule to expire untagged images older than 1 day
- Docker image tagged `latest`
If `USE_PREBUILT_SERVICE_BINARIES=true`, supported Dockerfiles are remapped:
| Dockerfile | Prebuilt Dockerfile |
| --- | --- |
| `Dockerfile` | `Dockerfile.prebuilt` |
| `Dockerfile.convert_service` | `Dockerfile.convert_service.prebuilt` |
| `Dockerfile.search_processing_service` | `Dockerfile.search_processing_service.prebuilt` |
Unsupported Dockerfile names throw an error when prebuilt binaries are enabled.
### ECS/Fargate service conventions
The `mcp-server` and `document-cognition-service` components show the current ECS service shape:
- tasks run in private subnets
- security group ingress is only from the ALB security group on the service port
- ALB accepts public HTTP/HTTPS when the service is public
- deployment circuit breaker is enabled with rollback
- task role is passed through `taskDefinitionArgs.taskRole`
- Route53 creates an A alias for the service domain
- CPU, memory, ALB 5xx, and target-tracking scaling policies are declared in the service component
`mcp-server` uses:
| Setting | Value |
| --- | --- |
| domain | `mcp-server.macro.com` in `prod`, `mcp-server-${stack}.macro.com` otherwise |
| container port | `8080` |
| health check | `/health` |
| desired count | `1` |
| autoscaling min/max | min `1`, max `10` in `prod`, max `3` otherwise |
| scaling targets | ALB request count `1000`, CPU `70%`, memory `70%` |
| service alarms | CPU `80%`, memory `80%`, ALB 5xx count `25` |
## Datadog integration
Datadog support is centralized in `packages/resources`.
### Secrets and sidecars
`DATADOG_API_KEY` is read from Secrets Manager secret `datadog-api-key`.
`fargateLogRouterSidecarContainer` uses:
- image `amazon/aws-for-fluent-bit:latest`
- FireLens type `fluentbit`
- config file `/fluent-bit/configs/parse-json.conf`
- `ECS_FARGATE=true`
- `DD_ENV=${stack}`
- memory reservation `50`
`datadogAgentContainer` uses:
- image `public.ecr.aws/datadog/agent:latest`
- `DD_SITE=us5.datadoghq.com`
- APM enabled
- OTLP gRPC endpoint `0.0.0.0:4317`
- sample rate `0.1` in `prod`, `1.0` otherwise
- max traces per second `100`
- memory reservation `256`
Service containers that use FireLens send logs to `http-intake.logs.us5.datadoghq.com` with `dd_service`, `dd_source=fargate`, `dd_tags`, and `provider=ecs`.
### Lambda logs
The shared `Lambda` component creates a CloudWatch log group at:
```text
/aws/lambda/${baseName}-${stack}
```
It also creates a log subscription filter to the shared Datadog Kinesis Firehose stream and filters out Lambda runtime boilerplate lines:
```text
START RequestId
REPORT RequestId
END RequestId
INIT_START
```
### Datadog Software Catalog
`DatadogServiceEntity` creates a Datadog Software Catalog service entity with:
- service name
- display name
- owner
- GitHub repository link
- GitHub code path
- language metadata
- dependency on `service:document-storage`
## Lambda components
### `Lambda`
The generic `Lambda<T>` component deploys Rust Lambdas using the `provided.al2023` runtime.
Key behavior:
| Field | Value |
| --- | --- |
| handler | `bootstrap` |
| runtime | `provided.al2023` |
| architecture | `x86_64` |
| timeout default | `30` seconds |
| log retention | `7` days |
| deployment package | `pulumi.asset.FileArchive(zipLocation)` |
If a VPC is supplied, the component creates a Lambda security group, attaches all-in/all-out rules, and places the Lambda in private subnets.
The component adds `SOURCE_CODE_FILE_HASH` to the Lambda environment and depends on a `SourceCodeHash` resource so Pulumi detects source-code changes after a build.
### `WorkerTrigger`
`WorkerTrigger` creates a Lambda that runs an ECS task.
Environment variables:
| Name | Value |
| --- | --- |
| `TASK_DEFINITION` | task definition ARN |
| `CLUSTER` | ECS cluster ARN |
| `SUBNETS` | comma-separated private subnet IDs |
| `ENVIRONMENT` | Pulumi stack |
| `RUST_LOG` | `worker_trigger=info` |
The trigger role can call `ecs:RunTask` for the supplied task definition and cluster, and can `iam:PassRole` to `ecs-tasks.amazonaws.com`.
## AI tools deployment inputs
`getAiToolsInfra()` returns environment variables and IAM resource ARNs needed by services that host the Rust `ai_tools` crate.
It reads stack outputs from:
| StackReference name | Target stack |
| --- | --- |
| `ai-tools-cloud-storage-stack` | `macro-inc/document-storage/${stack}` |
| `ai-tools-cloud-storage-service-stack` | `macro-inc/cloud-storage-service/${stack}` |
| `ai-tools-email-service-stack` | `macro-inc/email-service/${stack}` |
| `ai-tools-linksharing-stack` | `macro-inc/link-sharing/${stack}` |
Returned environment variables include:
| Name | Source |
| --- | --- |
| `INTERNAL_API_SECRET_KEY` | `document-storage-service-auth-key-${stack}` secret value |
| `DOCUMENT_STORAGE_SERVICE_URL` | shared service URL map |
| `EMAIL_SERVICE_URL` | shared service URL map |
| `SYNC_SERVICE_URL` | shared service URL map |
| `LEXICAL_SERVICE_URL` | shared service URL map |
| `STATIC_FILE_SERVICE_URL` | shared service URL map |
| `SYNC_SERVICE_AUTH_KEY` | `sync-service-key-${stack}` secret name |
| `MCP_CREDENTIALS_KEY_SECRET_NAME` | `mcp-credentials-key-${stack}` |
| `DOCUMENT_STORAGE_BUCKET` | document-storage stack output |
| `DOCX_DOCUMENT_UPLOAD_BUCKET` | cloud-storage-service stack output |
| `EMAIL_SCHEDULED_QUEUE` | email-service stack output |
| `DOCUMENT_STORAGE_SERVICE_CLOUDFRONT_DISTRIBUTION_URL` | link-sharing stack output |
| `DOCUMENT_STORAGE_SERVICE_CLOUDFRONT_SIGNER_PUBLIC_KEY_ID` | link-sharing stack output |
| `DOCUMENT_STORAGE_SERVICE_CLOUDFRONT_SIGNER_PRIVATE_KEY_SECRET_NAME` | `linksharing-private-key-${stack}` |
Returned IAM inputs include:
- secret ARNs for sync service auth, CloudFront private key, and MCP credentials
- queue ARN for the email scheduled queue
- bucket ARNs for document storage and DOCX uploads
`getAiToolsServiceRoleArns()` returns role ARNs for services that host `ai_tools`, currently from `mcp-server`, `document-cognition`, and `agent-schedule-service` stack outputs.
## MCP server stack
`infra/stacks/mcp-server` deploys an ECS/Fargate service using the shared document-storage ECS cluster.
Required Pulumi config consumed by the stack:
| Key | Meaning |
| --- | --- |
| `macro_db_secret_key` | Secrets Manager key for database URL |
| `jwt_secret_key` | Secrets Manager key name for JWT secret |
| `fusionauth_client_id` | Secrets Manager key for FusionAuth client ID |
| `fusionauth_base_url` | FusionAuth base URL |
| `fusionauth_issuer` | FusionAuth issuer |
| `fusionauth_client_secret` | Secrets Manager key for FusionAuth client secret |
| `fusionauth_tenant_id` | FusionAuth tenant ID |
| `fusionauth_api_key` | Secrets Manager key for FusionAuth API key |
| `google_client_id` | Secrets Manager key for Google client ID |
| `google_client_secret` | Secrets Manager key for Google client secret |
| `macro_cache_secret_key` | Secrets Manager key for Redis endpoint |
| `anthropic_api_key` | Secrets Manager key for Anthropic API key |
The service also receives all `getAiToolsInfra()` environment variables.
Exports:
| Output | Meaning |
| --- | --- |
| `mcpServerUrl` | service URL |
| `mcpServerRoleArn` | ECS task role ARN |
## FusionAuth stacks
Macro uses two separate FusionAuth-related stacks.
| Stack | Responsibility |
| --- | --- |
| `fusion-auth` | Runs FusionAuth itself on ECS/Fargate with an RDS PostgreSQL database |
| `fusionauth-instance` | Configures FusionAuth tenants, application, templates, lambdas, webhooks, signing key, and OAuth settings through `pulumi-fusionauth` |
### `fusion-auth`
The runtime stack creates:
- RDS PostgreSQL database named `fusionauth`
- ECS cluster named `fusionauth-${stack}`
- ECS/Fargate service using image `fusionauth/fusionauth-app:1.62.1`
- ALB and Route53 record
- CloudWatch alarms for CPU, memory, and ALB 5xx
- production-only service autoscaling
Domain names:
| Stack | Domain |
| --- | --- |
| `prod` | `auth.macro.com` |
| non-`prod` | `fusionauth-${stack}.macro.com` |
Database behavior:
| Stack | Behavior |
| --- | --- |
| `prod` | endpoint output is hard-coded to `fusionauthdb-prod.macro.com` |
| non-`prod` | endpoint comes from the created RDS instance |
| `dev` | database is publicly accessible |
| `prod` | database is not publicly accessible |
Required config:
| Key | Meaning |
| --- | --- |
| `fusion-auth:db-password-secret-key` | Secrets Manager key containing the database password |
### `fusionauth-instance`
The configuration stack uses the `pulumi-fusionauth` provider. Provider inputs are read from the `fusionauth` Pulumi config namespace:
| Key | Meaning |
| --- | --- |
| `fusionauth:host` | FusionAuth API host |
| `fusionauth:apiKey` | FusionAuth API key |
Stack config inputs include:
| Key | Meaning |
| --- | --- |
| `fusionauth-instance:fusionauth-issuer` | tenant issuer |
| `fusionauth-instance:fusionauth-signing-key-id` | signing key ID |
| `fusionauth-instance:fusionauth-license-key-secret-key` | Secrets Manager key for FusionAuth license |
| `fusionauth-instance:smtp-user-secret-key` | Secrets Manager key containing `{ username, password }` |
| `fusionauth-instance:default-from-email` | tenant default sender email |
| `fusionauth-instance:authentication-service-domain` | authentication service base URL |
| `fusionauth-instance:authentication-service-internal-secret-key` | optional Secrets Manager key for webhook auth |
| `fusionauth-instance:fusionauth-default-tenant-id` | required outside `local` |
| `fusionauth-instance:fusionauth-client-id` | required outside `local` |
| `fusionauth-instance:fusionauth-client-secret-key` | required outside `local` |
Configured FusionAuth resources:
- Reactor license
- system CORS for Apple ID POST callbacks
- passwordless login email template
- email verification template
- create-user and delete-user webhooks to the authentication service
- default tenant with SMTP, email verification, event configuration, theme, issuer, logout URL, and unverified-user deletion after 30 days
- `populate_macro_jwt` JWTPopulate lambda
- `reconcile_secondary_idp_link` OpenIDReconcile lambda
- HS256 signing key
- Macro application with passwordless login, JWT, refresh tokens, and OAuth redirects
OAuth redirect URLs include:
- `${AUTHENTICATION_SERVICE_DOMAIN}/oauth/redirect`
- `https://mcp-server.macro.com/oauth/callback` in `prod`
- `https://mcp-server-${stack}.macro.com/oauth/callback` outside `prod`
- localhost redirect patterns in `local` and `dev`
<Warning>
The Google/Gmail secondary IdP reconcile lambda is declared in Pulumi, but the IdP wiring is documented in code as an admin-UI step until the IdP itself is managed by Pulumi.
</Warning>
### Local FusionAuth setup
`infra/stacks/fusionauth-instance/justfile` provides local setup commands.
```bash
cd infra/stacks/fusionauth-instance
just setup
```
`just setup` installs workspace dependencies, downloads the FusionAuth Docker Compose `.env`, starts FusionAuth, waits for `/api/status`, initializes the local Pulumi stack, applies the FusionAuth configuration, patches the root `.env`, and stops the container.
<Warning>
The local `fusionauth-instance` stack is intended for a personal Pulumi account. Do not create it with the `macro-inc/` organization prefix.
</Warning>
## Deployment checklist
<Steps>
<Step title="Install dependencies">
Run `bun i` from `infra/` or use the stack-specific setup command when one exists.
</Step>
<Step title="Select the stack directory">
Run Pulumi from `infra/stacks/<stack-name>`, not from the repository root.
</Step>
<Step title="Verify config">
Confirm `Pulumi.<stack>.yaml` contains `aws:region: us-east-1` and every `config.require(...)` key consumed by that stack.
</Step>
<Step title="Check stack references">
For service stacks, verify upstream stacks have exported the referenced outputs. Common dependencies are `document-storage`, `cloud-storage-service`, `email-service`, and `link-sharing`.
</Step>
<Step title="Deploy">
Run `pulumi up --stack dev` or `pulumi up --stack prod`.
</Step>
</Steps>
## Related pages
<CardGroup>
<Card title="Service stack implementation" href="/infrastructure/service-stacks">
ECS/Fargate service component shape, load balancer behavior, and service-specific deployment patterns.
</Card>
<Card title="FusionAuth operations" href="/infrastructure/fusionauth">
Runtime FusionAuth service, tenant configuration, local setup, and OAuth integration details.
</Card>
</CardGroup>
---
## 26. Build, test, and quality gates
> Operations: Rust checks, clippy, SQLx preparation, TypeScript checks, Biome, Tailwind hygiene, Vitest, Playwright, and CI path filters.
- Page Markdown: https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/26-build-test-and-quality-gates.md
- Generated: 2026-06-01T01:04:48.525Z
### Source Files
- `justfile`
- `rust/cloud-storage/justfile`
- `js/package.json`
- `js/app/package.json`
- `js/app/justfile`
- `.github/workflows/code-check-cloud-storage.yml`
- `.github/workflows/web-app-check-main.yml`
---
title: "Build, test, and quality gates"
description: "Operations: Rust checks, clippy, SQLx preparation, TypeScript checks, Biome, Tailwind hygiene, Vitest, Playwright, and CI path filters."
---
The repository splits operational checks between `rust/cloud-storage` and `js/app`: Rust uses `just` recipes around Cargo, SQLx, and Nextest, while the web app uses Bun scripts, Biome, Vitest, Playwright, and GitHub Actions path filters to keep PR checks scoped.
## Command map
| Surface | Working directory | Command | Purpose |
|---|---:|---|---|
| Rust type check | `rust/cloud-storage` | `just check` | Runs `cargo check` with warnings denied and `SQLX_OFFLINE=true`. |
| Rust lint | `rust/cloud-storage` | `just clippy` | Runs workspace Clippy with all features, warnings denied, and `clippy::disallowed_methods` denied. |
| Rust format | `rust/cloud-storage` | `just format` | Runs `cargo fmt`; CI uses `cargo fmt --check`. |
| SQLx cache | `rust/cloud-storage` | `just prepare_db` | Runs workspace `cargo sqlx prepare --workspace -- --all-features`. |
| Rust tests | `rust/cloud-storage` | `cargo nextest run --all-features --lib --bins --tests` | Mirrors the CI test runner after database setup. |
| Web type check | `js/app` | `bun run type-check` | Runs `tsc --noEmit --skipLibCheck --project packages/app/tsconfig.json`. |
| Web combined check | `js/app` | `bun run check` | Runs type-check and `bun biome check`. |
| Web Biome CI | `js/app` | `bunx --bun biome ci --changed --no-errors-on-unmatched --error-on-warnings` | Matches the main web PR Biome gate. |
| Tailwind hygiene | `js/app` | `just check-tailwind` | Scans changed lines for prohibited raw Tailwind color/font utilities. |
| Vitest | `js/app` | `bunx vitest` or `bun run test` | Runs configured Vitest projects. |
| Local Playwright E2E | repository root | `just local-e2e` | Starts local dependencies, seeds deterministic data, then runs Playwright with `LOCAL_E2E=true`. |
<Note>
The JavaScript workspace declares `bun@1.3.5`. Rust uses the checked-in toolchain file with Rust `1.94.0` plus `clippy`, `rustfmt`, `rust-analyzer`, and `rust-src`.
</Note>
## Recommended pre-PR gate
<Steps>
<Step title="Run Rust gates after Rust changes">
```bash
cd rust/cloud-storage
just check
just clippy
just format
```
</Step>
<Step title="Refresh SQLx metadata after SQL changes">
```bash
cd rust/cloud-storage
just prepare_db
```
Commit any changed `.sqlx/query-*.json` files generated by SQLx.
</Step>
<Step title="Run web gates after TypeScript or UI changes">
```bash
cd js/app
bun run type-check
bunx --bun biome ci --changed --no-errors-on-unmatched --error-on-warnings
just check-tailwind
bunx vitest
bun run build
```
</Step>
<Step title="Run local E2E when behavior crosses the app/backend boundary">
```bash
just local-e2e
```
</Step>
</Steps>
## Rust cloud-storage gates
### Local checks
`rust/cloud-storage/justfile` owns the local Rust quality recipes:
```bash
just check
just clippy
just format
just build
```
`just check` sets `RUSTFLAGS=-Dwarnings`, `RUSTDOCFLAGS=-Dwarnings`, `CARGO_TERM_COLOR=always`, and `SQLX_OFFLINE=true` before `cargo check`.
`just clippy` applies the same warning policy and additionally denies `clippy::disallowed_methods`, then runs:
```bash
cargo clippy --workspace --all-features
```
`just format` runs `cargo fmt`. Use CI-style format checking with:
```bash
cargo fmt --check
```
### SQLx preparation
`rust/cloud-storage/sqlx.just` defines the database lifecycle wrappers. The default local database URL comes from `rust/cloud-storage/database.just`:
```text
postgres://user:password@localhost:5432/macrodb
```
The SQLx preparation recipe is workspace-level:
```bash
cd rust/cloud-storage
just prepare_db
```
It expands to:
```bash
cargo sqlx prepare --workspace -- --all-features
```
Use this after adding or changing SQLx compile-time checked queries, migrations that affect queried schemas, or Rust test queries that require refreshed offline metadata.
<Warning>
Do not hand-edit `.sqlx/query-*.json` files. Generate them with `just prepare_db` from `rust/cloud-storage`.
</Warning>
### Rust test database setup
The Rust test path uses live Postgres, not SQLx offline mode. The CI workflow starts `pgvector/pgvector:pg16` and Redis, configures Postgres for high-concurrency tests, then runs:
```bash
just rust/cloud-storage/setup_test_envs
just rust/cloud-storage/initialize_dbs
cd rust/cloud-storage
cargo nextest run --all-features --lib --bins --tests --test-threads 32
```
`setup_test_envs` writes `DATABASE_URL` into the crate-level `.env` files needed by SQLx-backed tests. `initialize_dbs` runs the macro database create/migrate setup.
<Warning>
Do not run Rust tests with `SQLX_OFFLINE=true`. Offline mode is for `cargo check`, `cargo build`, and `cargo clippy`; tests validate against the live local schema.
</Warning>
## Web app gates
### TypeScript
The web app TypeScript gate runs against `packages/app/tsconfig.json`:
```bash
cd js/app
bun run type-check
```
The top-level `js/package.json` also exposes:
```bash
cd js
bun run check
```
That delegates to the app type-check command.
In CI, the TypeScript job runs when either web files changed or Rust API inputs changed. If Rust API inputs changed, the job first runs generated client validation:
```bash
cd js/app
bun run gen-api -- --check
```
The generator builds Rust OpenAPI binaries with `SQLX_OFFLINE=true`, runs them, regenerates TypeScript clients through Orval, runs Biome over generated service clients, then fails in `--check` mode if `git diff` or untracked generated files remain.
### Biome
`js/app/biome.jsonc` configures Biome with Git VCS support, `origin/main` as the default branch, LF line endings, 2-space indentation, 80-column formatting, Solid rules enabled, React rules disabled, and generated-output exclusions.
CI runs:
```bash
cd js/app
biome ci --changed --no-errors-on-unmatched --error-on-warnings
```
Local equivalents include:
```bash
cd js/app
bunx --bun biome lint --skip=suspicious/noImportCycles
bunx --bun biome format --write
bunx --bun biome check --write
```
The web CI also has a separate import-cycle gate:
```bash
cd js/app
biome lint --changed --no-errors-on-unmatched --only=suspicious/noImportCycles
```
### Tailwind hygiene
`js/app/scripts/check-tailwind.ts` enforces theme hygiene on changed lines only. It scans changed `.tsx`, `.ts`, `.jsx`, and `.js` files under `packages/` and rejects raw utility classes matching:
- raw color utilities such as `bg-red-500`, `text-gray-700`, `border-black`, `fill-blue-400`
- raw `white` / `black` color utilities for the checked utility groups
- `font-berkeley` and `font-inter`
Run it with:
```bash
cd js/app
just check-tailwind
```
In CI, the base branch is `origin/${GITHUB_BASE_REF}`. Locally, the script defaults to `origin/dev`.
Expected success output includes:
```text
✅ No prohibited Tailwind classes found in changed lines!
```
On failure, the script prints file, line, column, matched class name, and a hint to replace raw utilities with semantic classes.
### Vitest
`js/app/vitest.config.ts` defines a multi-project Vitest setup for package groups including `websocket`, `core`, `queries`, `scripts`, `lexical-core`, `theme`, `block-channel`, `channel`, `notifications`, and `block-email`.
Run once:
```bash
cd js/app
bunx vitest
```
Run via package script:
```bash
cd js/app
bun run test
```
Watch mode is exposed through the app justfile:
```bash
cd js/app
just test-watch
```
The web PR workflow installs Bun dependencies with `bun install --frozen-lockfile`, installs/caches Playwright Chromium when requested by the setup action, then runs `bunx vitest`.
### Build
The app build script is:
```bash
cd js/app
bun run build
```
It runs from `packages/app` with:
```bash
MODE=development NODE_ENV=production bun run --bun build
```
The app justfile also provides mode-specific builds:
```bash
just build-dev
just build-staging
just build-prod
just build-tauri
```
## Playwright and local E2E
Playwright is configured in `js/app/playwright.config.ts` with `testDir: ./tests/e2e`, fully parallel execution, trace retention on failure, and `baseURL` set to `http://localhost:${PORT:-3000}/app`.
The repository-level local E2E harness is the safest entry point:
```bash
just local-e2e
```
It performs the full setup:
1. starts LocalStack with test AWS credentials,
2. starts the local service subset through Docker Compose,
3. seeds deterministic E2E data,
4. runs `LOCAL_E2E=true bunx playwright test` in `js/app`.
For UI mode:
```bash
just local-e2e-ui
```
`LOCAL_E2E=true` changes Playwright to a single `local-chromium` project and generates a local JWT unless `LOCAL_JWT` is already exported. Direct Playwright runs can fail token generation if `.env` and seeded local data are missing; prefer the repo-level harness.
## CI path filters
### Cloud Storage PR check
`.github/workflows/code-check-cloud-storage.yml` runs on pull requests to `main` for opened, synchronized, reopened, and ready-for-review events. The path filter enables the workflow for changes under:
- `rust/cloud-storage/**`
- `rust/rust-toolchain.toml`
- `flake.nix`
- `flake.lock`
- cloud-storage CI workflow and supporting GitHub actions/scripts
The workflow has three effective stages:
| Job | Runs when | Gate |
|---|---|---|
| `path-check` | Always on matching PR event | Computes `should_run` and an optional Nextest package filter. |
| `check` | `should_run == true` and PR is not draft | Runs Rust format check and Clippy. |
| `test` | `should_run == true` and PR is not draft | Starts Postgres/Redis, initializes DBs, and runs Nextest. |
| `status-check` | Always | Fails only if `path-check`, `check`, or `test` failed. Skipped jobs are accepted. |
For package-specific Rust changes, CI computes a Nextest expression such as:
```text
rdeps(=package_name)
```
Multiple package filters are joined with `|`. Workspace-level changes, lockfiles, toolchain changes, Cargo workspace changes, or CI/action changes force the full test suite.
### Web app PR check
`.github/workflows/web-app-check-main.yml` runs on pull requests to `main`. Its path filters produce two outputs:
| Output | Matching changes |
|---|---|
| `should_run` | Web package files, app packages/src, `biome.jsonc`, lexical workspaces, setup actions, and the workflow file. |
| `api_changed` | Rust cloud-storage Rust/Cargo files, flake files, API generation scripts, setup actions, and the workflow file. |
The jobs are scoped as follows:
| Job | Condition | Command |
|---|---|---|
| `typescript` | `should_run == true` or `api_changed == true` | Optionally `bun run gen-api -- --check`, then `tsc`. |
| `biome-check` | `should_run == true` | `biome ci --changed --no-errors-on-unmatched --error-on-warnings`. |
| `tailwind` | `should_run == true` | `just check-tailwind`. |
| `test` | `should_run == true` | `bunx vitest`. |
| `cycles` | `should_run == true` | Biome changed-file import-cycle lint. |
| `build` | `should_run == true` | `bun run build`. |
| `status-check` | Always | Fails only if a needed job failed. Skipped jobs are accepted. |
This means a Rust API-only change can trigger the TypeScript/API generation validation without running every web-only gate.
## Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| SQLx reports missing cached query data during check/build | `.sqlx` metadata is stale or absent | Run `cd rust/cloud-storage && just prepare_db`. |
| Rust tests fail when `SQLX_OFFLINE=true` is set | Tests need a live schema | Unset `SQLX_OFFLINE`, initialize DBs, and rerun tests. |
| Web TypeScript CI fails after Rust API changes | Generated service clients are out of sync | Run `cd js/app && bun run gen-api`, review generated files, commit updates. |
| Tailwind hygiene fails on new classes | Raw color/font utilities were added on changed lines | Replace with semantic theme classes. |
| Direct Playwright run cannot generate `LOCAL_JWT` | Local seed/env prerequisites are missing | Run `just local-e2e` or export `LOCAL_JWT` after preparing local data. |
| Biome changed-file CI finds no files locally | Wrong comparison branch | Fetch the expected base branch and rerun from `js/app`. |
## Next
For backend database work, pair this page with the repository’s SQLx migration and query-update conventions. For UI changes, run the web gates before opening a PR and use `just local-e2e` when the change depends on local services.
---
## 27. Deploy services and web app
> Operations: Generic service deployment, web app deployment, production release workflow, database migrations, Pulumi stacks, and deployment concurrency.
- Page Markdown: https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/27-deploy-services-and-web-app.md
- Generated: 2026-06-01T01:07:17.883Z
### Source Files
- `.github/workflows/deploy-service-generic.yml`
- `.github/workflows/reusable-deploy-service.yml`
- `.github/workflows/deploy-web-app.yml`
- `.github/workflows/release-production.yml`
- `.github/workflows/migrate-macro-db.yml`
- `infra/README.md`
- `rust/cloud-storage/README.md`
---
title: "Deploy services and web app"
description: "Operations: Generic service deployment, web app deployment, production release workflow, database migrations, Pulumi stacks, and deployment concurrency."
---
Macro deploys run through GitHub Actions workflows that build Rust service binaries or Lambda `bootstrap.zip` artifacts, install Pulumi TypeScript dependencies under `infra/`, configure AWS credentials, and run `pulumi up` against `macro-inc/dev` or `macro-inc/prod`.
## Deployment surfaces
| Surface | Entry point | Trigger | Target |
| --- | --- | --- | --- |
| Generic service deploy | `.github/workflows/deploy-service-generic.yml` | Manual `workflow_dispatch` | One service from the workflow choice list |
| Reusable service deploy | `.github/workflows/reusable-deploy-service.yml` | `workflow_call` | One service stack |
| Deploy all services | `.github/workflows/deploy-all-services.yml` | Manual or production release | Every key in `.github/services-config.json` |
| Dev cloud-storage deploy | `.github/workflows/deploy-cloud-storage-on-push.yml` | Push to `main` matching cloud-storage or infra paths | All configured services to `dev` |
| Manual Pulumi stack deploy | `.github/workflows/deploy-pulumi-stack.yml` | Manual `workflow_dispatch` | Any `infra/stacks/<name>` directory |
| Web app deploy | `.github/workflows/deploy-web-app.yml` | Manual or `workflow_call` | `infra/stacks/web-app` |
| Production release | `.github/workflows/release-production.yml` | GitHub release `released` event | DB migrations, services, web app, release artifact |
| Database migration | `.github/workflows/migrate-macro-db.yml` | Manual `workflow_dispatch` | `macro-db-dev` or `macro-db-prod` |
## Service deployment model
Service deployments are driven by `.github/services-config.json`. Each service key maps source ownership and deploy artifacts to a Pulumi stack directory.
```json
{
"services": {
"<service-key>": {
"source_paths": ["rust/cloud-storage/<crate-or-area>/**"],
"stack_path": "infra/stacks/<pulumi-project>/**",
"deploy_binaries": ["<cargo-bin-name>"],
"deploy_lambdas": ["<lambda-package-name>"]
}
}
}
```
| Field | Required | Used by |
| --- | --- | --- |
| `stack_path` | Yes | `.github/actions/get-project-name` derives the Pulumi project directory from this path. |
| `source_paths` | No | Documents source ownership and is available for change-detection workflows. |
| `deploy_binaries` | No | Enables Nix binary artifact builds through `.#deploy-service-binaries-<service>`. |
| `deploy_lambdas` | No | Enables Lambda artifact builds through `.github/scripts/build-cloud-storage-lambdas.sh`. |
<Note>
The current `deploy-cloud-storage-on-push.yml` workflow selects all service keys from `.github/services-config.json`; it does not narrow the dev deploy to changed `source_paths`.
</Note>
### Generic manual service deploy
Use **Deploy Service (Generic)** when the target service appears in `.github/workflows/deploy-service-generic.yml`.
Inputs:
<ParamField body="service" type="choice" required>
Service key to deploy. The key is passed to the reusable deploy workflow as `service-name`.
</ParamField>
<ParamField body="environment" type="choice" required>
`dev` or `prod`. Defaults to `dev`.
</ParamField>
The workflow forwards to `.github/workflows/reusable-deploy-service.yml` with inherited secrets. Its concurrency group is:
```text
deploy-${service}-${environment}
```
`cancel-in-progress` is `false`, so a second deploy for the same service and environment queues instead of canceling the active deploy.
### Reusable service deploy lifecycle
`reusable-deploy-service.yml` has three jobs:
1. `build-service-binaries`
- Checks `.services[$SERVICE].deploy_binaries`.
- Runs `nix build ".#deploy-service-binaries-${SERVICE}"` only when binaries are configured.
- Copies `result/bin/*` and the Nix runtime closure from `nix-store -qR result`.
- Uploads `prebuilt-binaries.tar.gz` as `cloud-storage-service-binaries-<service>-<sha>`.
2. `build-lambda-artifacts`
- Checks `.services[$SERVICE].deploy_lambdas`.
- Runs `.github/scripts/build-cloud-storage-lambdas.sh` only when Lambdas are configured.
- Uploads `lambda-artifacts.tar.gz` as `cloud-storage-lambdas-<service>-<sha>`.
3. `deploy`
- Resolves the Pulumi project name from `stack_path`.
- Calls `.github/actions/deploy-cloud-storage-pulumi`.
- Passes artifact names only when the service config enabled those artifact types.
The deploy job runs on an 8 CPU Linux runner with a per-run service identifier.
### Pulumi deploy action
`.github/actions/deploy-cloud-storage-pulumi` performs the stack update:
<Steps>
<Step title="Checkout and restore artifacts">
The action checks out the repository, downloads optional prebuilt binary and Lambda artifacts, and extracts them under `rust/cloud-storage/prebuilt` or `rust/cloud-storage/target/lambda`.
</Step>
<Step title="Prepare runtime tools">
It optionally sets up Docker Buildx, installs Bun `1.2.0`, and runs `bun install` in `infra/`.
</Step>
<Step title="Configure AWS and ECR">
It configures AWS credentials for `us-east-1` by default and logs Docker into the repository account ECR registry.
</Step>
<Step title="Run Pulumi">
It runs `pulumi/actions` with `command: up`, `stack-name: macro-inc/<environment>`, and `work-dir: ./infra/stacks/<pulumi-service-name>`.
</Step>
</Steps>
Required inputs include AWS keys, `PULUMI_ACCESS_TOKEN`, `DD_APP_KEY`, `DD_API_KEY`, and `pulumi-service-name`.
Optional inputs:
| Input | Default | Behavior |
| --- | --- | --- |
| `use-docker` | `true` | Controls Docker Buildx setup. |
| `use-lfs` | `false` | Enables LFS checkout. |
| `aws-region` | `us-east-1` | AWS region for credentials. |
| `prebuilt-binaries-artifact` | empty | Downloads and extracts prebuilt service binaries. |
| `lambda-artifacts` | empty | Downloads and extracts Lambda zips. |
| `cloud-storage-service-name` | empty | Allows inline Lambda builds when no Lambda artifact was supplied. |
| `dd-host` | `https://api.us5.datadoghq.com/` | Datadog API host for Pulumi resources. |
## Prebuilt binaries and Docker images
Rust service binaries are built by Nix through flake outputs named:
```text
deploy-service-binaries-<service-key>
```
The Nix package copies release binaries into `$out/bin` and strips them when possible. CI then packages the binaries with their Nix store closure.
When `USE_PREBUILT_SERVICE_BINARIES=true`, `infra/packages/service/src/ecr.ts` switches supported Dockerfiles to prebuilt variants:
| Normal Dockerfile | Prebuilt Dockerfile |
| --- | --- |
| `Dockerfile` | `Dockerfile.prebuilt` |
| `Dockerfile.convert_service` | `Dockerfile.convert_service.prebuilt` |
| `Dockerfile.search_processing_service` | `Dockerfile.search_processing_service.prebuilt` |
If a service uses a different Dockerfile and prebuilt binaries are enabled, deployment fails with:
```text
No prebuilt Dockerfile mapping configured for <Dockerfile>
```
The default prebuilt image copies `./prebuilt/nix-store/` into `/nix/store/`, copies `./prebuilt/${SERVICE_NAME}` to `/app/svc`, and starts it through `dumb-init`.
## Lambda artifacts
Lambda-backed stacks read `bootstrap.zip` files from:
```text
rust/cloud-storage/target/lambda/<lambda-name>/bootstrap.zip
```
`.github/scripts/build-cloud-storage-lambdas.sh` reads `deploy_lambdas` for the selected service, enters the Nix development shell, and runs:
```bash
cd rust/cloud-storage
just "<lambda-name>/build"
test -f "target/lambda/<lambda-name>/bootstrap.zip"
```
It packages the output as:
```text
lambda-artifacts.tar.gz
└── target/
└── lambda/
└── <lambda-name>/
└── bootstrap.zip
```
For local Pulumi deploys that touch Lambda-backed stacks, `rust/cloud-storage/README.md` instructs running `just build_lambdas` before `pulumi up`.
## Web app deployment
`.github/workflows/deploy-web-app.yml` builds `js/app` and deploys `infra/stacks/web-app`.
Inputs:
<ParamField body="environment" type="string" required>
`dev` or `prod`.
</ParamField>
<ParamField body="notify" type="boolean">
Accepted by the workflow interface. No current workflow step reads this input.
</ParamField>
Build command:
```bash
cd js/app
just build-${environment}
```
The `just` recipes set Vite build mode:
| Environment | Command | Build env |
| --- | --- | --- |
| `dev` | `just build-dev` | `MODE=development NODE_ENV=production` |
| `prod` | `just build-prod` | `MODE=production NODE_ENV=production` |
The workflow supplies these Vite variables:
| Variable | Source |
| --- | --- |
| `VITE_DD_WEB_APP_ID` | `DD_WEB_APP_ID` secret |
| `VITE_DD_WEB_APP_TOKEN` | `DD_WEB_APP_TOKEN` secret |
| `VITE_DD_HASH` | `github.sha` |
| `VITE_SEGMENT_WRITE_KEY` | `SEGMENT_WRITE_KEY` secret |
| `VITE_POSTHOG_API_KEY` | `POSTHOG_API_KEY` secret |
After building, the workflow installs infra dependencies, configures AWS credentials, and runs Pulumi:
```text
stack-name: macro-inc/<environment>
work-dir: ./infra/stacks/web-app
```
It then uploads sourcemaps through Datadog CI:
```bash
cd js/app/packages/app
bun run ddupload:<environment>
```
For `prod`, the workflow also uploads the built `js/app/packages/app/dist` directory as the GitHub Actions artifact `web-app-build`.
### Web app Pulumi stack behavior
`infra/stacks/web-app/index.ts` requires the Pulumi config key `macro-web-app:path`. For `dev` and `prod`, this points at:
```text
../../../js/app/packages/app/dist
```
The stack fails early when the build output directory is missing or empty:
```text
Local path of build output is empty
```
The stack also blocks local production deployment unless `CI` is set:
```text
You are trying to deploy to prod without the CI environment variable set
```
The stack creates and updates:
- an S3 website bucket for app assets;
- a public `app/app-archive.zip`;
- synced app files under `./output`;
- `Cache-Control: no-store` metadata on `app/index.html`;
- a content-encoding Lambda@Edge function;
- an app-route Lambda function URL;
- a Datadog software catalog entity;
- a production CloudFront invalidation against the `website-infra` stack output `cdnId`.
For production asset sync, JavaScript sourcemaps are excluded from the S3 sync command.
## Database migrations
Database migrations run through `.github/actions/migrate-cloud-storage-db`.
Entry points:
| Workflow | Environment |
| --- | --- |
| `.github/workflows/migrate-macro-db.yml` | Manual `dev` or `prod` |
| `.github/workflows/deploy-cloud-storage-on-push.yml` | `dev` before service deployment |
| `.github/workflows/release-production.yml` | `prod` before production service deployment |
The action checks out `.github/` and `rust/cloud-storage/macro_db_client`, then reads the database URL from AWS Secrets Manager:
| Environment | Secret name |
| --- | --- |
| `dev` | `macro-db-dev` |
| `prod` | `macro-db-prod` |
If the URL does not include `sslmode=`, the action appends `sslmode=require`.
Migration commands run from `rust/cloud-storage/macro_db_client`:
```bash
# dev
cargo sqlx migrate run --ignore-missing
# prod
cargo sqlx migrate run
```
The manual migration workflow runs on the `db-migrator` runner. The migration action itself does not configure AWS credentials; the runner environment must be able to call AWS Secrets Manager.
## Production release workflow
`.github/workflows/release-production.yml` runs when a GitHub release is published with the `released` event type.
```text
release: released
└─ migrate-db
└─ deploy-cloud-storage
└─ build web app
└─ deploy web app
└─ upload release artifact
```
Lifecycle:
1. `migrate-db`
- Runs production SQLx migrations on `db-migrator`.
2. `deploy-cloud-storage`
- Calls `.github/workflows/deploy-all-services.yml` with `environment: prod`.
- Deploys every service key in `.github/services-config.json`.
3. `build`
- Builds the web app with `just build-prod`.
- Uploads `web-app-prod-build` from `js/app/packages/app/dist`.
4. `deploy`
- Downloads `web-app-prod-build`.
- Runs Pulumi in `./infra/stacks/web-app` with `stack-name: macro-inc/prod`.
- Uploads production sourcemaps to Datadog.
5. `upload`
- Creates `web-app-${github.ref_name}.tar.gz` from the built `dist` directory.
- Uploads it to the GitHub release.
The production release workflow has a single workflow-level concurrency group and does not cancel in-progress releases.
## Pulumi stacks
Infrastructure is TypeScript Pulumi under `infra/`. Each stack lives in `infra/stacks/<stack-name>` and has its own `Pulumi.yaml`. CI runs Pulumi from the stack directory.
The repository uses AWS infrastructure in `us-east-1`. The workflows configure AWS with repository secrets and pass Datadog keys through Pulumi environment variables.
Common CI stack names:
| Environment input | Pulumi stack name |
| --- | --- |
| `dev` | `macro-inc/dev` |
| `prod` | `macro-inc/prod` |
Local stack deployment follows the infra README pattern:
```bash
cd infra/stacks/<stack-name>
pulumi up --stack dev
# or
pulumi up --stack prod
```
The web app package also exposes stack-specific scripts:
```bash
cd infra/stacks/web-app
bun run deploy:dev
bun run preview:dev
bun run deploy:prod
```
## Deployment concurrency
| Workflow | Concurrency group | Cancel in progress |
| --- | --- | --- |
| `deploy-service-generic.yml` | `deploy-<service>-<environment>` | `false` |
| `deploy-web-app.yml` | `<workflow>-web-app-<environment>` | `false` |
| `release-production.yml` | `<workflow>` | `false` |
| `deploy-all-services.yml` | `<workflow>-all-services-<environment>` | `false` |
| `deploy-cloud-storage-on-push.yml` | `deploy-cloud-storage-<git-ref>` | `false` |
| `deploy-pulumi-stack.yml` | `<workflow>-<pulumi-service-name>-<environment>` | `false` |
Matrix limits:
| Workflow | Job | `max-parallel` |
| --- | --- | --- |
| `deploy-all-services.yml` | Build service binaries | `8` |
| `deploy-all-services.yml` | Build Lambda artifacts | `8` |
| `deploy-all-services.yml` | Deploy services | `20` |
| `deploy-cloud-storage-on-push.yml` | Build service binaries | `8` |
| `deploy-cloud-storage-on-push.yml` | Build individual Lambda artifacts | `8` |
| `deploy-cloud-storage-on-push.yml` | Package Lambda artifacts | `8` |
| `deploy-cloud-storage-on-push.yml` | Deploy services | `20` |
A scheduled cleanup workflow, `.github/workflows/cancel-stuck-cloud-storage-deploys.yml`, runs hourly and cancels queued or in-progress `deploy-cloud-storage-on-push.yml` runs older than 90 minutes by default. It also supports manual `max-age-minutes` and `dry-run` inputs.
## Adding or changing a deployable service
<Steps>
<Step title="Add or update the service config">
Add the service key to `.github/services-config.json` with `stack_path`, optional `source_paths`, optional `deploy_binaries`, and optional `deploy_lambdas`.
</Step>
<Step title="Create or update the Pulumi stack">
Ensure `infra/stacks/<stack-name>/Pulumi.yaml` exists. The `stack_path` basename must match the directory that CI should pass as `pulumi-service-name`.
</Step>
<Step title="Wire binary builds when needed">
If `deploy_binaries` is set, ensure `flake.nix` exposes `deploy-service-binaries-<service-key>` and includes every listed Cargo binary.
</Step>
<Step title="Wire Lambda builds when needed">
If `deploy_lambdas` is set, ensure each `just "<lambda>/build"` recipe produces `rust/cloud-storage/target/lambda/<lambda>/bootstrap.zip`.
</Step>
<Step title="Check Dockerfile compatibility">
Services using prebuilt binary deploys must use a Dockerfile supported by `EcrImage` prebuilt mapping, or the mapping must be extended.
</Step>
<Step title="Expose manual deployment when needed">
If operators should deploy through **Deploy Service (Generic)**, add the service key to that workflow's `service` choice list.
</Step>
</Steps>
## Troubleshooting
| Symptom | Likely cause | Check |
| --- | --- | --- |
| `Service '<name>' not found in .github/services-config.json` | The service key is missing or misspelled. | Verify `.github/services-config.json` and the workflow input. |
| `Error: Service '<name>' not found in services-config.json` | `get-project-name` could not find `stack_path`. | Confirm the service key has a `stack_path`. |
| Lambda build exits after `test -f ... bootstrap.zip` | The `just <lambda>/build` recipe did not produce the expected zip. | Run the recipe from `rust/cloud-storage` and inspect `target/lambda/<lambda>/`. |
| `No prebuilt Dockerfile mapping configured` | A prebuilt binary deploy is using an unsupported Dockerfile name. | Update `infra/packages/service/src/ecr.ts` or use a supported Dockerfile. |
| `Local path of build output is empty` | Web app Pulumi ran before `js/app/packages/app/dist` was built. | Run `just build-dev` or `just build-prod` from `js/app`. |
| Production web app deploy fails outside CI | The web app Pulumi stack blocks `prod` when `CI` is not set. | Use the release workflow or a CI environment. |
| Dev cloud-storage deploy appears stuck | The on-push workflow has queued or in-progress runs older than the threshold. | Run `Cancel Stuck Cloud Storage Deploys` with `dry-run: true`, then without dry run if appropriate. |
## Related pages
- Cloud-storage local build and Lambda notes in `rust/cloud-storage/README.md`
- Pulumi stack authoring and local deploy notes in `infra/README.md`
---
## 28. Observability and debugging
> Operations: Backend tracing initialization, Datadog ECS sidecars, browser RUM and logs, sourcemaps, local debug signals, and iOS WebView freeze diagnosis.
- Page Markdown: https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/28-observability-and-debugging.md
- Generated: 2026-06-01T01:07:41.839Z
### Source Files
- `rust/cloud-storage/macro_entrypoint/src/lib.rs`
- `infra/packages/resources/src/resources/datadog.ts`
- `js/app/packages/observability/src/index.ts`
- `js/app/packages/app/index.tsx`
- `.github/workflows/deploy-web-app.yml`
- `js/app/AGENTS.md`
---
title: "Observability and debugging"
description: "Operations: Backend tracing initialization, Datadog ECS sidecars, browser RUM and logs, sourcemaps, local debug signals, and iOS WebView freeze diagnosis."
---
Macro’s observability surface is split between Rust entrypoint initialization, Pulumi-managed Datadog sidecars for ECS/Fargate services, the `@observability` browser package, and deployment-time sourcemap upload for the web app.
## Runtime map
```text
Rust service
MacroEntrypoint::init()
├─ local: pretty logs or hierarchical tracing tree
└─ dev/prod: JSON logs + OpenTelemetry OTLP -> 127.0.0.1:4317
ECS task
service container
├─ FireLens log driver -> log_router -> Datadog logs intake
└─ OTel spans -> datadog-agent sidecar -> Datadog APM
Web app
app/index.tsx idle import("@observability")
├─ Datadog RUM
├─ Datadog browser logs
└─ sourcemaps uploaded by GitHub Actions after deploy
```
## Backend tracing initialization
Rust binaries should initialize tracing through:
```rust
macro_entrypoint::MacroEntrypoint::default().init();
```
`MacroEntrypoint::default()` loads `.env` if present and resolves `macro_env::Environment` from `ENVIRONMENT`, falling back to production when the variable is missing or invalid. Valid values are:
| `ENVIRONMENT` | Runtime enum |
| --- | --- |
| `prod` | `Environment::Production` |
| `dev` | `Environment::Develop` |
| `local` | `Environment::Local` |
### Local behavior
For `Environment::Local`, the default subscriber uses `tracing_subscriber::fmt()` with:
| Setting | Value |
| --- | --- |
| ANSI | enabled |
| filter | `EnvFilter::from_default_env()` |
| file and line | enabled |
| formatter | pretty |
| OpenTelemetry exporter | disabled |
Set `RUST_LOG` to control local verbosity, for example:
```sh
RUST_LOG=document_storage_service=trace,tower_http=debug
```
For tree-shaped local output, build the entrypoint with `tree_tracing`:
```rust
macro_entrypoint::MacroEntrypoint::new(macro_env::Environment::Local)
.local()
.tree_tracing(Some(4))
.build()
.init();
```
### Develop and production behavior
For `Environment::Develop` and `Environment::Production`, `MacroEntrypoint::init()`:
- installs `tracing_panic::panic_hook`;
- creates an OpenTelemetry OTLP gRPC span exporter pointed at `http://127.0.0.1:4317`;
- sets OpenTelemetry resource attributes from:
- `DD_SERVICE`, defaulting to `unknown-service`;
- `DD_ENV`, defaulting to `unknown`;
- attaches a `tracing_opentelemetry` layer;
- emits JSON logs with current span, flattened event fields, file, and line number;
- injects Datadog correlation fields `dd.trace_id` and `dd.span_id` into each valid span-backed JSON log event.
<Note>
`InitializedEntrypoint::shutdown()` exists to flush the OpenTelemetry tracer provider before process exit. Retain the returned value and call `shutdown()` for short-lived jobs where trace flushing matters.
</Note>
## Datadog ECS sidecars
The shared Datadog containers live in Pulumi resources and are added to ECS tasks by the service components unless a component explicitly opts out with `noDatadog`.
| Container | Image | Purpose |
| --- | --- | --- |
| `log_router` | `amazon/aws-for-fluent-bit:latest` | FireLens log routing to Datadog logs intake |
| `datadog-agent` | `public.ecr.aws/datadog/agent:latest` | APM/trace collection and OTLP receiver |
### `log_router`
The FireLens sidecar is essential, reserves 50 MiB, enables ECS metadata, and uses Fluent Bit’s bundled `/fluent-bit/configs/parse-json.conf`.
Environment:
| Variable | Value |
| --- | --- |
| `ECS_FARGATE` | `true` |
| `DD_API_KEY` | AWS Secrets Manager secret `datadog-api-key` |
| `DD_ENV` | Pulumi stack name |
Service containers route logs with `awsfirelens` to `http-intake.logs.us5.datadoghq.com` and set Datadog options such as `dd_service`, `dd_source=fargate`, `dd_tags`, and `provider=ecs`.
### `datadog-agent`
The Datadog agent sidecar listens for OTLP gRPC spans on port `4317`.
| Variable | Value |
| --- | --- |
| `DD_SITE` | `us5.datadoghq.com` |
| `DD_APM_ENABLED` | `true` |
| `DD_OTLP_CONFIG_RECEIVER_PROTOCOLS_GRPC_ENDPOINT` | `0.0.0.0:4317` |
| `DD_APM_SAMPLE_RATE` | `0.1` in `prod`, `1.0` otherwise |
| `DD_APM_MAX_TPS` | `100` |
| `DD_APM_RECEIVER_SOCKET` | empty string |
The empty `DD_APM_RECEIVER_SOCKET` disables socket buffering in the agent configuration used by this repo.
## Browser RUM and logs
The web app initializes browser observability lazily from `packages/app/index.tsx`. During non-HMR runs, it schedules an idle import of `@observability` and calls:
```ts
Observability.init(import.meta.env.__APP_VERSION__);
```
During `vite dev`, `import.meta.hot` is present, so Datadog browser observability is not injected.
### Datadog browser configuration
`@observability` configures both RUM and browser logs with:
| Field | Value |
| --- | --- |
| `service` | `web-app` |
| `site` | `us5.datadoghq.com` |
| `env` | `prod` when `import.meta.env.MODE === "production"`, otherwise `dev` |
| `version` | `import.meta.env.__APP_VERSION__` by default |
| RUM `sessionSampleRate` | `100` |
| RUM `sessionReplaySampleRate` | `0` |
| RUM `trackViewsManually` | `true` |
| RUM `defaultPrivacyLevel` | `mask` |
| logs `telemetrySampleRate` | `0` |
RUM tracing uses `tracecontext` propagation. In production, tracing is allowed only for selected Macro service hosts: auth, cognition, document storage, email, and notification. In non-production modes, all configured `SERVER_HOSTS` are allowed.
### Event filtering
The browser package drops known noisy events before they leave the client:
| Source | Filter |
| --- | --- |
| RUM resource events | non-200 `unfurl-service` resources |
| RUM errors | `ResizeObserver loop completed with undelivered notifications` |
| browser logs | messages containing `unfurl-service` |
| browser logs | messages containing the same ResizeObserver notification |
### Logging API
Use `@observability` or `@observability/logger` for client-side application logs:
```ts
import { logger } from '@observability';
logger.log('Uploaded file', { documentId });
logger.warn('Retrying request', { attempt });
logger.error('Failed to save draft', { cause: error, draftId });
```
When Datadog is unavailable, not initialized, or the app is running under HMR, the logger falls back to `console.log`, `console.warn`, or `console.error`.
### Routing and actions
`useObserveRouting()` starts manual RUM views from the current route. It derives the view name from the first path segment after `/app`; for `/app/component/:name`, it uses the component name segment instead.
It also records a RUM action named `split changed` when the joined path segments change.
For explicit user actions:
```ts
import { startAction } from '@observability';
startAction('opened command palette', { source: 'hotkey' });
```
## Sourcemaps and release versions
The Vite app build always enables sourcemaps. The app version is computed from `packages/app/package.json` plus the current short Git SHA:
```text
<package-version>+<short-sha>
```
After `vite build`, `postbuild:semver` writes the same value to:
```text
js/app/packages/app/dist/semver.txt
```
Deployment workflows pass Datadog browser credentials into the build, deploy the web app with Pulumi, then upload sourcemaps from `js/app/packages/app`:
```sh
bun run ddupload:dev
bun run ddupload:prod
```
The upload scripts call `datadog-ci sourcemaps upload ./dist` with:
| Environment | `--service` | `--release-version` | `--minified-path-prefix` |
| --- | --- | --- | --- |
| dev | `web-app` | `$(cat ./dist/semver.txt)` | `https://dev.macro.com/app` |
| prod | `web-app` | `$(cat ./dist/semver.txt)` | `https://macro.com/app` |
Production web-app infrastructure excludes `*.js.map` files from the S3 sync, so production sourcemaps are uploaded to Datadog rather than served with the app assets.
## Local debug signals
### Web app startup signals
On startup, the app logs the resolved version:
```text
App Version <version>
```
The root document element also receives runtime attributes useful for local debugging and CSS/state inspection:
| Attribute | Value source |
| --- | --- |
| `data-platform` | `getPlatform()` |
| `data-touch-device` | `isTouchDevice()` |
| `data-modality` | last captured `keydown`, `mousedown`, or `touchstart` |
In development mode, rendering is wrapped in a Solid `ErrorBoundary` that shows `FatalError`. In non-development builds, the root renders directly.
### Dynamic import deployment mismatch
The app registers a `vite:preloadError` listener outside HMR. If a dynamic module load fails after a new version is deployed, the browser shows:
```text
Please refresh page to update app to new version
```
Use this as the expected signal for stale chunks after deployment.
## iOS WebView freeze diagnosis
On iOS WKWebView, eagerly constructing an ES module Web Worker from a `tauri://` custom URL can deadlock the WebContent process. The rule in this repository is:
<Warning>
Do not call `new Worker(...)` at module-load time in code that runs on iOS. Construct workers lazily on first use. The `import Worker from "./worker?worker"` import is safe; instantiating it is the dangerous operation.
</Warning>
A lazy singleton can use a proxy:
```ts
export const svc = new Proxy({} as Service, {
get: (_, p, r) => Reflect.get(Service.getInstance(), p, r),
});
```
### Symptom signature
The app loads HTML, starts initial JavaScript, then JavaScript silently stops. Safari Web Inspector may attach but show no useful page state.
### Diagnosis procedure
<Steps>
<Step title="Reproduce on the iOS Simulator">
Use the simulator so logs are available through macOS unified logging.
```sh
cargo tauri ios dev "iPhone 15"
```
</Step>
<Step title="Stream Macro process logs">
Use the full `/usr/bin/log` path because `zsh` has a `log` builtin.
```sh
/usr/bin/log stream --predicate 'process == "macro"' --info --debug --style compact
```
</Step>
<Step title="Find the last meaningful URL request">
If protocol-handler request logging is enabled, the last `tauri://` request before silence is usually the file being loaded when the freeze occurred.
```sh
tail -200 <logfile> | grep -vE 'tauri:// request|tauri_protocol.rs|^\s*\\134'
```
</Step>
<Step title="Check whether WebContent is parked">
Get the WebContent PID from a `[com.apple.WebKit:...] [...PID=N...]` log line, then inspect CPU.
```sh
ps -o pid,pcpu,comm -p <pid>
```
`0.0%` CPU with a still-alive process points to a deadlock or parked thread, not a hot loop.
</Step>
<Step title="Sample the WebContent process">
```sh
sample <webcontent_pid> 3 -file /tmp/sample.txt
```
Look for `WebCore: Worker` stacks containing:
```text
WorkerOrWorkletScriptController::loadModuleSynchronously
→ WorkerDedicatedRunLoop::runInMode
→ Condition::waitUntilUnchecked
```
</Step>
</Steps>
If the last request was a `*-worker.js?worker_file&type=module` URL and a worker thread is parked in `loadModuleSynchronously`, trace the corresponding `new Worker(...)` call and move construction behind a lazy path.
Do not treat these as primary causes unless the stack confirms them: `NSKeyedArchiver` main-thread warnings, IPC throttling warnings, or failed bundle-updater requests to `localhost:3001`.
## Related pages
- Web app deployment and sourcemap upload
- Backend service deployment on ECS/Fargate
- iOS/Tauri local debugging
---
## 29. Documentation site maintenance
> Contribution: Maintain the Mintlify docs site, navigation manifest, generated MCP pages, local preview, broken-link checks, and writing conventions.
- Page Markdown: https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/29-documentation-site-maintenance.md
- Generated: 2026-06-01T01:06:59.884Z
### Source Files
- `docs/README.md`
- `docs/docs.json`
- `docs/package.json`
- `docs/scripts/generate-mcp-tool-pages.ts`
- `docs/config/tool-pages.json`
- `docs/AGENTS.md`
- `docs/CONTRIBUTING.md`
---
title: "Documentation site maintenance"
description: "Contribution: Maintain the Mintlify docs site, navigation manifest, generated MCP pages, local preview, broken-link checks, and writing conventions."
---
The documentation site lives in `docs/` as a Mintlify project with `docs/docs.json` as the active site manifest, Bun scripts for local operations, and generated MCP tool reference pages under `docs/AI/mcp/tools/`.
## Maintenance surface
```text
docs/
├── docs.json # active Mintlify manifest
├── package.json # dev, lint, generate:tools, prepare
├── README.md # local setup and monorepo deployment notes
├── AGENTS.md # agent-facing writing conventions
├── CONTRIBUTING.md # contributor workflow and style guidance
├── development.mdx # local Mintlify preview guide
├── AI/mcp/tools/ # generated MCP tool reference pages
├── config/tool-pages.json # generated tool page list
├── config/navigation.json # secondary navigation fragment, not the active manifest
└── scripts/
├── generate-mcp-tool-pages.ts
└── generate-changelog.ts
```
<Warning>
The checked-in MCP generator writes to `docs/AI/mcp/tools/`. If a maintenance note mentions `docs/reference/tools/`, verify it against `docs/scripts/generate-mcp-tool-pages.ts` before moving files or changing navigation.
</Warning>
## Local commands
Run documentation commands from `docs/`.
| Task | Command | Notes |
| --- | --- | --- |
| Install dependencies | `bun install` | Uses `docs/package.json` and `docs/bun.lock`. |
| Generate MCP tool pages | `bun run generate:tools` | Runs `bun ./scripts/generate-mcp-tool-pages.ts`. |
| Preview locally | `bun run dev` | Wraps `mint dev`. |
| Check links | `bun run lint` | Wraps `mint broken-links`. |
| Run generator through lifecycle script | `bun run prepare` | Calls `bun run generate:tools`. |
```bash
cd docs
bun install
bun run generate:tools
bun run dev
bun run lint
```
<Note>
Mintlify CLI commands may reject unsupported runtimes. The repository README recommends switching to Node 20 or Node 22 when `mint dev` or `mint broken-links` fails because of the active Node version.
</Note>
## Active site manifest
`docs/docs.json` controls the rendered site structure and global presentation.
| Area | Current value |
| --- | --- |
| Schema | `https://mintlify.com/docs.json` |
| Theme | `mint` |
| Site name | `Macro` |
| Default appearance | `dark` |
| Primary color | `#ff8f05` |
| Favicon | `/favicon.svg` |
| Logos | `/logo/light.svg`, `/logo/dark.svg` |
| Navbar CTA | `Get Macro` → `https://macro.com` |
| Tabs | `Documentation`, `Changelog` |
The `Documentation` tab contains `Getting Started` and `Product` groups. The `AI Chat` product group nests the MCP overview and generated Tool Reference pages. The `Changelog` tab points at `changelog/introduction`.
When adding a handwritten page:
1. Create the `.mdx` file under the matching content folder.
2. Add the route without the `.mdx` extension to `docs/docs.json`.
3. Use root-relative internal links such as `/product/ai-chat`.
4. Run `bun run lint`.
<Warning>
`docs/config/navigation.json` contains a `$ref` to `config/tool-pages.json`, but the active manifest in this checkout is `docs/docs.json`. Keep `docs/docs.json` synchronized unless the site is intentionally migrated to consume the navigation fragment.
</Warning>
## Generated MCP tool pages
`docs/scripts/generate-mcp-tool-pages.ts` rebuilds tool docs from the Rust tool schema registry.
### Inputs and outputs
| Path | Role |
| --- | --- |
| `rust/cloud-storage/ai_tools` | Rust crate used to build and run the schema generator. |
| `rust/cloud-storage/ai_tools/src/bin/gen_tool_schemas.rs` | Binary that writes `schemas/tools.json`. |
| `rust/cloud-storage/ai_tools/schemas/tools.json` | Intermediate generated schema file, ignored by `rust/cloud-storage/.gitignore`. |
| `docs/AI/mcp/tools/*.mdx` | Generated MDX pages and generated index page. |
| `docs/config/tool-pages.json` | Generated route list for tool navigation. |
| `docs/docs.json` | Active navigation manifest that must expose generated pages. |
### Generation lifecycle
<Steps>
<Step title="Build the Rust schema binary">
The script runs Cargo from `rust/cloud-storage` with offline SQLx mode:
```bash
SQLX_OFFLINE=true cargo build --bin gen_tool_schemas
```
</Step>
<Step title="Regenerate the schema JSON">
The script removes `rust/cloud-storage/ai_tools/schemas`, then executes the compiled `gen_tool_schemas` binary from `rust/cloud-storage/ai_tools`.
Expected binary output:
```text
Generated ai_tools/schemas/tools.json
```
</Step>
<Step title="Rewrite generated MDX pages">
The script deletes and recreates `docs/AI/mcp/tools/`, sorts schemas by `name`, slugifies each tool name, writes one page per tool, and writes `docs/AI/mcp/tools/index.mdx`.
</Step>
<Step title="Rewrite the generated page list">
The script writes `docs/config/tool-pages.json` with routes such as:
```json
[
"AI/mcp/tools/index",
"AI/mcp/tools/content-search"
]
```
</Step>
</Steps>
### Slug and page rules
| Rule | Behavior |
| --- | --- |
| Sorting | Tool schemas are sorted with `name.localeCompare`. |
| Slug conversion | CamelCase boundaries become hyphens; underscores and whitespace become hyphens; output is lowercase. |
| Page title | Uses the tool schema `name`. |
| Page description | Uses the tool schema `description`, or `Generated from the Macro Rust tool registry.` |
| Parameters table | Built from `inputSchema.properties`; required fields come from `inputSchema.required`. |
| Output schema | Loaded but not rendered into the generated MDX pages. |
Example generated route mapping:
| Tool name | Route |
| --- | --- |
| `ContentSearch` | `/AI/mcp/tools/content-search` |
| `text_editor_code_execution` | `/AI/mcp/tools/text-editor-code-execution` |
| `web_fetch` | `/AI/mcp/tools/web-fetch` |
## Updating MCP documentation after Rust tool changes
Use this checklist when a Rust tool name, schema, field, or description changes.
<Steps>
<Step title="Update the Rust tool schema source">
Edit the Rust tool definition, schema annotations, or tool registry entry that owns the tool behavior.
</Step>
<Step title="Regenerate docs">
```bash
cd docs
bun run generate:tools
```
</Step>
<Step title="Review generated diffs">
Check these paths first:
```text
docs/AI/mcp/tools/
docs/config/tool-pages.json
```
If a tool was added, removed, or renamed, update the Tool Reference page list in `docs/docs.json`.
</Step>
<Step title="Validate local docs">
```bash
bun run lint
bun run dev
```
Open the local preview and verify the MCP overview, Tool Reference index, and the changed tool page.
</Step>
</Steps>
## Broken-link checks
`bun run lint` runs `mint broken-links`. Use it after:
- adding, moving, or deleting an MDX page;
- changing `docs/docs.json` navigation;
- regenerating MCP tool pages;
- changing root-relative links;
- changing changelog output.
If `mint broken-links` fails after generated tool changes, check for one of these causes:
| Symptom | Likely cause | Fix |
| --- | --- | --- |
| Tool page exists but is missing from sidebar | `docs/docs.json` was not updated | Add the route under the Tool Reference group. |
| Sidebar route points to a deleted file | Tool was renamed or removed | Remove or replace the stale route in `docs/docs.json`. |
| Link works in generated index but not navigation | `config/tool-pages.json` and `docs/docs.json` diverged | Sync the active manifest. |
| CLI fails before checking links | Unsupported Node runtime | Switch to Node 20 or Node 22 and rerun. |
## Writing conventions
Documentation files are MDX with YAML frontmatter. Follow the repository’s current contribution and agent guidance:
| Convention | Rule |
| --- | --- |
| Voice | Use active voice. |
| Reader address | Use second person in procedures. |
| Sentences | Keep sentences concise; one idea per sentence. |
| Headings | Use sentence case. |
| UI labels | Bold UI labels, for example **Settings**. |
| Technical identifiers | Use code formatting for file names, commands, paths, and code references. |
| Terminology | Do not alternate between synonyms for the same product concept. |
| Examples | Include concrete examples when documenting behavior or commands. |
Use existing nearby pages as the shape reference. Product pages live under `docs/product/`, MCP setup lives under `docs/AI/mcp/`, and generated tool reference pages should not be manually polished in place because the generator rewrites them.
## Changelog generation
`docs/scripts/generate-changelog.ts` is separate from the MCP generator. It fetches GitHub releases for `macro-inc/macro`, keeps tags matching `vYYYY.M.D.N`, rewrites `docs/changelog/introduction.mdx`, and rewrites the `Changelog` tab in `docs/docs.json`.
Operational constraints:
- It uses `GITHUB_TOKEN` when present.
- It can also rely on GitHub CLI authentication according to its file header.
- It removes old per-release MDX files in `docs/changelog/` except `introduction.mdx`.
- It escapes unsupported MDX angle brackets before writing release bodies.
Run it only when intentionally refreshing changelog content from GitHub releases.
## Deployment configuration
For Mintlify’s monorepo setup, configure the docs path as:
```text
/docs
```
Keep deployment-facing changes inside `docs/` unless the generated MCP pages require Rust schema changes. The docs project is independent from model-provider configuration: MCP reference generation reads repository-local Rust schemas and does not require an AI provider key.
## Related pages
<CardGroup cols={2}>
<Card title="MCP setup" href="/AI/mcp/overview">
Connect AI clients to the Macro MCP endpoint.
</Card>
<Card title="Tool reference" href="/AI/mcp/tools">
Browse generated MCP tool pages.
</Card>
<Card title="Development" href="/development">
Preview and validate the Mintlify site locally.
</Card>
<Card title="Changelog" href="/changelog/introduction">
Review release notes generated from GitHub releases.
</Card>
</CardGroup>
---