# Service topology

> Concept: Rust services, workers, queues, storage dependencies, ports, and deployment boundaries used by the local and cloud stacks.

- Repository: macro-inc/macro
- GitHub: https://github.com/macro-inc/macro
- Human docs: https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e
- Complete Markdown: https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/llms-full.txt

## Source Files

- `docker-compose.yml`
- `rust/cloud-storage/Cargo.toml`
- `rust/cloud-storage/AGENTS.md`
- `infra/stacks/document-storage/index.ts`
- `infra/packages/resources/src/index.ts`
- `infra/packages/shared/src/ai_tools.ts`

---

---
title: "Service topology"
description: "Concept: Rust services, workers, queues, storage dependencies, ports, and deployment boundaries used by the local and cloud stacks."
---

`macro-inc/macro` runs the cloud-storage surface as a Rust Cargo workspace deployed either as local Docker Compose containers or as Pulumi-managed AWS ECS/Fargate services, Lambda handlers, SQS queues, S3 buckets, DynamoDB tables, Redis caches, OpenSearch, and Postgres databases.

## Runtime boundaries

The repository has two primary topology descriptions:

| Boundary | Source of truth | Runtime shape |
| --- | --- | --- |
| Local stack | `docker-compose.yml`, `docker-compose-databases.yml`, `docker-compose.local-e2e.yml` | Docker services on bridge networks, one shared Rust service image, local Postgres/Redis/OpenSearch, FusionAuth compose include |
| Cloud stack | `infra/stacks/**`, `infra/packages/resources/**`, `infra/packages/shared/**` | Pulumi stacks that create ECS/Fargate services, ALBs, Route53 records, SQS queues, Lambda handlers, S3 buckets, DynamoDB tables, Redis, OpenSearch, and RDS |
| Rust service workspace | `rust/cloud-storage/Cargo.toml` | One workspace containing HTTP services, queue workers, Lambda handlers, clients, models, and shared infrastructure crates |

<Note>
Most Rust HTTP binaries default to `PORT=8080`. Local Compose maps each container port to a distinct host port; cloud ECS services keep container port `8080` behind HTTPS ALBs.
</Note>

## Local Docker topology

`docker-compose.yml` freezes the Compose project name to `macro`, includes the database and FusionAuth compose files, and builds `macro-local-rust-services:dev` from `rust/cloud-storage/Dockerfile.dev`.

The dev Dockerfile builds a bundle of standard Rust binaries into `/app/out/*` and each service chooses a command from that bundle. `search_processing_service` uses a separate Dockerfile and is pinned to `linux/amd64` because its Pdfium library is amd64-only.

| Compose service | Host port | Container command or runtime | Main dependencies |
| --- | ---: | --- | --- |
| `authentication-service` | `8080` | `/app/out/authentication_service` | Postgres, FusionAuth, Redis |
| `connection_gateway` | `8082` | `/app/out/connection_gateway_service` | Redis |
| `contacts_service` | `8083` | `/app/out/contacts_service` | Postgres/Redis via env |
| `document_cognition_service` | `8085` | `/app/out/document_cognition_service` | Document storage, email, static file, sync, lexical |
| `document_storage_service` | `8086` | `/app/out/document_storage_service` | Redis, connection gateway, auth |
| `document_upload_finalizer` | none | `/app/out/document_upload_finalizer_local_worker` | Postgres, sync, lexical; polls LocalStack SQS |
| `email_service` | `8087` | `/app/out/email_service` | Auth, document storage, connection gateway, static file, Redis |
| `notification_service` | `8089` | `/app/out/notification_service` | Redis, cognition, auth, connection gateway, document storage |
| `search_processing_service` | `8092` | `Dockerfile.search_processing_service.dev` | Email service; profile `processors` |
| `static_file_service` | `8094` | `/app/out/static_file_service` | DynamoDB/S3-style env |
| `static_file_cdn` | `8100` | `nginx:alpine` | Routes `/file/*` to LocalStack S3 and `/api/*` to `static_file_service` |
| `unfurl_service` | `8095` | `/app/out/unfurl_service` | No Compose dependency |
| `image_proxy_service` | `8097` | `/app/out/image_proxy_service` | No Compose dependency |
| `websocket_service` | `6969` | JS WebSocket service | Services network |
| `sync_service` | `8787` | Rust sync-service Dockerfile | Internal API/document permission secrets |
| `lexical_service` | `8096` | Bun/JS lexical-service Dockerfile | Sync service |

### Local infrastructure containers

`docker-compose-databases.yml` provides:

| Service | Ports | Notes |
| --- | ---: | --- |
| `postgres` | `5432` | `pgvector/pgvector:pg16`, external volume `macro_postgres_data` |
| `redis` | `6379`, `8001` | `redis/redis-stack:latest`, external volume `macro_redis_data` |
| `search` | `9200`, `9600` | OpenSearch single-node with security disabled |
| `fusionauth` | `9011` | Included from `infra/stacks/fusionauth-instance/docker-compose.yml`; uses its own Postgres container |

Local networks separate service traffic:

| Network | Purpose |
| --- | --- |
| `services` | Service-to-service HTTP/WebSocket traffic |
| `databases` | Postgres, Redis, OpenSearch, LocalStack-style AWS endpoints |
| `auth` | FusionAuth access from the auth service |
| `auth-internal` | FusionAuth internal database link |

<Warning>
The local document upload finalizer defaults `LOCAL_AWS_URL` to `http://localstack:4566`, but the base Compose files do not define a `localstack` service. Local E2E overrides assume that endpoint exists and provide deterministic bucket, table, and queue names.
</Warning>

## Cloud service boundary

Cloud services are provisioned with Pulumi stacks under `infra/stacks`. Public Rust HTTP services typically follow the same pattern:

1. Build a service-specific ECR image from `rust/cloud-storage/Dockerfile` with `SERVICE_NAME`.
2. Create an ECS/Fargate service in private subnets.
3. Put an Application Load Balancer in public subnets when `isPrivate: false`.
4. Terminate HTTPS on port `443`; redirect HTTP `80` to HTTPS.
5. Forward ALB traffic to container port `8080`.
6. Attach service-specific IAM policies for Secrets Manager, SQS, S3, DynamoDB, SNS, Lambda, or OpenSearch.

```mermaid
flowchart LR
  subgraph Local["Local Compose"]
    BrowserLocal["Host/browser"]
    ComposePorts["Host port mappings"]
    RustBundle["macro-local-rust-services:dev"]
    LocalDB["Postgres + Redis + OpenSearch"]
    Fusion["FusionAuth"]
    LocalS3["LocalStack-compatible AWS endpoint"]
  end

  subgraph Cloud["Pulumi AWS stacks"]
    Route53["Route53 + HTTPS ALB"]
    ECS["ECS/Fargate Rust services"]
    Workers["ECS workers + Lambda handlers"]
    Queues["SQS queues + DLQs"]
    Storage["S3 buckets + DynamoDB"]
    Data["RDS MacroDB + Redis + OpenSearch"]
  end

  BrowserLocal --> ComposePorts --> RustBundle
  RustBundle --> LocalDB
  RustBundle --> Fusion
  RustBundle --> LocalS3

  Route53 --> ECS
  ECS --> Data
  ECS --> Storage
  ECS --> Queues
  Queues --> Workers
  Storage --> Workers
```

## Core cloud stacks

| Stack | Main resources | Deployment boundary |
| --- | --- | --- |
| `document-storage` | Versioned document S3 bucket, lifecycle rules, replication bucket, ECS cluster `cloud-storage-cluster-${stack}` | Shared storage and cluster foundation for many Rust services |
| `cloud-storage-service` | `document_storage_service` ECS service, DOCX upload bucket, delete document/chat queues, docx unzip Lambda, document upload finalizer Lambda | Main document API plus document upload/conversion event handlers |
| `document-storage-bucket-integrations` | EventBridge S3 Object Created rules and DLQs | Connects document bucket events to search upload, text extraction, and upload finalization Lambdas |
| `document-cognition-service` | `document_cognition_service` ECS service | AI/document cognition API and tool host |
| `email-service` | `email_service` ECS API, `email-service-pubsub-workers` ECS worker service, Redis, attachment bucket, CloudFront, many SQS queues, refresh/scheduled Lambdas | Email API plus queue consumers |
| `connection-gateway` | `connection_gateway` ECS service, Redis, DynamoDB connection table | Realtime connection tracking and message gateway |
| `contacts-service` | `contacts_service` ECS service, contacts queue | Contact API plus SQS/outbox workers |
| `notification-service` | `notification_service` ECS service, notification queues, SNS platform applications, push event handler | Notification API and push/event workers |
| `static-file-service` | `static_file_service` ECS service, static-file S3 bucket, CloudFront, DynamoDB metadata table, S3 event queue, image optimizer Lambda | Static file API/CDN boundary |
| `search-processing-service` | `search_processing_service` ECS service, backfill job DynamoDB table | Search indexing workers plus backfill API |
| `convert-service` | `convert_service` ECS service, convert queue | LibreOffice-based conversion API/worker |
| `opensearch` | OpenSearch domain | Search index storage |
| `macrodb` | RDS Postgres primary and read replica | MacroDB storage boundary |

## Service URLs and stack routing

Cloud service URL values are centralized in `ServiceUrl` and resolved by stack (`dev` or `prod`). Pulumi stacks inject these values as environment variables instead of hard-coding peer endpoints inside Rust binaries.

| Env key | Dev URL | Prod URL |
| --- | --- | --- |
| `DOCUMENT_STORAGE_SERVICE_URL` | `https://cloud-storage-dev.macro.com` | `https://cloud-storage.macro.com` |
| `AUTHENTICATION_SERVICE_URL` | `https://auth-service-dev.macro.com` | `https://auth-service.macro.com` |
| `CONNECTION_GATEWAY_URL` | `https://connection-gateway-dev.macro.com` | `https://connection-gateway.macro.com` |
| `DOCUMENT_COGNITION_SERVICE_URL` | `https://document-cognition-dev.macro.com` | `https://document-cognition.macro.com` |
| `EMAIL_SERVICE_URL` | `https://email-service-dev.macro.com` | `https://email-service.macro.com` |
| `STATIC_FILE_SERVICE_URL` | `https://static-file-service-dev.macro.com` | `https://static-file-service.macro.com` |
| `NOTIFICATION_SERVICE_URL` | `https://notifications-dev.macro.com` | `https://notifications.macro.com` |
| `SYNC_SERVICE_URL` | `https://sync-service-dev3.macroverse.workers.dev` | `https://sync-service-prod2.macroverse.workers.dev` |
| `LEXICAL_SERVICE_URL` | `https://lexical-service-dev.macroverse.workers.dev` | `https://lexical-service.macroverse.workers.dev` |

## Storage dependencies

| Storage | Local implementation | Cloud implementation | Primary consumers |
| --- | --- | --- | --- |
| MacroDB | Postgres `pgvector/pgvector:pg16` on `5432` | RDS Postgres, plus read replica in `macrodb` stack | Document storage, auth, email, contacts, notification, cognition, search processing |
| Redis | Redis Stack on `6379` | ElastiCache Redis or secret-backed shared cache, depending on stack | Auth sessions/rate limits, connection gateway, email rate limits, contacts, notification |
| OpenSearch | Single-node OpenSearch on `9200` | OpenSearch domain with prod VPC placement and multi-AZ settings | Search processing, document storage search paths |
| Document files | LocalStack-compatible S3 endpoint when configured | Versioned S3 bucket from `document-storage` stack | Document storage, search/text extraction, upload finalizer |
| Static files | LocalStack bucket through `static_file_cdn` | `static-file-storage-${stack}` S3 bucket + CloudFront | Static file service, image optimizer |
| Connection state | Local DynamoDB-style table name from env overrides | DynamoDB connection table in `connection-gateway` | Connection gateway, cognition realtime integration |
| Metadata tables | Local DynamoDB-style env names | DynamoDB tables such as `static-file-metadata-${stack}` and search backfill jobs | Static file service, search processing |

## Queues, workers, and event flows

| Queue or event | Producer | Consumer | Notes |
| --- | --- | --- | --- |
| `search-event-queue-${stack}` | Document storage, email, cognition, auth flows | `search_processing_service` | Cloud queue has DLQ and production backlog/age alarms |
| Document text extractor queue | Document cognition | `document_text_extractor` Lambda | Lambda queue has DLQ and event source mapping with batch size `1` |
| Document bucket Object Created | S3 EventBridge | Search upload Lambda, text extractor Lambda, document upload finalizer Lambda | `document-storage-bucket-integrations` creates EventBridge rules and DLQs |
| `document-upload-finalizer-queue` local | LocalStack S3 notification | `document_upload_finalizer_local_worker` | Local worker long-polls SQS, processes up to 10 messages, deletes only after success |
| `convert-service-queue-${stack}` | DOCX unzip/document flows | `convert_service` | Convert service also exposes HTTP health/API and runs a queue worker unless built with `disable_worker` |
| Email queues | Gmail webhooks, scheduled sends, backfill, SFS mapping, link manager | `email_service`, `email-service-pubsub-workers`, refresh/scheduled Lambdas | Email stack owns multiple queues and exports names for other stacks |
| `contacts-queue-${stack}` | Document/email/channel flows | `contacts_service` worker | Contacts service also runs an outbox worker |
| Notification ingress/egress queues | Document/email/cognition/auth flows | `notification_service` | Notification service also integrates SNS platform applications |
| Static file S3 event queue | Static file bucket notifications | `static_file_service` | Used for static file metadata/event handling |

## Document upload and processing path

```mermaid
sequenceDiagram
  participant Client
  participant DSS as document_storage_service
  participant S3 as document-storage S3 bucket
  participant EB as EventBridge/SQS
  participant Finalizer as document_upload_finalizer
  participant Extractor as document_text_extractor
  participant Search as search_processing_service
  participant OS as OpenSearch

  Client->>DSS: Request document upload/update
  DSS->>S3: Write object or issue presigned upload path
  S3-->>EB: Object Created
  EB-->>Finalizer: Finalize versioned object
  EB-->>Extractor: Extract document text
  DSS-->>EB: Publish search event when applicable
  EB-->>Search: Search event queue message
  Search->>S3: Read source content
  Search->>OS: Index searchable representation
```

Local development replaces the EventBridge Lambda path with the `document_upload_finalizer_local_worker` polling a LocalStack SQS queue. Cloud uses EventBridge rules and Lambda targets with DLQs.

## AI tool hosting topology

`getAiToolsInfra()` defines the shared infrastructure contract for services that host the Rust `ai_tools` crate. It returns:

| Field | Meaning |
| --- | --- |
| `envVars` | Service URLs, bucket names, queue names, CloudFront signer names, and internal auth values required by `ai_tools` context construction |
| `secretArns` | Secrets Manager ARNs for sync auth, CloudFront signing, and MCP credentials |
| `queueArns` | Queue access required by hosted tools, including email scheduled queue |
| `bucketArns` | Document storage and DOCX upload bucket permissions |

`getAiToolsServiceRoleArns()` centralizes the IAM role ARNs for tool-hosting services such as MCP server, document cognition, and agent schedule service. Resource-side policies can grant access to that set without coupling the topology to one model provider.

<Info>
The AI tool infrastructure contract is provider-neutral at the topology layer: services receive AWS resources, service URLs, secrets, queues, and buckets. Model-provider keys are service-specific environment or secret inputs, not a requirement of the shared tool-hosting boundary.
</Info>

## Health checks and verification signals

| Runtime | Health path |
| --- | --- |
| Most Rust HTTP services | `/health` |
| `static_file_service` | `/api/health` |
| FusionAuth local container | `/api/status` |
| `sync_service` | `/health` on port `8787` |
| `lexical_service` | `/health` on port `8096` |

Local Compose health checks use the container port, usually `8080`, even when the host port differs. Cloud ALB target groups use the configured `healthCheckPath` and forward to the ECS task container port.

## Operations notes

### Local startup constraints

- Create the external Docker networks and volumes expected by Compose before starting the full stack.
- Use the `processors` profile only when the local machine can run the amd64 search processing image or QEMU emulation.
- Use `COMPOSE_FILE=docker-compose.yml:docker-compose.local-e2e.yml` for deterministic local E2E environment overrides.
- Ensure a LocalStack-compatible endpoint exists when using S3/SQS-backed local flows.

### Cloud deployment constraints

- New service environment variables must be added to the matching Pulumi stack so ECS tasks or Lambda handlers receive them.
- Services that call AWS APIs need both runtime environment values and IAM policy coverage.
- Queue-backed workflows should define DLQs and alarms; the shared `Queue` component creates a queue, DLQ, and DLQ alarm by default.
- Rust services use `SQLX_OFFLINE=true` during Docker builds; SQLx query metadata must be prepared from the Rust workspace when database queries change.

## Related pages

<CardGroup>
  <Card title="Document storage" href="/architecture/document-storage">
    Document API, bucket layout, upload/finalization flow, and storage permissions.
  </Card>
  <Card title="Search indexing" href="/architecture/search-indexing">
    Search event queues, extraction workers, OpenSearch indexes, and backfill operations.
  </Card>
  <Card title="Local development stack" href="/development/local-stack">
    Compose startup, environment overrides, LocalStack setup, and health checks.
  </Card>
</CardGroup>