# Run backend services locally

> Guide: Start databases, LocalStack, FusionAuth, Rust services, optional processor profiles, and local health checks.

- Repository: macro-inc/macro
- GitHub: https://github.com/macro-inc/macro
- Human docs: https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e
- Complete Markdown: https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/llms-full.txt

## Source Files

- `justfile`
- `docker-compose.yml`
- `docker-compose-databases.yml`
- `docker-compose.local-e2e.yml`
- `local_stack.just`
- `rust/cloud-storage/justfile`

---

---
title: "Run backend services locally"
description: "Guide: Start databases, LocalStack, FusionAuth, Rust services, optional processor profiles, and local health checks."
---

The local backend runtime is owned by `just` recipes and Docker Compose files at the repository root. `just setup` prepares `.env`, external Docker networks and volumes, LocalStack AWS resources, the local Postgres database, FusionAuth configuration, and prebuilt Rust service images; `just run_local` then starts the Compose stack.

<Warning>
Local Docker resources are intentionally single-instance. The Compose project is fixed to `macro`, so multiple checkouts or worktrees share the same containers, volumes, networks, LocalStack container, and FusionAuth instance.
</Warning>

## Prerequisites

Install the tools used directly by the local recipes:

| Tool | Used by |
| --- | --- |
| `just` | Repository command runner |
| Docker and Docker Compose | Databases, FusionAuth, Rust services, JS services |
| `sops` | Decrypting `.env-local*.enc` into `.env` |
| AWS CLI | Creating LocalStack SQS, DynamoDB, and S3 resources |
| Pulumi | Local FusionAuth stack outputs and configuration |
| SQLx CLI | Database create, migrate, setup, and reset commands |
| Bun and Node | FusionAuth setup dependencies and JS-backed services/tests |

If not using the repository shell environment, export `SOPS_KMS_ARN` before decrypting local environment files.

```bash
export SOPS_KMS_ARN="arn:aws:kms:us-east-1:569036502058:key/mrk-cab29bf948044eb79005a81f48d40e93,arn:aws:kms:us-west-1:569036502058:key/mrk-cab29bf948044eb79005a81f48d40e93"
```

## One-command setup

```bash
just setup
```

`just setup` performs this sequence:

1. Decrypts the root local environment into `.env`.
2. Creates external Docker networks: `databases` and `auth`.
3. Creates persistent local volumes for Postgres, Redis, OpenSearch, and FusionAuth.
4. Starts and provisions LocalStack.
5. Creates and migrates the local Macro Postgres database.
6. Starts FusionAuth, deploys the local Pulumi stack, patches root `.env`, then stops FusionAuth.
7. Builds the development Rust service images.

<Note>
`setup_local_dbs` starts Postgres and Redis to initialize the database, then stops the database Compose stack. `run_local` starts the services again when needed.
</Note>

## Start only infrastructure

Use these commands when you want to rebuild or inspect dependencies before starting application services.

<Steps>
  <Step title="Create shared Docker resources">
    ```bash
    just create_networks
    ```
  </Step>

  <Step title="Start Postgres and Redis for database work">
    ```bash
    just run_dbs -d
    ```

    This targets only `postgres` and `redis` from `docker-compose-databases.yml`.
  </Step>

  <Step title="Create or migrate the Macro database">
    ```bash
    just rust/cloud-storage/macro_db_client/create_db
    just rust/cloud-storage/macro_db_client/migrate_db
    ```

    The local database URL is `postgres://user:password@localhost:5432/macrodb`.
  </Step>

  <Step title="Provision LocalStack">
    ```bash
    AWS_ACCESS_KEY_ID=test AWS_SECRET_ACCESS_KEY=test AWS_DEFAULT_REGION=us-east-1 just setup_localstack
    ```
  </Step>

  <Step title="Set up FusionAuth">
    ```bash
    just setup_fusionauth
    ```
  </Step>
</Steps>

## Local infrastructure endpoints

| Component | Container/image | Host endpoint | Notes |
| --- | --- | --- | --- |
| Postgres | `pgvector/pgvector:pg16` | `localhost:5432` | User `user`, password `password`, database `macrodb` after setup |
| Redis Stack | `redis/redis-stack:latest` | `localhost:6379`, UI/tools on `localhost:8001` | Uses persistent `macro_redis_data` volume |
| OpenSearch | `opensearchproject/opensearch:latest` | `localhost:9200`, analyzer on `localhost:9600` | Defined in the database Compose file; not targeted by `just run_dbs` |
| LocalStack | `localstack/localstack:4` | `localhost:4566` | Runs SQS, DynamoDB, and S3 on the `databases` network |
| FusionAuth | `fusionauth/fusionauth-app:1.62.1` | `localhost:9011` | Exposed to backend services through the external `auth` network |

## LocalStack resources

`just setup_localstack` starts LocalStack with `SERVICES=sqs,dynamodb,s3`, waits for `/_localstack/health`, then creates queues, tables, buckets, and a document-upload S3 notification.

### S3 buckets

| Bucket | Used for |
| --- | --- |
| `macro-email-attachments` | Email attachments |
| `doc-storage` | Document storage and document-upload finalizer events |
| `docx-upload` | DOCX upload flow |
| `static-file-storage` | Static file objects |
| `bulk-upload-staging` | Bulk upload staging |

All buckets receive a local CORS configuration allowing `http://localhost:3000` through `http://localhost:3009`.

### DynamoDB tables

| Table | Key shape |
| --- | --- |
| `bulk-upload` | `PK` hash, `SK` range, `DocumentPkIndex` GSI |
| `connection-gateway-table` | `PK` hash, `SK` range, `ConnectionPkIndex` GSI |
| `static-file-metadata` | `file_id` hash |

### SQS queues

Local setup creates queues for notifications, email jobs, contacts, conversion, document deletion, document text extraction, search events, static-file events, and document-upload finalization. The document-upload finalizer wires S3 `ObjectCreated` events from `doc-storage` to `document-upload-finalizer-queue`.

Inside Docker, services use `http://localstack:4566`. Host-side commands and generated browser-facing LocalStack URLs use `http://localhost:4566`.

## FusionAuth local setup

FusionAuth local setup is split between Docker Compose and a Pulumi stack under `infra/stacks/fusionauth-instance`.

```bash
just setup_fusionauth
```

The setup recipe:

1. Downloads the FusionAuth container `.env` if missing.
2. Starts the FusionAuth Postgres database and app.
3. Waits for `http://localhost:9011/api/status` to report `{"status":"Ok"}`.
4. Initializes or updates the Pulumi `local` stack.
5. Writes local FusionAuth values into the root `.env`.
6. Stops FusionAuth after setup.

Useful local credentials:

| Field | Value |
| --- | --- |
| Admin username | `admin@macro.com` |
| Admin password | `macroIsGreat!` |
| API key | `bf69486b-4733-4954-a44e-2e1b5f2c8a91` |

`just run_local` calls `patch_local_fusionauth_env` before starting services. If FusionAuth is not already running, the patch recipe starts it temporarily, reads the client secret, updates `.env`, and stops the temporary container afterward.

## Start backend services

```bash
just run_local
```

By default, this runs `docker compose up` in the foreground. Pass Docker Compose arguments through `just` when you want detached startup, waiting, or a subset of services.

```bash
just run_local -d --wait
```

For a clean rebuild after changing service code:

```bash
just run_local --build
```

`run_local` always builds the shared Rust services image target `rust_services_image`. With `--build`, it also rebuilds `websocket_service`, `sync_service`, and `lexical_service`.

## Service ports and health endpoints

| Service | Host port | Health check | Runtime notes |
| --- | ---: | --- | --- |
| `authentication-service` | `8080` | `/health` | Depends on Postgres, FusionAuth, and Redis |
| `connection_gateway` | `8082` | `/health` | WebSocket gateway; service alias `connection-gateway` |
| `contacts_service` | `8083` | `/health` | Contact management service |
| `document_cognition_service` | `8085` | `/health` | Depends on document storage, email, static files, sync, and lexical |
| `document_storage_service` | `8086` | `/health` | Depends on Redis, connection gateway, and auth |
| `document_upload_finalizer` | none | none | Local SQS worker for `doc-storage` object-created events |
| `email_service` | `8087` | `/health` | Depends on auth, document storage, connection gateway, static files, and Redis |
| `notification_service` | `8089` | `/health` | Depends on Redis, cognition, auth, connection gateway, and document storage |
| `search_processing_service` | `8092` | `/health` | Optional `processors` profile |
| `static_file_service` | `8094` | `/api/health` | Uses DynamoDB/static-file metadata |
| `static_file_cdn` | `8100` | none | Nginx local CloudFront-style emulator |
| `unfurl_service` | `8095` | `/health` | URL unfurling service |
| `image_proxy_service` | `8097` | `/health` | External image proxy |
| `websocket_service` | `6969` | none | JS WebSocket service |
| `sync_service` | `8787` | `/health` | Cloudflare Worker/Durable Object sync runtime via Docker |
| `lexical_service` | `8096` | `/health` | Lexical conversion service |

Check the stack:

```bash
docker compose ps
curl -fsS http://localhost:8080/health
curl -fsS http://localhost:8086/health
curl -fsS http://localhost:8094/api/health
curl -fsS http://localhost:8787/health
curl -fsS http://localhost:9011/api/status
curl -fsS http://localhost:4566/_localstack/health
```

## Optional processor profile

`search_processing_service` is behind the Compose profile `processors`.

```bash
just run_local --profile processors
```

Rebuild it explicitly when changing processor code:

```bash
just run_local --build --profile processors
```

The processor image uses `Dockerfile.search_processing_service.dev` and is pinned to `linux/amd64` because the bundled `search_processing_service/pdfium-lib/linux/libpdfium.so` is amd64-only. Apple Silicon hosts run this service through emulation.

## Local E2E backend profile

The local E2E commands start LocalStack, start a narrowed backend service set with `docker-compose.local-e2e.yml` overrides, seed deterministic data, then run tests.

```bash
just local-e2e
```

Rust ignored integration tests use the same seeded stack:

```bash
just local-e2e-rust
```

Run Rust integration tests and Playwright after one stack startup and seed pass:

```bash
just local-e2e-all
```

Open Playwright UI mode:

```bash
just local-e2e-ui
```

The E2E override file forces services to use local dependencies instead of shared development infrastructure:

| Key | Local value |
| --- | --- |
| `DATABASE_URL` | `postgres://user:password@postgres:5432/macrodb` |
| `DATABASE_URL_READONLY` | `postgres://user:password@postgres:5432/macrodb` |
| `LOCAL_AWS_URL` | `http://localstack:4566` |
| `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` | `test` / `test` |
| `REDIS_URI` | `redis://redis:6379` |
| `DOCUMENT_STORAGE_SERVICE_REDIS_URI` | `redis://redis:6379` |

The deterministic seed command is guarded. It requires `LOCAL_E2E_SEED=true` and refuses database URLs outside the local Docker database shape `postgres://user:...@(localhost|127.0.0.1|postgres):5432/macrodb`.

## Stop and reset

| Command | Effect |
| --- | --- |
| `just stop-local` | Runs `docker compose down` for the main local stack |
| `just stop-databases` | Stops the database Compose stack |
| `just stop_fusionauth` | Stops the FusionAuth Compose stack |
| `docker rm -f localstack` | Removes the LocalStack container |
| `just rust/cloud-storage/macro_db_client/reset_db` | Drops and recreates the local Macro database |
| `just docker_cache_usage` | Shows BuildKit cache disk usage |
| `just docker_cache_clear_targets` | Clears Rust target cache mounts |
| `just docker_cache_clear` | Clears all BuildKit build cache |

## Troubleshooting

### `.env not found`

Run:

```bash
just get_environment
```

For a fresh checkout, prefer:

```bash
just setup
```

### FusionAuth local stack is missing

If `run_local` prints a warning that the Pulumi local stack was not found, run:

```bash
just setup_fusionauth
```

Then restart the backend stack.

### LocalStack resources are missing

Recreate LocalStack resources:

```bash
AWS_ACCESS_KEY_ID=test AWS_SECRET_ACCESS_KEY=test AWS_DEFAULT_REGION=us-east-1 just setup_localstack
```

The recipe uses `localstack/localstack:4`; it is intentionally not `latest`.

### Processor builds are slow on Apple Silicon

`search_processing_service` runs as `linux/amd64` because of the bundled PDFium library. Use the default stack unless you need the `processors` profile.

### Local E2E seed refuses to run

Check that the seed command has `LOCAL_E2E_SEED=true` and that `DATABASE_URL` points at the local Compose database, not a shared development database.

## Related pages

<CardGroup>
  <Card title="Frontend local E2E" href="/js-app-local-e2e">
    Run Playwright against the local backend stack and seeded fixtures.
  </Card>
  <Card title="Seed CLI" href="/seed-cli">
    Populate local Macro data with deterministic scenarios.
  </Card>
  <Card title="FusionAuth instance stack" href="/fusionauth-instance-stack">
    Maintain the local FusionAuth Pulumi stack and Docker runtime.
  </Card>
</CardGroup>