# Deploy services and web app

> Operations: Generic service deployment, web app deployment, production release workflow, database migrations, Pulumi stacks, and deployment concurrency.

- Repository: macro-inc/macro
- GitHub: https://github.com/macro-inc/macro
- Human docs: https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e
- Complete Markdown: https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/llms-full.txt

## Source Files

- `.github/workflows/deploy-service-generic.yml`
- `.github/workflows/reusable-deploy-service.yml`
- `.github/workflows/deploy-web-app.yml`
- `.github/workflows/release-production.yml`
- `.github/workflows/migrate-macro-db.yml`
- `infra/README.md`
- `rust/cloud-storage/README.md`

---

---
title: "Deploy services and web app"
description: "Operations: Generic service deployment, web app deployment, production release workflow, database migrations, Pulumi stacks, and deployment concurrency."
---

Macro deploys run through GitHub Actions workflows that build Rust service binaries or Lambda `bootstrap.zip` artifacts, install Pulumi TypeScript dependencies under `infra/`, configure AWS credentials, and run `pulumi up` against `macro-inc/dev` or `macro-inc/prod`.

## Deployment surfaces

| Surface | Entry point | Trigger | Target |
| --- | --- | --- | --- |
| Generic service deploy | `.github/workflows/deploy-service-generic.yml` | Manual `workflow_dispatch` | One service from the workflow choice list |
| Reusable service deploy | `.github/workflows/reusable-deploy-service.yml` | `workflow_call` | One service stack |
| Deploy all services | `.github/workflows/deploy-all-services.yml` | Manual or production release | Every key in `.github/services-config.json` |
| Dev cloud-storage deploy | `.github/workflows/deploy-cloud-storage-on-push.yml` | Push to `main` matching cloud-storage or infra paths | All configured services to `dev` |
| Manual Pulumi stack deploy | `.github/workflows/deploy-pulumi-stack.yml` | Manual `workflow_dispatch` | Any `infra/stacks/<name>` directory |
| Web app deploy | `.github/workflows/deploy-web-app.yml` | Manual or `workflow_call` | `infra/stacks/web-app` |
| Production release | `.github/workflows/release-production.yml` | GitHub release `released` event | DB migrations, services, web app, release artifact |
| Database migration | `.github/workflows/migrate-macro-db.yml` | Manual `workflow_dispatch` | `macro-db-dev` or `macro-db-prod` |

## Service deployment model

Service deployments are driven by `.github/services-config.json`. Each service key maps source ownership and deploy artifacts to a Pulumi stack directory.

```json
{
  "services": {
    "<service-key>": {
      "source_paths": ["rust/cloud-storage/<crate-or-area>/**"],
      "stack_path": "infra/stacks/<pulumi-project>/**",
      "deploy_binaries": ["<cargo-bin-name>"],
      "deploy_lambdas": ["<lambda-package-name>"]
    }
  }
}
```

| Field | Required | Used by |
| --- | --- | --- |
| `stack_path` | Yes | `.github/actions/get-project-name` derives the Pulumi project directory from this path. |
| `source_paths` | No | Documents source ownership and is available for change-detection workflows. |
| `deploy_binaries` | No | Enables Nix binary artifact builds through `.#deploy-service-binaries-<service>`. |
| `deploy_lambdas` | No | Enables Lambda artifact builds through `.github/scripts/build-cloud-storage-lambdas.sh`. |

<Note>
The current `deploy-cloud-storage-on-push.yml` workflow selects all service keys from `.github/services-config.json`; it does not narrow the dev deploy to changed `source_paths`.
</Note>

### Generic manual service deploy

Use **Deploy Service (Generic)** when the target service appears in `.github/workflows/deploy-service-generic.yml`.

Inputs:

<ParamField body="service" type="choice" required>
Service key to deploy. The key is passed to the reusable deploy workflow as `service-name`.
</ParamField>

<ParamField body="environment" type="choice" required>
`dev` or `prod`. Defaults to `dev`.
</ParamField>

The workflow forwards to `.github/workflows/reusable-deploy-service.yml` with inherited secrets. Its concurrency group is:

```text
deploy-${service}-${environment}
```

`cancel-in-progress` is `false`, so a second deploy for the same service and environment queues instead of canceling the active deploy.

### Reusable service deploy lifecycle

`reusable-deploy-service.yml` has three jobs:

1. `build-service-binaries`
   - Checks `.services[$SERVICE].deploy_binaries`.
   - Runs `nix build ".#deploy-service-binaries-${SERVICE}"` only when binaries are configured.
   - Copies `result/bin/*` and the Nix runtime closure from `nix-store -qR result`.
   - Uploads `prebuilt-binaries.tar.gz` as `cloud-storage-service-binaries-<service>-<sha>`.

2. `build-lambda-artifacts`
   - Checks `.services[$SERVICE].deploy_lambdas`.
   - Runs `.github/scripts/build-cloud-storage-lambdas.sh` only when Lambdas are configured.
   - Uploads `lambda-artifacts.tar.gz` as `cloud-storage-lambdas-<service>-<sha>`.

3. `deploy`
   - Resolves the Pulumi project name from `stack_path`.
   - Calls `.github/actions/deploy-cloud-storage-pulumi`.
   - Passes artifact names only when the service config enabled those artifact types.

The deploy job runs on an 8 CPU Linux runner with a per-run service identifier.

### Pulumi deploy action

`.github/actions/deploy-cloud-storage-pulumi` performs the stack update:

<Steps>
  <Step title="Checkout and restore artifacts">
    The action checks out the repository, downloads optional prebuilt binary and Lambda artifacts, and extracts them under `rust/cloud-storage/prebuilt` or `rust/cloud-storage/target/lambda`.
  </Step>
  <Step title="Prepare runtime tools">
    It optionally sets up Docker Buildx, installs Bun `1.2.0`, and runs `bun install` in `infra/`.
  </Step>
  <Step title="Configure AWS and ECR">
    It configures AWS credentials for `us-east-1` by default and logs Docker into the repository account ECR registry.
  </Step>
  <Step title="Run Pulumi">
    It runs `pulumi/actions` with `command: up`, `stack-name: macro-inc/<environment>`, and `work-dir: ./infra/stacks/<pulumi-service-name>`.
  </Step>
</Steps>

Required inputs include AWS keys, `PULUMI_ACCESS_TOKEN`, `DD_APP_KEY`, `DD_API_KEY`, and `pulumi-service-name`.

Optional inputs:

| Input | Default | Behavior |
| --- | --- | --- |
| `use-docker` | `true` | Controls Docker Buildx setup. |
| `use-lfs` | `false` | Enables LFS checkout. |
| `aws-region` | `us-east-1` | AWS region for credentials. |
| `prebuilt-binaries-artifact` | empty | Downloads and extracts prebuilt service binaries. |
| `lambda-artifacts` | empty | Downloads and extracts Lambda zips. |
| `cloud-storage-service-name` | empty | Allows inline Lambda builds when no Lambda artifact was supplied. |
| `dd-host` | `https://api.us5.datadoghq.com/` | Datadog API host for Pulumi resources. |

## Prebuilt binaries and Docker images

Rust service binaries are built by Nix through flake outputs named:

```text
deploy-service-binaries-<service-key>
```

The Nix package copies release binaries into `$out/bin` and strips them when possible. CI then packages the binaries with their Nix store closure.

When `USE_PREBUILT_SERVICE_BINARIES=true`, `infra/packages/service/src/ecr.ts` switches supported Dockerfiles to prebuilt variants:

| Normal Dockerfile | Prebuilt Dockerfile |
| --- | --- |
| `Dockerfile` | `Dockerfile.prebuilt` |
| `Dockerfile.convert_service` | `Dockerfile.convert_service.prebuilt` |
| `Dockerfile.search_processing_service` | `Dockerfile.search_processing_service.prebuilt` |

If a service uses a different Dockerfile and prebuilt binaries are enabled, deployment fails with:

```text
No prebuilt Dockerfile mapping configured for <Dockerfile>
```

The default prebuilt image copies `./prebuilt/nix-store/` into `/nix/store/`, copies `./prebuilt/${SERVICE_NAME}` to `/app/svc`, and starts it through `dumb-init`.

## Lambda artifacts

Lambda-backed stacks read `bootstrap.zip` files from:

```text
rust/cloud-storage/target/lambda/<lambda-name>/bootstrap.zip
```

`.github/scripts/build-cloud-storage-lambdas.sh` reads `deploy_lambdas` for the selected service, enters the Nix development shell, and runs:

```bash
cd rust/cloud-storage
just "<lambda-name>/build"
test -f "target/lambda/<lambda-name>/bootstrap.zip"
```

It packages the output as:

```text
lambda-artifacts.tar.gz
└── target/
    └── lambda/
        └── <lambda-name>/
            └── bootstrap.zip
```

For local Pulumi deploys that touch Lambda-backed stacks, `rust/cloud-storage/README.md` instructs running `just build_lambdas` before `pulumi up`.

## Web app deployment

`.github/workflows/deploy-web-app.yml` builds `js/app` and deploys `infra/stacks/web-app`.

Inputs:

<ParamField body="environment" type="string" required>
`dev` or `prod`.
</ParamField>

<ParamField body="notify" type="boolean">
Accepted by the workflow interface. No current workflow step reads this input.
</ParamField>

Build command:

```bash
cd js/app
just build-${environment}
```

The `just` recipes set Vite build mode:

| Environment | Command | Build env |
| --- | --- | --- |
| `dev` | `just build-dev` | `MODE=development NODE_ENV=production` |
| `prod` | `just build-prod` | `MODE=production NODE_ENV=production` |

The workflow supplies these Vite variables:

| Variable | Source |
| --- | --- |
| `VITE_DD_WEB_APP_ID` | `DD_WEB_APP_ID` secret |
| `VITE_DD_WEB_APP_TOKEN` | `DD_WEB_APP_TOKEN` secret |
| `VITE_DD_HASH` | `github.sha` |
| `VITE_SEGMENT_WRITE_KEY` | `SEGMENT_WRITE_KEY` secret |
| `VITE_POSTHOG_API_KEY` | `POSTHOG_API_KEY` secret |

After building, the workflow installs infra dependencies, configures AWS credentials, and runs Pulumi:

```text
stack-name: macro-inc/<environment>
work-dir: ./infra/stacks/web-app
```

It then uploads sourcemaps through Datadog CI:

```bash
cd js/app/packages/app
bun run ddupload:<environment>
```

For `prod`, the workflow also uploads the built `js/app/packages/app/dist` directory as the GitHub Actions artifact `web-app-build`.

### Web app Pulumi stack behavior

`infra/stacks/web-app/index.ts` requires the Pulumi config key `macro-web-app:path`. For `dev` and `prod`, this points at:

```text
../../../js/app/packages/app/dist
```

The stack fails early when the build output directory is missing or empty:

```text
Local path of build output is empty
```

The stack also blocks local production deployment unless `CI` is set:

```text
You are trying to deploy to prod without the CI environment variable set
```

The stack creates and updates:

- an S3 website bucket for app assets;
- a public `app/app-archive.zip`;
- synced app files under `./output`;
- `Cache-Control: no-store` metadata on `app/index.html`;
- a content-encoding Lambda@Edge function;
- an app-route Lambda function URL;
- a Datadog software catalog entity;
- a production CloudFront invalidation against the `website-infra` stack output `cdnId`.

For production asset sync, JavaScript sourcemaps are excluded from the S3 sync command.

## Database migrations

Database migrations run through `.github/actions/migrate-cloud-storage-db`.

Entry points:

| Workflow | Environment |
| --- | --- |
| `.github/workflows/migrate-macro-db.yml` | Manual `dev` or `prod` |
| `.github/workflows/deploy-cloud-storage-on-push.yml` | `dev` before service deployment |
| `.github/workflows/release-production.yml` | `prod` before production service deployment |

The action checks out `.github/` and `rust/cloud-storage/macro_db_client`, then reads the database URL from AWS Secrets Manager:

| Environment | Secret name |
| --- | --- |
| `dev` | `macro-db-dev` |
| `prod` | `macro-db-prod` |

If the URL does not include `sslmode=`, the action appends `sslmode=require`.

Migration commands run from `rust/cloud-storage/macro_db_client`:

```bash
# dev
cargo sqlx migrate run --ignore-missing

# prod
cargo sqlx migrate run
```

The manual migration workflow runs on the `db-migrator` runner. The migration action itself does not configure AWS credentials; the runner environment must be able to call AWS Secrets Manager.

## Production release workflow

`.github/workflows/release-production.yml` runs when a GitHub release is published with the `released` event type.

```text
release: released
  └─ migrate-db
      └─ deploy-cloud-storage
          └─ build web app
              └─ deploy web app
                  └─ upload release artifact
```

Lifecycle:

1. `migrate-db`
   - Runs production SQLx migrations on `db-migrator`.

2. `deploy-cloud-storage`
   - Calls `.github/workflows/deploy-all-services.yml` with `environment: prod`.
   - Deploys every service key in `.github/services-config.json`.

3. `build`
   - Builds the web app with `just build-prod`.
   - Uploads `web-app-prod-build` from `js/app/packages/app/dist`.

4. `deploy`
   - Downloads `web-app-prod-build`.
   - Runs Pulumi in `./infra/stacks/web-app` with `stack-name: macro-inc/prod`.
   - Uploads production sourcemaps to Datadog.

5. `upload`
   - Creates `web-app-${github.ref_name}.tar.gz` from the built `dist` directory.
   - Uploads it to the GitHub release.

The production release workflow has a single workflow-level concurrency group and does not cancel in-progress releases.

## Pulumi stacks

Infrastructure is TypeScript Pulumi under `infra/`. Each stack lives in `infra/stacks/<stack-name>` and has its own `Pulumi.yaml`. CI runs Pulumi from the stack directory.

The repository uses AWS infrastructure in `us-east-1`. The workflows configure AWS with repository secrets and pass Datadog keys through Pulumi environment variables.

Common CI stack names:

| Environment input | Pulumi stack name |
| --- | --- |
| `dev` | `macro-inc/dev` |
| `prod` | `macro-inc/prod` |

Local stack deployment follows the infra README pattern:

```bash
cd infra/stacks/<stack-name>
pulumi up --stack dev
# or
pulumi up --stack prod
```

The web app package also exposes stack-specific scripts:

```bash
cd infra/stacks/web-app
bun run deploy:dev
bun run preview:dev
bun run deploy:prod
```

## Deployment concurrency

| Workflow | Concurrency group | Cancel in progress |
| --- | --- | --- |
| `deploy-service-generic.yml` | `deploy-<service>-<environment>` | `false` |
| `deploy-web-app.yml` | `<workflow>-web-app-<environment>` | `false` |
| `release-production.yml` | `<workflow>` | `false` |
| `deploy-all-services.yml` | `<workflow>-all-services-<environment>` | `false` |
| `deploy-cloud-storage-on-push.yml` | `deploy-cloud-storage-<git-ref>` | `false` |
| `deploy-pulumi-stack.yml` | `<workflow>-<pulumi-service-name>-<environment>` | `false` |

Matrix limits:

| Workflow | Job | `max-parallel` |
| --- | --- | --- |
| `deploy-all-services.yml` | Build service binaries | `8` |
| `deploy-all-services.yml` | Build Lambda artifacts | `8` |
| `deploy-all-services.yml` | Deploy services | `20` |
| `deploy-cloud-storage-on-push.yml` | Build service binaries | `8` |
| `deploy-cloud-storage-on-push.yml` | Build individual Lambda artifacts | `8` |
| `deploy-cloud-storage-on-push.yml` | Package Lambda artifacts | `8` |
| `deploy-cloud-storage-on-push.yml` | Deploy services | `20` |

A scheduled cleanup workflow, `.github/workflows/cancel-stuck-cloud-storage-deploys.yml`, runs hourly and cancels queued or in-progress `deploy-cloud-storage-on-push.yml` runs older than 90 minutes by default. It also supports manual `max-age-minutes` and `dry-run` inputs.

## Adding or changing a deployable service

<Steps>
  <Step title="Add or update the service config">
    Add the service key to `.github/services-config.json` with `stack_path`, optional `source_paths`, optional `deploy_binaries`, and optional `deploy_lambdas`.
  </Step>
  <Step title="Create or update the Pulumi stack">
    Ensure `infra/stacks/<stack-name>/Pulumi.yaml` exists. The `stack_path` basename must match the directory that CI should pass as `pulumi-service-name`.
  </Step>
  <Step title="Wire binary builds when needed">
    If `deploy_binaries` is set, ensure `flake.nix` exposes `deploy-service-binaries-<service-key>` and includes every listed Cargo binary.
  </Step>
  <Step title="Wire Lambda builds when needed">
    If `deploy_lambdas` is set, ensure each `just "<lambda>/build"` recipe produces `rust/cloud-storage/target/lambda/<lambda>/bootstrap.zip`.
  </Step>
  <Step title="Check Dockerfile compatibility">
    Services using prebuilt binary deploys must use a Dockerfile supported by `EcrImage` prebuilt mapping, or the mapping must be extended.
  </Step>
  <Step title="Expose manual deployment when needed">
    If operators should deploy through **Deploy Service (Generic)**, add the service key to that workflow's `service` choice list.
  </Step>
</Steps>

## Troubleshooting

| Symptom | Likely cause | Check |
| --- | --- | --- |
| `Service '<name>' not found in .github/services-config.json` | The service key is missing or misspelled. | Verify `.github/services-config.json` and the workflow input. |
| `Error: Service '<name>' not found in services-config.json` | `get-project-name` could not find `stack_path`. | Confirm the service key has a `stack_path`. |
| Lambda build exits after `test -f ... bootstrap.zip` | The `just <lambda>/build` recipe did not produce the expected zip. | Run the recipe from `rust/cloud-storage` and inspect `target/lambda/<lambda>/`. |
| `No prebuilt Dockerfile mapping configured` | A prebuilt binary deploy is using an unsupported Dockerfile name. | Update `infra/packages/service/src/ecr.ts` or use a supported Dockerfile. |
| `Local path of build output is empty` | Web app Pulumi ran before `js/app/packages/app/dist` was built. | Run `just build-dev` or `just build-prod` from `js/app`. |
| Production web app deploy fails outside CI | The web app Pulumi stack blocks `prod` when `CI` is not set. | Use the release workflow or a CI environment. |
| Dev cloud-storage deploy appears stuck | The on-push workflow has queued or in-progress runs older than the threshold. | Run `Cancel Stuck Cloud Storage Deploys` with `dry-run: true`, then without dry run if appropriate. |

## Related pages

- Cloud-storage local build and Lambda notes in `rust/cloud-storage/README.md`
- Pulumi stack authoring and local deploy notes in `infra/README.md`