# Release pipeline

> Building and publishing a release: `build-release.mjs` byte-identical worker bundles and manifest generation, the placeholder contract (`$ACCOUNT_ID`, `$WORKER_NAME(...)`, `$SECRET(...)`, `$PUBLIC_BASE_URL`), `upload-release.mjs --candidate` staging, and the all-or-nothing manifest copy in `promote-release.mjs`. Covers required R2 variables, `NO_DEFAULT_CRED_INPUTS`, promote serialization, and golden-file regeneration with `UPDATE_GOLDEN=1`.

- Repository: cloudflare/cloudflare-os
- GitHub: https://github.com/cloudflare/cloudflare-os
- Human docs: https://grok-wiki.com/public/docs/cloudflare-cloudflare-os-838773bb92dd
- Complete Markdown: https://grok-wiki.com/public/docs/cloudflare-cloudflare-os-838773bb92dd/llms-full.txt

## Source Files

- `scripts/release/build-release.mjs`
- `scripts/release/manifest-lib.mjs`
- `scripts/release/upload-release.mjs`
- `scripts/release/promote-release.mjs`
- `scripts/release-manifest.test.js`
- `scripts/release-promote.test.js`
- `AGENTS.md`

---

---
title: "Release pipeline"
description: "Building and publishing a release: `build-release.mjs` byte-identical worker bundles and manifest generation, the placeholder contract (`$ACCOUNT_ID`, `$WORKER_NAME(...)`, `$SECRET(...)`, `$PUBLIC_BASE_URL`), `upload-release.mjs --candidate` staging, and the all-or-nothing manifest copy in `promote-release.mjs`. Covers required R2 variables, `NO_DEFAULT_CRED_INPUTS`, promote serialization, and golden-file regeneration with `UPDATE_GOLDEN=1`."
---

A release is an immutable, content-addressed bundle set plus one manifest. `scripts/release/build-release.mjs` builds every deployable worker exactly as `wrangler deploy` would upload it (`--dry-run --outdir`, using the repo's pinned wrangler), writes each module and static asset under its content hash, and emits `manifest.json` describing the whole set. `scripts/release/upload-release.mjs` mirrors that directory to R2 over the S3 API, blobs first and manifest last. `scripts/release/promote-release.mjs` publishes a candidate by copying a single key: `candidates/<id>/manifest.json` → `releases/<id>/manifest.json`. The manifest is the contract between this repo's CI and the deploy service, which PUTs the bundles into customer accounts via the Workers script-upload API.

## Pipeline stages

| Stage | Command | Effect |
| --- | --- | --- |
| Build | `node scripts/release/build-release.mjs --out <dir> [--release-id <id>]` | Writes `<out>/modules/<sha256>`, `<out>/assets/<cfHash>`, then `<out>/manifest.json` |
| Stage | `node scripts/release/upload-release.mjs --release <dir> --candidate` | Uploads blobs, then manifest to `candidates/<id>/manifest.json` |
| Publish | `node scripts/release/promote-release.mjs --release-id <id>` | Copies the candidate manifest to `releases/<id>/manifest.json` |
| Direct publish | `node scripts/release/upload-release.mjs --release <dir>` | Uploads blobs, then manifest straight to `releases/<id>/manifest.json` |

```text
build-release.mjs                 R2 bucket
┌──────────────────────┐          ┌──────────────────────────────────┐
│ frontend (Access)    │  upload  │ blobs/modules/<sha256>           │  content-addressed,
│ wrangler --dry-run   │ ───────► │ blobs/assets/<cfHash>            │  deduped across releases
│ manifest.json        │          ├──────────────────────────────────┤
└──────────────────────┘          │ candidates/<id>/manifest.json    │  invisible to deploy
        │ --candidate                                                    service
        │                         │              │ promote-release.mjs
        └────────────────────────►│              ▼
                                  │ releases/<id>/manifest.json      │  scanned by deploy
                                  └──────────────────────────────────┘     service
```

<Info>
The manifest-last protocol is what makes a release atomic: the deploy service scans only `releases/`, so a crashed blob upload never leaves a manifest pointing at missing blobs.
</Info>

## Building a release

`build-release.mjs` runs in a fixed order because the router's `wrangler.jsonc` points its assets directory at `workshop-frontend/dist`.

<Steps>
<Step title="Resolve release identity">
The commit comes from `CI_COMMIT_SHA` or `git rev-parse HEAD`. When `--release-id` is omitted, the id is derived: `r<CI_PIPELINE_IID padded to 6>-<short sha>` in CI, otherwise `dev-<base36 unix seconds>`. `CI_PIPELINE_IID` (per-project, monotonic) is used deliberately, not `CI_PIPELINE_ID` — run numbers must form one monotonic sequence because `promote-release.mjs` compares them. Ids stay short because downstream worker version tags (`gd:<id>:<fp8>`) have a hard 25-character cap.
</Step>
<Step title="Build the Access-mode frontend">
`pnpm run build` runs in `packages/workshop-frontend` with `VITE_CF_ACCESS_MODE=true` (a build-time flag read by `workshop-frontend/src/useAuth.ts`). Its `dist/` output is collected as the `access` asset variant — the one variant every release carries — and each blob is written to `<out>/assets/<hash>`.
</Step>
<Step title="Bundle every deployable worker">
For each package returned by `findDeployablePackages(packages/)`, the script runs `pnpm exec wrangler deploy --dry-run --outdir <tmp>/<pkg>` **from the package directory**, so custom build commands (`capnweb-validate`) resolve their bins. Modules are written to `<out>/modules/<sha256>`.
</Step>
<Step title="Generate the manifest">
`generateManifest({ releaseId, commit, createdAt, wranglerVersion, workers, assetVariants })` produces the manifest, stringified with `stableStringify` and written last — mirroring the R2 upload order. `wranglerVersion` is read from `node_modules/wrangler/package.json`, so the recorded version is the pinned one.
</Step>
</Steps>

The final log line reports worker count, module count, and unique asset blob count.

<ParamField body="--out" type="path" required>
Output directory. Removed recursively before the build, then recreated with `modules/` and `assets/` subdirectories.
</ParamField>

<ParamField body="--release-id" type="string">
Overrides the derived release id. Any other argument throws `unknown argument: <arg>`.
</ParamField>

## The placeholder contract

`scripts/release/manifest-lib.mjs` parses each package's `wrangler.jsonc` and emits binding *templates*: every account-specific value is replaced by a placeholder the deploy service resolves from instance state. The list is closed — the deploy-side renderer fails on any `$` token it does not recognize, so `manifest-lib.mjs` and the renderer must evolve together. `MANIFEST_VERSION` (currently `1`) guards that coupling.

| Placeholder | Resolves to |
| --- | --- |
| `$ACCOUNT_ID` | The user's account tag |
| `$KV_<BINDING>_ID` | A KV namespace provisioned at deploy time |
| `$R2_<BINDING>_NAME` | An R2 bucket provisioned at deploy time |
| `$WORKER_NAME(<pkg>)` | The instance's chosen name for another worker in this release |
| `$SECRET(<name>)` | A user-supplied secret, passed through as `secret_text` |
| `$PUBLIC_BASE_URL` | The instance's public origin (the router's URL) |

Examples produced by the generator and asserted by the golden tests:

```json
{ "type": "kv_namespace", "name": "BLUEPRINTS", "namespace_id": "$KV_BLUEPRINTS_ID" }
{ "type": "r2_bucket", "name": "BLUEPRINT_CONTENT", "bucket_name": "$R2_BLUEPRINT_CONTENT_NAME" }
{ "type": "service", "name": "WORKSHOP_BACKEND", "service": "$WORKER_NAME(workshop-backend)" }
{ "type": "secret_text", "name": "CLIENT_SECRET", "text": "$SECRET(CLIENT_SECRET)" }
```

Placeholder-free bindings pass through as `{ type, name }` only: `worker_loader` (`LOADER`), `ai` (`WORKERS_AI`, which always ships because `webFetch`'s `toMarkdown` conversion depends on it), `browser` (Browser Rendering is generally available, and wrangler's dev-only `remote` flag is dropped), and `assets` (binding name defaults to `ASSETS`).

Var templating is per worker kind: the backend carries `PUBLIC_BASE_URL: "$PUBLIC_BASE_URL"`, and each gatekeeper carries `BASE_URL: "$PUBLIC_BASE_URL/gatekeeper/<shortName>"` where `shortName` is the package name minus the `gatekeeper-` prefix and matches the router's path scan.

<Warning>
`HANDLED_CONFIG_KEYS` fails closed. A `wrangler.jsonc` key the generator does not know throws `<pkg>/wrangler.jsonc has key(s) this generator doesn't handle: ...`, forcing an explicit decision about how customer instances receive it. Similarly, an `artifacts` binding outside `ARTIFACTS_CUT_ALLOWED` (`gatekeeper-context` only) is a hard error; `gatekeeper-context`'s closed-beta Artifacts binding is dropped from customer manifests and the gatekeeper degrades gracefully.
</Warning>

## Worker kinds, inputs, and installability

`workerKind()` classifies each deployable package: `workshop-backend` → `backend`, `router` → `router`, anything starting with `gatekeeper-` → `gatekeeper`. An unclassifiable package throws `cannot classify deployable package: <name>`.

Installable gatekeepers default to the two `DEFAULT_CRED_INPUTS` secret inputs, which materialize as `$SECRET(...)` bindings:

```js
// scripts/release/manifest-lib.mjs
export const DEFAULT_CRED_INPUTS = [
  { name: "CLIENT_ID",     kind: "secret", label: "OAuth client ID" },
  { name: "CLIENT_SECRET", kind: "secret", label: "OAuth client secret" },
];
```

`NO_DEFAULT_CRED_INPUTS` names the installable gatekeepers that take no third-party OAuth app credentials:

| Package | Reason |
| --- | --- |
| `gatekeeper-context` | No third-party service; uses its own storage |
| `gatekeeper-homeassistant` | Users connect their own Home Assistant URL + token in-app |
| `gatekeeper-scheduler` | Auto-provisioned; no third-party OAuth app |
| `gatekeeper-mcp` | MCP OAuth uses dynamic client registration, not a static app |
| `gatekeeper-mcp-portal` | Same MCP OAuth chain as `gatekeeper-mcp` |

`NOT_INSTALLABLE` contains `gatekeeper-email`: Email Routing needs a zone, which workers.dev-hosted instances don't have. Its bundle still ships in the release so the entry stays auditable, with `installable: false` and `inputs: []`. Per-package `deploy-inputs.json` (read by `readDeployInputs`, `undefined` when absent) overrides the default input set.

The backend entry also carries `gatekeeperBindingExpansion.entrypoint === "GatekeeperVendor"` and its full ordered `migrations` history verbatim from `wrangler.jsonc` (tag `v0` onward, including `new_sqlite_classes` such as `UserDurableObject`).

## Staging with `--candidate`

`upload-release.mjs` reads `<dir>/manifest.json`, enumerates `modules/` and `assets/`, and maps each file to its R2 key via `moduleR2Key(sha256)` and `assetR2Key(hash)`. Eight concurrent workers (`UPLOAD_CONCURRENCY = 8`) drain a shared queue; each key is `HEAD`'d first and skipped when it already exists, so unchanged blobs dedupe across releases. Any HEAD status other than `200`/`404` is a hard error, as is a failed PUT.

The manifest PUT is last, and `--candidate` only changes its key:

```text
default:     releases/<manifest.releaseId>/manifest.json
--candidate: candidates/<manifest.releaseId>/manifest.json
```

Blob handling is identical either way. A candidate manifest is invisible to the deploy service, which scans only `releases/`, until promotion. The script logs `candidate uploaded (not yet visible to the deploy service): <key>` or `release complete: <key>`.

<RequestExample>
```bash Stage a candidate
export R2_ENDPOINT="https://<account>.r2.cloudflarestorage.com"
export R2_BUCKET="<bucket>"
export R2_ACCESS_KEY_ID="<key-id>"
export R2_SECRET_ACCESS_KEY="<secret>"

node scripts/release/build-release.mjs --out ./release-out
node scripts/release/upload-release.mjs --release ./release-out --candidate
```
</RequestExample>

<ResponseExample>
```text Output
blobs: 41 uploaded, 128 already present
candidate uploaded (not yet visible to the deploy service): candidates/r000123-abc1234/manifest.json
```
</ResponseExample>

### Required R2 variables

Both `upload-release.mjs` and `promote-release.mjs` call `requireEnv` for the same four variables and throw `missing required environment variable: <name>` when one is absent. Requests are signed with `aws4fetch` (`service: "s3"`, `region: "auto"`).

<ParamField body="R2_ENDPOINT" type="string" required>
`https://<account>.r2.cloudflarestorage.com`. A trailing slash is stripped.
</ParamField>

<ParamField body="R2_BUCKET" type="string" required>
Bucket holding `blobs/`, `candidates/`, and `releases/`.
</ParamField>

<ParamField body="R2_ACCESS_KEY_ID" type="string" required>
S3-compatible access key id.
</ParamField>

<ParamField body="R2_SECRET_ACCESS_KEY" type="string" required>
S3-compatible secret access key.
</ParamField>

## Promotion

Because blobs are already in place — content-addressed and uploaded before the candidate's manifest PUT — publishing is a single manifest copy. `promote-release.mjs` prefers a server-side `CopyObject` (`x-amz-copy-source: /<bucket>/candidates/<id>/manifest.json`), avoiding a second body transfer. S3 can answer `200` with an error document, so success requires the response body to contain `<CopyObjectResult`; otherwise the script logs `CopyObject unavailable (status ...); falling back to PUT` and PUTs the already-buffered candidate body.

```mermaid
stateDiagram-v2
    [*] --> HeadPublished: promote-release.mjs --release-id <id>
    HeadPublished --> AlreadyPromoted: 200
    HeadPublished --> GetCandidate: 404
    HeadPublished --> HardError: other status
    GetCandidate --> HardError: 404 candidate not found
    GetCandidate --> SupersedeCheck: 200 (body buffered)
    SupersedeCheck --> Superseded: higher CI run published
    SupersedeCheck --> CopyObject: no superseder
    CopyObject --> Promoted: 200 + CopyObjectResult
    CopyObject --> PutFallback: copy unavailable
    PutFallback --> Promoted: PUT ok
    PutFallback --> HardError: PUT failed
    AlreadyPromoted --> [*]: exit 0
    Superseded --> [*]: exit 0 (warning)
    Promoted --> [*]: exit 0
    HardError --> [*]: throw
```

### Exit-0 guards versus hard errors

| Condition | Behavior |
| --- | --- |
| `releases/<id>/manifest.json` already exists (HEAD 200) | Logs `already promoted: <key>`, exits 0 — idempotent re-runs |
| A CI-format id with a **higher** run number already published | Warns and returns without copying, exits 0 |
| `candidates/<id>/manifest.json` missing (GET 404) | Throws `candidate not found: <key> — was the release uploaded with --candidate?` |
| Unexpected HEAD status on the published key | Throws `HEAD <key>: unexpected status <n>` |
| Failed fallback PUT | Throws `PUT <key>: <status> <body>` |

A missing candidate manifest is deliberately a hard error: the caller asked to promote something never uploaded (or since cleaned up), and silently succeeding would report a phantom publish.

### Ordering guard

"Latest" is decided by manifest upload time in the deploy service (`release.ts`), so promoting an older candidate after a newer one shipped would **roll production back** — the stale PUT would carry the newest timestamp. Two exported pure functions encode the guard:

```js
ciRunNumber("r000123-abc1234") // 123
ciRunNumber("r1-abc1234")      // 1
ciRunNumber("dev-mdxk3f2a")    // null
ciRunNumber("release-42")      // null
ciRunNumber("r-abc1234")       // null

supersededBy("r000123-abc1234", ["r000122-aaaaaaa", "r000124-bbbbbbb"]) // "r000124-bbbbbbb"
supersededBy("r000123-abc1234", ["r000123-abc1234"])                    // null
supersededBy("r000123-abc1234", ["dev-zzzzzzzz"])                       // null
supersededBy("dev-mdxk3f2a", ["r999999-abc1234"])                       // null
```

Only `r<run#>-<sha>` ids participate in ordering. `dev-<ts>` ids carry no ordering claim on either side: a dev candidate is never superseded, and a published dev release never supersedes a CI candidate. Published ids are discovered by paginated `ListObjectsV2` over `prefix=releases/` (1000 keys per round trip, `<NextContinuationToken>` followed); only key names are parsed, manifest bodies are never fetched.

<Warning>
Promotion is all-or-nothing but **not** isolated. The newer-release guard is check-then-act, so concurrent promotions can interleave between the LIST and the copy. The caller must serialize runs of this script — `gadgets-internal`'s CI runs it in a resource group — and the guard then catches the remaining hazard: a promote that starts after a newer release has already published.
</Warning>

## Golden manifest test

`scripts/release-manifest.test.js` runs the manifest generator against the repo's **real** `wrangler.jsonc` files, substituting fixture bundles and assets so no compilation is needed. It pins deterministic inputs (`releaseId: "r000000-fixture"`, an all-zero commit, `createdAt: "2026-01-01T00:00:00.000Z"`, `wranglerVersion: "0.0.0-fixture"`) and compares `stableStringify(manifest)` against `scripts/testdata/golden-manifest.json`.

<Tabs>
<Tab title="Run">
```bash
node --test scripts/release-manifest.test.js
```
</Tab>
<Tab title="Regenerate">
```bash
UPDATE_GOLDEN=1 node --test scripts/release-manifest.test.js
```
</Tab>
</Tabs>

With `UPDATE_GOLDEN` set, the test writes the rendered manifest to the golden path and returns without asserting. Without it, a missing golden file fails with `golden manifest missing; run with UPDATE_GOLDEN=1`.

This is a deliberate tripwire: changing any deployable package's `wrangler.jsonc` fails the test until the golden file is regenerated, forcing a conscious decision about how the change reaches customer instances. The failure message is explicit — verify the deploy service handles the change, then regenerate.

Adding a new deployable package also requires a fixture: the test asserts `scripts/testdata/fixture-bundles/<pkg>/` exists, with the message `missing fixture bundle for new deployable package: add scripts/testdata/fixture-bundles/<pkg>/ with a single .js module`.

Two further tests in the same file lock the deploy contract:

- **Placeholder syntax.** Every `$`-token found in each worker's `bindings`, `vars`, and `gatekeeperBindingExpansion` must match the closed placeholder regex `^\$(ACCOUNT_ID|PUBLIC_BASE_URL|KV_[A-Z0-9_]+_ID|R2_[A-Z0-9_]+_NAME|WORKER_NAME\([a-z0-9-]+\)|SECRET\([A-Z0-9_]+\))`, recursing through arrays and objects.
- **Contract shape.** Worker kinds (`workshop-backend` → `backend`, `router` → `router`, `gatekeeper-google` → `gatekeeper`), the backend's provisioned-resource placeholders and migration history, `gatekeeper-google`'s `shortName`/`BASE_URL`/`CLIENT_ID`+`CLIENT_SECRET` inputs, `gatekeeper-email`'s `installable: false` with empty `inputs`, and the router's `assetsConfig` (`run_worker_first` includes `/gatekeeper/*`, `not_found_handling === "single-page-application"`, variants exactly `["access"]`). Every asset hash referenced by a variant manifest must exist in `manifest.assets` with `r2Key === "blobs/assets/<hash>"`.

`scripts/release-promote.test.js` covers the promote guard's pure logic only (`ciRunNumber`, `supersededBy`); the R2 round trip is exercised by the deploy e2e pipeline, not by unit tests.

```bash
node --test scripts/release-promote.test.js
```

## Troubleshooting

<AccordionGroup>
<Accordion title="manifest changed. If the wrangler.jsonc change is intentional...">
The golden manifest test detected a config drift. Confirm the deploy service handles the new shape, then run `UPDATE_GOLDEN=1 node --test scripts/release-manifest.test.js` and commit the regenerated `scripts/testdata/golden-manifest.json`.
</Accordion>
<Accordion title="<pkg>/wrangler.jsonc has key(s) this generator doesn't handle">
The key is outside `HANDLED_CONFIG_KEYS` in `scripts/release/manifest-lib.mjs`. Decide how customer instances should receive that config, add explicit handling (and a placeholder if the value is account-specific), then regenerate the golden manifest.
</Accordion>
<Accordion title="candidate not found: candidates/<id>/manifest.json — was the release uploaded with --candidate?">
Promotion requires a staged candidate. Re-run `upload-release.mjs --release <dir> --candidate`, or confirm the candidate was not cleaned up. Promotion never publishes without a candidate body.
</Accordion>
<Accordion title="WARNING: not promoting <id> — a newer release (<id>) is already published">
Expected exit-0 behavior, not a failure. A higher CI run number already shipped; promoting now would roll production back because "latest" is decided by manifest upload time. Ship a new build instead.
</Accordion>
<Accordion title="missing required environment variable: R2_*">
`upload-release.mjs` and `promote-release.mjs` both require `R2_ENDPOINT`, `R2_BUCKET`, `R2_ACCESS_KEY_ID`, and `R2_SECRET_ACCESS_KEY`. `build-release.mjs` requires none of them.
</Accordion>
<Accordion title="missing fixture bundle for new deployable package">
A new package with a `wrangler.jsonc` was discovered by `findDeployablePackages`. Add `scripts/testdata/fixture-bundles/<pkg>/` containing a single `.js` module, then regenerate the golden manifest.
</Accordion>
<Accordion title="cannot classify deployable package: <name>">
`workerKind()` only recognizes `workshop-backend`, `router`, and `gatekeeper-*` prefixes. Rename the package or extend the classifier and the manifest contract together.
</Accordion>
</AccordionGroup>

## Related pages

<CardGroup cols={2}>
<Card title="Build, lint, and test" href="/build-lint-test">
The commands CI enforces and their ordering, including `node --test scripts/*.test.js`.
</Card>
<Card title="Routing and worker bindings" href="/routing-and-bindings">
The bindings and DO migration tags the manifest templates, and how the router resolves `/gatekeeper/<name>/*`.
</Card>
<Card title="Environment variables" href="/environment-variables">
Backend environment variables behind `PUBLIC_BASE_URL` and the rest of the instance config.
</Card>
<Card title="Configure gatekeeper credentials" href="/configure-gatekeeper-credentials">
How `CLIENT_ID`/`CLIENT_SECRET` inputs and `deploy-inputs.json` overrides are consumed.
</Card>
</CardGroup>