Current operations: see the September 27 repair receipt.
The 21-PR integration plan below is historical; that code is already on the
Production line. Use the canonical release flow and scoped regional deployment
controls for subsequent repairs.
The current selected-access model uses one blue-green control-plane machine per region, matching EU.
US placement requires an organization grant and ready capacity.
Validate EU continuity and complete the isolated tenant canary before leaving US ready.
Use the linked receipt for current gates; the original redundancy targets below describe the historical integration plan.
This is the deploy plan for the multi-region work. It assumes
docs/us-east-readiness.md for what was built and whether it works; this
file is only about getting it there without a surprise.
Every stage has a gate. A gate that does not pass is a stop, not a note.
push to main → Development (deploy-cp.yml, env=development, profile=dev)push to staging → Stagingpush to prod → Production
main is the integration line. The promotion workflows fast-forwardmain → staging, then staging → prod; a branch can only deploy to the
environment it is. So nothing reaches production except by having sat on main,
then staging, first. Two other workflows fire on main: deploy-hosts.yml
(hosts/**, apps/edge/**) and db-migrations.yml.
db-migrations.yml is a verification job. It migrates a throwaway
pt_shadow database inside the runner — it does not touch dev, staging or
production. What actually migrates a real database is the deploy, which runs
db:migrate and then assert-journal-applied.ts on the target.
There are 21 pull requests. They are all CI-green and all mergeable against
their own base, which is not the same as mergeable after each other: main is
45 commits ahead, main enforces strict up-to-date status checks, and
replaying the queue on top of main conflicts on seven of them.
Land it as one integration merge.us-east/integration is already the
resolved merge of all 21, it is what every suite ran against, and main has been
merged into it once — one conflict resolution instead of twenty-one. The slice
PRs stay as the review record and close as "landed via ".
The alternative is a serial train: for each PR, update the branch, wait for 16
checks, merge, and watch that merge invalidate the other twenty. That is ~21 CI
cycles, and every branch update is a fresh chance to hit the migration trap
below. It buys a per-slice merge commit that a squash discards anyway.
drizzle reads one watermark, once, before it applies anything:
select id, hash, created_at from drizzle.__drizzle_migrations order by created_at desc limit 1if (!last || Number(last.created_at) < migration.folderMillis) …apply
An entry whose when is at or below that watermark is not an error and not a
warning. The loop does not enter, no row is inserted, and the process exits 0
saying "up to date". The SQL never runs.
This already happened. The stacked branches numbered their migrations from
a base main later overtook, picking when values above main's 0101
(snap_refs_capture_token) and 0102 (billable_since). us-east applied the
branch migrations first, its watermark jumped past both of main's entries, and
they became permanently unreachable — sandboxes was missing two columns and
nothing failed. assert-journal-applied.ts is the only reason anyone knew.
Only an environment that applied a branch's migrations before main's is
affected. Staging and production have main's entries and none of the
branch ones, so the new migration lands normally there.
Repair (dev regions only): back up and delete the ledger rows above the
last entry the journal and the database agree on, then re-run db:migrate.
\copy (select id,hash,created_at from drizzle.__drizzle_migrations where created_at > <last agreed>) to '/tmp/ledger-backup.csv' csv headerdelete from drizzle.__drizzle_migrations where created_at > <last agreed>;bun run db:migrate && bun src/db/assert-journal-applied.ts
This is only safe because every statement being re-applied is idempotent.
Never raise the when of an entry main has already shipped to get around
this. On every database that already applied it, it would run a second time —
and ALTER TABLE … ADD COLUMN has no IF NOT EXISTS.
The five multi-region tables (template_regions, billing_settlement_outbox,
billing_settlement_claims, audit_outbox, region_state) are now one
migration regenerated on top of main's head, not five renumbered by hand. A
hand-renumbered snapshot still describes the schema before main's 0101 and
0102, so the next db:gen reads their columns as missing and emits a DROP.
Every statement in it is CREATE TABLE IF NOT EXISTS, CREATE INDEX IF NOT EXISTS or a guarded DO block, so it re-applies as a no-op where the tables
already exist.
Historically us-cp's lane file named its local netem rig, while node.env
overrode the serving process to the managed US database. The old deployment
migrated the rig and reported success, leaving the real region behind its code.
The current infra/deploy-bundle.sh resolves, in this order:
PT_MIGRATE_DATABASE_URL in /etc/platinum/migrate.env;
the same key in node.env, then /etc/platinum.env (legacy fallbacks);
DATABASE_URL in node.env, then /etc/platinum.env;
a running process only when no file supplies a target.
It compares the migration target's hostname/database with the running CP and
refuses a mismatch before the swap. Different roles are intentional. Keep the
schema-owner credential in root-owned mode-0600 migrate.env, which the
serving unit does not load; keep the US app credential in node.env.
Both migration lanes must pass: the Drizzle journal and the idempotent
operational SQL scripts. Never replay the historical destructive
bauth_cutover.sql. After schema changes, reconcile the publication,
subscription relations and regional read-only grants against
apps/api/src/db/ownership.ts. DDL and privileges do not replicate.
Check sequence USAGE separately from table DML: the application can have full
regional-table access while a trigger's nextval() still fails. In the
September 27 repair this blocked volume creation on
volume_billing_intervals_id_seq; volume_merge_jobs_seq_seq also needed USAGE.
Grant only regional sequences to the app role, leaving global copies read-only.
A worker PID or a recent keepalive is insufficient replication proof. Check
advancing received/applied positions, publisher confirmed_flush_lsn, bounded
retained WAL, stable apply/sync error counters and every subscription relation
in ready state. Verify fresh and revoked credentials through both regional APIs;
the positive auth-cache TTL currently permits up to 30 seconds of cross-process
revocation delay in addition to replication delay.
The September 27 repair applied the five missing migrations from already
released c772f373ca47; the apply worker recovered without skipping WAL. The
publication also needed volume_billing_config_epochs, whose regional app
write grants were revoked. Receipts and current launch gates are tracked in
#1300.
Everything multi-region is conditional on a separate global database, which dev
does not have, so dev must behave exactly as before. Run against dev:
verify/feature-matrix-e2e.sh expect the same score as before the mergeverify/e2e-deep.sh 45 pass / 0 fail / 3 skippedverify/stability-e2e.sh 4 scenarios
Gate: no check that passed before the merge fails after it. A single-region
deployment must not be able to tell this landed.
Fast-forward staging → prod. us-east is not serving customer traffic yet;
this stage only puts the multi-region code under EU.
Gates:
assert-journal-applied.ts on the production database.
EU's own suites at their pre-merge scores.
platinum_global_db_circuit_open is 0 and
platinum_settlement_outbox_pending is not growing.
Rollback: revert the integration merge on main and fast-forward again. The
migration does not need reverting — five unused tables and one column are inert
to code that does not read them. Do not drop them to "clean up"; a re-land
would then re-create them and the ledger would disagree.
A second host and a second control plane in us-east. One of each is a
regional outage waiting for a hardware fault, and no code fixes that.
The private VLAN with Hivelocity (ticket open — the reply they need is the
private NIC ens8 02:01:01:b0:00:02 172.16.8.2/22, node port 117052
84:16:0c:bb:4c:d0 172.16.8.10/22, VLAN 2039; 1e:00:ef:00:07:4a is the
VPS public NIC and is not the one to tag).
Decide the admin credential (below), then re-run verify/region-e2e.sh on
both regions and expect 11/11.
Run verify/region-outage-e2e.sh against us-east one final time: 12/12.
placement.regions → api_origin — set 2026-09-18: eu-west →
https://api.platinum.dev (EU serves the apex, so the derived
eu-west.api.platinum.dev never resolved), us-east →
https://us-east.api.platinum.dev. Datacenters preserved verbatim.
Per-region bucket — deliberately NOT set. Declaring buckets arms the
template-residency gate, which then refuses a placement whose template is
not resident in the target region. That is a production placement behaviour
change and belongs in its own change with its own gate, not in this one.
The admin credential. The fan-out forwards the caller's own credential by
design. api_keys and bauth_session are global tables read from each
region's replica, so a dashboard session or API key authenticates in every
region. ADMIN_TOKEN is a per-deployment environment variable and does not:
an ops token from us-east gets a 401 from EU. Either share one token across
the deployments or drive the admin with a session. Until one is chosen, the
fan-out check in verify/region-e2e.sh stays red, and that is honest.
A region's dashboard listing is opt-in (?regions=all). The picker
(GET /v1/regions) is not — it merges siblings by default. If the frontend
never sends regions=all, a user with boxes in both regions sees half of
them and nothing reports an error.
Session replication lag: a user who signs in on EU and immediately hits
us-east can be rejected until the session row replicates.
PT_FANOUT_TIMEOUT_MS is 3 s. A sibling slower than that is reported
unavailable, never waited for.