us-east: what is ready, what is not
us-east: what is ready, what is not
Status on 2026-09-17, measured on the real boxes (CP VPS + node at Hivelocity OGB1, PlanetScale prod-us-east, Wasabi platinum-prod-us, Frankfurt as the global database).
Short answer: not yet. The software is there and it is fast; the region is one host, one control plane, one template, with no warm pool and an unconfigured secret keyring. Nothing below is unknown — each line is something that was run or measured.
Ready
| evidence | |
|---|---|
| Latency beats EU on every route | create 865 ms vs 1586, start 502 vs 1586, stop 450 vs 1097, fork 403 vs 870, delete 230 vs 559, GET 55 vs 179, exec 16 vs 34 (p50 of 5 warmed runs) |
| Regional database split works | 111 tables, both migration lanes applied, app role writes only regional tables, global copies REVOKEd; zero permission denied in normal operation |
| Replication is healthy | platinum_global_replication_lag_bytes in the hundreds, worker up, subscription pt_us_east current |
| Money settles exactly once | flush, uptime sampler and request meter write billing_settlement_outbox; the shipper applies them in the global database under billing_settlement_claims; drained to 0 after every drill |
| Audit survives the distance | lifecycle audits commit locally and ship globally; sandbox.create/stop/start/delete rows verified present in Frankfurt |
| A Frankfurt outage does not slow this region | with the global database firewalled off: create 920 ms, stop 496, start 508, delete 309, exec 63, GET 46 — against 50.1 s / 5.1 / 5.3 / 5.6 before the breaker and audit outbox |
| Health is observable | replication lag, both outbox backlogs, circuit state and refusals on /metrics; the scrape answers during an outage (2 s cap on its one global read) |
| The rig database is closed | local Postgres bound to localhost; nothing off-box used it |
| Concurrency matches production | the spawn cap is derived from the database pool (pool − 2); us-east ran pool 8 against production's 16, so it shed at 6 concurrent creates where production sheds at 14. Now 16: 20 of 20 parallel creates succeed, and a burst of 30 sheds 7 with the documented 503 and a Retry-After, exactly as production does |
| A fresh database comes up complete | every migration applied to an empty database yields 112 tables, including billing_settlement_outbox, billing_settlement_claims, region_state, audit_outbox and template_regions |
| Org secrets work, end to end | create in the region (global write), bind at sandbox create (regional binding), host leases and arms it, the proxy injects it into the allowed header over HTTPS (destination answered 204), plaintext is refused, a response that echoes it is blocked, and the guest's environment never holds the value |
| A customer can use an image the region has never seen | POST /v1/sandboxes with an image spec: pull, build on a US host, chunks to the US bucket, boot — 6.3 s cold, 940 ms for the next one; the row lands ready with no host pin and its manifest in platinum-prod-us |
Not ready
-
One host, one control plane.
aec_hosts_total 1. A node failure is a regional outage; a CP failure is a regional outage. Production needs at least two of each, and the placement/drain paths need to be exercised with more than one host in-region. -
Templates built in EU are not here. A template is only usable in the region that holds its chunks. Building in-region works and is fast; replicating an existing EU template is a per-template admin call. Anyone routed here who expects their existing templates must have them replicated first.
Not a gap, despite what this document said earlier: no warm pool is missing in us-east, because no region has one.
SNAPSHOT_CONFIGSnameskortix-computer, which no longer exists in the catalogue, and the dynamic list is empty — there are zerowarm-*templates and zero pool sandboxes fleet-wide. EU production cold-boots too. Warming is worth doing, but it is a fleet-wide improvement, not a US deficit. -
The org-secret keyring is unset.Done. The home region's kidK1is installed on the regional control plane, and the node was promoted to a secret carrier withinfra/scripts/promote-host-secret-carrier.sh(anti-spoof enforce; it advertisessecretsV=2and the scheduler will place credential-bearing sandboxes on it). A new region needs both steps, and neither is automatic:PT_SECRETS_ACTIVE_KIDis absent, so/v1/secrets/*and sealed host leases answer 503 (the host retries ~155 times an hour, harmlessly). This is deliberate rather than half-configured: the ciphertext lives in the globalorg_secret_versions, so a regional control plane must carry the same keyring as the home region. A different key here would write secrets the home region cannot read. -
No private network. The Hivelocity VLAN still passes no frames between the CP VPS and the node, so CP↔node traffic uses public addresses. The cutover (node
PT_INTERNAL_IP, vsock cert SAN, edge database route) is written and waiting on their side. -
The multi-region code is not merged. It lives on
us-east/integrationas a stack of pull requests. Production runsmain. -
A build whose chunks never reach object storage now fails in a regional control plane instead of falling back to a host pin, because the global catalogue cannot name a regional host. That is deliberate (
templateHostPin.ts), but it means S3 credentials on a regional host are load-bearing: if they break, builds fail loudly rather than degrading.
Before routing real traffic here
Each line is a thing to do, not a thing to believe. The four suites in
tests/README.md are how the first three are checked.
-
Merge the queue in order. It is 21 pull requests, not the seven this line used to name — the region also needs the multi-region groundwork and the round-trip work it was built on:
1018 1019 1023 1026 1030 1031 independent 1029 → 1140 residency, then prewarm 1040 1041 in-process waits, installer lessons 1110 → 1111 → 1120 → 1112 → 1113 round trips, in this order 1138 1139 1141 1144 independent 1142 → 1143 the two multi-region slicesmultiregion/base, which #1142 targets, is a staging branch with no pull request of its own: merging #1142 into it lands nothing in main. Retarget #1142 at main once its predecessors are in, or the two largest slices have no path home.A green "Mergeable" on a pull request means it merges into its own base, not that it merges after its siblings. Seven of these conflict against the queue ahead of them —
regionsDefault.tsandserver.ts(both additive, keep both sides),api/hosts.tsandapi/sandboxes.ts(sibling perf edits to the same statement), and the migration chain.us-east/integrationis the already-resolved merge of all of them and is what the suites ran against, so it is the reference for every one of those resolutions. -
Roll EU and re-run the battery there. Every multi-region path is conditional on a separate global database, so EU's behaviour should be byte-identical — and that is a claim to test, not to assert. The same
verify/feature-matrix-e2e.shagainst EU proves it. -
Set the two
api_originrows in the regions map, then re-runverify/region-e2e.shon both regions: the fan-out check turns green only when each region can actually reach the other. -
Add a second host and a second control plane in us-east. One of each is a regional outage waiting for a hardware fault, and no code fixes that.
-
Re-run the outage drill after any change to
db/global.ts,audit.ts, the breaker or adbGlobalcaller. It is the only check that can see the home region going away. -
Check both halves report a clean build. The control plane answers its health port with
{"version":"git:<sha>"}and the host advertisesagent_versioninGET /v1/admin/hosts. A-dirtysuffix on either means the binary was built from a tree with uncommitted changes, so nothing reviewed can be said to be what is running — us-east served for three hours ongit:996fa72ff68c-dirtybefore anyone looked.deploy-hosts.shrefuses a staledist/, but it will happily roll a dirty one.
What a launch checklist looks like from here
- Merge the stack, roll EU, confirm no regression there (EU gains the same guest-boot and resume fixes).
- Add a second node and a second control plane in us-east; re-run the drills with two of each.
- Replicate the production templates, bake their pools in-region, and re-measure create from the pool.
- Ship the home region's secret keyring to the regional control plane, then re-enable
/v1/secrets/*. - Finish the private VLAN and move CP↔node and database traffic off public addresses.
- Point monitoring at the new gauges with the thresholds in
docs/multi-region-db.md.
How to re-run the evidence
- Routes:
bash /root/pt-routes-us.shon the CP (medians of at least three; one run straight after a deploy reads cold). - Cold-boot breakdown:
bash /root/pt-guest-probe.sh N, then the host agent's[perf]lines. - Home-region outage: the drill drops outbound 5432 to the global database's addresses on the CP, exercises the sandbox path, a route that must write globally, and
/metrics, then restores the firewall.