Shared Volume mounted-RW gateway runbook

Gateway v1 is a default-off gated path. Merging or deploying the code does not enable mounted writes: the global Admin switch remains off, and hosts advertise v0/null unless a root-owned test or rollout profile explicitly arms v1. Direct file APIs and read-only mounts remain the supported fallback.

Safety model

The write path is:

sandbox -> virtiofs -> rclone (VFS off) -> loopback gateway
        -> root-only bounded spool -> operation-scoped S3 staging object
        -> final S3 object

The sandbox and rclone receive only attachment-scoped loopback credentials. Provider credentials stay in the host agent. The control plane receives no file bytes: it opens and fences a single writer, atomically reserves quota, commits logical deltas after provider evidence, and retains ambiguous operations.

Rclone's multipart API is virtual. The gateway fsyncs a CP-issued upload ID, part revisions, and response WAL before acknowledging them. Complete streams the selected bounded spool into one operation-tagged staging PutObject, then copy-promotes it. This deliberately avoids a second provider multipart initiation identity that could be lost during a host crash.

Hard limits and compatibility

• One active or pending RW attachment per volume; RO attachments may coexist.
• One GiB maximum dirty spool per attachment, 256 MiB per virtual part, four GiB aggregate host spool, and eight active virtual uploads per host.
• One rclone transfer/checker/upload worker; VFS cache mode is off so a rejected close returns a filesystem error instead of acknowledging dirty cache.
• Sequential create/truncate/close is the MVP filesystem contract. Seek-write, append, locks, hard links, transactional rename, and multi-writer semantics are not promised.
• Existing direct APIs, RO mounts, snapshots, detach, and recovery retain their existing behavior. Snapshots still require RW attachments to be detached.

Every numeric safety limit is part of the exact v1 host fingerprint. A rolled back, stale, differently configured, cordoned, or non-heartbeating host is not eligible for new reservations.

Rollout gate

Do not turn on the Admin switch or a real host advertisement until all of these are recorded against the exact candidate SHA:

1. Fresh PostgreSQL migration twice, API typecheck/tests, web, SDK/CLI/MCP, Go race/vet/build, and the migration tripwires pass.
2. Linux fake-CP/fake-S3 tests prove quota rejection happens before provider bytes, no direct-S3 route exists, dot/prefix/tenant escape is refused, and create/part/complete/copy/delete replay converges after lost responses.
3. Host restart recovers the quota image, gateway WAL, staging object, and CP reservations before rclone or the guest is re-exposed.
4. A designated isolated KVM host passes the two-sandbox gold flow: one RW plus one RO, checksum visibility, detach, role swap, hard quota rejection, delete frees quota, restart, reattach, and persistence.
5. Fault tests pass for CP outage, S3 outage, emergency lock, lowered quota, rclone/virtiofsd/agent crash, bounded disk use, detach flush, and stale capability replay.
6. The frozen diff passes adversarial SOL review and Codex Security with no unresolved reportable finding.

Exact rollout sequence

1. Merge the normal PR to main; allow the additive migration and default-off CP code to deploy to Development.
2. Provision PT_GATEWAY_OBJECT_HASH_KEY through the documented environment secret flow. It must be stable across CP restarts and independent of auth key rotation. Never print or put it in git. Missing key fails closed.
3. Roll the candidate host binary while v1 advertisement remains off. Verify RO/direct API behavior and v0/null heartbeats. The canonical installer, manual host installer, and in-place host rollout all run the same root-only runtime reconciler: pinned rclone, fuse3, virtiofsd, pt-vms/pt-builds slices, root-owned attachment roots, and the narrow AppArmor mount rule. It refuses to mutate an active attachment and preserves its cache/spool. Before any host mutates, the rollout verifies the active CP manifest is the exact candidate; after each serial restart it reads the authoritative CP row for fresh shared_volume_v=1, exact storage fingerprint, candidate agent SHA, and gateway v0/null. A failed receipt stops the fleet immediately.
4. Arm v1 only on one designated test/canary host. Confirm its exact fresh fingerprint before granting any organization RW permission.
5. Enable the Admin RW switch only for the approved organization, run the gold flow again, then expand one host and organization at a time.
6. Promote main -> staging -> prod only through the normal fast-forward workflows and prove the exact SHA separately in each lane.

Emergency lock and rollback

The emergency Shared Volume write lock rejects subsequent growth reservations immediately. Reads, authorized deletes, aborts, detach, recovery, and reconciliation continue. Lowering quota below usage has the same data-safety shape: growth stops; reads and deletion remain available.

Rollback order is:

1. Disable global RW admission and per-org grants.
2. Stop new writes on every gateway; flush or retain every durable WAL and ambiguous reservation. Do not delete evidence to make detach succeed.
3. Detach guests, prove no RW rclone/gateway remains, and close capabilities.
4. Retract host v1 advertisements, then roll hosts back.
5. Roll back the control plane only after all live capabilities are closed.

Never release an uncertain reservation because a request timed out. A rollback that cannot resolve provider outcome leaves the reservation ambiguous and blocks that object target until recovery or reconciliation proves the result.

Recovery and observability

On restart the host mounts the attachment's quota image before reading WAL, queries CP state, resolves final/staging object metadata, replays the exact operation key, and only then starts rclone and virtiofs. Legacy RW metadata with no valid v1 gateway state is drained fail-closed.

The admin metrics endpoint exposes these rollout-critical series:

For reservation recovery, first disable new RW admission, identify the exact attachment/generation and operation state, and preserve its root-only WAL and quota image. Let the same-version host recovery query CP and provider metadata, then replay the stored idempotency key. An open pre-provider virtual upload is aborted centrally on restart/detach; a provider-inflight operation stays held until operation metadata proves success or absence. Never edit the ledger, delete a spool, or release a reservation merely to make the gauges reach zero. If recovery cannot converge, keep the capability closing and escalate with the WAL, capability generation, CP operation ID, and provider evidence intact.