Volumes end-to-end test plan

This is the release test for both Platinum storage products. Run it against an isolated control plane, database, object prefix, and designated KVM host. Never point it at the CAS namespace or a production bucket. Record real identifiers, timestamps, and failures in TESTED.md; a source-text assertion, compile, or green CI job is not end-to-end evidence.

Required topology

• Local control plane and disposable database with migrations 0052-0059 applied.
• One real Linux/KVM host running the candidate host-agent and Cloud Hypervisor.
• rclone, fuse3, and virtiofsd installed by infra/scripts/host-install.sh.
• A private disposable S3-compatible bucket or isolated non-CAS prefix.
• volumes_enabled, shared_volumes_enabled, and the test org's sharedVolumes grant enabled only for that org.
• A test API token, the CLI, TypeScript SDK, Python SDK, MCP server, and local dashboard pointed at the same control plane.

Before testing, prove that PT_VOLUME_S3_BUCKET and PT_VOLUME_S3_PREFIX do not equal, contain, or sit beneath the CAS bucket/prefix. Run cd apps/api && bun run env:check. Do not print credentials.

Evidence rules

For each case capture the API status/body, volume and sandbox ids, guest command output, attachment row state, and relevant host-agent log line. After cleanup, prove that no test VM, FUSE mount, rclone/virtiofsd process, cache loop device, credential file, or object prefix remains. Measure lifecycle latency where the case waits for host work; do not replace measurements with estimates.

1. Static and component gates

2. Feature gate and tenant boundary

1. With volumes_enabled=false, all volume routes and UI entry points are absent or denied as designed.
2. With base volumes enabled but Shared disabled, Local operations work and Shared create/attach/file/snapshot operations fail closed.
3. Grant Shared to org A only. Org A can use it; org B receives no volume data, attachment data, object metadata, presigned URL, or existence oracle.
4. Attempt every GET and mutation using another org's volume id, attachment id, snapshot id, and file cursor. Expect 404/denial, never cross-org content.
5. Verify admin access resolves storage under the volume owner's org, not the admin's current org.
6. Submit bucket names, endpoints, credentials, raw object keys, .., encoded traversal, invalid Unicode, control characters, and reserved marker names in public payloads. Expect rejection and no host command containing credentials.

3. Common volume resource API

Run each applicable operation through raw HTTP, CLI, TypeScript SDK, Python SDK, MCP, and dashboard. The clients must agree on snake_case public models.

4. Local Volume matrix

1. Create an unattached 1 GiB Local Volume. Prove available, selected host, sparse ext4 backing, and no sandbox attachment.
2. Create with sandbox_id. Prove placement on that sandbox's host and mounted state when the sandbox is running.
3. Create a sandbox using legacy volume_ids, then using the explicit volumes[] descriptor. Write a random marker and checksum it in the guest.
4. Hot-attach at a valid custom path. Verify the block device, ext4 mount, read/write, reported device path, and exactly one attachment.
5. Attempt attach to a sandbox on another host, a stopped/deleted sandbox, an occupied path, /, /etc, aliases, and traversal. All must fail without a leaked device or changed volume state.
6. Attempt a second writer/attachment. It must be rejected.
7. Detach while files are closed, then reattach to another sandbox on the same host. The marker and checksum must survive and the old guest must lose the device cleanly.
8. Stop/start the sandbox three times. The attachment returns at the same path and the marker survives each cycle.
9. Restart the host-agent while the VM runs. The VM and Local Volume remain usable and are re-adopted.
10. Delete the sandbox. The volume becomes available, data remains, and a new sandbox can attach it.
11. Kill Cloud Hypervisor and exercise failed-start/stale-reap cleanup. The volume must not remain falsely attached and must be recoverable.
12. Create a Local snapshot only while detached. Verify offline inode-holder proof, read-only e2fsck, reflink capture, second validation, and ready.
13. Modify the volume, detach, restore the snapshot, reattach, and prove the old marker/version is restored. A failed fsck or unsupported reflink must fail without changing the live backing inode.
14. Verify snapshot create/restore/delete races are fenced; attach and volume delete are denied while snapshot work is active; retained snapshots block volume deletion.
15. Delete every Local snapshot, delete the volume, and prove backing files and snapshot directories are gone while unrelated host data is untouched.

5. Shared Volume direct file API

Use nested Unicode paths and binary payloads in addition to simple text.

Inject S3 403, 404, 429, 5xx, timeout, malformed listing XML, and mid-stream disconnects. Responses must be bounded and redact endpoint, bucket, key, credential, and provider internals. Retrying must converge without escaping the org/volume namespace.

6. Shared Volume mount matrix

Mounted RW is currently unavailable. First prove that the Admin flag, API, stale host commands, both host constructors, and persisted recovery all fail closed. Run the RW cases below only after a versioned host gateway has durable mutation identity and ambiguous-outcome recovery; they are release gates, not claims about current behavior.

1. Attach rw to a running sandbox at /mnt/shared; verify API attachment state, host metadata, rclone mount, dedicated cache filesystem, virtiofsd, Cloud Hypervisor fs device, and guest mount.
2. Write in the guest, close the file, then poll the direct API from a separate client until its checksum appears. Write through the API and prove the guest sees it after the documented cache window.
3. Keep the same volume ro in a second sandbox while the first owns the sole future rw attachment. Prove the reader sees a completed write. Detach and swap roles, then prove the former reader can write and the former writer can read. Race two rw requests and prove exactly one wins; never claim multi-writer support.
4. Attach ro; reads work and create/write/rename/delete fail. Confirm host-side --read-only, not merely guest mount flags.
5. Attach a confined subpath. It exposes only that prefix. Test sibling access, absolute paths, .., %2e%2e, backslashes, duplicate separators, invalid Unicode, symlinks, and host mount aliases.
6. Verify unsupported semantics: no hard links, device nodes, advisory locks, or transactional multi-file rename; document expected symlink-dependent workload failures instead of silently corrupting them.
7. Reject quota before S3 accepts bytes and return a deterministic filesystem error from flush/close. Detach must preserve every ambiguous reservation and provider operation for recovery rather than acknowledge data loss.
8. Detach/reattach repeatedly and prove Cloud Hypervisor removes the exact fs device id; no PCI/fs-device, mount, process, loop device, or directory leak.
9. Stop/start and pause/resume three times. The attachment is reconstructed only after guest readiness, at the same path/mode/subpath, with data intact.
10. Restart the host-agent under a mounted volume. Heartbeats start promptly; recovery is bounded/concurrent; rclone and virtiofsd are adopted or safely replaced; the control plane converges to reality.
11. Relocate/restore a sandbox to another eligible host. Attachment ownership moves with generation fencing; stale commands from the old host cannot detach or overwrite the new attachment.
12. Kill rclone, virtiofsd, and Cloud Hypervisor separately. Each failure must be visible, bounded, reconciled, and recoverable without cross-attachment kills or credential/config leakage.
13. Fill the 10 GiB attachment cache and approach host low-space admission. The loopback cache boundary and 20 GiB free-space gate must protect the host; unrelated sandboxes and the host root filesystem remain healthy.
14. Apply CPU, memory, task, file-descriptor, and process churn to rclone and virtiofsd. Verify their cgroup/resource budgets contain one tenant without starving host-agent heartbeats or other VMs.
15. Simulate S3 outage during reads, writes, detach, stop, and host restart. Reads/writes fail honestly; acknowledged durable data remains; dirty cache is retained; recovery uploads after S3 returns.

7. Explicit volume snapshots and sandbox policies

For Shared snapshots, create a generation, mutate live data, restore, and prove the exact manifest generation returns. Verify manifest-last publication, staged object cleanup, writer quiescing/admission, immutable generation keys, usage accounting, snapshot deletion, reaper behavior, and stale-run fencing. Snapshot or restore while an rw attachment/direct-write lease is active must fail or quiesce according to the API contract; ro readers must never turn into writers.

For sandbox snapshot, backup, clone, fork, and restore, prove volume bytes are never silently embedded or copied. Attachment metadata must follow the explicit omit, reattach, or clone policy, be revalidated at commit time, and fail closed if the volume was deleted, renamed, reattached, or its beta grant was revoked during the operation. Shared ids in legacy volume_ids must be rejected.

8. Dashboard and client acceptance

• Dashboard: create both types, validation/errors, list/get, usage, rename, attach/detach, attachment list, file browser operations, snapshot create/restore/delete, tombstone/reaper progress, and responsive/loading/empty states. Refresh midway through each operation and verify state recovery.
• CLI: run every pt volume and pt volume files command, JSON and table output, stdin/stdout streaming, paths with spaces/Unicode, and non-zero exits.
• SDK TS/Python: run identical golden flows, including async Python, streaming bodies, pagination, 200 versus 202 attach, typed error fields, and no unsafe automatic retry of non-idempotent writes.
• MCP: confirm tools require the intended permission, never expose S3 configuration, and return bounded content rather than embedding large files.

9. Final soak, security, and cleanup gate

Run mixed Local + Shared attachments on several sandboxes for at least one hour: continuous checksum writes/reads, periodic stop/start, attach/detach, snapshots, agent restart, and one controlled S3 fault. Report operation counts, error rate, p50/p95/p99 latency, maximum unavailable interval, host root/data free space, process/RSS/task counts, and reconciliation lag.

Release behind the rollout gates only if there is no data mismatch, cross-org observation, false attached state, unbounded recovery, credential exposure, host-root growth, or leaked process/mount/device. Any one of those is a ship blocker. Finish by deleting test sandboxes, detaching and deleting volumes and snapshots, waiting for the Shared reaper, and proving the isolated object prefix is empty.