Enrolling a cache node on an md array

Audience: whoever is standing up a regional cache node on a box whose disks all carry the OS.

Scope: device-mode enrollment where the target is a Linux RAID array the operator assembled, rather than a spare whole disk.

Durability rule, unchanged: cache bytes are disposable and Object Storage remains authoritative. Nothing in this runbook risks data that matters. What it risks is a reinstall window and a one-time enrollment token.

1. The ordering trap

Real bare metal has no spare disk. Every disk carries a partition of the OS, so whole-disk device mode has nothing to offer and directory mode has nowhere with enough free space. The way out is to reinstall with the OS on a small mirror and give the remainder to an array the cache owns outright:

nvme0n1p1 + nvme1n1p1   64 GiB each   -> md0  RAID1  ext4  /
nvme0n1p2 + nvme1n1p2   remainder     -> md1  RAID0  ext4  /var/platinum/cache

Do not reinstall the box until array support has shipped to the control plane this box enrolls against. After the reinstall both disks carry OS partitions, so whole-disk device mode is dead, and the 64 GiB mirror is below the directory-mode floor. The box would have no enrollable target at all and would sit with zero cache until the code lands.

While you wait, the existing capacity-raise route can lift a directory-mode node to 0.7 × (available + used) on its current filesystem. One click, no code, and it costs nothing to undo.

2. Build the array

Partition at install time: p1 = 64 GiB on each disk, RAID1, ext4, /. p2 = the remainder on each disk, no filesystem and no RAID unless the installer offers RAID0 on p2 directly.

If you are building it by hand:

mdadm --create /dev/md1 --homehost=any --metadata=1.2 --level=0 --chunk=512 \
      --raid-devices=2 /dev/nvme0n1p2 /dev/nvme1n1p2

--homehost=any is mandatory, not a preference. mdadm records a homehost in the array metadata; an array pinned to the name of the box that created it does not auto-assemble on a box that answers to a different name, which is what a rename or a reimage produces. udev then names it /dev/md/<host>:<name> instead of /dev/md/<name>, the enrollment refuses it, and the refusal names the fix:

mdadm --assemble --update=homehost --homehost=any /dev/md1

RAID0 is deliberate. One member failure takes the whole cache, and that is the correct trade: the bytes are disposable, the node drops out of region topology within the freshness window, and hosts fall through to Object Storage at a reduced hit rate. RAID1 would halve the capacity to buy redundancy the cache does not need.

3. Check the array before minting anything

sudo infra/scripts/install-cache-node.sh --check-array /dev/md1

Read-only. It takes the device from the command line and from nowhere else: no token, no control plane, no persisted state, and it writes nothing, mounts nothing and reports nothing. It prints every fact the inventory will report — the array's uuid link, its size, its members with the disks, models and serials under them — and every reason a plan for it would be refused, not just the first. It exits non-zero when it would be refused.

Fix everything it names before minting a token. The refusals it can print:

wipefs -a /dev/md1 is the one unguarded destructive command in this whole flow. It is on a non-boot device, and --check-array tells you in advance whether you need it and names the exact device. Re-run --check-array after it.

4. Enrol

1. Mint a discovery token in the admin panel.
2. Paste the printed install command on the box. Phase 1 probes disks and NICs read-only, reports an inventory, installs a systemd oneshot to continue in the background, and exits. It is safe to close the terminal.
3. In the panel the array appears in "format a whole device", typed md · nvme, with its members listed.
4. The confirm dialog enumerates every member — partition, size, parent disk, model, serial — and states what the array currently carries. Read it against the hardware. The type-to-confirm phrase is the device and its size, e.g. md1 3.96TB.
5. Accept the prefilled capacity. It is the whole device, and that is the number to take: the installer formats this array for the cache alone, so the budget is the filesystem. The 1–2% that mkfs spends on the md data offset and ext4's own metadata is not knowable from the panel, and the box measures the real filesystem after mounting it and clamps the budget to what is actually there, logging the clamp and posting it as a stage message.
6. Approve. Phase 2 formats, mounts by UUID with nofail, writes the marker, and registers.

What phase 2 does that is worth knowing

• mkfs.ext4 -m 1. ext4 reserves 5% for root by default, which on a 3.96 TB array is ~197 GB the cache can never reach. -m 1 recovers ~158 GB of it and still leaves ~39 GB the kernel enforces against the unprivileged platinum-cache account, underneath the store's own sampled free-space floor. Two independent layers, one of them not ours to get wrong. No -i, no -N, no -E stride: default geometry already clears the inode invariant by ~3.7x.
• The inode invariant is asserted, not assumed. After the mount, the installer checks the filesystem holds at least 1.5× the store's index-entry cap in inodes. A filesystem this run created that fails it is a die naming the shortfall — reformat with a larger inode count and re-run. A filesystem it was handed is not its to reformat and runs conservatively instead.
• Watermarks are measured, not chosen. 0.92/0.89 is written only when the freshly mounted filesystem has ≥98% of its blocks available and the inode assertion passed. Anything else gets 0.80/0.70 with the reason logged. This is never keyed on the storage mode, because a device-mode apply also runs over a filesystem an earlier crashed attempt left behind.
• A capacity the filesystem cannot hold is clamped, not fatal — unless the clamp would cost more than 10% of the plan, which means the wrong device, not an overhead estimate. Before this, the mismatch was a die after mkfs had already run, and re-planning a used token answers 409, so the oneshot retried the identical plan forever against a destroyed disk.

5. Accept the node

Do all of this before it carries traffic.

1. Reboot the box twice. After each: systemctl is-active platinum-cache-agent, no refusing to serve in the journal, and no fault badge in the panel.
2. Confirm the panel shows filesystem_size as the raise basis and a capacity within 2% of the array size.
3. Confirm on the box:

   stat -f -c '%a %b %c' /var/platinum/cache        # bavail blocks inodes
   grep WATERMARK /etc/platinum/cache-agent.env      # expect 0.92 / 0.89
   systemctl is-enabled platinum-cache-mount-missing.service

4. Record the cold-scan time from /startupz at a representative fill. That number is the day-2 acceptance gate for the deploy timeout.

The mount-missing alert

The agent unit carries ConditionPathIsMountPoint on the data root. That is what stops it from writing a whole array's worth of cache onto the OS disk if the array does not come back from a reboot — but systemd counts a unit skipped by a failed condition as a successful no-op, so OnFailure= never fires and nothing tells anyone.

platinum-cache-mount-missing.service inverts the same condition, so exactly one of the two runs on every boot and the other one shouts. It is installed and enabled only where the data root really is a mount point, and it uses the same alert script as the agent's own failure unit. If that unit is missing or its alert path is broken, a missing mount is silent again — which is exactly why step 5.3 checks it is enabled.

6. When a member disk dies

The array is gone and so is the cache. Nothing durable is lost.

mdadm --stop /dev/md1
# replace the disk, then recreate the array exactly as in section 2
mdadm --create /dev/md1 --homehost=any --metadata=1.2 --level=0 --chunk=512 \
      --raid-devices=2 /dev/nvme0n1p2 /dev/nvme1n1p2
mkfs.ext4 -q -F -m 1 -L ptcache /dev/md1
mount /dev/md1 /var/platinum/cache
sudo infra/scripts/install-cache-node.sh --rearm-storage-guard

The recreated array is a new filesystem: the marker naming this node went with the old one and the fstab entry addresses a UUID that no longer exists, so the storage guard correctly refuses to serve. --rearm-storage-guard reads the node id out of the installed env file, refuses a filesystem whose marker names a different node, rewrites the marker, refreshes the fstab entry, points the guard at the mount and restarts the agent.

No token, no control-plane call, no re-enrollment. The node keeps its id, its credentials and its place in region topology — re-enrolling would churn the one identifier all three are keyed on, for a disk swap.

7. Testing this without hardware

infra/test/loop-md-rig.sh builds a real md array out of two non-sparse loop images and runs the real managed enrollment against it: real mkfs, real fstab, real systemd units, real service account, real marker, real registration. It runs on any Linux box.

sudo infra/test/loop-md-rig.sh \
  --control-plane-url http://100.x.x.x:3002 --admin-token pt_live_... \
  --size-gib 4 --level 0 --scenario happy

Scenarios: happy, m1, m5, inodes, clamp, clamp10x, signature, homehost. Then --keep, reboot, and --reboot-check; or --break-array to prove what happens when the array does not come back. --teardown-only is complete and idempotent.

PT_CACHE_INSTALL_ALLOW_LOOP_MEMBERS=1 is the one documented door through the loop-member refusal. It must be set in the environment of the install itself; no plan and no control plane can set it, and every other guard still applies in full.

What a loop rig cannot prove: loop devices do not survive a reboot, so --reboot-check re-attaches them. Boot-time reassembly of a real NVMe array by that box's initramfs, the array's exact data offset and therefore the final df size to within 1–2%, real NVMe throughput at multi-terabyte fill, and days-long fill-and-evict behaviour are all only observable on the hardware.

8. Residual risk you are accepting

1. RAID0 has no redundancy. One NVMe failure takes the entire cache. The mitigation is the stale-node alert and fall-through to Object Storage, not redundancy.
2. One reinstall window with zero cache on that node.
3. wipefs -a /dev/mdX is unguarded. It is the one place a typo is destructive, and it is on a non-boot device that --check-array names for you in advance.
4. filesystem_total_bytes, min_free_bytes and the storage guard are self-reported by a root-run agent. The raised ceiling is a guardrail against an admin typo, not a security boundary. Root already edits PT_CACHE_CAPACITY_BYTES on the box directly. This is consistent with the threat model and is stated so nobody later mistakes it for a boundary.
5. The final budget may land 1–2% under the array size. It will be a logged clamp visible in the panel as a stage message, not a failure.
6. A disk replacement is handled on the box, not in the control plane. --rearm-storage-guard covers the box; the control-plane-side story for a replaced node is unchanged.