Storage nodes

Status: Implemented, opt-in, OFF by default

Scope: A disposable GET-only read accelerator in front of Scaleway S3 for the JuiceFS object store

Durability rule: Scaleway S3 remains authoritative. Nothing changes about it.

Governing decision: ADR-002 (reports/ADR-002-decision.md on the unified-volumes branch)

1. What a storage node is

One Go binary, storage-agent, on an enrolled box. It terminates an S3-compatible endpoint on the tailnet and does exactly one thing well: it answers a GET or HEAD for an object it already holds a confirmed copy of, without going to Scaleway.

Everything else — PUT, POST, DELETE, every listing, every multipart operation, every request it cannot confidently accelerate — it forwards to the provider unchanged and streams the provider's answer back verbatim.

JuiceFS client -> storage node -> Scaleway S3   (reads, sometimes short-circuited)
JuiceFS client -> storage node -> Scaleway S3   (writes, always, unbuffered)

2. What a storage node is NOT

This section is the point of the document. Read it before changing anything.

• It is not authoritative. Every byte under its cache root is a second copy of bytes that are already durable in Scaleway S3.
• It is not a write path. No write is buffered locally and acknowledged early. No write is transformed. A write's durability is byte-identical to talking to S3 directly, because it is talking to S3, through a socket this process owns. There is one function that can write a response header for a mutating verb, and it copies a response the provider has already sent.
• Losing it is a zero-data-loss event by construction. Disk failure, a mistaken mkfs, the whole rack: nothing is lost. There is nothing to drain, nothing to back up, nothing to replicate.
• It is not a step towards a write-back tier. ADR-002 gates any write-back tier behind measurement at equivalent durability, a second node so N>1, and an existing maintained S3 server. None of that is in scope here.
• It cannot break a read. Every local failure degrades to transparent pass-through. See §4.

3. Cache validity — the hard problem

A JuiceFS metadata restore (juicefs load, or a PITR of the metadata Postgres) regresses the slice counter. The same object key can therefore legitimately hold different bytes afterwards, sometimes under the same ETag. A cache keyed by object key alone would serve the pre-restore body for a post-restore key and corrupt the filesystem silently, with no error anywhere.

Four mechanisms, all of which must hold:

1. Entries are keyed by (generation, key, ETag). The on-disk name of an entry is sha256(generation || key || ETag). A body is never returned for an ETag this node has not confirmed upstream.
2. A control-plane cache generation, delivered on every heartbeat. Any change to it retires the entire index synchronously, before the heartbeat returns, and the retired generation's directory is unlinked in the background. A generation that moves backwards is adopted too, and logged: a restore is precisely the event that regresses a counter.
3. Unknown generation means bypass. Before the first heartbeat, after a heartbeat that omits the generation, and after the control plane has been unreachable past the fail-closed window, the generation is 0 and the node caches nothing and serves nothing from cache. It is a plain proxy.
4. Conditional revalidation on a TTL. An entry whose confirmation is older than PT_STORAGE_REVALIDATE_TTL_SECONDS is re-proved with an upstream If-None-Match before a byte of it is served. A 304 restarts the clock without moving data; anything else is streamed from the provider and the old entry is superseded.

The test that pins this is TestGenerationBumpPreventsAStaleBodyFromEverBeingServed: it replays the restore — same key, same ETag, different bytes — and asserts the pre-restore body can never come back.

4. Fail-open

A storage node must never be able to break a read. Each of the following degrades to pass-through, not to an error:

The single exception is authentication, which fails closed. Fail-open is about this node's storage never breaking a read; it is not about letting an unauthenticated caller reach a bucket through a credential it does not hold.

5. Security boundary

• Listens on the tailnet or the loopback only. The process refuses to start if PT_STORAGE_LISTEN is the wildcard or resolves to any public address, and the installer refuses the same addresses before you get that far.

• Never port 18048 — that is the vsock-RPC port every host already serves. The default is 18055, and the agent refuses 18048 explicitly. The loopback admin/metrics listener is 127.0.0.1:19093.

• Callers authenticate with SigV4 against a local identity. The caller is a JuiceFS client, which already speaks S3 and already carries an access-key/secret pair, so this costs one configuration line and no new machinery. mTLS would need a per-host certificate lifecycle for a client whose S3 backend does not expose client-certificate configuration, and — the deciding point — a client certificate authenticates the connection while SigV4 authenticates each request against its own method, path, query and signed headers. A node that re-signs every request with a provider credential should be verifying the request, not the pipe.

• The provider credential is unreachable from the request path. The agent drops the caller's Authorization header and re-signs with the provider credential in exactly one function. The two credentials are required to differ, and the agent refuses to start if they do not. Neither ever appears in argv, in a unit file, in a log line, or in the installer.

• The data endpoint is TLS. The installer generates a self-signed P-256 keypair at install time whose only SAN is the approved endpoint host — an IP SAN, because on this fleet that host is a tailnet address. The certificate is 0644 root-owned at /etc/platinum/storage-data-plane-cert.pem, the key is 0600 owned by the service account at /etc/platinum/storage-data-plane-key.pem, and the agent parses both at startup: an unreadable, malformed or group-readable key is a process that refuses to bind, never one that quietly serves plaintext. A non-loopback listener requires a keypair outright. The loopback admin listener stays plain HTTP — it binds a concrete loopback address and carries no credential. Both the SPKI pin and the certificate's own SHA-256 are reported at register and recorded on the node row.

• Credentials live in 0600 files owned by the platinum-storage service account, delivered in the register response over TLS. Group- or world-readable credential files are refused at startup.

• The object-store credential a node is issued must be prefix-scoped and carry NO delete permission. A storage node is a disposable box on a tailnet that terminates an S3 endpoint for other machines; the credential it holds is the blast radius of losing one. The control plane issues PT_STORAGE_NODE_UPSTREAM_ACCESS_KEY / PT_STORAGE_NODE_UPSTREAM_SECRET_KEY when they are set, and that pair is the production posture: scoped to the bucket prefix whose objects the node may serve, read-only.

  The PT_S3_* fallback is a rig convenience and NOT the production posture.

  When the scoped pair is unset, the control plane falls back to its OWN PT_S3_ACCESS_KEY / PT_S3_SECRET_KEY so a node can be enrolled on a development fleet before a scoped key has been cut. That hands the node the control plane's full-bucket credential — delete included — which is far more authority than a storage node should ever hold. The split exists so closing the gap is a configuration change, not a code change. GET /v1/admin/storage-nodes reports metadata.upstream_credential_scoped per node; on a production fleet every node must read true.

• The node pins itself to one bucket and one prefix, independently of what its credential could reach. PT_STORAGE_UPSTREAM_BUCKET is required — the agent refuses to start without it — and PT_STORAGE_UPSTREAM_PREFIX bounds it further; both come from the approved plan. Every request outside them is refused with a 403 before it is re-signed with the provider credential, so a caller's reach here is the intersection of the credential and the pin. A bucket-level listing must name a prefix under the pinned one, or it is refused too. This is the in-code half of the scoping above: the scoped key is a deployment concern and this is what holds when it has not been done, when it was done wrongly, or when a bug in this process would otherwise have turned one node into fleet-wide object access.

• A caller's preconditions are the provider's to evaluate. A request carrying If-None-Match, If-Match, If-Modified-Since, If-Unmodified-Since or If-Range is forwarded untouched and never served from cache or used to revalidate one. The node's own revalidation strips them and sends only the ETag it holds. Forwarding them was a real hole: under S3's precedence rule a caller-supplied future If-Modified-Since turned the node's "is this still my ETag?" into a 304 for an object that had changed, after which the node served the pre-change body as a confirmed hit — a caller could make the node lie to itself.

• A Range is answered as a Range, or not by this node. A ranged request it cannot satisfy from cache is forwarded with its Range intact and the provider's 206 is returned verbatim. It is never answered with a full-body 200: JuiceFS's s3client.Get does not check 206 against 200, so such a response is accepted and read at the requested offset — silent corruption of a tenant's data rather than a cache miss.

• The unit runs as a dedicated non-root account with an empty CapabilityBoundingSet, NoNewPrivileges=true, ProtectSystem=strict, MemoryDenyWriteExecute=true, and a generated drop-in that names the one mount it may write.

• The installer is served unauthenticated on purpose, at GET /install-storage-node.sh, because it embeds no secret. It is shaped after infra/scripts/install-cache-node.sh, not after /install.sh, which is known to embed fleet root SSH material and object-storage credentials. apps/api/src/installStorageNodeSh.guards.test.ts pins both halves of that claim.

6. Opting a host in and out

The feature is OFF by default and reversible by configuration alone.

• PT_STORAGE_NODE_ENABLED in /etc/platinum/storage-agent.env is the switch. Anything but 1 and the agent logs one line and exits 0 without binding a socket. Flip it and systemctl restart platinum-storage-agent.
• A compute host is pointed at a storage node separately, and deliberately by hand. ADR-002 cancelled the automatic data-plane change (juicefs config --bucket, the forwarder, epochs), so there is no code in this repo that redirects a JuiceFS mount at a storage node. Doing it is a per-host, per-filesystem operator decision, and undoing it is the same decision reversed; neither touches this agent.

Because the node holds nothing authoritative, "opting out" needs no coordination: stop it, and every reader goes to Scaleway as it did before.

7. Enrollment

Two phases, like the cache node.

# On the box, as root. The token is minted in the admin panel and pasted at a
# hidden prompt; it never reaches argv.
curl -fsSLo /tmp/pt-storage-install.sh https://api.platinum.dev/install-storage-node.sh
read -rs PT_TOKEN; printf '%s\n' "$PT_TOKEN" | \
  bash /tmp/pt-storage-install.sh --enroll \
    --control-plane https://api.platinum.dev \
    --node <short-name> \
    --token-stdin
unset PT_TOKEN

Phase 1 is read-only: it posts a device and network inventory, writes the token to /etc/platinum/storage-enroll-token (0600 root), installs itself at /usr/local/libexec/platinum-storage-enroll-apply, and enables platinum-storage-enroll.service.

An admin then approves a plan in the admin panel — either a whole blank disk or, on a box whose disks all back /, a directory on an existing filesystem. The approved plan is what the box polls for, and the control plane answers it with, in one response:

Nothing in that response is a secret, which is the rule the route is held to: it is polled repeatedly by a bearer sitting on the disk of a box that is not yet finished being provisioned. The credentials arrive exactly once, in the register response, and are written straight to 0600 files:

The box verifies binary_sha256 before installing the binary, and generates its data-plane certificate against the approved endpoint host before it registers.

Phase 2 then re-verifies every claim the plan makes on the box itself: /dev/disk/by-id addressing, not an OS disk, not mounted, no holders, not an LVM PV, not an md member, the approved serial still present, an exclusive flock, and a blank signature (or an operator re-typing the serial at the terminal). Only then does it format.

Follow it with:

journalctl -fu platinum-storage-enroll

8. Verification

# The process is up and knows which generation it serves. generation 0 means it
# is proxying everything, which is correct before the first heartbeat lands.
curl -sS localhost:19093/statusz | python3 -m json.tool

# Counters. hits/misses/bypass/passthrough are the four that matter.
curl -sS localhost:19093/metrics | grep platinum_storage_node_

# It is listening on the tailnet, not on the wildcard and not on the public IP.
ss -ltnp | grep platinum-storage-agent

# The unit is hardened as intended.
systemd-analyze security platinum-storage-agent.service | head -20

# The boot-drift marker names this node.
cat /var/lib/platinum-storage/.platinum-storage-node
systemctl show -p Environment platinum-storage-agent   # EnvironmentFile is INVISIBLE here
grep PT_STORAGE_NODE_ENABLED /etc/platinum/storage-agent.env

# End to end, as a caller: a signed GET, twice. The second must not reach
# Scaleway, which the miss/hit counters prove.
curl -sS localhost:19093/metrics | grep -E 'hits_total|misses_total'

A healthy node's heartbeat carries cache_generation non-zero, disk_healthy true, and no serve_refusal. A serve_refusal names why the node stopped caching; it never means the node stopped answering.

9. Drain

drain means stop being selected for reads and let placements move away. It does not flush anything, because there is nothing un-shipped to flush.

Set drain on the node in the admin panel. The next heartbeat response carries it, and the node then:

• reports /readyz as 503, so placement stops choosing it;
• stops admitting new cache entries;
• keeps answering every request still pointed at it, from cache or from the provider.

Clearing drain is the same directive with false. Nothing is lost either way.

10. Upgrade

# Build (the binary lands in dist/storage-agent alongside the others)
bash infra/build-host-binaries.sh

# Roll it: the node is disposable, so a hard restart is a valid deploy.
install -m 0755 -o root -g root dist/storage-agent /usr/local/bin/platinum-storage-agent
systemctl restart platinum-storage-agent
curl -sS localhost:19093/statusz

A restart starts with generation 0, so the node proxies everything until its first heartbeat, then re-indexes the on-disk bodies for that generation. The worst case of a botched upgrade is a cold cache, never a lost byte.

11. Rollback

In increasing order of severity, each of which is complete on its own:

1. Turn it off. PT_STORAGE_NODE_ENABLED=0 in /etc/platinum/storage-agent.env, then systemctl restart platinum-storage-agent. The unit stays installed and inert.
2. Stop it. systemctl disable --now platinum-storage-agent.
3. Remove it. Stop the unit, rm -rf /var/lib/platinum-storage/objects, remove the unit files and /etc/platinum/storage-*. Rotate the two credentials the node held at their provider.

There is no step where data has to be recovered from the node first, and no ordering constraint with anything else. That is the whole point of ADR-002.

12. Known gaps

• A node's local_credentials are returned once and never stored. The control plane keeps only the access key id (metadata.local_access_key_id), so an operator can see which identity a node enforces without the control plane holding the means to use it. The consequence is that there is no distribution path to a compute host yet: pointing a JuiceFS mount at a node means capturing the pair from the register output at enrollment time and configuring the client by hand. That is consistent with §6 — a compute host is pointed at a node deliberately, by an operator — but it is the piece to build next.
• A HEAD is answered from the index, a GET from the verified body. A HEAD on an entry whose body has been corrupted still returns 200 with the right length while a GET falls through to the provider. There is no body, so no wrong byte reaches anyone — but the two answers disagree until the GET discards the entry. Verifying a body to answer a request that has no body was not judged worth the read.
• An out-of-band overwrite is served stale for up to revalidate_ttl (300 s by default). Nothing closes that window except the immutable-key restriction below, which is what makes it unreachable in practice rather than merely unlikely.
• The cache serves only write-once keys. Admission is restricted to JuiceFS slice blocks and CAS chunks; every other object, including anything under a prefix the agent has not seen, is proxied uncached. That is what makes the node safe without a coherence protocol — a mutation landing at the provider directly cannot invalidate anything, because no cached key can be given different bytes. A new immutable namespace on this bucket will be proxied rather than accelerated until it is named.
• The heartbeat request body must be validated non-strictly on the control plane. A .strict() heartbeat schema caused a P0 in PR #759: one field an agent did not send turned every beat in the fleet into a 400. The agent omits optional fields rather than sending nulls, and the CP must ignore fields it does not know. The RESPONSE has the mirrored hazard and it has already bitten: the agent required cache_generation and the control plane never sent it, so every beat failed its own decode and the cache was permanently bypassed. The response shape is now recorded from the real route into hosts/host-agent/cmd/storage-agent/testdata/control-plane-heartbeat.json and fed to the real decoder by the agent's own test; neither side hand-writes it.
• The heartbeat request body must be validated non-strictly on the control plane. A .strict() heartbeat schema caused a P0 in PR #759: one field an agent did not send turned every beat in the fleet into a 400. The agent omits optional fields rather than sending nulls, and the CP must ignore fields it does not know.
• The installer implements a subset of install-cache-node.sh's device forensics. The deeper closure walks (loop members, md parent chains, non-NVMe parents) are not reproduced; the signature gate is what stands between a mistake and a wipe.
• Nothing here is measured. Every latency claim is NOT MEASURED until the ADR-002 benchmark harness produces one at matched durability.