Guest kernel modules on demand

A template carries only the modules a guest needs to reach its host (vsock, virtiofs). dockerd, Podman, k3s and WireGuard need more: bridge, veth, overlay, nf_tables and so on. The host attaches its guest kernel's whole module tree to every VM as one read-only disk, and the guest loads what a workload asks for. No template carries the tree, and no template needs a rebuild beyond picking up a current in-VM agent.

How it works

• At start, every host builds /var/platinum/data/templates/_kernel/kmod-<kver>-<sha16>.sqfs from its canonical module tree (host-agent guestkmod_disk.go). The build is reproducible and the tree is identical fleet-wide, so every host holds the same file under the same name. It takes about a second on the first start after a kernel change, and the file is 137 MiB per kernel. Images are never deleted, because memory snapshots name them.
• With the flag on, a cold boot attaches it readonly=on with serial pt-kmod.
• The in-VM agent mounts it ro,nosuid,nodev,noexec at /lib/modules/.pt-kmod and sets kernel.modprobe to /usr/local/bin/pt-modprobe, which is the agent itself (invm-agent modprobe.go). It loads Ubuntu's .ko.zst files through finit_module(2), so images need neither kmod nor a zstd-capable one.
• A resume whose snapshot names an image this host lacks cold-boots instead (loadAndResumeLocked).

On by default

Every lane ran with PT_GUEST_KMOD=1 by 2026-10-07 (both dev hosts, the 5 EU prod hosts, the US host), so the disk is now attached unless PT_GUEST_KMOD=0. A host builds its image at start, before it takes a guest, so a new host can no longer come up as the one host whose guests lose Docker.

To switch it off on a host, set PT_GUEST_KMOD=0 in /etc/platinum/host.env and restart platinum-host-agent. Snapshots taken with the disk keep restoring, because the image stays on disk.

What the guest sees

• A template built with an agent from before this change ignores the disk and boots exactly as before.
• The agent overlays the host's tree on the template's own /lib/modules/<kver>. The template's directory is the overlay's upper layer, so its files win and anything written there (depmod, dpkg, DKMS) lands on the root disk as before. An image's own modprobe (kmod with zstd, as in ubuntu and debian) then finds every module. The kernel's automatic loading (mount, socket, netlink, iptables) goes through pt-modprobe, which needs no kmod. Run pt-modprobe NAME to load one by hand.
• /etc/modprobe.d is not read by pt-modprobe.
• The agent sets kernel.modprobe last. pt-init (templates built from this change on) waits for it, at most 3 s, before it loads modules-load.d and starts the image's entrypoint, so an entrypoint that runs set -e; modprobe br_netfilter in its first millisecond works.

The rest of a normal VM's boot

pt-init runs no systemd or udev. What they do on a normal VM, a guest now gets too (platinum#1444):

The image's sysctl.d wins over the command-line values, as on any VM, except kernel.panic: it stays the host's, which reboots a panicked guest into the boot watchdog.

Security

The VM is the boundary. A sandbox user is already root in the guest, and root can already load any module. Modules bind only to the virtio devices the VM has. The disk is read-only on the host and in the guest. An alias never resolves to a module that Ubuntu's blacklist-rare-network.conf names, so an unprivileged socket(2) cannot load those protocols. A requested name is never read as a path.

Cost (Development, 2026-10-06)