Three Proxmox-on-ZFS installs, three UEFI shells

Wanted to bring up one Proxmox node on new hardware. Familiar layout: ZFS RAID-1 across two NVMe disks, root on the pool, snapshots + send/receive into Proxmox Backup Server. Hour and a half from order to production.

Burned a day instead.

What happened

The installer runs clean: GPT layout on both disks, EFI System Partitions written to each NVMe (mirror requires it), ZFS pool assembles, system lands in rpool/ROOT/pve-1. Install finishes. Reboot.

Server drops into the UEFI Interactive Shell.

Three attempts, three paths:

→ Native installer via netboot.xyz — network auto-detect tripped on the multi-VLAN config (host IP + dedicated subnet on different VLANs), rolled back before installer started → qemu-iso from rescue mode — installer in a QEMU VM with disk passthrough, ISO in RAM, VNC on :5900. Install completes. Reboot. UEFI shell. → Native installer again with IP/Gateway/DNS pre-filled — install completes. Reboot. UEFI shell.

After each attempt I checked the Mapping table from inside the UEFI shell — the GPT GUIDs of the EFI partitions changed between attempts. The installer was writing to disk correctly. Something else was failing systematically.

Where the bug is

Proxmox boots differently depending on root filesystem.

On LVM, Proxmox uses GRUB. The installer drops grubx64.efi in two places: the path registered through efibootmgr into UEFI NVRAM — and the fallback path \EFI\BOOT\BOOTX64.EFI. This fallback is the removable-media path that every UEFI firmware on the planet checks if no NVRAM entry matches. It's the safety net for "NVRAM got wiped, firmware was reflashed, disk was moved to another server."

On ZFS, Proxmox can't use GRUB reliably (the GRUB ZFS module exists but is a lottery across pool versions and feature flags). It switches to systemd-boot instead. The bootloader systemd-bootx64.efi lives only in \EFI\systemd\ on each ESP. Proxmox does not populate the \EFI\BOOT\BOOTX64.EFI fallback for systemd-boot. Booting depends entirely on the UEFI NVRAM entries that proxmox-boot-tool refresh registers via efibootmgr.

If a particular BMC doesn't persist that NVRAM entry across cold reboots — and on whitebox AMI BIOS dedicated servers, this happens — the fallback would have saved you, but the fallback isn't there.

So:

→ On LVM the symptom doesn't reproduce — fallback covers it → On ZFS it reproduces only when the BMC mishandles UEFI variables → Vendor docs don't warn about this specifically. I couldn't find anything

Safety-net recipe (will verify on the next install — don't apply blindly without testing)

Once Proxmox is up — through a manual UEFI shell boot, a fresh install via virtual media, or any other path — copy the systemd-boot binary into the fallback path on each ESP:

cp /boot/efi/EFI/systemd/systemd-bootx64.efi /boot/efi/EFI/BOOT/BOOTX64.EFI
 
# second ESP — needs to be mounted manually
proxmox-boot-tool init /dev/nvme1n1p2
mount /dev/nvme1n1p2 /mnt/esp
cp /mnt/esp/EFI/systemd/systemd-bootx64.efi /mnt/esp/EFI/BOOT/BOOTX64.EFI
umount /mnt/esp

After this, the server should boot from disk regardless of what's in NVRAM. Either disk in the mirror can boot the system through the standard fallback path. That's the whole point of the mirror — at the bootloader level too, not just the data layer.

Belt and suspenders, as they say.

→ ZFS-on-root for Proxmox is great. Snapshots, send/receive into PBS, compression, ARC. Not giving it up. → But on someone else's hardware with unpredictable BMC firmware — after install, manually drop a fallback BOOTX64.EFI copy of systemd-boot on both ESPs. Costs a minute. → Vendor docs don't cover the UEFI + ZFS + systemd-boot intersection anywhere I've found. A gap, not a bug — but you only think of it after you've reinstalled the system from scratch three times in one evening.

(though you'd think)

A day gone. The lesson stays.