Thought it was one thing, turned out to be another: 4Kn drives broke the ZFS boot
This morning I wrote: three Proxmox-on-ZFS installs, three UEFI shells. The root cause supposedly in the boot chain — systemd-boot doesn't populate the \EFI\BOOT\BOOTX64.EFI fallback path, everything depends on UEFI NVRAM, and on whitebox hardware NVRAM can lose entries. Workaround was to copy the bootloader into the fallback path by hand.
Good theory. Not my actual bug.
What was actually happening
The provider's support spent several hours on diagnosis: kernel config tests, ZFS parameter sweeps, different install paths. They went a layer deeper than where I was looking.
Two NVMe drives, Intel P4510, with native 4 KB sector size (4Kn). On paper that's fine: the UEFI spec supports 4Kn boot. In practice — the specific combination of PM9 + ZFS + this BIOS firmware doesn't handle it. The installer writes the disks, the pool assembles, the EFI partitions get created. But the BIOS can't correctly read an ESP off a 4Kn drive at boot time. Result — UEFI shell.
This is why most providers ship 512e drives (physically 4 KB, emulated 512 B at the logical level) — to avoid this exact class of bug. The P4510 in its 4Kn variant is an older enterprise series, optimized for peak performance, not universal compatibility.
Resolution — physical swap to Micron 9300 Pro with native 512 B. The install runs clean, the server boots from disk without ceremony.
Where I was wrong
Symptom — yes, correct: boot drops to UEFI shell, NVRAM-dependent. Mechanism — I invented a more general one: systemd-boot doesn't write the fallback. That is a real characteristic of Proxmox + ZFS on whitebox hardware with unpredictable BMC firmware, and the workaround (manually copying BOOTX64.EFI) is generically useful. Just not my bug this time.
Honestly: I was looking at the OS + bootloader layer. The provider's support went a layer below — firmware vs physical sector format of the drive. That's a layer I didn't think to check.
Misdiagnosis acknowledged. Useful — next time I'll look there earlier.
Takeaways
→ The "UEFI shell on ZFS install" symptom has at least two distinct roots: bootloader-level (systemd-boot fallback gap) and hardware-level (4Kn drive + finicky BIOS). Both traps are real.
→ When checking GPT GUIDs after install confirms the drives ARE being rewritten but the server still won't boot — ask about the drive sector format (nvme id-ns /dev/nvme0n1 shows 512 vs 4096). I didn't ask.
→ The cp BOOTX64.EFI recipe stays in the toolkit as a generic insurance against a different class of bug. Belt and suspenders earn their keep.
→ Good support is when they swap hardware instead of giving you a templated reply. Several hours of real work, transparent technical answer, physical drive replacement. Respect earned back.
The day's gone. The lesson stayed — different from the one I expected this morning.