Fixed NVFP4 on Blackwell — it was missing CUDA 12.9
This morning I wrote that NVFP4 on Blackwell only worked for one model — Gemma-26B and Qwen3.6-27B both failed. By evening I'd found why.
The attention-backend workaround got the dense Qwen3.6-27B further, but the server died again — a new error: No supported CUDA architectures found for major versions [12]. Not FlashInfer's fault this time — the FP8 kernel JIT-compile for SM120 needs CUDA ≥12.9, and the box had 12.8.
Installed 12.9 alongside it, left the old one untouched. Qwen came up.
Tested Gemma-26B in the same environment — came up too, no tie_word_embeddings:false hack, tied embeddings intact. Looks like this morning's tie_weights theory in vLLM was a red herring: the CUDA mismatch alone was killing weight loading, and the checkpoint re-export was never needed.
Then I measured memory. This morning's ~14GB for Gemma and ~15GB for Qwen were plain arithmetic — params × 4 bits. The real number at load time:
→ Gemma-4-26B-A4B-NVFP4 — 18.16 GB → Qwen3.6-27B-NVFP4 — 20.0 GB
An NVFP4 checkpoint carries block-level scaling factors and unquantized attention/embedding layers — that's the extra 4-5GB the formula missed. Neither model fits 16GB — that tier still belongs to Gemma-12B-fp8 alone. Both fit 24GB, but tight: Gemma leaves ~5.8GB for KV cache, Qwen about 4GB.
Ran the full benchmark (ruAIME + ruMMLU) on both overnight:
| Model | Quant | ruAIME | ruMMLU |
|---|---|---|---|
| Gemma-4-26B-A4B | NVFP4 | 0.7333 | 0.8280 |
| Qwen3.6-27B | NVFP4 | 0.7333 | 0.8162 |
| Qwen3.6-27B | bf16 | 0.833 | 0.8156 |
| Gemma-4-12B | fp8 | 0.700 | 0.8103 |
Gemma-26B-A4B in NVFP4 beat both the 12B and its own bf16 flagship on knowledge. Makes sense — it's bigger, 18GB against the 12B's 16, and that's the honest price of the gain, not a free lunch. The gap against this morning's QAT-GGUF row for the same model (0.567 / 0.7006) is huge, +0.13-0.17, but that's not a same-precision comparison — two different quantization methods. The quant was weak, not the model.
The dense Qwen3.6-27B tells a plainer story — the shape a real quantization cost should have on the same model: knowledge holds (0.8162 vs 0.8156, noise), math drops ten points (0.7333 vs 0.833).
The fix is reproducible: two environment variables plus one toolkit package, installed once on the box. Table's live: bench.csylabs.com/ru.