Russian open models on my own Blackwell — and where NVFP4 breaks
I rented an RTX PRO 6000 (96 GB, Blackwell) on Selectel for a week, ~314₽/hour, and ran a ladder of open models through two Russian benchmarks: ruAIME-2025 (30 olympiad problems) and ruMMLU from MERA (14,012 knowledge questions).
The question wasn't "which model is smartest" — closed cloud models lead the raw score anyway. It was the practical one: what open model actually fits a card you own inside Russia, and at what quality.
Spoiler: the measurement beat intuition again. And NVFP4, the format you buy this card for, came up for only one of three.
The rakes: where NVFP4 breaks on Blackwell
Starting here, honestly. NVFP4 is a 4-bit format that on Blackwell gives near-fp8 quality at half the memory. It's what lets a 26–35B model fit a client's 24 GB card. That was the whole point.
Of the three models I wanted in NVFP4, one came up.
Qwen3.6-35B-A3B (MoE) ran clean: 0.800 on AIME, ~22 GB of weights. It's the base we were already planning to serve.
Gemma-4-26B-A4B died at init. A bug in vLLM itself: after a recent refactor, tying the embedding weights to the model head was never implemented for quantized methods. Any tied-embedding model from an NVFP4 checkpoint crashes on startup, and Gemma is exactly that.
The next part is on me, not the model. I figured I'd just turn tying off with a flag — tie_word_embeddings: false. The crash stopped; the model started emitting garbage. Turning tying off left the head uninitialized: I didn't fix it, I broke it more quietly. Reading one answer with my own eyes corrected me in a minute. The right path is different: re-export the checkpoint with a materialized, untied head. Or TensorRT-LLM, which doesn't have this class of bug by engine design.
Qwen3.6-27B (dense) died differently: FlashInfer requires GPUs with sm75 or higher. Absurd on its face, since SM120 is newer than SM75. It's a version-comparison bug in FlashInfer on Blackwell, specifically on the dense-attention path: the MoE path (that same 35B) ran fine on the same card. Workaround: switch the attention backend. Root fix: FlashInfer built for SM120.
Honest takeaway from the rakes: NVFP4 on Blackwell via vLLM is, today, immature for exactly the two classes you most need — dense models and tied-embedding models. MoE landed first. It's not the hardware; the software is catching up to SM120 reactively.
The leaderboard
32B is still queued. What I have so far, top-down by knowledge:
| Model | Quant | ruAIME | ruMMLU |
|---|---|---|---|
| Qwen3.6-27B | bf16 | 0.833 | 0.8156 |
| Gemma-4-12B | fp8 | 0.700 | 0.8103 |
| T-pro-2.0 (published) | — | 0.646 | 0.790 |
| Qwen3-14B | bf16 | 0.467 | 0.7765 |
| Gemma-4-E4B | bf16 | 0.467 | 0.7688 |
| Qwen3.6-35B-A3B | NVFP4 | 0.800 | 0.7355 |
| Qwen3-8B | bf16 | 0.667 | 0.7180 |
| Gemma-4-26B-A4B | QAT-GGUF | 0.567 | 0.7006 |
| Qwen3-4B | bf16 | 0.400 | 0.6566 |
The measurement beat intuition: a 12B half a step behind the 27B
The interesting part wasn't the top row. The dense Qwen3.6-27B took both benchmarks — ruMMLU 0.8156, ruAIME 0.833. Expected: it's bigger.
What matters more than size is this. Gemma-4-12B landed half a step behind on knowledge — 0.8103 vs 0.8156, a 0.005 gap — while being less than half the 27B's size. And it beat both the published T-pro-2.0 (0.790) and our own flagship 35B (0.7355). Intuition said "take the 27B." The measurement clarified it: on knowledge the 27B-to-12B gap is five thousandths of a point, while on hardware and cost it's a chasm. For knowledge work on a cheap card, the 12B earns its place.
One more twist, not about size: the dense 27B clearly beat the MoE 35B (0.8156 vs 0.7355). On knowledge, a dense model of this class wins over the "sparse" one — even though the 35B is larger in total parameters.
A caveat on Gemma-26B: its QAT-GGUF version (0.567/0.7006) lost to the smaller 12B-fp8. Not because it's a weak model. Two things overlap here: 4-bit QAT vs ~8-bit fp8, and NVFP4, which would separate precision from architecture, is blocked (see above). Until I isolate it, I won't judge.
The honest caveat
This is our own harness on T-Bank's named datasets, directional, not a byte-for-byte reproduction of their numbers. The quants in the table differ, noted per row, and what's compared is accuracy, not speed: tok/s is backend-dependent and doesn't collapse into one column. The closed cloud incumbent is named by category, its API requires that. The numbers aren't a universal verdict; they answer one question: what do I put on my own card.
What's next
→ Finish the ladder: 32B is running in the queue. → Unblock NVFP4 for Gemma-26B and the dense 27B: checkpoint re-export plus FlashInfer for SM120, or TensorRT-LLM. → The cost/quality call for 24 GB today: Gemma-4-12B in fp8. On knowledge it's half a step from the 27B, fits with room, none of NVFP4's fragility. The 35B-A3B-NVFP4 flagship goes on a 32–48 GB card.
An interactive version of the ladder that updates as runs land is at bench.csylabs.com/ru.
NVFP4 either comes up for the dense 27B and Gemma-26B — we'll see. Or we live on TensorRT-LLM — we'll see that too. Which is why the benchmark ships before "we fixed everything."