Russian open models on my own Blackwell — and where NVFP4 breaks
I rented an RTX PRO 6000 (96 GB, Blackwell) on Selectel for a week, ~314₽/hour, and ran a ladder of open models through two Russian benchmarks: ruAIME-2025 (30 olympiad problems) and ruMMLU from MERA (14,012 knowledge questions).
The question wasn't "which model is smartest" — closed cloud models lead the raw score anyway. It was the practical one: what open model actually fits a card you own inside Russia, and at what quality.
Spoiler: the measurement beat intuition again. And NVFP4, the format you buy this card for, came up for only one of three.
The rakes: where NVFP4 breaks on Blackwell
Starting here, honestly. NVFP4 is a 4-bit format that on Blackwell gives near-fp8 quality at half the memory. It's what lets a 26–35B model fit a client's 24 GB card. That was the whole point.
Of the three models I wanted in NVFP4, one came up.
Qwen3.6-35B-A3B (MoE) ran clean: 0.800 on AIME, ~22 GB of weights. It's the base we were already planning to serve.
Gemma-4-26B-A4B died at init. A bug in vLLM itself: after a recent refactor, tying the embedding weights to the model head was never implemented for quantized methods. Any tied-embedding model from an NVFP4 checkpoint crashes on startup, and Gemma is exactly that.
The next part is on me, not the model. I figured I'd just turn tying off with a flag — tie_word_embeddings: false. The crash stopped; the model started emitting garbage. Turning tying off left the head uninitialized: I didn't fix it, I broke it more quietly. Reading one answer with my own eyes corrected me in a minute. The right path is different: re-export the checkpoint with a materialized, untied head. Or TensorRT-LLM, which doesn't have this class of bug by engine design.
Qwen3.6-27B (dense) died differently: FlashInfer requires GPUs with sm75 or higher. Absurd on its face, since SM120 is newer than SM75. It's a version-comparison bug in FlashInfer on Blackwell, specifically on the dense-attention path: the MoE path (that same 35B) ran fine on the same card. Workaround: switch the attention backend. Root fix: FlashInfer built for SM120.
Honest takeaway from the rakes: NVFP4 on Blackwell via vLLM is, today, immature for exactly the two classes you most need — dense models and tied-embedding models. MoE landed first. It's not the hardware; the software is catching up to SM120 reactively.
The leaderboard (still partial)
27B is still finishing ruMMLU, 32B is queued. What I have so far, top-down by knowledge:
| Model | Quant | ruAIME | ruMMLU |
|---|---|---|---|
| Gemma-4-12B | fp8 | 0.700 | 0.8103 |
| Qwen3-14B | bf16 | 0.467 | 0.7765 |
| Gemma-4-E4B | bf16 | 0.467 | 0.7688 |
| Qwen3.6-35B-A3B | NVFP4 | 0.800 | 0.7355 |
| Qwen3-8B | bf16 | 0.667 | 0.7180 |
| Gemma-4-26B-A4B | QAT-GGUF | 0.567 | 0.7006 |
| Qwen3-4B | bf16 | 0.400 | 0.6566 |
| Qwen3.6-27B | bf16 | 0.833 | running |
| T-pro-2.0 (published) | — | 0.646 | 0.790 |
The measurement beat intuition: a 12B topped everyone on knowledge
The interesting part wasn't the overall score. Gemma-4-12B scored 0.8103 on ruMMLU, above both the published T-pro-2.0 (0.790) and our own flagship 35B (0.7355). The smallest model in the run beat everything several times its size on Russian knowledge.
Intuition said "get the bigger model." The measurement said the opposite: for broad Russian knowledge, Gemma's 12B is denser than the giants. And it's not just the 12B, the same shows on E4B. The Gemma family is disproportionately strong on Russian factual knowledge for its size.
A caveat on Gemma-26B: its QAT-GGUF version (0.567/0.7006) lost to the smaller 12B-fp8. Not because it's a weak model. Two things overlap here: 4-bit QAT vs ~8-bit fp8, and NVFP4, which would separate precision from architecture, is blocked (see above). Until I isolate it, I won't judge.
The honest caveat
This is our own harness on T-Bank's named datasets, directional, not a byte-for-byte reproduction of their numbers. The quants in the table differ, noted per row, and what's compared is accuracy, not speed: tok/s is backend-dependent and doesn't collapse into one column. The closed cloud incumbent is named by category, its API requires that. The numbers aren't a universal verdict; they answer one question: what do I put on my own card.
What's next
→ Finish the ladder: 27B is running, 32B queued. → Unblock NVFP4 for Gemma-26B and the dense 27B: checkpoint re-export plus FlashInfer for SM120, or TensorRT-LLM. → The cost/quality call for 24 GB today: Gemma-4-12B in fp8. Top on knowledge, fits with room, none of NVFP4's fragility. The 35B-A3B-NVFP4 flagship goes on a 32–48 GB card.
NVFP4 either comes up for these two — we'll see. Or we live on TensorRT-LLM — we'll see that too. Which is why the benchmark ships before "we fixed everything."