18 quants on one card: 3x faster doesn't mean smarter
Wanted an evening to answer one question: which quant to run. Ran 18 configurations — three models, five-six quants each, one set of 22 real tasks — and got the answer plus two I wasn't looking for.
The card is a rented Selectel RTX PRO 6000 (96 GB), partly funded through their AI grant program.
| Model | Quant | Weight | tok/s |
|---|---|---|---|
| Ornith-9B | IQ3_M → Q8_0 | 4.7-9.6 GB | 229 → 145 |
| Ornith-35B (MoE) | IQ3_XXS → Q8_0 | 16-37.8 GB | 259 → 224 |
| Qwen3.8-27B | UD-IQ2_XXS → UD-Q6_K_XL | 7.3-23.6 GB | 125 → 54 |
Cleanly monotonic within every family: smaller weight, faster answer. Expected. The interesting part is next.
Faster doesn't mean smarter
Ornith-35B is MoE — 35B parameters nominally, roughly 3B active per token. That's where the speed comes from: nearly 3x Qwen3.8-27B on decode. Ran a head-to-head on two substantive prompts: tied on code. On a five-apples split-in-half word problem, Ornith-35B mixed an Arabic word into the Russian sentence and concluded with "5.5 apples" — arithmetically correct, physically nonsensical for whole fruit. Neither Ornith-9B nor Qwen reproduced the defect.
Ornith-35B's vendor-claimed numbers (Terminal-Bench 67.8 vs. 52.5 for the prior version) didn't show up on this check. A model that beats its own predecessor on someone else's benchmark couldn't split five apples without a word in the wrong language. MoE really is faster — that stands. "Faster" and "smarter" are just different axes.
Two bits isn't quantization anymore, it's a failure mode
Qwen at UD-IQ2_XXS (7.3 GB) returned an empty response on 2 of 22 real tasks — not garbage, a literal empty string, on an Ansible firewall playbook and a Python data-sanitization function. Disqualified regardless of how tempting the file size looks.
Which card, which model
96 GB of rented card covers both production models at once (90.8 GB combined, with concurrency headroom). For smaller cards — what's measured versus what's arithmetic:
| VRAM | Run this | Basis |
|---|---|---|
| 16 GB | Ornith-9B, Q5_K_M-Q6_K | Measured |
| 24 GB | Ornith-9B with headroom; Ornith-35B at an aggressive quant, reduced context | 9B measured, 35B at 24GB untested |
| 32 GB | Ornith-35B, Q4_K_M-Q5_K_M (real ~31 GB at 262K context) | Measured |
| 48 GB | One model, not both — together they need ~91 GB | Measured (an out-of-memory crash) |
| 96 GB | Both models at production quality and full context | Measured — this is my card |
Caveat
One run per configuration. The 35B language defect is two prompts, not a benchmark — needs a larger blind-scored set before it's a verdict. Quality can't be fairly compared between Ollama and production vLLM for these models: vLLM's GGUF plugin doesn't yet support their architecture (support merged 6 days before this test).