2-bit DeepSeek, Ornith-35B, and a bug in my own measurement
Last time I compared Qwen3.8-27B on my own card against DeepSeek V4 Flash over the API — 0.789 vs 0.731. Obvious next question: what if I put DeepSeek on the same card, quantized down to 2 bits?
| Model | Score | Pass | tok/s |
|---|---|---|---|
| Qwen3.8-27B, NVFP4, own card | 0.789 | 46/50 | 56-59 |
| DeepSeek V4 Flash, API, full precision | 0.731 | 42/50 | 53.3 |
| Ornith-35B (MoE), own card, first time measured | 0.729 | 42/50 | 236.5 |
| DeepSeek V4 Flash, IQ2_XXS, own card | 0.697 | 35/50 | 66.8 |
| Ornith-9B, own card, first time measured | 0.628 | 31/50 | 123.6 |
Two bits cost real score — minus 0.034 against the same model at full precision over the API. Judges cite specifics: invented performance numbers for combinations that don't exist, a quantization recommendation that directly contradicts the task's stated constraint. Exactly what you'd expect from 2-bit compression — precision suffers, coherence doesn't. DeepSeek is honestly a good model at full precision — 304B parameters, 13B active, MIT — I just can't justify two 96GB cards for 32K of context.
Claimed a win — then caught my own bug in the measurement
The first pass used a single judge and scored 0.681 — higher than any local Qwen configuration I had at the time. I wrote it up as: "the local model beats everything we can self-host."
I didn't catch the mistake. "Are we sure about these numbers?" — I compared judge rosters and found every prior comparable run used a 3-judge ensemble (gemini + gpt-5.1 + opus-4.8); mine used gemini alone. Different judges, different scale — not comparable. Re-ran on the matched ensemble: 0.697. And the best local Qwen number wasn't the 0.628 I'd cited from an old table — it was 0.789 from a later run I'd simply failed to pull out of the archive. The headline flipped 180 degrees in one evening.
The second round of measurement brought its own surprise: part of the task set came back with PermissionDeniedError from all three judges at once. Not a rate limit — the OpenRouter account balance, $3.91 left out of $460. The per-key limit field showed 70-odd dollars of headroom the whole time — that field isn't what decides it.
The model I hadn't measured in six months
Ornith-35B (MoE, roughly 3B active parameters per token) had never run through this bench before — I just hadn't gotten to it. Result: 0.729, 42 of 50 tasks passed — the exact same pass count as DeepSeek over the API at full precision, at 4.4x the decode speed. Not close enough to replace Qwen3.8 in production, but the first number worth keeping in mind next time I pick a model.
Caveat
One run per configuration — same spread as last time. bash-002 failed with the identical exit code on both Ornith sizes — probably a defect in the task itself (grep with no matches exits 1, that's not an error), not the model. Didn't verify.