Qwen3.8-Flash-Next vs production 27B across three runs: 0.789 was the wrong config
The 0.789 result from my last bench in the last post - wrong config. Qwen3.8-Flash-Next dropped, and I'm back testing less than a day later. Testing the new architecture on the old RTX PRO 6000, and sizing it up against a 5090 with 32GB VRAM.
| Model | Runs | Score | σ | Pass | tok/s |
|---|---|---|---|---|---|
| Flash-Next, Q3_K_XL | 3 | 0.786 | 0.017 | 43/50 | 79.1 |
| Flash-Next, Q4_K_XL | 3 | 0.785 | 0.021 | 44/50 | 77.4 |
| Qwen3.8-27B, production (corrected) | 3 | 0.769 | 0.007 | 41/50 | 70.0 |
| Ornith-35B (MoE) | 2 | 0.732 | 0.004 | 42/50 | 238.1 |
| Ornith-9B | 3 | 0.590 | 0.037 | 30/50 | 121.4 |
The production row was stale for two weeks
Everyone's probably already read that Qwen3.8-Flash-Next is a preview of the architecture that becomes Qwen4: 125B parameters, 6B active per token, plus a 51B n-gram embedding that isn't computed dense at all - the n-gram data gets read as point lookups off disk. Obvious question: how does it compare to what's already running in production?
I'm a simple person - new model drops, I go verify the config for a fair comparison. Found that the "Qwen3.8-27B, own card" row in last post's table (0.789/46) wasn't actually the live server. Leftover from a deployment three weeks stale: third-party unsloth checkpoint, MTP off, through a tunnel to a port that hasn't existed in a while. Scary? You'd think 🙂
The real production config is the official Qwen3.8-27B-NVFP4A16, MTP on, sharing the card with Ornith-35B, which runs background compute for me - constantly adapting and updating my core repos. Nobody had checked the two side by side, because both looked plausible on their own. Re-ran it clean, bothered to do three passes: 0.769.
Three runs instead of one - and it all landed
My own methodology has known its instrumental delta for about a year now - 0.043 at identical config (reminder: perfect is 1.0). Three of the five models landed within 0.02 of each other. Flash-Next and production 27B are statistically indistinguishable: 0.786 vs 0.769, less than half the spread apart. The finer quant (Q4 vs Q3) bought nothing either - 0.785 vs 0.786, despite Q4 having a measurably better accuracy-recovery number by unsloth's own data. On this task set the extra bit apparently just ate VRAM (4.3GB instead of 24.7GB) without moving the score.
Ornith-35B sits a consistent 0.05 below the leaders - but it's an honest tradeoff: 3.4x the decode speed. Ornith-9B is predictably last, and its spread is the real one (0.037), unlike the rest - but for the money (I honestly run it on my Mac fairly often)!
The judge models got tired five times overnight
I mostly run benches in overnight sessions, hence the title. Last post already caught one run pulling PermissionDeniedError from OpenRouter from all three judges at once. Thought - lucky streak, one-off, bad timing. Last night alone I caught it five times, anywhere from 3 to 18 tasks at once per run (the moment errors start piling up, I get notified). The answers were substantive and complete (finish_reason: stop) - the judges just went quiet for a minute or two under load. Same fix every time: "wait for recovery, rerun exactly those tasks with a pause between calls, never mix with the stale score." The money difference is small - 111 judge calls for a full run cost about 70 cents - but the time cost is real. Accounted for, corrected.
Bottom line
Flash-Next only really works on llama.cpp today (looks like they shipped a production release earlier today) - neither vLLM nor SGLang has builds for the new architecture yet (Gated DeltaNet, sparse attention, the n-gram table). Production's MTP speculation and the whole mature vLLM stack are a real switching cost worth chasing eventually. For me the main takeaway is probably that three runs is still a small sample for the categories with real spread - I want to run more, but that costs more money too. A dilemma.