Third-party NVFP4 vs official FP8 on Qwen3.8-27B: 1.35× faster

Ran a full cross-engine pass on Qwen3.8-27B on a rented RTX PRO 6000 (96GB, Blackwell): vLLM, an NVFP4 checkpoint from unsloth (a community quant, not an official release) against the official Qwen/Qwen3.8-27B-FP8, MTP on and off, plus SGLang with DSpark on top. The throughput number is clean and backend-controlled: my NVFP4 checkpoint beats the official FP8 release by 1.35× (643 vs 476 tok/s — aggregate throughput at concurrency 16, 1024-in/256-out tokens, not single-request speed), MTP adds another 16–19% on either quant. Then the trouble started — not with the card, with my own reading of the results.

Measurement-setup failures I hit before accepting results

Before accepting any run's results, the harness (lii-code-bench-ru, 50 real architect tasks) tripped four times in a row:

→ Downloading a second checkpoint (RadixArk) silently overwrote the one already on disk (unsloth) — both resolved to the same directory on a matching basename, and the config/tokenizer got overwritten with someone else's. Caught it within two minutes, while arm 1 was already running against a mismatched checkpoint. → None of my launch commands across two days had the tool-calling flags. A tool-call task failed with a 400. → My default config had reasoning off — even though the repo's own changelog already had a precedent: a run served with reasoning off had been ruled invalid. → And the real one: on these 50 tasks, a 4096-token answer budget was regularly not enough. 13–15 of 50 tasks per arm came back empty — the whole budget spent on thinking, none left for an answer.

I fixed the first three before anything counted. Not the fourth: I deliberately kept the budget as-is, for comparability with the existing leaderboard, so the truncations it causes are in every scored arm. None of these issues are in the model. All of them are in my own measurement setup.

Quality: a band, not a verdict

Among the tasks the model actually answered — excluding the 13–15 of 50 where the 4096-token budget ran out on thinking before an answer started — all four configs land in a 0.743–0.792 band (a plain average, not task-weighted). No large collapse like the one I saw with AWQ, at least on this subset. That's not quantization parity, though: the band is built on answered tasks only and unweighted, at one run per config — and the raw leaderboard scores (0.605 / 0.580 / 0.549 / 0.628, weighted per the harness's own methodology) cross in a way that doesn't add up: the NVFP4 checkpoint beats FP8 with MTP off, and loses with MTP on. Part of the gap between the two sets of numbers is the weighting switch itself, not just the excluded truncations. At n=1 that's either noise or a real quant/speculative-decoding interaction — can't tell which yet.

Single judge (Gemini), not an ensemble — my own historical board has no other comparable Gemini-judged run on this model. Honestly: this isn't settled. It needs the same n≥3 repeat I already ran for the throughput matrix.

SGLang and DSpark — no fireworks either

The community claimed 200+ tok/s single-stream on this exact checkpoint+card pair via SGLang with DSpark. Didn't reproduce — observed 68→126 tok/s (×1.85), one measurement per config, not repeated. At batch-16 SGLang doesn't trail vLLM dramatically (603 vs 643), but that's two different checkpoints (RadixArk/NVIDIA ModelOpt for SGLang, unsloth for vLLM) on two different stacks — not an engine comparison at matched quant, and SGLang wasn't pinned to the same backend-control standard vLLM's number was.

Got corrected twice — fairly

An early draft of this called ninfer (a third-party engine targeting the RTX 5090) "confirmed incompatible." Not true — cmake genuinely failed on a missing CUDA 13.1 (that part's real, I have 13.0.3), but the README's claim of being specific to that exact card was never build-tested — the CUDA blocker stopped things before I got there. That went into the repo as "confirmed incompatible," got caught in review, fixed in a follow-up commit.

The second one was worse. I compared Qwen3.8 against a model labeled "qwen36" on an old board, and assumed by analogy with the label that it was a same-class Qwen3.6-27B. Never checked the manifest. Turned out to be Qwen3.6-35B-A3B — different architecture, different size class, and reasoning had been off on that run too. The whole comparison was invalid. Caught before it shipped anywhere — it lived in chat, never made it to a committed doc.

Next

→ Repeat the quality run at n≥3 to settle the NVFP4/FP8×MTP crossing → A parallel run is going through OpenRouter right now — same harness, same task set, three judges instead of my one, actually comparable to the historical board. A separate run will update the board once it finishes. → ninfer stays parked — a second CUDA stack to chase an unverified README claim isn't worth it without a real reason to force it.