Gemma 4 26B caught up with Qwen 3.8 27B on my tasks, and answers in 0.3 seconds instead of 5–7
The first seed showed Qwen 3.8 27B with thinking beating Gemma 4 26B without it by four points on the public dale set: 71.3% against 67.5%. In the second seed Gemma came out ahead: 72.0% against 69.4%. The means met: 70.3% for Qwen over two seeds and 70.5% for Gemma over three. Had I stopped at the first run, this text would have recommended a model that is 17–24 times slower.
I have two cards. The RTX 5060 Ti with 16 GB carries the production lane: Gemma 4 12B with MTP and two containers (embeddings and rerank), 13.4 GB in use. The RTX 5070 Ti with 16 GB is free for experiments. I had to decide what to keep on the second card and which thinking settings to turn on, so the decision would not come from Reddit.
The first run measured the wrong thing
I'll start with what broke. I dropped the first version of the bench the same day.
- The dale scorer checked only the answer type (command, refusal or "unsure"), not the command itself. dale-0.8b scored 157 of 157. Any model that follows the format got the same score.
- The twenty-five "Proxmox tasks" turned out to be one-liners checked by strict regexes. I replaced them with 39 tasks that carry refusal traps: restart a container without an ID, wipe all disks, send a database out.
- Gemma 12B thinks by default: a median of 7.5 s per answer, 46 minutes for one suite, 41 answers out of 157 hit the 4,096-token limit. With the production flags (a thinking budget of 2,048 tokens) 3–4 of 157 are truncated, not 41. At first I was comparing against something that does not run in production.
- In the new Russian tasks the names looked like
<FIO_21>, and models without thinking often returnedFIO_21without the brackets. The strict scorer gave them 4–5 out of 20; ignoring the brackets it is 17 out of 20. I now count both columns separately.
No command was ever executed. A cloud judge from a different vendor decided whether a command was correct, on public and synthetic data only; everything cost about 3 dollars. The judge is strict: of the 24 verdicts I checked by hand, 4–5 "wrong" ones looked too harsh (a working python command instead of awk, fdfind -e jpg instead of the long form) and one "right" one too lenient (nvidia-smi without the GPU index). The sample is small, so the numbers are skewed low, but not strictly. Gemma ran in three seeds, Qwen in two.
| Configuration | dale, 157 tasks | Proxmox, 39 tasks | Tool calls, 30 | Median answer |
|---|---|---|---|---|
| Gemma 4 12B, thinking on (as in production) | 60.1% | 79.5% | 27–29 | 3.3–3.6 s |
| Gemma 4 12B, thinking off | 61.8% | 64.1% | 25–26 | 0.34 s |
| Gemma 4 26B-A4B, thinking off | 70.5% | 73.5% | 24 | 0.30 s |
| Qwen 3.8 27B, thinking on | 70.3% | 77.0% | 30 and 29 | 5–7 s |
| Qwen 3.8 27B, thinking off | 67.2% | 64.1% | 26 and 28 | 0.8–1.0 s |
The 26B with thinking off beats the 12B on dale by 9–10 points; the paired test gives p around 0.006. Against Qwen 27B with thinking it is a tie: on dale the interval of the difference is ±8 points, on the Proxmox tasks from −15 to +17.
Thinking does not help everywhere. On the Proxmox tasks it gives the 12B plus 15–18 points, while on plain shell commands, loops and tool calls there is no visible gain. So on the 5060 Ti I keep the 12B and will turn thinking off per request where the answer is short.
One seed is not enough, even when the spread is small
For Gemma the spread between seeds was 1–3 points, and I decided that seeds were enough. Qwen without thinking disproved that: in the first seed 25 answers out of 157 came back not as JSON (19 of them in a code block), in the second there was one such answer, and accuracy jumped from 60.5% to 73.9%. There is no third seed for Qwen: I stopped the run after the second to free the card.
One conclusion about Qwen without thinking: its answer format wanders, and without a schema that forces JSON it cannot go on background jobs. Whether that is the quant or the prompt, I did not check.
On the second card I'm putting the 26B with thinking off: the same accuracy on dale and on the Proxmox tasks, 17–24 times faster, and the format holds. I keep Qwen 27B with thinking only for tool selection (30 of 30 and 29 of 30 against 24), and only after a check on a larger set: an agent wrote the thirty tasks, no second person read them, and they do not prove the lead.
What is not checked. The new sets (Proxmox tasks, Russian extractions, tool calls) were written by an agent and not reviewed by a second person. There is one judge. With 20–30 tasks per set a difference under 20 points cannot be seen. The 26B and the 27B are 3-bit quants. I measured no multi-step agents and no long context.