7 min read#ai#llm#infrastructure

Qwen3.8-Flash-Next, 125B parameters at home: 16 GB of VRAM, 40 GiB of experts in RAM, 32–40 tokens/s

In half a minute the server's swap grew by 259 MiB, and the guard stopped the model load. I never ran the Cyrillic vocabulary check. Before that, to make room for the trial, I had switched off the regular services: corpus search and camera recording were down while the experiment ran.

I had assumed a model like this would not fit into a home server with 16 GB cards. Per the model card, Qwen3.8-Flash-Next has 125B parameters, 6B of them active, plus 51B in n-gram embeddings and 4B in MTP. It is an MoE: only a small part of the experts works during generation, but the rest still has to be stored somewhere and read from somewhere. Now the model answers through Strata on one card, while Gemma, search and video live on the other. The setup is good enough for trials; I have not yet tested a long shared load.

Two cards, one pool of RAM

Ryzen 7 7700, 124 GiB of DDR5, Proxmox with LXC containers. An RTX 5070 Ti with 16 GB in the x16 slot is given to Flash-Next. The second card, an RTX 5060 Ti (81,999 ₽ by the shop's price tag), sits in a PCIe Gen4 x2 slot and carries Gemma 4 12B on llama.cpp, the embedding service, the reranker and Frigate. It is one machine with shared RAM, and two cards do not add up to one pool of VRAM: they split the duties.

Before, everything lived on one card, and any trial of a big model meant stopping the neighbours. Afterwards I restored the processes and checked that Frigate was writing fresh segments again and that search answered real queries.

A green /health does not mean a working GPU

After the second card went in, the container with Gemma and search came up, the HTTP checks were green, and cuInit returned 999. The services had quietly fallen back to CPU. Restarting the container brought CUDA back but did not show why. The cause was boot order: the NVIDIA device number changed between boots, and the hook that fixes it ran after Proxmox had already prepared the container config. I moved it ahead of guest startup. It passed a check with a deliberately stale device number; I have not yet repeated a full cold boot of the host since the fix.

Frigate was a different picture. CUDA saw the card, but hardware decoding (NVDEC) on the 5060 Ti failed with CUDA_ERROR_NO_DEVICE. I took an old video segment for a sign that recording was running, when recording had been down for about 44 minutes until the video went back to the main card. I found a workaround with a separate probe: expose both cards to the container through the NVIDIA toolkit and pick the right one for CUDA. Why NVDEC breaks with a single visible card, I don't know, and upgrading Frigate to 0.18.2 did not fix it. Now I look at the age of the last completed segment, not at the container's green badge.

The experts live in RAM

Strata keeps the hot parts of the model and the KV cache in VRAM and places the experts in RAM within a set budget; misses go to the file tier. My settings: ISTA IQ3_S quant, k8v4 KV cache, speculative decoding through MTP, a resident-expert budget of 40 GiB. The container limit is 60 GiB, not 40: the process, the cache and the neighbours need memory beyond the experts.

The first attempt, with a smaller budget, came out worse. Short answers ran at 13.7–17.8 tokens/s, then a swap-growth guard stopped a long probe. I capped the ZFS ARC at 12 GiB, lowered swappiness to 10 and added a separate watchdog. During loading, the host's available memory fell from 79 to 33 GiB; that is a host snapshot, not the usage of one process. Later the same guard killed an already running Strata: I was loading a vLLM checkpoint into memory, swap-in rose to 3,144 pages per second with 36 GiB free. The rule turned out to be too strict; now it fires only when available memory drops below 20 GiB.

Measurements

CheckResultWhat was measured
Flash-Next, English / Russian / tool call40.3 / 32.2 / 31.6 tokens/sshort separate probes on the 5070 Ti, JSON valid
Flash-Next, prefill2,076 tokens/scold synthetic input of 32,235 tokens
Flash-Next, resources131,072 tokens, 15.2 GiB VRAM, load about 100 sconfigured window, KV k8v4
Gemma 4 12B, llama.cpp + MTP84.4 / 83.9 tokens/sthree repeats of 500 words, English / Russian
Gemma 4 12B NVFP4, vLLM without MTP40.8 / 41.5 tokens/ssame run, 32,768 window

Prefill, time to first token and answer generation are different stages. 2,076 tokens/s is the speed of reading the prompt, not of writing the answer. The 131,072 window is a configured ceiling: how well it holds on live documents, I haven't tested.

On the local check, ISTA returned valid JSON in two requests out of three. On the long-context set it passed six checks out of seven at an input of about 38 thousand tokens and five out of seven at 117 thousand, one pass per length. Another Flash-Next variant, Swift, returned valid JSON in three requests out of three, but in one of them the check confirmed none of the 20 verbatim quotes. For document work that is worse than a broken bracket. I am not declaring a quality winner: the number of repeats differs between variants, and this run had no external judge.

Gemma on the neighbouring card: llama.cpp with MTP is twice as fast as vLLM

Gemma 4 12B (UD-Q4_K_XL and Q8 MTP) is the workhorse for the daily flow: triage and bounded tasks with tools. llama-server takes 9,514 MiB, and each of the two search services takes 1,324 MiB.

Separately I tried Gemma in NVFP4 on vLLM 0.33: it reads images, which my llama.cpp cannot do without a separate multimodal file. First test (08.11): 9.25 GiB of weights, FP8 KV cache, 65,536 window, 34.3 tokens/s in English and 41.0 in Russian, a valid tool-call JSON, an invoice read exactly, a needle found at 59 thousand tokens. At 131 thousand, memory ran out under load. Only later did I notice that the embedding container was missing at that moment (how it vanished I did not establish) and recreated it. So 65,536 fit without that service, not next to the whole stack as I had thought.

The repeat test on 09.11, with both services and Frigate on the card and 11.7 GiB free, gave a 32,768 window and 40.8 / 41.5 tokens/s, half the speed of llama.cpp with MTP in the same run. MTP in vLLM with Google's native assistant did not start with one, two or three draft tokens: the compile autotuner asked for 1.9 GiB with 1.6 free. Gemma on llama.cpp stays. vLLM would make sense for images, but then search would have to leave this card. I did not check audio.

What is not measured

→ Long shared load on both cards: temperature, stability, effect on search and on recording freshness.

→ A cold host boot after the device-order fix and Strata's autostart after it; the cause of the NVDEC failure with one visible card.

→ Reliability across the whole context window on real documents, Russian and tool calling on a large sample, the rate of errors with valid JSON.

→ The pure MTP gain on this card, token cost, electricity and savings against the cloud.

→ The Cyrillic vocabulary comparison, the Q4 quant and audio in the NVFP4 probe: I did not run them.

I was wrong twice: I decided a model like this would not fit at home, and I decided 64K for vLLM fit next to the whole stack. The first was corrected by measurements: the experts found room in shared RAM, and the card stayed with the everyday services. The second was my inattention to what was actually running. The headroom in this setup is shared across the whole host, so the next test is a simultaneous load on memory and both cards.

Related reading