More memory doesn't mean faster: prefill, decode, and the Apple M5 Ultra launch

I'd been using "faster" as one word — for a GPU, for context, for everything at once. Apple shipped M6 (its first 2nm chip) and M5 Ultra today — a new Mac Studio with 512GB of unified memory and 1.2TB/s of bandwidth, up to 80 GPU cores each with a Neural Accelerator. "Faster" shows up on nearly every page of the press release. Annoying to catch myself unable to actually explain the difference between "computes faster" and "responds faster" — had to work through it properly, with the math, not "whoever has more gigabytes."

Roofline: one question decides which wall you hit

The answer lives in a performance model HPC engineers have used for decades — the roofline model. Any hardware has a compute ceiling (FLOPS) and a memory-bandwidth ceiling (bytes/sec). Which one you hit isn't decided by the hardware alone — it's the task's operational intensity: how much computation happens per byte read from memory. High intensity, you hit the compute ceiling. Low intensity, you hit the memory ceiling. Prefill and decode are literally two different values of that intensity inside the same inference call.

Prefill: one weight load, thousands of tokens

Processing a prompt is parallel — every token passes through the model at once, in one batch. Rough estimate: a forward pass takes about 2 × N × T FLOPs, where N is parameter count and T is token count (ignoring attention's quadratic term for very long context — honest caveat below). Weights load into memory once and get reused across all T tokens immediately. The longer the prompt, the higher the operational intensity — the harder you hit compute specifically. This is where Tensor Cores decide: specialized fused-multiply-accumulate units that multiply an entire matrix tile per clock, not one number.

Decode: the same weight load, one token

Generating the next token is the opposite: the model processes exactly one new token at a time, but still has to re-read all the weights plus the KV-cache from memory to compute that one token. Operational intensity drops near zero. This is where Tensor Core count stops mattering and memory bandwidth decides — how many bytes per second the card can push from memory to the compute units.

A Tensor Core isn't the same thing as a CUDA core

A CUDA core is a general-purpose arithmetic unit — does anything, slowly, in parallel. A Tensor Core is a specialized unit for one operation: fused matrix-multiply-accumulate on a matrix tile per clock. Nvidia introduced them with Volta in 2017; on Blackwell it's 5th-generation, with native FP4. Apple's equivalent — a Neural Accelerator inside each GPU core — only landed this generation, for the first time on an Ultra chip. First-gen vs fifth-gen isn't just silicon: the software tuned against matmul operations (cuBLAS, TensorRT-LLM, vLLM's continuous batching) has had eight years; Apple's MLX has had about two.

	M5 Max	M5 Ultra	RTX PRO 6000 Blackwell
Memory	up to 128GB	up to 512GB	96GB GDDR7 ECC
Bandwidth	614GB/s	1,200GB/s	1,792GB/s
Matrix units	Neural Accelerator × 40 cores, 1st-gen	Neural Accelerator × 80 cores, 1st-gen (first on Ultra)	752 Tensor Cores, 5th-gen, native FP4
Price	from $2,499	from $5,499 (256GB realistically $10-18k)	rented, not bought

Numbers from NVIDIA's own datasheet and Apple's press release, not marketing slides claiming "Nx faster." I do actually rent the RTX PRO 6000 from Selectel — for this piece it's not the hero, just a real comparison point instead of a hypothetical one.

What the engineering community says, not just vendors

In the Hacker News thread under the announcement (91 comments), engineers who actually run both hardware types put the same thing different ways: "RTX 6000 will run circles around the Mac Studio in just about every way. Memory bandwidth is literally the only spec where Apple is competitive" — and separately: "M5 is excellent for inference; but if you like to train, data format support and effective performance is limited... pales in comparison to what RTX 6000 Pro can do for compute/matmuls/training." There's also a credible rumor (9to5mac, cited across several threads): Apple is skipping M6 Pro/Max/Ultra entirely to accelerate M7 — a chip reworked specifically for AI — next year. One commenter put it plainly: "I'd skip M5 and M6 chips for LLM work and wait a year for M7."

Who actually wants what

Single user, one large model, decode-heavy work (generating long text, not processing long prompts), privacy and power efficiency mattering more than raw speed — Apple's 512GB unified memory is a real argument, not marketing: matching that memory footprint on discrete cards costs an order of magnitude more. Training, fine-tuning, prefill on long prompts, or handling several requests concurrently — Tensor Cores and eight years of CUDA ecosystem decide, not gigabytes.

Caveat to myself: the 2 × N × T formula for prefill is a rough approximation — it ignores attention's quadratic growth on very long context (intensity runs higher there, a separate conversation). And this isn't a universal verdict, it's a breakdown of which hardware fits which task, without the vendor's thumb on the scale.