Feeding the Cores

Two numbers from the RTX 4090's spec sheet. Its lanes can do 82.6 trillion floating-point operations a second, which is 41.3 trillion fused multiply-adds. Its memory delivers 1,008 gigabytes a second, which at four bytes per number is about 252 billion numbers a second. Every multiply-add wants two fresh numbers in. So if each number arriving from memory were used exactly once, the lanes would be busy for one tick in every 330, and the most expensive part of the card would sit idle over 99 per cent of the time.

This is the memory wall from Part 7, now measured in bandwidth rather than latency, and it is the central design problem of a GPU. The whole memory system, from the register file to the stacks of DRAM beside the die, exists to make sure that a number fetched from memory gets used many times before it is thrown away. This part walks that system from the top down, then gives you the one diagram that tells you, for any job, whether you are short of arithmetic or short of bytes.

Reuse is everything

The quantity to hold in your head is arithmetic intensity: how many floating-point operations a job does for each byte it moves from memory. The 4090's ratio of 82.6 trillion operations to 1,008 gigabytes works out to about 82 operations per byte. A job with intensity above that has enough arithmetic to keep the lanes busy while the bytes trickle in. A job below it will have the lanes waiting on memory, and no amount of extra compute will help.

Consider two jobs. Adding two arrays together, y[i] = a[i] + b[i], does one operation for every twelve bytes moved (two numbers in, one out): an intensity of 0.08. On a 4090 it runs at about 80 GFLOPS, one thousandth of the card's peak, and there is nothing to be done about it because every number is used once and discarded. Now multiply two 4,096 by 4,096 matrices. That takes about 137 billion operations, and the three matrices together are 200 MB, so the intensity is around 680. Every element of the input matrices participates in 4,096 separate multiply-adds. The job is drowning in reuse, and it can run at close to the card's full speed, provided the hardware manages to hold each number close by while it is used those 4,096 times. Holding numbers close by, in the registers, the shared memory and the caches next to the lanes rather than out in DRAM, is what the hierarchy is for.

The ladder, GPU edition

Here is the whole ladder before we climb down it: four levels, each larger, slower and more widely shared than the one above, with the lanes sitting on the top rung.

At the top are the registers: each thread's private handful, in the SM's 256 KB register file, reachable every tick. A number in a register is free to reuse. But registers are per thread, and a matrix multiply needs threads to share.

Next is shared memory, and it is the thing that most distinguishes a GPU's memory system from a CPU's. It is a block of SRAM inside the SM, up to 100 KB of it on Ada and up to 228 KB on Hopper, which the program manages directly rather than the hardware. On a CPU the L1 cache decides for itself what to keep. On a GPU, the programmer says "load this 64 by 64 tile of the matrix into shared memory", and then every thread in the block reads from it as often as it likes at about 20 ticks a time. This is how the reuse in a matrix multiply is actually captured: a tile is fetched from DRAM once, parked in shared memory, and used thousands of times by the SM's threads before the next tile replaces it. The same physical block also serves as an ordinary L1 cache for whatever the program did not explicitly manage.

Below that is the L2 cache, one for the whole chip, shared by every SM. It is the band down the middle of the die photograph. The full Ada die has 96 MB, sixteen times the previous generation's, of which the RTX 4090 gets 72, and Hopper's is 50 MB. It costs around 200 ticks to reach, and it is the last stop before leaving the silicon.

And then there is DRAM, the main memory of the card, which comes in two very different flavours, and the next section is about them. First, though, a question the ladder raises and rarely answers: how does a number get onto a rung, and what pushes it off?

How numbers move between the rungs

The answer is different at the top of the ladder and at the bottom, and the difference is the point. The two rungs nearest the lanes are managed by the program and the compiler. The two below are managed by the hardware.

A number reaches a register in one of two ways: a load instruction fetches it from memory and writes it there, or it is the result of an arithmetic instruction. How many registers each thread gets is fixed when the program is compiled, and a thread keeps its registers for its whole life. Nothing is ever evicted from a register by the hardware. A value stays until the program overwrites it, and when the thread finishes its registers are simply released, contents and all. So a thread does not "keep" its results in registers when it is done. Anything it wants to survive it must store to memory with an explicit store instruction before it ends, and that store passes down through the L2 and on to DRAM. If a program needs more registers than a thread has, the compiler makes room itself by inserting stores and loads that park the surplus values in memory and fetch them back, which is called spilling, and it shows up as a slower program rather than as anything the hardware does at run time.

Shared memory works the same way one level down. Numbers get there because the program copies them there on purpose, usually a tile at a time, either by loading them into registers and storing them again or, on Hopper, by asking a copy engine to move a whole tile from DRAM straight into shared memory. Nothing is pushed out. When the threads have finished with a tile they wait at a barrier, so that nobody is still reading it, and then overwrite it with the next one. When the block of threads finishes, its share of the shared memory is released.

The L1 and L2 caches are the opposite. They fill themselves: any load that goes to DRAM brings the 128-byte line it belongs to into the L2 and usually into the L1 as well, without the program asking. And they empty themselves: when a new line needs the space, the hardware evicts one, roughly the one used longest ago. The L1 never holds anything that needs saving, because stores write straight through it to the L2, so evicting a line from L1 just drops it. The L2 does hold modified lines, and when it evicts one it writes it back to DRAM first. That write-back, and the store instructions from the SMs, are how results finally reach the memory on the board, and from there they leave only when the program copies them back to the CPU over PCI Express.

GDDR and HBM

A gaming card uses GDDR (graphics double data rate memory): a dozen or so ordinary-looking DRAM chips soldered onto the circuit board around the GPU, each talking to it over its own set of wires. The RTX 4090 has a 384-bit bus, meaning 384 wires' worth of data lanes, and the GDDR6X chips push 21 billion bits a second down each wire by encoding two bits per signal (a scheme called PAM4). Multiply out and you get the 1 TB/s. The RTX 5090 moves to GDDR7 on a 512-bit bus for 1.79 TB/s. GDDR is cheap, it is made in enormous volume, and its limit is the wires: you can only route so many across a circuit board, and only drive them so fast before the signals blur.

A data centre card uses HBM (high bandwidth memory), which attacks the wire problem by not using a circuit board at all. DRAM dies are stacked eight or twelve high, connected vertically by thousands of tiny holes etched through the silicon, and the stack sits on the same package as the GPU, a few millimetres from the die, connected through a slab of silicon called an interposer that can carry far more wires than any circuit board. Each stack has a 1,024-bit interface. The wires are individually slower than GDDR's, but there are many more of them. An H100 has five stacks of HBM3 and reaches 3.35 TB/s. A B200 has eight stacks of HBM3e for 8 TB/s. Vera Rubin's HBM4 targets 22 TB/s. HBM is also how a data centre GPU gets its capacity, 80, 141, 192, 288 GB, because each stack holds many dies' worth. The price is that stacking and interposers are expensive and difficult, which is why your gaming card does not have it and why, as we reach the last few parts, HBM supply turns out to be the thing that paces the whole industry.

The roofline

In 2009 three researchers at Berkeley published a diagram that has become the standard way to think about all of this, called the roofline model. Put arithmetic intensity on the horizontal axis and achieved performance on the vertical, both on log scales. A machine's peak compute is a flat ceiling. Its memory bandwidth is a sloping line rising from the bottom left, because at intensity I and bandwidth B the most you can possibly do is I × B operations per second. The two lines meet at the ridge point, and the resulting shape looks like a roof. Any job sits under the roof at its own intensity. To the left of the ridge it is memory-bound: the sloping line is the limit and the lanes are idle. To the right it is compute-bound: the flat line is the limit and the memory is keeping up.

Try the language-model job on each machine. That is the case I most want you to notice, and it deserves its own section.

Why a chatbot typing is memory-bound

When a language model generates text, it produces one token at a time, and to produce each token it runs the whole network once. How LLMs Talk described this as pushing the text through billions of frozen numbers. In the terms of this part: every weight in the model is read from memory once per token, and each weight is used for one multiply-add. Two operations per weight, and at 16 bits per weight that is two bytes. Intensity: one operation per byte. Look at where that lands on the roofline. It is far to the left of the ridge on every GPU ever made. An 8-billion-parameter model at 16 bits is 16 GB of weights, so a 4090 at 1 TB/s can read them at most about 60 times a second, and 60 tokens a second is its ceiling no matter how fast its lanes are. The lanes are almost entirely idle. The bytes are the bottleneck.

Two things follow directly, and both are ideas you have met before. First, quantisation, the Q4 and Q8 suffixes from How LLMs Talk, is not mainly about fitting the model in memory. It is about bytes per token. A 4-bit model moves a quarter of the bytes of a 16-bit one, so it can generate up to four times faster on the same card. Second, batching: if the server handles 64 users' requests at once, it reads each weight once and uses it 64 times, one per user. The intensity goes up 64-fold, the job moves right along the roofline, and the same card produces 64 times the tokens for roughly the same memory traffic. That is why serving a model to many people is so much cheaper per token than running it for yourself, and why every inference system on earth is built around batching.

There is a third consequence, which becomes a whole chip in Part 18. Before a model generates anything it first reads your prompt, all of it at once, and that phase, called prefill, uses each weight for every token in the prompt. A 4,000-token prompt has intensity in the thousands. So a single request has two phases with opposite shapes: a compute-bound prefill followed by a memory-bound generation. No one machine is ideal for both, and Nvidia's most recent designs split them apart.

One more thing the hardware wants

There is a rule about how threads read memory that catches everyone once. When the 32 threads of a warp each ask for a number, the hardware looks at the 32 addresses. If they are adjacent, 32 consecutive floats, it fetches them as one 128-byte transaction. If they are scattered, it makes up to 32 separate transactions, and the effective bandwidth drops by that factor. This is called coalescing, and it means the layout of data in memory matters as much as the arithmetic done on it. Well-written GPU code arranges its data so that neighbouring threads read neighbouring addresses, and a surprising amount of the engineering in libraries like cuBLAS is about exactly that.

Where this leaves us

Compute is cheap. Bytes are expensive. That sentence explains more about GPU design than any other, and every generation since Volta has been built around it: bigger L2 caches to catch more reuse, bigger shared memory so tiles can be larger, HBM stacked beside the die so more bytes arrive per second, and lower-precision number formats so each number is fewer bytes. Where a job sits on the roofline tells you which of those will help it, and for the most common job in AI today, generating text, the answer is almost always bytes.

There is one more lever, and it is the biggest of them all. The lanes of Part 9 do one multiply-add per tick each, and each one needs its operands delivered from the register file. For matrix arithmetic, where the same numbers are reused across many multiply-adds, that register traffic is itself a bottleneck. The unit that fixes it is the green box in the SM diagram, and it is the reason a GPU can be called an AI chip at all.

Next: Tensor Cores: a unit that does a whole small matrix multiply in one instruction, why fewer bits per number multiplies its speed, and the sequence of formats from FP32 to FP4 that defines each Nvidia generation.