Appendix · Module 10

GPU Hardware

A GPU is a throughput machine: thousands of tiny cores grouped into streaming multiprocessors, fed by a steep memory hierarchy. These animations walk through how a kernel's grid of thread blocks gets scheduled, how warps execute in lockstep, where data lives, and why decode is so often bottlenecked by memory bandwidth rather than raw compute.

The GPU execution model: grid → blocks → SMs → warps

step-through

A kernel launches a grid of thread blocks; the hardware dispatches blocks onto many streaming multiprocessors, where threads run in 32-thread warps in SIMT lockstep.

kernel / grid thread block dispatch SM cores / threads at work divergent / idle thread

The contrast with a CPU is the whole point. A CPU has a few big, stateful cores tuned for latency and complex control flow; a GPU has thousands of simple, stateless cores organized for throughput. An SM — a Streaming Multiprocessor — is the unit that most resembles a CPU core: it bundles cores, registers, caches, and a scheduler, and runs a huge amount of simultaneous multithreading. (AMD calls these Compute Units, and the green CUDA Cores are Stream Processors.)

Throughput vs latency: many small cores, not a few big ones

auto-play

A CPU finishes a few items fast with big out-of-order cores; a GPU floods a huge grid of simple cores and wins on total work per second. Watch both chew through the same workload.

CPU core (big, latency) GPU core (small, throughput) work item in flight

The memory hierarchy: registers → SRAM → L2 → HBM

step-through

As you climb down the hierarchy capacity grows but bandwidth and latency get worse. Keeping data high in the hierarchy is the single biggest lever for kernel performance.

memory level (SRAM / HBM) thread / SM accessing it KV cache + weights live in HBM data movement

Ties back to Module 5 FlashAttention is a memory-hierarchy trick. Naïve attention writes the full N×N score matrix out to HBM and reads it back — a flood of slow global traffic. FlashAttention instead streams tiles of Q, K and V into on-chip SRAM (shared memory / L1), computes the softmax incrementally there, and never materializes the big matrix in HBM. Same math, far less data movement — which is exactly why it is faster.

Kernels & Triton: one program instance, one tile

step-through

A kernel is a function run across the whole grid. CUDA (NVIDIA) and ROCm (AMD) are the native stacks; Triton lets you write tiled kernels in Python where each program instance owns one tile — load to SRAM, compute, write back.

kernel / program code HBM tensor & on-chip SRAM tile load / store compute on the tile

One kernel, many program instances (Triton) or thread blocks (CUDA/ROCm) — each handling a different tile of the same tensor, all running at once across the SMs. The art of a fast kernel is choosing a tile that fits in SRAM and registers so the cores stay busy instead of waiting on HBM.

Memory-bound vs compute-bound: why decode waits

step-through

Arithmetic intensity = math done per byte moved. Low intensity work is starved by bandwidth; high intensity work finally saturates the cores. This is the roofline intuition.

memory-bandwidth roof peak-compute roof decode (low intensity) prefill / big matmul (high intensity)