The tile in HBM and the threads in a block are {}(readout-text = \text "thread ( {sel-y}) owns column {owned-rows}") shown here.

Subtracting from each value makes the calculation stable (no overflows) without altering the output.

Shared memory is physically split into 32 parallel banks, each serving one 4-byte word per cycle. A warp's 32 lanes can be served in one cycle in two…

A warp issues one instruction per cycle. When threads in a warp take different if/else paths, the warp cannot split. So all threads run the if block…

A sector is a fundamental unit of data transfer. Even if you want to read or write just a single word, the entire sector the word's memory belongs to is…

Threads in a block are divided into fixed groups of 32 threads called warps. Each thread lives in a warp lane.

To build a good enough mental model of CUDA, it is useful to think in terms of how memory and compute are organized.

Generating text with a transformer happens in two stages. First comes prefill, where the whole prompt goes through at once, so attention has hundreds or…

For a single query row , stream the key/value tiles of width . Carry a running max , denominator , and output :

A floating-point number has three components: - a sign bit - a biased exponent - a mantissa

Flash Attention builds on the ideas behind online softmax. While online softmax requires two fetches from HBM, Flash Attention only requires one. Just…

A normalizing flow turns an unknown data distribution into a known one by learning an invertible map .Thus, log likelihood of an unknown distribution can…

Three different components of a simple ODE are optimized. I work through it the way one would with pen and paper, prioritizing correctness over…

Four representation types are illustrated here: uint, two's complement, one's complement and biased representation.