Tiny VoltAnatomy of a floating point
Sign, exponent, mantissa: 32 bits split three ways. What each field controls, why the spacing between representable numbers grows, and where the familiar rounding surprises come from.
Read article →Machine Learning for Programmers
Featured
Binary representations of integers
How a row of bits becomes a number: place values, why the leftmost bit is worth the most, and what changes when the same eight bits have to hold negative numbers too.
Solving neural ODEs from first principles
A network that defines a derivative instead of a layer. Follow one trajectory through the vector field, take the steps by hand, and see where the solver and the adjoint method come from.
A simple 1D normalizing flow
Push a Gaussian through an invertible function and watch the density bend. One dimension is enough to see why the change-of-variables formula needs that derivative term.
Flash attention 1
Attention without ever writing the score matrix to memory. Stream one block of keys and values at a time, keep a running softmax, and see why the speedup comes from what you do not store.
Flash attention 2
Same algorithm, better occupancy. Swapping the loop order puts query blocks on the outside so they run in parallel, and moving the rescaling out of the inner loop cuts the non-matmul work.
Flash decoding
One query token against a long KV cache leaves most of the GPU idle. Split the cache into chunks, attend to each in parallel, and combine the partial results with their softmax statistics.
CUDA essentials: memory hierarchy and threads
Where a value actually lives: registers per thread, shared memory per block, global memory for everyone. The distances between them are what most kernel performance comes down to.
CUDA essentials: warps and interleaving
Threads are scheduled 32 at a time. Follow the issue slots as the scheduler switches between warps to cover memory latency, and see why more resident warps can beat faster ones.
CUDA essentials: warp divergence
A branch inside a warp costs both sides. Watch the lanes that take the if run while the others sit masked off, then swap, and work out when divergence is worth avoiding.
CUDA essentials: shared memory bank conflicts
Shared memory is split into banks, and two lanes hitting the same bank serialise. Trace a strided access pattern to its conflicts, then pad the array by one and watch them disappear.
Kernel visualization: online softmax
Softmax in a single pass. Carry a running maximum and a running sum, rescale the sum whenever the maximum moves, and end up with the same answer as the two-pass version.
Kernel visualization: efficient matrix transpose
The naive transpose reads coalesced and writes scattered. Stage the tile in shared memory to fix the writes, then pad it by one column to clear the bank conflicts you just introduced.