Machine Learning for Programmers

Different representations of binary integers are seen through a unified framework. Using mathematical objects like ring, affine map and dot product uncovers insights that are not obvious otherwise.

A detailed tour of what is, in my opinion, one of the least understood topics in ML. It covers the motivating problem, and solves three problems from scratch without any of the deep learning machinery.

Push a Gaussian through an invertible function and watch the density bend. One dimension is enough to see why the change-of-variables formula needs that derivative term.

A visual explainer of how FA1 works, focusing on data movement across HBM. It builds on the ideas behind online softmax.

FA2 is often described as a "transposed" version of FA1. This articles makes it clear why. A visual guide to how data moves in FA2.

The often ignored cousin of Flash attention. It implements the same recurrence formula as FA but uses a different parallelization strategy that is more suitable for decoding stage.

Introduction to the foundations of GPU programming in CUDA.

A really short (and silly) animation of warp interleaving.

An illustrated article of what warp divergence is, and how to avoid it.

Shared memory is split into banks, and what it means to have a bank conflict.

Naive (safe) softmax takes three passes. Can we do better? This algorithm set the stage for Flash attention algorithms.

A visualization to understand a simple kernel for efficient matrix transpose (with no bank conflicts!).