Threads in a block are divided into fixed groups of 32 threads called warps. Each thread lives in a warp lane.

Threads and warp lanes

A thread is one instance of your code running on one piece of data. A warp lane is just a thread's fixed slot inside its warp - lane 0 through lane 31. Each thread keeps the same lane for its whole life. The lane number is what lets all 32 threads in a warp run the same instruction at the same time but on different data: lane 5 grabs A[5], lane 6 grabs A[6], and so on, all in the same cycle.

• In each cycle the SM picks one ready warp and issues its next instruction across all 32 of its lanes at once (SIMT). Issuing is fast - 1 cycle - but the instruction can take far longer to actually finish.
  • A LOAD from HBM takes ~400 cycles to land, same cost as in the memory-hierarchy file.

Warp size

• Threads divided into four warps: each warp contains 32 threads.

---

Warp interleaving

The gap between issuing an instruction and it finishing is why the SM works with many warps at once: while one warp's LOAD is still traveling, the SM issues another warp's LOAD instead of waiting idle. Watch the data packets in the two scenes - in the sequential one, only one packet moves at a time and the rest sit parked at HBM; in the interleaved one, several packets are moving at once.

---

Sequential data transfer

The scheduler waits for one warp's data to fully land before issuing the next, only one packet ever moves at a time - the rest sit untouched.

• Visualize sequential data transfer (no interleaving).

---

• Reset the HBM packets: reset packets

Interleaved data transfer

In this case the scheduler doesn't wait for one warp's data to land before issuing the next.

• Visualize interleaved data transfer.

Interleaving exists to hide LOAD/STORE latency specifically. This file uses 4 resident warps for a readable demo; real occupancy on an SM can be dozens of warps deep, all issuing in the gaps left by each other's HBM waits.

Is more warps always better?

No. The SM can still only start one instruction per cycle. Once enough warps are loaded on the SM to keep it busy during the ~400-cycle wait for data, loading even more warps doesn't hide any more waiting time - there's no more waiting time left to hide.

Past that point, loading more warps only costs you. Each thread's storage on the chip - its registers and its shared memory - gets divided up among all the warps sharing the SM. So more warps means less storage per thread. If that forces a thread to spill data out to slower memory, or to fetch things more often, the kernel can end up slower even though more warps are running.

The opposite can also win: a kernel that keeps more data in each thread's own storage, using fewer warps, can beat a version that runs more warps but has to fetch data more often per thread. The goal is "enough warps to fill the waiting time that's actually there," not "as many warps as the hardware allows."