Attention needs a softmax over each row of an N by N score matrix, which looks like it forces you to build the whole matrix. FlashAttention does not. It walks over small tiles, keeps a running maximum and running sum per row, and fixes earlier work with a rescale factor whenever the maximum grows. The tab below runs the real arithmetic on random data and checks the answer against the naive method. Everything here is computed live in your browser, nothing is measured on a GPU.
Score matrix S = QKᵀ
rows are queries, columns are keys
What this tile does
One row, block by block
A single query row with 12 scores. Blocks arrive one at a time. Bars show each weight relative to the current maximum. Watch earlier bars shrink the moment a bigger score shows up.
Three ways to get the same answer
How much data moves through slow memory? model, not a benchmark
Counts reads and writes to GPU main memory (HBM) per attention head, using the I/O analysis from the FlashAttention paper. The arithmetic is identical for both methods, only the traffic differs.