From 2636da1685384c4cfe2d4d5f62072b2018fd76f8 Mon Sep 17 00:00:00 2001 From: RaviTejGuntuku <64335725+RaviTejGuntuku@users.noreply.github.com> Date: Sun, 17 May 2026 22:59:04 -0500 Subject: [PATCH] Fix algorithm name in dataflow walkthrough the paper that proposed the online softmax algorithm to avoid manifesting the entire O(N^2) QK^T matrix was the ORIGINAL FlashAttention paper, not FlashAttention3. --- dataflow_walkthrough.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataflow_walkthrough.md b/dataflow_walkthrough.md index 6e6e906..766636e 100644 --- a/dataflow_walkthrough.md +++ b/dataflow_walkthrough.md @@ -156,7 +156,7 @@ LSU fires `ISSUE_VEC_U softmax_online, scores`. This is the cleverest microcode Naive softmax computes `exp(scores - max(scores)) / sum(exp(...))` — but this requires materializing the *entire* attention score matrix in SRAM before you can compute the max and the sum. With 3000-token context × 8 heads × 16-bit, that's 50 KB just for the matrix — already pushing the activation buffer. -The **FlashAttention-3 online algorithm** instead processes scores in tiles of (say) 32 at a time. Per attention row, VecU keeps just two running scalars — `m_i` (running max) and `l_i` (running sum-of-exps) — plus the running output accumulator `O_i`. When a new tile arrives: +The **FlashAttention online algorithm** instead processes scores in tiles of (say) 32 at a time. Per attention row, VecU keeps just two running scalars — `m_i` (running max) and `l_i` (running sum-of-exps) — plus the running output accumulator `O_i`. When a new tile arrives: 1. Compute the new tile's max: `m_new = max(m_old, max(tile))` 2. Rescale the existing accumulators: `l_old *= exp(m_old - m_new)`, `O_old *= exp(m_old - m_new)`