Skip to content

Fix algorithm name in dataflow walkthrough #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
RaviTejGuntuku wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Fix algorithm name in dataflow walkthrough #2

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion dataflow_walkthrough.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -156,7 +156,7 @@ LSU fires `ISSUE_VEC_U softmax_online, scores`. This is the cleverest microcode

Naive softmax computes `exp(scores - max(scores)) / sum(exp(...))` — but this requires materializing the *entire* attention score matrix in SRAM before you can compute the max and the sum. With 3000-token context × 8 heads × 16-bit, that's 50 KB just for the matrix — already pushing the activation buffer.

The **FlashAttention-3 online algorithm** instead processes scores in tiles of (say) 32 at a time. Per attention row, VecU keeps just two running scalars — `m_i` (running max) and `l_i` (running sum-of-exps) — plus the running output accumulator `O_i`. When a new tile arrives:
The **FlashAttention online algorithm** instead processes scores in tiles of (say) 32 at a time. Per attention row, VecU keeps just two running scalars — `m_i` (running max) and `l_i` (running sum-of-exps) — plus the running output accumulator `O_i`. When a new tile arrives:

1. Compute the new tile's max: `m_new = max(m_old, max(tile))`
2. Rescale the existing accumulators: `l_old *= exp(m_old - m_new)`, `O_old *= exp(m_old - m_new)`
Expand Down