Skip to content
Scrollchart
AI & Machine Learning Rich #flash-attention#gpu#kernel

FlashAttention: Tiled & Fused

Compute attention in blocks that fit in SRAM. Fused, online-softmax, no materialized N x N.

A free, animated flashattention: tiled & fused you can read here or embed on any website, from Scrollchart.

FlashAttention: Tiled & Fused

FlashAttention: Tiled and Fused Attention KernelTiles Q, K, V into SRAM blocks. Online softmax. No N x N matrix in HBM. Memory O(N^2) -> O(N).Naive Attentionwrites full N x N score matrix to HBMHBM: N x N floats (e.g. 512^2 = 262 k, fp16 = 0.5 MB per head)Q (seq)K (seq)Memory: O(N^2) HBM passes: 2 (write scores, read for softmax)FlashAttentiontiles into SRAM, online softmax, no HBM write for scoresQtileKtileVtileall 3 tiles fit in SRAM (~4-16 MB on A100)online softmax+ running statsOutput Owritten onceMemory: O(N) HBM passes: O(N^2 / M) where M = SRAM sizeHBM read/write operations (relative, N=4096, d=128, A100 SRAM=20MB)Naive~6x more HBM trafficFlashO(N) HBM writes. 2-4x end-to-end GPU speedup.FlashAttention (Dao et al. 2022) is IO-bound, not compute-bound. Tiling keeps all active data in SRAM and avoids round-tripping the N x N matrix through HBM, enabling longer contexts at practical speed.

Naive attention materializes the N x N score matrix in HBM. FlashAttention tiles Q, K, V into SRAM-sized blocks, computes attention output and online-softmax stats incrementally. Memory drops from O(N^2) to O(N). HBM read/write counts plotted.

Good for

  • GPU kernel and inference optimization tutorials for ML engineers
  • Long-context LLM architecture articles explaining why 100k context is now practical
  • FlashAttention vs standard attention comparisons in training and serving frameworks

Source & accuracy

This flashattention: tiled & fused is an editorial illustration built to represent the concept accurately. Where it shows figures, they are typical or representative values chosen to make the relationship clear, not a single underlying dataset. The diagram and its explainer are reviewed and maintained centrally, and updated over time as understanding improves.

Computing attention without materializing the full matrix

Standard attention computes a full N x N matrix of pairwise similarities (where N is sequence length), then softmax and multiplies by values. For a sequence of 4K tokens, this is a 16M-element matrix, consuming gigabytes of memory and bandwidth. FlashAttention reduces memory and compute by processing attention in tiles that fit in fast SRAM.

The trick is recomputing softmax incrementally. As you process tiles, you update the running sum and normalization separately, avoiding the need to store the full attention matrix.

Speed and memory improvements in practice

FlashAttention is 2-4x faster than standard attention on modern GPUs, with proportional memory savings. For long contexts (8K to 100K+ tokens), this difference is transformative: standard attention becomes infeasible, while FlashAttention remains practical.

The fused kernel (combining multiple operations into one) is key. Separate calls to compute attention, softmax, and multiply incur overhead; one fused kernel avoids it.

Enabling longer contexts in training and inference

FlashAttention made long-context pretraining feasible. Models like Llama 2 with 4K context and later 8K were enabled by this. Without FlashAttention, context length is limited by available memory, forcing shorter sequences and shorter-range dependencies.

Embed this diagram

Add this animated flashattention: tiled & fused to your own site. Copy one line of HTML, or use the embed builder for theme and sizing options.

Reference

What this is
A free, embeddable, animated flashattention: tiled & fused for any website.
Who uses it
AI/ML blogs, DevOps / SRE sites.
How to embed
Copy one line of HTML. No signup. No watermark. Works in WordPress, Webflow, Ghost, Substack, plain HTML.
File size
iframe embed, ~80 KB gzipped (loads on demand, does not block your page paint).
License
Free forever. Editorial explainer text included; updated centrally over time.
URL
scrollchart.com/diagram/flash-attention

Embed format options

Copy the universal HTML snippet, the WordPress shortcode, or an iframe fallback - see the WordPress plugin page for details. Any format keeps the same Core Web Vitals profile and the same explainer text.

Embed snippet 
<div data-scrollchart="flash-attention" data-scrollchart-v="1"></div>
<script src="https://scrollchart.com/embed.js" async></script>

By embedding, you acknowledge the diagram and explainer text are served from scrollchart.com and may be updated from time to time to reflect new research or to link to relevant sources within the explainer.

Frequently asked questions

Where can I get a free animated "FlashAttention: Tiled & Fused" for my website?
Scrollchart provides "FlashAttention: Tiled & Fused" as a free, embeddable animated diagram you can add to any website with one line of HTML. No signup is required and there is no watermark. The diagram and its explainer text are served from scrollchart.com, so the embed stays current without any maintenance on your end.
How do I embed a flashattention: tiled & fused in a developer or tech blog?
Copy the one-line snippet from the Scrollchart diagram page and paste it into your post HTML. It works in any static site generator, CMS, or hand-coded HTML page. The embed is a thin loader, not an iframe, so the content is fully in your DOM.
Related diagrams

Other ai & machine learning explainers

See all ai & machine learning →
AI & Machine Learning Rich

Multi-Head Attention

Each head learns its own Q/K/V projections. Many heads, many relational lenses, concatenated.
AI & Machine Learning Rich

KV Cache: Why Generation Speeds Up

During generation, past K and V are cached. Each new token only computes its own.
AI & Machine Learning Rich

MLA: Multi-Head Latent Attention

Compress K and V to a low-rank latent before caching. Massive cache shrink, near-MHA quality.