FlashAttention from first principles

Lately with all the buzz around new LLM releases, claude code limits and workflow or agents, skills and agents orchestration. I think it is nice every now and then to step back and actually understand some of the foundational stuff too.

This week I had some time and spent it going back to understand FlashAttention from first principles.

Standard attention is memory-bound, meaning it does not account for the GPU memory hierarchy and repeatedly shuffles large intermediate matrices between slow and fast GPU memory. FlashAttention addresses this by making attention IO-aware. It computes exact standard attention by restructuring the computation to minimize data movement between these memory levels. The result is faster training, longer context length support and lower attention memory footprint.

I wrote a short blog on it. It is not an exhaustive deep dive but it goes deep enough to build intuition around why standard attention is slow and memory-bound and how FlashAttention fixes it using ideas like kernel fusion, tiling, recomputation, and online softmax.

You can find the blogpost here: https://aayushgarg.dev/posts/2026-03-27-flash-attention/