The MathemaTricks behind FlashAttention
Fast and memory efficient exact attention
Datta's Blog
First-principles notes on LLM systems
I write visual, math-backed explanations of attention, fine-tuning, GPU kernels, and the engineering details behind modern deep learning systems.
The lore behind LoRA
LoRA imagined from the ground up
Exploring the Mixture of Experts
An intuitive build up to Mixture of Experts
Understanding multi-GPU Parallelism paradigms
A practical guide to data, tensor, and pipeline parallelism for LLM inference and training
Attention and Transformer Imagined
An intuitive build up to Attention and Transformer
Transformer showdown MHA vs MLA vs nGPT vs Differential Transformer
Comparing various transformer architectures like MHA, GQA, Multi Latent Attention, nGPT, Differential Transformer.
Rethink LoRA initializations for faster convergence
A better initialization for LoRA to make convergence faster