All the attentions

The whole picture in one table

The techniques fall into five groups:

• Move data faster: FlashAttention
• Make the cache smaller: GQA, MLA
• Read fewer old tokens: Sliding Window, DSA
• Replace the cache with a fixed-size memory: Gated DeltaNet, KDA
• Manage the cache during serving: PagedAttention

Words you need

Read this list once. The rest of the note is easier after it.

Why we need any of this: two costs

This note uses one example model throughout, shaped like Llama 3 8B: 32 layers, 32 query heads, 8 KV heads, head size 128, and numbers in 16-bit form (2 bytes).

Cost 1: the notebook (KV cache). For each token, each layer stores one key and one value for each KV head:

2 (K and V) x 32 layers x 8 KV heads x 128 dims x 2 bytes = 131,072 bytes, about 128 KB per token

A model with a 128k context can fill most of a 24 GB machine with the cache of one chat, and the model weights still need room. Serve 50 users at once, and the cache decides how many GPUs you need.

Cost 2: the score table. To decide where to look, attention compares each token with every earlier token. For 8,000 tokens, that is about 67 million comparisons for each head in each layer. If the text is two times as long, the work is four times as large.

Each technique below lowers one of these costs or both.

1. FlashAttention: move data faster

In one line: the same exact result, computed faster because the model avoids slow memory trips.

The problem. A standard implementation builds the full table of scores. Then it writes the table to slow GPU memory (HBM), reads it back to apply softmax, writes it again, and reads it again to multiply with the values. For 8k tokens, the table is about 134 MB for each head in each layer. The math units wait for the data.

The idea. Think of a cook who walks to the pantry for each ingredient, against a cook who brings a tray to the counter. FlashAttention splits Q, K, and V into small tiles that fit in the fast SRAM of the chip. The hard part is softmax, which normally needs all scores in a row. FlashAttention uses an online softmax: it keeps a running maximum and a running sum, and it corrects earlier results as new tiles arrive. The large table never exists in slow memory.

import torch.nn.functional as F
out = F.scaled_dot_product_attention(q, k, v, is_causal=True)
# On supported GPUs this call uses a FlashAttention-style kernel.

The trade: special GPU kernels are necessary. The output matches standard attention, so quality does not change. It saves the memory of the score table and a lot of time. It does not shrink the KV cache.

2. GQA: make the notebook smaller

In one line: several query heads share one set of keys and values.

The problem. In standard multi-head attention, each of the 32 heads has its own keys and values. The notebook then holds 32 sets for each token.

The idea. Keep 32 query heads (32 readers), but keep only 8 sets of keys and values (8 notebooks). Each group of 4 readers shares one notebook. The extreme case, one notebook for all heads, is called MQA. The in-between case is GQA.

At 8k tokens, a switch from MHA to GQA turns a 4 GB cache into a 1 GB cache. In code, the only change is that k_proj and v_proj output 8 heads, and each KV head is copied to line up with its group of query heads:

k = k.repeat_interleave(n_heads // n_kv_heads, dim=1)  # 8 heads -> 32 heads
v = v.repeat_interleave(n_heads // n_kv_heads, dim=1)
out = F.scaled_dot_product_attention(q, k, v, is_causal=True)

The trade: the heads share information, so there is a little less variety. In practice, quality barely drops. GQA saves memory, not math: 32 query heads still do the same number of comparisons.

3. MLA: compress the notebook

In one line: store a small compressed summary of each token, and rebuild the keys and values from it when needed.

The problem. GQA shrinks the cache by making heads share, and that can cost some quality. Can the cache shrink and keep each head distinct?

The idea. Do not save a full-size photo of each page. Save a small compressed picture, and expand it when you need it.

1. Project the token vector down to a small latent vector. It has a few hundred numbers, not thousands.
2. Cache only that latent vector.
3. At attention time, project it back up into keys and values for each head.

The project-back-up matrices can be merged into the query and output projections. At decode time, the model then does not rebuild full keys and values. A short extra key carries position information for RoPE, the rotary position encoding. The model caches that key with the latent.

DeepSeek caches about a 512-number latent and a 64-number position key for each token in each layer. For a model of the size of DeepSeek-V3, that is tens of KB per token, where full per-head keys and values would need several MB. DeepSeek reported a cache reduction of about 93% in the V2 paper, against their earlier dense model. Check the model card for the exact numbers of your configuration.

GQA and MLA in one sentence: GQA shrinks the cache by sharing heads. MLA shrinks it by compression, with each head kept distinct.

The trade: more complex to build, with extra projections and special position handling. Optimized kernels matter. The choice is made at training time, so an existing model cannot switch to it.

4. Sliding Window: read fewer old tokens

In one line: each token looks only at the last W tokens.

The problem. Attention to every earlier token costs more as the text grows, and the cache grows too.

The idea. When you read a novel, you remember the last few pages in detail and earlier pages only vaguely. A sliding window does the same. Token i may read only tokens i-W+1 to i. Older entries are never needed again, so the cache becomes a rolling buffer of fixed size W. New entries replace the oldest ones.

Mistral 7B uses W = 4,096. At a 32,000-token context, a full cache would be about 4 GB. The windowed cache stays at about 512 MB, no matter how long the text gets. The mask is a few lines:

import torch

def sliding_window_mask(T, W):
    i = torch.arange(T)[:, None]   # query position
    j = torch.arange(T)[None, :]   # key position
    return (j <= i) & (j > i - W)  # True = allowed to attend

For T=6 and W=3, token 3 can see tokens 1, 2, and 3, but not token 0.

Information still travels far. Layer 1 lets a token see W tokens back. Layer 2 sees tokens that already hold information from W further back. After L layers, the reach is about L x W, but it becomes less sharp with distance.

The trade: the model cannot look up a fact from far outside the window. Many models mix window layers with some full (global) layers. Gemma 3 uses about 5 local layers, with a window of 1,024, for each global layer. gpt-oss alternates windowed and full layers.

Note: a plain mask still computes the full score table and then hides most of it. Real savings need a kernel that skips the hidden blocks, such as the window option in FlashAttention.

5. DSA: pick the old tokens that matter

In one line: a cheap scorer picks the old tokens that matter, and full attention runs on those only.

The problem. A sliding window is blunt. It keeps the most recent tokens, not the most relevant tokens. If the answer sits at token 3,000 of a 100,000-token document, a window will not find it.

The idea. Use the index of a book, not every page. A small, fast part called the lightning indexer scores each earlier token for relevance to the current token. The real attention then reads only the top-k scores. DeepSeek uses about 2,048. The choice is based on content, not position.

At 128,000 tokens of context, each new token reads about 2,048 tokens instead of 128,000. That is about 60 times fewer. The indexer still looks at every token, but it is small and cheap compared with full attention.

Window and DSA: a window keeps a fixed recent part. DSA picks the relevant part, wherever it is.

The trade: DSA saves compute and memory reads, not cache storage. The full cache stays, so the indexer can choose from it. The indexer needs training, and it can miss a token that mattered. DeepSeek added it by continued training of an existing MLA model, not from zero.

6. Gated DeltaNet: a fixed-size memory

In one line: a fixed-size memory, updated by corrections, with a fade gate.

Background: linear attention. Do not store every key and value. Keep one matrix S for each head, as a summary. Each new token writes its key-to-value link into S. Each query reads from S. The size of S never changes, from 1,000 tokens to 1,000,000 tokens. This is the same idea as an RNN, a model with a hidden state.

The problem with the simple form. If you only add, S becomes a blur. Old and new information pile up and interfere.

Two fixes give the name:

• Delta rule (update by correction). Before a write, check what the memory already predicts for this key. Write only the difference between that and the true value. If the memory knows the fact, almost nothing changes. If the memory is wrong, it gets a correction.
• Gate (fading). A learned number decides how much of the old memory fades before each update. Old information can then go away.

Think of a fixed-size whiteboard. When new information contradicts a line, you erase that line and write it again. You also have a dimmer that fades old content.

Toy example, one head, no normalization:

import torch

d_k, d_v = 4, 4
S = torch.zeros(d_v, d_k)          # the whole memory: fixed size

def step(S, q, k, v, alpha, beta):
    S = alpha * S                  # gate: fade the old memory
    err = v - S @ k                # what the memory gets wrong for this key
    S = S + beta * torch.outer(err, k)   # write only the correction
    out = S @ q                    # read
    return S, out

At 1M tokens, the cache of a standard layer is in gigabytes. The state S here is one d_v x d_k matrix for each head, for example 128 x 128. It has the same size at token 1 and at token 1,000,000.

The trade: a fixed memory cannot hold everything exactly. It is good at the gist and weaker at an exact phrase from 400k tokens ago. Real models are hybrids. Qwen3-Next uses Gated DeltaNet layers for most of the stack, and one full attention layer about every fourth layer for exact lookups.

7. KDA: one fade control for each channel

In one line: Gated DeltaNet, but with a separate fade control for every channel.

The problem. The gate in Gated DeltaNet is coarse: one fade value for a whole head. Some features must stay for a long time, such as a name. Others must fade fast, such as the grammar of the last sentence.

The idea. Instead of one master dimmer for the whole whiteboard, KDA has one dimmer for each column. In code, the gate goes from one number to a vector:

S = alpha[None, :] * S   # alpha now has one value per key channel
# the correction and read steps stay the same

Example: Kimi Linear. This model uses KDA layers and MLA layers in a 3:1 ratio: three KDA layers for each full-attention MLA layer. It has 48B parameters, about 3B active for each token, and a 1M-token context. The authors report up to 75% less KV cache and about 6 times faster decode at 1M tokens, against a full-attention MLA baseline. The MLA layers here use no positional encoding. The KDA layers carry the sense of position.

The trade: more complexity, and kernels that know the hardware are necessary. The paper sets the math so it works well in parallel on GPUs. As with Gated DeltaNet, a hybrid with a few full-attention layers covers exact long-range recall.

8. PagedAttention: manage the cache during serving

In one line: store the KV cache in small fixed-size blocks, like an operating system stores memory in pages.

The problem. This one is about many users, not about the math. Early serving systems reserved one large continuous block of GPU memory for each request, sized for the maximum possible length. A request that used 300 tokens still held space for 8,000. Free space also broke into small gaps that no request could use. The vLLM paper found that existing systems wasted a large part of the cache this way, 60% to 80% in their measurements.

The idea. Think of a car park where each car can take any free space, instead of reserving 100 spaces in a row. The cache splits into fixed-size blocks, for example 16 tokens. A block table maps each request to the physical place of its blocks. Blocks are allocated only when needed.

Because blocks are separate, sharing is easy. If 100 requests start with the same system prompt, they can all point at the same physical blocks. A block is copied only when a request writes to it (copy-on-write).

Example: your chatbot has a 2,000-token system prompt for every user. With paging, the cache of that prompt is stored once and shared, not 100 times. The vLLM paper reports up to 2 to 4 times more throughput than earlier systems at the same latency, mostly because more requests fit in memory at once.

The trade: extra bookkeeping, and the attention kernel must follow the block table. The output of the model does not change.

How real models combine these

The techniques are ingredients, not competitors.

The common pattern is a hybrid. Cheap layers do most of the work. A few expensive full-attention layers keep exact recall over a long range.

Cheat sheet

Rule of thumb: FlashAttention and PagedAttention are free speed for most teams. You can usually turn them on. The other six are design decisions, made when a model is trained.

Takeaways

• Attention costs two things: memory (the KV cache) and time (comparisons and data movement).
• GQA and MLA make the notebook smaller. Sliding Window and DSA read less of it. Gated DeltaNet and KDA replace it with a fixed-size memory. FlashAttention reads it faster. PagedAttention stores it with less waste.
• The best models mix several of these. You will often see two or three in one architecture.
• The numbers here use one example configuration. Always read the config file of the model for the real head counts, window sizes, and layer ratios.

Sources and further reading

• Attention Lab, the live guide this note follows.
• Dao et al., FlashAttention (2022).
• Ainslie et al., GQA: Training Generalized Multi-Query Transformer Models (2023).
• DeepSeek-AI, DeepSeek-V2 (MLA) and the DeepSeek-V3.2-Exp release (DSA).
• Jiang et al., Mistral 7B (2023).
• Yang et al., Gated Delta Networks (2024).
• Kimi Team, Kimi Linear: An Expressive, Efficient Attention Architecture (arXiv 2510.26692).
• Kwon et al., Efficient Memory Management for LLM Serving with PagedAttention (2023).