Attention's Memory Problem Has Three Solutions: MQA, GQA, and MLA

1. The KV Cache Is the Bottleneck You Cannot Ignore

In transformer inference, the forward pass during decode is not compute-bound. It is memory-bound — specifically, by the key-value (KV) cache that accumulates for every token in context. The roofline arithmetic from article #5 applies here with surgical precision: at typical serving batch sizes, the arithmetic intensity of the attention operation sits near 1 FLOP/byte. The H100’s ridge is at 295 FLOP/byte. You are 295× below compute saturation. Every cycle spent in decode is a cycle spent waiting on HBM.

The KV cache grows as:

KV cache size = 2 × n_layers × n_heads × d_head × seq_len × sizeof(dtype)

For Llama-3-70B at FP16 with 80 layers, 64 attention heads, head dimension 128, serving a 32K context at batch size 32:

= 2 × 80 × 64 × 128 × 32,768 × 2 bytes
≈ 137 GB

The H100 has 80 GB of HBM3. The KV cache alone, at this context length and batch size, does not fit. You have three structural choices: evict tokens (lossy), offload to CPU memory (slow), or compress the KV cache architecturally. The third option is what MQA, GQA, and MLA are doing — and they are not doing it equally well.

2. Multi-Query Attention (MQA) — The Aggressive Baseline

Noam Shazeer’s 2019 paper (arXiv:1911.02150) introduced Multi-Query Attention. The idea is radical in its simplicity: all query heads share a single key head and a single value head. If the model has $n_h$ query heads each of dimension $d_h$, standard MHA projects to $n_h$ key heads and $n_h$ value heads. MQA collapses both to one.

The projection at inference time becomes:

Q ∈ R^{n_h × d_h}   (per-head queries, unchanged)
K ∈ R^{1 × d_h}     (one key head, shared)
V ∈ R^{1 × d_h}     (one value head, shared)

The KV cache reduction is therefore $n_h \times$ — for a 64-head model, you cut KV memory by 64×. For the Llama-3-70B example above, that 137 GB cache compresses to ~2.1 GB. Suddenly a single H100 can serve 30+ concurrent long-context sessions without running out of HBM.

The engineering appeal is obvious. The cost is quality. The original Shazeer paper noted degradation relative to MHA; follow-up analysis (Ainslie et al., 2305.13245) found that the quality gap is “significant” particularly on tasks demanding fine-grained per-head attention differentiation. Different query heads exist precisely because different heads specialize — one head tracks syntactic dependencies, another tracks coreference, another attends to position. When all heads share a single K and V, you destroy the representational diversity those keys and values were providing. MQA essentially asks every query head to ask its distinct question while the keys and values answer from an undifferentiated consensus. The information bottleneck is structural.

Despite this, MQA was deployed. PaLM (540B) and Falcon used it. At the scales and latency requirements where KV memory is immediately catastrophic, MQA bought breathing room at an acceptable quality cost. But it was always a blunt instrument. The question was whether you could get most of the memory reduction with far less of the quality damage.

3. Grouped-Query Attention (GQA) — The Production Standard

Ainslie et al. (2023) provided the answer (arXiv:2305.13245). Grouped-Query Attention is the interpolation between MHA and MQA that turned out to matter enormously in practice. Instead of forcing all $n_h$ query heads to share a single K/V pair, GQA divides query heads into $g$ groups, each group sharing its own K and V head:

Q ∈ R^{n_h × d_h}    (all query heads)
K ∈ R^{g × d_h}      (g key heads, one per group)
V ∈ R^{g × d_h}      (g value heads, one per group)

Each group contains $n_h / g$ query heads. The KV cache reduction is $n_h / g \times$ relative to MHA.

The key empirical finding: quality loss falls dramatically at very small group counts. GQA with 8 groups retains nearly MHA-level quality on standard benchmarks while achieving an 8× KV cache reduction. The sweet spot identified in the paper is exactly what Meta deployed: Llama 2-70B uses 8 KV heads for 64 query heads — 8 groups of 8 queries each. The same architecture carries forward into Llama 3.

For the 70B example above:

KV cache = 137 GB × (8/64) = 137 GB / 8 ≈ 17 GB

At batch 32, 32K context, you now sit comfortably inside a single H100. Raise context to 128K:

= 2 × 80 × 8 × 128 × 131,072 × 2 bytes ≈ 27 GB

Still fits. The weight file (70B × 2 bytes ≈ 140 GB) doesn’t fit on one H100 regardless — but with 8-way tensor parallelism across an HGX node, each GPU holds ~17.5 GB of weights and ~3.4 GB of KV cache per 32K session. The arithmetic works.

Why does 8 groups retain quality where 1 group (MQA) loses it? The interpretive answer is that attention heads cluster into semantic roles, and those roles are coarser than $n_h$. Empirically, 8 clusters capture most of the per-head specialization that 64 heads represent. You lose information at the margin. You keep the structure that matters.

GQA’s implementation complexity is minimal — it is a straightforward change to how K and V projections are sized and how the attention kernel indexes into the KV heads. Flash Attention 2 supports GQA natively. The Triton kernels in vLLM handle it. Deployment is friction-free.

This is why GQA is the current production standard for 7B–70B class models. It is the correct default until the context lengths push you somewhere GQA can’t reach — which is exactly where MLA lives.

4. Multi-head Latent Attention (MLA) — The Compression Frontier

DeepSeek-V2 (arXiv:2405.04434, 2024) introduced Multi-head Latent Attention, which attacks the KV cache problem from a fundamentally different angle. GQA reduces the number of key and value heads. MLA reduces the rank of the key-value representation itself, projecting into a low-dimensional latent space and reconstructing full attention at compute time.

The architecture is defined by two projections at each layer:

Compression (write):

c_t^{KV} = W^{DKV} h_t

where c_t^{KV} ∈ R^{d_c},  d_c ≪ d_h × n_h

$h_t$ is the hidden state at position $t$. $W^{DKV}$ is the down-projection that compresses into a latent vector of dimension $d_c$. This latent vector, not the full KV tensors, is what gets cached.

Reconstruction (read):

K_t = W_K^{UK} c_t^{KV}
V_t = W_V^{UV} c_t^{KV}

At attention time, the cached latent $c_t^{KV}$ is up-projected back to full key and value tensors via learned weight matrices $W_K^{UK}$ and $W_V^{UV}$. The up-projection happens on-chip, during the attention computation, paid in FLOPs — which, as established, are not the binding constraint.

DeepSeek-V2 sets $d_c = 512$ for a model with $n_h = 128$ heads of dimension $d_h = 128$. Full MHA KV dimensionality per position would be $2 \times 128 \times 128 = 32{,}768$ values. MLA stores 512. That is a 64× compression of KV cache per position in the limit. In practice, DeepSeek-V2 also caches a decoupled rotary position embedding (RoPE) key of dimension $d_c^R = 64$, raising the effective cached size to 576 per position per layer, giving approximately ~5.75× compression vs MHA on the full KV tensor including position components, or roughly ~14× compression vs standard MHA by the DeepSeek team’s reporting methodology, which compares against their full n_h × d_h baseline.

The math for why this is lossless at sufficient $d_c$: the full key and value tensors $K, V \in R^{n_h \times d_h}$ lie in a space of dimension $n_h \times d_h = 16{,}384$. If the actual information in that space is low-rank — which empirical evidence increasingly suggests it is, particularly for long-context sequences where keys and values become highly correlated across positions — then a $d_c = 512$ latent captures the essential subspace while discarding noise.

The quality results from DeepSeek-V2 are striking: despite the aggressive compression, MLA matches or exceeds MHA on standard evaluation benchmarks when trained from scratch with MLA’s compression objective baked in. This is the crucial point — MLA is not a post-hoc compression applied to an MHA model. The model is trained to route information through the latent bottleneck. The bottleneck becomes a regularizer.

The RoPE complication deserves an honest note. Standard RoPE applies position-dependent rotation to the query and key vectors. If K is computed as $W_K^{UK} c_t^{KV}$, applying RoPE after the up-projection breaks the ability to cache $c_t^{KV}$ and recover the correctly rotated K at attention time — you would need to cache the rotated K instead, defeating the compression. DeepSeek-V2 addresses this by decoupling position: a small additional RoPE key $k_t^R$ is cached alongside $c_t^{KV}$, and the content key $K_t$ from the latent is concatenated with $k_t^R$ at attention time. This adds $d_c^R = 64$ dimensions to the cache per position per layer — a modest overhead that the 64× latent compression more than absorbs.

5. The Tradeoff Matrix

No architecture dominates on all axes. The decision depends on context length regime, model size, quality requirements, and engineering resources:

The “uptraining feasible” entry for GQA reflects the Ainslie et al. finding that existing MHA models can be converted to GQA by mean-pooling existing KV heads into groups, followed by fine-tuning for a fraction of original training compute — and quality is retained. This matters enormously: you can take a pretrained MHA checkpoint and GQA-ize it without retraining from scratch. MLA does not share this property. The compression pathway must be present during training because the model learns to route useful information through the latent bottleneck rather than distributing it freely across all KV dimensions.

The FLOPs overhead of MLA’s up-projection ($W_K^{UK}$ and $W_V^{UV}$ applied to cached latents at each attention step) is real but modest. For a single decode step: the up-projection cost is $2 \times d_c \times n_h \times d_h = 2 \times 512 \times 128 \times 128 \approx 16.8M$ FLOPs per layer. Against the full attention compute for a 128K context sequence, this is negligible. But it does require a fused kernel that handles the latent-to-KV reconstruction inline with the attention computation, and current FlashAttention implementations do not natively support this. Custom CUDA or Triton kernels are required, which raises the engineering bar.

6. Why MLA Wins at Long Context — and What It Costs You

The scaling argument for MLA is straightforward. For a 128K context sequence, a 70B-class model with 80 layers:

GQA (g=8) KV cache at 128K, batch=1:

2 × 80 × 8 × 128 × 131,072 × 2 bytes ≈ 27 GB

Fits on a single H100, barely. At batch=4, you are at 108 GB — spilling off-chip or evicting.

MLA KV cache at 128K, batch=1 (14× compression):

27 GB / 14 ≈ 1.9 GB

At batch=32: ~61 GB — still fits, with 19 GB left for weights in a sharded configuration.

MLA does not just improve KV cache size; it changes the throughput curve qualitatively. Because KV cache is the binding memory constraint at long context, MLA shifts the bottleneck back toward the weight matrix bandwidth — which is constant with sequence length. GQA buys you headroom; MLA buys you a different regime of operation entirely.

The production winner determination is therefore workload-conditional:

For serving ≤70B models at contexts ≤32K: GQA with g=8 is the answer. Zero kernel complexity, universal framework support, proven quality, fits in HBM. No contest.

For serving 70B+ models at contexts ≥128K, or for next-generation architectures being trained from scratch: MLA is the correct investment. The 14× KV compression changes the system architecture: you can increase batch size by 14×, serve more concurrent users, or reduce the number of GPUs required by allowing fewer shards to hold the same KV state. At the model sizes where 128K context is a production requirement — code generation over large repositories, long-document analysis, multi-turn agents — MLA’s economics dominate.

The kernel complexity cost is real. DeepSeek’s custom attention kernel, integrated into their inference stack, is not a drop-in for vLLM or TensorRT-LLM today. The reconstruction path ($c_t^{KV} \to K, V$) must be fused with the attention computation to avoid materializing full KV tensors in HBM — which would defeat the purpose. An unfused implementation that computes $K = W_K^{UK} c_t^{KV}$, writes K to HBM, then reads it back for attention is strictly worse than GQA. Fusion is not optional; it is load-bearing.

The Triton implementation path for MLA requires a custom kernel that accepts cached latents as input, performs the up-projection inside shared memory, computes attention scores against the query, and outputs the attention-weighted values — all without a full KV materialization in HBM. As of mid-2025, this kernel exists in DeepSeek’s inference stack and in a small number of community implementations, but production-grade support in major inference frameworks is still maturing. The trajectory is clear — MLA will be universally supported by 2026 — but if you need it today, you are writing or adapting kernels.

Synthesis: The Decision Tree

The correct mental model is not “which is best” but “which constraint is currently binding”:

If your KV cache fits in HBM with GQA at your target context and batch size, use GQA. The quality, ecosystem, and zero implementation overhead are decisive. If your model was trained with MHA and you need to compress it, GQA uptraining is the only viable path — MLA is not retrofittable.

If you are training a new model and your target deployment regime involves contexts above 64K at meaningful batch sizes, architect for MLA from the start. The quality results from DeepSeek-V2 establish that you do not sacrifice model capability by doing so. What you gain is the ability to serve 14× more concurrent users at long context for the same HBM budget — and at the system economics of GPU-hour pricing, that factor of 14× is the difference between a profitable inference endpoint and an underwater one.

The memory hierarchy is ultimately the arbiter of all these choices. Registers, SRAM, L2, HBM — each level of the hierarchy that you can avoid crossing is latency and bandwidth you don’t pay. MQA avoided it crudely. GQA avoided it elegantly. MLA avoided it mathematically. The physics hasn’t changed; only the cleverness with which we negotiate it.

References

• Shazeer, N. (2019). Fast Transformer Decoding: One Write-Head is All You Need. arXiv:1911.02150.
• Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebrón, F., & Sanghai, S. (2023). GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints. arXiv:2305.13245.
• DeepSeek-AI. (2024). DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model. arXiv:2405.04434.
• Dao, T. et al. (2022). FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. arXiv:2205.14135.
• NVIDIA. (2022). H100 Tensor Core GPU Architecture Whitepaper.

@article{fp4-2606015,
  title   = {Attention's Memory Problem Has Three Solutions: MQA, GQA, and MLA},
  author  = {fp4 editorial desk},
  year    = {2026},
  url     = {https://fp4.dev/algorithm/gqa-mqa-mla-inference/},
  journal = {fp4}
}