Skip to content

fix(flex): support GQA/MQA in attention (q_heads % kv_heads == 0)#5142

Open
AnayGarodia wants to merge 2 commits into
tracel-ai:mainfrom
AnayGarodia:fix/4930-flex-attention-gqa
Open

fix(flex): support GQA/MQA in attention (q_heads % kv_heads == 0)#5142
AnayGarodia wants to merge 2 commits into
tracel-ai:mainfrom
AnayGarodia:fix/4930-flex-attention-gqa

Conversation

@AnayGarodia

Copy link
Copy Markdown

Fixes #4930.

Problem

attention_impl (flash path) and attention_naive_impl asserted strict head-count equality between Q and K/V, panicking on ONNX-compliant Grouped-Query / Multi-Query Attention where kv_heads < q_heads.

Fix

Relax the check to require q_heads % kv_heads == 0 (matching the ONNX Attention spec and burn-ndarray) and map each query head to its shared K/V head kv_h = h / (q_heads / kv_heads). K/V batch strides now use kv_heads. Standard MHA is the q_per_kv == 1 special case; mask/bias broadcasting is unchanged (still keyed on q-heads). Applied identically to both the flash and naive paths.

Testing

Added test_attention_gqa_matches_repeated_kv (4 q-heads / 2 kv-heads) and test_attention_mqa_matches_repeated_kv (4 q-heads / 1 kv-head); both assert grouped output equals plain MHA with K/V heads repeated q_heads/kv_heads times, exercising both loops.

cargo test -p burn-flex attention                                 # 14 passed
cargo clippy -p burn-flex --all-targets -- -D warnings            # clean
cargo fmt --check                                                 # clean

Copilot AI review requested due to automatic review settings July 8, 2026 22:13

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

`attention_impl` and `attention_naive_impl` asserted `k_heads == q_heads`
and `v_heads == q_heads`, panicking on ONNX-compliant Group Query Attention
(GQA) and Multi-Query Attention (MQA) where `kv_heads < q_heads`.

Relax the head-count checks to require `q_heads % kv_heads == 0` (matching
the ONNX `Attention` spec and burn-ndarray), and map each query head `h` to
its shared K/V head `h / (q_heads / kv_heads)`. K/V batch strides now use
`kv_heads`. Plain MHA is the special case `q_heads == kv_heads`.

Add tests asserting GQA (4 q-heads, 2 kv-heads) and MQA (4 q-heads, 1
kv-head) match plain MHA with K/V heads repeated, across both the naive and
flash inner loops.
@AnayGarodia
AnayGarodia force-pushed the fix/4930-flex-attention-gqa branch from 25231dc to 15ff2b5 Compare July 17, 2026 19:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

burn-flex: attention_naive rejects ONNX-compliant Group Query Attention (mismatched K/V heads)

4 participants