[EXTERNAL][AMDGPU] Fold constant offset into SV form addressing in SVS#2435
Open
bogdan-petkovic wants to merge 1 commit into
Open
[EXTERNAL][AMDGPU] Fold constant offset into SV form addressing in SVS#2435bogdan-petkovic wants to merge 1 commit into
bogdan-petkovic wants to merge 1 commit into
Conversation
Backport of two upstream llvm/llvm-project commits to fix an AMDGPU
frame-index elimination miscompile that shows up on gfx950:
* 5f371b2add3e [AMDGPU] Fold constant offset into SV form addressing
in SVS (llvm/llvm-project#208629)
* 6ce86d47ca09 [NFC][AMDGPU] Simplify scratch address materialization
in SVS (llvm/llvm-project#208624)
Under extreme VGPR pressure (~512) the double-buffer direct-to-LDS
schedule used by padded attention kernels hit a bad frame-index
elimination for the flat-scratch SVS (scalar+vector+scalar) addressing
form: the constant offset was not correctly folded/materialized into the
scavenged vaddr VGPR, producing wrong scratch addresses and NaNs in the
kernel output.
The fix folds as much of the constant offset as possible into the
instruction's immediate offset field and materializes the remainder
(plus the frame register, if any) into the scavenged VGPR. The upstream
regression test (eliminate-frame-index-flat-scratch-svs.mir) is included.
Verified against the June LLVM bump: an O0-vs-O3 reproducer of the
gfx950 padded-attention kernel goes from FAIL to PASS (3/3) with this
patch applied.
Signed-off-by: Bogdan Petkovic <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Padded attention kernels that use the double-buffer direct-to-LDS schedule produce NaNs on gfx950. The root cause is an AMDGPU backend miscompile, not an MLIR codegen issue. This PR backports the upstream LLVM fix so those configurations are numerically correct on gfx950 and no MLIR-side workaround (gating double-buffer to single-buffer when padding is present) is needed.
Technical Details
Clean backport of two upstream llvm/llvm-project commits (external subtree only):
5f371b2add3e[AMDGPU] Fold constant offset into SV form addressing in SVS (the fix)6ce86d47ca09[NFC][AMDGPU] Simplify scratch address materialization in SVS (prerequisite refactor + test)Under extreme VGPR pressure (~512), the double-buffer direct-to-LDS schedule hit a bad frame-index elimination for the flat-scratch SVS (scalar+vector+scalar) addressing form: the constant offset was not correctly folded/materialized into the scavenged vaddr VGPR, producing wrong scratch addresses and NaNs. The fix folds as much of the constant offset as possible into the instruction's immediate offset field and materializes the remainder (plus the frame register, if any) into the scavenged VGPR.
Files touched:
external/llvm-project/llvm/lib/Target/AMDGPU/SIRegisterInfo.cppand the added upstream regression testexternal/llvm-project/llvm/test/CodeGen/AMDGPU/eliminate-frame-index-flat-scratch-svs.mir.Test Plan
Built a standalone O0-vs-O3 reproducer of the failing gfx950 padded-attention kernel (extracted LLVM IR + HIP harness), run against the June LLVM bump with and without this patch. The upstream
.mirregression test (eliminate-frame-index-flat-scratch-svs.mir) is also included and runs as part of the LLVM test suite.Test Result
.mirtest passes.Submission Checklist