[release/2.11] Fix ROCm Triton wheel version to match torch's triton pin#3477
Draft
ethanwee1 wants to merge 5 commits into
Draft
[release/2.11] Fix ROCm Triton wheel version to match torch's triton pin#3477ethanwee1 wants to merge 5 commits into
ethanwee1 wants to merge 5 commits into
Conversation
The manylinux wheel build succeeded, but the downstream `pip install torch`
step (torchvision stage) failed to resolve torch's Triton dependency:
ERROR: Could not find a version that satisfies the requirement
triton==3.8.0+rocm7.2.4.git4cff872c
(from versions: ..., 3.8.0+git4cff872c.rocm7.2.4.git4cff872c)
ERROR: No matching distribution found for
triton==3.8.0+rocm7.2.4.git4cff872c
torch pins `triton==<ver>+rocm<rocm>.git<sha>` (from triton_version.txt +
rocm version + short commit hash), but the built Triton wheel was versioned
`<ver>+git<sha>.rocm<rocm>.git<sha>` with an extra `+git<sha>`.
Root cause: build_triton_wheel.py checks out the Triton pin as a detached
HEAD. At the pin, ROCm/triton's setup.py computes
`TRITON_VERSION = "<ver>" + get_triton_version_suffix()`, and
get_triton_version_suffix() appends `+git<sha>` unless the current branch name
starts with "release". Combined with the ROCm env suffix, this produced the
mismatched version. Triton 3.7.x did not embed the git hash this way, so this
only started failing after the 3.8.x pin bump.
Fix (ROCm only): check the pinned commit out onto a "release/"-prefixed local
branch so triton's git suffix stays empty and the wheel is versioned
canonically as `<ver>+rocm<rocm>.git<sha>`, matching torch's dependency. The
commit hash is still preserved via the rocm suffix. CUDA/XPU paths are
unchanged.
…ild-script workaround Point the Triton pin at ROCm/triton#957 (72c9556a1, "Remove git version suffix" on release/internal/3.8.x), which fixes the wheel version at the source. With that commit, TRITON_VERSION no longer appends "+git<sha>", so the build-script workaround in build_triton_wheel.py is unnecessary and is reverted to upstream behavior.
… break
The manylinux wheel build failed compiling torch_hip against the AOTriton
0.12.50tp2 headers under GCC 11 (gcc-toolset-11):
In file included from /pytorch/torch/include/aotriton/flash.h:7,
from .../ATen/native/transformers/hip/sdp_utils.cpp:31:
/pytorch/torch/include/aotriton/config.h:27:22: error: expected identifier before '__attribute__'
27 | #define AOTRITON_API __attribute__ ((visibility ("default")))
/pytorch/torch/include/aotriton/v2/cpp_tune.h:26:104: error: expected initializer before ':' token
26 | enum [[deprecated(...)]] AOTRITON_API CppTuneSpecialKernelIndex : int {
ninja: build stopped: subcommand failed.
The 0.12.50tp2 preview package applies the AOTRITON_API visibility attribute
(__attribute__((visibility("default")))) to enum/struct declarations in the
deprecated V2 API headers. GCC does not accept a visibility attribute in that
position on an enum with a fixed underlying type, so the build fails (clang
tolerates it). The 0.13b package used by release/2.12 and upstream main builds
cleanly with the same toolchain.
Bump the AOTriton pin to 0.13b (matching release/2.12 and main), updating the
version, CI commit, and runtime/image SHA256 lists accordingly.
… fix Temporarily build torchaudio from ethanwee1/audio@7eeeffe7 (release/2.11 base 34c52a67 + "pass extension sources as str") so the release/2.11 wheel-stack build can get past the torchaudio `AssertionError: 'sources' must be a list of strings` and validate the full stack (triton pin + AOTriton 0.13b + torchaudio fix) end to end. Revert this once the torchaudio fix lands upstream (pytorch/audio release/2.11) and restore the pytorch/audio pin.
…r fix Build torchaudio from the ROCm/audio release/2.11.0.2 sources-str fix (ROCm/audio#19) so the release/2.11 wheel-stack build gets past the torchaudio `AssertionError: 'sources' must be a list of strings` and validates the full stack (triton pin + AOTriton 0.13b + torchaudio fix) end to end. The commit is hosted on the PR #19 head branch (ethanwee1/audio, no push access to ROCm/audio directly). Once ROCm/audio#19 merges, repoint this at https://github.com/ROCm/audio release/2.11.0.2 @ the merged SHA.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
release/2.11 manylinux wheel-build fix stack
This is the integration PR for the ROCm
release/2.11wheel build. The whole stack below was validated together in one green combined wheel build (torch + triton + torchaudio all build, andpip install torchresolves cleanly).Component PRs
TDMCommon.hinpackage_triton_wheel.sh(merged)0.12.50tp2->0.13b(fixes the GCCtorch_hipbuild break)sourcesasstr(fixes'sources' must be a list of strings)What this branch carries
.ci/docker/ci_commit_pins/triton.txt-> the [release/internal/3.8.x] Remove git version suffix triton#957 commitcmake/External/aotriton.cmake-> AOTriton0.13brelated_commits[TEST ONLY] -> torchaudio built from the [release/2.11] Pass extension sources as str, not pathlib.Path audio#19 fix, so all changes build togetherOnce ROCm/triton#957 and ROCm/audio#19 merge, repoint the pins/
related_commitsat the merged SHAs and drop the[TEST ONLY]commits.https://rocm-ci.amd.com/view/Release-7.2/job/pyt_whl_docker_rel-7.2/967/
https://rocm-ci.amd.com/job/pytorch-manylinux-wheel-builder_rel-7.2/531/
Jira: https://amd-hub.atlassian.net/browse/AIPYTORCH-827