Skip to content

ci: bump base image/runner to v26.4 and drop v26.3-only workarounds#896

Open
WangLingxun wants to merge 4 commits into
mainfrom
chore/ci-bump-primus-v26.4
Open

ci: bump base image/runner to v26.4 and drop v26.3-only workarounds#896
WangLingxun wants to merge 4 commits into
mainfrom
chore/ci-bump-primus-v26.4

Conversation

@WangLingxun

@WangLingxun WangLingxun commented Jul 21, 2026

Copy link
Copy Markdown
Collaborator

Upgrade the Primus runtime image from rocm/primus:v26.3 to v26.4 across CI Dockerfile/workflow, example launch scripts, benchmark scripts and docs, drop v26.3-only CI workarounds now that they're no longer needed, and fix a stale-container bug found while verifying the CI change. JAX/MaxText base image not upgraded this round.

Image bump

  • .github/workflows/ci.yaml & docker/Dockerfile: BASE_IMAGE -> v26.4, sync v26.3 comments
  • README/examples/docs/benchmark scripts: bump documented image tag to v26.4, release branch to release/v26.4, pip pin to primus==26.4.0
  • runs-on stays on the v26.3 self-hosted runner label (no v26.4 runner registered yet)

Workaround cleanup

  • Drop the v26.3-only "Purge stale TorchTitan install" step: v26.4 no longer bundles a torchtitan that shadows the submodule.
  • Drop the origami (rocm-libraries@223648a) install from Dockerfile and CI: this base image ships no origami, so primus_turbo falls back to heuristic kernel selection for MoE grouped-gemm. Left the pip install one-liner in a comment to add it back if ever needed.
  • aiter_deepbind_patches.py self-detects whether the RTLD_DEEPBIND workaround is still needed: skips installing it once transformer_engine.__version__ >= 2.14 (fixed in v26.4), still applies on v26.3 and older. Added unit tests for the gating logic.

CI fix

  • The unit-test container reuse check compared only the image tag string (docker.io/tasimage/primus:${IMAGE_TAG}), not content. On main, IMAGE_TAG is always latest, so after build-docker repushes under that tag, the check still saw an identical string and reused the stale container instead of pulling the new image. Fixed by pulling first and comparing the resolved image id instead of the tag name.

@WangLingxun
WangLingxun marked this pull request as draft July 21, 2026 06:21
- ci.yaml: BASE_IMAGE + runs-on v26.3 -> v26.4; drop the v26.3-only
  'Purge stale TorchTitan' step (v26.4 ships no torchtitan to shadow).
  Keep 'Install fixed origami' (v26.4 still ships no origami).
- Dockerfile: default BASE_IMAGE + comments v26.3 -> v26.4.
- docs/scripts: bump default image refs to v26.4, release branch to
  release/v26.4, and pip pin to primus==26.4.0.
@WangLingxun
WangLingxun force-pushed the chore/ci-bump-primus-v26.4 branch 2 times, most recently from a6e9036 to 90e0389 Compare July 21, 2026 09:24
- Dockerfile/ci.yaml: drop the origami install; base image ships none,
  so primus_turbo falls back to heuristic MoE grouped-gemm kernel
  selection.

- aiter_deepbind_patches.py: skip the RTLD_DEEPBIND hook once
  transformer_engine >= 2.14 (fixed in rocm/primus:v26.4); still
  applies on older images.

- Add unit tests for the new gating logic.
Container reuse compared only the image tag string, not content. On
main, IMAGE_TAG is always "latest", so after build-docker repushes
under that tag, the check still saw an identical string and reused the
stale container instead of pulling the new image.

Fix: pull first, then compare docker image inspect .Id (target) against
the container's docker inspect .Image (actual) instead of the tag name.
@WangLingxun
WangLingxun force-pushed the chore/ci-bump-primus-v26.4 branch from 62f555d to 11e1885 Compare July 23, 2026 10:49
@WangLingxun
WangLingxun marked this pull request as ready for review July 23, 2026 10:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant