Fix GitHub Actions CI: unblock auto-cancelled runs, improves test gating, adds checkpointing fixes for older JAX - #339
Merged
Conversation
copybara-service
Bot
force-pushed
the
test_951479789
branch
from
July 21, 2026 16:25
6cb0e98 to
afe07cf
Compare
…ing, adds checkpointing fixes for older JAX
## Summary
The GitHub actions test suite passes on internal infrastructure but has never gone green on GitHub. This PR fixes two root causes:
1. A poisoned Actions concurrency group was auto-cancelling all runs on `main` since mid-June.
2. CPU-only GitHub runners failed on tests requiring TPU capabilities (int4, Pallas, `scaled_matmul` lowering, multi-device shardings, and TPU-specific fp8 numerics).
After these changes, the CI-equivalent suite passes locally on CPU (except two einsum tests blocked by an XLA:CPU bug). No TPU-side behavior or tolerance is changed.
## 1. CI Workflows
Re-keyed concurrency groups to prevent pushes to `main` from being starved or cancelled:
* **Pushes to `main`:** Unique group per run (`github.run_id`) — never queued or cancelled.
* **PR runs:** Shared per-PR group — superseded commits are cancelled.
```yaml
concurrency:
group: ${{ github.workflow }}-${{ github.event_name == 'pull_request' && format('pr-{0}', github.event.pull_request.number) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
```
## 2. Test Gating
We apply targeted gating for TPU only-tests
| Test File | Issue on CPU | Fix / Resolution |
| :--- | :--- | :--- |
| **`core/mxfp_dot_test.py`** | `scaled_matmul` MLIR lowering is missing in released PyPI JAX. | Gates the 4 affected tests using a dynamic runtime capability probe (`_scaled_matmul_supported()`). Tests auto-enable once JAX CPU gains lowering.
| **`kernels/fused_hadamard_quantize_test.py`** | TPU-only Pallas kernel requires `CpuDevice` unsupported attributes during setup. | Added class-level `setUp` skip on non-TPU platforms. |
| **`core/conv_general_qt_test.py`** | Dilated fp8-backward conv (`fp8_bwd_nhwc_dilated`) diverges deterministically on XLA:CPU. | Skip only this specific parameterization when off-TPU. |
| **`core/dot_general_qt_test.py`** | Untiled `fp8_bwd` grads slightly exceed the strict 0.04 TPU-tuned bound on CPU. | Use a platform-conditional bound of `0.25` off-TPU (TPU keeps `0.04`). |
## 3. Checkpoint Sharding (`utils/checkpoint_util.py`)
Introduced a library-level fix to resolve sharding loss on released JAX versions.
| Component | Details |
|---|---|
| Symptom | 2 tests building `NamedSharding` over `Mesh('fsdp': 2, 'tp': 2)` with 4 virtual CPU devices passed quantized-param assertions, but the bias landed as `SingleDeviceSharding`. |
| Root Cause | On released JAX, `nnx.eval_shape` yields template leaves whose unboxed `ShapeDtypeStruct` has `sharding=None`. Because of this, `_apply_sharding_and_dtype` skipped calling `jax.device_put`, leaving fp params on a single device (at JAX-at-head they carry `NamedSharding` over an `AbstractMesh`). |
| **Resolution** | Implemented a metadata fallback. When the primary template-sharding resolution yields `None` (and `use_checkpoint_sharding=False`), `_apply_sharding_and_dtype` reconstructs the sharding from the boxed template's `nnx` metadata via Flax's `nnx.spmd.get_var_pspec` against the active concrete mesh. |
| Verification | Verified on JAX nightly where the primary path resolves every sharding and the helper is never invoked. Passes on both JAX `0.9.0.1` and JAX nightly (`0.11.0.dev20260618`). |
## 4. Type Checks: `qwix/contrib/kernels/`
* Fixed `pyrefly check` errors on `pl.kernel(...)` calls.
* Kernels are written against the newer JAX-at-head API (using `out_type`, `scratch_types`, `mesh`), which differs from released JAX signatures on CPU runners.
PiperOrigin-RevId: 951532859
copybara-service
Bot
force-pushed
the
test_951479789
branch
from
July 21, 2026 16:26
afe07cf to
8d0dddb
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix GitHub Actions CI: unblock auto-cancelled runs, improves test gating, adds checkpointing fixes for older JAX
Summary
The GitHub actions test suite passes on internal infrastructure but has never gone green on GitHub. This PR fixes two root causes:
mainsince mid-June.scaled_matmullowering, multi-device shardings, and TPU-specific fp8 numerics).After these changes, the CI-equivalent suite passes locally on CPU (except two einsum tests blocked by an XLA:CPU bug). No TPU-side behavior or tolerance is changed.
1. CI Workflows
Re-keyed concurrency groups to prevent pushes to
mainfrom being starved or cancelled:main: Unique group per run (github.run_id) — never queued or cancelled.2. Test Gating
We apply targeted gating for TPU only-tests
core/mxfp_dot_test.pyscaled_matmulMLIR lowering is missing in released PyPI JAX._scaled_matmul_supported()). Tests auto-enable once JAX CPU gains lowering.kernels/fused_hadamard_quantize_test.pyCpuDeviceunsupported attributes during setup.setUpskip on non-TPU platforms.core/conv_general_qt_test.pyfp8_bwd_nhwc_dilated) diverges deterministically on XLA:CPU.core/dot_general_qt_test.pyfp8_bwdgrads slightly exceed the strict 0.04 TPU-tuned bound on CPU.0.25off-TPU (TPU keeps0.04).3. Checkpoint Sharding (
utils/checkpoint_util.py)Introduced a library-level fix to resolve sharding loss on released JAX versions.
NamedShardingoverMesh('fsdp': 2, 'tp': 2)with 4 virtual CPU devices passed quantized-param assertions, but the bias landed asSingleDeviceSharding.nnx.eval_shapeyields template leaves whose unboxedShapeDtypeStructhassharding=None. Because of this,_apply_sharding_and_dtypeskipped callingjax.device_put, leaving fp params on a single device (at JAX-at-head they carryNamedShardingover anAbstractMesh).None(anduse_checkpoint_sharding=False),_apply_sharding_and_dtypereconstructs the sharding from the boxed template'snnxmetadata via Flax'snnx.spmd.get_var_pspecagainst the active concrete mesh.0.9.0.1and JAX nightly (0.11.0.dev20260618).4. Type Checks:
qwix/contrib/kernels/pyrefly checkerrors onpl.kernel(...)calls.out_type,scratch_types,mesh), which differs from released JAX signatures on CPU runners.