[CI] Stabilize MORI overlap and mGPU tests

[leo-automation]

Summary

• Bind the default NCCL process group to each rank's GPU and explicitly clean up model-parallel/custom process groups between topology changes.
• Treat MORI shared memory as process-scoped, keep its reusable outputs private across overlapped launches, and only exercise valid node-spanning MORI EP layouts.
• Run chunk-schedule MORI coverage in a fresh torchrun file, group all four data variants under each TP topology lifetime, and make numerical failure collective so one rank cannot strand the others.
• Give the two measured long-running checkpoint files a 60-minute per-file timeout instead of the normal 30 minutes.

Root cause

There were two independent failures:

1. The original node-dependent GPU wedge followed the host amdgpu/KFD version, not the container. The investigation nodes were remediated to the stable 6.12.12 driver.
2. After that remediation, untouched rocm_dev still reproduced a MORI ShmemStates::CheckStatusValid() abort. MORI shared memory/operator state is process-scoped. The chunk-schedule file first ran 38 topology-changing non-MORI cases and then repeatedly tore down/recreated MORI state across 16 cases. On e04 this corrupted later MORI cases. A fresh process alone was insufficient; keeping one operator alive and running the four data variants under each TP topology lifetime was stable.

The experimental core barriers/synchronization attempted during diagnosis are not included because they did not fix the corruption.

Validation

Final exact commit 00c29b338 passed on both allowed 8-GPU nodes:

• quanta-ccs-aus-e04-01: exit 0, 21m10s
• dell300x-ccs-aus-e11-21: exit 0, 18m59s
• Base chunk schedule: 26 passed, 12 skipped per rank
• Isolated/grouped MORI schedule: 4 passed per rank, covering all 16 TP/padding/layer combinations
• Formerly hanging target files all passed: discrepancy (32), grouped MLP (3), A2A dispatcher (29), token dispatcher (7), MoE layer (63), and layer schedule (11) per rank
• The grouped 16-combination MORI file also passed five complete fresh-process repetitions on e04 during diagnosis
• py_compile, bash -n run_unit_tests.sh, and git diff --check passed

Earlier full-CI evidence while narrowing the fix:

• e04: https://github.com/ROCm/Megatron-LM/actions/runs/29968413569
• Dell: https://github.com/ROCm/Megatron-LM/actions/runs/29904875199

Both produced all 177 JUnit reports: 4,207 tests, 789 skipped, 0 failures, 0 errors. The PR's new full CI runs after the final targeted two-node validation above.