Megatron CI improvement

[sudhu2k]

Problem

The single Build and Test on GPU job has been flaky and operationally painful:

• Jobs hang for 10+ hours and get cancelled. A single RCCL/NCCL collective deadlock would hang torchrun forever, and the only timeout was the 12h job-level timeout-minutes: 720, so one stuck test stalled the entire run until it was force-cancelled.
• Logs are enormous. set -x traced every shell command, and pytest ran with --showlocals --tb=long -v -s, dumping full local tensors, long tracebacks, and uncaptured per-rank stdout (including RCCL chatter). The reporting step also ran ls -R over the whole repo.
• No flaky-test mitigation. A single transient failure failed the whole job.
• The test report silently showed all-green on crashes. dorny/test-reporter only reads the JUnit XML. On a hard exit (SIGTERM from a timeout, segfault, RCCL abort, OOM, worker death), pytest never finishes writing the XML, so crashed files simply disappeared from the report even though the job failed.
• The report check often didn't publish at all because the workflow had no permissions: block for checks: write.

Changes

run_unit_tests.sh

• Removed set -x; kept set -u -o pipefail. Added export NCCL_DEBUG=WARN to cut RCCL/NCCL noise.
• Added a hard per-file wall-clock cap via coreutils timeout (--signal=SIGTERM --kill-after=60, default 1200s/20 min) so a deadlocked file is killed and the loop moves on instead of hanging the job.
• Added a graceful in-pytest timeout (--timeout=900 --timeout-method=thread) that fires first and produces a traceback before the hard kill.
• Redirected all verbose pytest output to a per-file log (output/pytest_<name>.log); the console now prints a single concise line per file ([PASS]/[FAIL]/[TIMEOUT] + pytest's one-line summary).
• Slimmed pytest flags: dropped --showlocals, -v, -s; added --tb=short --capture=fd.
• Added flaky retries: --reruns 2 --reruns-delay 5.
• Added write_crash_xml: when a run dies hard (exit code not 0/1) or the JUnit XML is missing/empty, a valid synthetic JUnit file is emitted so the crash shows up as a red entry in the report (with exit code, a human label like SIGTERM/segfault/timeout, and the last 50 log lines).
• Reap stray pytest/torchrun processes between files to avoid leaking GPU memory after a kill.

Dockerfile_rocm.ci

• Added pytest-timeout and pytest-rerunfailures to the pip install block.

.github/workflows/megatron-ci.yml

• Added a top-level permissions: block (contents: read, checks: write, pull-requests: write) so the test report check can publish.
• Bumped dorny/test-reporter @v1 → @v3 and set collapsed: auto (collapse on all-pass, auto-expand on failure).
• Upload output/pytest_*.log as artifacts alongside the JUnit/CSV reports.
• Replaced the whole-workspace ls -R with ls -R output.
• Lowered timeout-minutes from 720 to 240.

Impact

• A hung test can no longer run the job to the 12h limit; worst case is bounded by the per-file cap.
• Console logs are dramatically smaller; full detail remains in per-file log artifacts and the test report.
• Transient failures auto-retry instead of failing the run outright.
• Crashes and timeouts now appear as failures in the report instead of silently showing green.