Add live remote training status and performance reporting

[iskandr]

Motivation

Remote pan-allele training should expose a concise live status view while a run is active. Today the pipeline writes useful pieces (release_driver.log, heartbeat snapshots, GPU occupancy CSVs in some stages, worker logs, and old jobs/ perf monitors), but there is no maintained command that turns those into an operator-friendly progress report.

Desired status shape:

Instance: RUNNING / READY
Active stage: processing
Active variant: with_flanks
Active cell: mb=512, lr=0.0005, held_out_samples=10
Progress: 166/512 models, 32%
Elapsed for this variant: ~2h18m
Projected finish for this variant: roughly 7h total at current pace

GPU utilization:
GPU0: 99%, 17.4 GiB
GPU1: 100%, 30.6 GiB
GPU2: 99%, 28.6 GiB
GPU3: 100%, 27.7 GiB

Proposed design

• Have the training pipeline write a structured run-status file, e.g. <run-dir>/status/current.json, updated atomically.
• Keep append-only event logs for history, but make the status file the stable machine-readable interface.
• Track at least:
  • backend and instance state when available (local, brev-existing, ssh, future brev-provision)
  • active stage (affinity, processing, presentation, calibration, eval, plots, deploy-validation)
  • active processing variant / hyperparameter cell where applicable
  • completed model count and total model count
  • elapsed time per stage/variant
  • moving-average model throughput and projected finish time
  • GPU utilization, memory, and power where nvidia-smi is available
  • recent error / stalled-worker signal if detected
• Add a CLI command, e.g. mhcflurry training-status --run-dir RUN_DIR, that prints the concise human-readable view and can also emit JSON.
• Make scripts/release/retrain_evaluate_deploy.sh and remote backends able to call or document this command without special-case log scraping.

Notes

The old ignored jobs/pan_allele_release_full_mb1024.py contains useful prior art: it sampled nvidia-smi, /proc/meminfo, and ps, then summarized GPU/system/process metrics. The maintained affinity stage also has heartbeat and GPU occupancy logs. This issue is to promote those ideas into maintained status reporting instead of keeping them as experiment-only instrumentation.

Acceptance criteria

• A running local or remote training run writes structured status under the run directory.
• mhcflurry training-status --run-dir RUN_DIR prints a compact status report suitable for periodically watching from a laptop.
• --json emits the same data for automation.
• GPU utilization appears when nvidia-smi is available and degrades cleanly otherwise.
• Progress and ETA are computed from model manifests / planned work items / event logs, not by parsing human prose logs.
• The status command works after rsyncing a remote run directory back locally.
• Documentation includes examples for local, brev-existing, and ssh runs.