Test fixtures for training & checkpointing use fabricated DLIO metric keys real DLIO never emits

[FileSystemGuy]

Summary

The training and checkpointing summary.json test fixtures use metric keys that real DLIO never emits. Tests therefore validate reportgen/aggregation against a metric shape that does not exist in production, and the _aggregate_* code paths grow columns that can never populate from a real run.

Evidence

Real DLIO output (dlio_benchmark/utils/statscounter.py, DLIO_local_changes):

• Training (statscounter.py:201-205):
  • train_throughput_samples_per_second — list
  • train_io_mean_MB_per_second — scalar (np.mean(train_throughput)*record_size/1024/1024; logged "MiB/second" — the MB label is MiB)
  • train_io_stdev_MB_per_second — scalar
• Checkpointing (statscounter.py:178-185):
  • save_checkpoint_io_mean_GB_per_second, save_checkpoint_duration_mean_seconds (+ _stdev_) — scalars
  • load_checkpoint_io_mean_GB_per_second, load_checkpoint_duration_mean_seconds (+ _stdev_) — scalars

What the fixtures contain instead:

• Training fixtures — metric keys: train_au_percentage, train_throughput_samples_per_second, train_io_throughput_MB_per_second (a list key that does not exist in real DLIO — real is the scalar train_io_mean_MB_per_second).
• Checkpointing fixtures — metric keys: checkpoint_read_throughput_GB_per_second, checkpoint_write_throughput_GB_per_second (list keys; real DLIO emits save/load scalar means + duration means, with no read/write throughput lists).

Affected fixtures/helpers (non-exhaustive):

• tests/fixtures/sample_results/training/**/summary.json (and multi_orgname/, degenerate/ trees)
• tests/fixtures/sample_results/checkpointing/**/summary.json
• tests/conftest.py checkpointing fixture builder
• referenced in tests/unit/test_aggregation.py, tests/unit/test_rules_extractors.py

Impact

1. Coverage gap. The reportgen aggregation and rules extractors are exercised against a metric layout production never produces, so bugs specific to the real DLIO key names/shapes can slip through.
2. Dead aggregation columns. _aggregate_training / _aggregate_checkpointing build train_mean_of_* / checkpoint_mean_of_* columns by iterating the list metric keys present. Against the fabricated fixtures this yields train_mean_of_io_throughput_MB_per_second and checkpoint_mean_of_read/write_throughput_GB_per_second — columns that can never populate from a real run because the source keys don't exist.
3. Trap for future work. The reportgen v3.0 final-table slices had to source the real metrics (Read B/W, checkpoint bw/durations) from the real scalar keys and ship self-contained realistic summary.json in their new tests, precisely because the shared fixtures were unrealistic. Anyone reusing the shared fixtures inherits the mismatch.

Note: the fixed webpage-parity results.{csv,json} schema no longer emits the dead *_mean_of_* columns, so this is not a user-visible output bug today — it is test-fidelity / tech-debt that should be corrected so fixtures reflect real DLIO.

Proposed fix

Regenerate the training & checkpointing summary.json fixtures (and the conftest.py builder) to use the real DLIO metric keys:

• Training: replace the fabricated train_io_throughput_MB_per_second list with the real scalars train_io_mean_MB_per_second / train_io_stdev_MB_per_second, keeping the real train_throughput_samples_per_second list and train_au_percentage.
• Checkpointing: replace checkpoint_{read,write}_throughput_GB_per_second with the real save/load scalars *_io_mean_GB_per_second and *_duration_mean_seconds (+ stdevs).

Update test_aggregation.py / test_rules_extractors.py expectations accordingly. Decide whether to keep the *_mean_of_* aggregation path at all (it aggregates list metrics; with the real scalar-only checkpoint/IO keys there are fewer list metrics to aggregate).