Summary
mlpstorage reports reportgen emits a spurious CLOSED violation for
two-invocation checkpointing submissions:
Gap between write-phase end and read-phase start is <gap>s, exceeding the
30s maximum required by Rules.md §4.7.1.
even when the invocation bookends (invocation_start_time /
invocation_end_time) show the real quiet window is well under 30 seconds.
This is the same class of issue as the earlier submission-checker fixes
(storage #714 / #782 and the 2.1.24 checkpointingTimestampGap fix): the gap is
being measured from the wrong timestamps.
Root cause
A CLOSED checkpointing submission (per Rules.md §4.7.1) can consist of two
invocations — a write phase and a read phase — separated by a short quiet
window (cache flush). Two different timestamp sources exist per invocation:
- DLIO summary
start / end — the DLIO benchmark loop bookends. These
include read-side framework startup (Python imports, MPI spawn,
env-validation) and write-side post-benchmark cluster collection.
metadata.invocation_start_time / metadata.invocation_end_time — the
mlpstorage invocation bookends. These exclude that startup/collection
overhead and represent the true quiet window between phases.
The submission checker was already corrected to measure the gap from the
invocation bookends:
submission_checker/checks/directory_checks.py — rule 2.1.24
submission_checker/checks/checkpointing_checks.py — rule 4.7.1
(cache_flush_validation)
But the report generator validates through a separate code path
(BenchmarkVerifier → CheckpointSubmissionRulesChecker.check_invocation_structure
in rules/submission_checkers/checkpointing.py), and that checker still
measured the gap from the DLIO summary fields:
write_end = ordered[0].end_datetime → summary["end"]
read_start = ordered[1].run_datetime → summary["start"]
rules/models.py populates BenchmarkRunData.run_datetime / end_datetime
from the summary start / end and never captured the invocation bookends, so
the reportgen checker had no access to them.
Behavior
| Measurement |
Source |
Result |
| Fixed submission checker |
read.invocation_start_time − write.invocation_end_time |
small gap → PASS |
reportgen (buggy) |
read.summary.start − write.summary.end |
inflated gap → FAIL |
The difference between the two is exactly the overhead the invocation bookends
were introduced to exclude:
- read-side framework startup charged as "gap"
- write-side post-benchmark cluster collection charged as "gap"
The effect grows with node count: on large topologies the fixed startup and
cluster-collection overhead can dwarf the actual quiet window, so the buggy
summary-based gap easily exceeds the 30-second bound and trips the false
violation. The larger the node count, the more pronounced the discrepancy.
Summary
mlpstorage reports reportgenemits a spurious CLOSED violation fortwo-invocation checkpointing submissions:
even when the invocation bookends (
invocation_start_time/invocation_end_time) show the real quiet window is well under 30 seconds.This is the same class of issue as the earlier submission-checker fixes
(storage #714 / #782 and the
2.1.24 checkpointingTimestampGapfix): the gap isbeing measured from the wrong timestamps.
Root cause
A CLOSED checkpointing submission (per Rules.md §4.7.1) can consist of two
invocations — a write phase and a read phase — separated by a short quiet
window (cache flush). Two different timestamp sources exist per invocation:
start/end— the DLIO benchmark loop bookends. Theseinclude read-side framework startup (Python imports, MPI spawn,
env-validation) and write-side post-benchmark cluster collection.
metadata.invocation_start_time/metadata.invocation_end_time— themlpstorage invocation bookends. These exclude that startup/collection
overhead and represent the true quiet window between phases.
The submission checker was already corrected to measure the gap from the
invocation bookends:
submission_checker/checks/directory_checks.py— rule2.1.24submission_checker/checks/checkpointing_checks.py— rule4.7.1(
cache_flush_validation)But the report generator validates through a separate code path
(
BenchmarkVerifier→CheckpointSubmissionRulesChecker.check_invocation_structurein
rules/submission_checkers/checkpointing.py), and that checker stillmeasured the gap from the DLIO summary fields:
write_end = ordered[0].end_datetime→summary["end"]read_start = ordered[1].run_datetime→summary["start"]rules/models.pypopulatesBenchmarkRunData.run_datetime/end_datetimefrom the summary
start/endand never captured the invocation bookends, sothe reportgen checker had no access to them.
Behavior
read.invocation_start_time − write.invocation_end_timereportgen(buggy)read.summary.start − write.summary.endThe difference between the two is exactly the overhead the invocation bookends
were introduced to exclude:
The effect grows with node count: on large topologies the fixed startup and
cluster-collection overhead can dwarf the actual quiet window, so the buggy
summary-based gap easily exceeds the 30-second bound and trips the false
violation. The larger the node count, the more pronounced the discrepancy.