Skip to content

False 4.7.1 inter-phase gap violation in reportgen #825

Description

@wolfgang-desalvador

Summary

mlpstorage reports reportgen emits a spurious CLOSED violation for
two-invocation checkpointing submissions:

Gap between write-phase end and read-phase start is <gap>s, exceeding the
30s maximum required by Rules.md §4.7.1.

even when the invocation bookends (invocation_start_time /
invocation_end_time) show the real quiet window is well under 30 seconds.

This is the same class of issue as the earlier submission-checker fixes
(storage #714 / #782 and the 2.1.24 checkpointingTimestampGap fix): the gap is
being measured from the wrong timestamps.

Root cause

A CLOSED checkpointing submission (per Rules.md §4.7.1) can consist of two
invocations — a write phase and a read phase — separated by a short quiet
window (cache flush). Two different timestamp sources exist per invocation:

  • DLIO summary start / end — the DLIO benchmark loop bookends. These
    include read-side framework startup (Python imports, MPI spawn,
    env-validation) and write-side post-benchmark cluster collection.
  • metadata.invocation_start_time / metadata.invocation_end_time — the
    mlpstorage invocation bookends. These exclude that startup/collection
    overhead and represent the true quiet window between phases.

The submission checker was already corrected to measure the gap from the
invocation bookends:

  • submission_checker/checks/directory_checks.py — rule 2.1.24
  • submission_checker/checks/checkpointing_checks.py — rule 4.7.1
    (cache_flush_validation)

But the report generator validates through a separate code path
(BenchmarkVerifierCheckpointSubmissionRulesChecker.check_invocation_structure
in rules/submission_checkers/checkpointing.py), and that checker still
measured the gap from the DLIO summary fields:

  • write_end = ordered[0].end_datetimesummary["end"]
  • read_start = ordered[1].run_datetimesummary["start"]

rules/models.py populates BenchmarkRunData.run_datetime / end_datetime
from the summary start / end and never captured the invocation bookends, so
the reportgen checker had no access to them.

Behavior

Measurement Source Result
Fixed submission checker read.invocation_start_time − write.invocation_end_time small gap → PASS
reportgen (buggy) read.summary.start − write.summary.end inflated gap → FAIL

The difference between the two is exactly the overhead the invocation bookends
were introduced to exclude:

  • read-side framework startup charged as "gap"
  • write-side post-benchmark cluster collection charged as "gap"

The effect grows with node count: on large topologies the fixed startup and
cluster-collection overhead can dwarf the actual quiet window, so the buggy
summary-based gap easily exceeds the 30-second bound and trips the false
violation. The larger the node count, the more pronounced the discrepancy.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions