2nd: feat(validate): warnings-only submission-flaw checks — 3.3.7 node capability, 4.7.4 simultaneous R/W#827
Draft
FileSystemGuy wants to merge 4 commits into
Draft
Conversation
Replaces the misleading 'satisfied by construction' INFO for an
explicitly-declared simultaneous_{write,read}=false with a warn_violation:
a completed checkpointing run demonstrates simultaneous R/W via CAP-02, so
a false declaration contradicts the run and its system description is
likely wrong. Warnings-only — always returns valid. Missing fields remain
silent-skipped (SystemYamlSchemaCheck owns them, no double-emit).
) Rules.md 3.3.7 explicitly wants a warning (not a failure) when physical nodes differ widely in capability. New tests: >1.5x per-host memory or CPU divergence warns; uniform hosts and no-cluster-information stay silent; collector host_consistency_issues are surfaced. Adds a build_submission run_metadata_cluster_information knob to inject per-host vectors.
Implements Rules.md 3.3.7 as a warnings-only check: reads each run's metadata['cluster_information'] per-host memory / CPU-core vectors and warns (never fails) when the max/min ratio exceeds 1.5x, or when the cluster collector recorded host_consistency_issues. No cluster_information or a single-host cluster is silently passed. Replaces the unconditional 'satisfied by construction' INFO.
|
MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Turns two stubbed submission-checker rules into real checks that surface submission flaws to reviewers as warnings — never as hard failures. Both follow the submission-window doctrine: they call
warn_violation(...)and still return valid, so a flagged submission is reviewable but never blocked.This is the first slice of the deferred validate-check backlog (the
[CHECK]items). Two more clusters were scoped and intentionally deferred — see "Deferred" below.Checks
① 3.3.7
trainingNodeCapabilityConsistency(training_checks.py)Rules.md 3.3.7 explicitly asks the validator to warn (not fail) when the physical nodes in a distributed submission differ widely in capability. The check reads each run's
metadata['cluster_information']per-host memory / CPU-core vectors and warns when the max/min ratio exceeds 1.5×, or when the cluster collector already recordedhost_consistency_issues. Nocluster_information, or a single-host cluster, is silently passed (nothing to compare). Replaces the previous unconditional "satisfied by construction" INFO.② 4.7.4
checkpointSimultaneousRwSupport(checkpointing_checks.py)A completed checkpointing run has already demonstrated simultaneous read/write on the shared namespace via the CAP-02 runtime probe. So a system description that declares
simultaneous_writeorsimultaneous_readasfalsecontradicts the run it accompanies. The check now warns on that inconsistency (naming the offending field) instead of mislabelling it "satisfied by construction". Missing capability fields stay silent-skipped —SystemYamlSchemaCheckowns the missing-field violation (D-A3), so warning here too would double-emit.Notes for reviewers
1.5×divergence threshold for 3.3.7 is a documented, tunable constant (_NODE_CAPABILITY_DIVERGENCE_RATIO) — happy to adjust if there's a spec-preferred value.build_submission(run_metadata_cluster_information=…)test knob was added to inject per-host capability vectors.Deferred (scoped, not in this PR)
config.pyyet.pool_structure_checks.pyCHECK-02 across thecode-<hash8>pool; the only missing piece is comparison againstREFERENCE_CHECKSUMS, which is currently all-None("not yet pinned"), so it can't produce a real signal until the v3.0 reference digests are pinned.Testing
All four CI suites pass locally:
tests/mlpstorage_py/testsvdb_benchmark/testskv_cache_benchmark/testsEach check landed RED→GREEN.
Stacked on #826 (reportgen final-results-tables). Base will retarget to
mainonce #826 merges; this PR's diff is independent of it.