Skip to content

2nd: feat(validate): warnings-only submission-flaw checks — 3.3.7 node capability, 4.7.4 simultaneous R/W#827

Draft
FileSystemGuy wants to merge 4 commits into
reportgen-v3-final-results-tablesfrom
validate-flaw-warnings
Draft

2nd: feat(validate): warnings-only submission-flaw checks — 3.3.7 node capability, 4.7.4 simultaneous R/W#827
FileSystemGuy wants to merge 4 commits into
reportgen-v3-final-results-tablesfrom
validate-flaw-warnings

Conversation

@FileSystemGuy

Copy link
Copy Markdown
Contributor

Summary

Turns two stubbed submission-checker rules into real checks that surface submission flaws to reviewers as warnings — never as hard failures. Both follow the submission-window doctrine: they call warn_violation(...) and still return valid, so a flagged submission is reviewable but never blocked.

This is the first slice of the deferred validate-check backlog (the [CHECK] items). Two more clusters were scoped and intentionally deferred — see "Deferred" below.

Checks

① 3.3.7 trainingNodeCapabilityConsistency (training_checks.py)
Rules.md 3.3.7 explicitly asks the validator to warn (not fail) when the physical nodes in a distributed submission differ widely in capability. The check reads each run's metadata['cluster_information'] per-host memory / CPU-core vectors and warns when the max/min ratio exceeds 1.5×, or when the cluster collector already recorded host_consistency_issues. No cluster_information, or a single-host cluster, is silently passed (nothing to compare). Replaces the previous unconditional "satisfied by construction" INFO.

② 4.7.4 checkpointSimultaneousRwSupport (checkpointing_checks.py)
A completed checkpointing run has already demonstrated simultaneous read/write on the shared namespace via the CAP-02 runtime probe. So a system description that declares simultaneous_write or simultaneous_read as false contradicts the run it accompanies. The check now warns on that inconsistency (naming the offending field) instead of mislabelling it "satisfied by construction". Missing capability fields stay silent-skipped — SystemYamlSchemaCheck owns the missing-field violation (D-A3), so warning here too would double-emit.

Notes for reviewers

  • Warnings only, always valid. Neither check can flip a submission INVALID. The 1.5× divergence threshold for 3.3.7 is a documented, tunable constant (_NODE_CAPABILITY_DIVERGENCE_RATIO) — happy to adjust if there's a spec-preferred value.
  • No double-emit. 4.7.4 defers missing-field reporting to the schema check; 3.3.7 does not re-fire on absent data.
  • Existing "satisfied by construction" tests for both rules were updated to the new warning behavior; a build_submission(run_metadata_cluster_information=…) test knob was added to inject per-host capability vectors.

Deferred (scoped, not in this PR)

  • VDB scale / recall-minimum / query-count checks (5.1.1 / 5.3.2 / 5.3.3) — need threshold tables that don't exist in config.py yet.
  • Code-image upstream-identity checksum (3.6.1 / 5.6.1) — self-consistency is already enforced by pool_structure_checks.py CHECK-02 across the code-<hash8> pool; the only missing piece is comparison against REFERENCE_CHECKSUMS, which is currently all-None ("not yet pinned"), so it can't produce a real signal until the v3.0 reference digests are pinned.
  • KVCache §6 (17 rules) — a green-field checker, sized as its own effort.

Testing

All four CI suites pass locally:

Suite Result
tests/ 2966 passed, 1 skipped, 13 deselected
mlpstorage_py/tests 894 passed
vdb_benchmark/tests 228 passed
kv_cache_benchmark/tests 238 passed, 2 deselected

Each check landed RED→GREEN.


Stacked on #826 (reportgen final-results-tables). Base will retarget to main once #826 merges; this PR's diff is independent of it.

)

A completed checkpointing run demonstrates simultaneous R/W via the CAP-02
probe, so a system YAML declaring simultaneous_{write,read}=false
contradicts the run. Surface as a warning (never a failure); missing
fields stay silent-skipped (SystemYamlSchemaCheck owns them, no double-emit).
Replaces the misleading 'satisfied by construction' INFO for an
explicitly-declared simultaneous_{write,read}=false with a warn_violation:
a completed checkpointing run demonstrates simultaneous R/W via CAP-02, so
a false declaration contradicts the run and its system description is
likely wrong. Warnings-only — always returns valid. Missing fields remain
silent-skipped (SystemYamlSchemaCheck owns them, no double-emit).
)

Rules.md 3.3.7 explicitly wants a warning (not a failure) when physical
nodes differ widely in capability. New tests: >1.5x per-host memory or CPU
divergence warns; uniform hosts and no-cluster-information stay silent;
collector host_consistency_issues are surfaced. Adds a build_submission
run_metadata_cluster_information knob to inject per-host vectors.
Implements Rules.md 3.3.7 as a warnings-only check: reads each run's
metadata['cluster_information'] per-host memory / CPU-core vectors and
warns (never fails) when the max/min ratio exceeds 1.5x, or when the
cluster collector recorded host_consistency_issues. No cluster_information
or a single-host cluster is silently passed. Replaces the unconditional
'satisfied by construction' INFO.
@github-actions

Copy link
Copy Markdown

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

@FileSystemGuy FileSystemGuy changed the title feat(validate): warnings-only submission-flaw checks — 3.3.7 node capability, 4.7.4 simultaneous R/W 2nd: feat(validate): warnings-only submission-flaw checks — 3.3.7 node capability, 4.7.4 simultaneous R/W Jul 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant