fix(3.1.2): grandfather pre-3.0.43 datasets in dataset-size check#831
Merged
Merged
Conversation
…heck Datasets generated before the floor->ceil datagen alignment (commit f9d414d, first shipped in mlpstorage 3.0.43) were sized with integer floor division in rules/utils.py. Rule 3.1.2 in the submission checker now re-derives the threshold with ceil, so it rejects a valid pre-alignment dataset by exactly one file (a submitter hit "actual files 4710038 < minimum required 4710039" on a Jun-28 run made with 3.0.24). Relax the 3.1.2 threshold to ceil-minus-one for the v3.0 round. We can't just floor it: datagen floored via integer `//` while this check recomputes the same formula as a float chain, so float drift can push floor() one above the recorded count and still reject. ceil-1 equals floor() for non-integer ratios and grants one file of slack exactly at the integer boundary where that drift bites, absorbing the bounded <=1-file divergence. Stays int-vs-int (no "N < N" display bug) and still rejects genuinely undersized runs (short by >1 file). Validator-only change; rules/utils.py (datagen/runtime verifier) stays on ceil. Revert to plain ceil once the v3.0 round closes -- post-3.0.43 datagen ceils, so newer datasets pass either way.
|
MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A submitter re-validating against 3.0.46 is blocked on rule 3.1.2:
Their dataset was generated with mlpstorage 3.0.24 (Jun 28), before the floor→ceil datagen alignment in
f9d414d(first shipped in 3.0.43). Datagen back then sized datasets with integer floor division inrules/utils.py; the validator now re-derives the threshold with ceil, so it rejects a valid pre-alignment dataset by exactly one file. The submitter did nothing wrong — the official tool sized the dataset and reported it compliant.Fix
Relax the rule 3.1.2 threshold from
ceiltoceil - 1for the v3.0 round (validator only;rules/utils.pydatagen/runtime verifier stays onceil).Why
ceil()-1and notfloor()Datagen floored via integer
//(exact), while this check recomputes the same formula as a float chain — so float drift can pushfloor()one above the recorded count and still reject the dataset we're grandfathering.ceil-1:floor()for non-integer ratios (no change in the common case),Testing
All four CI suites pass (2938 + 890 + 238 + 228, zero failures). No assertion flipped, including
test_true_undersized_submission_still_fails_3_1_2(short by ~50k files, far beyond the 1-file relaxation).Follow-up
Revert to plain
math.ceilonce the v3.0 round closes — post-3.0.43 datagen ceils, so newer datasets pass either way. Tracked inline in the code comment.🤖 Generated with Claude Code