4th: feat(reportgen): fixed webpage-parity schema for results.{csv,json}#829
Draft
FileSystemGuy wants to merge 5 commits into
Draft
4th: feat(reportgen): fixed webpage-parity schema for results.{csv,json}#829FileSystemGuy wants to merge 5 commits into
FileSystemGuy wants to merge 5 commits into
Conversation
…chema The v3.0 results web page is 8 tables sharing a System-Under-Test block. reportgen must emit ONE flat results.csv/json that a staff member reduces to any single table by deleting non-applicable columns. That requires a FIXED schema (exact reference columns + Division/Benchmark Type/Model discriminators), not the current data-driven column set. Pins the emitted-file contract: header is EXACTLY the 54-column fixed schema; JSON keys match in order; internal machine columns (category/ orgname/systemname/benchmark_type prefix, *_mean_of_* dynamic means, trailing issues) are gone; values cherry-picked into fixed columns (Division=CLOSED/OPEN, Model=display label, kvcache option N -> its fixed group). Fails against the current dynamic _ordered_fieldnames.
results.{csv,json} is now a FIXED contract — exactly the reference columns
for all 8 v3.0 tables (Results Table Structure.xlsx) plus 3 discriminator
columns (Division / Benchmark Type / Model), never data-driven. A staff
member opens results.csv in Excel and reduces it to any single webpage
table by deleting the workload blocks + discriminators that don't apply.
Approach: output projection, not a rewrite of aggregation. The internal
_aggregate_* helpers (train_mean_of_* / checkpoint_mean_of_* / vdb_* /
kvcache_* machine keys, and the empty-metric StatisticsError -> INVALID
gate) are untouched, so validity behavior is unchanged. A new _final_row()
projection runs inside write_json_file/write_csv_file and cherry-picks each
fixed column from the in-memory row; the dynamic *_mean_of_* means and the
trailing issues column are simply no longer emitted (output-only change).
Schema (54 cols): 13-col left edge (Public ID, Organization, Division,
Benchmark Type, Model, Name, Description, Type, Access Protocol,
Availability, RU's, Integrated Client Storage, Usable Capacity) then
workload-qualified Training/Checkpointing/VDB/KVCache blocks (shared metric
names are unique-per-column, so they carry a 'Training - ' etc. prefix).
- Division = category upper-cased; Model = reference display label
(llama3-1t -> 1250B), blank for kvcache/vdb (single-table workloads).
- Name shows the system name -> .yaml; Description shows 'PDF' -> .pdf.
- Code/Logs are per-workload (an OPEN row may ship its own code), routed
into the row's own block from the code-image pointer.
- KVCache option 1/2/3 -> the reference groups (llama3.1-8b Storage Only /
Storage + Mem / llama3.1-70b Storage Only).
- Removed the machine-key prefix (category/orgname/systemname/
benchmark_type/model/accelerator), the issues column, and the now-dead
_ordered_fieldnames helper.
Output-file tests migrated to the fixed-schema contract; internal-structure
tests (test_aggregation helper keys) unchanged. All 4 CI suites green.
|
MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅ |
…scores Two-invocation checkpoint submissions (Rules §2.1.23/§4.7.1: mandatory write→read split when checkpoint-per-node < 3x client RAM) blank all four score columns. Each phase dir carries only its own scalars (write dir: save_*; read dir: load_*), so _scalar_mean's all-summaries-must-be-numeric gate sees [4.64, None] and returns None. Adds three cases to TestCheckpointingFinalTableColumns: - split write→read populates all four (RED — reporter's Alluxio bug) - two combined invocations average per direction (locks averaging) - a metric missing from a phase that WAS configured to produce it stays a loud blank, never a silent partial-mean (locks the guard)
…t invocations
Two-invocation checkpoint submissions (Rules §2.1.23/§4.7.1 write→read
split, mandatory when checkpoint-per-node < 3x client RAM) blanked all four
score columns: the write dir carries only save_* and the read dir only
load_*, so the old all-summaries-must-be-numeric gate saw [4.64, None] and
returned None.
Replace _scalar_mean with _directional_mean: classify each invocation as a
write/read producer from its configured checkpoint.num_checkpoints_{write,
read} (CLOSED forces 10 or 0), then take the mean over the producing phase
only. The loud-failure guard (D-23 / PITFALLS #3: no silent partial-mean)
is preserved but scoped to the producing phase — a scalar missing from an
invocation that WAS configured to produce it stays a blank, never a partial
mean over the survivors. Falls back to a present-only mean when the phase
signal is absent (legacy / OPEN packages) so those rows don't regress.
Fixes the reporter's Alluxio CLOSED v3.0 case: llama3-8b/70b/405b now
populate Write/Read B/W and durations from their split timestamp dirs.
The VectorDB workload grouping key (D-06) is (category, orgname, systemname, engine, index_type) — it does not include `command`, so a workload's datasize/datagen/run leaves all group under one key and arrive at `_aggregate_vdb` as a single `runs` list in discovery order. Only the `run` leaf carries the native query-phase metrics (statistics.json / summary.json); the datasize and datagen leaves have none. `_aggregate_vdb` used `run = runs[0]`, so whenever a non-run leaf sorted first the QPS / latency / recall / read-B/W columns came out blank even though the run leaf's statistics.json held the real values (identity columns still populated because they live in every leaf's parameters). Select the `run` invocation for metrics and recall, falling back to runs[0] only when no run leaf is present (e.g. a datasize-only tree) so identity columns still populate. Resolve identity columns across all grouped leaves (run first) so a value recorded only on the datasize leaf (e.g. num_vectors / dimension) still fills in. Regression test: a [datasize, datagen, run] group now reads its metrics from the run leaf.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
4th (stacked on #828): fixed webpage-parity schema for
results.{csv,json}Stacked on top of
reportgen-validator-fixes(#828). Part of the reportgen review-tooling stack #826 ← #827 ← #828 ← this. Retarget base tomainonce the lower PRs land.What
reports reportgennow emitsresults.csv/results.jsonas a fixed contract — exactly the reference columns for all 8 v3.0 results-web-page tables (Results Table Structure.xlsx) plus three discriminator columns (Division,Benchmark Type,Model) so one flat file can carry every table's rows. The column set is never data-driven: a MLCommons staff member opensresults.csvin Excel and reduces it to any single web table by deleting the workload blocks + discriminator columns that don't apply.How
Output projection, not an aggregation rewrite. The internal
_aggregate_*helpers (train_mean_of_*/checkpoint_mean_of_*/vdb_*/kvcache_*machine keys, and the empty-metricStatisticsError→ INVALID gate) are untouched, so validity behavior is unchanged. A new_final_row()runs insidewrite_json_file/write_csv_fileand cherry-picks each fixed column from the in-memory row. The dynamic*_mean_of_*per-metric means and the trailingissuescolumn are simply no longer emitted — this is an output-only change with no path from pass to fail.Schema (54 columns)
Public ID,Organization,Division,Benchmark Type,Model,Name,Description,Type,Access Protocol,Availability,RU's,Integrated Client Storage,Usable Capacity (TiB).Training - …,Checkpointing - …,VDB - …,KVCache - …. Shared reference names (# Client Nodes,Code,Read B/W (GiB/s), …) are workload-qualified so every column key is unique.Division= category upper-cased (CLOSED/OPEN);Model= reference display label (llama3-1t→1250B), blank for kvcache/vdb (single-table workloads).Nameshows the system name and links tosystem-description.yaml;DescriptionshowsPDFand links to the.pdf.Code/Logsare per-workload (an OPEN row may ship its own code), routed into the row's own block from the run's code-image pointer.1/2/3map to the reference groupsllama3.1-8b Storage Only/Storage + Mem/llama3.1-70b Storage Only.category/orgname/systemname/benchmark_type/model/accelerator), theissuescolumn, and the now-dead_ordered_fieldnameshelper.Tests
RED-first. Output-file tests migrated to the fixed-schema contract; internal-structure tests (aggregation helper keys) unchanged. All 4 CI suites green:
tests/2975p·1skip,mlpstorage_py/tests898p,vdb_benchmark/tests228p,kv_cache_benchmark/tests238p.