reportgen and validate cannot populate result tables from submission logs

[paglimo]

Summary

mlpstorage reports reportgen and mlpstorage validate --csv currently do not reliably populate user-facing result tables from an existing submission tree.

In the observed case, the submission logs were discovered and validated, and reportgen produced workload-level rollup files. However, the final CSVs intended for reporting/summary use were either partially populated with internal metric fields or entirely blank, making it difficult to obtain the final benchmark result table from submission logs alone.

Observed Behavior

Running report generation on a canonical submission tree discovers the runs and writes result files:

mlpstorage reports reportgen --results-dir /path/to/submission

The tool reports that it found benchmark runs and writes files such as:

/path/to/submission/closed/<submitter>/results/<system>/training/<model>/run/results.csv
/path/to/submission/closed/<submitter>/results/<system>/training/<model>/run/results.json
/path/to/submission/closed/<submitter>/results/results.csv
/path/to/submission/closed/<submitter>/results/results.json

The top-level results.csv contains rows and non-empty metric values, but the schema is reportgen-internal, for example:

category,orgname,systemname,benchmark_type,model,accelerator,train_mean_of_au_percentage,train_mean_of_throughput_samples_per_second,...

Separately, running validation with CSV output:

mlpstorage validate /path/to/submission \
  --csv /path/to/submission/summary.csv

creates a public-summary-style CSV with headers such as:

Public ID,Organization,Submission Name,Description,Type,Access Protocol,Availability,RUs,Integrated Client Storage,Accelerator Type,# Client Nodes,...

but all data rows are empty strings. In the observed case, the CSV had multiple data rows, but zero non-empty data cells.

Why This Appears to Happen

The validator CSV exporter appears to be a placeholder.

In mlpstorage_py/submission_checker/results.py, ResultExporter.add_result() initializes an empty row and appends it without extracting any fields from the submission logs:

def add_result(self, submission_logs: SubmissionLogs):
    row = {key: "" for key in self.head}
    # TODO: extract values from submission logs
    self.rows.append(row.copy())

The validator main loop calls this exporter for valid workloads:

# TODO: Add results to summary
if valid:
    exporter.add_result(logs)

As a result, validate --csv can successfully validate the submission and still generate a CSV whose rows are entirely blank.

There also appears to be a schema mismatch between the report generator output and the validator summary output:

• reportgen emits internal fields such as:

  • train_mean_of_au_percentage
  • train_mean_of_throughput_samples_per_second
  • kvcache_*
  • workload-specific flattened metric names

• validate --csv expects public summary fields such as:

  • 3D-Unet - Read B/W (GiB/s)
  • 8B - Write B/W (GiB/s)
  • 70B - Read B/W (GiB/s)

There does not appear to be a completed mapping layer from submission logs / reportgen rollups into the public summary table.

Expected Behavior

Given a completed canonical submission tree, the tools should be able to populate the final result table from submission logs.

At minimum, one of the following should be true:

1. reports reportgen should generate a final public-facing benchmark result table.
2. validate --csv should populate its public summary CSV from submission logs and/or reportgen rollups.
3. If the CSV summary path is not implemented yet, the tool should explicitly warn or fail rather than generating rows with all empty values.

Suggested Direction

A possible fix would be to implement a shared result-table extraction layer that:

• reads validated SubmissionLogs
• extracts system metadata from systems/<system>.yaml
• extracts benchmark metrics from run summaries or reportgen rollups
• maps internal metric fields to public result-table columns
• is shared by reportgen and validate --csv, or clearly separates their schemas in documentation

This would avoid the current situation where reportgen and validate both process the submission successfully, but neither directly produces a fully populated final public result table from the submission logs.

[FileSystemGuy]

@paglimo Unfortunately, the --csv argument to mlpstorage validate is a leftover from a prior era and is not supported. The supported path to generating a summary of your results is "mlpstorage reports reportgen".

Thank you for filing this Issue! It had the very happy effect of asking me to look closer at the reportgen code, which lead me to both fix some bugs and enhance it quite a bit. I will not PR the changes I'm working on into main yet so I don't want it to bring any risk of destabilizing anyone's last days of runs, but you are welcome to run the code on it's development branch against your <results-dir> to both see a summary of your results and to help by creating any Issues you see while running it!

git clone --branch reportgen-column-parity --single-branch https://github.com/mlcommons/storage

That branch has much better support for extracting all the right info from all the right json files stored by the workload generators (DLIO, VDB, and KVCache) and generating the <results-dir>/results.{csv,json} files. Specifically, the output .csv file should exactly match the columns in the Results Table Structure.

If you do run that code, please tell me (or file Issues) if any of the columns doesn't look like what you expect it to look like, if it doesn't accurately represent the results you think you got from the benchmark. We will do this more formally at the end of the peer review process, but I'd really like to know this code is working perfectly before then.

[junjiangwu]

@FileSystemGuy Thanks for the clarification and help. We tested the latest reportgen as suggested and found a VectorDB-related issue.

reports reportgen groups VectorDB datasize, datagen, and run directories into the same workload, then reads metrics from runs[0], which can be a non-run directory, causing VDB QPS/latency/recall fields to be blank.

Environment / code version:

Repository: mlcommons/storage
Branch/tag: reportgen-column-parity
mlpstorage version: 653ef88703b3ec8ec2d48581eb37c0c9aec0c12a

Command used:

./mlpstorage reports reportgen -rd /root/Suzhou_Zishan_Longlin

Observed output problem:

The report is generated, but the VectorDB row has only identity fields populated, while query metrics are blank.

For example, vector_database/milvus/DISKANN/results.csv contains values for:

VDB - Vector Count
VDB - Vector Dimension
VDB - Index Type

but these fields are empty:

VDB - Queries per Sec
VDB - Query Latency (ms)
VDB - Recall Percentage
VDB - Read B/W (GiB/s)

The raw run data does contain these metrics. For example, under:

vector_database/milvus/DISKANN/run/<timestamp>/statistics.json

there are fields such as:

{
  "throughput_qps": 7045.832447836199,
  "mean_latency_ms": 7.3530224407536275,
  "p95_latency_ms": 8.0702900886535,
  "recall": {
    "mean_recall": 0.4009
  }
}

Code-level root cause:

In mlpstorage_py/report_generator.py, _derive_workload_key() groups VectorDB only by:

(category, orgname, systemname, engine, index_type)

It does not distinguish command, so the following directories are grouped together:

vector_database/milvus/DISKANN/datasize/<timestamp>
vector_database/milvus/DISKANN/datagen/<timestamp>
vector_database/milvus/DISKANN/run/<timestamp>

Then _aggregate_vdb() uses:

run = runs[0]
summary = self._load_vdb_summary(run)

If runs[0] is the datasize or datagen directory, there is no summary.json, statistics.json, or recall_stats.json, so the VDB metric fields become None / blank even though the real run//statistics.json files contain the data.

Suggested fix:

For VectorDB aggregation, reportgen should select command == "run" entries when reading query metrics, instead of blindly using runs[0].

It may also need to collect identity fields such as num_vectors, dimension, vdb_engine, and vdb_index from all available VDB metadata files, because some identity fields may be present in datasize metadata while query metrics are present in run outputs.

[FileSystemGuy]

@junjiangwu Please do a git pull on the reportgen-column-parity branch, that will contain the fix.