Clarification on client host memory requirement for 1T model checkpointing – changed between v2.0 and v3.0?

[litianqi00315]

Hi MLCommons Storage team,

I have a question regarding the client host memory requirement for the checkpointing benchmark, specifically for the 1T model.

According to the current README (https://github.com/mlcommons/storage/tree/main/checkpointing), Table 2 shows that the checkpoint size for the 1T model is 18 TB. The README also states:

"Note: the aggregate simulated accelerator memory across all nodes must be sufficient to accommodate the model’s checkpoint size."

Based on this, I previously understood that the total client host memory across all nodes must be at least 18 TB to run the 1T checkpointing benchmark.

However, when I actually ran the benchmark this year (presumably with the MLPerf Storage v3.0 codebase), I found that the benchmark ran successfully even though the total client host memory was less than 18 TB. This was not possible with the v2.0 codebase, where the benchmark would fail if the memory was insufficient.

This leads me to believe that there may have been a change between v2.0 and v3.0 regarding how the client host memory requirement is enforced. Could you please clarify:

Has the memory requirement been relaxed or removed in v3.0?

If so, what is the new rule or validation logic for client host memory?

Is the --client-host-memory-in-gb parameter still strictly validated against the checkpoint size, or is it now only used for cache clearing decisions (e.g., the "3× memory" rule mentioned in the docs)?

I would appreciate any clarification on this. Thank you!

[FileSystemGuy]

@russfellows please confirm that I'm getting the details right.

Yes, the codepath in v3.0 has been optimized (by Russ) to require MUCH less memory. In the v2.0 codebase, for an 18TB checkpoint the DLIO code allocated a sum total of 18TB of DRAM, filled those buffers with random data and then wrote them to storage. In v3.0, the codebase allocates a couple of MB on each host, fills it with random data, writes it out, then refills the buffer and writes it out. It can do it that way because of the highly optimized random data generator that Russ wrote.

The rules do not require 18TB of RAM, the "Note: the aggregate simulated accelerator memory across all nodes must be sufficient to accommodate the model’s checkpoint size." text was a warning to the user rather than a rule that we would check for adherence to.

I don't have any guidance on how much memory the new version of the code will use, and therefore how many processes will fit into a typical client with 256GB of RAM, you may have to just try it a few times. Perhaps @russ has some insight he can pass along.

[russfellows]

To answer the original question, and to add to what Curtis has stated, here is what I know.

• Yes, the memory requirement has been DRASTICALLY reduced in 3.0, because we now perform "streaming writes"
• This means several data buffers are allocated, filled with random data, then written, with new buffers being re-filled after they have been written
• What this means is that at no time do we utilize more than 8 GB of RAM when creating checkpoints, regardless of the checkpoint size

So technically, this means you could create a 100 TB checkpoint from a single system with only 8 GB of RAM. Which brings us back to the rule.

The rule states, "You must have a TOTAL memory capacity equivalent to the TOTAL checkpoint size." Rather than being a technical limitation, this is now a rule, which must be checked.

[FileSystemGuy]

@russfellows @idevasena @dslik

My reading and memory is:
The rules do not require 18TB of RAM, the "Note: the aggregate simulated accelerator memory across all nodes must be sufficient to accommodate the model’s checkpoint size." text was a warning to the user rather than a rule that we would check for adherence to.

The following rule seems like the one we're talking about:
4.3.4. checkpointAggregateAcceleratorMemory -- The aggregate simulated accelerator memory across all nodes must be sufficient to accommodate the model’s checkpoint size. That is, the GB of memory associated with the chosen accelerator (eg: H100) times the accelerator count must be equal to or greater than the total checkpoint size for that scale of checkpoint. (see table 2)

IMHO this could be interpreted as "you need this much RAM" or it could be interpreted as "you need this many GPUs". Given the optimization of the datagen code, I'd like us to consider the second interpretation. Would that be valid? What is the right/best answer here?

I'm trying to be cautious since it's very late to be changing rules.