add hook for building GROMACS on NVIDIA Grace CPUs with hwloc support#225
Open
bedroge wants to merge 12 commits into
Open
add hook for building GROMACS on NVIDIA Grace CPUs with hwloc support#225bedroge wants to merge 12 commits into
bedroge wants to merge 12 commits into
Conversation
Contributor
Author
|
bot: build repo:eessi.io-2025.06-software instance:eessi-bot-jsc for:arch=aarch64/nvidia/grace |
Contributor
Author
|
Ah, need to wait for a dirty frag mitigation to be deployed. |
Contributor
Author
|
bot: build repo:eessi.io-2025.06-software instance:eessi-bot-jsc for:arch=aarch64/nvidia/grace |
|
New job on instance
|
Contributor
Author
|
bot: build repo:eessi.io-2025.06-software instance:eessi-bot-jsc for:arch=aarch64/nvidia/grace |
|
New job on instance
|
Contributor
Author
|
Hmm, it now runs it with cc @al42and |
|
Huh, fun. The GPU NUMA node seems to have gone away (processorsinNumaNudes went down from 9*72 to 8*72; need to make the name less lewd), but there are still seven mystery NUMA nodes left. Any chance you can get hwloc XML from the machine? |
Contributor
Author
Sure, here it is. I have to confirm it, but it looked like it worked with |
Contributor
Author
|
Let me try it here with bot: build repo:eessi.io-2025.06-software instance:eessi-bot-jsc for:arch=aarch64/nvidia/grace |
|
New job on instance
|
Contributor
Author
|
bot: build repo:eessi.io-2025.06-software instance:eessi-bot-jsc for:arch=aarch64/nvidia/grace |
|
New job on instance
|
Contributor
Author
|
Hmm, same error now. The other main difference between my manual attempt to build it and the builds done by the bot is that the latter uses a container. I can give it a try in the same container. |
Have you generated the XML in the container or not? The XML does not have any of the "phantom" numa nodes GROMACS is complaining about. So if it was done bare-metal, than everything points to a side-effect of containerization. |
|
@bedroge. Can you simply add a call to |
al42and
reviewed
May 11, 2026
Co-authored-by: Andrey Alekseenko <[email protected]>
Contributor
Author
|
bot: build repo:eessi.io-2025.06-software instance:eessi-bot-jsc for:arch=aarch64/nvidia/grace |
|
New job on instance
|
Contributor
Author
That one was indeed generated without the container. The bot always uses the container for doing the software builds, so I'm now trying your approach. |
Contributor
Author
|
@al42and Here it is: |
|
I have a GH200 node, and there is a similar pattern if I look at However, bare-metal hwloc correctly filters them out, unless $ hwloc-ls
Machine (856GB total)
Package L#0
NUMANode L#0 (P#0 119GB)
NUMANode(GPUMemory) L#1 (P#4 95GB)
L3 L#0 (114MB)
...
$ HWLOC_ALLOW=all hwloc-ls
Machine (856GB total)
Package L#0
NUMANode L#0 (P#0 119GB)
NUMANode(GPUMemory) L#1 (P#4 95GB)
NUMANode L#2 (P#5)
NUMANode L#3 (P#6)
NUMANode L#4 (P#7)
NUMANode L#5 (P#8)
NUMANode L#6 (P#9)
NUMANode L#7 (P#10)
NUMANode L#8 (P#11)
L3 L#0 (114MB)
...There does not seem to be a clean way to filter them out except by the lack of memory. I guess this means that either Skipping this test seems like the right solution now; in the next GROMACS version, we can filter invalid NUMA nodes ourselves. To me, the hwloc behavior seems wrong here, but I don't know to what extent you can/want debug the containerization. We don't have any containers on our nodes, so I don't think I could dig any deeper. But at least I can reproduce the thing on my end, so I won't pester you about testing the workaround :) |
Contributor
Author
|
I tried it on the host itself (i.e. without the container), and then I do get the same output as you. Inside the container (same hwloc version) I get the additional NUMA nodes, even though that environment variable is not set. So, looks like you're right and the containerization somehow messes things up... I think I'll try disabling that test for now. Thanks again for your help! |
Contributor
Author
|
Looks like bind mounting edit: I figured it may be cgroup-related (since it's running as a Slurm job), and indeed, bind mounting |
Contributor
Author
|
bot: build repo:eessi.io-2025.06-software instance:eessi-bot-jsc for:arch=aarch64/nvidia/grace |
|
New job on instance
|
bedroge
commented
May 18, 2026
bedroge
commented
May 18, 2026
Contributor
Author
|
bot: build repo:eessi.io-2025.06-software instance:eessi-bot-jsc for:arch=aarch64/nvidia/grace |
|
New job on instance
|
Contributor
Author
|
bot: build repo:eessi.io-2023.06-software instance:eessi-bot-jsc for:arch=aarch64/nvidia/grace |
|
New job on instance
|
|
New job on instance
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Trying the suggestion from EESSI/software-layer#1497 (comment).