SCHED-1897: Remove alloc_mem_used health check#2778
Draft
fabrizio2210 wants to merge 1 commit into
Draft
Conversation
fabrizio2210
force-pushed
the
agent/sched-1897-remove-alloc-mem-used
branch
from
July 24, 2026 11:06
6a7c404 to
1e5f492
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
alloc_mem_useddrain and undrain checks[user_problem] alloc_mem_usedas the replacement check's stable drain-reason identifier so nodes drained before the upgrade can still be recoveredWhy
This PR is related to SCHED-1897.
The periodic
idle_mem_usedcheck introduced by #2773 replaces the old prolog-timealloc_mem_usedbehavior for detecting suspicious memory use. Removing the old check avoids running two overlapping memory checks with different execution models.Dependency
This PR is stacked on #2773 and deliberately targets
agent/sched-1897-idle-memory-checkinstead ofmain. Merge #2773 first, then retarget this PR tomain.This ordering ensures that
alloc_mem_usedis not removed before itsidle_mem_usedreplacement is available.For safe upgrades with persisted Helm values, merge the independent,
main-based #2779 before this PR. It adds generic validation for enabled built-ins whose bundled files no longer exist.Upgrade impact
[user_problem] alloc_mem_usedremain recoverable because the replacement drain and undrain schedules preserve that stable reason prefix.CHECKS_JOB_ALLOC_MEM_BYTESsupport remains available to custom checks.Persisted-values warning
Deployments that explicitly retain
slurmScripts.builtIn.alloc_mem_used.*withenabled: truemust remove those entries during the upgrade. Without #2779, Helm can retain the removed built-in keys, render missing script/configuration content, and produce a malformed checks configuration that prevents the check runner from running health checks.#2779 changes this failure mode into a clear Helm rendering error. It is kept separate so this PR contains only the check removal and its direct drain-reason compatibility.
Validation
python3 -m unittest discover -s helm/slurm-cluster/slurm_scripts -p '*_test.py'— 21 tests passedhelm unittest helm/slurm-cluster— 101 tests passed across 18 suiteshelm templaterendering passedgit diff --checkpassed