Skip to content

SCHED-1897: Remove alloc_mem_used health check#2778

Draft
fabrizio2210 wants to merge 1 commit into
agent/sched-1897-idle-memory-checkfrom
agent/sched-1897-remove-alloc-mem-used
Draft

SCHED-1897: Remove alloc_mem_used health check#2778
fabrizio2210 wants to merge 1 commit into
agent/sched-1897-idle-memory-checkfrom
agent/sched-1897-remove-alloc-mem-used

Conversation

@fabrizio2210

@fabrizio2210 fabrizio2210 commented Jul 24, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • remove the obsolete built-in alloc_mem_used drain and undrain checks
  • remove their default Helm registrations
  • keep [user_problem] alloc_mem_used as the replacement check's stable drain-reason identifier so nodes drained before the upgrade can still be recovered

Why

This PR is related to SCHED-1897.

The periodic idle_mem_used check introduced by #2773 replaces the old prolog-time alloc_mem_used behavior for detecting suspicious memory use. Removing the old check avoids running two overlapping memory checks with different execution models.

Dependency

This PR is stacked on #2773 and deliberately targets agent/sched-1897-idle-memory-check instead of main. Merge #2773 first, then retarget this PR to main.

This ordering ensures that alloc_mem_used is not removed before its idle_mem_used replacement is available.

For safe upgrades with persisted Helm values, merge the independent, main-based #2779 before this PR. It adds generic validation for enabled built-ins whose bundled files no longer exist.

Upgrade impact

  • Existing nodes whose drain reason starts with [user_problem] alloc_mem_used remain recoverable because the replacement drain and undrain schedules preserve that stable reason prefix.
  • CHECKS_JOB_ALLOC_MEM_BYTES support remains available to custom checks.

Persisted-values warning

Deployments that explicitly retain slurmScripts.builtIn.alloc_mem_used.* with enabled: true must remove those entries during the upgrade. Without #2779, Helm can retain the removed built-in keys, render missing script/configuration content, and produce a malformed checks configuration that prevents the check runner from running health checks.

#2779 changes this failure mode into a clear Helm rendering error. It is kept separate so this PR contains only the check removal and its direct drain-reason compatibility.

Validation

  • python3 -m unittest discover -s helm/slurm-cluster/slurm_scripts -p '*_test.py' — 21 tests passed
  • helm unittest helm/slurm-cluster — 101 tests passed across 18 suites
  • default helm template rendering passed
  • YAML and JSON parsing passed
  • git diff --check passed

@fabrizio2210
fabrizio2210 force-pushed the agent/sched-1897-remove-alloc-mem-used branch from 6a7c404 to 1e5f492 Compare July 24, 2026 11:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant