SCHED-1897: Add idle node memory health check

[fabrizio2210]

Historical status

This PR was accidentally merged into the #2743 feature branch. It is retained as the historical review record and should not be used as the active dependency stack.

The clean replacement stack is:

1. Expose node RealMemory without controller RPC #2774 — expose node RealMemory without a controller RPC
2. SCHED-1897: Add idle node memory health check #2773 — add the dependent idle-memory health check

Original scope

This PR introduced the SCHED-1897 idle_mem_used periodic health check. It uses local scontrol listjobs for RPC-free idle detection, consumes authoritative Slurm RealMemory, derives the idle-memory threshold from MemTotal - RealMemory, and drains idle nodes with an actionable diagnostic when available memory is unexpectedly low.

See #2773 for the complete replacement description, validation, and isolated five-file feature diff.

[rdjjke]

This env var is populated based on scontrol show node output, so it won't allow us to get rid of these RPCs completely. It doesn't run in the Prolog context, so it's better, but just highlighting that RPCs are still in place.
Theoretically, we can check for node idleness by using scontrol listjobs, which doesn't make any RPCs to the Slurm controller (it requests the local worker instead of the controller).

[rdjjke]

os.environ["CHECKS_NODE_STATE_FLAGS"] = "+".join(get_node_info().state_flags)

[fabrizio2210]

I was aware of keeping an RPC, but I didn't know about scontrol listjobs. I will substitute it

[fabrizio2210]

In an empty node it returns RC 1, is it correct?

$ kubectl -n soperator exec worker-0 -- chroot /mnt/jail  scontrol listjobs
No slurmstepd's found on this node
command terminated with exit code 1

While during the job:

$ kubectl -n soperator exec login-0 -- chroot /mnt/jail srun --mem=2048 -N1 -w worker-1 scontrol listjobs
cpu-bind=MASK - worker-1, task  0  0 [3560]: mask 0x3 set
JOBID
1011    

[fabrizio2210]

My understanding is that RC=1 is expected, also when running hc_program because it doesn't spawn the slurmstepd process. I updated the code

[rdjjke]

This is a good start, but I think we should eliminate the need to pass this setting to Helm values for each cluster.
How it is now:

• Slurm nodes have RealMemory setting. It shows the amount of RAM on this node that Slurm will use.
• Soperator guarantees that this setting is equal to the worker K8s pods memory limit.
  But when we have a separate Helm value, then there is no such guarantee anymore. A person can update one of the values but forget to update another, which can lead to nodes being drained wrongly.

What I suggest:

• Update the nodeset reconciliation logic.
• When Soperator creates the worker statefulset with some specific memory limit, it also adds the same memory limit to the environment variable for this container
• Container entrypoint (slurmd_entrypoint.sh) reads this env var, and stores it to a file on the ephemeral container filesystem
• The check runner reads this file, instead of using the Helm value + env var

The solution can also be generalized for other cases and values where we need to communicate some metadata about the node from Soperator to various scripts (i.e., VM ID, GPU platform, the presence of IB, or any other metadata that would be useful in other scripts without querying KubeAPI or other services).

For example, we can have a Helm value like "Soperator exported metadata env", which is a list of environment variable names that Soperator should export inside the node.
Then, in the entrypoint script, we write an env file to some node-local volume inside the jail (e.g., /etc/soperator_exported_meta.env).
And then any script (ours or the user's) that runs there can execute source /etc/soperator_exported_meta.env and have this metadata available.
It's also customizable and configurable. We can export more vars if needed by adding them to the StatefulSet spec and allowing them in the Helm value.
Or maybe we can put all environment variables there, even without allowlists.

[fabrizio2210]

I see, it seems interesting, so we can be reusable. But it seems a bigger/separate change. Do you want me to create a separate PR? And then stack this (i.e. 2742) PR on top of it?

[fabrizio2210]

By mistake I collapsed this request onto #2743