Nexus: Failed jobs not caught when wallclock limit exceeded (slurm)

[prckent]

Describe the bug

Slurm jobs that are killed due to the wallclock limit being reached are not recognized as failed by Nexus. Instead the subsequent jobs are run. Obviously this is at best wasteful. The error is incorrectly assigned to the subsequent job(s) if/when they fail (assuming failure due to missing files and not incorrect but successful execution).

Example:

$ ./qmc_scan.py --status_only|grep aa2_
...
    111111  0  35110     aa2_lowest_5_each_geom_type_0004  ./runs/aa2_lowest_5_each_geom_type_0004/PBE0/ccecp-aug-cc-pV5Z/SCF
    111111  1  37295     c4q     ./runs/aa2_lowest_5_each_geom_type_0004/PBE0/ccecp-aug-cc-pV5Z/SCF
    000000  0  ------    opt_J12_1  ./runs/aa2_lowest_5_each_geom_type_0004/PBE0/ccecp-aug-cc-pV5Z/opt_J12_1
    000000  0  ------    opt_J123_1  ./runs/aa2_lowest_5_each_geom_type_0004/PBE0/ccecp-aug-cc-pV5Z/opt_J123_1
....

Notice that the SCF job is considered successful while the c4q wavefunction conversion job failed. However, looking at the specifics:

$ cd ./runs/aa2_lowest_5_each_geom_type_0004/PBE0/ccecp-aug-cc-pV5Z/SCF
$  tail -n 10 aa*.out aa*.err
==> aa2_lowest_5_each_geom_type_0004.out <==
  2.44587563e+01  2.46048591e+01  2.47231764e+01  2.47497489e+01
  2.49176545e+01  2.49440755e+01  2.50440220e+01  2.50698014e+01
  2.51035157e+01  2.51324647e+01  2.53741177e+01  2.53881410e+01
  2.54181575e+01  2.54285586e+01  2.55597784e+01  2.55686291e+01
  2.56874991e+01  2.57723998e+01  2.57837897e+01  2.57948679e+01
  2.58880298e+01  2.59715996e+01  2.59932008e+01  2.61416840e+01
  2.62687226e+01  2.62802764e+01  2.72429252e+01  2.72868739e+01
  3.76959870e+01  3.78467014e+01]
nelec by numeric integration = [58.00004233 58.00004233]
    CPU time for vxc   2987.61 sec, wall time     93.86 sec

==> aa2_lowest_5_each_geom_type_0004.err <==
<class 'pyscf.df.df_jk.DFROKS'> does not have attributes  tol
slurmstepd: error: *** JOB 35110 ON blc111 CANCELLED AT 2026-06-30T06:08:35 DUE TO TIME LIMIT ***

i.e. PySCF was killed due to hitting the wallclock limit. There is therefore no final energy or wavefunction .h5 file created for the subsequent step.

Expected behavior

Catch the job failure. Either check the job status in slurm (via sacct) or parse the error code in the job error output. Consider the job failed. Jobs like the SCF job above should register as failed.

System:

pathfinder at ORNL (same as baseline), slurm. Likely not pathfinder specific.

Additional context

I have not investigated how uniform & consistent the error reporting is between slurm installations and versions, but looking for JOB.*CANCELLED in the error file would be a good start if we want to add (missing?) parsing of error file output. Perhaps the error simply changed(?). Alternatively (my preference), a cleaner new route that avoids potentially brittle parsing of files would be to use sacct -j JobID and look for the existence of TIMEOUT and perhaps also CANCELLED.

$ sacct -b -j 35110
JobID             State ExitCode
------------ ---------- --------
35110           TIMEOUT      0:0
35110.batch   CANCELLED     0:15
35110.extern  COMPLETED      0:0

[brockdyer03]

Using sacct may work, however it is unclear how long ago you can expect sacct to be able to report. From some testing on Perlmutter, the maximum time window is ~4 weeks, which you can consistently get with the following command:

$ sacct -u <user> --starttime=now-4weeks --endtime=now --format="JobID,ExitCode,State"

This seems a good enough time window for most users, however I am unsure if other clusters store job data for that long, so they may only have shorter windows available.