Numba LLVM error when doing chunked PBP indexing

[jadball]

Sometimes I see an error like this:

LLVM ERROR: Symbol not found: __gufunc__._ZN8ImageD119sinograms10tensor_map13unitcell_to_bB3v69B52c8tJTIeFImbKsAWlBR0FJMfbgsMFGDwYEQhVCwwAMEsTAA_3d_3dE5ArrayIdLi1E1A7mutable7alignedE5ArrayIdLi2E1A7mutable7alignedE5ArrayIdLi2E1A7mutable7alignedE
/var/lib/slurm/slurmd/job31322315/slurm_script: line 12: 3379103 Aborted                 (core dumped)

when doing chunked PBP indexing.
I think this is a race condition for generating/loading cached Numba functions - see here #4807
In a SLURM job array task, the first job will crash instantly, but others will hang forever.
This seems to generate 'poisoned' cached Numba functions which will always crash on future attempts to import them (nice!).
The solution is to delete the __pycache__ - this is normally in ImageD11/sinograms in the git checkout you are running on.

[jadball]

I will try to fix by importing ImageD11.sinograms.tensor_map inside pbp_1_indexing.ipynb before the chunked call is run on SLURM. I'd like to leave this open for a while just in case it happens again though to motivate further debugging. Would rather not turn off caching because it slows things down a lot.

[jonwright]

There should be a per-cpu-flavor cache. If the front-end node differs from the nodes running the code, you will still encounter the problem. Doesn't it need to run (and wait on) a compilation job if the cache is not valid?

[jadball]

I think the problem lies here:

• You work on a fresh checkout of ImageD11
• Nothing has been compiled yet
• You start a chunked PBP indexing job on a cluster machine
• These jobs run parallel per-machine and also across multiple machines
• Each parallel process tries to compile the Numba code (when exactly? don't know)
• They write the compiled code to __pycache__ - a race condition in the write leads to garbled cached code
• The garbled cached code causes SIGABRT

This could also happen if you import and cache the Numba code on one machine, then submit cluster jobs to a different machine. Numba is architechture-sensitive so will trigger re-caching (e.g. if you run the notebooks on an Intel system then run the slurm jobs on an AMD system)