Lima segmenter not working on Windows

[rehogenbirk]

Using the lima_segmenter function from \nbGui\S3DXRD\0_segment_and_label.ipynb does not work on Windows as it generates a bash script to perform the actions. An alternative is shown in the s3DXRD example: \nbGui\S3DXRD\0_segment_frelon.ipynb, but the input options and output are different, so cannot work as a direct substitute.

The code I tried to run is from \nbGui\S3DXRD\0_segment_and_label.ipynb:

create batch file to send to SLURM cluster

sbat = ImageD11.sinograms.lima_segmenter.setup(ds.dsfile, **ui.getopts(), pythonpath=PYTHONPATH)
if sbat is None:
raise ValueError("This scan has already been segmented!")
print(sbat)

utils.slurm_submit_and_wait(sbat, 60)

The error is as follows:

[rehogenbirk]

Script generated from successful segmetnation run on esrf jupyter slurm
lima_segmenter_slurm.sh

[rehogenbirk]

I am unable to do it within python as I ran into issues passing the variables to the segment function within lima_segmenter.py.

[jadball]

Hi,

Thanks for getting in touch. In general, the provided notebooks for 3DXRD/S3DXRD reconstruction piplines assume that you are running them on the ESRF Slurm cluster. The script causing you issues is to prepare a job array task, to segment (well, actually just sparsify at this step) many detector frames in parallel across multiple cluster tasks (e.g. multiple machines).

Can I get some more info on what you're trying to do, and the size of your input data? If it's the sort of thing that will fit on one machine, I think together we can make a quick script that calls the right lima_segmenter code to run locally for you.

Thanks,
James

[jonwright]

I would expect that "ImageD11.sinograms.lima_segmenter.sbatchlocal" runs on a local machine that has bash installed. This should work under git bash on a windows machine.

Otherwise I think you just set files_per_core=len(ds.imagefiles) and cores_per_job=1 so that you get one job. Then call:

   python -m ImageD11.sinograms.lima_segmenter segment dsname.h5 0

We could change sbatchlocal to do that instead of calling bash.

[rehogenbirk]

Hi James and Jon,

I am trying to get the segmentation to work locally for ma4985/2_SSB2_PCNMC811/2_SSB2_PCNMC811_CCCD_C10_2_cycles, whose folder is 245 GB. I am unsure what the limits are with 32GB of RAM.
I have run this segmentation on SLURM last week when I got my access back, but would like to know how to run this locally to make sure I can run it again in the future when I do not have access to SLURM.

I tried to modify the bash file to run locally and I got the following error from the log file. Attached you can find the slurm and edited bash files and the log file.
File "C:\Users\rijkelberthoge\AppData\Local\miniconda3\envs\esrf\Lib\site-packages\ImageD11\sinograms\lima_segmenter.py", line 514, in segment
jobid = int(sys.argv[3])
~~~~~~~~^^^
IndexError: list index out of range

Am I looking in the right direction?
I do not know how to define the python variables in the bash script or where to make in lima_segmenter.py to make the edits.

lima_segmenter_slurm.sh
lima_segmenter_.log
lima_segmenter_local.sh

[jonwright]

Are you running via ImageD11.sinograms.lima_segmenter.sbatchlocal? Something like:

from ImageD11.sinograms.lima_segmenter import sbatchlocal
sbatchlocal( 'lima_segmenter_local.sh' )

[rehogenbirk]

I was not. I tried, but I am getting empty log files numbered 0 to 29.

Command run in jupyter notebook:
from ImageD11.sinograms.lima_segmenter import sbatchlocal
slurm_bash = r"U:\phdrehogenbirk\ma4985\2_SSB_PCNMC811_bothcycles\PROCESSED_DATA\position_0\nmc_101\2_SSB2_PCNMC811\2_SSB2_PCNMC811_CCCD_C10_both_cycles\slurm\lima_segmenter_local.sh".replace("\", "/")
sbatchlocal(slurm_bash)

lima_segmenter_local.sh

[jonwright]

You can probably replace "bash" by "cmd /c" in the string that gets executed, if it is saying something like "bash" command does not exist.

[rehogenbirk]

The bash command works within the cmd as you can see in the image. I replaced bash by "cmd /c" in sbatchlocal and got the same result. I suspect something else is happening as log files get made and are made as expected, but are just empty.

[jonwright]

Can you try to execute the script command manually and see what happens?

[rehogenbirk]

That works! Next problem is that the dataset file contains links to the sparse data on SLURM instead of the local data and the path cannot be updated through ds.sparsefile (or ds.sparsefiles), but that is a seperate issue. Highlighted in the image are the paths I tried to update but could not.

update_paths.ipynb

[jonwright]

Try calling dataset.update_paths(self, force=False, verbose = False) but set force=True and verbose=True before saving...

[rehogenbirk]

That works for the dataset, thanks.

Next error I get is that it cannot create nor access the sparsefiles it should create, both when I keep the sparsefiles folder and when I remove it. I checked the path and filename and they are identical.

lima_segmenter_test3.log

[jonwright]

The filename it wants to create looks longer than what I am expecting:
'U:\phdrehogenbirk\ma4985\2_SSB_PCNMC811_bothcycles\PROCESSED_DATA\position_0\nmc_101\2_SSB2_PCNMC811\2_SSB2_PCNMC811_CCCD_C10_both_cycles\sparsefiles_data_visitor_ma4985_id11_20211201_2_SSB2_PCNMC811_2_SSB2_PCNMC811_CCCD_C10_2_cycles_scan0010_eiger_0000_sparse.h5'

It should usually be something like:
'U:\phdrehogenbirk\ma4985\2_SSB_PCNMC811_bothcycles\PROCESSED_DATA\position_0\nmc_101\2_SSB2_PCNMC811\2_SSB2_PCNMC811_CCCD_C10_both_cycles\sparsefiles\scan0010_eiger_0000_sparse.h5'

Does the sparsefiles subfolder exist? That should have been created during lima_segmenter.setup_slurm_array.

I am starting to think we should just add some option to setup to run the segmentation locally instead of on slurm...

[rehogenbirk]

Hi, sorry for the late response. I can confirm the subfolder and the files with the long names exists. The sparsefiles are already there from the segmenter run on SLURM. I do not know why these filenames are so long. It looks like the path with the slashes replaced by underscores.