Read Pilatus4/Lima2 data

[jonwright]

Currently: sinograms.dataset cannot interpret ID15a lima2 Pilatus4 data. The VDS map is interleaving frames over 16 files. The ImageD11 code was not expecting this.

Unclear how to work around in a clean way. Either:

• read the frames via the master file (we need to use the master file + scan number + detector in lima_segmenter).

Or:

• segment the underlying files (as before) and interpret the VDSmap, eg:
• [ vsrc.vspace.get_select_hyper_blocklist() for vsrc in frames.virtual_sources() ]
• Either dataset could map the omega angles (and monitor and dty and etc etc) to follow the image file order
• or: assemble_label could re-map the sparse frames to follow the experiment order

There is something coming up about multiple thresholds, but for now I didn't find an example.

Example data are in /data/id11/jon/Pilatus4_ID15a

Might be a relevant example for @abmajith. Thanks to Stefano Checchia for the dataset.

[jonwright]

Related - the master file encodes an absolute path and not a relative path. Problematic to copy the data around.

[jonwright]

For file paths it has:

• 1.1/measurement/data = Soft Link to instrument group
• 1.1/instrument/detector/data = External Link to scan0001/master.h5
• scan0001/master.h5 = VDSmap

[AxelHenningsson]

I am very interested in any progress on this. Is there something I could do to help out here?

We have recent data from ID15 we would very much like to proccess in ImageD11.

Cheers
Axel

[jonwright]

Just learned, there is frame_idx in each individual file...

[jonwright]

There is something (hacky) working for the silicon calibration in:

/data/id11/jon/Pilatus4_ID15a/PROCESSED_DATA/silicon/silicon_diffuse/silicon_calib.ipynb

I just need to figure out how to send this to git.

[abmajith]

Hi @jonwright

Could you confirm is the detector name is Pilatus4, i could not see its name under instrument or measurement of raw data

/data/id11/jon/Pilatus4_ID15a/RAW_DATA/align_89keV/align_89keV_0001/align_89keV_0001.h5::/

[jonwright]

@abmajith : In scan 12.1 the detector is "pilatus4_raw_frame". The roi counter names do not match the detector name (requires a change in segmenter gui). Are you trying to use ImageD11 in /data/id11/jon/Pilatus4_ID15a/SCRIPTS? If not, we still need to merge some changes.

[abmajith]

Hmm we use the ImageD11 adapted api in the ewoks3dxrd like this

def get_frame_image(
    file_path: str | Path,
    detector: str,
    scan_id: str,
    frame_idx: int | None = None,
    counter: str = "_roi1",
) -> np.ndarray:
    """
    extract a frame image from master file path
    """
    with h5py.File(file_path, "r") as h5file:
        if frame_idx is None:
            frame_idx = get_optimal_frame_index(
                masterfile_path=file_path,
                scan_number=scan_id[:-2],
                detector=detector,
                counter=counter,
            )
        return h5file[f"{scan_id}/measurement/{detector}"][frame_idx]

but in the future, we ewoks3dxrd) has to adpat this new detectors,

[jonwright]

Does someone in your group know how to ask the bliss file for "the stack of 2D images" and "the roi counters associated with those images" ? It should be a nexus navigation issue.

For now, I have just made counter be the full string, and not assume it starts with the detector name. This is an easy fix if we put the counter names into our notebooks. For your gui, I have no idea how you can decide which counter to use. For sure, it should not be hard wired into the code :-)

[abmajith]

Does someone in your group know how to ask the bliss file for "the stack of 2D images" and "the roi counters associated with those images" ? It should be a nexus navigation issue.

For now, I have just made counter be the full string, and not assume it starts with the detector name. This is an easy fix if we put the counter names into our notebooks. For your gui, I have no idea how you can decide which counter to use. For sure, it should not be hard wired into the code :-)

I will ask them asap.

[woutdenolf]

I will describe the situation as it currently is.

ask the bliss file for the stack of 2D images

Loop over the NXdetector groups under instrument and find the ones with a dataset called data that is three-dimensional.

Regardless of the scan dimensions, in the HDF5 file the data is always saved as a flat list of nD data-points. So for images that results in a 3D dataset.

Alternatively loop over all datasets in the measurement group and select the ones at are three-dimensional.

the roi counters associated with those images

There is no rigorous link except relying on the names. When the image NXdetector is called "eiger" then all associated counters are called "eiger_*". This is for ROI's, bpm's and any other associated signals.

Note: lima2 is not stable yet and can change any time.

[jonwright]

Thanks @woutdenolf . In the bliss master file for lima2, it seems to be using raw_frame instead of data below:

hpc5-0704:RAW_DATA/silicon/silicon_diffuse % h5ls silicon_diffuse.h5/1.1/measurement/pilatus4_raw_frame
pilatus4_raw_frame       Soft Link {/1.1/instrument/pilatus4/raw_frame}

hpc5-0704:RAW_DATA/silicon/silicon_diffuse % h5ls  silicon_diffuse.h5/1.1/instrument/pilatus4 
acq_params               Group
description              Dataset {SCALAR}
raw_frame                External Link {scan0001/pilatus4_raw_frame_master.h5//entry_0000/ESRF-ID00/pilatus4/data}
roi_statistics_roi1_avg  Dataset {1440/Inf}
roi_statistics_roi1_max  Dataset {1440/Inf}
roi_statistics_roi1_min  Dataset {1440/Inf}
roi_statistics_roi1_std  Dataset {1440/Inf}
roi_statistics_roi1_sum  Dataset {1440/Inf}
type                     Dataset {SCALAR}
x_pixel_size             Dataset {SCALAR}
y_pixel_size             Dataset {SCALAR}

Can we infer that the roi counters belong to the detector when they are in the same instrument group?

For now, I understand there is a frame_idx within the data files as well as the VDSmap layout in the raw_frame_master.h5 file. I guess these should always agree, but Manu Papillon suggested the frame_idx is the one coming in the Dectris stream and should always give the correct ordering. For processing, we need the frames in the proper order, and frame_idx might be easier to read than a VDS.

[woutdenolf]

Again, Lima2 can change at any moment from beamline to beamline until Bliss 2.3

The structure you show is unknown to me and created by patching the blisswriter I presume. I don't know how the ROI's are finally going to be saved. Probably like Lima1 but I have not seen any data yet. But in this case I assume yes, the ROI counters in the same NXdetector group should be coming from that detector.

Because of the multiple receivers, the frames are all over and the Lima2 master file has the correct order in the VDS. I'm not aware of any saving of the frame_idx. I'm guessing in the Lima2 files from the receivers?

[abmajith]

For processing, we need the frames in the proper order, and frame_idx might be easier to read than a VDS.

Is it possible from the omega values (diffrz), and the corresponding 2D frame, we could order the 2D stack frame images for now?

[AxelHenningsson]

I am trying to read data from

/data/visitor/ma6975/id15a/20251127/RAW_DATA/Ag2S_dogbone_pristine_2_94kev_crl/Ag2S_dogbone_pristine_2_94kev_crl_3dxrd

Does anyone know what is the good omega motor (i.e the rotation stage angular values)? I find that file are heterogenous and was expecting hrrz_center to be found in all files. As an example, see the below code and output:

paths = ['/data/visitor/ma6975/id15a/20251127/RAW_DATA/silicon/silicon_diffuse/silicon_diffuse.h5',
 '/data/visitor/ma6975/id15a/20251127/RAW_DATA/Ag2S_dogbone_pristine_2_94kev_crl/Ag2S_dogbone_pristine_2_94kev_crl_3dxrd_0010/Ag2S_dogbone_pristine_2_94kev_crl_3dxrd_0010.h5',
 '/data/visitor/ma6975/id15a/20251127/RAW_DATA/Ag2S_dogbone_pristine_2_94kev_crl/Ag2S_dogbone_pristine_2_94kev_crl_3dxrd_0009/Ag2S_dogbone_pristine_2_94kev_crl_3dxrd_0009.h5',
 '/data/visitor/ma6975/id15a/20251127/RAW_DATA/Ag2S_dogbone_pristine_2_94kev_crl/Ag2S_dogbone_pristine_2_94kev_crl_diffuse/Ag2S_dogbone_pristine_2_94kev_crl_diffuse.h5',
 '/data/visitor/ma6975/id15a/20251127/RAW_DATA/Ag2S_dogbone_pristine_2_94kev_crl/Ag2S_dogbone_pristine_2_94kev_crl_3dxrd_0008/Ag2S_dogbone_pristine_2_94kev_crl_3dxrd_0008.h5']
for p in paths:
    print(f"\n\n In File {p.split('/')[-1]} I found: \n ")
    with h5py.File(p, "r") as hin:
        measurements = hin['1.1']["measurement"]
        for key in measurements.keys():
            if 'hrrz' in key:
                print(end = '    ' + key)

With the heterogenous output:

In File silicon_diffuse.h5 I found: 

   hrrz    hrrz_center    hrrz_delta    hrrz_period    hrrz_raw    hrrz_trig

In File Ag2S_dogbone_pristine_2_94kev_crl_3dxrd_0010.h5 I found: 

   hrrz    hrrz_cen360    hrrz_period    hrrz_raw    hrrz_trig

In File Ag2S_dogbone_pristine_2_94kev_crl_3dxrd_0009.h5 I found: 

   hrrz    hrrz_cen360    hrrz_period    hrrz_raw    hrrz_trig

In File Ag2S_dogbone_pristine_2_94kev_crl_diffuse.h5 I found: 

   hrrz    hrrz_center    hrrz_delta    hrrz_period    hrrz_raw    hrrz_trig

In File Ag2S_dogbone_pristine_2_94kev_crl_3dxrd_0008.h5 I found: 

   hrrz    hrrz_cen360    hrrz_period    hrrz_raw    hrrz_trig

Sometimes I see hrrz_cen360 and sometimes hrrz_center etc...