Eiger2CBF parallel conversion script

[w.l...@sydney.edu.au]

Hi All

Attached is a python script I have written to speed up using Eiger2CBF. The script splits up the run into a specified number of pools and spawns a copy Eiger2CBF for each pool, enabling parallel conversion of the dataset into CBF files. 

On my computer the optimum number of parallel instances is 12, which gives a 5x increase in rate of file conversion.

The script was written with python 3.8.0, so any version of python newer than that should work. The script assumes that the eiger2cbf executable is available in the directory it is run (it is in the path or in the directory).

The script is invoked by typing:

python eiger2cbf_spawn.py [name of master h5 file] [number of instances to spawn] [filename stem for CBF] [optional test]

Optionally the word 'test' can be put after the above, which will test a +-5 range from the number of instances specified, and calculate the time per frame converted. This only does a short batch (10 frames) in each pool, so is quicker than running the full conversion.

If you have any questions or find any bugs, please let me know, either via here or emailing me directly.

Cheers

William

[w.l...@sydney.edu.au]

Hi All

Attached is an updated version of the script. This version should avoid the problem where eiger2cbf occasionally skips 2 frames and writes them as blank files (thanks to Stephanie for pointing out this problem with eiger2cbf). This version extracts each frame as a separate process, rather than spawning a number of processes which convert a batch each. Asides from the details of how the script works it is still started the same way as before.

Cheers

William

[w.l...@sydney.edu.au]

This is now available at https://github.com/WilliamLewis-X-ray/eiger2cbf-spawn as well.