MRIQC in parallel processing

[zeit problem]

Hey all,

Here I am with another problem :)

I'm trying tp speed up the qc by running multiple processes in parallel (using python's multiprocessing). The thing is, it fails when I'm running mriqc from python in parallel, but when I'm executing the generated command via terminal, it works. 

Here some more information:

I'm running a singularity image (v0.16.1) on a limnux machine. Some of the errors I get are: 

FileNotFoundError: [Errno 2] No such file or directory: '/users/rubendorfel/work/mriqc_wf/anatMRIQC/ComputeIQMs/_in_file_..data..sub-f1081..ses-1..anat..sub-f1081_ses-1_run-1_T1w.nii/metadata/_0x76900ad800180ae11eefae989e0b8241_unfinished.json' -> '/users/rubendorfel/work/mriqc_wf/anatMRIQC/ComputeIQMs/_in_file_..data..sub-f1081..ses-1..anat..sub-f1081_ses-1_run-1_T1w.nii/metadata/_0x76900ad800180ae11eefae989e0b8241.json'

This is the command I'm calling:

singularity run --cleanenv --bind <path_to_data>/bids:/data --bind  <path_to_data>/  mriqc:/out  <path_to_sif>/mriqc-0.16.1.sif /data /out participant --participant_label <sub_id> --no-sub 

If I run it within the parallel processing, it fails, but If I run it separately in a terminal, everything works ... I'm kinda puzzled right now. 

[Oscar Esteban]

That error typically arises when there's a race condition between two processes. It seems to me you are running two subjects with same subject ID, session ID, and run index (in other words, highly likely to be the same image) at the same time.

I would suggest that you leverage MRIQC's internal parallelization by running groups of ~4 subjects. Then using bash (no need to wrap singularity within python, or else you will quickly build up memory issues), just throw several singularity instances of those 4-subject singularity calls.

Cheers,

oe

--
You received this message because you are subscribed to the Google Groups "mriqc-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mriqc-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mriqc-users/942841c4-f16d-4127-a8b9-b92b983b7a8dn%40googlegroups.com.

[zeit problem]

As I said, I'm printing out the command that is executed in each process, and it is definitely not the exact same image. Maybe the issue is that the working directory is the same? 

Anyway, I ended up the second approach you suggested :)

Best,

Ruben.