Support of parallelization across sessions

[Chris Gorgolewski]

Dear BIDS Apps developers,

We have just discussed a long requested feature - parallelization across sessions. Here is the proposal I would love your opinion on:

Additional analysis level: session*

Additional parameter: --session_label

If a given BIDS App supports per session parallelization it should accept the "session" analysis level. This means that separate independent jobs can be send for specific sessions and subjects. 

For example an app supporting the following levels: "session", "participant" will allow for the following execution schedule (assuming dataset with two participants and two sessions).

First parallel batch:

job 1) /bids_in /out session --participant_label 01 --session_label 01

job 2) /bids_in /out session --participant_label 01 --session_label 02

job 3) /bids_in /out session --participant_label 02 --session_label 01

job 4) /bids_in /out session --participant_label 02 --session_label 02

Second parallel batch (starts when all jobs of the first batch ends)

job 5) /bids_in /out participant --participant_label 01 --session_label 01 02

job 6) /bids_in /out participant --participant_label 02 --session_label 01 02

Jobs that don't specify session level work as usual and should not expect the --session_label parameter. For example an app accepting "participant" and "group" levels will allow for the following execution scheme:

First parallel batch:

job 1) /bids_in /out participant --participant_label 01

job 2) /bids_in /out participant --participant_label 02

Second parallel batch (starts when all jobs of the first batch ends)

job 3) /bids_in /out group --participant_label 01 02

Let me know what do you think!

Best,

Chris

[Eric Earl]

Chris,

I think this is a great idea and would help anyone that can parallelize their data processing at the (optional) session level.

~Eric

[Ali Khan]

Are there existing or in-the-works BIDS-App-wide standards to support parallelization (subject or session), or is it up to the App developer to build it in internally? E.g. I am thinking of batch job parallelization on HPC with slurm/sge/condor etc, as to do that currently we are using wrappers.

Ali