Iterables on list defined only when workflow is run: any solution?

[Michael]

I often face the same issue: i would like to define an iterable on a list only known after an interface is run.

The most recent example is the one when i run a preprocessing pipeline, the output for each subject being saved with the following format:

subject_id_mymask1.nii.gz, subject_id_mymask2.nii.gz, ..., subject_id_mymaskN.nii.gz

For the main pipeline i need an iterable on these masks, but i can't get the iterable list before running first a datagrabber or selectfiles node and so the following won't work:

subjectIDs = pe.Node(util.IdentityInterface(fields=["subject_id"]),"subjectIDs")

subjectIDs.iterables = ('subject_id', ['subject1','subject2', ..., 'subjectK'])

masks = dict(mask1 = '{subject_id}_mymask1.nii.gz',

             mask2 = '{subject_id}_mymask2.nii.gz',

                     ...

             maskN = '{subject_id}_mymaskN.nii.gz')

allmymasks = pe.Node(SelectFiles(masks, base_directory='/path/to/my/masks'), "allmymasks")

MergeMasks= pe.Node(Merge(N), "MergeMasks")

MyOpOnAllMasks = pe.Node(Op(fields=["Mask"]), "MyOpOnAllMasks")

MyOpOnAllMasks.iterables = [("Mask", Can't put my mask list here because masks haven't been grabbed yet)] 

MyWf.connect([

                    (subjectIDs, allmymasks,

                    [('subject_id', 'subject_id')]),

                    (allmymasks, MergeMasks,

                    [('mask1', 'in1'), etc.,('maskN', 'inN') ]),

                    (MergeMasks, MyOpOnAllMasks,

                    [('out', 'Mask')]),

                    ...

           ])  Won't work

How to solve this issue elegantly?

[Satrajit Ghosh]

hi mike,

is N constant? or does N vary by subject?

if N is fixed, you can setup another iterable, use a joinnode to collect those masks and merge.

if N is variable, i would use a mapnode.

the other thing to consider is that post the merge, the iterables don't have an effect on the merged file. so mapnodes might be more useful here.

cheers,

satra

--

---
You received this message because you are subscribed to the Google Groups "NiPy Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nipy-user+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Michael]

In that particular case N is constant but sometimes i face the same situation with N variable.

So if N is constant you would suggest to setup another iterable on a dummy variable going from 1 to N (since the masks path cannot be obtain before runtime) and then somehow get the masks path within the iterated pipeline?

[Satrajit Ghosh]

On Fri, Mar 28, 2014 at 5:16 PM, Michael <mdayan....@gmail.com> wrote:

In that particular case N is constant but sometimes i face the same situation with N variable.

So if N is constant you would suggest to setup another iterable on a dummy variable going from 1 to N (since the masks path cannot be obtain before runtime) and then somehow get the masks path within the iterated pipeline?

further decision points would be:

-  are the masks are generated by a single node or multiple nodes?

- if single node, is that computation expensive and therefore you might want to parallelize - use mapnode.

- perhaps feed to fslmerge directly to get 4d mask file

i think this is a place where you have a fair bit of flexibility as to what you want to do.

cheers,

satra

[Michael]

Hi Satra,

The mask are collected with SelectFiles with an iterable on subjects (1 subject -> 1 collection of masks) and i cannot use MapNode as the subsequent steps use multiple nodes, even possibly MapNodes so i would need to use an iterable on the masks and a JoinNode. The problem is that these masks are collected during runtime and thus i cannot define an iterable on a list not already known.