Nipype: Reuse workflows within workflows

[Eric]

Hi all,

I'm trying to figure out how to build one workflow, for instance, and reuse it within other workflows.  From what I've read of what Nipype can do I know this should be easy, but it's hard to understand from the documentation and I couldn't find any other users here with the same question, so...

Example!

Let's say I've created a generic workflow, "workflow1", I'd like to reuse utilizing nodes and connecting them in a python file called workflow1.py.  Now I would like to use workflow1.py's workflow within another workflow, "workflow2".  What should I import, if anything, and from where to be able to use workflow1 in workflow2?  Also, what are the input and output connections for workflow1 now called?

Thank you all in advance.  Any help is much appreciated.

~Eric

[Eric]

Hello again,

I dug a little deeper and found this:

http://miykael.github.io/nipype-beginner-s-guide/firstLevelExample.html#define-a-framework-workflow-that-contains-the-preprocess-and-the-volume-analysis

... which I think explains how to use and connect multiple workflows, but still begs the question for me, "How do I contain one workflow within one python file and connect it to another workflow within another python file?  Is IdentityInterface what I should use for a larger pipeline's inputs and is DataSink what I should use for outputs?"

Thanks some more!

~Eric

[Fred Loney]

Hi Eric,

I raised this question in https://groups.google.com/forum/?fromgroups#!topic/nipy-user/w4e15-vkAfg. See also https://groups.google.com/forum/?fromgroups#!topic/nipy-user/qSpLzv1oYLw.

I experimented with four styles:

• Connect nodes across workflows
• Wrap the reused workflow in a CommandLine interface
• Use Nippype Memory to connect workflows
• Execute workflows successively in Python functions

I favor the last, and most primitive, approach. I abandoned the promising Memory style because it doesn't parallelize easily. I find little advantage in wrapping workflows in a callable Command interface. It made more sense in my case to split a master workflow into three coarse-grained subworkflows with no cross-workflow node connection. These are run successively, with inputs set and outputs collected in a Python control flow. In my experience, reusing nodes across workflows is not a workable solution.

Nipype would be improved by implementing basic workflow constructs like sub-process, fork and join (MapNode notwithstanding) found in standard Process Modeling. However, that requires a significant architectural change.

Perhaps the Nipype experts can offer insightful dissent from this view, which I welcome.

Fred

[Fred Loney]

I should clarify that the 'master workflow' is a Python pipeline script, not a Nipype Workflow object. The subworkflows are Nipype Workflow objects which are built and run by the pipeline script.

[Satrajit Ghosh]

hi eric and fred,

i'll fast answer eric's question first as that is much simpler:

```

wf = Workflow(name='outer_workflow')

node1 = Node(...)

wf1 = create_inner_workflow()

wf.connect(node1, 'output1', wf1, 'inner_node.input1')

```

as an example see here:

https://github.com/nipy/nipype/blob/9595f272aa4086ea28f7534a8bd05690f60bf6b8/nipype/workflows/fmri/fsl/preprocess.py#L8

https://github.com/nipy/nipype/blob/9595f272aa4086ea28f7534a8bd05690f60bf6b8/nipype/workflows/fmri/fsl/preprocess.py#L963

now for fred's question:

we should have this discussion in greater length, but here are some thoughts from my perspective, other developers may have different thoughts. 

- we already have sub-process from BPMN, i.e., the workflow above.

- i think the constraints boil down to graph surgery in the context of iterables and the fact that they may exist at many different levels (including on a node in a sub-process) - BPMN cannot address this.

- recently michael hanke has implemented a nipype workflow that actually does take care of fork and join quite nicely

- we need to have conditionals/gateways in nipype

if someone can figure out how to deal with iterables with BPMN, i think that would really help. 

cheers,

satra

--
 
---
You received this message because you are subscribed to the Google Groups "NiPy Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nipy-user+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Chris Filo Gorgolewski]

@Fred please remind me why you prefer not to connect nodes across workflows?

[Fred Loney]

Satra, thanks for the clarification. I tried that back in February without success, but I must have been doing something wrong. It works now.

Chris, I must have misinterpreted the response in an earlier post.

BTW, BPMN Process has both an iterative and parallel indicator. But of course that's just a modeling construct. The hard part is making it work!

Thanks,

Fred

[Eric]

Thank you all for the prompt and excellent responses.  That really clears up a lot.  I think I'll use the function returning a workflow method and I didn't even realize IdentityInterface was there before this.  So glad for the help.  Thanks again!

~Eric

[Eric]

Okay.  So, I did the method where I...

call a function to create a pipeline which...

calls functions to create workflows which...

have nodes and mapnodes wrapping interfaces.

Now, I have a tricky predicament.  If I truly want to reuse a workflow (let's call it WORKFLOW_A).where I use the exact same WORKFLOW_A multiple times in a single pipelines with different input and output connections, how can I rename each reused WORKFLOW_A so Nipype can differentiate between WORKFLOW_A_EPI, WORKFLOW_A_MPR, and WORKFLOW_A_T2?

So far, I have tried resetting WORKFLOW_A's ".name" property after creating the workflow to WORKFLOW_A_EPI, WORKFLOW_A_MPR, and WORKFLOW_A_T2.  That seems to differentate them at the name level, but the folders in the current working directory still come out as only WORKFLOW_A.

In other words, the new problem is getting the output directories renamed as well as the workflows.

Thanks for any help or ideas anyone can offer.

~Eric

[Eric]

Hey All,

I think I answered my own question again... I just passed an extra optional argument into my functions creating workflows, "workflow_name", instead of always using the same name.  Duh.  Sorry for the excess post.

~Eric