Workflow iterating on internal workflow

[Oscar Esteban]

Dear all, I'd like to extend the Workflow class to behave as an Iterable. But it would not iterate on inputs, it'd be over a number of inserted internal workflows that share the inputs.

The idea is as follows: one would define an interface for the internal workflow (e.g. in a segmentation task, the input image and parameters), and then it should be possible to insert as many different segmentation methods (fulfilling the inputs/outputs interface) as one wants to run. Output ports should be generated automatically as outputnode0, outputnode1, etc. Calling run() would launch all the methods inserted this way.

Is it feasible with the current architecture? Can anyone shed a light on this so I can start with my implementation?

These new workflows would be interesting when one wants to compare several methods/algorithms.

Thanks in advance

Oscar

[Oscar Esteban]

Dear all,

I've been advancing with this idea, and now I can easily build graphs this fashion:

where testname1 and testname2 are workflows that meet a certain i/o interface and can be automatically inserted.

This is useful when comparing algorithms, if you build a workflow that meets the general interface, N methods can be grafted in the graph.

However, in runtime, the graph is flattened and I want to propagate the outputs at outXX identity interfaces to the external outputnode (that has been modified to allow several input connections, that should be collated to build a list).

Is this possible? Where is the point in the pipeline engine (I guess somewhere in pipeline/utils.py) where the outputs are propagated to inputs? OR at which point the connection is pruned before I can build the list?

Best,

Oscar

[Satrajit Ghosh]

hi oscar,

take a look at joinnode. can you describe the mechanism you are using to build these graphs?

cheers,

satra

--

---
You received this message because you are subscribed to the Google Groups "NiPy Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nipy-user+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Oscar Esteban]

Hi Satra,

Yes, as most of the times it is desiderable to have two identity interfaces as input/outputnodes, I've created an "InterfacedWorkflow" that sets them up by default. Some additional improvements include that inputs can be referred to without the leading 'inputnode.' like 'inputnode.fieldname'. This makes workflows more readable. Other features include automatic connection of inputs and outputs when both interfaces are the same (field by field connection is still allowed).

Then, I have a "GraftWorkflow" derived from InterfacedWorkflow, which adds a method "insert" to allow inserting InterfacedWorkflows meeting the in/out interface. For this, identity nodes named outXX are created for each inserted workflow.

My idea is to build something like the JoinNode, but in this case I have to look at all the output nodes (outXX) that point to the input of the final outputnode. I recall that here I want to join a set of IdentityInterfaces and for each field of the interface generate a list with the fields of the outputs.

In my github, this code is in the branch enh/GraftWorkflow (https://github.com/oesteban/nipype/tree/enh/GraftWorkflow).

I've been looking at JoinNode, but this situation of having several source-nodes I don't know how to deal with it.

Before flattening the graph, I can see that the outputnode has several predecessors.

Thanks!

[Oscar Esteban]

Maybe not the most elegant solution, but works well and is simple.

I finally wrote a new IdentityInterface-like interface in which inputs are dynamic and output lists are generated depending on the prefix of the input ports.

If you want me to do a PR, just let me know :)

[Satrajit Ghosh]

hi oscar,

do send a PR. this comparative tool is going to be quite useful for persistent evaluations.

however, many comparative evaluations take place in the context of a larger dataflow. some node in a graph changes algorithm for example. but what one wants to evaluate is the impact on some downstream node. within this context using iterables for algorithms and then joinnode comes in handy.

we can work through this in the PR.

cheers,

satra