conditionals & loops

[Peter Amstutz]

Hello everyone,

Here's a sketch for adding conditionals and sequential loops in CWL.

For conditionals, Introduce an "if" field which is evaluated when the
step is ready to run (there are values ready on all the input links.)
If true, run the step as normal. If false, the output parameters are
all assigned "null".

Downstream steps or output parameters would use the multilple input
links feature to list several conditional steps as sources. To
accommodate condition steps that don't execution, introduce new
linkMerge methods "select" and "non_null", the first of which selects
the first non-null value from a list of incoming links, the second
filters out null values.

inputs:
switch: bool
outputs:
out:
type: string
outputSource [iftrue/out, iffalse/out]
linkMerge: select # select the first non-null input link
steps:
iftrue:
in: {switch: switch}
out: [out]
if: $(inputs.switch == true)
run: do-this.cwl
iffalse:
in: {switch: switch}
out: [out]
if: $(inputs.switch == false)
run: do-that.cwl

In this example, if "switch" is true, then run "do-this.cwl" and
assign the value of "iftrue/out", "iffalse/out" is set to null. The
"out" parameter has two source links for a value of ["foo", null] and
the "select" merge method would take the first non-null value ("foo").
If "switch" is false, everything is reversed, "iftrue/out" is null and
"iffalse/out" is assigned, the "out" source is [null, "bar"] and the
first non-null value is "bar".

For sequential loops, use the "if" field to determine whether to run
the step; each time the step runs, assign the value of output
parameters to input parameters as listed in "loop" and evaluated "if"
again. The output value of the step is the output of the final
iteration. As with the "if" example above, if the the conditional is
false on the first test, the output parameters are all assigned
"null".

inputs:
startvalue: int
endvalue: int
outputs:
out:
outputSource: loop/out
steps:
loopstep:
in: {startvalue: startvalue, endvalue: endvalue}
out: [out]
if: $(inputs.startvalue < inputs.endvalue)
loop:
startvalue: out
run: add-one.cwl

Example trace:
startvalue: 1, endvalue: 3
if $(inputs.startvalue < inputs.endvalue) -> true
add-one.cwl -> {out: 2}
loop and assign to startvalue the value of out
startvalue: 2, endvalue: 3
if $(inputs.startvalue < inputs.endvalue) -> true
add-one.cwl -> {out: 3}
loop and assign to startvalue the value of out
startvalue: 3, endvalue: 3
if $(inputs.startvalue < inputs.endvalue) -> false
final step output -> {out: 3}

Thoughts?

Thanks,
Peter

[karl.no...@gmail.com]

Hi Peter,

I can only speak from a user's point-of-view, but I was looking for something like the switch statement. Allowing the Javascript expression in the if block seems very convenient. I could definitely work with with this.

I havn't found need for looping yet, but perhaps that is just a matter of time. One concern I have here is that the tool must be written with the loop in mind. I'm not sure if it is possible, but perhaps the loop block could contain an increment: {javascript expression} tag somehow.

Another aspect is how to transport information to the next iteration. As I see it, a step's dependence on a previous iteration is a main argument to use a loop over the already present scatter-functionality. Perhaps I missed it?

Not sure if the discussion continued elsewhere, but I leave this here where I found it.

Best,
Karl

[Michael Miller]

hi all,

 

yes, loops are a definitely useful construct.  there are algorithms that loop until a certain threshold is reached, is one example, where a biological population reaches a steady state.

 

cheers,

michael

 

Michael Miller

Software Engineer

Institute for Systems Biology

 

 

--
You received this message because you are subscribed to the Google Groups "common-workflow-language" group.
To unsubscribe from this group and stop receiving emails from it, send an email to common-workflow-la...@googlegroups.com.
To post to this group, send email to common-workf...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/common-workflow-language/1412e3fd-e52f-418f-9259-025bca16afe6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[karl.no...@gmail.com]

Until proper conditionals are implemented I realized that it is possible to do something similar to tapply with a simple ExpressionTool. Each output variable can be redirected to different work steps and if needed the results can be merged later.

cwlVersion: v1.0
class: ExpressionTool

inputs:
  array: string[]
  cases: string[]

outputs:
  case1: string[]
  case2: string[]
  case3: string[]
  case4: string[]
  unknown: string[]

expression: |
  ${
    var case1= [];
    var case2= [];
    var case3= [];
    var case4= [];
    var unknown= [];

    for ( var i= 0; i < inputs.array.length; i++){
      switch (inputs.cases[i]) {
        case "case1":
          case1.push(inputs.array[i]);
          break;
        case "case2":
          case2.push(inputs.array[i]);
          break;
        case "case3":
          case3.push(inputs.array[i]);
          break;
        case "case4":
          case4.push(inputs.array[i]);
          break;
        default:
          unknown.push(inputs.array[i]);
      }
    }

    return {"case1":case1, "case2":case2, "case3":case4, "case4": case4, "unknown": unknown};
  }

This is obviously a bit clumsy and there are probably better ways of achieving the same thing. Problems are that I don't think you can have an arbitrary number of outputs and then the conditions are hard-coded. I think the latter could be overcome with a smarter implementation. The former, though, forces you to write a separate tool depending on the number of cases.

To unsubscribe from this group and stop receiving emails from it, send an email to common-workflow-language+unsub...@googlegroups.com.

[Stian Soiland-Reyes]

I see this resurrects an old thread from Peter.

I think conditionals are important - we found Taverna users needed
branching, but they did similar tricks to the ones proposed here,
which often led to weird behaviour. For instance, Peter's suggestion
with using `null` can introduce two weird cases: Where neither test
conditions are true, or where both test conditions are true.


Yes, you can use "null" as a way to 'cancel out' a branch of the
workflow - however it will be very unclear to the user which part of
the workflow would be cancelled out as it depends on if downstream
steps can handle the "null" or not (without looking at their types and
defaults in detail) - in a way there is no explicit { .. } blocks in
such an approach.


In Taverna we found that users started doing conditionals using fake
errors - where a first step step would check a condition, and return
an error, which then caused downstream steps in that part of the
workflow to not run. Steps not relying on that first step would
however run. In some cases we found workflows that ran both the "if"
and "else" branch because users didn't cover an overlapping edge case
in their check (neither failed!) - with a "pick first" merge strategy
this would be silently swallowed, but still kill performance and
introduce side effects (e.g. database updates) - as well as having a
race condition as to which branch would arrive "first" - a second run
of the same workflow could give a different result! Not recommended
at all for scientific workflows..

A similar approach is using empty (false) or singleton lists (true)
in combination with scattering and list merging, need to flatten again
afterwards when merging. This however has the same problem that you
could end up with two empty lists or two non-empty lists.

So I think those kind of workarounds don't work well. Karl's
suggestion is an elaboration of this - with a single 'dispatcher' step
it should be easier to verify that you don't accidentally
double-branch or no-branch (unless that is what you explicitly want) -
but this would just be a pattern which users could (and would) get
wrong - and also which can't be reasoned over.

An explicit dispatcher syntax would have the benefit that it can do
extra checking - but still this only works if there's a single kind of
input that needs to be passed on - extra inputs needs to be passed on
out of bands - which gives the overall workflow impression that the
step is wired up - only to find it being "turned off" by the empty
list from the dispatcher. However with an explicit syntax it's
easier for visualizer to highlight this, for instance a different
colour or icon on the dispatcher step or its outgoing links.


In my view we should do more dataflow-focus and less control-flow -
and only have cordoned off branching - basically it should either be a
single step or a bunch of steps in a nested workflow (possibly
embedded). That way the "interface" to that branch remains the same -
it must provide the same outputs no matter which branch it takes. Any
'incompatible' outputs must then be filled with hard-coded fallback
values - or null/empty list workaround.

This means the workflow is still flowing in an understandable manner,
and it becomes difficult to end up in "dual state" situations.

I think a syntax for "if" should be very similar to "while" loop.

Perhaps "if" should support more than 2 ordered conditions, like
Clojure's "cond" - but with an explicit :else step (not implicit
nulls) - or does that get tricky to represent in Schema Salad?


In Taverna we didn't implement if-blocks - but we did do a while loop
layer - which runs a step again until a condition is satisfied. It's
possible to wire outputs from the previous loop iteraton as input to
the next iteration - or to modify them from within the loop condition
checking script. It is typically used with a nested workflow inside -
which then effectively becomes the while { } block. See:
http://dev.mygrid.org.uk/wiki/display/tav250/Loops

>> email to common-workflow-la...@googlegroups.com.

>> To post to this group, send email to common-workf...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/common-workflow-language/1412e3fd-e52f-418f-9259-025bca16afe6%40googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>

> --
> You received this message because you are subscribed to the Google Groups
> "common-workflow-language" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to common-workflow-la...@googlegroups.com.

> To post to this group, send email to
> common-workf...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/common-workflow-language/5f74db50-1fc0-4cf1-b622-af5a8c9ac5a5%40googlegroups.com.

>
> For more options, visit https://groups.google.com/d/optout.

--
Stian Soiland-Reyes, eScience Lab
School of Computer Science
The University of Manchester
http://orcid.org/0000-0001-9842-9718

[Luka Stojanovic]

I completely agree with everything said. Steps very naturally behave
like blocks, so adding conditionals, looping or error handling mechanism
to them seem like the right approach. I also agree that we should think
in a term of switch/cond/guards rather than ifs. For looping though, I
would rather workflow not having loops, and loop-controlling expression
having the results of previous run(s) by virtue of us deciding so,
rather than creating cyclic workflows.

[Pjotr Prins]

On Fri, Dec 16, 2016 at 05:52:01PM +0100, Luka Stojanovic wrote:
> I completely agree with everything said. Steps very naturally behave
> like blocks, so adding conditionals, looping or error handling mechanism
> to them seem like the right approach. I also agree that we should think
> in a term of switch/cond/guards rather than ifs. For looping though, I
> would rather workflow not having loops, and loop-controlling expression
> having the results of previous run(s) by virtue of us deciding so,
> rather than creating cyclic workflows.

Hi Luka,

I have been reading this thread with interest and it confirms my idea
that CWL is not going anywhere when we keep trying to please
everyone. What started out as a 'simple' document of a workflow, now
is trying to be a full computer language.

The original idea of CWL representing a workflow as a document is
great. And the abstraction of a generalized workflow, in my opionion,
is something that runs serially (in sequence) or fans out and collects
(in parallel). That idea is captured in CWL and can be implemented
fairly easily. Implementation matters for general adoption of
CWL. Note that, despite the hype, we only have two working
implementations.

I propose a CWL-lite - which is a minimum version of a workflow description.
Strip out the fancy support for Docker, Javascript and such. That can all be
handled by the underlying implementation. The CWL should not try to be a
language.

Pj.

[Michael Miller]

hi all,

a couple of comments:

> For looping though, I
> would rather workflow not having loops, and loop-controlling expression
> having the results of previous run(s) by virtue of us deciding so,
> rather than creating cyclic workflows.

i might be misunderstanding this, but in our lab, one of the most common
workflows is the simulation of cell growth where there is 'set state from
previous step' then 'run step' and repeat until steady state. do you see
this type of workflow being possible to specify?

> > ... - but still this only works if there's a single kind of
> > input that needs to be passed on - ...
> >
> > ...it must provide the same outputs no matter which branch it takes.

this certainly isn't the general case, the reason there is an else is that
there is a different condition, perhaps the if needs the equivalent of
stdout and the else, stderr.

cheers,
michael

Michael Miller
Software Engineer
Institute for Systems Biology

> To post to this group, send email to common-workflow-
> lang...@googlegroups.com.

> To view this discussion on the web visit
> https://groups.google.com/d/msgid/common-workflow-

> language/91ee3278-b3a5-1a8c-b5ae-462f0407a1f3%40sbgenomics.com.

[Karl Nordström]

Hi everybody,

I couldn't let this go. I understand that finding workarounds perhaps isn't the best solution, but I find the inclusion of conditionals would be a great improvement on CWL. 

My first solution was to allow javascript expressions as input to the run statement, but that would make it hard to control if a workflow is valid before execution. I appreciate those checks. I also think it would allow to high a degree of freedom and would make it hard to follow/understand a workflow.

I liked the the idea of keeping it within one step and enforce common input and output variables. Below is an alternative implementation:

steps:

  conditional:

    condition: $(expr) #allows graceful termination of a line of calculation. (Could be used as try/catch)

    in:

      inputA: ...

      inputB: ...

    out:

    - outputA

    - outputB

    run:

      ifelse: #ifelse instead of switch. Will only execute the first tool with a true condition. Takes a list as input... not just two

        - run: toolA.cwl

          condition: $(inputs.inputA == 12)

        - while:

          condition: $(inputs.inputA < 12)

          update: #allow access to the output of the last iteration of the loop

          - inputA:

              valueFrom: $(inputs.inputA + 1)

          - inputB:

              source: "#conditional/outputB"

          run: #allow nesting

            ifelse:

            - condition: $(inputs.inputA == 0)

              run: toolB.cwl

            - run: toolC.cwl #default behaviour, else statement

        - run: toolD.cwl

          condition: $(...)

        #when no condition can be fulfilled, terminate thread of calculations

• No need of a merge function. No relying on null output
  
• Each step happens once (multiplied by scatter and loop iterations)
• I think it would be beneficial to have a way to terminate a pipeline without getting an error from a calculation step. A crude version of try/catch. Hence I added the condition block to WorkStep as well.
  
• The tools needs to have the same set of outputs. On the input-side I don't think there is a problem with overloading as long as there are no two same named parameters with different functionality. Output naming must be the same though. This is a restraint, but I think it is a good one. It goes in the direction of harmonizing tool design. And in a worst case, it can we resolved with a wrapper-workflow.

I have never used Clojure and don't know how cond works, but maybe an "ifelse chain" is similar?

And at last, I'm not yet versed enough in CWL to be able to judge how the above suggestion plays along with other parts. Call it blessed ignorance if you wish :)

Best,

Karl

> >>> email to common-workflow-language+unsub...@googlegroups.com.

> >>> To post to this group, send email to common-
> workf...@googlegroups.com.
> >>> To view this discussion on the web visit
> >>> https://groups.google.com/d/msgid/common-workflow-
> language/1412e3fd-e52f-418f-9259-025bca16afe6%40googlegroups.com.
> >>> For more options, visit https://groups.google.com/d/optout.
> >> --
> >> You received this message because you are subscribed to the Google
> Groups
> >> "common-workflow-language" group.
> >> To unsubscribe from this group and stop receiving emails from it, send
> >> an

> >> email to common-workflow-language+unsub...@googlegroups.com.

> >> To post to this group, send email to

> >> common-workflow-language@googlegroups.com.

> >> To view this discussion on the web visit
> >> https://groups.google.com/d/msgid/common-workflow-
> language/5f74db50-1fc0-4cf1-b622-af5a8c9ac5a5%40googlegroups.com.
> >>
> >> For more options, visit https://groups.google.com/d/optout.
> >
> >
>
> --
> You received this message because you are subscribed to the Google Groups
> "common-workflow-language" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to common-workflow-language+unsub...@googlegroups.com.

> To post to this group, send email to common-workflow-
> lang...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/common-workflow-

> language/91ee3278-b3a5-1a8c-b5ae-462f0407a1f3%40sbgenomics.com.

> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "common-workflow-language" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/common-workflow-language/JU7PSEKr97M/unsubscribe.
To unsubscribe from this group and all its topics, send an email to common-workflow-language+unsub...@googlegroups.com.
To post to this group, send email to common-workflow-language@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/common-workflow-language/b78efbba87f34144970da27712b844ea%40mail.gmail.com.

[Peter Amstutz]

I think your suggestion of embedding the conditional into the workflow step is a good one.  Personally I would prefer to not have nesting since you can already use subworkflows.  A step could contain either an if/else chain or a loop.

What should the behavior be when if/else is nonexhaustive or there are zero loop iterations?

Interesting point about terminating a workflow without forcing a tool failure.  Related is how to permit partial failure (eg a single failed scatter step) so a workflow can keep going.

> >>> email to common-workflow-language+unsubscr...@googlegroups.com.

> >>> To post to this group, send email to common-
> workf...@googlegroups.com.
> >>> To view this discussion on the web visit
> >>> https://groups.google.com/d/msgid/common-workflow-
> language/1412e3fd-e52f-418f-9259-025bca16afe6%40googlegroups.com.
> >>> For more options, visit https://groups.google.com/d/optout.
> >> --
> >> You received this message because you are subscribed to the Google
> Groups
> >> "common-workflow-language" group.
> >> To unsubscribe from this group and stop receiving emails from it, send
> >> an

> >> email to common-workflow-language+unsubscr...@googlegroups.com.

> >> To post to this group, send email to
> >> common-workflow-language@googlegroups.com.
> >> To view this discussion on the web visit
> >> https://groups.google.com/d/msgid/common-workflow-
> language/5f74db50-1fc0-4cf1-b622-af5a8c9ac5a5%40googlegroups.com.
> >>
> >> For more options, visit https://groups.google.com/d/optout.
> >
> >
>
> --
> You received this message because you are subscribed to the Google Groups
> "common-workflow-language" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to common-workflow-language+unsubscr...@googlegroups.com.

> To post to this group, send email to common-workflow-
> lang...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/common-workflow-

> language/91ee3278-b3a5-1a8c-b5ae-462f0407a1f3%40sbgenomics.com.

> For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to a topic in the Google Groups "common-workflow-language" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/common-workflow-language/JU7PSEKr97M/unsubscribe.

To unsubscribe from this group and all its topics, send an email to common-workflow-language+unsubscr...@googlegroups.com.

To post to this group, send email to common-workflow-language@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/common-workflow-language/b78efbba87f34144970da27712b844ea%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "common-workflow-language" group.
To unsubscribe from this group and stop receiving emails from it, send an email to common-workflow-language+unsub...@googlegroups.com.
To post to this group, send email to common-workflow-language@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/common-workflow-language/CAD%3DV0dSdiXLhbDny5%2BjhMv1RoNqoVX3xtD%2BsbKTtoujfOwPnKg%40mail.gmail.com.

[Paul Grosu]

We should be careful how we extend the language, or we possibly risk making it hard to maintain later on.  Yes, conditionals and loops are useful, but their implementation now can have implications in the long run when extending the language with other concepts.  I prefer we stay within the framework of graphs, and keep it as simple as possible.  So loops are naturally recursive conditionals.  For example, a conditional and a loop as a graph are as follows – which I know we all recognize:

 

  A Conditional

  ~~~~~~~~~~~~~

 

  { input (x) } --------> { func1(x) & true => store_to(output) } --------> { output }

                   |                                                  |

                   +----> { func2(x) & true => store_to(output) } ----+

                   |                                                  |

                   +----> { func3(x) & true => store_to(output) } ----+

 

 

  A Loop (Recursive Conditional)

  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

 

   { input ( x: [start, end] ) }

             │

             │

             ▼

       { base_case:  ( base_func( first(x) ) == second(x) ) & true  => store_to(output) ;   

         x = local_update_first_elem_func(x) => base_case ;  } ──┐

                                                                 │

                                                                 │

                                                                 ▼

                                                            { output }

 

The functions first() and second() functions denote access and updates to the first and second indexed element in the (input) array, which in this case is x.  Also a node in the graph is denoted as a {...}, which can be a CWL file to run containing the required checks.  We can even have standard CWL files for specific parts of the language, such as a conditional CWL (conditional.CWL) and an iteration CWL (interation.CWL) that would be called provided to the run field.  We do not need both the start and end conditions to be part of x, but can each be provided by other parts of the graph if necessary as it is being evaluated.

 

~p

> >>> email to common-workflow-language+unsub...@googlegroups.com.

> >>> To post to this group, send email to common-
> workf...@googlegroups.com.
> >>> To view this discussion on the web visit
> >>> https://groups.google.com/d/msgid/common-workflow-
> language/1412e3fd-e52f-418f-9259-025bca16afe6%40googlegroups.com.
> >>> For more options, visit https://groups.google.com/d/optout.
> >> --
> >> You received this message because you are subscribed to the Google
> Groups
> >> "common-workflow-language" group.
> >> To unsubscribe from this group and stop receiving emails from it, send
> >> an

> >> email to common-workflow-language+unsub...@googlegroups.com.
> >> To post to this group, send email to

> >> common-workf...@googlegroups.com.

> >> To view this discussion on the web visit
> >> https://groups.google.com/d/msgid/common-workflow-
> language/5f74db50-1fc0-4cf1-b622-af5a8c9ac5a5%40googlegroups.com.
> >>
> >> For more options, visit https://groups.google.com/d/optout.
> >
> >
>
> --
> You received this message because you are subscribed to the Google Groups
> "common-workflow-language" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to common-workflow-language+unsub...@googlegroups.com.

> To post to this group, send email to common-workflow-
> lang...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/common-workflow-

> language/91ee3278-b3a5-1a8c-b5ae-462f0407a1f3%40sbgenomics.com.

> For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to a topic in the Google Groups "common-workflow-language" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/common-workflow-language/JU7PSEKr97M/unsubscribe.

To unsubscribe from this group and all its topics, send an email to common-workflow-language+unsub...@googlegroups.com.

To post to this group, send email to common-workf...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/common-workflow-language/b78efbba87f34144970da27712b844ea%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "common-workflow-language" group.
To unsubscribe from this group and stop receiving emails from it, send an email to common-workflow-language+unsub...@googlegroups.com.

To post to this group, send email to common-workf...@googlegroups.com.

[Michael Miller]

hi paul,

 

'I prefer we stay within the framework of graphs'

 

if/else conditionals fit into directed acyclic graphs, loops fit into general graphs.  taverna allows conditionals and a restricted looping, galaxy appears to be in the same position as CWL, fielding requests for a conditional feature but not implemented, as does pegasus.  in a paper from 2014 about pegasus, they had this to say:

'These workflows are interpreted to

coordinate data flow and task execution on distributed infrastructures

including grids, clusters and clouds. Moteur [86] is a

workflow engine that uses a built-in language to describe workflows

that targets the coherent integration of: 1) a data-driven

approach to achieve transparent parallelism; 2) arrays manipulation

to enable data parallel application in an expressive and

compact framework; 3) conditional and loop control structures

to improve expressiveness; and 4) asynchronous execution to

optimize execution on a distributed infrastructure. These systems

provide conditional and loop functionalities that are not

provided by Pegasus, however the complexity of their workflow

language can limit scalability and adoption.'

so even though i'm all for conditionals, the general opinion for CWL to move cautiously on conditionals is well supported.

> >>> email to common-workflow-la...@googlegroups.com.

> >>> To post to this group, send email to common-
> workf...@googlegroups.com.
> >>> To view this discussion on the web visit
> >>> https://groups.google.com/d/msgid/common-workflow-
> language/1412e3fd-e52f-418f-9259-025bca16afe6%40googlegroups.com.
> >>> For more options, visit https://groups.google.com/d/optout.
> >> --
> >> You received this message because you are subscribed to the Google
> Groups
> >> "common-workflow-language" group.
> >> To unsubscribe from this group and stop receiving emails from it, send
> >> an

> >> email to common-workflow-la...@googlegroups.com.

> >> To post to this group, send email to
> >> common-workf...@googlegroups.com.
> >> To view this discussion on the web visit
> >> https://groups.google.com/d/msgid/common-workflow-
> language/5f74db50-1fc0-4cf1-b622-af5a8c9ac5a5%40googlegroups.com.
> >>
> >> For more options, visit https://groups.google.com/d/optout.
> >
> >
>
> --
> You received this message because you are subscribed to the Google Groups
> "common-workflow-language" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to common-workflow-la...@googlegroups.com.

> To post to this group, send email to common-workflow-
> lang...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/common-workflow-

> language/91ee3278-b3a5-1a8c-b5ae-462f0407a1f3%40sbgenomics.com.

> For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to a topic in the Google Groups "common-workflow-language" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/common-workflow-language/JU7PSEKr97M/unsubscribe.

To unsubscribe from this group and all its topics, send an email to common-workflow-la...@googlegroups.com.

To post to this group, send email to common-workf...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/common-workflow-language/b78efbba87f34144970da27712b844ea%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "common-workflow-language" group.

To unsubscribe from this group and stop receiving emails from it, send an email to common-workflow-la...@googlegroups.com.

To post to this group, send email to common-workf...@googlegroups.com.

[Paul Grosu]

Hi Michael,

I agree, but I am trying to push us to something more fundamental, in order to save us major headaches down the line.  So Taverna has Scufl, Askalon has AGWL, Pegasus has DAX, CWL has YAML, Galaxy has JSON, as representations of the language.  But what we want is a common (workflow) language that can be portable everywhere for all domains.  Now Taverna has similarities to the definitions I proposed, but we want something more.

So in order to keep the language as clean as possible - by continuing to build up types from a small set of defined types - let's have both a complete second-order type system with evaluation rules for our current grammar.  Then we can have judgements, type rules, type derivation to ensure well-typing for the environments that these types are declared in.  Type inference will follow naturally, and macros will enable language definitions to be extended for different domains.  These all can be defined by a graph framework, which follow fundamentally the composability, associativity and identity operations.  Then all interpreters just utilize these definitions to guarantee soundness, and equivalence via the same results.  This ensures we have correctness, by allowing hierarchy to simply progress the language evolution in a natural and type-safe manner. 

Thanks,
Paul

> >>> email to common-workflow-language+unsub...@googlegroups.com.

> >>> To post to this group, send email to common-
> workf...@googlegroups.com.
> >>> To view this discussion on the web visit
> >>> https://groups.google.com/d/msgid/common-workflow-
> language/1412e3fd-e52f-418f-9259-025bca16afe6%40googlegroups.com.
> >>> For more options, visit https://groups.google.com/d/optout.
> >> --
> >> You received this message because you are subscribed to the Google
> Groups
> >> "common-workflow-language" group.
> >> To unsubscribe from this group and stop receiving emails from it, send
> >> an

> >> email to common-workflow-language+unsub...@googlegroups.com.

> >> To post to this group, send email to
> >> common-workf...@googlegroups.com.
> >> To view this discussion on the web visit
> >> https://groups.google.com/d/msgid/common-workflow-
> language/5f74db50-1fc0-4cf1-b622-af5a8c9ac5a5%40googlegroups.com.
> >>
> >> For more options, visit https://groups.google.com/d/optout.
> >
> >
>
> --
> You received this message because you are subscribed to the Google Groups
> "common-workflow-language" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to common-workflow-language+unsub...@googlegroups.com.

> To post to this group, send email to common-workflow-
> lang...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/common-workflow-

> language/91ee3278-b3a5-1a8c-b5ae-462f0407a1f3%40sbgenomics.com.

> For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to a topic in the Google Groups "common-workflow-language" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/common-workflow-language/JU7PSEKr97M/unsubscribe.

To unsubscribe from this group and all its topics, send an email to common-workflow-language+unsub...@googlegroups.com.

To post to this group, send email to common-workf...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/common-workflow-language/b78efbba87f34144970da27712b844ea%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "common-workflow-language" group.

To unsubscribe from this group and stop receiving emails from it, send an email to common-workflow-language+unsub...@googlegroups.com.

To post to this group, send email to common-workf...@googlegroups.com.

[Nebojsa Tijanic]

I think your suggestion of embedding the conditional into the workflow step is a good one.  Personally I would prefer to not have nesting since you can already use subworkflows.  A step could contain either an if/else chain or a loop.

+1, but what happens if someone needs both? And how does one specify to evaluate conditions in each iteration vs picking the branch/implementation first, then loop with same one? Go for the more common case (possibly disallow both in one step) and have people use nested WFs for the complex cases?

 

What should the behavior be when if/else is nonexhaustive or there are zero loop iterations?

Make it a repeat-until loop so there's always at least one iteration? Require the "else"? Embedding a small ExpressionTool allows for the initial "return nulls" idea (though Stian advised against it and I agree - see Errors below).

Interesting point about terminating a workflow without forcing a tool failure.  Related is how to permit partial failure (eg a single failed scatter step) so a workflow can keep going.

+1. Not the topic of this thread, but I think both of these can be achieved by introducing an Error type. Processes that fail produce Errors on all outputs, process with Errors on any input fail but WF author has option to filter out Errors in step config. In any case, it should probably be solved separately.

The need for conditions is common and there are cases, as Michael said, that need to loop over a step until e.g. a threshold is reached. I suggest we take several concrete use cases and post individual proposals (i.e. one proposal handles all use cases) on GH so that the edge cases and problems can be discussed for each individual solution. Then we pick some, implement as CWL extensions (each with own Requirement), wait a bit to see if other implementations include it, and eventually consider making it part of CWL without the need for explicit Requirement.

> >>> email to common-workflow-language+unsubscr...@googlegroups.com.

> >>> To post to this group, send email to common-
> workf...@googlegroups.com.
> >>> To view this discussion on the web visit
> >>> https://groups.google.com/d/msgid/common-workflow-
> language/1412e3fd-e52f-418f-9259-025bca16afe6%40googlegroups.com.
> >>> For more options, visit https://groups.google.com/d/optout.
> >> --
> >> You received this message because you are subscribed to the Google
> Groups
> >> "common-workflow-language" group.
> >> To unsubscribe from this group and stop receiving emails from it, send
> >> an

> >> email to common-workflow-language+unsubscr...@googlegroups.com.

> >> To post to this group, send email to
> >> common-workf...@googlegroups.com.
> >> To view this discussion on the web visit
> >> https://groups.google.com/d/msgid/common-workflow-
> language/5f74db50-1fc0-4cf1-b622-af5a8c9ac5a5%40googlegroups.com.
> >>
> >> For more options, visit https://groups.google.com/d/optout.
> >
> >
>
> --
> You received this message because you are subscribed to the Google Groups
> "common-workflow-language" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to common-workflow-language+unsubscr...@googlegroups.com.

> To post to this group, send email to common-workflow-
> lang...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/common-workflow-

> language/91ee3278-b3a5-1a8c-b5ae-462f0407a1f3%40sbgenomics.com.

> For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to a topic in the Google Groups "common-workflow-language" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/common-workflow-language/JU7PSEKr97M/unsubscribe.

To unsubscribe from this group and all its topics, send an email to common-workflow-language+unsubscr...@googlegroups.com.

To post to this group, send email to common-workf...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/common-workflow-language/b78efbba87f34144970da27712b844ea%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "common-workflow-language" group.

To unsubscribe from this group and stop receiving emails from it, send an email to common-workflow-language+unsubscr...@googlegroups.com.

To post to this group, send email to common-workf...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/common-workflow-language/CAD%3DV0dSdiXLhbDny5%2BjhMv1RoNqoVX3xtD%2BsbKTtoujfOwPnKg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

 

--

You received this message because you are subscribed to the Google Groups "common-workflow-language" group.
To unsubscribe from this group and stop receiving emails from it, send an email to common-workflow-language+unsub...@googlegroups.com.

To post to this group, send email to common-workflow-language@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/common-workflow-language/508c05c4-6078-41d5-99be-1dcfda2e5324%40googlegroups.com.

[Stian Soiland-Reyes]

Yes, it should be nested workflows if you need to do more complicated
stuff, that would show what happens at what level. As nested workflows
(and their steps) can be referenced or inlined it becomes more of a
style question when to split out or not.

Not sure if in CWL you can do the common Taverna trick of having the
outer step eat a whole array on an input (depth 1), and then iterate
over it inside the nested workflows (step with depth0)? I think that
should work.

Add loops/branching for a step, then that would allow you to have a
repeat..until loop over a whole list using conditionals inside - for
instance to segment a list into "good" or "bad" sublists with the
outer loop incrementally changing some threshold until good > bad.

Agree on requiring at least one iteration in while loop so you always
have outputs and don't need a start condition, as well as a forced
:else in any condition structure.

Btw, here is Clojure's condition syntax:

(defn blah [x y]
(str "x is "
(cond
(= x y) "equal"
(< x y) "smaller"
(> x y) "bigger"
:else "weird!" )))

Not a good example as people freak out over < and = being functions
rather than inline operators, but it shows how cond just takes a list
with boolean, possible return values. (Tip: to get "weird" try (blah
(/ 1 2) 0.5) as the fraction is not exactly equal to the float)

These are evaluated in order, so they don't have to be mutually
exclusive. In Clojure's the return of cond is a value, here becoming
an argument to the (str) concatenation function. If there is fall thru
and no :else you get nil returned from cond.

Now for CWL I think we would of course use a yaml list and not (s
expressions), but similarly evaluate the conditional steps in order
depending on the cond (they might have side effects) but force an
:else clause to avoid accidental null (but null could be explicit).

The left hand side would have to be either a boolean-like value (from
input parameter or candidate output parameter (null=false if this was
the first step to evaluate) or a Javascript expression (which can
utilise multiple parameters and arithmetic) that return a boolean. The
right hand side would be the "run" to do to populate candidate output
parameters, as normally defined under "step".

The steps need to have compatible inputs or outputs, but can be mapped
if needed (as is already possible on steps vs tool).

Could the :else step also be a map of hard coded outputs, or would
default values on declared output parameters make more sense, to be
like input defaults?


A good question is how this would look in YAml. Something like:


inputs:
doSomething: bool
outputs:
out:
type: string
outputSource: doit/out

steps:
doit:
requirements:
- class: ConditionalRequirement
in:
doSomething: doSomething
out: [out]
run:
cond:
if: inputs.doSomething
run: do-this.cwl
else:
run: do-that.cwl

ConditionalRequirement could then be expanded to use
InlineJavascriptRequirement and multiple conditions:

inputs:
name: string
outputs:
out:
type: string
outputSource: doit/out

steps:
doit:
requirements:
- class: ConditionalRequirement
- class: InlineJavascriptRequirement
in:
name: name
out: [out]
run:
cond:
- if: $(inputs.name == "Alice")
run: call-alice.cwl
- if: $(inputs.name == "Bob")
run: call-bob.cwl
else:
run:
class: ExpressionTool
outputs:
out: string
expression: { "out": "Sorry mate" }


(I might have got that ExpressionTool wrong)

Note the use of an ordered list under "cond".


On 22 Dec 2016 1:36 pm, "Nebojsa Tijanic"

>>> > >>> email to common-workflow-la...@googlegroups.com.

>>> > >>> To post to this group, send email to common-
>>> > workf...@googlegroups.com.
>>> > >>> To view this discussion on the web visit
>>> > >>> https://groups.google.com/d/msgid/common-workflow-
>>> > language/1412e3fd-e52f-418f-9259-025bca16afe6%40googlegroups.com.
>>> > >>> For more options, visit https://groups.google.com/d/optout.
>>> > >> --
>>> > >> You received this message because you are subscribed to the Google
>>> > Groups
>>> > >> "common-workflow-language" group.
>>> > >> To unsubscribe from this group and stop receiving emails from it, send
>>> > >> an

>>> > >> email to common-workflow-la...@googlegroups.com.

>>> > >> To post to this group, send email to
>>> > >> common-workf...@googlegroups.com.
>>> > >> To view this discussion on the web visit
>>> > >> https://groups.google.com/d/msgid/common-workflow-
>>> > language/5f74db50-1fc0-4cf1-b622-af5a8c9ac5a5%40googlegroups.com.
>>> > >>
>>> > >> For more options, visit https://groups.google.com/d/optout.
>>> > >
>>> > >
>>> >
>>> > --
>>> > You received this message because you are subscribed to the Google Groups
>>> > "common-workflow-language" group.
>>> > To unsubscribe from this group and stop receiving emails from it, send an

>>> > email to common-workflow-la...@googlegroups.com.

>>> > To post to this group, send email to common-workflow-
>>> > lang...@googlegroups.com.
>>> > To view this discussion on the web visit
>>> > https://groups.google.com/d/msgid/common-workflow-
>>>
>>> > language/91ee3278-b3a5-1a8c-b5ae-462f0407a1f3%40sbgenomics.com.
>>>
>>> > For more options, visit https://groups.google.com/d/optout.
>>>
>>> --
>>>
>>> You received this message because you are subscribed to a topic in the Google Groups "common-workflow-language" group.
>>> To unsubscribe from this topic, visit https://groups.google.com/d/topic/common-workflow-language/JU7PSEKr97M/unsubscribe.

>>> To unsubscribe from this group and all its topics, send an email to common-workflow-la...@googlegroups.com.

>>>
>>>
>>> To post to this group, send email to common-workf...@googlegroups.com.
>>>
>>> To view this discussion on the web visit https://groups.google.com/d/msgid/common-workflow-language/b78efbba87f34144970da27712b844ea%40mail.gmail.com.
>>>
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups "common-workflow-language" group.

>>> To unsubscribe from this group and stop receiving emails from it, send an email to common-workflow-la...@googlegroups.com.

>>> To post to this group, send email to common-workf...@googlegroups.com.
>>>
>>> To view this discussion on the web visit https://groups.google.com/d/msgid/common-workflow-language/CAD%3DV0dSdiXLhbDny5%2BjhMv1RoNqoVX3xtD%2BsbKTtoujfOwPnKg%40mail.gmail.com.
>>>
>>>
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups "common-workflow-language" group.

>> To unsubscribe from this group and stop receiving emails from it, send an email to common-workflow-la...@googlegroups.com.

>> To post to this group, send email to common-workf...@googlegroups.com.

>> To view this discussion on the web visit https://groups.google.com/d/msgid/common-workflow-language/508c05c4-6078-41d5-99be-1dcfda2e5324%40googlegroups.com.
>>
>> For more options, visit https://groups.google.com/d/optout.
>
>

> --
> You received this message because you are subscribed to the Google Groups "common-workflow-language" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to common-workflow-la...@googlegroups.com.

> To post to this group, send email to common-workf...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/common-workflow-language/CACnO1SGhFwOPY%3DjtQpcdoU%2BtoAYTYE%2BCtOTRCW1s8Bt%2B8RJAXQ%40mail.gmail.com.

[Paul Grosu]

Hi Stian,

In my opinion every symbol can be a defined as a function :)  So your examples are great!  Now how about having optional parameters for the CWL file that be utilized - basically adding or overriding the default ones.  You can then treat these CWL files as functions, which can push their outputs to custom environments if necessary.  Loops can actually become lazily infinite if one prefers to.  For example, one can run them like this, including a couple of nested examples with optional environments to push data into that can be referenced later:

run: |
     conditional.CWL
     case: $(inputs.name == "Alice"), run: call-alice.cwl
     case: $(inputs.name == "Bob")  , run: call-bob.cwl

run: |
     loop.CWL
     in: stdin , basecase: !NULL , run: print.CWL

run: |
     loop.CWL
     in: {1,2,3,4}, basecase: !EMPTY || 4, run: add1-and-print.CWL

run: |
     loop.CWL
     in: {1,2,3,4} , basecase: !EMPTY || 4 , run: add1-and-print.CWL
     out: |
          environment: add-collection

run: |
     loop.CWL
     in: [ environment: add-collection ]
     basecase: !EMPTY || 4
     run: add1-and-print.CWL
     environment: new-add-collection

run: |
     loop.CWL
     in: {"Alice" , "Bob", "Charlie"}
     basecase: !EMPTY
     run: |
          conditional.CWL
          case: $(value == "Alice"), run: call-alice.cwl
          case: $(value == "Bob"), run: call-bob.cwl

run: |
     conditional.CWL
     case: $(inputs.name == "Alice")
         run: |
              loop.CWL
              in: $(input.name.split(''))
              basecase: !EMPTY
              run: to-upper-case-AND-print.CWL
     case: $(inputs.name == "Bob")
         run: |
              loop.CWL
              in: $(input.name.split(''))
              basecase: !EMPTY
              run: to-lower-case-AND-print.CWL

What do you think?

~p

>>> > >>> email to common-workflow-language+unsub...@googlegroups.com.

>>> > >>> To post to this group, send email to common-
>>> > workf...@googlegroups.com.
>>> > >>> To view this discussion on the web visit
>>> > >>> https://groups.google.com/d/msgid/common-workflow-
>>> > language/1412e3fd-e52f-418f-9259-025bca16afe6%40googlegroups.com.
>>> > >>> For more options, visit https://groups.google.com/d/optout.
>>> > >> --
>>> > >> You received this message because you are subscribed to the Google
>>> > Groups
>>> > >> "common-workflow-language" group.
>>> > >> To unsubscribe from this group and stop receiving emails from it, send
>>> > >> an

>>> > >> email to common-workflow-language+unsub...@googlegroups.com.

>>> > >> To post to this group, send email to
>>> > >> common-workf...@googlegroups.com.
>>> > >> To view this discussion on the web visit
>>> > >> https://groups.google.com/d/msgid/common-workflow-
>>> > language/5f74db50-1fc0-4cf1-b622-af5a8c9ac5a5%40googlegroups.com.
>>> > >>
>>> > >> For more options, visit https://groups.google.com/d/optout.
>>> > >
>>> > >
>>> >
>>> > --
>>> > You received this message because you are subscribed to the Google Groups
>>> > "common-workflow-language" group.
>>> > To unsubscribe from this group and stop receiving emails from it, send an

>>> > email to common-workflow-language+unsub...@googlegroups.com.

>>> > To post to this group, send email to common-workflow-
>>> > lang...@googlegroups.com.
>>> > To view this discussion on the web visit
>>> > https://groups.google.com/d/msgid/common-workflow-
>>>
>>> > language/91ee3278-b3a5-1a8c-b5ae-462f0407a1f3%40sbgenomics.com.
>>>
>>> > For more options, visit https://groups.google.com/d/optout.
>>>
>>> --
>>>
>>> You received this message because you are subscribed to a topic in the Google Groups "common-workflow-language" group.
>>> To unsubscribe from this topic, visit https://groups.google.com/d/topic/common-workflow-language/JU7PSEKr97M/unsubscribe.

>>> To unsubscribe from this group and all its topics, send an email to common-workflow-language+unsub...@googlegroups.com.

>>>
>>>
>>> To post to this group, send email to common-workf...@googlegroups.com.
>>>
>>> To view this discussion on the web visit https://groups.google.com/d/msgid/common-workflow-language/b78efbba87f34144970da27712b844ea%40mail.gmail.com.
>>>
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups "common-workflow-language" group.

>>> To unsubscribe from this group and stop receiving emails from it, send an email to common-workflow-language+unsub...@googlegroups.com.

>>> To post to this group, send email to common-workf...@googlegroups.com.
>>>
>>> To view this discussion on the web visit https://groups.google.com/d/msgid/common-workflow-language/CAD%3DV0dSdiXLhbDny5%2BjhMv1RoNqoVX3xtD%2BsbKTtoujfOwPnKg%40mail.gmail.com.
>>>
>>>
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups "common-workflow-language" group.

>> To unsubscribe from this group and stop receiving emails from it, send an email to common-workflow-language+unsub...@googlegroups.com.

>> To post to this group, send email to common-workf...@googlegroups.com.
>> To view this discussion on the web visit https://groups.google.com/d/msgid/common-workflow-language/508c05c4-6078-41d5-99be-1dcfda2e5324%40googlegroups.com.
>>
>> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups "common-workflow-language" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to common-workflow-language+unsub...@googlegroups.com.

[Nebojsa Tijanic]

+1 to all Stian said except one: the requirements should be listed top-level not step-level. Step requirements combine with impl. requirements; it doesn't make much sense to have feature requirements for a subset of steps I think.

Could the :else step also be a map of hard coded outputs, or would
default values on declared output parameters make more sense, to be
like input defaults? 

Since it can be done with ExpressionTool (as you did), I think this is "sugar" that can be added independently either by allowing an object (without "class" property) as target for "run" property or by introducing a "result" property that is mutually exclusive with "run".

Not sure if in CWL you can do the common Taverna trick of having the
outer step eat a whole array on an input (depth 1), and then iterate
over it inside the nested workflows (step with depth0)?  I think that
should work.

Yes, that works.

@Paul I'm sorry, I don't understand your notation. You may be trying to introduce too many new concepts in a single example for me to handle :(

>>> > >>> email to common-workflow-language+unsubscr...@googlegroups.com.

>>> > >>> To post to this group, send email to common-
>>> > workf...@googlegroups.com.
>>> > >>> To view this discussion on the web visit
>>> > >>> https://groups.google.com/d/msgid/common-workflow-
>>> > language/1412e3fd-e52f-418f-9259-025bca16afe6%40googlegroups.com.
>>> > >>> For more options, visit https://groups.google.com/d/optout.
>>> > >> --
>>> > >> You received this message because you are subscribed to the Google
>>> > Groups
>>> > >> "common-workflow-language" group.
>>> > >> To unsubscribe from this group and stop receiving emails from it, send
>>> > >> an

>>> > >> email to common-workflow-language+unsubscr...@googlegroups.com.

>>> > >> To post to this group, send email to
>>> > >> common-workf...@googlegroups.com.
>>> > >> To view this discussion on the web visit
>>> > >> https://groups.google.com/d/msgid/common-workflow-
>>> > language/5f74db50-1fc0-4cf1-b622-af5a8c9ac5a5%40googlegroups.com.
>>> > >>
>>> > >> For more options, visit https://groups.google.com/d/optout.
>>> > >
>>> > >
>>> >
>>> > --
>>> > You received this message because you are subscribed to the Google Groups
>>> > "common-workflow-language" group.
>>> > To unsubscribe from this group and stop receiving emails from it, send an

>>> > email to common-workflow-language+unsubscr...@googlegroups.com.

>>> > To post to this group, send email to common-workflow-
>>> > lang...@googlegroups.com.
>>> > To view this discussion on the web visit
>>> > https://groups.google.com/d/msgid/common-workflow-
>>>
>>> > language/91ee3278-b3a5-1a8c-b5ae-462f0407a1f3%40sbgenomics.com.
>>>
>>> > For more options, visit https://groups.google.com/d/optout.
>>>
>>> --
>>>
>>> You received this message because you are subscribed to a topic in the Google Groups "common-workflow-language" group.
>>> To unsubscribe from this topic, visit https://groups.google.com/d/topic/common-workflow-language/JU7PSEKr97M/unsubscribe.

>>> To unsubscribe from this group and all its topics, send an email to common-workflow-language+unsubscr...@googlegroups.com.

>>>
>>>
>>> To post to this group, send email to common-workf...@googlegroups.com.
>>>
>>> To view this discussion on the web visit https://groups.google.com/d/msgid/common-workflow-language/b78efbba87f34144970da27712b844ea%40mail.gmail.com.
>>>
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups "common-workflow-language" group.

>>> To unsubscribe from this group and stop receiving emails from it, send an email to common-workflow-language+unsubscr...@googlegroups.com.

>>> To post to this group, send email to common-workf...@googlegroups.com.
>>>
>>> To view this discussion on the web visit https://groups.google.com/d/msgid/common-workflow-language/CAD%3DV0dSdiXLhbDny5%2BjhMv1RoNqoVX3xtD%2BsbKTtoujfOwPnKg%40mail.gmail.com.
>>>
>>>
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups "common-workflow-language" group.

>> To unsubscribe from this group and stop receiving emails from it, send an email to common-workflow-language+unsubscr...@googlegroups.com.

>> To post to this group, send email to common-workf...@googlegroups.com.

>> To view this discussion on the web visit https://groups.google.com/d/msgid/common-workflow-language/508c05c4-6078-41d5-99be-1dcfda2e5324%40googlegroups.com.
>>
>> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups "common-workflow-language" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to common-workflow-language+unsubscr...@googlegroups.com.

[Alan R Williams]

Hello everyone

For the looping the examples have a single input list that is iterated
over (or are they run in parallel?). Is there support for handling
multiple lists e.g. with the equivalent of a Python zip?

Thanks

Alan

[Peter Amstutz]

Hi Alan,

CWL already specifies a parallel scatter operation which includes both
dot product and cross product options for working with multiple input
arrays.

The feature being discussed here is a sequential looping construct for
situations where it is necessary to feed the output of a previous
iteration into the next iteration.

Thanks,
Peter

> --
> You received this message because you are subscribed to the Google Groups
> "common-workflow-language" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to common-workflow-la...@googlegroups.com.

> To post to this group, send email to
> common-workf...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/common-workflow-language/9a5ee51c-293b-d8f0-149e-0e93ba7dd22e%40manchester.ac.uk.

[Nebojsa Tijanic]

Alan,

specifically, you want to set the "scatterMethod" step property to "dotproduct" and list both ports in the "scatter" property, in order: http://www.commonwl.org/v1.0/Workflow.html#WorkflowStep

On Fri, Dec 23, 2016 at 4:59 PM, Peter Amstutz <peter....@curoverse.com> wrote:

Hi Alan,

CWL already specifies a parallel scatter operation which includes both
dot product and cross product options for working with multiple input
arrays.

The feature being discussed here is a sequential looping construct for
situations where it is necessary to feed the output of a previous
iteration into the next iteration.

Thanks,
Peter

On Fri, Dec 23, 2016 at 10:46 AM, Alan R Williams
<alan.r.williams@manchester.ac.uk> wrote:
> Hello everyone
>
> For the looping the examples have a single input list that is iterated over
> (or are they run in parallel?). Is there support for handling multiple lists
> e.g. with the equivalent of a Python zip?
>
> Thanks
>
> Alan
>
> --
> You received this message because you are subscribed to the Google Groups
> "common-workflow-language" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to common-workflow-language+unsub...@googlegroups.com.

> To post to this group, send email to

> common-workflow-language@googlegroups.com.

> To view this discussion on the web visit
> https://groups.google.com/d/msgid/common-workflow-language/9a5ee51c-293b-d8f0-149e-0e93ba7dd22e%40manchester.ac.uk.
>
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "common-workflow-language" group.

To unsubscribe from this group and stop receiving emails from it, send an email to common-workflow-language+unsub...@googlegroups.com.
To post to this group, send email to common-workflow-language@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/common-workflow-language/CAEXjzRsAt9%3Dv5amhV%3DwjVk3sFH5qpRo4hOyVUFB3kSsMg4L-ZQ%40mail.gmail.com.

[Alan R Williams]

On 23-Dec-16 15:59, Peter Amstutz wrote:
> Hi Alan,
>
> CWL already specifies a parallel scatter operation which includes both
> dot product and cross product options for working with multiple input
> arrays.
>
> The feature being discussed here is a sequential looping construct for
> situations where it is necessary to feed the output of a previous
> iteration into the next iteration.

Thanks Peter and Nebosa

> Thanks,
> Peter

Alan

[Paul Grosu]

Sorry to reply so late - the Holidays are keeping me a bit busy :)  I hope everyone is also enjoying a Happy Holiday Season, and will have a Happy New Year!

Nebojsa, you wanted a bit more clarification on what I was proposing as an extension to Stian's approach.  So for that let me define CWL in terms of some non-scary mathematical definitions, without getting too abstract.  That will allow us to augment the language dynamically with what I was suggesting that can take place, in order to make the language even more flexible.  If anything I wrote might seem redundant, I just included it for completeness and please feel free to ask any questions if something is confusing.

I'll start with the grammar of a Workflow, which can be either a Step or a collection of Steps using BNF (Backus-Naur Form):

  <Workflow> ::= <Step>
               | ListOf<Step>

The term ListOf is defined as follows, with the symbol ≡ denoting that it is identical:

  ListOf ≡ Set (Collection) of Ordered Steps
         ≡ {Step_1, Step_2, ..., Step_n}

Next I'll define a Step as follows:

  <Step> ::= <Expression>
           | <Command>
           | ListOf<Step>

But to simplify, we know that a Function will take an Input type and produce an Output type, which can be defined as follows - where these types can be void:

  (Input_Type -> Output_Type)

Thus an Expression and a Command are elements of the set of Function, which is a supertype - the symbol (∈) denotes that the left element is part of the set on the right:

  Expression ∈ Function

     Command ∈ Function

But a Function has the following composability properties:

  Function_2( Function_1( ListOf<Input> ) )

  Function_1( ListOf<Input> ).Function_2()

But Function_1(Input) is also a composition, which can be written as follows:

   ListOf<Input>.Function_1()

  Function_1 ° ListOf<Input>

The second definition is more powerful using the composition (°) symbol, in that Input can actually become a function via the Identity function, which just returns the same contents as Input:

  Function_1 ° Identity( ListOf<Input> )

This is evaluated right-to-left - read as Function_1 after Identity - where the Output of the Identity function will be the Input to Function_1.  This now allows for also associativity and the identity functions to be performed.  Associativity allows composable steps to be run in any grouping and Identity allows the same (self) object to be returned, such as:

  Function_3 ° (Function_2 ° Function_1) ≡ (Function_3 ° Function_2) ° Function_1

                              Function_1 ≡ Identity(Function_1)

This allows us to build and guarantee Directed Acyclic Graphs (DAGs) constructs of the language using the composability, associativity and identity operations! 

Now we can define the previous construct even more completely as a function, as follows:

  ( (Function_1 ° Identity( ListOf<Input> )) -> ListOf<Output> )

We now have a definition of a Step as a function!  This allows us to expand an instance of Workflow to the following:

  Workflow_instance_1 = { Step_1_Function ° Input_1    -> Step_1_out, 
                          Step_2_Function ° Step_1_out -> Step_2_out,
                          Step_3_Function ° Step_2_out -> Step_3_out  }

This now allows us to go-back-in-time if we want to, and re-run the same Workflow instance with modification - using the memoized previous run - without duplicating any of the previous steps, as follows:

  Workflow_instance_2 = { Workflow_instance_1,
                          Modified_Step_3_Function ° step_3_out -> step_3_out }

  Workflow_instance_2 = { Workflow_instance_1,
                          Modified_Step_3_Function ° Workflow_instance_1[step_3_out] -> step_3_out }

Since an Expression and a Command are elements of the Function category, then a collection of Steps are also a member of Function - since a collection of steps have a composition - which start with an Input and produce an Output.  So we can now add the following three definitions to our language:

          Step ∈ Function
  ListOf<Step> ∈ Function
      Workflow ∈ Function

Now a Workflow is just a function!  But a Workflow is just a CWL file, now with an overridable Input.  This means that we can augment a Workflow with an Input as follows (either as a file or by overridding):

   Workflow ° Input -> Output

Now Input can be defined as a collection of cases (in terms of a Conditional) as such - where Workflow can also be redefined as a CWL file, which understands how to process the format of the appropriate Input:

      Workflow ° [ Case: <test> then Run: Case1.CWL,
                   Case: <test> then Run: Case1.CWL ] -> Output

      Condition.CWL ° [ Case: <test1> then Run: Case1.CWL,
                        Case: <test2> then Run: Case2.CWL ] -> Output

Or in the case of a Loop, that can be the following:

      Workflow ° [ Input: {1,2,3,4},
                   BaseCase: IteratedInput != EMPTY,
                   Run: Add1_to_Each_Iteration.CWL ] -> Output

      Loop.CWL ° [ Input: {1,2,3,4},
                   BaseCase: IteratedInput != EMPTY,
                   Run: Add1_to_Each_Iteration.CWL ] -> Output

But now we can compose Workflows, which is given to us for free by the definition!  This means that you can compose Loops, Conditionals, and any other language constructs to make it as ubiquitous or as domain-specific as you like for all applications:

      Loop.CWL ° [ Input: {1,2,3,4},
                   BaseCase: IteratedInput != EMPTY,
                   Run: Conditional.CWL] °
                     [ Case: <test> then Run: Case1.CWL,
                       Case: <test> then Run: Case1.CWL] -> Output
     
      Condition.CWL ° [ Case: $(inputs.name == "Alice") then Run: Loop.CWL,
                        Case: $(inputs.name == "Bob") then Run: Loop.CWL ] °
                          (
                            [ Input: $(input.name.split('')),
                              BaseCase: IteratedInput != EMPTY,
                              Run: To-Upper-Case-AND-Print.CWL ] ,

                            [ Input: $(input.name.split('')),
                              BaseCase: IteratedInput != EMPTY,
                              Run: To-Lower-Case-AND-Print.CWL ]
                          ) -> Output

These will naturally transform appropriately, though inlining is also possible but might make things less readable. 

This now makes type construction a little bit simpler, and the evaluation rules come for free making type-checking automatic under this construct.  It also allows us to guarantee Directed Acyclic Graphs (DAGs) constructs of the language using the composability, associativity and identity operations! 

There are even more powerful generalizations of abstractions that can be made to the language - while still remaining strongly-typed - but I will leave that for another time, as it will dilute the message. 

Again you can perform what I just did for all workflow languages, and you'll see that they all will pretty much simplify to very simple definitions.  The key important part is how strongly generalizable can we make CWL, in order to have it be common and re-definable to all domains.

Please feel free to ask anything in case I still left something that is confusing.

Hope this helps and Happy New Year to Everyone!
Paul

>>> > >>> email to common-workflow-language+unsub...@googlegroups.com.

>>> > >>> To post to this group, send email to common-
>>> > workf...@googlegroups.com.
>>> > >>> To view this discussion on the web visit
>>> > >>> https://groups.google.com/d/msgid/common-workflow-
>>> > language/1412e3fd-e52f-418f-9259-025bca16afe6%40googlegroups.com.
>>> > >>> For more options, visit https://groups.google.com/d/optout.
>>> > >> --
>>> > >> You received this message because you are subscribed to the Google
>>> > Groups
>>> > >> "common-workflow-language" group.
>>> > >> To unsubscribe from this group and stop receiving emails from it, send
>>> > >> an

>>> > >> email to common-workflow-language+unsub...@googlegroups.com.

>>> > >> To post to this group, send email to
>>> > >> common-workf...@googlegroups.com.
>>> > >> To view this discussion on the web visit
>>> > >> https://groups.google.com/d/msgid/common-workflow-
>>> > language/5f74db50-1fc0-4cf1-b622-af5a8c9ac5a5%40googlegroups.com.
>>> > >>
>>> > >> For more options, visit https://groups.google.com/d/optout.
>>> > >
>>> > >
>>> >
>>> > --
>>> > You received this message because you are subscribed to the Google Groups
>>> > "common-workflow-language" group.
>>> > To unsubscribe from this group and stop receiving emails from it, send an

>>> > email to common-workflow-language+unsub...@googlegroups.com.

>>> > To post to this group, send email to common-workflow-
>>> > lang...@googlegroups.com.
>>> > To view this discussion on the web visit
>>> > https://groups.google.com/d/msgid/common-workflow-
>>>
>>> > language/91ee3278-b3a5-1a8c-b5ae-462f0407a1f3%40sbgenomics.com.
>>>
>>> > For more options, visit https://groups.google.com/d/optout.
>>>
>>> --
>>>
>>> You received this message because you are subscribed to a topic in the Google Groups "common-workflow-language" group.
>>> To unsubscribe from this topic, visit https://groups.google.com/d/topic/common-workflow-language/JU7PSEKr97M/unsubscribe.

>>> To unsubscribe from this group and all its topics, send an email to common-workflow-language+unsub...@googlegroups.com.

>>>
>>>
>>> To post to this group, send email to common-workf...@googlegroups.com.
>>>
>>> To view this discussion on the web visit https://groups.google.com/d/msgid/common-workflow-language/b78efbba87f34144970da27712b844ea%40mail.gmail.com.
>>>
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups "common-workflow-language" group.

>>> To unsubscribe from this group and stop receiving emails from it, send an email to common-workflow-language+unsub...@googlegroups.com.

>>> To post to this group, send email to common-workf...@googlegroups.com.
>>>
>>> To view this discussion on the web visit https://groups.google.com/d/msgid/common-workflow-language/CAD%3DV0dSdiXLhbDny5%2BjhMv1RoNqoVX3xtD%2BsbKTtoujfOwPnKg%40mail.gmail.com.
>>>
>>>
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups "common-workflow

...

[Michael Miller]

hi paul,

 

this is a nice approach but i'm trying to work my head around it.  one thing that would help is that you start with BNF but then wander off into transformations that lead to another set of operations and symbols than BNF.  it would be nice to wander back to the BNF notation to see how the loop and conditional would look with that notation.

"It also allows us to guarantee Directed Acyclic Graphs (DAGs) constructs of the language using the composability, associativity and identity operations!"

typically workflows in a loop would not be represented as a DAG, usually it would be represented as:

are you saying, instead, that each 'loop' is actually unwound, to something like this?

 

also, since the top level rule in the BNF is step followed by step, i don't quite understand how scatter and gather branches would be represented, such as in this taverna pathways_and_gene_annotations_forqtl_region_290738.t2flow workflow?  could you give an example (doesn't have to be this particular workflow)?

 

thanks,

>>> > >>> email to common-workflow-la...@googlegroups.com.

>>> > >>> To post to this group, send email to common-
>>> > workf...@googlegroups.com.
>>> > >>> To view this discussion on the web visit
>>> > >>> https://groups.google.com/d/msgid/common-workflow-
>>> > language/1412e3fd-e52f-418f-9259-025bca16afe6%40googlegroups.com.
>>> > >>> For more options, visit https://groups.google.com/d/optout.
>>> > >> --
>>> > >> You received this message because you are subscribed to the Google
>>> > Groups
>>> > >> "common-workflow-language" group.
>>> > >> To unsubscribe from this group and stop receiving emails from it, send
>>> > >> an

>>> > >> email to common-workflow-la...@googlegroups.com.

>>> > >> To post to this group, send email to
>>> > >> common-workf...@googlegroups.com.
>>> > >> To view this discussion on the web visit
>>> > >> https://groups.google.com/d/msgid/common-workflow-
>>> > language/5f74db50-1fc0-4cf1-b622-af5a8c9ac5a5%40googlegroups.com.
>>> > >>
>>> > >> For more options, visit https://groups.google.com/d/optout.
>>> > >
>>> > >
>>> >
>>> > --
>>> > You received this message because you are subscribed to the Google Groups
>>> > "common-workflow-language" group.
>>> > To unsubscribe from this group and stop receiving emails from it, send an

>>> > email to common-workflow-la...@googlegroups.com.

>>> > To post to this group, send email to common-workflow-
>>> > lang...@googlegroups.com.
>>> > To view this discussion on the web visit
>>> > https://groups.google.com/d/msgid/common-workflow-
>>>
>>> > language/91ee3278-b3a5-1a8c-b5ae-462f0407a1f3%40sbgenomics.com.
>>>
>>> > For more options, visit https://groups.google.com/d/optout.
>>>
>>> --
>>>
>>> You received this message because you are subscribed to a topic in the Google Groups "common-workflow-language" group.
>>> To unsubscribe from this topic, visit https://groups.google.com/d/topic/common-workflow-language/JU7PSEKr97M/unsubscribe.

>>> To unsubscribe from this group and all its topics, send an email to common-workflow-la...@googlegroups.com.

>>>
>>>
>>> To post to this group, send email to common-workf...@googlegroups.com.
>>>
>>> To view this discussion on the web visit https://groups.google.com/d/msgid/common-workflow-language/b78efbba87f34144970da27712b844ea%40mail.gmail.com.
>>>
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups "common-workflow-language" group.

>>> To unsubscribe from this group and stop receiving emails from it, send an email to common-workflow-la...@googlegroups.com.

>>> To post to this group, send email to common-workf...@googlegroups.com.
>>>
>>> To view this discussion on the web visit https://groups.google.com/d/msgid/common-workflow-language/CAD%3DV0dSdiXLhbDny5%2BjhMv1RoNqoVX3xtD%2BsbKTtoujfOwPnKg%40mail.gmail.com.
>>>
>>>
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups "common-workflow

...

--

You received this message because you are subscribed to the Google Groups "common-workflow-language" group.

To unsubscribe from this group and stop receiving emails from it, send an email to common-workflow-la...@googlegroups.com.

To post to this group, send email to common-workf...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/common-workflow-language/487785e9-3658-44aa-8f2e-98e53784a436%40googlegroups.com.

[Jeff Gentry]

Hi Michael -

When we were designing WDL we decided to view things like loops & scatters as a single node in the DAG which would then be unrolled/exploded dynamically at runtime, much like you're describing here. So e.g. with a loop if you view the loop as a single node you can still have a DAG, and then once the workflow is complete you'd have a final DAG, but you don't necessarily have a DAG in the interim. In a sense it is like how tail recursion can always be transformed into a for loop.

We don't yet support loops in Cromwell but for instance with scatters that's exactly what we do, the scatter is a single node in the graph until it is triggered at which the shards appear in the workflow graph dynamically.

I believe that's what you're saying here and IMO it's the way to view it. if I'm remembering the SBG Bunny paper correctly that's also more or less what they've done with their graph processing.

J

>>> > >>> email to common-workflow-language+unsub...@googlegroups.com.

>>> > >>> To post to this group, send email to common-
>>> > workf...@googlegroups.com.
>>> > >>> To view this discussion on the web visit
>>> > >>> https://groups.google.com/d/msgid/common-workflow-
>>> > language/1412e3fd-e52f-418f-9259-025bca16afe6%40googlegroups.com.
>>> > >>> For more options, visit https://groups.google.com/d/optout.
>>> > >> --
>>> > >> You received this message because you are subscribed to the Google
>>> > Groups
>>> > >> "common-workflow-language" group.
>>> > >> To unsubscribe from this group and stop receiving emails from it, send
>>> > >> an

>>> > >> email to common-workflow-language+unsub...@googlegroups.com.

>>> > >> To post to this group, send email to
>>> > >> common-workf...@googlegroups.com.
>>> > >> To view this discussion on the web visit
>>> > >> https://groups.google.com/d/msgid/common-workflow-
>>> > language/5f74db50-1fc0-4cf1-b622-af5a8c9ac5a5%40googlegroups.com.
>>> > >>
>>> > >> For more options, visit https://groups.google.com/d/optout.
>>> > >
>>> > >
>>> >
>>> > --
>>> > You received this message because you are subscribed to the Google Groups
>>> > "common-workflow-language" group.
>>> > To unsubscribe from this group and stop receiving emails from it, send an

>>> > email to common-workflow-language+unsub...@googlegroups.com.

>>> > To post to this group, send email to common-workflow-
>>> > lang...@googlegroups.com.
>>> > To view this discussion on the web visit
>>> > https://groups.google.com/d/msgid/common-workflow-
>>>
>>> > language/91ee3278-b3a5-1a8c-b5ae-462f0407a1f3%40sbgenomics.com.
>>>
>>> > For more options, visit https://groups.google.com/d/optout.
>>>
>>> --
>>>
>>> You received this message because you are subscribed to a topic in the Google Groups "common-workflow-language" group.
>>> To unsubscribe from this topic, visit https://groups.google.com/d/topic/common-workflow-language/JU7PSEKr97M/unsubscribe.

>>> To unsubscribe from this group and all its topics, send an email to common-workflow-language+unsub...@googlegroups.com.

>>>
>>>
>>> To post to this group, send email to common-workf...@googlegroups.com.
>>>
>>> To view this discussion on the web visit https://groups.google.com/d/msgid/common-workflow-language/b78efbba87f34144970da27712b844ea%40mail.gmail.com.
>>>
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups "common-workflow-language" group.

>>> To unsubscribe from this group and stop receiving emails from it, send an email to common-workflow-language+unsub...@googlegroups.com.

>>> To post to this group, send email to common-workf...@googlegroups.com.
>>>
>>> To view this discussion on the web visit https://groups.google.com/d/msgid/common-workflow-language/CAD%3DV0dSdiXLhbDny5%2BjhMv1RoNqoVX3xtD%2BsbKTtoujfOwPnKg%40mail.gmail.com.
>>>
>>>
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups "common-workflow

...

--
You received this message because you are subscribed to the Google Groups "common-workflow-language" group.

To unsubscribe from this group and stop receiving emails from it, send an email to common-workflow-language+unsub...@googlegroups.com.
To post to this group, send email to common-workflow-language@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/common-workflow-language/487785e9-3658-44aa-8f2e-98e53784a436%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "common-workflow-language" group.

To unsubscribe from this group and stop receiving emails from it, send an email to common-workflow-language+unsub...@googlegroups.com.
To post to this group, send email to common-workflow-language@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/common-workflow-language/966678ed35d4deba5e450e137a478fd7%40mail.gmail.com.

[Peter Amstutz]

Hello everyone, thanks for your input!

Here's some more design ideas for conditionals and loops inspired from the conversation.

First, conditionals:

>>>
steps:
  step1:
    in:
      a: a
    out: [out]

    cond:

      - case: $(inputs.a > 1)
        run: greater.cwl
      - case: $(inputs.a < 1)
        run: less.cwl
      - case: default
        run: same.cwl
<<<

A conditional step has a "cond" field.  It lists one or more cases and the process that should be run if the condition is true.  The "default" case is executed if no other case matches.  Provided the cases are orthogonal (i.e. not order dependent), this can also be rolled up into a more compact syntax:

>>>
steps:
  step1:
    in:
      a: a
    out: [out]

    cond:

      "$(inputs.a > 1)": greater.cwl
      "$(inputs.a < 1)": less.cwl
      default: same.cwl
<<<

A couple points of naming:
- would consider "if" or "switch" instead of "cond"
- would consider "else" or "true" instead of "default"

Next, loops:

>>>

steps:
  step2:
    in:
      a: a

    out: [out]

    while: $(inputs.a > 1)
    run: blah.cwl

    next:

      - dest: a
        valueFrom: $(self.out)

    default:

      out: "the base case"

<<<

The "while" field is the loop condition.  The "next" field adjusts the input object going into the next iteration of the loop based on current iteration ("self" is bound to the output of the run).  The "default" field provides a base case output object if the loop is never entered (i.e. the loop condition is initially false).

The compact syntax version of "next" wolud be:

>>>

    next:

      a: $(self.out)

<<<

The spec currently has a single type "WorkflowStep" with optional scatter.  I propose making "WorkflowStep" abstract, and having several subtypes: SimpleStep, ScatterStep, ConditionalStep, and LoopStep.  So, a workflow step could be one of those four types.  Nesting (such as a condition inside a loop inside a scatter) may be accomplished using subworkflows.

I haven't started prototyping this yet, but I feel like this design strikes about the right balance between simplicity and expressiveness.

Thanks,

Peter

On Sat, Jan 7, 2017 at 2:28 PM, Jeff Gentry <jge...@broadinstitute.org> wrote:

Hi Michael -

When we were designing WDL we decided to view things like loops & scatters as a single node in the DAG which would then be unrolled/exploded dynamically at runtime, much like you're describing here. So e.g. with a loop if you view the loop as a single node you can still have a DAG, and then once the workflow is complete you'd have a final DAG, but you don't necessarily have a DAG in the interim. In a sense it is like how tail recursion can always be transformed into a for loop.

We don't yet support loops in Cromwell but for instance with scatters that's exactly what we do, the scatter is a single node in the graph until it is triggered at which the shards appear in the workflow graph dynamically.

I believe that's what you're saying here and IMO it's the way to view it. if I'm remembering the SBG Bunny paper correctly that's also more or less what they've done with their graph processing.

J

>>> > >>> email to common-workflow-language+unsubscr...@googlegroups.com.

>>> > >>> To post to this group, send email to common-
>>> > workf...@googlegroups.com.
>>> > >>> To view this discussion on the web visit
>>> > >>> https://groups.google.com/d/msgid/common-workflow-
>>> > language/1412e3fd-e52f-418f-9259-025bca16afe6%40googlegroups.com.
>>> > >>> For more options, visit https://groups.google.com/d/optout.
>>> > >> --
>>> > >> You received this message because you are subscribed to the Google
>>> > Groups
>>> > >> "common-workflow-language" group.
>>> > >> To unsubscribe from this group and stop receiving emails from it, send
>>> > >> an

>>> > >> email to common-workflow-language+unsubscr...@googlegroups.com.

>>> > >> To post to this group, send email to
>>> > >> common-workf...@googlegroups.com.
>>> > >> To view this discussion on the web visit
>>> > >> https://groups.google.com/d/msgid/common-workflow-
>>> > language/5f74db50-1fc0-4cf1-b622-af5a8c9ac5a5%40googlegroups.com.
>>> > >>
>>> > >> For more options, visit https://groups.google.com/d/optout.
>>> > >
>>> > >
>>> >
>>> > --
>>> > You received this message because you are subscribed to the Google Groups
>>> > "common-workflow-language" group.
>>> > To unsubscribe from this group and stop receiving emails from it, send an

>>> > email to common-workflow-language+unsubscr...@googlegroups.com.

>>> > To post to this group, send email to common-workflow-
>>> > lang...@googlegroups.com.
>>> > To view this discussion on the web visit
>>> > https://groups.google.com/d/msgid/common-workflow-
>>>
>>> > language/91ee3278-b3a5-1a8c-b5ae-462f0407a1f3%40sbgenomics.com.
>>>
>>> > For more options, visit https://groups.google.com/d/optout.
>>>
>>> --
>>>
>>> You received this message because you are subscribed to a topic in the Google Groups "common-workflow-language" group.
>>> To unsubscribe from this topic, visit https://groups.google.com/d/topic/common-workflow-language/JU7PSEKr97M/unsubscribe.

>>> To unsubscribe from this group and all its topics, send an email to common-workflow-language+unsubscr...@googlegroups.com.

>>>
>>>
>>> To post to this group, send email to common-workf...@googlegroups.com.
>>>
>>> To view this discussion on the web visit https://groups.google.com/d/msgid/common-workflow-language/b78efbba87f34144970da27712b844ea%40mail.gmail.com.
>>>
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups "common-workflow-language" group.

>>> To unsubscribe from this group and stop receiving emails from it, send an email to common-workflow-language+unsubscr...@googlegroups.com.

>>> To post to this group, send email to common-workf...@googlegroups.com.
>>>
>>> To view this discussion on the web visit https://groups.google.com/d/msgid/common-workflow-language/CAD%3DV0dSdiXLhbDny5%2BjhMv1RoNqoVX3xtD%2BsbKTtoujfOwPnKg%40mail.gmail.com.
>>>
>>>
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups "common-workflow

...

--
You received this message because you are subscribed to the Google Groups "common-workflow-language" group.

To unsubscribe from this group and stop receiving emails from it, send an email to common-workflow-language+unsubscr...@googlegroups.com.

To post to this group, send email to common-workflow-language@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/common-workflow-language/487785e9-3658-44aa-8f2e-98e53784a436%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "common-workflow-language" group.

To unsubscribe from this group and stop receiving emails from it, send an email to common-workflow-language+unsubscr...@googlegroups.com.

To post to this group, send email to common-workflow-language@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/common-workflow-language/966678ed35d4deba5e450e137a478fd7%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "common-workflow-language" group.
To unsubscribe from this group and stop receiving emails from it, send an email to common-workflow-language+unsub...@googlegroups.com.
To post to this group, send email to common-workflow-language@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/common-workflow-language/CA%2BBw4v16fq-CGLP9EquvBs8-hQ_3j5_SZDWf6j6uJyFcPiaw7A%40mail.gmail.com.

[Michael Miller]

hi jeff,

 

i hadn't thought of tail recursion, but that is right on

 

cheers,

michael

 

From: Jeff Gentry [mailto:jge...@broadinstitute.org]
Sent: Saturday, January 07, 2017 11:28 AM
To: Michael Miller
Cc: Paul Grosu; Luka Stojanovic; karl.no...@gmail.com; common-workflow-language
Subject: Re: conditionals & loops

 

Hi Michael -

>>> > >>> email to common-workflow-la...@googlegroups.com.

>>> > >>> To post to this group, send email to common-
>>> > workf...@googlegroups.com.
>>> > >>> To view this discussion on the web visit
>>> > >>> https://groups.google.com/d/msgid/common-workflow-
>>> > language/1412e3fd-e52f-418f-9259-025bca16afe6%40googlegroups.com.
>>> > >>> For more options, visit https://groups.google.com/d/optout.
>>> > >> --
>>> > >> You received this message because you are subscribed to the Google
>>> > Groups
>>> > >> "common-workflow-language" group.
>>> > >> To unsubscribe from this group and stop receiving emails from it, send
>>> > >> an

>>> > >> email to common-workflow-la...@googlegroups.com.

>>> > >> To post to this group, send email to
>>> > >> common-workf...@googlegroups.com.
>>> > >> To view this discussion on the web visit
>>> > >> https://groups.google.com/d/msgid/common-workflow-
>>> > language/5f74db50-1fc0-4cf1-b622-af5a8c9ac5a5%40googlegroups.com.
>>> > >>
>>> > >> For more options, visit https://groups.google.com/d/optout.
>>> > >
>>> > >
>>> >
>>> > --
>>> > You received this message because you are subscribed to the Google Groups
>>> > "common-workflow-language" group.
>>> > To unsubscribe from this group and stop receiving emails from it, send an

>>> > email to common-workflow-la...@googlegroups.com.

>>> > To post to this group, send email to common-workflow-
>>> > lang...@googlegroups.com.
>>> > To view this discussion on the web visit
>>> > https://groups.google.com/d/msgid/common-workflow-
>>>
>>> > language/91ee3278-b3a5-1a8c-b5ae-462f0407a1f3%40sbgenomics.com.
>>>
>>> > For more options, visit https://groups.google.com/d/optout.
>>>
>>> --
>>>
>>> You received this message because you are subscribed to a topic in the Google Groups "common-workflow-language" group.
>>> To unsubscribe from this topic, visit https://groups.google.com/d/topic/common-workflow-language/JU7PSEKr97M/unsubscribe.

>>> To unsubscribe from this group and all its topics, send an email to common-workflow-la...@googlegroups.com.

>>>
>>>
>>> To post to this group, send email to common-workf...@googlegroups.com.
>>>
>>> To view this discussion on the web visit https://groups.google.com/d/msgid/common-workflow-language/b78efbba87f34144970da27712b844ea%40mail.gmail.com.
>>>
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups "common-workflow-language" group.

>>> To unsubscribe from this group and stop receiving emails from it, send an email to common-workflow-la...@googlegroups.com.

>>> To post to this group, send email to common-workf...@googlegroups.com.
>>>
>>> To view this discussion on the web visit https://groups.google.com/d/msgid/common-workflow-language/CAD%3DV0dSdiXLhbDny5%2BjhMv1RoNqoVX3xtD%2BsbKTtoujfOwPnKg%40mail.gmail.com.
>>>
>>>
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups "common-workflow

...

--
You received this message because you are subscribed to the Google Groups "common-workflow-language" group.

To unsubscribe from this group and stop receiving emails from it, send an email to common-workflow-la...@googlegroups.com.

To post to this group, send email to common-workf...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/common-workflow-language/487785e9-3658-44aa-8f2e-98e53784a436%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "common-workflow-language" group.

To unsubscribe from this group and stop receiving emails from it, send an email to common-workflow-la...@googlegroups.com.

To post to this group, send email to common-workf...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/common-workflow-language/966678ed35d4deba5e450e137a478fd7%40mail.gmail.com.

[Michael Miller]

hi peter,

 

this looks good.  i think there could also be a non-abstract nested WorkflowStep for embedded workflows.  i imagine this all would be part of CWL 2.0?

 

cheers,

michael

 

From: Peter Amstutz [mailto:peter....@curoverse.com]
Sent: Sunday, January 08, 2017 6:48 PM
To: Jeff Gentry
Cc: Michael Miller; Paul Grosu; Luka Stojanovic; Karl Nordström; common-workflow-language
Subject: Re: conditionals & loops

 

Hello everyone, thanks for your input!

>>> > >>> email to common-workflow-la...@googlegroups.com.

>>> > >>> To post to this group, send email to common-
>>> > workf...@googlegroups.com.
>>> > >>> To view this discussion on the web visit
>>> > >>> https://groups.google.com/d/msgid/common-workflow-
>>> > language/1412e3fd-e52f-418f-9259-025bca16afe6%40googlegroups.com.
>>> > >>> For more options, visit https://groups.google.com/d/optout.
>>> > >> --
>>> > >> You received this message because you are subscribed to the Google
>>> > Groups
>>> > >> "common-workflow-language" group.
>>> > >> To unsubscribe from this group and stop receiving emails from it, send
>>> > >> an

>>> > >> email to common-workflow-la...@googlegroups.com.

>>> > >> To post to this group, send email to
>>> > >> common-workf...@googlegroups.com.
>>> > >> To view this discussion on the web visit
>>> > >> https://groups.google.com/d/msgid/common-workflow-
>>> > language/5f74db50-1fc0-4cf1-b622-af5a8c9ac5a5%40googlegroups.com.
>>> > >>
>>> > >> For more options, visit https://groups.google.com/d/optout.
>>> > >
>>> > >
>>> >
>>> > --
>>> > You received this message because you are subscribed to the Google Groups
>>> > "common-workflow-language" group.
>>> > To unsubscribe from this group and stop receiving emails from it, send an

>>> > email to common-workflow-la...@googlegroups.com.

>>> > To post to this group, send email to common-workflow-
>>> > lang...@googlegroups.com.
>>> > To view this discussion on the web visit
>>> > https://groups.google.com/d/msgid/common-workflow-
>>>
>>> > language/91ee3278-b3a5-1a8c-b5ae-462f0407a1f3%40sbgenomics.com.
>>>
>>> > For more options, visit https://groups.google.com/d/optout.
>>>
>>> --
>>>
>>> You received this message because you are subscribed to a topic in the Google Groups "common-workflow-language" group.
>>> To unsubscribe from this topic, visit https://groups.google.com/d/topic/common-workflow-language/JU7PSEKr97M/unsubscribe.

>>> To unsubscribe from this group and all its topics, send an email to common-workflow-la...@googlegroups.com.

>>>
>>>
>>> To post to this group, send email to common-workf...@googlegroups.com.
>>>
>>> To view this discussion on the web visit https://groups.google.com/d/msgid/common-workflow-language/b78efbba87f34144970da27712b844ea%40mail.gmail.com.
>>>
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups "common-workflow-language" group.

>>> To unsubscribe from this group and stop receiving emails from it, send an email to common-workflow-la...@googlegroups.com.

>>> To post to this group, send email to common-workf...@googlegroups.com.
>>>
>>> To view this discussion on the web visit https://groups.google.com/d/msgid/common-workflow-language/CAD%3DV0dSdiXLhbDny5%2BjhMv1RoNqoVX3xtD%2BsbKTtoujfOwPnKg%40mail.gmail.com.
>>>
>>>
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups "common-workflow

...

--
You received this message because you are subscribed to the Google Groups "common-workflow-language" group.

To unsubscribe from this group and stop receiving emails from it, send an email to common-workflow-la...@googlegroups.com.

To post to this group, send email to common-workf...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/common-workflow-language/487785e9-3658-44aa-8f2e-98e53784a436%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "common-workflow-language" group.

To unsubscribe from this group and stop receiving emails from it, send an email to common-workflow-la...@googlegroups.com.

To post to this group, send email to common-workf...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/common-workflow-language/966678ed35d4deba5e450e137a478fd7%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "common-workflow-language" group.

To unsubscribe from this group and stop receiving emails from it, send an email to common-workflow-la...@googlegroups.com.

To post to this group, send email to common-workf...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/common-workflow-language/CA%2BBw4v16fq-CGLP9EquvBs8-hQ_3j5_SZDWf6j6uJyFcPiaw7A%40mail.gmail.com.

[Paul Grosu]

Hi Michael,

So I wanted to use BNF to describe the general grammar of CWL in order to prove a few guarantees, which would allow for the language to be used for extending the language itself, while managing complexity in a structured way.  That required a few other mathematical notations to illustrate it.  The most concise BNF representation of a loop I can think of is the following, but it becomes hard to read:

 <iteration_element> ::= <condition>
                       | <iteration_element>*

The asterisk symbol indicates that there are zero or more of those elements, while the condition performs a check that might be met.

So besides the continuation-passing style approach that Jeff mentioned - or a distributed message-based approach - you can think of a scatter/gather as a function-addressable data structure.  So if you want some data to be transformed by passing through several functions like this, where each step can be a scatter/gather:

  f3 ° f2 ° f1 ° data

Visually you can think of it like this:

Basically you are accessing different levels of data-transformed spaces, by using the function as an address, which behaves like a dynamic hash table.  This can be achieved by constructing a relation of the function composition via hash-mapped addressable transformations:

  ((data[ f1 ])[ f2 ])[ f3 ]

Which can be thought of more concisely as this:

  data[ f1 -> f2 -> f3 ]

This also has the added benefit of providing natural provenance as well.

I need to think a little more about Peter's updated approach, so I'll answer that a bit later.

Paul

On Saturday, January 7, 2017 at 11:45:23 AM UTC-5, Michael Miller wrote:

hi paul,

 

this is a nice approach but i'm trying to work my head around it.  one thing that would help is that you start with BNF but then wander off into transformations that lead to another set of operations and symbols than BNF.  it would be nice to wander back to the BNF notation to see how the loop and conditional would look with that notation.

...

[Paul Grosu]

Hi Peter, et al.,

I like parts of it :)  After thinking about it from different angles, I keep simplifying to the same answer which uses the base term match for pattern matching.  So we can treat a conditional as a single match, while a loop is a recursive match denoted here having the rematch field set to true (which by default is false).  The loop has the option to bind values to variables with the field name bind, but you can use (re)assign if that is preferred.  For the default match, I replaced it with none but that can be substituted with ( default, rest, true, * , _ , ... ) if that is more preferable.  I also moved the out at the end to create an enclosure.  Below are the updated examples of the conditional and loop:

Conditionals

>>>
steps:
  step1:
    in:
      a: inputs.a
    match:
      - $(a > 1):
          run: greater.cwl
      - $(a < 1):
          run: less.cwl
      - none:
          run: same.cwl
    out: [out]
<<<

Loops

>>>
steps:
  step2:
    in:
      a: inputs.a
    match:
      - $(a > 1):
          run: blah.cwl
          bind:
            - a: $(self.out)
          rematch:
            true
    out: [out]
<<<

I also played with the idea of having loops be denoted via match* instead of using rematch like in the following example, which might be an option but for some folks it could have other interpretations:

>>>
steps:
  step2:
    in:
      a: inputs.a
    match*:
      - $(a > 1)
          run: blah.cwl
          bind:
            - a: $(self.out)
    out: [out]
<<<

I experimented with other keywords such as with match-once, match-rec, match-while, match-until, match-all, match-exists, match-any but my goal was to keep the language as clean as I possibly could, with as few required key words as necessary.  I'll keep thinking in case I come up with other approaches later on, that might improve on the above.

Let me know what you think.

Thanks,
Paul

...