The Space of Workflow Definition Languages

[Kiani, Amir]

Hi guys,

I do appreciate the abundance of different workflow definition languages. I have been looking into the tradeoffs between them, and wonder if others have done a comparative study that they could share.

I'm specifically curious about the differences/tradeoffs between CWL, Nextflow and WDL. It seems like Nextflow is pretty focused on performance (e.g. streaming) vs. CWL is focused on reproducibility and interoperability (and seems to have better community support ;) )

I don't mean to distract the group, but in case anyone has good pointers/opinions on this, I'd love to hear them.

Many thanks,

Amir

[Peter Amstutz]

Hi Amir,

CWL is a standards effort intended to be implemented by many workflows engines.  From a technical standpoint, the focus is on interoperability and support for tooling and extensibility.

Nextflow is a specific workflow engine. As far as I understand, it uses a flow based programming model which is a little different from the standard dependency DAG.   Paolo has expressed interest in supporting CWL in Nextflow, but I don't know what his plans are.

WDL is developed by the Broad and implemented in the Cromwell workflow engine.  There is a meeting at Broad next week to discuss how CWL and WDL could interoperate, such as compiling WDL to CWL.  From a technical standpoint, WDL focuses on having terse syntax that is convenient to write by hand.

Thanks,
Peter

--
You received this message because you are subscribed to the Google Groups "common-workflow-language" group.
To unsubscribe from this group and stop receiving emails from it, send an email to common-workflow-la...@googlegroups.com.
To post to this group, send email to common-workf...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/common-workflow-language/CAPaaX%2B-Nh7516cj1epw0SN0hy05SCHj%3DXdgQe2c8oCcvrsET5g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Scott Frazer]

I can only really comment on WDL and CWL.  The way I see it, CWL and WDL are mostly the same with one major philosophical difference: CWL is optimized for the average developer to implement.  WDL is optimized for the people reading and writing these tools/workflows.  WDL emerged because we realized that we're going to have workflows that are a thousand lines long and this needs to be reviewable in a pull request by a diverse set of people.  These workflows also need to be maintained and easily understandable to people reading them.  The goal with WDL is to create a language so easy to read and write that it requires little explanation to get people to start authoring and modifying tasks and workflows.

WDL is also designed to be implemented by other workflow engines besides Cromwell.  The WDL Project provide parsers (Java, Python) so developers don't need specialized parsing knowledge to parse a WDL file.  Cromwell itself is an execution engine, but it also provides Scala language bindings to WDL (i.e. you can use Cromwell to understand WDL without using the execution engine part of Cromwell).

Hope that helps!

On Friday, November 6, 2015 at 10:56:12 PM UTC-5, Peter Amstutz wrote:

Hi Amir,

CWL is a standards effort intended to be implemented by many workflows engines.  From a technical standpoint, the focus is on interoperability and support for tooling and extensibility.

Nextflow is a specific workflow engine. As far as I understand, it uses a flow based programming model which is a little different from the standard dependency DAG.   Paolo has expressed interest in supporting CWL in Nextflow, but I don't know what his plans are.

WDL is developed by the Broad and implemented in the Cromwell workflow engine.  There is a meeting at Broad next week to discuss how CWL and WDL could interoperate, such as compiling WDL to CWL.  From a technical standpoint, WDL focuses on having terse syntax that is convenient to write by hand.

Thanks,
Peter

On Nov 6, 2015 5:44 PM, "Kiani, Amir" <amir....@bina.roche.com> wrote:

Hi guys,

I do appreciate the abundance of different workflow definition languages. I have been looking into the tradeoffs between them, and wonder if others have done a comparative study that they could share.

I'm specifically curious about the differences/tradeoffs between CWL, Nextflow and WDL. It seems like Nextflow is pretty focused on performance (e.g. streaming) vs. CWL is focused on reproducibility and interoperability (and seems to have better community support ;) )

I don't mean to distract the group, but in case anyone has good pointers/opinions on this, I'd love to hear them.

Many thanks,

Amir

--
You received this message because you are subscribed to the Google Groups "common-workflow-language" group.

To unsubscribe from this group and stop receiving emails from it, send an email to common-workflow-language+unsub...@googlegroups.com.

[Kiani, Amir]

Many thanks Scott and Peter. Both of your responses were enlightening. Hearing the genesis and philosophy of the language/tools helps very much with understanding their use cases.

 

I guess it'd be very nice to hear from Paolo (who seems to be a Nextflow author according to Peter) as well on his thoughts.

I think a blog post/article on this discussion would be pretty useful for the community.

Amir

On Mon, Nov 16, 2015 at 8:25 AM, Scott Frazer <sfr...@broadinstitute.org> wrote:

I can only really comment on WDL and CWL.  The way I see it, CWL and WDL are mostly the same with one major philosophical difference: CWL is optimized for the average developer to implement.  WDL is optimized for the people reading and writing these tools/workflows.  WDL emerged because we realized that we're going to have workflows that are a thousand lines long and this needs to be reviewable in a pull request by a diverse set of people.  These workflows also need to be maintained and easily understandable to people reading them.  The goal with WDL is to create a language so easy to read and write that it requires little explanation to get people to start authoring and modifying tasks and workflows.

WDL is also designed to be implemented by other workflow engines besides Cromwell.  The WDL Project provide parsers (Java, Python) so developers don't need specialized parsing knowledge to parse a WDL file.  Cromwell itself is an execution engine, but it also provides Scala language bindings to WDL (i.e. you can use Cromwell to understand WDL without using the execution engine part of Cromwell).

Hope that helps!

On Friday, November 6, 2015 at 10:56:12 PM UTC-5, Peter Amstutz wrote:

Hi Amir,

CWL is a standards effort intended to be implemented by many workflows engines.  From a technical standpoint, the focus is on interoperability and support for tooling and extensibility.

Nextflow is a specific workflow engine. As far as I understand, it uses a flow based programming model which is a little different from the standard dependency DAG.   Paolo has expressed interest in supporting CWL in Nextflow, but I don't know what his plans are.

WDL is developed by the Broad and implemented in the Cromwell workflow engine.  There is a meeting at Broad next week to discuss how CWL and WDL could interoperate, such as compiling WDL to CWL.  From a technical standpoint, WDL focuses on having terse syntax that is convenient to write by hand.

Thanks,
Peter

On Nov 6, 2015 5:44 PM, "Kiani, Amir" <amir....@bina.roche.com> wrote:

Hi guys,

I do appreciate the abundance of different workflow definition languages. I have been looking into the tradeoffs between them, and wonder if others have done a comparative study that they could share.

I'm specifically curious about the differences/tradeoffs between CWL, Nextflow and WDL. It seems like Nextflow is pretty focused on performance (e.g. streaming) vs. CWL is focused on reproducibility and interoperability (and seems to have better community support ;) )

I don't mean to distract the group, but in case anyone has good pointers/opinions on this, I'd love to hear them.

Many thanks,

Amir

--
You received this message because you are subscribed to the Google Groups "common-workflow-language" group.

To unsubscribe from this group and stop receiving emails from it, send an email to common-workflow-la...@googlegroups.com.

[Paolo Di Tommaso]

Hi, 

Nextflow shares with CWL the intent to make workflows portable, however our approach is to enable portability targeting multiple execution platforms. In the meaning that you can have the same Nextflow workflow running, for example, on a grid engine or in the cloud with having to modify it. Instead CWL aims to make your workflow portable across different workflow managers that will eventually support that specification. 

As Peter said, I'm planning to include in Nextflow a support for CWL, however this is a goal in the long run.

 

Nextflow implements also a DSL for workflow definitions, so in this aspect it is similar to WDL. However Nextflow is a multi-paradigm computing environment. It mixes a full-featured object-oriented programming environment with an high-level declarative programming model for parallel task executions (dataflow) and a DSL to simplify the writing of complex tasks that execute user scripts and system commands.

The dataflow programming model is surely what distinguish Nextflow from CWL and WDL (and other frameworks). This is a powerful paradigm that brings a lot of benefits when writing parallel applications, because radically simplifies the management and the synchronisation of the of concurrent task executions. 

Finally Nextflow has a built-in support for Docker (like CWL) and popular source code management platforms (Bitbucket, GitLab and GitHub). This allows you to upload your project into one of these services, create a Docker image including all workflow dependencies and execute it, on any platform having the Docker engine installed, with a single command in your terminal. 

Hope you will find this useful. 

Cheers,

Paolo

[Scott Frazer]

I think a blog post/article on this discussion would be pretty useful for the community.

That's a good idea, I'll offer to contribute to it.

I think there are a lot of interesting discussions here...  Even conversations about "what problem are we trying to solve?" and "who are we trying to solve it for?" are areas that could use more discussion.

[Kiani, Amir]

Great! Peter and Paolo, are you guys okay if I quote you on my blog post?

I will definitely give you a pre-read as well. All I'm planning to do is to write a comparative article and not make a claim on which WDL is better or worse. Just a note on the diversity that exists and what each languages philosophy is. This will be posted on Bina's blog.

I will get right to it once I have you guys' permission.

[Paolo Di Tommaso]

That's fine on my side.

Thank you.
Paolo

--
You received this message because you are subscribed to a topic in the Google Groups "common-workflow-language" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/common-workflow-language/qQ8-jCNkAXE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to common-workflow-la...@googlegroups.com.

To post to this group, send email to common-workf...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/common-workflow-language/CAPaaX%2B-%3DpF7Z2qB3UCrWXVskuqUA9-ahXT7%2B84n44oi9k-068A%40mail.gmail.com.

[Kiani, Amir]

Many thanks!

[Peter Amstutz]

Go for it. I can provide more background if you have any questions.

> --
> You received this message because you are subscribed to the Google Groups
> "common-workflow-language" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to common-workflow-la...@googlegroups.com.
> To post to this group, send email to
> common-workf...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/common-workflow-language/CAPaaX%2B-%3DpF7Z2qB3UCrWXVskuqUA9-ahXT7%2B84n44oi9k-068A%40mail.gmail.com.