Can I submit Nextflow job to Docker Swarm?

[Kai Liu]

Hi, 

I'd like to start a research project that using Docker Swarm as the execution engine for Nextflow job. Is it possible?

Best Regards,

Kai LIU

[Paolo Di Tommaso]

Hi Kai, 

This sounds interesting. I've never tried that but in principle it should be possible since Docker swark allows to run transparently containers in a cluster. 

The critical point I see is the file system. Nextflow mounts the hosting file system in the container depending the process inputs/outputs. I don't know how are managed host-mounted volumes by Docker when using Swarm. 

Anyway this is something that deserve to be investigated and I would happy to help you if needed.

Best,
Paolo

--
You received this message because you are subscribed to the Google Groups "Nextflow" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nextflow+u...@googlegroups.com.
Visit this group at https://groups.google.com/group/nextflow.
For more options, visit https://groups.google.com/d/optout.

[Kai Liu]

Dear Paolo,

Thank you for your information and I'm sorry for the late reply.

I've started to do some investigation about how to better combine Nextflow with Docker Swarm. In my opinion, there are three major problems:

Storage mechanism: How to handle input data, output data and data flow?

Scheduling algorithm: How to schedule tasks across multiple nodes?

Resource allocation: How to allocate suitable amount of CPU, memory, IO?

And I believe the problem is the same for supporting other container cluster systems like Kubernetes, Mesos and YARN.

In addition, if you can publish a long paper about Nextflow, which will benefit a lot for the development of Nextflow community. 

Best Regards,

Kai LIU

--
You received this message because you are subscribed to a topic in the Google Groups "Nextflow" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/nextflow/NHYylOjOM9s/unsubscribe.
To unsubscribe from this group and all its topics, send an email to nextflow+u...@googlegroups.com.

[Maria Chatzou]

Hi Kai,

Good points.

Nextflow already supports and works like a charm with Kubernetis and we are currently trying Mesos.

We are also now submitting the Nextflow paper, which is more oriented towards usecases, and planning to submit a second extended paper that is going to be targeting mainly developers and explaining in details the algorithmic and implementation details behind Nextflow.

Cheers,
Maria

[Kai Liu]

Hi Maria,

Thank you for your reply.

How do you handle data when submitting Nextflow task to Kubernetes? Do you use Network File System(NFS)?

Best Regards,

Kai LIU

[Maria Chatzou]

Hi Kai,

Exactly Nextflow uses NFS, which can be problematic in a cloud-setting env that you don't have that.

We are now working with UNIVA and GoogleCloud to create a Cloud solution that will be running over Kubernetes, creating a NFS like system and then Nextflow will be running over that.

I don't know if what you want to do is related to cloud...

These are all new features of Nextflow, soon to be launched out there and properly documented  for the Nextflow community to use.

Hopefully we will  soon be having Nextflow working with Mesos as well. So then we will be able to give more details on how we are doing things.

Cheers,
Maria

[Paolo Di Tommaso]

Let me add some details. I gave a try to Swarm and it works transparently as expected, so it should be feasible to run nextflow tasks with it. 

About the problem of the storage, you will need to install a shared file system in your cloud environment such as Gluster or Ceph. I agree that this is not a trivial task. Some tools like Ansiable or Terraform could help here. Also would be nice to coordinate a community effort to make it easy to deploy such file systems in a easy manner. 

Regarding the problem of scheduling and resource allocation I'm not completely sure, but I think that Docker Swarm *can* take resource allocation in consideration when scheduler the execution of containers. At least so it seems readying the Docker Swarm Strategies documentation at this link. 

https://docs.docker.com/swarm/scheduler/strategy/

Could be really interesting to investigate in this way. 

Finally, we have a preliminary working support for Kubernetes. Currently it's only available in the kube branch.  We have tested it in local cluster. Data is expected to be managed using a shared file system as described before.  

Best,
Paolo

[Kai Liu]

Hi Maria, Paolo,

Using shared file system like NFS as the storage system greatly simplifies the scheduling. Because we don't have to consider the data distribution and movement. 

However, we may need to sacrifice scalability and performance... It's tricky.

Best Regards,

Kai LIU

[Paolo Di Tommaso]

Hi Kai, 

If you need that for a production deployment of your pipelines I would surely suggest the shared file system approach. 

Instead if you are aimed for research purpose other approaches can be taken in consideration. 

For example Nextflow has a built-in integration with Apache Ignite that provides an in-memory distributed file system which can be used as a replacement for a shared file system. In this approach instead of executing a container per task, you will need to executed the nextflow/ignite workers as docker container instances in your Swarm cluster and use as nextflow working directory a Ignite path e.g. igfs://work

This should work well as long your process input/output are not too big because in practice files are saved in in-memory cache distributed in all cluster nodes. Ignite provides also some interesting features for data-affinity and co-location for task execution. Nextflow currently does not take advantage of these features however it would be very interesting to investigate how use them in order to optimise the execution of workloads. 

Hope this helps

Best,

Paolo