Nextflow with AWS Batch and EFS

[stephen mclaughlin]

Hi Paolo,

   Recently we got AWS Batch working (without using nextflow) by setting the volume and mount points for the container properties to be set to an EFS mount on an AWS image i.e.:

cprop["volumes"]           = [{host: {sourcePath:"/efs/"}, name:"EFS"}]

cprop["mountPoints"]       = [{containerPath:"/efs/", readOnly:false, sourceVolume:"EFS"}]

  

  When a job runs, you are then able to read and write to this /efs/ volume from within the running docker container.  Is there a way in nextflow to use this sort of approach where intermediate files (the work directory) could be used with AWS Batch?   I really like kicking off AWS Batch jobs from a lightweight instance with EFS mounted then being able to see the inputs and outputs the AWS Batch docker containers are using right off a mount on that same instance with all of the benefits of using unix command line tools to explore/debug the outputs.  nextflow currently uses AWS Batch with S3 inputs and outputs which makes it harder to explore the intermediate files.  The idea of hjaving all the intermediate files and final outputs read and write to EFS seems very appealing.  Do you have any thoughts about this or recommendations for how this could currently be done with nextflow?  Have you thought about this being a supported feature in the future?  Is there anything about this approach that you think makes this a bad solution?

  I appreciate any feedback.  Thank you!

Stephen

[Paolo Di Tommaso]

This is an approach that deserve to be investigated, I think it's not working with the current version of NF. 

A few question, how is EFS mounted in the Ec2 instance? is that a cloudformation snippet? 

Then how is the /efs host path mounted in the container?

I think this could be a good solution, the only bad thing are the EFS costs I guess. 

Cheers,
Paolo

--
You received this message because you are subscribed to the Google Groups "Nextflow" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nextflow+unsubscribe@googlegroups.com.
Visit this group at https://groups.google.com/group/nextflow.
For more options, visit https://groups.google.com/d/optout.

[stephen mclaughlin]

Hi Paolo,

   A standard ECS AMI from amazon was used and a mountpoint was created at /efs/ then nfs-utils was intalled.  It was mounted like so:

sudo mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2

  Then it was added to fstab so each AMI when it boots to the cluster has it mounted as described here:

  https://docs.aws.amazon.com/AmazonECS/latest/developerguide/using_efs.html

  The same procedure was used to mount the EFS to the lightweight instance where the jobs are kicked off from (also using /efs/).  

  Lastly, the job definitions for AWS Batch are configured as follows:

  This means that any files you provide as input on the command line when you launch AWS Batch will have precisely the same paths in the Docker containers as they do from the lightweight instance the AWS Batch jobs are launched from which is convenient for tabbing out arguments on the command line etc etc.   I was using the AWS Batch API to create the job definitions and launch the jobs.  It would love to use nextflow this way though!  

Thank you,

Stephen

To unsubscribe from this group and stop receiving emails from it, send an email to nextflow+u...@googlegroups.com.

[Paolo Di Tommaso]

I see. In principle it should be possible though currently NF tries to stage and copy the data from/to S3. 

However what's your use case? do you want to use it only for scratch data storage? or also to store input dataset and the workflow results? What's the main advantage you are expecting using EFS over S3  ? 

p

To unsubscribe from this group and stop receiving emails from it, send an email to nextflow+unsubscribe@googlegroups.com.