Hybrid workloads

[Andreas Wilm]

Hi Paolo and all,

I'm pretty excited about the hybrid workload functionality. Does this also mean that it's now in theory possible to use AWS S3 transparently for data storage/staging, even if I run the compute jobs locally? For example, could I store all input data on S3, run the workflow locally and use S3 again as publishdir and all that without having to invoke s3 cp manually? 

Thanks, 

Andreas 

[Paolo Di Tommaso]

It was possible also before to have a publishDir on S3 while running the workflow locally. 

The hybrid allows you to deploy the execution in the local/cluster as usual, while some processes can be executed on Batch. NF manages transparently the required uploads/downloads from/to S3. 

In this way you can offload only specific tasks to Batch, for example only the ones particularly compute intensive or requiring GPU or other special hardware. 

p

--
You received this message because you are subscribed to the Google Groups "Nextflow" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nextflow+u...@googlegroups.com.
Visit this group at https://groups.google.com/group/nextflow.
For more options, visit https://groups.google.com/d/optout.

[Andreas Wilm]

Wonderful. Good to know. I seem to remember that transparent s3 access was only possible with the batch executor.

Andreas