AWS sync step as first step of pipeline

[Rohit Sharma]

Hi everyone,

How can i use 'aws sync' command as first step in pipeline to download R1 and R2 fastq file pairs and pass them to next steps as inputs. Any help here would be great .

Thanks

 

[Rohit Sharma]

any help here would be highly appreciated

[Simon Sadedin]

Hi Rohit,

I would think you would just make it the first step in your pipeline and then use the forward command to make those files become inputs to the downstream stages, eg:

download = {

    exec """

    """

} 

Cheers,

Simon

--
You received this message because you are subscribed to the Google Groups "bpipe-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bpipe-discuss+unsubscribe@googlegroups.com.
To post to this group, send email to bpipe-...@googlegroups.com.
Visit this group at https://groups.google.com/group/bpipe-discuss.
For more options, visit https://groups.google.com/d/optout.

[Simon Sadedin]

Sorry, sent that too early!

The example would be:

download = {

    requires r1_input : 'The R1 file to download',

             r2_input : 'The R2 file to download'

    produce(r1_input, r2_input) {

      exec """

          aws s3 sync s3://mybucket/$r1_input $output1.fastq.gz

          aws s3 sync s3://mybucket/$r2_input $output2.fastq.gz

      """

    }

} 

In this case you should not even need the forward command. 

Cheers,

Simon

On Fri, Mar 2, 2018 at 6:35 PM, Simon Sadedin <ssad...@gmail.com> wrote:

Hi Rohit,

I would think you would just make it the first step in your pipeline and then use the forward command to make those files become inputs to the downstream stages, eg:

download = {

    exec """

    """

} 

Cheers,

Simon