Download Fastq files from Basespace and Upload to AWS S3 Automatically

[Edson Zandamela]

Hi there, 

I'm looking for options to get fastq files from basespace to AWS S3 automatically.
I've tried using basespace CLI on a docker image but I have to download the files to the image and then upload to S3 which is not ideal so I'm wondering if people have done it in a simplified way maybe using Basemout???
Anyway, I appreciate any ideas you have.

[kjth...@me.com]

We use a python script combined with Basemount. The python script takes a BaseSpace project name and an AWS S3 folder as input. Then the Boto3 package is used to connect to AWS, check that the FASTQs are already uploaded in that AWS folder and then uploads them if they aren't.

It's possibly not the fastest way, so if you're doing something like WGS you might want to look for a different solution, but for our application, it works perfectly fine. It could also be further automated to check for new data regularly.

[Edson Zandamela]

Ohhh cool, so do you have to download the "new/missing" FASTQ files from Basespace to the local environment before uploading them to S3 at all or you do the upload from Basemount directly to S3?
Thanks