Upload too slow / transfert to aws S3

[Quentin Chartreux]

Hello,

I'm New on basespace and I have several questions but I will ask them in different discussions so that the title corresponds well and that it remains readable by all.

I wanted if it was normal for the file transfer to basespace to be so slow? to transfer the two fastq.gz from a 30x wgs it took 1 hour. Knowing that our servers are equipped with a very good connection and that when we transfer our files to aws it is much faster.

Another question about transfers: is there a way to easily transfer files which are on basespace to aws s3? As the files are actually on Amazon's servers, I find it a bit absurd to transfer them to our servers and then transfer them back to aws ..

Quentin. 

[Al Maynard]

Welcome Quentin - thanks for splitting the questions.

How are you transferring the files? The fastest solution is usually our CLI tools. With low-latency 10 Gb connection and default settings I have seen 3 TB/hr.

https://basespace.illumina.com is indeed backed by a bucket in us-east-1. We don't support direct S3 access to the files, but the API and CLI do provide a pre-signed AWS URL that you can use for multipart download directly. If you have an instance in AWS (or indeed if you are running a native basepsace app) then there is no need to do the round-trip.

Al

[Quentin Chartreux]

Thank you for your reply. 

To transfert the file i use the basespace cli with a command like this :  bs upload dataset -p 21646627 --recursive .

When i transfert directly to aws s3 it's at least 4 time faster.

Quentin

[Quentin Chartreux]

I upload a common file and it was really faster so i don't understand.