Multipart and Multithread file downloads

[Thomas Gatter]

Hi,

we are currently trying to embed BaseSpace  into our own Amazon-based cloud-framework.
In order to use the full bandwidth of the used instances we need to increase the transfer speed by downloading all files in multiple parts in a heavily parallelized fashion.

For this reason we have created a tool doing just this for files on S3 using the JAVA SDK. [1, 2]

Since BaseSpace files are also S3 files we would like to access them in the same fashion.

However the BaseSpace REST API only specifies a method to retrieve a pre-signed download URL. [3]
While it is basically possible to specify the range attribute in the header of the GET and therefore also possible to build a program to execute parallel downloads, we would like to handle BaseSpace files the same way as we do normal S3 files.

Thus we would like know if it possible to get the bucket and object of BaseSpace files in order to to download them using the S3 API and possibly session credentials.

If not I would like to know if there as a better way to implement multi-part downloads than using the range attribute in the header and the pre-signed URL.

Thank you very much for you help!



---------------------------
[1] http://aws.amazon.com/sdkforjava/
[2] http://wiki.techfak.uni-bielefeld.de/bibiserv/BiBiS3
[3] https://developer.basespace.illumina.com/docs/content/documentation/rest-api/api-reference#GET%3a_files%2f{Id}%2fcontent

[Greg Roberts]

Hi Thomas,

Thanks for your question.

Currently we don't give out temporary tokens for resources in S3, as you stated we use the pre-signed urls.  I would say in the short term, your best bet is to write a simple utility or modify the JAVA SDK transfer code to take in a url as input.  We've internally created our own that is agnostic of S3 and works with any web server that accepts byte range requests, which is a nice/simple tool to have in your belt.  We will look into adding this credential capability in the future, or possibly adding better download code to our Java SDK , but for what is possible now, it seems like you will need to do some custom code.  

Sorry that this is causing problems.  Let us know if you have any more questions..

Thanks!

Greg