Copying a sample's files from S3 to EBS with BaseSpace API

[Sébastien Boisvert]

Hi,

I used the BaseSpace API [1] to get a list of files for
"2x150 HiSeq 2500 demo NA12878". I followed these URIs:

1. https://api.basespace.illumina.com/v1pre3/users/current/projects
2. https://api.basespace.illumina.com/v1pre3/projects/183184
3. https://api.basespace.illumina.com/v1pre3/projects/183184/samples
4. https://api.basespace.illumina.com/v1pre3/samples/262264
5. https://api.basespace.illumina.com/v1pre3/samples/262264/files
6. https://api.basespace.illumina.com/v1pre3/files/25033024
7. https://api.basespace.illumina.com/v1pre3/files/25033024/content

The URI # 7 redirects to
https://basespace-data.s3.amazonaws.com/29d632250d84443ebb6da10a013f675a/sorted_S1_L001_R1_001.fastq.gz

with this trailing QUERY_STRING:

AWSAccessKeyId=...&Expires=1358695548&response-content-disposition=attachment&response-content-encoding=binary&response-content-type=application/octet-stream&Signature=...


To run Ray as a native application on Amazon EC2, we need to get
all the files associated to a sample inside a EBS volume attached to a
EC2 instance.

What is the best course of action to do that ? Does BaseSpace have a
framework to pull from S3 to EBS, run the native app, and then push
AppResult back in S3 using BaseSpace API ?

As I understand, I can not register a native app that does that at the
moment, right ?

Otherwise, I can do this with the access_token [2] and then push back a
result [3,4].

From the Amazon documentation, data transits from S3 to EBS and vice
versa are free.


---
[1] https://developer.basespace.illumina.com/docs/content/documentation/rest-api/api-reference
[2] https://developer.basespace.illumina.com/docs/content/documentation/authentication/obtaining-access-tokens
[3] https://developer.basespace.illumina.com/docs/content/documentation/rest-api/api-reference#POST%3a_projects%2f{Id}%2fappresults
[4] https://developer.basespace.illumina.com/docs/content/documentation/app-integration/writing-back-to-basespace#Upload_files

[Mayank Tyagi]

Hey Sebastien,

The workflow you described to pull Sample files is correct, to pull each of the files you would simply repeat steps 5-7 and iterate through the file list.  At the moment, we don't support native apps, so we wouldn't handle the instance for you.  However, to be supported in the current implementation of BaseSpace, you could stand up a website that only redirects to S3 at which point you're free to operate between S3 and EBS.  The website would essentially always be up and listening for redirects from BaseSpace.  

We're working on integrating native apps into BaseSpace, and we'll keep developers updated on that through the groups and the BaseSpace Blog.  Thanks for posting and please let us know if there is anything else we can provide help for.

Sincerely,

The BaseSpace Platform Team

[Mayank Tyagi]

Hey Sebastien,

Just wanted to add, the documents you have listed for writing back to BaseSpace are correct as well.  Please let us know if you run into any issues during this process.

Sincerely,

The BaseSpace Platform Team