Downloading a file shared with me using wget

[John Carroll]

Hi guys

I am a developer working with others to try and automate the downloading of customers files.  The customer selects a file(s) from a GUI that uses the Basespace API.

So the idea is I get a URL that allows me to download a file shared with me.

https://basespace-data-east.s3.amazonaws.com/c77e8484d04847f6b0eba191010abd37/HHCBYD_S1_L001_R2_001.fastq.gz?AWSAccessKeyId=mykeyissecret&Expires=123456789&response-content-disposition=filename%3HHCBYD_S1_L001_R2_001.fastq.gz&response-content-type=application/json&Signature=4Ujn_testing_38%3D

I have changed the keys in the url and wget output below.

On a windows machine when I click on the link the file begins to download in a new tab in the browser and here is a small snippet:

‹ ‰3¼Q? HCC1187C_S1_R2.fastq?¬ý»r$=Ò,Šêë-–T Nö… ìe)l?f€•2 „­ ÷ ‹S w 3 "{Ö_3_“, iŒ w Àÿ“¶ò Cú ÿßÿßö ~üŸ ?¶×?ÿgûÿþü_¡„ôú¯¦ôúPKª¯ ^_ÕRJMÿy}ó?õ?á?§GýOIé?ÿ)¯ÏÊë ì3¼¦¾~¶„ ÊëwµÏÚoyýªÿõÿù_1Æ<>žÏ×ÿíñúâ oñõßë §Çë‰=¶ ÿ{ÛŽˆÏðšmÛ÷ íW¶ b|}{ß ÿõÿL¬ü +_ L©¯¿¬¾þ—Bû¼™ ^ °Ù]›íí ¯ç_Æ¿Þ„öH¡ÙUƒ™òúFm†½^ ^Ÿ¿Þ#ûuí•Á^,+ 7± ‰ÏrÌÏÃ_ðzªýýí 3öe-ž|êûí¡·­ýâØþ Zù‹V¦ö·¿þœ× Ø {ý ÉþÈÒL)ÿ™=Úwþ“¦ß

When I try wget to download the file, it fails.  

wget -d https://basespace-data-east.s3.amazonaws.com/c77e8484d04847f6b0eba191010abd37/HHCBYD_S1_L001_R2_001.fastq.gz?AWSAccessKeyId=mykeyissecret&Expires=123456789&response-content-disposition=filename%3DHHCBYD_S1_L001_R2_001.fastq.gz&Signature=4Ujn_testing_38%3D

[1] 7559

[2] 7560

DEBUG output created by Wget 1.13.4 on linux-gnu.

URI encoding = `UTF-8'

[3] 7561

--2014-09-11 13:59:34--  https://basespace-data-east.s3.amazonaws.com/c77e8484d04847f6b0eba191010abd37/HHCBYD_S1_L001_R2_001.fastq.gz?AWSAccessKeyId=mykeyissecret

[2]   Done                    Expires=123456789

workflow@gear3:~/workflows/slivaBlasttest/victor$ Resolving basespace-data-east.s3.amazonaws.com (basespace-data-east.s3.amazonaws.com)... 54.231.2.209

Caching basespace-data-east.s3.amazonaws.com => 54.231.2.209

Connecting to basespace-data-east.s3.amazonaws.com (basespace-data-east.s3.amazonaws.com)|54.231.2.209|:443... connected.

Created socket 3.

Releasing 0x00000000013650a0 (new refcount 1).

Initiating SSL handshake.

response-content-disposition=filename%3DHHCBYD_S1_L001_R2_001.fastq.gz: command not found

Handshake successful; connected socket 3 to SSL handle 0x00000000013653f0

certificate:

  subject: /C=US/ST=Washington/L=Seattle/O=Amazon.com Inc./CN=*.s3.amazonaws.com

  issuer:  /C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=Terms of use at https://www.verisign.com/rpa (c)10/CN=VeriSign Class 3 Secure Server CA - G3

X509 certificate successfully verified and matches host basespace-data-east.s3.amazonaws.com

---request begin---

GET /c77e8484d04847f6b0eba191010abd37/HHCBYD_S1_L001_R2_001.fastq.gz?AWSAccessKeyId=mykeyissecret HTTP/1.1

User-Agent: Wget/1.13.4 (linux-gnu)

Accept: */*

Host: basespace-data-east.s3.amazonaws.com

Connection: Keep-Alive

---request end---

HTTP request sent, awaiting response...

---response begin---

HTTP/1.1 403 Forbidden

x-amz-request-id: BCC2E38D25E75EC6

x-amz-id-2: KMS7x0OhhMz+i25+kkI1clGfBLHlb/AhK3H7NUVBUXBGfY/d581jIFvv+Z9yXsYzfCTdzxubBHU=

Content-Type: application/xml

Transfer-Encoding: chunked

Date: Thu, 11 Sep 2014 12:59:34 GMT

Server: AmazonS3

---response end---

403 Forbidden

Registered socket 3 for persistent reuse.

Skipping 243 bytes of body: [<?xml version="1.0" encoding="UTF-8"?>

<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>BCC2E38D25E75EC6</RequestId><HostId>KMS7x0OhhMz+i25+kkI1clGfBLHlb/AhK3H7NUVBUXBGfY/d581jIFvv+Z9yXsYzfCTdzxubBHU=</HostId></Error>] done.

2014-09-11 13:59:35 ERROR 403: Forbidden.

My first guess is an authentication problem, but then it downloads in a browser?.   Tried curl as well with the same response.  Any help and I would be very grateful!

John

[John Carroll]

Hi All

I created a cookie with my own basespace creditials.

wget --keep-session-cookies --save-cookies cookies.txt --post-data="Username=us...@email.com&Password=guest" "https://icom.illumina.com/login?service=basespace"

That works fine.

wget -d --load-cookies cookies.txt --content-disposition --keep-session-cookies  https://basespace-data-east.s3.amazonaws.com/c77e8484d04847f6b0eba191010abd37/HHCBYD_S1_L001_R2_001.fastq.gz?AWSAccessKeyId=mykeyissecret &Expires=1410974453&response-content-disposition=filename%3DHCC1187C_S1_L001_R2_001.fastq.gz&response-content-type=application/json&Signature=4Ujn_testing_38%3D3D

---response end---

403 Forbidden

Registered socket 3 for persistent reuse.

Skipping 231 bytes of body: [<?xml version="1.0" encoding="UTF-8"?>

<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>E                             65BF3AB77982C5E</RequestId><HostId>u5NizUU5+5ePN3ou2AoDl5rskTzryrFXJOJm3Gpgk                             V6FhShWaJBnbPGl8vziEBLf</HostId></Error>] done.

2014-09-12 17:49:04 ERROR 403: Forbidden.

Any help would be great.

Thanks