Re: S3 Bucket#SignedUrl

[Gustavo Niemeyer]

The problem is apparently that the bucket is
"com.typekit.entitlement_files-staging", yet the signature in the
error message seems to include only "/entitlement_files-staging/", I
haven't seen that kind of mangling before, and it's not clear how such
a signature could validate the prefix as well.

I'll be looking for more documentation/information on this. If someone
knows what's going on, please chime in.

On Thu, Oct 3, 2013 at 2:17 AM, George Blazer <gbl...@gmail.com> wrote:
> Hi there,
>
> I'm having trouble getting a SignedUrl for a file in the bucket up on S3.
>
> I'm definitely using correct AWS credentials, but getting an error. I have
> found an article that seems relevant
> http://kitcarrau.tumblr.com/post/12841485836/amazon-s3-signed-urls-with-nodejs,
> but can't quite figure out what's causing the problem.
>
> Please help.
>
> ~/workspace/polka curl -i
> "https://s3.amazonaws.com/com.typekit.entitlement_files-staging/73b5da7f4e7280e18af2441f58acab8b39c391b8?AWSAccessKeyId=AKIAJDHEEQAZEKYIVKNQ&Expires=1380777486&Signature=OuA2uAk9HOKsuC4F4ktfvuvshhk%3D"
> HTTP/1.1 403 Forbidden
> x-amz-request-id: 026ABF4EDA9578C2
> x-amz-id-2: H7OIcP8WNwiuPCJOVOr4vm8CjDWECE93I6qQAANRkjXGtamUdYdjQGMpqBjZtfhd
> Content-Type: application/xml
> Transfer-Encoding: chunked
> Date: Thu, 03 Oct 2013 05:15:13 GMT
> Server: AmazonS3
>
> <?xml version="1.0" encoding="UTF-8"?>
> <Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we
> calculated does not match the signature you provided. Check your key and
> signing method.</Message><StringToSignBytes>47 45 54 0a 0a 0a 31 33 38 30 37
> 37 37 34 38 36 0a 2f 63 6f 6d 2e 74 79 70 65 6b 69 74 2e 64 65 73 6b 74 6f
> 70 5f 66 6f 6e 74 5f 65 6e 74 69 74 6c 65 6d 65 6e 74 5f 66 69 6c 65 73 2d
> 73 74 61 67 69 6e 67 2f 37 33 62 35 64 61 37 66 34 65 37 32 38 30 65 31 38
> 61 66 32 34 34 31 66 35 38 61 63 61 62 38 62 33 39 63 33 39 31 62
> 38</StringToSignBytes><RequestId>026ABF4EDA9578C2</RequestId><HostId>H7OIcP8WNwiuPCJOVOr4vm8CjDWECE93I6qQAANRkjXGtamUdYdjQGMpqBjZtfhd</HostId><SignatureProvided>OuA2uAk9HOKsuC4F4ktfvuvshhk=</SignatureProvided><StringToSign>GET
>
>
> 1380777486
> /entitlement_files-staging/73b5da7f4e7280e18af2441f58acab8b39c391b8</StringToSign><AWSAccessKeyId>AKIAJDHEEQAZEKYIVKNQ</AWSAccessKeyId></Error>%
>
> --
> You received this message because you are subscribed to the Google Groups
> "goamz" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to goamz+un...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.



--

gustavo @ http://niemeyer.net

[George Blazer]

You mean it's the underscores and dots that the code doesn't like?

[George Blazer]

Actually, I'm not sure that's the issue?

This is the ouput

~/workspace/goamz/s3 curl -i -v "https://s3.amazonaws.com/com.typekit.desktop_font_entitlement_files-staging/73b5da7f4e7280e18af2441f58acab8b39c391b8?AWSAccessKeyId=AKIAJDHEEQAZEKYIVKNQ&Expires=1380822585&Signature=EmIavAJD%2B9iXdDFSr566pXoNwhM%3D"

* About to connect() to s3.amazonaws.com port 443 (#0)

*   Trying 207.171.189.80...

* connected

* Connected to s3.amazonaws.com (207.171.189.80) port 443 (#0)

* SSLv3, TLS handshake, Client hello (1):

* SSLv3, TLS handshake, Server hello (2):

* SSLv3, TLS handshake, CERT (11):

* SSLv3, TLS handshake, Server finished (14):

* SSLv3, TLS handshake, Client key exchange (16):

* SSLv3, TLS change cipher, Client hello (1):

* SSLv3, TLS handshake, Finished (20):

* SSLv3, TLS change cipher, Client hello (1):

* SSLv3, TLS handshake, Finished (20):

* SSL connection using RC4-MD5

* Server certificate:

* subject: C=US; ST=Washington; L=Seattle; O=Amazon.com Inc.; CN=s3.amazonaws.com

* start date: 2013-09-09 00:00:00 GMT

* expire date: 2014-09-10 23:59:59 GMT

* subjectAltName: s3.amazonaws.com matched

* issuer: C=US; O=VeriSign, Inc.; OU=VeriSign Trust Network; OU=Terms of use at https://www.verisign.com/rpa (c)10; CN=VeriSign Class 3 Secure Server CA - G3

* SSL certificate verify ok.

> GET /com.typekit.desktop_font_entitlement_files-staging/73b5da7f4e7280e18af2441f58acab8b39c391b8?AWSAccessKeyId=AKIAJDHEEQAZEKYIVKNQ&Expires=1380822585&Signature=EmIavAJD%2B9iXdDFSr566pXoNwhM%3D HTTP/1.1

> User-Agent: curl/7.24.0 (x86_64-apple-darwin12.0) libcurl/7.24.0 OpenSSL/0.9.8x zlib/1.2.5

> Host: s3.amazonaws.com

> Accept: */*

> 

< HTTP/1.1 403 Forbidden

HTTP/1.1 403 Forbidden

< x-amz-request-id: F2E08281C87CA763

x-amz-request-id: F2E08281C87CA763

< x-amz-id-2: T55f++1apKy/H0B/AOljvdjVSBmT98bGh0Mb5ggv2niNvozzI9dGNkPGjM0Tlgar

x-amz-id-2: T55f++1apKy/H0B/AOljvdjVSBmT98bGh0Mb5ggv2niNvozzI9dGNkPGjM0Tlgar

< Content-Type: application/xml

Content-Type: application/xml

< Transfer-Encoding: chunked

Transfer-Encoding: chunked

< Date: Thu, 03 Oct 2013 17:46:54 GMT

Date: Thu, 03 Oct 2013 17:46:54 GMT

< Server: AmazonS3

Server: AmazonS3

< 

<?xml version="1.0" encoding="UTF-8"?>

<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><StringToSignBytes>47 45 54 0a 0a 0a 31 33 38 30 38 32 32 35 38 35 0a 2f 63 6f 6d 2e 74 79 70 65 6b 69 74 2e 64 65 73 6b 74 6f 70 5f 66 6f 6e 74 5f 65 6e 74 69 74 6c 65 6d 65 6e 74 5f 66 69 6c 65 73 2d 73 74 61 67 69 6e 67 2f 37 33 62 35 64 61 37 66 34 65 37 32 38 30 65 31 38 61 66 32 34 34 31 66 35 38 61 63 61 62 38 62 33 39 63 33 39 31 62 38</StringToSignBytes><RequestId>F2E08281C87CA763</RequestId><HostId>T55f++1apKy/H0B/AOljvdjVSBmT98bGh0Mb5ggv2niNvozzI9dGNkPGjM0Tlgar</HostId><SignatureProvided>EmIavAJD+9iXdDFSr566pXoNwhM=</SignatureProvided><StringToSign>GET

1380822585

* Connection #0 to host s3.amazonaws.com left intact

/com.typekit.desktop_font_entitlement_files-staging/73b5da7f4e7280e18af2441f58acab8b39c391b8</StringToSign><AWSAccessKeyId>AKIAJDHEEQAZEKYIVKNQ</AWSAccessKeyId></Error>* Closing connection #0

* SSLv3, TLS alert, Client hello (1):

On Thursday, October 3, 2013 10:32:16 AM UTC-7, Gustavo Niemeyer wrote:

[Gustavo Niemeyer]

No, I mean that the URL contains "com.typekit." in the bucket name,
but the <StringToSign> content in the error message provided by the
server does not. That's definitely not okay.

[Gustavo Niemeyer]

This is a different URL than the one you provided in the first
message. Was the content manipulated by hand in the first message?

[George Blazer]

Yes, sorry

You received this message because you are subscribed to a topic in the Google Groups "goamz" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/goamz/uNaPCEBt5EA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to goamz+un...@googlegroups.com.

[Gustavo Niemeyer]

Okay, no problem. I do clean up messages in some cases as well, but in
that specific case it's a problem because what is failing is likely a
detail in the signed content.

Do you have a trash access key pair and a self-contained example I
could use to reproduce the issue locally?

[George Blazer]

Can I just email it to you directly instead of the list?

[George Blazer]

Sent to your email.

[Gustavo Niemeyer]

Yes please.

[George Blazer]

The bucket is com.typekit.desktop_font_entitlement_files-staging , the file 73b5da7f4e7280e18af2441f58acab8b39c391b8

Please let me know if you can access it ok.

[Gustavo Niemeyer]

Thanks, I can reproduce the issue. I should have a bug description/fix soon.

[George Blazer]

Super, thanks! Also any direction how I can run goamz from the local source?

[George Blazer]

Gustavo,

Is there an ETA or should I attempt to work around it?

Thanks.

[Gustavo Niemeyer]

It looks like your key pair is corrupted.

You can replace that bucket with any other bucket, and you'll get the
exact same failure. I've also attempted to sign any other content with
your credentials, and it consistently fails in the same manner. I've
also attempted to sign any URLs with your credentials using two other
toolkits from two other languages, and it fails in the exact same way.

Regenerate your keys and I bet things will start working.

[George Blazer]

This is strange, I use other keys and can't get the Url signed.

[Gustavo Niemeyer]

I believe your secret key is too long. See if you don't have an issue
somewhere else generating these keys.

Here is a working example: http://paste.ubuntu.com/6189518/

These credentials are valid, and generate valid URLs. You'll get
Access Denied when trying to use them, though, for lack of
permissions.

Now, try adding one extra byte at the end of the secret key and you'll
get the same behavior you get with your own key.

[George Blazer]

That sounds like a problem though right? I don't generate secret keys myself. I'm using the keys that we have been using in prod for a long time, so if anything it's not the AWS that doesn't like the keys, but I think the s3 package.

[Gustavo Niemeyer]

On Thu, Oct 3, 2013 at 5:19 PM, George Blazer <gbl...@gmail.com> wrote:
> That sounds like a problem though right? I don't generate secret keys
> myself. I'm using the keys that we have been using in prod for a long time,
> so if anything it's not the AWS that doesn't like the keys, but I think the
> s3 package.

Well, everything is possible when we don't know the server code, but
from my tests:

1. The payload is verifiably correct according to what the server says
2. Your private key cannot sign any URL with the s3 Go package
3. Your private key cannot sign any URL with the boto Python package
4. Your private key cannot sign any URL with the nodejs
amazon-url-signer package
5. Other keys I've tested (including the one I handed you) can sign
URLs with any of these
6. The error message with an incorrect S3 secret key matches exactly
the error with your key

I don't really have much to work with here, other than guessing your
key isn't sound.


gustavo @ http://niemeyer.net

[Gustavo Niemeyer]

Oh, and if you manage to produce a valid URL signature with *any*
library with your key, please let me know. I'm definitely interested.

[George Blazer]

I think I found the problem. When I copy/paste by access key, and secret key everything works, but when the values get read from the env variables I get an error.

I pasted the secret key in the file and 41 bytes, instead of 40 like it should be.

I ran `od -A n -t uC secret` and the last number was 10, which is a linefeed.

I think your library should be trimming linefeeds off the strings that it gets from env variables.

[Gustavo Niemeyer]

Heh.. how has an incorrect key appeared in my inbox then? Or perhaps
that key was made up and I wasted my time with invalid credentials?

Glad you've found the issue, though. Good luck with your application.

> You received this message because you are subscribed to the Google Groups
> "goamz" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to goamz+un...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

[George Blazer]

Gustavo, actually the keys that I gave you were correct, but the last byte was a linefeed.

I think you should change your library to chomp the strings when it gets the values from environment variables.

Thanks for your help though.

[Gustavo Niemeyer]

Which of these bytes is a line feed, George?

> This user should have access to the bucket I want
>
> Access Key ID:
> AKIAJSS2TSS4BBKEKDKQ
> Secret Access Key:
> STp4EVGVRZ4g1ln8nXydcZBNhn0f6AejK536NswGa
>

[George Blazer]

Paste the secret I gave you into the file 'secret' and run `od -A n -t uC secret`. What's the last byte? If it's 10, then it's a line feed that needs to be removed

[George Blazer]

So your code should do something like this

auth := aws.Auth{strings.TrimRight(os.Getenv("AWS_ACCESS_KEY_ID"), "\r"), strings.TrimRight(os.Getenv("AWS_SECRET_KEY"), "\r")}

[Gustavo Niemeyer]

The 10 you see in your secret file is from your editor George. The
last character in that secret key is 'a', which is 97 in ASCII:

$ echo -n STp4EVGVRZ4g1ln8nXydcZBNhn0f6AejK536NswGa > secret
$ od -t uC secret
0000000 83 84 112 52 69 86 71 86 82 90 52 103 49 108 110 56
0000020 110 88 121 100 99 90 66 78 104 110 48 102 54 65 101 106
0000040 75 53 51 54 78 115 119 71 97
0000051

Honestly this whole thread is full of nonsense. I'll step out of it right now.

> You received this message because you are subscribed to the Google Groups
> "goamz" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to goamz+un...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.