Creating a POST policy and signing it for Browser based uploads to Amazon S3

[ravi2...@gmail.com]

I am trying to create a POST Policy and sign it using amazon access key, but I am not able to create the policy correctly. The steps for creating the signed policy are :

Step 1. Create a policy using UTF-8 encoding.
Step 2. Convert the UTF-8-encoded policy to Base64. The result is the string to sign.
Step 3. Create the signature as an HMAC-SHA256 hash of the string to sign. You will provide the signing key as key to the hash function.
Step 4. Encode the signature by using Base64.

The trick for testing :

I am following this document : 

Using HTTP POST to upload directly in Amazon S3

Amazon Signature Version 4 - Browser based POST example

They have provided a base64 encoded policy, and using this policy, an encoded signature. I am reverse engineering these inputs to verify my program.

For the step 1 and 2:

I took an example base64 encoded string from Amazon tutorial and decoded it here https://www.base64decode.org/. I used that JSON-Policy in my program to re-encode it in base64. The string I am receiving is different from the one I decoded initially. 

package main

import "fmt"

import "encoding/base64"

func main() {

bytePolicy := []byte(`{ "expiration": "2013-08-07T12:00:00.000Z",

  "conditions": [

    {"bucket": "examplebucket"},

    ["starts-with", "$key", "user/user1/"],

    {"acl": "public-read"},

    {"success_action_redirect": "http://examplebucket.s3.amazonaws.com/successful_upload.html"},

    ["starts-with", "$Content-Type", "image/"],

    {"x-amz-meta-uuid": "14365123651274"},

    ["starts-with", "$x-amz-meta-tag", ""],

    {"x-amz-credential": "AKIAIOSFODNN7EXAMPLE/20130806/us-east-1/s3/aws4_request"},

    {"x-amz-algorithm": "AWS4-HMAC-SHA256"},

    {"x-amz-date": "20130806T000000Z" }

  ]

}`)

fmt.Println(base64.StdEncoding.EncodeToString(bytePolicy))

}

Gaoling Play - http://play.golang.org/p/tC6DX-Xls9

As, the JSON-based text is UTF-8 encoded, so my policy is getting created using UTF-8 encoding(or is it?). So, what am I missing here? Why my base64 encoded string is different from the one I used for encoding itself?

For the Step 3 and 4,

I used the example base64 provided in amazon document(which is correct) and signed it using the method provided here. Now, the answer which I am getting should match the signature already provided in amazon document.  Golang Play - http://play.golang.org/p/Gh0igvyOzv

These steps are independent from steps 1 and 2, for my verification of the program. Am I doing some mistake while signing the base64 encoded policy or while creating the signing key itself?

[Tamás Gulácsi]

You may need to sort the keys first.

[Ravi Kant]

Which keys? I have used the order exactly same as provided in the document.

[Tamás Gulácsi]

Sorry, that's for query params.

Is the example's policy the same as yours?
The base64 encoded values differ? Where?

[Ravi Kant]

1. I have the Amazon's base64 encoded policy(example).

2. I decoded it and got the policy in JSON-format.

3. I re-encoded it in base64 using my program.

4. Now, this base64 encoded string should be exactly same as I used in step 1 above. It is not.

[Tamás Gulácsi]

Where is the difference? Base64 encodes in 3-byte chunks to 4-char chunks.
Whitespace, std vs. Urlsafe encoding issue?

[Ravi Kant]

"The two strings are not same, though they should be!" - thats the difference.

I don't know where I am doing wrong, but because of that, later I am receiving SignatureDoesNotMatch Error from Amazon S3. Are you familiar with Amazon S3?

[Piers]

The reason the base64 result is different is just line-endings in the literal string (newline vs carriage-return newline, \n vs \r\n, 0x0D vs 0x0A 0x0D)

See https://play.golang.org/p/2gzca215Va which gives your probable result and then the expected result.

I'm not sure this would affect the signature. 

I didn't know S3 could do this (even though I know S3 well) so I might carry on and see if I can guess what else is not working.

[Piers]

(Tiny error in the text: Of course I meant 0x0A vs 0x0D 0x0A. Teach me to go overboard.)

[Piers]

So,

This works (calculates the same signature as on the example page): http://play.golang.org/p/7PItbEVN2r 

I don't really understand the code you had there before, it wasn't calculating the HMAC-SHA256 at least not by AWS's definition.

The other thing I found is that despite one piece of doc, S3 expects the Signature value in the HTML Form to be in lower-case hex, not base64. 

So something like: 

<input type="hidden" name="X-Amz-Signature"  value="21496b44de44ccb73d545f1a995c68214c9cb0d41c45a17a5daeec0b1a6db047" />

This matches their example but not step 4 on this page http://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-UsingHTTPPOST.html 

cheers.

On Friday, 16 October 2015 15:16:29 UTC+1, Piers wrote:

[Ravi Kant]

Piers,

You pointed out the exact things, my silly mistakes. Great!

I did't saw your post until now. I just resolved the issue and thought of updating it here. 

I am a beginner in Go, so was confused in implementing HMAC and sha256.

Thanks for your efforts :)

Best Reagards.

[Piers]

Good news - glad you got it sorted. And I learnt something new too :)