SignatureDoesNotMatch error when S3 object name contains “=”

[nonnikcm]

i am struggling with using the s3manager to create files on S3. the file names need to be in the following format "set=2012-04-3", containing an "=". uploading with out the "=" works perfectly...

sess := session.Must(session.NewSession()) uploader := s3manager.NewUploader(sess) _, err = uploader.Upload(&s3manager.UploadInput{ Bucket: aws.String("/testbucket/data/set=2012-04-3/"), Key: aws.String("test.json"), Body: bytes.NewReader([]byte(message.Body)), }) if err != nil { fmt.Printf("\n Error: There was an issue uploading to s3: %s \n", err.Error()) }

Would be grateful for any help here

[Jesper Louis Andersen]

On Thu, Apr 15, 2021 at 7:01 PM nonnikcm <b...@compado.com> wrote:

i am struggling with using the s3manager to create files on S3. the file names need to be in the following format "set=2012-04-3", containing an "=". uploading with out the "=" works perfectly...

sess := session.Must(session.NewSession()) uploader := s3manager.NewUploader(sess) _, err = uploader.Upload(&s3manager.UploadInput{ Bucket: aws.String("/testbucket/data/set=2012-04-3/"), Key: aws.String("test.json"), Body: bytes.NewReader([]byte(message.Body)), }) if err != nil { fmt.Printf("\n Error: There was an issue uploading to s3: %s \n", err.Error()) }

It may just be me, but having a Bucket equal to "/testbucket/data/set=..." seems wrong. I would expect something like

{ Bucket: aws:String("testbucket")

  Key: aws.String("/data/set=2012-04-3/test.json"),

  Body: bytes.NewReader([]byte(message.Body)),

}

I'm guessing you get a signature error because in the underlying S3 request, the = messes with the signature of the request. If you read https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-keys.html they explicitly state that = might require special handling (and while any UTF-8 string is valid, they also suggest a safe character set, not including =). In S3, it's just a flat string, so if you use something like /set/2012-04-03/ you are probably going to have an easier time working with the data, both in Go, and in other languages.

--

J.

[nonnikcm]

hi j,

thanks for your answer...

this is something we have had to port from PHP recently.

we are actually dependent on having the "=" in the path for processing downstream by PySpark.

thanks for the reply!

[Nick White]

Hi,

Quoth nonnikcm:

> i am struggling with using the s3manager to create files on S3. the file names
> need to be in the following format "set=2012-04-3", containing an "=".
> uploading with out the "=" works perfectly...

It would probably be worth submitting a bug to the AWS SDK for Go
issue tracker; if it is a bug, they're the ones to fix it.
https://github.com/aws/aws-sdk-go/issues

That said, Jesper's guess sounds about right. If you can try to find
the exact S3 request that is sent from your old PHP code, and
compare it to what gets sent with the new code - maybe the '=' was
escaped in some manner.

Nick

[nonnikcm]

thanks to both of you for your response!

I was indeed able to solve the problem, by going back to one of Jasper's suggestions.

the bucket needed to be a single value, not a path.

the item then contained the full sub-path even with the "=".

hopefully this saves someone else.

thanks again!