Problem encoding pluses for Amazon S3 keys using net/url/URL (via goamz/s3)

[Bryan Murphy]

Hello All,

I've been trying to fix a few bugs in the launchpad.net/goamz/s3 library and have run into a problem.  Amazon S3 does not encode URL paths the same way as other web servers (apache, nginx).  It treates + as space in the path (instead of +).  Please see these two threads for more context:

https://forums.aws.amazon.com/message.jspa?threadID=16361

https://forums.aws.amazon.com/message.jspa?messageID=208136

In order to properly integrate with S3, +'s need to be encoded as %2B, so a file named "Users + Email.txt" would be encoded as either "Users+%2B+Email.txt", "Users%20%2B%20Email.txt" or "Users %2B Email.txt".  Essentially, the path submitted to S3 needs to be URL encoded.

The problem I have is that net/http/client takes net/url/URL (via net/http/Request) as input.  net/http/client.doFollowingRedirects calls net/url/URL.String() which returns an a properly encoded url, per the HTTP spec, but not what S3 needs.  If I specify + in the path, + gets treated as a space (by S3).  If I specify %2B in the path, it gets double encoded as %252B (by URL.String).  

Is there a way I can change the behavior of either client.doFollowingRedirects or URL.String without having to fork the library code?  In Python/Ruby I would monkey-patch net/url/URL or net/http/client for this particular case, but I have not yet been able to figure out a way to fix this in go.

I'm willing to work on the net/http library to allow better control over url path encoding, but I wanted to know if there were any other approaches I could try.

Thanks!

Bryan

[r...@rafiksalama.com]

I just came across this same problem. I was hoping to come across a solution here, but after a few hours, I was able to come up with one myself. The URL that's sent in the HTTP header is not url.String() but url.ResourceURI(). To get around the path component getting escaped, manually construct a url.URL yourself, leave the Path component blank, and use Opaque instead. ResourceURI() won't perform any encoding on the Opaque portion of a URL.