Problem with Amazon S3

[Miguel]

Hi everybody,
I am trying to make a simple call to S3 storage:

gsutil ls s3://<name-of-my-bucket>

.. and I always get this error:
S3ResponseError:: status=301, code=PermanentRedirect, reason=Moved
Permanently.

I checked the "boto" file and everything seems fine:
aws_access_key_id = <my-access-key>
aws_secret_access_key = <secret-access>

The same configuration works fine with s3cmd, so it is not a problem
with authentication!

Thank you

[Google Storage Team]

Hi Miguel,

I'd like to start by determining whether this problem is transient or persistent - could you retry and let us know if you continue to observe this problem? If so, can you share any information about how we might be able to reproduce it? (e.g. Is this a newly created bucket or existing bucket? Anything unusual about the bucket, its ACL or its contents? Is the bucket fixed in a particular region? Is this problem observable only with one bucket or with multiple/all of your buckets?).

Thanks,

Marc

Google Cloud Storage Team

[lex...@gmail.com]

Hi, Miguel,

please check if the command

gsutil mb s3://mytestbucket

will return successful. After that:

gsutil ls s3://mytestbucket

If this is successful, too, it is a problem with your bucket region. As always, you might wan to replace "mytestbucket" with an alternate name.

Regards,
Dirk

[lex...@gmail.com]

Just fixed "my same problem" by setting

s3_host=s3-eu-west-1.amazonaws.com

in the .boto file, right after the aws settings.

@gsutil team: is this right? Or am I missing some options/properties?

[Mike Schwartz (Google Storage Team)]

Hi,

Modifying the s3_host as you suggest will work for that bucket, but has the downside that you would need to edit the file to use a different s3_host if you want to access a bucket in a different S3 region (or, you could have multiple .boto files, one for each region).

It shouldn't be necessary to set the s3_host this way, because gsutil is implemented atop boto, and boto handles redirects for regionally named S3 buckets. If you can send us the output of running:

gsutil -d ls s3://your_bucket_name

(stripping out the Authorization: header), we'll take a look. This debug output should show the HTTP permanent redirect, followed by an attempt by boto to connect to the regionally named end point.

Thanks,

Mike

[lex...@gmail.com]

Hi, Mike,

no problem. See here:

gsutil -d ls s3://ds2mvn
***************************** WARNING *****************************
*** You are running gsutil with debug output enabled.
*** Be aware that debug output includes authentication credentials.
*** Do not share (e.g., post to support forums) debug output
*** unless you have sanitized authentication tokens in the
*** output, or have revoked your credentials.
***************************** WARNING *****************************
DEBUG: wildcard=*, prefix=None, delimiter=/, prefix_wildcard=*, suffix=
send: 'GET /ds2mvn/?&delimiter=/ HTTP/1.1\r\nHost: s3.amazonaws.com\r\nAccept-Encoding: identity\r\nDate: Sat, 28 Apr 2012 21:12:08 GMT\r\nContent-Length: 0\r\nx-goog-api-version: 2\r\nAuthorization: AWS <myaccesskey>:<anotherhash>=\r\nUser-Agent: Boto/2.3.0 (linux2) gsutil/3.3 (linux2)\r\n\r\n'
reply: 'HTTP/1.1 301 Moved Permanently\r\n'
header: x-amz-request-id: 9C1948118808606C
header: x-amz-id-2: FwIZhAPSfS3CdRMj4susmYaVZgPPJsFzdMif7pTMktxNvO8uQJ2cCL1eXJLmPfcU
header: Content-Type: application/xml
header: Transfer-Encoding: chunked
header: Date: Sat, 28 Apr 2012 21:12:09 GMT
header: Server: AmazonS3

S3ResponseError:: status=301, code=PermanentRedirect, reason=Moved Permanently.

Shouldn't there be a Location header?

Regards,
Dirk

[Mike Schwartz (Google Storage Team)]

Hi Dirk,

You're right, that response does not contain a Location header. The boto redirect handling code only handles redirects that contain a Location header, which explains why gsutil isn't working for you in this case.

I'm afraid I'm at the limit of my knowledge about why the 301 response contains no Location header. I suggest one of two options at this point:

1) You can continue using gsutil the way you have it configured, with the regionally named endpoint configured in the .boto file in order to access your bucket.

2) You could inquire at Amazon's technical support (or, possibly, on the boto-users mailing list) about why there is no HTTP Location header with 301 redirects.

Mike