get_bucket leads to 403 Forbidden while s3cmd ls works

[Yaroslav Halchenko]

What could be the reason, e.g. which policy setting would forbid (I am not administering that bucket) me being able to connect to the bucket at all

$> ./s3-ls ~/.s3cfg-hcp500 s3://hcp-openaccess/HCP/142626/.xdlm/142626_3T_tfMRI_LANGUAGE_preproc.json
Connecting to bucket: hcp-openaccess
Traceback (most recent call last):
  File "./s3-ls", line 34, in <module>
    bucket = conn.get_bucket(bucket_name)
  File "/usr/lib/python2.7/dist-packages/boto/s3/connection.py", line 502, in get_bucket
    return self.head_bucket(bucket_name, headers=headers)
  File "/usr/lib/python2.7/dist-packages/boto/s3/connection.py", line 535, in head_bucket
    raise err
boto.exception.S3ResponseError: S3ResponseError: 403 Forbidden

while e.g. "s3cmd ls" works (although info doesn't):

$> s3cmd -c ~/.s3cfg-hcp500 ls s3://hcp-openaccess                                                      
                       DIR   s3://hcp-openaccess/HCP/

$> s3cmd -c ~/.s3cfg-hcp500 info s3://hcp-openaccess
ERROR: Access to bucket 'hcp-openaccess' was denied
ERROR: S3 error: 403 (AccessDenied): Access Denied

for completeness, that s3-ls script of mine is here: https://github.com/yarikoptic/datalad/blob/rf-crawler-new-design/tools/s3-ls

Any help would be much appreciated.

[rosswil...@gmail.com]

With the boto library when a bucket connection is made boto will attempt to validate that the bucket exists by listing all buckets and seeing if its in the list. The policy needed to allow this to go through might look something like:

        {
            "Effect": "Allow",
            "Action": "s3:ListAllMyBuckets",
            "Resource": "*",
            "Condition": {}
        }

Added to the list of statements in the policy that governs the account. 

If you can modify the s3-ls code (not ideal) you can skip the check to see if the bucket exists by adding "validate=False" to the boto call for get_bucket:

http://stackoverflow.com/questions/12571217/python-amazon-s3-cannot-get-the-bucket-says-403-forbidden

[Yaroslav Halchenko]

Thank you!  I thought I had tried with validate=False with no success, but have tried it again now -- indeed works as desired

Meanwhile I have overcome the problem by finding this recipe somewhere -- surprisingly I could get_all_buckets() (not sure how that is possible) and match the bucket by desired name. (see e.g. https://github.com/yarikoptic/datalad/blob/rf-crawler-new-design/tools/s3-ls#L36)