Bucket doesn't exist but not getting S3ResponseError with get_bucket() method

[Diogo Vieira]

Hello,

I'm using boto's S3 API and whenever I try to get a bucket from a connection with a certain name (of a bucket that doesn't exist but probably existed in the past, not entirely sure; if it did in fact existed it would have been removed weeks or months ago) I'm not getting a S3ResponseError exception as would be expected. The bucket doesn't show up in S3's management console nor is it returned from get_all_buckets method. Can someone help me understand what is going on and how can I check if a bucket exists or not?

Thank you in advance,

Diogo Vieira

[Adrian Klaver]

Not sure what is going on, but it would help to have the exact version
of boto you are using and the actual code snippet(bucket name changed if
necessary).

Also what does connection.lookup(bucket_name) show?

>
> Thank you in advance,
> Diogo Vieira
>

--
Adrian Klaver
adrian...@aklaver.com

[Diogo Vieira]

Sure, no problem:

>>> import boto

>>> boto.__version__

'2.25.0'

>>> from boto.s3.connection import S3Connection

>>> connection = S3Connection("secret","secret")

>>> connection.lookup("ola")

<Bucket: ola>

>>> connection.get_bucket("ola")

<Bucket: ola>

Thank you,

Diogo Vieira

[Adrian Klaver]

So it exists.

Yet you say it does not show up in get_all_buckets?

Is this a location specific bucket?


>
>
> Thank you,
> Diogo Vieira
>
> --

--
Adrian Klaver
adrian...@aklaver.com

[Diogo Vieira]

Not sure what is going on, but it would help to have the exact version
of boto you are using and the actual code snippet(bucket name changed
if necessary).

Also what does connection.lookup(bucket_name) show?

Sure, no problem:

    >>> import boto
    >>> boto.__version__
   '2.25.0'
    >>> from boto.s3.connection import S3Connection
    >>> connection = S3Connection("secret","secret")
    >>> connection.lookup("ola")
   <Bucket: ola>
    >>> connection.get_bucket("ola")
   <Bucket: ola>

So it exists.

Yet you say it does not show up in get_all_buckets?

Is this a location specific bucket?

That's the problem. It doesn't show up in management console and no, it doesn't show up in get_all_buckets.

>>> [bucket for bucket in connection.get_all_buckets() if bucket.name == "ola"]

[]

Not as far as I know.

Thank you,

Diogo Vieira

[Adrian Klaver]

What if you do not do the == "ola" and just look at the list, on the
chance it is a case/encoding issue?

>
>
> Not as far as I know.

Well on the bucket object you return from lookup() do

bucket.get_location()


I have to run, so I will be able to answer for awhile. My suspicion is
it it is region and/or permissions issue(s).

FYI boto is now up to 2.27. A search of the issues:

https://github.com/boto/boto/issues

might indicate that upgrading is the way to go.

>
> Thank you,
> Diogo Vieira
>

--

Adrian Klaver
adrian...@aklaver.com

[Diogo Vieira]

Diogo Vieira <d...@eurotux.com>
Programador 
Eurotux Informática, S.A. | www.eurotux.com

(t) +351 253 680 300

On Apr 16, 2014, at 6:06 PM, Adrian Klaver <adrian...@aklaver.com> wrote:

On 04/16/2014 09:57 AM, Diogo Vieira wrote:

Not sure what is going on, but it would help to have the exact version
of boto you are using and the actual code snippet(bucket name changed
if necessary).

Also what does connection.lookup(bucket_name) show?

Sure, no problem:

   >>> import boto
   >>> boto.__version__
  '2.25.0'
   >>> from boto.s3.connection import S3Connection
   >>> connection = S3Connection("secret","secret")
   >>> connection.lookup("ola")
  <Bucket: ola>
   >>> connection.get_bucket("ola")
  <Bucket: ola>

So it exists.

Yet you say it does not show up in get_all_buckets?

Is this a location specific bucket?

That's the problem. It doesn't show up in management console and no, it
doesn't show up in get_all_buckets.

    >>> [bucket for bucket in connection.get_all_buckets() if
   bucket.name == "ola"]
   []

What if you do not do the == "ola" and just look at the list, on the chance it is a case/encoding issue?

I did. I just didn't want to list all the buckets in the email :)

Not as far as I know.

Well on the bucket object you return from lookup() do

bucket.get_location()

Even more strange, I believe:

>>> connection.get_bucket("ola").get_location()

Traceback (most recent call last):

  File "<stdin>", line 1, in <module>

  File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 1131, in get_location

    response.status, response.reason, body)

boto.exception.S3ResponseError: S3ResponseError: 403 Forbidden

Followed by an XML specifying Access Denied as the error. But I can't even see it in the management console so I can't change permissions.

I have to run, so I will be able to answer for awhile. My suspicion is it it is region and/or permissions issue(s).

No problem, thank you for your help!

FYI boto is now up to 2.27. A search of the issues:

https://github.com/boto/boto/issues

might indicate that upgrading is the way to go.

I tried searching for similar issues but haven't found anything so far.

Thank you,

Diogo Vieira

[Mitchell Garnaat]

I'm not sure what is going on but remember that the S3 bucket name space is global.  So, if someone else created a bucket called "ola" and happened to have the permissions set up such that everyone had read access to the bucket (but not to it's content) I think you would get this behavior.  It doesn't show up when you list all of your buckets because it's not your bucket.

Mitch

--

You received this message because you are subscribed to the Google Groups "boto-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to boto-users+...@googlegroups.com.
To post to this group, send email to boto-...@googlegroups.com.
Visit this group at http://groups.google.com/group/boto-users.
For more options, visit https://groups.google.com/d/optout.

[Adrian Klaver]

On 04/16/2014 10:15 AM, Diogo Vieira wrote:
>
> Diogo Vieira <d...@eurotux.com <mailto:d...@eurotux.com>>
> Programador
> Eurotux Informática, S.A. | www.eurotux.com <http://www.eurotux.com>

> (t) +351 253 680 300
>
> On Apr 16, 2014, at 6:06 PM, Adrian Klaver <adrian...@aklaver.com

> <mailto:adrian...@aklaver.com>> wrote:
>

>> Well on the bucket object you return from lookup() do
>>
>> bucket.get_location()
>
> Even more strange, I believe:
>
> >>> connection.get_bucket("ola").get_location()
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line
> 1131, in get_location
> response.status, response.reason, body)
> boto.exception.S3ResponseError: S3ResponseError: 403 Forbidden
>
>
> Followed by an XML specifying Access Denied as the error. But I can't
> even see it in the management console so I can't change permissions.

Well this would seem to verify Mitchs post on the global nature of
bucket names. In other words you are looking at someone else's bucket
that has the same name as the bucket you created in the past.

[Diogo Vieira]

>>> Well on the bucket object you return from lookup() do
>>>
>>> bucket.get_location()
>>
>> Even more strange, I believe:
>>
>> >>> connection.get_bucket("ola").get_location()
>> Traceback (most recent call last):
>> File "<stdin>", line 1, in <module>
>> File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line
>> 1131, in get_location
>> response.status, response.reason, body)
>> boto.exception.S3ResponseError: S3ResponseError: 403 Forbidden
>>
>>
>> Followed by an XML specifying Access Denied as the error. But I can't
>> even see it in the management console so I can't change permissions.
>
> Well this would seem to verify Mitchs post on the global nature of bucket names. In other words you are looking at someone else's bucket that has the same name as the bucket you created in the past.
>

> On Apr 16, 2014, at 6:25 PM, Mitchell Garnaat <mi...@garnaat.com> wrote:
>
>> I'm not sure what is going on but remember that the S3 bucket name space is global. So, if someone else created a bucket called "ola" and happened to have the permissions set up such that everyone had read access to the bucket (but not to it's content) I think you would get this behavior. It doesn't show up when you list all of your buckets because it's not your bucket.
>>
>> Mitch

Well, that's a bit counterintuitive while reading through the API, but OK, I guess it makes sense. So in order to check if a bucket exists and is mine I would have to list my buckets and see if it exists? My only problem with that is that it seems to be a slower method to check if it exists since you have to list them all.

Thank you,
Diogo Vieira