TooManyAuthHandlerReadyToAuthenticate Error on Ubuntu but not on Mac?

[Paul Wiseman]

I wrote a server which needs to access both S3 and GS, and all seems
to be working fine on mac which I develop on. Now has come the time to
test it on a server, I put it on a Google instance and the same code
that runs fine on Mac is throwing an error
(TooManyAuthHandlerReadyToAuthenticate) on the Google box (Ubuntu).

I can replicate the error with these lines:

from boto.s3.connection import S3Connection
from boto.gs.connection import GSConnection
s3conn = S3Connection(ACCESS_KEY_S3, SECRET_KEY_S3)
gsconn = GSConnection(ACCESS_KEY_GS, SECRET_KEY_GS)

Creating the gsconn throws the error on the Ubuntu box only.

Both machines are running python 2.7.3 and both have boto-2.6.0_dev
installed. I can't figure out why they'd behave differently?

Should I not be creating 2 connections? If not why does it work on Mac?

All I need the connection objects for is the generate_url methods, so
maybe there's a work around? Possibly I could use one connection to
make urls for both as I think they're both signed the same way?

Any insight would be great! This has me stumped

Paul

[Mitchell Garnaat]

Hmm, I'm unable to reproduce this.  I created a fresh Ubuntu quantal EC2 instance, cloned the develop branch of boto and installed it in a virtualenv.  I then followed the exact steps you have below and did not receive the error.  I also tried it without the virtualenv.  Again, no error.

I've never seen this error come up before.  Basically, when a connection is created it advertises the type of authentication mechanism it needs and then a list of pluggable authentication modules are searched.  If no modules provide the requested authentication, you get a boto.exception.NoAuthHandlerFound exception.  If one module is found, it uses it.  If more than one module claims to support that authentication method, you get the error you are receiving.  So, somehow, there are multiple pluggable auth modules that claim to support the "s3" authentication.

Mitch

Paul

--
You received this message because you are subscribed to the Google Groups "boto-users" group.
To post to this group, send email to boto-...@googlegroups.com.
To unsubscribe from this group, send email to boto-users+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/boto-users?hl=en.

[Paul Wiseman]

On 18 December 2012 14:27, Mitchell Garnaat <mi...@garnaat.com> wrote:
> Hmm, I'm unable to reproduce this. I created a fresh Ubuntu quantal EC2
> instance, cloned the develop branch of boto and installed it in a
> virtualenv. I then followed the exact steps you have below and did not
> receive the error. I also tried it without the virtualenv. Again, no
> error.
>

I just tried it on EC2 and it didn't raise the error either, however
on this google instance I'm running I get it without fail.

I might create a fresh one and see if I still get the problem.. very weird!

> I've never seen this error come up before. Basically, when a connection is
> created it advertises the type of authentication mechanism it needs and then
> a list of pluggable authentication modules are searched. If no modules
> provide the requested authentication, you get a
> boto.exception.NoAuthHandlerFound exception. If one module is found, it
> uses it. If more than one module claims to support that authentication
> method, you get the error you are receiving. So, somehow, there are
> multiple pluggable auth modules that claim to support the "s3"
> authentication.
>

Is there anyway that this plugin.Plugin list could change? Could I
just drop the HmacAuthV1Handler from the list?

I just got boto with 'pip install boto' so I don't think it's been
modified at all.

PS full trace back:

Python 2.7.3 (default, Aug 1 2012, 05:14:39)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.

>>> from boto.s3.connection import S3Connection
>>> from boto.gs.connection import GSConnection

>>> s3conn = S3Connection("","")
>>> gsconn = GSConnection("","")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python2.7/dist-packages/boto/gs/connection.py",
line 46, in __init__
suppress_consec_slashes=suppress_consec_slashes)
File "/usr/local/lib/python2.7/dist-packages/boto/s3/connection.py",
line 169, in __init__
validate_certs=validate_certs)
File "/usr/local/lib/python2.7/dist-packages/boto/connection.py",
line 548, in __init__
host, config, self.provider, self._required_auth_capability())
File "/usr/local/lib/python2.7/dist-packages/boto/auth.py", line
647, in get_auth_handler
requested_capability))
boto.exception.TooManyAuthHandlerReadyToAuthenticate: 2 AuthHandlers
['ComputeAuth', 'HmacAuthV1Handler'] ready to authenticate for
requested_capability ['s3'], only 1 expected. This happens if you
import multiple pluging.Plugin implementations that declare support
for the requested_capability.

[Adrian Klaver]

On 12/18/2012 07:15 AM, Paul Wiseman wrote:
> On 18 December 2012 14:27, Mitchell Garnaat <mi...@garnaat.com> wrote:

>> On Tue, Dec 18, 2012 at 2:50 AM, Paul Wiseman <poa...@gmail.com> wrote:
>>>
>>> I wrote a server which needs to access both S3 and GS, and all seems
>>> to be working fine on mac which I develop on. Now has come the time to
>>> test it on a server, I put it on a Google instance and the same code
>>> that runs fine on Mac is throwing an error
>>> (TooManyAuthHandlerReadyToAuthenticate) on the Google box (Ubuntu).
>>>
>>> I can replicate the error with these lines:
>>>
>>> from boto.s3.connection import S3Connection
>>> from boto.gs.connection import GSConnection
>>> s3conn = S3Connection(ACCESS_KEY_S3, SECRET_KEY_S3)
>>> gsconn = GSConnection(ACCESS_KEY_GS, SECRET_KEY_GS)
>>>
>>> Creating the gsconn throws the error on the Ubuntu box only.
>>>
>>> Both machines are running python 2.7.3 and both have boto-2.6.0_dev
>>> installed. I can't figure out why they'd behave differently?
>>>
>>> Should I not be creating 2 connections? If not why does it work on Mac?
>>>
>>> All I need the connection objects for is the generate_url methods, so
>>> maybe there's a work around? Possibly I could use one connection to
>>> make urls for both as I think they're both signed the same way?
>>>
>>> Any insight would be great! This has me stumped

From this Google tutorial, see the Note:

https://developers.google.com/compute/docs/images

4. Upload Your Image to Google Cloud Storage

You must store your image on Google Cloud Storage. You will later load
the image from that location during instance startup.

To upload your image to Google Cloud Storage, use the gsutil command
line tool that comes preinstalled on your instance.

Set up gsutil.
If this is your first time using gsutil on this instance AND you didn't
set up your instance to use a service account to talk to Google Cloud
Storage, run gsutil config and follow the instructions. Otherwise, you
can skip this step.
*Note:* If your instance is set up with a service account scope to
Google Cloud Storage and you attempt to run gsutil config, the command
will fail with an error similar to the following:
AuthHandlers ['ComputeAuth', 'OAuth2Auth'] ready to authenticate for

requested_capability ['s3'], only 1 expected.

In this case gsutil is Googles version of boto.

>>>
>>> Paul
>>>



--
Adrian Klaver
adrian...@gmail.com

[Michael Schwartz]

Hi,

The problem in this case isn't the fact that gsutil has its own bundled version of boto. It's that Google Compute Engine installs a plugin capable of performing service account auth in /etc/boto.cfg, and the user ran gsutil config (which sets up a ~/.boto file with OAuth2 credentials). This causes two different plugins to be loaded that both declare the same auth capability ('s3'), and the boto plugin framework disallows that.

I've been talking with the GCE folks about how to solve this problem, and I'd appreciate the boto community's feedback on the idea we discussed: Instead of raising an exception when two different auth plugins are ready to authenticate for the same capability, we propose a change so that the last one in the path is used. This is similar in spirit to how the Unix shell will pick the first version of a program along the PATH if there is more than one program with the same name found in the path - however in this case we think the logic should be to pick the last auth plugin on the boto config path, to allow a user to customize their auth. To clarify this latter point, the boto config file path allows users to load / mix in multiple configurations, which can be useful for various sharing scenarios (e.g., see https://developers.google.com/storage/docs/collaboration?hl=en#using). In this arrangement, the first config (/etc/boto.cfg) is shared across users, and the second config (~/.boto) is the user's personal config. So, if the boto auth plugin framework simply picked the last plugin capable of performing then specified auth, it would solve the problem.

(I would still leave the code raising an exception if there is more than one same-capa plugin found from one boto config file.)

The original reason the plugin framework disallowed multiple auth plugins willing to perform the same auth is:

# NOTE: Even though it would be nice to accept more than one handler

# by using one of the many ready handlers, we are never sure that each

# of them are referring to the same storage account. Since we cannot

# easily guarantee that, it is always safe to fail, rather than operate

# on the wrong account.

I was one of the reviewers of the auth plugin framework code, and although I tthink the above comment captures a potential problem, I think the path-based interpretation we're proposing would not make that problem worse.



Thoughts welcome.


Thanks,


Mike

 

Paul

--
Adrian Klaver
adrian...@gmail.com

--
You received this message because you are subscribed to the Google Groups "boto-users" group.
To post to this group, send email to boto-...@googlegroups.com.

To unsubscribe from this group, send email to boto-users+unsubscribe@googlegroups.com.

[Paul Wiseman]

I don't have a ~/.boto file (In /root or in /home/paul/), I only have
the /etc/boto.cfg

In /usr/share/google/boto/boto_plugins/ there's a compute_auth.py - I
moved it out this folder to see what would happen and now I can
successfully create the GSConnection object, although I don't know if
it will auth properly yet with that file missing.

>
> I've been talking with the GCE folks about how to solve this problem, and
> I'd appreciate the boto community's feedback on the idea we discussed:
> Instead of raising an exception when two different auth plugins are ready to
> authenticate for the same capability, we propose a change so that the last
> one in the path is used. This is similar in spirit to how the Unix shell
> will pick the first version of a program along the PATH if there is more
> than one program with the same name found in the path - however in this case
> we think the logic should be to pick the last auth plugin on the boto config
> path, to allow a user to customize their auth. To clarify this latter point,
> the boto config file path allows users to load / mix in multiple
> configurations, which can be useful for various sharing scenarios (e.g., see
> https://developers.google.com/storage/docs/collaboration?hl=en#using). In
> this arrangement, the first config (/etc/boto.cfg) is shared across users,
> and the second config (~/.boto) is the user's personal config. So, if the
> boto auth plugin framework simply picked the last plugin capable of
> performing then specified auth, it would solve the problem.
>

+1 That sounds like a good idea

btw I've worked around the problem for now with this, I have no idea
how well this will work for all tasks but it's enough so that it
doesn't raise the exception I had originally. I can upload to GS and
generate urls which is all I need.

def get_gsconn(secure=False):
gs_conn= S3Connection(settings.ACCESS_KEY_GS,settings.SECRET_KEY_GS,
is_secure=secure)
gs_conn.QueryString = 'Signature=%s&Expires=%d&GoogleAccessId=%s'
gs_conn.provider = Provider("google", settings.ACCESS_KEY_GS,
settings.SECRET_KEY_GS)
gs_conn.host = gs_conn.provider.host = gs_conn.DefaultHost =
'storage.googleapis.com'
return gs_conn

> (I would still leave the code raising an exception if there is more than one
> same-capa plugin found from one boto config file.)
>
> The original reason the plugin framework disallowed multiple auth plugins
> willing to perform the same auth is:
>
> # NOTE: Even though it would be nice to accept more than one handler
> # by using one of the many ready handlers, we are never sure that each
> # of them are referring to the same storage account. Since we cannot
> # easily guarantee that, it is always safe to fail, rather than operate
> # on the wrong account.
>
>
> I was one of the reviewers of the auth plugin framework code, and although I
> tthink the above comment captures a potential problem, I think the
> path-based interpretation we're proposing would not make that problem worse.
>
>
>
> Thoughts welcome.
>
>
> Thanks,
>
>
> Mike
>
>
>
>
>>>>>
>>>>> Paul
>>>>>
>>
>>
>>
>> --
>> Adrian Klaver
>> adrian...@gmail.com
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "boto-users" group.
>> To post to this group, send email to boto-...@googlegroups.com.
>> To unsubscribe from this group, send email to

>> boto-users+...@googlegroups.com.

[Michael Schwartz]

Paul,

Sorry, I should have been more clear. The problem arises if you have a plugin_directory specified in one or more of the boto config files boto finds (by default, /etc/boto.cfg and ~/.boto), and that plugin directory contains a plugin that declares 's3' capability, and you have OAuth2 credentials in your boto config file (gs_oauth2_refresh_token). Do you have gs_oauth2_refresh_token in your boto config file?

Also, do you have either the BOTO_CONFIG or BOTO_PATH env variables set? If so, you may be pulling in additional boto config files beyond /etc/boto.cfg.

[Paul Wiseman]

Na, the only thing in there about plugins is this:

[Plugin]
plugin_directory = /usr/share/google/boto/boto_plugins

There is also this comment at the top (I don't know if this is typical)

# This file is automatically created at boot time by the
# /usr/share/google/boto/boot_setup.py script. Do not edit this file
# directly. If you need to add items to this file, create/edit
# /etc/boto.cfg.template instead and then re-run the script.

The compute_auth.py in the boto plugins folder I now notice has a
class variable capability = ['google-oauth2', 's3'] - So I guess this
is the problem? Presumably I can just take out s3 of that list and it
shouldn't collide with the other auth handler?

[Michael Schwartz]

Hi Paul,

You don't want to remove that capability spec from the plugin.

I have a fix for this problem; I posted a note on boto-dev about it, to give the boto developer community a chance to look and comment before I commit the change.

Mike