ec2 module works...ec2.py does not (confused!)

[Mark Casey]

I have a role that uses the ec2 module and it is working fine, so I'm assuming I'm ok on dependencies like boto... but when I try to run ec2.py --list I get back:

root@ip-1-1-1-1:/etc/ansible/elastirax# ./ec2.py --list
Looks like AWS is down again:
EC2ResponseError: 401 Unauthorized
<?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>AuthFailure</Code><Message>AWS was not able to validate the provided access credentials</Message></Error></Errors><RequestID> removed </RequestID></Response>

That host is an ec2 instance itself. I tried it on another host (a rackspace box) after exporting the credentials the exact same way and the problem isn't there; both the role and ec2.py work there.

I've tried comparing installed packages between the two hosts (both Ubuntu 12.04.3, ansible 1.4.3) and using pip to make sure I have latest boto, but I'm not sure how to troubleshoot from here. Do I need to do something to run ec2.py on an ec2 instance...change a url to an internal one or something?

Thanks for any insight/suggestions,
Mark

[Mark Casey]

No change specifically with ec2.py... it doesn't work on the host where I wanted to use it, but it does on another host I tried it on.

However, on the host where ec2.py is not working, I thought I'd mention that I am able to use boto on the python shell:

ubuntu@ip-172-16-20-21:/etc/ansible$ python
Python 2.7.3 (default, Sep 26 2013, 20:03:06)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import boto.ec2
>>> conn = boto.ec2.connect_to_region("us-west-2",
... aws_access_key_id="redacted",
... aws_secret_access_key="redacted")
>>> reservations = conn.get_all_reservations()
>>> instances = reservations[1].instances
>>> inst = instances[0]
>>> inst.private_dns_name
u'ip-172-16-20-39.us-west-2.compute.internal'
>>>

I'm exporting credentials in bash for ec2.py, and added them temporarily to /etc/boto.cfg for the test on the console.

Thank you,
Mark

[Yves Dorfsman]

Could it be a security group issue (have you limited your outbound ports etc...?)?

Or DNS? Does the aws console resolved to the same place when digged from inside?

On 2014-01-13 16:57, Mark Casey wrote:
> No change specifically with ec2.py... it doesn't work on the host where I
> wanted to use it, but it does on another host I tried it on.
>
> However, on the host where ec2.py is not working, I thought I'd mention that I
> am able to use boto on the python shell:
>
> ubuntu@ip-172-16-20-21:/etc/ansible$ python
> Python 2.7.3 (default, Sep 26 2013, 20:03:06)
> [GCC 4.6.3] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
> >>> import boto.ec2
> >>> conn = boto.ec2.connect_to_region("us-west-2",
> ... aws_access_key_id="redacted",
> ... aws_secret_access_key="redacted")
> >>> reservations = conn.get_all_reservations()
> >>> instances = reservations[1].instances
> >>> inst = instances[0]
> >>> inst.private_dns_name
> u'ip-172-16-20-39.us-west-2.compute.internal'
> >>>
>
> I'm exporting credentials in bash for ec2.py, and added them temporarily to
> /etc/boto.cfg for the test on the console.
>
> Thank you,
> Mark
>
>
> On Monday, January 13, 2014 10:40:00 AM UTC-6, Mark Casey wrote:
>
> I have a role that uses the ec2 module and it is working fine, so I'm

> assuming I'm ok on dependencies like boto... but when I try to run *ec2.py
> --list* I get back:

>
> root@ip-1-1-1-1:/etc/ansible/elastirax# ./ec2.py --list
> Looks like AWS is down again:
> EC2ResponseError: 401 Unauthorized
> <?xml version="1.0" encoding="UTF-8"?>
> <Response><Errors><Error><Code>AuthFailure</Code><Message>AWS was not
> able to validate the provided access
> credentials</Message></Error></Errors><RequestID> removed
> </RequestID></Response>
>
>
> That host is an ec2 instance itself. I tried it on another host (a
> rackspace box) after exporting the credentials the exact same way and the
> problem isn't there; both the role and ec2.py work there.
>
> I've tried comparing installed packages between the two hosts (both Ubuntu
> 12.04.3, ansible 1.4.3) and using pip to make sure I have latest boto, but
> I'm not sure how to troubleshoot from here. Do I need to do something to
> run ec2.py on an ec2 instance...change a url to an internal one or something?
>
> Thanks for any insight/suggestions,
> Mark
>

> --
> You received this message because you are subscribed to the Google Groups
> "Ansible Project" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to ansible-proje...@googlegroups.com.
> To post to this group, send email to ansible...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.


--
Yves.

[Jharrod LaFon]

Check the version of boto. I ran into the same issue and found that the version of boto on the system was old enough that it 

wouldn't automatically pull in EC2 role credentials.

-Jharrod

[Joshua Conner]

IIRC the `ec2` module expects AWS_ACCESS_KEY and AWS_SECRET_KEY environment variables, but ec2.py expects AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables. Are you pulling your credentials from the environment? Could this be the problem?

[Mark Casey]

Thanks for all the great suggestions! Not sure if it will be today but I'll try them and post back how I get on.

[Mark Casey]

Mmkay, so...

• No outbound firewalls (within the OS or at the security group level)
• As far as DNS, I assumed the console url is the same as the endpoint and tried digging ec2.us-west-2.amazonaws.com. The response only lists one IP at a time in an alternating fashion, but after running it numerous times I've seen a few IPs show up as return values on both the host that is working and the host that is not working, so I am pretty sure they're resolving to the same list of addresses.
• I've run 'sudo pip install boto --upgrade' and got version 2.23.0
• I was actually pretty hopeful about this last one :).... but I've confirmed I get a different error (No handler was ready to authenticate. 1 handlers were checked. Check your credentials) whenever the credentials are exported to the wrong var names or are not exported at all. I also made sure and they are exported as the same names on the host that is working.
  

Finally, I downloaded a new copy of ec2.py and ec2.ini today from GitHub, but it didn't change anything.

[Jharrod LaFon]

First, verify that the IAM role credentials are present by querying them from the metadata service:

curl http://169.254.169.254/latest/meta-data/iam/security-credentials/<role-name>

Install the aws cli:

pip install aws

Then, using the credentials returned from the metadata service, query EC2:

aws ec2 describe-instances

That should help you determine if the problem is indeed with the credentials, or with ec2.py.

-Jharrod

[Mark Casey]

Heh, was just looking at this...

I think boto is throwing a non-fatal exception that ec2.py is interpreting as fatal. If I comment out the sys.exit call on line line 288 I still get the error message, but then the script prints my ec2 inventory as if nothing is wrong.

I discovered this when I made a new test.py script and basically started pulling out pieces of ec2.py to try and find where it was getting lost. I was boiler-plating in the get_instances_by_region function but didn't have add_instance yet, so I simply changed the call to add_instance to a print statement instead. At that point I got output from the print that showed it was working, but then the exception was still catching and exiting.

It makes sense to me... but I just recently started learning python so if this is in fact what is happening (while I'll be happy to try things on my end) I don't think I'm yet comfortable enough with the try/except particulars to do anything about it myself.

As a final thought, the access keys I am using are in fact for an IAM user. I keep wanting to wonder if perhaps my account is only allowed to query certain regions... but again, I don't have this problem on the other host I'm using to test with. On that host I can use these same access keys work with stock ec2.py and ec2.ini, querying all regions without issue.

HTH,
Mark

[Mark Casey]

FWIW...here is that output...

bobtheuser@ip-172-20-fake-124:~/jobs/thajob/workspace/bin/devtools/ansible$ ./ec2.py --list --refresh-cache

Looks like AWS is down again:
EC2ResponseError: 401 Unauthorized
<?xml version="1.0" encoding="UTF-8"?>

<Response><Errors><Error><Code>AuthFailure</Code><Message>AWS was not able to validate the provided access credentials</Message></Error></Errors><RequestID>cefef..... ... ...</RequestID></Response>
{
  "_meta": {
    "hostvars": {
      "54.fake.141.999": {
        "ec2__in_monitoring_element": false,
        "ec2_ami_launch_index": "0",
        "ec2_architecture": "x86_64",
        "ec2_client_token": "test9",
        "ec2_dns_name": "ec2-54-fake-141-999.us-west-2.compute.amazonaws.com",
        "ec2_ebs_optimized": false,

[Mark Casey]

For posterity... I downloaded a fresh copy of ec2.py today and this was no longer an issue. In the new copy there was an 'if' that changed to an 'if not' near the lines of the exception catch I had to comment out before, so I'm assuming that was it.