AWS China Support/Issues - AWS was not able to validate the provided access credentials - Launching Instance

[K Cheng]

Hi,

Apologies if this has already been asked previously - I couldn't seem to find the relevant information I required.

I'm having issues running a playbook to launch a EC2 instance in AWS China. The same playbook works in non-China AWS accounts.

The error I am receiving is the following:

TASK [Launch instance] *********************************************************

ESTABLISH LOCAL CONNECTION FOR USER: ubuntu

127.0.0.1 EXEC (<REDACTED>)

127.0.0.1 PUT<REDACTED>

127.0.0.1 EXEC LANG=C LC_MESSAGES=C LC_CTYPE=C <REDACTED> > /dev/null 2>&1

An exception occurred during task execution. The full traceback is:

Traceback (most recent call last):

  File "/Users/Etherdaemon/.ansible/tmp/ansible-tmp-1445294739.44-143915179278919/ec2", line 3429, in <module>

    main()

  File "/Users/Etherdaemon/.ansible/tmp/ansible-tmp-1445294739.44-143915179278919/ec2", line 1392, in main

    (tagged_instances, instance_dict_array, new_instance_ids, changed) = enforce_count(module, ec2, vpc)

  File "/Users/Etherdaemon/.ansible/tmp/ansible-tmp-1445294739.44-143915179278919/ec2", line 771, in enforce_count

    reservations, instances = find_running_instances_by_count_tag(module, ec2, count_tag, zone)

  File "/Users/Etherdaemon/.ansible/tmp/ansible-tmp-1445294739.44-143915179278919/ec2", line 558, in find_running_instances_by_count_tag

    reservations = get_reservations(module, ec2, tags=count_tag, state="running", zone=zone)

  File "/Users/Etherdaemon/.ansible/tmp/ansible-tmp-1445294739.44-143915179278919/ec2", line 617, in get_reservations

    results = ec2.get_all_instances(filters=filters)

  File "/usr/local/lib/python2.7/site-packages/boto/ec2/connection.py", line 585, in get_all_instances

    max_results=max_results)

  File "/usr/local/lib/python2.7/site-packages/boto/ec2/connection.py", line 681, in get_all_reservations

    [('item', Reservation)], verb='POST')

  File "/usr/local/lib/python2.7/site-packages/boto/connection.py", line 1186, in get_list

    raise self.ResponseError(response.status, response.reason, body)

boto.exception.EC2ResponseError: EC2ResponseError: 401 Unauthorized

<?xml version="1.0" encoding="UTF-8"?>

<Response><Errors><Error><Code>AuthFailure</Code><Message>AWS was not able to validate the provided access credentials</Message></Error></Errors><RequestID>88e083e5-bdcf-4bf0-95ba-740ceb5ec790</RequestID></Response>

fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "parsed": false}

My playbook looks like this:

---

- name: Create an All In One Environment

  hosts: localhost

  vars_files:

    - "../variable_files/{{ env }}_vars.yml"

  gather_facts: False

  tags:

    - create

  tasks:

    - include: tasks/iam.yml

    - name: Launch instance

      ec2:

        key_name: "{{ key_name }}"

        group: "{{ security_group }}"

        instance_type: "{{ instance_type }}"

        image: "{{ image }}"

        wait: true

        region: "{{ region }}"

        vpc_subnet_id: "{{ subnet_id }}"

        assign_public_ip: no

        instance_tags:

          Name: "{{ role }}{{ '%02d'|format(instance_count|int) }}.{{ env }}.aws-{{ region_shortname }}.{{ tld_name_internal }}"

          Environment: "{{ environment_name }}"

          Role: "{{ role }}"

        exact_count: "{{ instance_count }}"

        count_tag: 

          Environment: "{{ environment_name }}"

          Role: "{{ role }}"

        instance_profile_name: "{{ game }}_{{ env }}_ec2_role"

        volumes:

          - device_name: /dev/xvdb

            ephemeral: ephemeral0

          - device_name: /dev/xvdc

            ephemeral: ephemeral1

        user_data: "{{ lookup('file', 'files/userdata.txt') }}"

      register: ec2

    - name: Add new instance to host group

      add_host: hostname={{ item.public_ip }} groupname="{{ environment_name }}"

      with_items: ec2.instances

    - name: Wait for SSH to come up

      wait_for: host={{ item.public_dns_name }} port=22 delay=60 timeout=320 state=started

      with_items: ec2.instances

• I am running Ansible from source - v2.0.0-0.3.beta1
  
• I am also running boto (2.38.0) and boto3 (1.1.4)
• I am using /.boto config with a profile named china
• If I run the ec2.py inventory script by itself using the boto profile and credentials it runs correctly with no errors
• If I run from a local inventory instead - the same error occurs
• I have checked my IAM credentials and it has full access
• I suspect the issue is around boto using the get_all_instances command 

Can anyone advise if they have Ansible running in AWS China atm or if anyone has an idea on what the issue is?

Any help greatly appreciated.

Cheers

Karen

[Brian Coca]

Only accounts created in the China region can access the China region
https://www.amazonaws.cn/en/sign-up/, this is an AWS restriction and
Ansible cannot do anything about it.



--
Brian Coca

[K Cheng]

I have an AWS China account with IAM credentials which is what I am trying to use with Ansible.

I've found a handful of github Ansible/China related issues/posts but I can't tell whether Ansible supports China or not. 

Karen

[Brian Coca]

Ansible does not restrict the regions, that is what your credentials +
AWS API do, are you sure you are using the correct credentials?
The error message you got seems to indicate that this is not the case.


--
Brian Coca

[K Cheng]

Yes - only thing that is different is that China and some other regions AWS has supports only Signature v4 while other AWS regions support both v2 and v4 Signatures so not sure if that has been catered for.

In terms of credentials - yes they are correct - they work for both the ec2.py dynamic inventory script, terraform scripts, etc. I even created a second set of API keys and tried those as well.

Karen

[Brian Coca]

So they should work 'in principle' I know other people have been using
the China region, not sure about the signatures though, is that a
recent change?



--
Brian Coca

[K Cheng]

Thanks Brian, in terms of the Signature - its been there from day 1 from memory.

In terms of getting help on this issue, would this be the best place to ask for help on this matter? Not sure how to proceed and AWS China is something we need to have working.

I was working on terraform originally but wasn't liking the way the variables and structure was going so thought I would take a look at Ansible instead. I would really prefer to stick with Ansible as I like what I've seen so far.

Karen

[K Cheng]

Hmm - looks like its something to do with the profile lookup in boto.

I've hardcoded the credentials into the playbook temporarily to test and launching of an instance now works - creating an IAM role does not at the moment and returns:

The security token included in the request is invalid."

[Brian Coca]

Open a github issue, we might be able to work around this, afaik those
using AWS in the China zone were using 1.x versions of ansible, there
were many changes introduced in 2.0 (current devel).



--
Brian Coca

[K Cheng]

No worries - thanks Brian.