Working around lookup plugin limitation with environment variables

[Rob White]

Hi all,

I'm looking for any suggestions on how to work around the following issue:

I am trying to use a lookup plugin but it is ignoring my environment variables.

The lookup plugin I am trying to use is credstash and the environment variables are being set because I am using AWS STS to assume a role.

It looks a little like this:

---
- name: Get STS token
  hosts: "{{ hosts }}"
  connection: local
  tasks:
    - name: Assume role for passed account
      sts_assume_role:
        region: "{{ aws_region }}"
        role_arn: "arn:aws:iam::{{ aws_account_id }}:role/ansible-is_assumed"
        role_session_name: ansible_assumed_role
      register: assumed_role
      when: use_sts

    - name: Set environment with AWS creds when available
      set_fact:
        my_environment:
          AWS_ACCESS_KEY_ID: "{{ assumed_role.sts_creds.access_key }}"
          AWS_SECRET_ACCESS_KEY: "{{ assumed_role.sts_creds.secret_key }}"
          AWS_SECURITY_TOKEN: "{{ assumed_role.sts_creds.session_token }}"
      when: assumed_role.skipped is not defined

    - name: Set environment blank when no AWS creds
      set_fact:
        my_environment: {}
      when: assumed_role.skipped is defined

- name: Deploy infrastructure role
  hosts: "{{ hosts }}"
  connection: local
  roles:
    - "{{ role_to_deploy }}"
  environment: "{{ my_environment }}"

Now in the role_to_deploy i have lots of tasks that work brilliantly calling out to AWS using the STS credentials supplied as intended.

Unfortunately, any lookup plugins do not work properly because they do not use the environment variables I have set.  They end up calling out to AWS using the default credentials rather than being overridden by the STS creds.

I don't think this is a bug but it is never the less annoying.

I'm hoping maybe one of the core team can think of a way I can get the lookup plugin to 'see' my STS environment variables I have set.

[Brian Coca]

The environment keyword ONLY affects remote task execution (invoked
inside connection), not the 'local' environment which is what lookups
run under.

----------
Brian Coca

[Rob White]

Indeed, hence why it doesn't work.

Is there any other way to pass an environment variable to a lookup plugin?

The only thing I've come up with is a not very elegant:

lookup('pipe', '(export AWS_ACCESS_KEY_ID=' + assumed_role.sts_creds.access_key + ' && export AWS_SECRET_ACCESS_KEY=' + assumed_role.sts_creds.secret_key + ' && export AWS_SECURITY_TOKEN=' + assumed_role.sts_creds.session_token + ' && credstash -r ' + aws_region + ' -t ' + env + '-credstash get rds_instance_mysql-1_username)')

This then fails when I am not using STS.

[Brian Coca]

set them in the environment before executing Ansible

AWS_SECURITY_TOKEN=xxx ansible-playbook ....
----------
Brian Coca

[Rob White]

The problem is it is the ansible-playbook that is actually going off to get the STS creds using sts_assume_role module so until i run the play, i don't have the creds.

I guess I could invoke a playbook from within another playbook :|

[Nico H]

Hey, have you ever find a solution here ? 

I face the same problem  trying to pass my sts credentials in environment for a role, like this : 

pre_tasks :

    - name: Assume role

      sts_assume_role:

        role_arn: "arn:aws:iam:::role/myrole"

        role_session_name: "mysession"

        region: "{{ my_region }}"

      register: assumed_role

roles:

 - myrole

environment:

    AWS_ACCESS_KEY_ID: "{{ assumed_role.sts_creds.access_key }}"

    AWS_SECRET_ACCESS_KEY: "{{ assumed_role.sts_creds.secret_key }}"

    AWS_SECURITY_TOKEN: "{{ assumed_role.sts_creds.session_token }}"

however it doesn't work because environment is evaluated before everything else, so my pre-task has not been run yet and the credentials (assumed_role variable) doesn't exist .

not sure how to pass credential as environment in this case ?

[Stefan Hornburg (Racke)]

Hello Nico,

I recommended to switch to tasks only in your playbook instead of using pre_tasks and roles keywords.
This should solve your problem and in my opinion it also looks more consistent:

tasks:

- name: Assume role
sts_assume_role:
role_arn: "arn:aws:iam:::role/myrole"
role_session_name: "mysession"
region: "{{ my_region }}"
register: assumed_role

- name: Import my role
import_role:
name: myrole

environment:
AWS_ACCESS_KEY_ID: "{{ assumed_role.sts_creds.access_key }}"
AWS_SECRET_ACCESS_KEY: "{{ assumed_role.sts_creds.secret_key }}"
AWS_SECURITY_TOKEN: "{{ assumed_role.sts_creds.session_token }}"

Regards
Racke

--
Ecommerce and Linux consulting + Perl and web application programming.
Debian and Sympa administration. Provisioning with Ansible.

[Nico H]

Thanks Racke, 

sure , so in this case environment is assigned to the tasks level and should therefore not fail cause it is not loaded before runtime. 

In the case I have multiple roles, I have to repeat the task as many times as I need a role , lots of duplicate environment then .

Can we pass a list or role in import_role or include_role ? I couldn't make this working with "loop"

cheers

- include_role:
        - name: myrole1, when: something | default(None) 
        - name: myrole2, when: somethingElse | default(None)
  environment:
      AWS_ACCESS_KEY_ID: "{{ assumed_role.sts_creds.access_key }}"
      AWS_SECRET_ACCESS_KEY: "{{ assumed_role.sts_creds.secret_key }}"
      AWS_SECURITY_TOKEN: "{{ assumed_role.sts_creds.session_token }}" 

[Stefan Hornburg (Racke)]

On 7/17/20 8:02 PM, Nico H wrote:
> Thanks Racke, 

> sure , so in this case */environment /*is assigned to the /*tasks */level and should therefore not fail cause it is not

> loaded before runtime. 
> In the case I have multiple roles, I have to repeat the task as many times as I need a role , lots of

> duplicate /*environment */then .
>
> Can we pass a list or role in /*import_role */or /*include_role */? I couldn't make this working with "/*loop*/"
>
> cheers
>

Hello Nico,

Loop is the wrong approach. But you can certainly set up a variable with your desired environment:

- name: My env
set_fact:
myenv:

AWS_ACCESS_KEY_ID: "{{ assumed_role.sts_creds.access_key }}"
AWS_SECRET_ACCESS_KEY: "{{ assumed_role.sts_creds.secret_key }}"
AWS_SECURITY_TOKEN: "{{ assumed_role.sts_creds.session_token }}"

And in your import_role/include_role:

- name: Import my role
import_role:
name: myrole

environment: "{{ myenv }}"

Regards
Racke

> --
> You received this message because you are subscribed to the Google Groups "Ansible Project" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to
> ansible-proje...@googlegroups.com <mailto:ansible-proje...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/ansible-project/7cf3f429-db66-4543-a5a7-8a26a0b049ado%40googlegroups.com
> <https://groups.google.com/d/msgid/ansible-project/7cf3f429-db66-4543-a5a7-8a26a0b049ado%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Nico H]

Thank you for the proposition, I ve been trying for days to put  this in every direction I could but unfortunately 'environment '
was never  passed to 'import_role' .

I am not sure how to investigate about import_role module and why environment is not supported , seems impossible to reuse sts.creds then .

Any idea is welcome 

[Dick Visser]

Your setting it in one play and then try to use it in another. That won't work.

Other question. You run this sts task for all hosts, but is that needed for all of them?

--

You received this message because you are subscribed to the Google Groups "Ansible Project" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/b52315eb-366c-48b6-b377-d2f745bb5452o%40googlegroups.com.

--

Sent from a mobile device - please excuse the brevity, spelling and punctuation.

[Nico H]

not sure if that needs new topic but I see that other AWS modules also fail to pickup variables in Environment as they should .

Example is rds_instance_facts 

again, I set up the sts credentials in environment block, and then the module doesn't use them ,why is that ? 

Saisissez le code ici...  
# playbook.yml

    pre_tasks:

    - name: Assume teardown devel stack role

      sts_assume_role:

        role_arn: "arn:aws:iam:::role/role"

        role_session_name: "session"

        region: "{{ region }}"

      register: assumed_role

      # no_log: false

    - name: import roles 

      block:

      - import_role:

          name: test2

        when: cf_region is defined

      environment:

        aws_access_key: "{{ assumed_role.sts_creds.access_key }}"

        aws_secret_key: "{{ assumed_role.sts_creds.secret_key }}"

        security_token: "{{ assumed_role.sts_creds.session_token }}

# test2.yml

- name: Get rds facts

    rds_instance_facts:

      # command: facts

      db_instance_identifier: "{{ cf_stack_name }}"

      region: "{{ cf_region }}"

    register: rds

in the exec I see the temporary keys: 

 EXEC /bin/sh -c 'aws_secret_key=cHAyHECUKdRXeMNELOADIjPKRwdYfJR/BIY8nVMZ aws_access_key=ASIA3K27TU7xxxxxxxxxxxx

security_token=FwoGZXIvYXdzEOv//////////wEaDBTRsHoNaa4W+L0bkiK+ASTwafNPh9h+q2E6IqgZAR/MjX4eDpo/Vh2zLfFX+D/XHH/B1fCcLUaGp+6AXc0H/wqJG58dt9SQP5Dym1iLq3Xzn3rVwmEc9U+0Q3PcdbczO4qwJS/A2mr1lnZ03HV+PsT0lFbActJJ65VHcNvVCqy4sSvRd+ykeBgkkiM6L35icm8eTr5RJdnTMb2tmcHogNXteaXbaag1AjABC/114coEMspTOCPjexiPtic3io67lQvIJPK9L3gqKsMMn4Ioq7js+AUyLer/WPTStS/PBgsU8WsLrlxd/iuaU4qGgquy4+6vY5H/X7vEaSMyT/8OKo6MVg== 

but the module doesn't care and bail  with :

ClientError: An error occurred (AccessDenied) when calling the DescribeDBInstances operation: User: arn:aws:iam:::user/myuser is not authorized to perform: rds:DescribeDBInstances on resource

I am losing hope in ansible :( 

What is the point of having a global environment if each and single AWS module requires a set of local credentials ? 

please help me understand the logic here, 

cheers

[Stefan Hornburg (Racke)]

On 7/24/20 7:44 PM, Nico H wrote:
> not sure if that needs new topic but I see that other AWS modules also fail to pickup variables in Environment as they
> should .
>

You are sure these environment variables are lowercase? Usually they are uppercase, like AWS_ACCESS_KEY.

Ansible just passes them along as you see in your output.

Regards
Racke

> Example is rds_instance_facts 
>
> again, I set up the sts credentials in environment block, and then the module doesn't use them ,why is that ? 
>
>
> |

> Saisissezle code ici... 

> --
> You received this message because you are subscribed to the Google Groups "Ansible Project" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to

> ansible-proje...@googlegroups.com <mailto:ansible-proje...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/ansible-project/c38b32ad-0f8a-46c3-aa06-d1e429c0f1cbo%40googlegroups.com
> <https://groups.google.com/d/msgid/ansible-project/c38b32ad-0f8a-46c3-aa06-d1e429c0f1cbo%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Brian Coca]

Another option, is to use 'default' filter:

environment:
AWS_ACCESS_KEY_ID: "{{ ars.access_key|default('') }}"
AWS_SECRET_ACCESS_KEY: "{{ ars.secret_key|default('') }}"
AWS_SECURITY_TOKEN: "{{ ars.session_token|default('') }}"
vars:
ars: '{{ assumed_role|default({}).sts_creds|default({}) }}'


--
----------
Brian Coca

[Nico H]

same problem, assumed_role is not defined yet, so ansible failed for Undefined variable, although I couldn't get the syntax you provided to work for the ars var , is that the good syntax ?

I got :

An unhandled exception occurred while templating '{{assumed_role|default({}).sts_creds|default({})}}'. Error was a <class 'ansible.errors.AnsibleError'>, original message: template error while templating string: expected token 'end of print statement', got '.'. String: {{assumed_role|default({}).sts_creds|default({})}}"

[Nico H]

Le vendredi 24 juillet 2020 12:42:34 UTC-7, Stefan Hornburg (Racke) a écrit :

On 7/24/20 7:44 PM, Nico H wrote:
> not sure if that needs new topic but I see that other AWS modules also fail to pickup variables in Environment as they
> should .
>

You are sure these environment variables are lowercase? Usually they are uppercase, like AWS_ACCESS_KEY.

Ansible just passes them along as you see in your output.

Regards
         Racke

yes you r right, in uppercase it then works better with the "block / import_role /  environment " structure .

Thank you 

 

> ansible...@googlegroups.com <mailto:ansible-project+unsub...@googlegroups.com>.