Re: [ansible-project] How to use ec2 modules user_data field?

[Michael DeHaan]

(a)  can you please show the line from your playbook where you are using the user data variable?

(b)  with the above, how are you determining the way it fails?  I.e. what does failure look like?

On Fri, Aug 15, 2014 at 5:36 PM, Jonathan Nakatsui <jnak...@gmail.com> wrote:

From the ansible docs the ec2 module (http://docs.ansible.com/ec2_module.html) mentions a user_data field. However, there is no example showing the use of user_data.

I've tried a variety of things to pass through to the user field to no avail including

user_data: file.txt

user_data: "{{ lookup('file', file.txt') }}"

user_data: """#!/bin/bash
                      apt-get update
                      apt-get install htop"""

And variations thereof. My googlefu has failed me so far to find any results for how to solve this issue.

Thanks if you have any input.

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
To post to this group, send email to ansible...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/64b86c47-036a-471f-8a64-9cbd38c04f5c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Erick Barros]

I got the following error:

ERROR: Syntax Error while loading YAML script, /etc/ansible/playbooks/search/roles/aws/tasks/main.yml
Note: The error may actually appear before this position: line 13, column 17


   state: running
   user_data: """#!/bin/bash
                ^
This one looks easy to fix.  It seems that there is a value started
with a quote, and the YAML parser is expecting to see the line ended
with the same kind of quote.  For instance:


    when: "ok" in result.stdout


Could be written as:


   when: '"ok" in result.stdout'


or equivalently:


   when: "'ok' in result.stdout"

[Michael DeHaan]

I don't believe YAML takes Python-like docstrings.

So you would need to start with a single or double quote and escape any internal quotes as need be.

The "file" lookup plugin may also be helpful. 

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/a1c6540e-6ece-49b8-84f8-83802cb97ebb%40googlegroups.com.

[Chris Church]

The file lookup plugin is your friend.

I'm using the following task for launching Windows instances and configuring PowerShell remoting:

- ec2:

    aws_access_key: '{{ aws_access_key }}'

    aws_secret_key: '{{ aws_secret_key }}'

    region: '{{ win_ec2_region }}'

    instance_type: '{{ win_ec2_instance_type }}'

    instance_tags:

        Name: '{{win_ec2_name_prefix}}-{{item.name}}'

    group: '{{ win_ec2_security_group }}'

    key_name: '{{ win_ec2_key_name }}'

    image: '{{ item.id }}'

    user_data: "{{ lookup('file', 'win_ec2_user_data') }}"

    exact_count: 1

    count_tag:

        Name: '{{win_ec2_name_prefix}}-{{item.name}}'

    wait: yes

  with_items: win_ec2_images

  register: win_ec2

My win_ec2_user_data file contains:

<powershell>

iex ((new-object net.webclient).DownloadString('https://raw.githubusercontent.com/ansible/ansible/devel/examples/scripts/ConfigureRemotingForAnsible.ps1'))

</powershell>

 

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/CA%2BnsWgyG79kBFTw3O_i62_FTd7MeBtuvLWUNnEauFLfNkQw66w%40mail.gmail.com.

[Rob White]

Hi Chris,

What's your approach for logging in to your Win boxes using WinRM after this step?

Ansible docs direct you to configure a vars file with ansible_ssh_pass set however every ec2 instance is going to have a different password.  I've created a module for using boto to get the ec2 password but i dont know how to chain this through to then use it for a following play which would configure my windows instances.

[Chris Church]

I'm planning to get some of these roles onto Galaxy, but until then...

The next tasks in that particular play are:

- name: wait for instances to listen on port 5986 (winrm https)

  wait_for:

    state=started

    host={{ item.tagged_instances[0].public_ip }}

    port=5986

  with_items: win_ec2.results

  when: win_ec2_images and win_ec2 is defined

- name: obtain windows passwords for instances

  ec2_win_pass:

    aws_access_key: '{{ aws_access_key }}'

    aws_secret_key: '{{ aws_secret_key }}'

    region: '{{ win_ec2_region }}'

    instance_id: "{{ item.tagged_instances[0].id }}"

    private_key: "{{ lookup('file', win_ec2_private_key) }}"

  with_items: win_ec2.results

  register: win_ec2_passwords

  when: win_ec2_images and win_ec2 is defined

- name: add_host

  add_host:

    name: '{{ item.item.tagged_instances[0].public_dns_name }}'

    groups: 'cloud,ec2,windows,{{win_ec2_name_prefix}}-{{ item.item.item.name }}'

    ansible_ssh_host: '{{ item.item.tagged_instances[0].public_ip }}'

    ansible_ssh_port: 5986

    ansible_ssh_user: '{{ item.item.item.user }}'

    ansible_ssh_pass: '{{ item.password }}'

    ansible_connection: 'winrm'

  with_items: win_ec2_passwords.results

  when: win_ec2_images and win_ec2_passwords is defined

The second task is using my module from https://github.com/ansible/ansible-modules-core/pull/378

You can also save these new hosts to an inventory file (using template module).

After the play above runs, I can then run another play to ping the newly-launched Windows hosts:

- name: ping new windows instances

  hosts: windows

  gather_facts: false

  tasks:

    - action: win_ping

Hope that helps!

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/3960ea12-56fa-4ee1-a902-6c090ea72ae0%40googlegroups.com.

[Daniel Neades]

On Saturday, 25 October 2014 20:33:32 UTC+1, Chris Church wrote:

The file lookup plugin is your friend.

It is – if one wishes to supply text-based user data. Unfortunately, I have a requirement to provide a binary blob of user data, and the file lookup plugin does not seem to handle that – looking at the source code, it expects the file to be a UTF8-encoded text file.

What I’d like to do is make use of FreeBSD’s configinit capability, which can treat the user data as a tar file, extract it, and then process each of the contained files. This is documented here:

http://www.daemonology.net/blog/2013-12-09-FreeBSD-EC2-configinit.html

As I mention in the comments on that page, I did try passing the contents of a base64-encoded tar file (via the file lookup plugin) in as the user_data parameter, but (unsurprisingly), that gets passed-through as-is (i.e. still base64-encoded) to the ec2 instance user data, whereas I need the user data to be set to the raw binary data of the unencoded tar file.

Now, the Ansible ec2 module documentation specifically says that the user_data parameter is an ‘opaque blob of data which is made available to the ec2 instance’, so it is obviously anticipating that directly passing a binary blob should be possible. The question, though, is how?

Any help would be greatly appreciated. I am new to both Ansible and EC2, so it is entirely possible that I am missing something blindingly obvious.

Thanks in advance!

[Daniel Neades]

Incidentally, I also tried using the file lookup plugin on a base64-encoded tar file, and then filtering that with Jinja2’s b64decode filter. That also fails, though I think it does so at a later stage than trying to use the file lookup plugin on a binary file. Using the b64decode filter on a looked-up base64-encoded file, one gets an error whose stack trace ends something like this:

  File "/usr/local/Cellar/ansible/1.8.2/libexec/lib/python2.7/site-packages/ansible/utils/template.py", line 362, in template_from_string
    res = jinja2.utils.concat(rf)
  File "<template>", line 7, in root

UnicodeDecodeError: 'ascii' codec can't decode byte 0xfd in position 0: ordinal not in range(128)

I then thought I’d do a little experiment, and provide the user data as a literal (copy-and-pasted) string of base64-encoded data, again filtered with the Jinga2 b64decode filter, like this:

user_data: "{{ '/Td6WFoAAATm1rRGA … VZiascRn+wIAAAAABFla' | b64decode }}"

That took the file lookup plugin out of the equation, but resulted in the same error.

I next tried the pipe lookup plugin (with ‘cat my-binary-file’), but that also fails in the same way as the file lookup plugin. Looking at the source code, the pipe lookup plugin follows the file lookup one and unconditionally tries to decode the output from the command as a UTF-8 encoded string.

I provisionally conclude from all this that Ansible generally doesn’t want to handle binary parameter values. That seems limiting. I suppose one solution to my specific problem (without fixing the wider issue) would be for the ec2 module to accept something like a user_data_file parameter (as an alternative to user_data), which would specify a local file whose content was to be used without any processing at all as an opaque user data binary object. In fact, that’s pretty much exactly what I want, as I could give it the FreeBSD configinit .tar file directly.

I am still hoping that someone has a solution that will enable binary user data to be provided to an instance.

Again, my thanks in advance for any assistance.

-- 

Daniel

On Tuesday, 23 December 2014 23:06:38 UTC, Daniel Neades wrote:

On Saturday, 25 October 2014 20:33:32 UTC+1, Chris Church wrote:

The file lookup plugin is your friend.

It is – if one wishes to supply text-based user data. Unfortunately, I have a requirement to provide a binary blob of user data, and the file lookup plugin does not seem to handle that – looking at the source code, it expects the file to be a UTF8-encoded text file.

<snip>

[Chris Church]

One issue may be that boto is encoding unicode user data into utf-8, then base64-encoding it:

https://github.com/boto/boto/blob/develop/boto/ec2/connection.py#L927

So, if we can somehow pass a str() instance to boto instead of a unicode() instance, it'll skip the utf-8 encoding step.

I've implemented a version of the ec2 module that accepts a user_data_b64 parameter as a base64-encoded string of the user data, and decodes it into a str() instance before passing to boto:

https://github.com/cchurch/ansible-modules-core/blob/f6522309ffaedace2305fd67a5a5721213f59199/cloud/amazon/ec2.py

You can read the local binary file into a base64-encoded string using the slurp module, then pass that blob to the ec2 module.  The slurp module needs an absolute path, so you can use role_path to specify a path relative to a role, or playbook_dir relative to your top-level playbook.

- local_action: slurp src="{{role_path}}/files/ec2_user_data"

  register: ec2_user_data_b64

- ec2:

    user_data_b64: '{{ ec2_user_data_b64.content }}'

    ...

--

You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
To post to this group, send email to ansible...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/9708972a-8a15-4819-9837-4a172b9cf608%40googlegroups.com.

[Catalin Costache]

Yaml has a notation for multiline string. I set user data like so:

  user_data: |
      #!/bin/bash
      echo "Defaults:{{admin_user}} !requiretty" > /etc/sudoers.d/disable_requiretty

Note the pipeline character.

# Join multiple lines without new line
value: >
        part 1
        part 2
        
# Join with newline
value2: |
        line 1
        line 2

[Daniel Neades]

Chris, thank you *very* much for looking at that. I have tried your modified ec2.py but, unfortunately, even using it exactly as you describe, the user data is still being stored on AWS in its base64 form. From within the instance:

# fetch http://169.254.169.254/latest/user-data

# cat user-data

/Td6WFoAAATm1rRGAgAhARYAAAB … /sCAAAAAARZWg==

# base64 -d user-data test.tar.xz

# tar tvf test.tar.xz 

-rw-r--r--  0 djn    staff       2 Dec 26 10:40 1.txt

-rw-r--r--  0 djn    staff       2 Dec 26 10:40 2.txt

-rw-r--r--  0 djn    staff     322 Dec 24 12:03 configinit

By the way, looking at line 930 of the boto connection.py source file you referenced, I don’t understand why boto is base64-encoding a parameter that its own documentation (on lines 774-776) indicates is apparently already to be base64-encoded. (Note, however, that I am not familiar with the boto source.) Still, since you are passing in a decoded version of the Ansible user_data_b64 parameter, I’d have thought what you’ve done ought to work. Empirically, though, it doesn’t seem to.

Finally, perhaps I should note that to test this, I dropped your ec2.py over the top of the existing ec2.py in a stock Ansible 1.8.2 installation that was installed by Homebrew on my OS X machine. This installation of Ansible is using boto 2.34.0, I believe.

Thank you again for your help!

[Daniel Neades]

Chris, please ignore my previous response, which was premature. I had misinterpreted the (perhaps slightly ambiguous) documentation for the slurp module, and thought that it needed to be given a base64-encoded file. Apparently, it does not. If I give it a raw .tar.xz, your ec2.py revision seems to work perfectly, thank you!

Perhaps an example in the documentation for the ec2 module using slurp and user_data would be in order, however, before you submit a pull request for your modifications to be incorporated into Ansible proper?

Many thanks again for your solution, let’s hope it can be incorporated into the next Ansible release.

[Bryan Larsen]

On Friday, 26 December 2014 05:05:33 UTC-5, Catalin Costache wrote:

Yaml has a notation for multiline string. I set user data like so:

  user_data: |
      #!/bin/bash
      echo "Defaults:{{admin_user}} !requiretty" > /etc/sudoers.d/disable_requiretty

I'm having trouble using this notation.

     - ec2_lc:

        name: "{{env}}-render-premium-{{gitsha}}"

        ...

        user_data: |

          {

            "services": ["worker-render-premium@4100"],

            "autoScalingGroupName": "{{env}}-render-premium"

          }

but when I do a `curl http://169.254.169.254/latest/user-data` to read it back I get:

    {'services': ['worker-render-premium@4100'], 'autoScalingGroupName': 'staging-render-premium'}

The loss of whitespace I can deal with, but translating double quotes into single quotes makes this invalid JSON and breaks things.

I'm using ansible 1.9.4.   Even with -vvvv user_data isn't printed.

[einarc]

Hey,

You want to put your data in a variable, let's say in /defaults/main.yml and then invoke that from the ec2_lc module:

 - ec2_lc:

        name: "{{env}}-render-premium-{{gitsha}}"

        ...

        user_data: "{ {my_user_data }}"

I did that exact thing for Launch Configs and it works prefectly.

Other than that you may try to rewrite your do to include the pipe WITHIN the brackets. But I wouldn't recommend that as a best practice.

     - ec2_lc:

        name: "{{env}}-render-premium-{{gitsha}}"

        ...

        user_data:

          { |

            "services": ["worker-render-premium@4100"],

            "autoScalingGroupName": "{{env}}-render-premium"

          }

my_user_data: |

                        #!/bin/bash 
           echo "Defaults:{{admin_user}} !requiretty" > /etc/sudoers.d/disable_requiretty

And then use that variable from within the module:

[Cesar Ramirez]

Hello Guys, new tot this Group, trying to provision an EC2 Windows  using Ansible with user_data.

[Dick Visser]

And your question is....?

--

You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/a876f870-7867-4bc7-b20e-10175ee9659en%40googlegroups.com.

--

Sent from Gmail Mobile