Trouble parsing JSON data returned from URI call - making use of user-data in EC2

Ben Turner

unread,

Mar 13, 2014, 8:35:16 PM3/13/14

to ansible...@googlegroups.com

I've been trying to use the user-data provided by EC2 servers to determine a servers hostname.

So currently new server come up with a hostname of "ec2-123-234-345-456" or similar, and I'd like in my ansible script to rename that server to something like "demonstration".

To do this, I've gone into my EC2 console, and added the following as user-data to my server:

{"hostname": "demonstration"}

Then, in my ansible playbook, I've tried the following:

- name: Load user data

uri: url=http://169.254.169.254/latest/user-data return_content=yes

register: user_data

- debug: var=user_data.content

- name: Set hostname from user data

command: hostname {{user_data.content.hostname}}

when: user_data is defined

This produces the output (using the -v flag):

PLAY [ec2] ********************************************************************

GATHERING FACTS ***************************************************************

ok: [ec2-54-253-114-###.ap-southeast-2.compute.amazonaws.com]

TASK: [common | Load user data] ***********************************************

ok: [ec2-54-253-114-###.ap-southeast-2.compute.amazonaws.com] => {"accept_ranges": "bytes", "changed": false, "connection": "close", "content": "{\"hostname\":\"demonstration\"}", "content_length": "28", "content_location": "http://169.254.169.254/latest/user-data", "content_type": "application/octet-stream", "date": "Fri, 14 Mar 2014 00:30:30 GMT", "etag": "\"1016948274\"", "item": "", "last_modified": "Thu, 13 Mar 2014 22:02:44 GMT", "redirected": false, "server": "EC2ws", "status": 200}

TASK: [common | debug var=user_data.content] **********************************

ok: [ec2-54-253-114-###.ap-southeast-2.compute.amazonaws.com] => {

"item": "",

"user_data.content": {

"hostname": "demonstration"

}

TASK: [common | Set hostname from user data] **********************************

fatal: [ec2-54-253-114-###.ap-southeast-2.compute.amazonaws.com] => One or more undefined variables: 'unicode object' has no attribute 'hostname'

So, whilst it seems that debugging user_data returns a JSON-like return strucutre, and drilling down to user_data.content also returns a JSON-like return structure, the actual JSON content returned by the uri task is unable to be parsed using the Jinja2 dot notation, and instead appears to be an impenetrable object, at least as far as I can determine.

Is there a way of extracting this URI return content using core ansible modules such that I can use it's JSON payload as ansible variables ?

Regards,

Ben

Romeo Theriault

unread,

Mar 13, 2014, 9:15:47 PM3/13/14

to ansible...@googlegroups.com

Hi Ben, If you want to access the JSON data it's available in the json
subkey. So instead of:

{{ user_data.content.hostname }}

I believe it would be

{{ user_data.json.hostname }}

or something similar.

Take a look at the {{ user_data.json }} or {{ user_data }} output with
the debug module to get a better idea.

Hope that helps,
Romeo

> --
> You received this message because you are subscribed to the Google Groups
> "Ansible Project" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to ansible-proje...@googlegroups.com.
> To post to this group, send email to ansible...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/ansible-project/978e1838-7964-4130-827f-2f82c0daa4f8%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

--
Romeo

Ben Turner

unread,

Mar 13, 2014, 10:27:53 PM3/13/14

to ansible...@googlegroups.com

I tried out some more debug statements; the .json suffix doesn't appear to be present ?

The user-data "json" appears to be coming back as a string...

TASK: [common | debug var=user_data] ******************************************

ok: [ec2-54-253-114-32.ap-southeast-2.compute.amazonaws.com] => {

"item": "",

"user_data": {

"accept_ranges": "bytes",

"changed": false,

"connection": "close",

"content": "{\"hostname\":\"qa\"}",

"content_length": "17",

"content_location": "http://169.254.169.254/latest/user-data",

"content_type": "application/octet-stream",

"date": "Fri, 14 Mar 2014 02:26:12 GMT",

"etag": "\"1016948274\"",

"invocation": {

"module_args": "url=http://169.254.169.254/latest/user-data return_content=yes",

"module_name": "uri"

},

"item": "",

"last_modified": "Thu, 13 Mar 2014 22:02:44 GMT",

"redirected": false,

"server": "EC2ws",

"status": 200

}

TASK: [common | debug var=user_data.json] *************************************

ok: [ec2-54-253-114-32.ap-southeast-2.compute.amazonaws.com] => {

"item": "",

"user_data.json": "{{ user_data.json }}"

}

TASK: [common | debug var=user_data.content.json] *****************************

ok: [ec2-54-253-114-32.ap-southeast-2.compute.amazonaws.com] => {

"item": "",

"user_data.content.json": "{{ user_data.content.json }}"

}

--
You received this message because you are subscribed to a topic in the Google Groups "Ansible Project" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ansible-project/FWSJulCzsQ8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ansible-proje...@googlegroups.com.

To post to this group, send email to ansible...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/CACUx58M%3D361LMPj4DFDjTrZQ0nNT57c32zLfKBm4HNupuxVtvQ%40mail.gmail.com.

Michael DeHaan

unread,

Mar 14, 2014, 10:39:09 AM3/14/14

to ansible...@googlegroups.com

In your Debug output it looks like it came out that way because that was what got stored there.

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/CAGxBzUE-B76djVXnVZrxJcE%3DXoT54zkPTiqC9fbFZnVpcUU3ig%40mail.gmail.com.

Ben Turner

unread,

Mar 17, 2014, 5:57:24 AM3/17/14

to ansible...@googlegroups.com

Looks like all "user-data" in AWS is just a "string" at the end of the day.

So there is no way for Ansible to parse that string and pull out values, without a third-party module such as https://github.com/jpmens/ansible-ec2-userdata say ?

Ben

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/CAEVJ8QMV-vmK04X5wpfkm2YWg%2BveHsSjAvZda4MvZzvgCLqRcw%40mail.gmail.com.

Matt Martz

unread,

Mar 17, 2014, 7:28:50 AM3/17/14

to ansible...@googlegroups.com

Doesn't the ec2_facts module already query user-data and parse into usable data?

I was pretty sure it did.

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/CAEVJ8QMV-vmK04X5wpfkm2YWg%2BveHsSjAvZda4MvZzvgCLqRcw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
To post to this group, send email to ansible...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/CAGxBzUGZvyNkjYTnpa5E16o90To9BqUUtX-pO9EcbHwmy0L3QQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
Matt Martz
ma...@sivel.net
http://sivel.net/

Dag Wieers

unread,

Mar 17, 2014, 7:49:56 AM3/17/14

to ansible...@googlegroups.com

On Mon, 17 Mar 2014, Ben Turner wrote:

> Looks like all "user-data" in AWS is just a "string" at the end of the day.
>
> So there is no way for Ansible to parse that string and pull out values,
> without a third-party module such as
> https://github.com/jpmens/ansible-ec2-userdata say ?

It would be a very simple improvement to set_fact to allow a string to
become a json-object, however the syntax of set_fact was made so that
every parameter is a fact name.

However a set_fact_from_json module would be super-easy to implement, so
that you can combine it with uri (or other registered variables).

Another option is to extend the uri module with an option like:
content_as_json=yes

My prefered solution is to include this in the inventory step (outside of
Ansible) so that it can be processed asynchronous (and cached) to Ansible.

--
-- dag wieers, d...@wieers.com, http://dag.wieers.com/
-- dagit linux solutions, con...@dagit.net, http://dagit.net/

[Any errors in spelling, tact or fact are transmission errors]

Matt Martz

unread,

Mar 17, 2014, 8:30:48 AM3/17/14

to Dag Wieers, ansible...@googlegroups.com

It is actually possible to use set_fact with json data already, or to use info from json data for pretty much anything. You just need to use the from_json filter.

- set_fact:

hostname: "{{ (user_data.content|from_json).hostname }}"

or in the example given:

- name: Set hostname from user data

command: "hostname {{ (user_data.content|from_json).hostname }}"

when: user_data is defined

Although 2 things to mention:

1) As mentioned, I do think that the ec2_facts module already grabs user-data <https://github.com/ansible/ansible/blob/devel/library/cloud/ec2_facts#L61>

If I am reading the code correctly, would end up looking like:

{{ hostvars[inventory_hostname]['ansible_ec2_user-data']['hostname'] }}

2) Instead of issuing a hostname change via the command module, look at the hostname module <http://docs.ansible.com/hostname_module.html>

--
Matt Martz
ma...@sivel.net

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
To post to this group, send email to ansible...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/alpine.LRH.2.02.1403171242390.6649%40pikachu.3ti.be.

Dag Wieers

unread,

Mar 17, 2014, 8:38:44 AM3/17/14

to Matt Martz, ansible...@googlegroups.com

On Mon, 17 Mar 2014, Matt Martz wrote:

> It is actually possible to use set_fact with json data already, or to use info from json data for pretty much anything. You just need to use the from_json filter.
>
> - set_fact:
> hostname: "{{ (user_data.content|from_json).hostname }}"
>
> or in the example given:
>
> - name: Set hostname from user data
> command: "hostname {{ (user_data.content|from_json).hostname }}"
> when: user_data is defined

Right, didn't think of doing it like this. Quite powerful indeed.

Thanks :)

Ben Turner

unread,

Mar 23, 2014, 10:58:53 PM3/23/14

to ansible...@googlegroups.com

Ok, so the ec2_facts module DOES pull down the EC2 information - thanks for that.

When keys are created, all hypens (-) are converted to underscores (_), so the key is ansible_ec2_user_data

The value of this key still comes down as "a string, which happens to be json", so it doesn't work to ask for {{ ansible_ec2_user_data.hostname }}

However, adding in the "from_json" filter to that does work, and got me to this final solution:

- action: ec2_facts

- hostname: name={{ (ansible_ec2_user_data|from_json).hostname }}

Thanks for all the tips - putting them together, and going through the source code, got me to this final solution !

Ben

Reply all

Reply to author

Forward