How to use the JSON part of an HTTP response in the subsequent tasks?

[Roman Revyakin]

Hi guys,

I am facing the problem of not being able to use the JSON part of the HTTP response in my subsequent tasks in an Ansible playbook. Here are the relevant tasks from the playbook:

  - name: Login to account as the specified user and obtain its id

    local_action: >

      uri url='http://{{ account_app }}.{{ target_domain}}/auth/user'

      body='{ "email": "{{ userId }}", "password": "{{ sa_password }}" }'

      method=POST

      return_content=yes status_code=200

      HEADER_Content-Type="application/json"

    register: usp_user

    tags:

      - config-manager

      - bootstrap

      - api

  - name: Archive previous top-level settings config

    local_action: >

       command

       curl -i -f -H "content-type: application/json"

        -H "{{ lookup('template', './user_header.json.j2') }}"

        -XPUT http://{{ config_manager }}.{{ target_domain }}/archive/settings

#      uri url='http://{{ config_manager }}.{{ target_domain }}/archive/settings'

#      method=PUT

#      return_content=yes status_code=200

#      HEADER_Content-Type="application/json"

#      HEADER_user="{{ usp_user.json }}"

    tags:

      - config-manager

      - bootstrap

      - api

  - name: Create new top-level settings config

    local_action: >

       command

       curl -i -f -H "content-type: application/json"

        -H "{{ lookup('template', './user_header.json.j2') }}"

        -XPOST http://{{ config_manager }}.{{ target_domain }}/config/settings

        -d '{{ lookup('template', './stages_request.json.j2') }}'

#      uri url='http://{{ config_manager }}.{{ target_domain }}/config/settings'

#      body='{{ lookup('template', './stages_request.json.j2') }}'

#      method=POST

#      return_content=yes status_code=200

#      HEADER_Content-Type="application/json"

#      HEADER_user="{{ usp_user.json }}"

    tags:

      - config-manager

      - bootstrap

      - api

The `user_header.json.j2` is a simple header template to overcome a YAML parsing problem of having a ':' followed by the curly braces

user: {{ usp_user.json }}

Basically what I am trying to do it to log in as a particular user to an account-managing app, and then use the JSON part of the response that contains required credentials (like the user id and the token) to do some requests to another app. It seems to work if I form the JSON header manually using the individual values from the JSON response, but not when I try to use the JSON as it is by referring to `variable.json`.

In the commented out code I tried to use the `uri` module which I've become a fan of lately and used `curl` command to test whether the problem is in the way `uri` supplies header in the request. Both `curl` and `uri` versions fail with the 400 on the "Archive previous top-level settings config" task (where no body is used in the request) and in the verbose output I see that the `usp_user.json` has 'u' prefixes before any of the key/value pairs, like follows:

"user: {u'customer': u'52afd279fa33dd1f00000004',...

I suspect this is at least part of the problem if not the problem on itself - please correct me if I am wrong and this is just the weird formatting that is only visible in the verbose output. I tried inserting the task `debug var=usp_user.json` and it seems to output well-formatted JSON though, without any of the 'u' attached.

I would appreciate a guidance on how to use the json part of the response in forming headers for succeeding requests.

With kind regards,

Roman

[Roman Revyakin]

Just one more thing that I forgot to mention: I am able to use the JSON response as it is in subsequent requests using a bash script which is currently in place until this issue with the Ansible playbook is fixed:

# Getting the user credentials in JSON format

user_info="$(curl -f -v -XPOST $account.$target_domain/auth/user -d "$user_login" -H "$content_type_header" | awk '$1=$1' RS= OFS=" " )"

# Using them in the header for subsequent requests

curl -i -H "$user_header" -XPUT $config_manager.$target_domain/archive/settings -H "$content_type_header"

...

[Michael DeHaan]

The easiest way to check the response code of a module is 

- debug: var=registered_variable_name

Start there.

The fact that you're getting back unicode shouldn't be a problem.   

Possible point of confusion -- What you print from "debug: var" is actually a Python datastructure, not JSON per se.   The 'json' structure you get back is the datastructure itself that came from the JSON.

i.e.    registered_variable_name.json.some_value_from_your_web_service should be directly accessible as a variable.

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
To post to this group, send email to ansible...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--

Michael DeHaan <mic...@ansibleworks.com>
CTO, AnsibleWorks, Inc.
http://www.ansibleworks.com/

[Roman Revyakin]

Hi Michael,

Thanks for your reply. 

I think I have already mentioned in my post that I used the debug statements and the structures returned seemed ok to me.

I believe that there is a problem with the way `uri` module handles the HTTP headers. When I fall back to using curl in my playbook everything works correctly:

  - name: Login to account-ng as the specified user and obtain its id

    local_action: >

      shell curl -v -f -H "content-type: application/json" -d '{ "email": "{{ userId }}", "password": "{{ sa_password }}" }' -XPOST 'http://{{ account_app }}.{{ target_domain}}/auth/user' | awk '$1=$1' RS= OFS=" "

    register: usp_user

  - name: Archive previous top-level settings config

    local_action: >

       command curl -i -f -H "content-type: application/json" -H 'user: {{ usp_user.stdout }}' -XPUT http://{{ config_manager }}.{{ target_domain }}/archive/settings

However if I try to replace `curl` in the second task with `uri` module I get 400 status. Here's the second task with `uri` used in place of `curl`:

  - name: Archive previous top-level settings config

    local_action: >

       uri url='http://{{ config_manager }}.{{ target_domain }}/archive/settings'

       method=PUT

       return_content=yes status_code=200

       HEADER_Content-Type="application/json"

       HEADER_user="{{ usp_user.stdout }}"

Best regards, 

Roman 

[Roman Revyakin]

And looks like JSON that is returned from `uri` module is also bad in some way. Here's a playbook where I replaced the first call to `curl` to obtain user credentials with `uri` and then using `usp_user.content` in the header of my subsequent curl POST requests, and that works. If I use `usp_user.json` however, the second request fails with 400 again:

  - name: Login to account-ng as the specified user and obtain its id

    local_action: >

      uri url='http://{{ account_app }}.{{ target_domain}}/auth/user'

      body='{ "email": "{{ userId }}", "password": "{{ sa_password }}" }'

      method=POST

      return_content=yes status_code=200

      HEADER_Content-Type="application/json"

    register: usp_user

  - name: Archive previous top-level settings config

    local_action: >

       command curl -i -f -H "content-type: application/json" -H 'user: {{ usp_user.content | replace("\n", "") }}' -XPUT http://{{ config_manager }}.{{ target_domain }}/archive/settings

Regards,

Roman

[Roman Revyakin]

Just to provide a summary of my findings, in a nutshell: 

Ansible module `uri` that I started to admire previously seems to be only usable in the cases where one:

1. does not need to supply custom headers in requests

2. does not use JSON part of response in its entirety in the subsequent requests (one can use individual key/values to form another JSON structure though)

which renders it to be of very limited use unfortunately. 

I would be glad if someone proves that the above statements are wrong

With kind regards,

Roman

[Roman Revyakin]

Here's a bit late update on how I managed to resolve the issues with the `uri` module, hopefully that would be useful to someone else:

First of all, I was wrong about the custom headers supplied in the `uri` module - they do work, as I was able to confirm it both from debugging and looking at the Python code.

What does not work though is using a JSON structure in its entirety when trying to supply it as part of a custom header to `uri`. The problem is, as I had assumed before and was able to confirm by going into a deep-debugging mode of the `uri` module (I had to modify the sources of the `httplib2` to write out headers of all requests to a file) is the fact that the JSON structure in the response from `uri` is in the form of the unicode string and in case it is used in the header of a subsequent request to a `uri` module, it is passed in the unicode form and triggers the 400 errors.

So taking slightly modified above examples, this will not work (400 error in the "Archive ... " task):

 - name: Login to account as the specified user and obtain its id

    local_action: >

      uri url='http://{{ account_app }}.{{ target_domain}}/auth/user'

      body='{ "email": "{{ userId }}", "password": "{{ sa_password }}" }'

      method=POST

      return_content=yes status_code=200

      HEADER_Content-Type="application/json"

    register: usp_user

 

- name: Archive previous top-level settings config

    local_action: >

       command

       uri url='http://{{ config_manager }}.{{ target_domain }}/archive/settings'

       method=PUT

       return_content=yes status_code=200

       HEADER_Content-Type="application/json"

       HEADER_user="{{ usp_user.json }}"

But this one will:

  - name: Archive previous top-level settings config

    local_action: >

      uri url='http://{{ config_manager }}.{{ target_domain }}/archive/settings'

      method=PUT

      return_content=yes status_code=200

      HEADER_Content-Type="application/json"

      HEADER_user='{{ usp_user.content | replace("\n", "") }}'

`usp_user.content` contains the same JSON structure as `usp_user.json`, only it has got the newlines that I need to remove using the `replace` Jinja2 filter.

Regards,

Roman