json_query/jmespath filtering question

[Nick Schendel]

Trying to wrap my head about how to query json results from a URI call & could use some guidance.  I am guessing there are plenty in this group which this will be an easy question for.

I am registering the json response from a URI call to our backup system to 'snapshotlist'

Here is an example chunk of the json:

  "json": {

        "vmwareTagList": [],

        "isCorrupt": false,

        "snapshotDiskDetails": [

            {

                "virtualDiskId": "23478e90-0228-4tab-ba99-d3894f94c8c837-vm-98-2001",

                "deviceKey": 2001,

                "size": 214748364800,

                "id": "83fecf84-26db-42db-9f33-ec7e10b5f736",

                "fileName": "[DC1_LINUX_VM_STORAGE] host2restore/host2restore_1.vmdk"

            },

            {

                "virtualDiskId": "23478e90-0228-4tab-ba99-d3894f94c8c837-vm-98-2002",

                "deviceKey": 2002,

                "size": 214748364800,

                "id": "1a9c094f-f19b-4ea7-1406-ab145b0ec2f1",

                "fileName": "[DC1_LINUX_VM_STORAGE] host2restore/host2restore_2.vmdk"

            },

            {

                "virtualDiskId": "23478e90-0228-4tab-ba99-d3894f94c8c837-vm-98-2000",

                "deviceKey": 2000,

                "size": 42949672960,

                "id": "83182b0a-bc21-4ef8-8582-7ec95465380b",

                "fileName": "[DC1_LINUX_VM_STORAGE] host2restore/host2restore.vmdk"

            }

        ],

        "date": "2020-03-10T05:42:32.675Z",

        "cloudState": 0,

I am trying to get the value of "id" for the object with fileName that ends with host2retore_1.vmdk

This is what I currently have to try to pull that value out based on some example I found on a blog somewhere.  However I get errors.  Anyone able to help me figure out what I am doing wrong?

- name: set variable for disk_snap_id

    set_fact:

      disk_snap_id: "{{ json_query('snapshotdetails.json.snapshotDiskDetails[?fileName.contains(@, 'host2restore_1.vmdk')].virtualDiskId') }}"

[Dick Visser]

You need to filter a variable through json_query.

Also, I don't see any key called 'snapshotdetails'.

And that json string should be filtered through from_json, ideally.

It looks like your "json" var isn't really what you are using.

If I shuffle things around slightly, then below playbook works:

---
- hosts: localhost
  connection: local
  gather_facts: no

  vars:

    json: |
      {
        "vmwareTagList": [],
        "isCorrupt": false,
        "snapshotDiskDetails": [
          {
            "virtualDiskId": "23478e90-0228-4tab-ba99-d3894f94c8c837-vm-98-2001",
            "deviceKey": 2001,
            "size": 214748364800,
            "id": "83fecf84-26db-42db-9f33-ec7e10b5f736",
            "fileName": "[DC1_LINUX_VM_STORAGE] host2restore/host2restore_1.vmdk"
          },
          {
            "virtualDiskId": "23478e90-0228-4tab-ba99-d3894f94c8c837-vm-98-2002",
            "deviceKey": 2002,
            "size": 214748364800,
            "id": "1a9c094f-f19b-4ea7-1406-ab145b0ec2f1",
            "fileName": "[DC1_LINUX_VM_STORAGE] host2restore/host2restore_2.vmdk"
          },
          {
            "virtualDiskId": "23478e90-0228-4tab-ba99-d3894f94c8c837-vm-98-2000",
            "deviceKey": 2000,
            "size": 42949672960,
            "id": "83182b0a-bc21-4ef8-8582-7ec95465380b",
            "fileName": "[DC1_LINUX_VM_STORAGE] host2restore/host2restore.vmdk"
          }
        ],
        "date": "2020-03-10T05:42:32.675Z",
        "cloudState": 0
      }

  tasks:
    - set_fact:
        disk_snap_id: "{{ json | from_json | json_query('snapshotDiskDetails[?fileName.contains(@, `host2restore_1.vmdk`)].virtualDiskId') }}"
    - debug: var=disk_snap_id

Output:

(ansible-2.9.6) dnmvisser@NUC8i5BEK playbooks$ ansible-playbook json_vmware.yml
[WARNING]: No inventory was parsed, only implicit localhost is available

PLAY [localhost] *****************************************************************************************************************************************************************************************************************************

TASK [set_fact] ******************************************************************************************************************************************************************************************************************************
ok: [localhost]

TASK [debug] *********************************************************************************************************************************************************************************************************************************
ok: [localhost] =>
  disk_snap_id:
  - 23478e90-0228-4tab-ba99-d3894f94c8c837-vm-98-2001

PLAY RECAP ***********************************************************************************************************************************************************************************************************************************
localhost                  : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/518e3077-690f-42ea-905c-d6be12954161%40googlegroups.com.

--

Dick Visser
Trust & Identity Service Operations Manager
GÉANT

[Nick Schendel]

Oops I said the wrong register variable name in my question.  snapshotlist is the registered output of a previous task, snapshotdetails is the register of the task right before this particular task.  the chunk of json I pasted in is pulled from the previous task's output.

I was previously able to use snapshotdetails.json.snapshotDiskDetails[1].virtualDiskId to get a value in the play, it was the filtering that I have been struggling with.

Thanks for the reply I will give that a shot.

To unsubscribe from this group and stop receiving emails from it, send an email to ansible...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/518e3077-690f-42ea-905c-d6be12954161%40googlegroups.com.

[Nick Schendel]

Still struggling to get this working.  Posting my entire playbook in case it can help understand what I might be doing wrong:

- hosts: '{{inventory_hostname}}'

  gather_facts: no

  vars:

    cluser_ip: "{{ rubrik_hostname }}"

  tasks:

  - name: login to rubrik to generate token

    uri:

        url: https://{{ cluser_ip }}/api/v1/session # URL to generate an auth token to be used by subsequent calls

        force_basic_auth: yes 

        user: '{{ lookup("env", "rubrik_username") }}'

        password: '{{ lookup("env", "rubrik_password") }}'

        method: POST 

        status_code: 200

        return_content: yes

        validate_certs: yes

    register: authtoken # token is available as authtoken.json.token

    delegate_to: localhost

    become: no 

  - name: get target vm id

    uri:

        url: https://{{ cluser_ip }}/api/v1/vmware/vm?name={{ inventory_hostname_nfq }}

        headers:

          Content-Type: "application/json"

          Authorization: "Bearer {{ authtoken.json.token  }}" # pass authentication token from earlier rather than basic auth

        method: GET 

        status_code: 200

        return_content: yes 

        validate_certs: yes 

    register: vminfo

    delegate_to: localhost

    become: no 

  - name: set variable for target_vm_id

    set_fact:

      target_vm_id: "{{ vminfo.json.data[0].id }}"

  - name: get list of snapshots

    uri:

        url: https://{{ cluser_ip }}/api/v1/vmware/vm/{{ source_vm_id }}/snapshot 

        headers:

          Content-Type: "application/json"

          Authorization: "Bearer {{ authtoken.json.token  }}" # pass authentication token from earlier rather than basic auth

        method: GET 

        status_code: 200

        return_content: yes

    register: snapshotlist

    delegate_to: localhost

    become: no 

  - name: get info on the snapshot to be used

    uri:

      url: https://{{ cluser_ip }}/api/v1/vmware/vm/snapshot/{{ snapshotlist.json.data[0].id }}

      headers:

          Content-Type: "application/json"

          Authorization: "Bearer {{ authtoken.json.token  }}" # pass authentication token from earlier rather than basic auth

      method: GET 

      status_code: 200

      return_content: yes

    register: snapshotdetails

    delegate_to: localhost

    become: no 

  - name: set variable for disk_snap_id

    set_fact:

      disk_snap_id: "{{ snapshotdetails | from_json | json_query('json.snapshotDiskDetails[?fileName.contains(@, 'host2restore_1')].virtualDiskId') }}"

Error I am getting currently:

TASK [set variable for disk_snap_id] *******************************************

task path: /tmp/awx_56722_7ypjkkvu/project/oracle_refresh_fixorder.yml:67

fatal: [host2]: FAILED! => {

"msg": "template error while templating string: expected token ',', got 'host2restore_1'. String: {{ snapshotdetails | from_json | json_query('json.snapshotDiskDetails[?fileName.contains(@, 'host2restore_1')]') }}"

}

I also attempted setting the actual query string in a separate var, and passing that to the json_query as I have seen in many examples but that also did not work. not sure what I am doing wrong.  A few permutations I have tried have run successfully but then returned a blank value.  As you can see from the playbook snapshotdetails is a registered variable from the previous task.

[Dick Visser]

The query for json_query is between single quotes. But you also use
single quotes to quote 'host2restore_1' - that doesn't work.
You need to use backticks instead.
If you look closely you can see that those were in the previous
example. I agree that it's not easy to spot :)

Dick

> To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/c49e57b6-43a5-4f64-a645-a121d7733eda%40googlegroups.com.

[Nick Schendel]

Yup I absolutely did not notice that, thank you for pointing it out. now I am getting the following error:

fatal: [host2]: FAILED! => {

"msg": "Unexpected templating type error occurred on ({{ snapshotdetails | from_json | json_query('json.snapshotDiskDetails[?fileName.contains(@, `host2restore_1`)].virtualDiskId') }}): expected string or buffer"

}

[Dick Visser]

Can you put a debug task between the uri and set_fact tasks and post
what that outputs?


- debug: var=snapshotdetails


If you put this in your ansible.cfg then it's more readable:

stdout_callback = yaml
bin_ansible_callbacks = True

> To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/aba2b485-cffc-484f-9740-39352ea0b806%40googlegroups.com.

[Nick Schendel]

managed to get it working with this:

  - name: set variable for disk_snap_id

    set_fact:

      disk_snap_id: "{{ snapshotdetails | to_json | from_json | json_query('json.snapshotDiskDetails[?fileName.contains(@, `host2restore_1`)].virtualDiskId') }}"

after finding this:

https://stackoverflow.com/questions/48318708/unexpected-templating-type-error-occured-on-ec2-instances-from-json-exp