Fetching varialble data into loops iterate/repeat

524 views
Skip to first unread message

Aharonu

unread,
Mar 4, 2023, 7:28:13 AM3/4/23
to ansible...@googlegroups.com, Todd Lewis
Hi Team,

Greetings for the day!

I have a variables called 'aggregates' under vars which ontaints cluster1,cluster3,cluster2..etc and aggr1,aggr2,aggr3..etc list as mentioned below.

{
    "aggregates": [
        [
            "Cluster1",
            "aggr1"
        ],
        [
            "Cluster1",
            "aggr2"
        ],
        [
            "Cluster1",
            "aggr4"
        ],
        [
            "Cluster2",
            "aggr1"
        ],
        [
            "Cluster3",
            "aggr2"
        ]
    ]
}

********************************************************************
Based on data from 'aggregates', Below playbook has 2 items to work on:
aggregates.name: here we need to use aggr1,aggr3,aggr2...from variable 'aggregates'
hostname: here we need to use Cluster1,Cluster2,Cluster3...from variable 'aggregates'

What looking for is:
it has to go through loop iteration  until 'aggregates' list completed.   First loop iteration has to go for Cluster1, aggr1, Second  loop iteration to go for Cluster1, aggr2 based on data from 'aggregates'.

1st loop iteration:
  aggregates.name: 'aggr1'  
hostname: 'Cluster1'            
2nd loop iteration:
  aggregates.name: 'aggr2'  
hostname: "Cluster1}"            


Playbook:
---
- hosts: localhost
  gather_facts: false
  collections:
    - netapp.ontap
  vars_files:
    - < login file path >
  vars:
        aggregates: []  
    login: &login
      username: "{{ username }}"
      password: "{{ password }}"
      https: true
      validate_certs: false
      feature_flags:
        trace_apis: true
  tasks:
    - name: Get info 
      na_ontap_rest_info:
          <<: *login
          gather_subset:
            - storage/volumes
          parameters:
            aggregates.name: "{{ ?? }}"   --> here we need to use aggr1,aggr3,aggr2...from variable 'aggregates'
          hostname: "{{ ?? }}"            -->  here we need to use Cluster1,Cluster2,Cluster3...from variable 'aggregates'
          use_python_keys: true
      register: vol_data
      loop: "{{ ?? }}"
      loop_control:
        label: "{{ ?? }}"
      no_log: false
    - debug: var=vol_data
...

Rowe, Walter P. (Fed)

unread,
Mar 6, 2023, 7:55:05 AM3/6/23
to ansible...@googlegroups.com
Your 'aggregates' is a list of lists. For each aggregate list item, you have a two-element list where element [0] is cluster name and element [1] is aggregate name.

Loop through aggregates, and use item.0 for cluster and item.1 for hostname ?

    - name: Get info 
      na_ontap_rest_info:
          <<: *login
          gather_subset:
            - storage/volumes
          parameters:
            aggregates.name: "{{ item.1 }}"
          hostname: "{{ item.0 }}"
          use_python_keys: true
      register: vol_data
      loop: "{{ aggregates }}"

      loop_control:
        label: "{{ ?? }}"
      no_log: false
    - debug: var=vol_data

Walter
--
Walter Rowe, Division Chief
Infrastructure Services, OISM
Mobile: 202.355.4123

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/CANGEjuX20ndh8%2Bh_oCx8nGGBJTZ4oeEFfGQb0tWy6MXAuhTucA%40mail.gmail.com.

Aharonu

unread,
Mar 6, 2023, 8:24:42 AM3/6/23
to ansible...@googlegroups.com
Yes, I got it what I was looking for.

Thank you very much for your help Todd and Walter.



Rowe, Walter P. (Fed)

unread,
Mar 6, 2023, 9:14:08 AM3/6/23
to ansible...@googlegroups.com
If you are defining aggregates (vs getting from some other task register var), there is another way you could construct your aggregates var.

aggregates:
  - { cluster: "Cluster 1", aggregate: "aggr1" }
  - { cluster: "Cluster 1", aggregate: "aggr2" }
  - { cluster: "Cluster 1", aggregate: "aggr4" }
  - { cluster: "Cluster 2", aggregate: "aggr1" }
  - { cluster: "Cluster 3", aggregate: "aggr2" }

This syntax makes it really easy to add more items and maintain proper syntax.

Your item.0 changes to item.cluster, and your item.1 change to item.aggregate.

    - name: Get info 
      na_ontap_rest_info:
          <<: *login
          gather_subset:
            - storage/volumes
          parameters:
            aggregates.name: "{{ item.cluster }}"
          hostname: "{{ item.aggregate }}"

          use_python_keys: true
      register: vol_data
      loop: "{{ aggregates }}"
      no_log: false

    - debug: var=vol_data

This might be more description and easier for someone else to make the connection between your aggregates var and your loop references. The aggregates var is still a list. It is a list of JSON dictionary items.

---

- name: test loop with JSON dict list

  hosts: all

  become: false

  gather_facts: false

  vars:

    aggregates:

      - { cluster: "Cluster 1", aggregate: "aggr1" }

      - { cluster: "Cluster 1", aggregate: "aggr2" }

      - { cluster: "Cluster 1", aggregate: "aggr4" }

      - { cluster: "Cluster 2", aggregate: "aggr1" }

      - { cluster: "Cluster 3", aggregate: "aggr2" }


  tasks:

    - name: show reference

      debug: msg="cluster {{ item.cluster }}, hostname {{ item.aggregate }}"

      loop: "{{ aggregates }}"



% ansible-playbook -i localhost, netapp.yml 


PLAY [test loop with JSON dict list] ***********************************************************************************


TASK [show reference] **************************************************************************************************

ok: [localhost] => (item={'cluster': 'Cluster 1', 'aggregate': 'aggr1'}) => {

    "msg": "cluster Cluster 1, hostname aggr1"

}

ok: [localhost] => (item={'cluster': 'Cluster 1', 'aggregate': 'aggr2'}) => {

    "msg": "cluster Cluster 1, hostname aggr2"

}

ok: [localhost] => (item={'cluster': 'Cluster 1', 'aggregate': 'aggr4'}) => {

    "msg": "cluster Cluster 1, hostname aggr4"

}

ok: [localhost] => (item={'cluster': 'Cluster 2', 'aggregate': 'aggr1'}) => {

    "msg": "cluster Cluster 2, hostname aggr1"

}

ok: [localhost] => (item={'cluster': 'Cluster 3', 'aggregate': 'aggr2'}) => {

    "msg": "cluster Cluster 3, hostname aggr2"

}


PLAY RECAP *************************************************************************************************************

localhost                  : ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   


Just another idea ...

Walter
--
Walter Rowe, Division Chief
Infrastructure Services, OISM
Mobile: 202.355.4123

Aharonu

unread,
Mar 6, 2023, 10:37:12 AM3/6/23
to ansible...@googlegroups.com
Hi, All and Walter,

Thank you for good suggestions.

That's a really good way to deal with it but there is something I should mention here to request help.

Actually below  aggregates data coming from another set_task ( see below) which contains a list data. 

vars:
    aggregates: []

aggregates data coming below set_fact.

- set_fact:
        aggregates: "{{ aggregates|default([]) + ([item.item]| product(item | json_query(get_attrs)))| map('flatten')|list }}"
   vars:
        get_attrs: "ontap_info.storage_aggregates.records[*].[name]"
   loop: "{{ ontap.results }}"
   no_log: true

Looking for:
As above mentioned set_fact giving data into list into lists, Can we make aggregates data into dict structure as mentioned below from same set_fact ?

aggregates:
  - { cluster: "Cluster 1", aggregate: "aggr1" }
  - { cluster: "Cluster 1", aggregate: "aggr2" }
  - { cluster: "Cluster 1", aggregate: "aggr4" }
  - { cluster: "Cluster 2", aggregate: "aggr1" }
  - { cluster: "Cluster 3", aggregate: "aggr2" }

Thank you for your time and helping out.


Rowe, Walter P. (Fed)

unread,
Mar 6, 2023, 10:58:27 AM3/6/23
to ansible...@googlegroups.com
No problem.

    - name: convert list of lists to list of dicts

      set_fact:

        json_list: "{{ aggregates2 | map('zip',['cluster','aggregate']) | map('map','reverse') | map('community.general.dict') }}"


produces

    "json_list": [

        {

            "aggregate": "aggr1",

            "cluster": "Cluster 1"

        },

        {

            "aggregate": "aggr2",

            "cluster": "Cluster 1"

        },

        {

            "aggregate": "aggr4",

            "cluster": "Cluster 1"

        },

        {

            "aggregate": "aggr1",

            "cluster": "Cluster 2"

        },

        {

            "aggregate": "aggr2",

            "cluster": "Cluster 3"

        }

    ]



Walter
--
Walter Rowe, Division Chief
Infrastructure Services, OISM
Mobile: 202.355.4123

Aharonu

unread,
Mar 6, 2023, 12:11:09 PM3/6/23
to ansible...@googlegroups.com
Hi,

Is there any version dependance to use community.general.dict? my current Ansible version 2.9.13.

When i use 
 - name: convert list of lists to list of dicts
      set_fact:
        json_list: "{{ aggregates | map('zip',['cluster','aggregate']) |  map('map','reverse')  | map('community.general.dict') }}"

output:
ok: [localhost] => {
    "json_list": "<generator object sync_do_map at 0x7f47f2a99fc0>"

When i use:
 - name: convert list of lists to list of dicts
      set_fact:
        json_list: "{{ aggregates | map('zip',['cluster','aggregate']) |  map('map','reverse')  | map('community.general.dict')|list }}"
Output:
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: jinja2.exceptions.TemplateRuntimeError: No filter named 'community.general.dict'.
fatal: [localhost]: FAILED! => {
    "changed": false
}

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/BFABBF7B-29C9-4299-B0A8-3F6FCB1EE68D%40nist.gov.

Rowe, Walter P. (Fed)

unread,
Mar 6, 2023, 12:40:59 PM3/6/23
to ansible...@googlegroups.com
Absent community.general.dict you can do it in two steps:

    - name: convert list of lists to list of dicts 1

      set_fact:

        json_list: "{{ aggregates2 | map('zip',['cluster','aggregate']) | map('map','reverse') }}"

      loop: "{{ aggregates2 }}"


    - name: convert list of lists to list of dicts 2

      set_fact:

        new_list: "{{ (new_list | default([])) + [ { item.0.0: item.0.1, item.1.0: item.1.1 } ] }}"

      loop: "{{ json_list }}"


Walter
--
Walter Rowe, Division Chief
Infrastructure Services, OISM
Mobile: 202.355.4123

Aharonu

unread,
Mar 6, 2023, 1:05:07 PM3/6/23
to ansible...@googlegroups.com
Hi Walter,

Sorry for the trouble. I got still some error

TASK [convert list of lists to list of dicts] ***********************************************************************************************************************************************
ok: [localhost] => (item=['cluster1', 'aggr1'])
ok: [localhost] => (item=['cluster1', 'aggr2'])
ok: [localhost] => (item=['cluster1', 'aggr3'])
ok: [localhost] => (item=['cluster1', 'aggr2'])
ok: [localhost] => (item=['cluster2', 'aggr4'])

TASK [convert list of lists to list of dicts 2] *********************************************************************************************************************************************
fatal: [localhost]: FAILED! => {}

MSG:

Invalid data passed to 'loop', it requires a list, got this instead: <generator object sync_do_map at 0x7f086aad40f8>. Hint: If you passed a list/dict of just one element, try adding wantlist=True to your lookup invocation or use q/query instead of lookup.

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/464F213D-3BF5-401F-ADC3-E917E992DDFC%40nist.gov.

Rowe, Walter P. (Fed)

unread,
Mar 6, 2023, 1:11:01 PM3/6/23
to ansible...@googlegroups.com

---

- name: test loop with JSON dict list

  hosts: all

  become: false

  gather_facts: false

  vars:

    aggregates:

      - { cluster: "Cluster 1", aggregate: "aggr1" }

      - { cluster: "Cluster 1", aggregate: "aggr2" }

      - { cluster: "Cluster 1", aggregate: "aggr4" }

      - { cluster: "Cluster 2", aggregate: "aggr1" }

      - { cluster: "Cluster 3", aggregate: "aggr2" }

    aggregates2: [

      [ "Cluster 1", "aggr1" ],

      [ "Cluster 1", "aggr2" ],

      [ "Cluster 1", "aggr4" ],

      [ "Cluster 2", "aggr1" ],

      [ "Cluster 3", "aggr2" ]

    ]


  tasks:


    - name: convert list of lists to list of dicts 1

      set_fact:

        json_list1: "{{ aggregates2 | map('zip',['cluster','aggregate']) | map('map','reverse') | map('community.general.dict') }}"

        json_list2: "{{ aggregates2 | map('zip',['cluster','aggregate']) | map('map','reverse') }}"

      loop: "{{ aggregates2 }}"


    - name: convert list of lists to list of dicts 2

      set_fact:

        json_list3: "{{ (json_list3 | default([])) + [ { item.0.0: item.0.1, item.1.0: item.1.1 } ] }}"

      loop: "{{ json_list2 }}"


    - name: show reference

      debug:

        msg:

          - "{{ json_list1 }}"

          - "{{ json_list2 }}"

          - "{{ json_list3 }}"



Walter
--
Walter Rowe, Division Chief
Infrastructure Services, OISM
Mobile: 202.355.4123

Aharonu

unread,
Mar 6, 2023, 1:56:26 PM3/6/23
to ansible...@googlegroups.com
The same playbook also giving issue from end. May be i will look into in deep if any version dependance.

the output i got for you provided playbook:

TASK [convert list of lists to list of dicts 1] *********************************************************************************************************************************************
ok: [ciccdot1] => (item=['Cluster 1', 'aggr1'])
ok: [ciccdot1] => (item=['Cluster 1', 'aggr2'])
ok: [ciccdot1] => (item=['Cluster 1', 'aggr4'])
ok: [ciccdot1] => (item=['Cluster 2', 'aggr1'])
ok: [ciccdot1] => (item=['Cluster 3', 'aggr2'])


TASK [convert list of lists to list of dicts 2] *********************************************************************************************************************************************
fatal: [ciccdot1]: FAILED! => {}

MSG:

Invalid data passed to 'loop', it requires a list, got this instead: <generator object sync_do_map at 0x7f1b5adf7830>. Hint: If you passed a list/dict of just one element, try adding wantlist=True to your lookup invocation or use q/query instead of lookup.


To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/6051A287-FFAC-416E-B098-85A767239113%40nist.gov.

Rowe, Walter P. (Fed)

unread,
Mar 6, 2023, 2:00:55 PM3/6/23
to ansible...@googlegroups.com
You probably need to upgrade your ansible.


Walter
--
Walter Rowe, Division Chief
Infrastructure Services, OISM
Mobile: 202.355.4123

Aharonu

unread,
Mar 6, 2023, 2:07:18 PM3/6/23
to ansible...@googlegroups.com
Yes guess so, Thanking you once again.


To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/AF580880-6BE1-4F34-853E-BF41847E0AE7%40nist.gov.
Reply all
Reply to author
Forward
0 new messages