Loopless alternative?

[Dick Visser]

Hii

I have a list of dicts, where I want to inject a number of helper keys.

I can do this with set_facts in a loop. Example playbook (hopefully this displays OK):

---

- name: Add helper keys to list of dicts

  hosts: localhost

  connection: local

  gather_facts: no

  tasks:

    - set_fact:

        backup_objects: "{{ backup_objects|default([]) | union(

          [

            item | combine(

              {

                'basename': item.Key|regex_replace('^' ~ prefix ~ '\\d{10}_(.*)\\.pgdump', '\\1')

              }

            )

          ]

          )

        }}"

      loop: "{{ all_objects }}"

      loop_control:

        label: "{{ item.Key }}"

    - debug: var=backup_objects

  vars:

    prefix: backup/database/

    all_objects:

      - ETag: '"d41d8cd98f00b204e9800998ecf8427e"'

        Key: backup/database/1689953756_dev_wss_db.pgdump

        LastModified: '2023-07-21T15:36:01.000Z'

        Owner:

          ID: 2d8917bbcab5a8e0d3d7f5f39d147cd6de38e883357d7ae16323398c302fe97e

        Size: 0

        StorageClass: STANDARD

      - ETag: '"d41d8cd98f00b204e9800998ecf8427e"'

        Key: backup/database/1689953756_dev_wss_db_requests.pgdump

        LastModified: '2023-07-21T15:36:08.000Z'

        Owner:

          ID: 2d8917bbcab5a8e0d3d7f5f39d147cd6de38e883357d7ae16323398c302fe97e

        Size: 0

        StorageClass: STANDARD

      - ETag: '"d41d8cd98f00b204e9800998ecf8427e"'

        Key: backup/database/1689953756_dev_bss_service_database.pgdump

        LastModified: '2023-07-21T15:36:13.000Z'

        Owner:

          ID: 2d8917bbcab5a8e0d3d7f5f39d147cd6de38e883357d7ae16323398c302fe97e

        Size: 0

        StorageClass: STANDARD

      - ETag: '"d41d8cd98f00b204e9800998ecf8427e"'

        Key: backup/database/1689953756_dev_bss_frontend_db.pgdump

        LastModified: '2023-07-21T15:36:19.000Z'

        Owner:

          ID: 2d8917bbcab5a8e0d3d7f5f39d147cd6de38e883357d7ae16323398c302fe97e

        Size: 0

        StorageClass: STANDARD

      - ETag: '"d41d8cd98f00b204e9800998ecf8427e"'

        Key: backup/database/1689953756_dev_mss_db.pgdump

        LastModified: '2023-07-21T15:36:25.000Z'

        Owner:

          ID: 2d8917bbcab5a8e0d3d7f5f39d147cd6de38e883357d7ae16323398c302fe97e

        Size: 0

        StorageClass: STANDARD

This works, but in my real world use case this list is very long, so there will be a lot of output because of each iteration. I have already set the loop label to something less noisy.

Is there a way to add keys to a list of dicts, where those new keys are based on an operation of another key?

thx !

Dick

[Todd Lewis]

Here's the best I've got to offer. I tried pipelining filters etc but couldn't quite get over the hump. So, back to old school loops.

    - set_fact:
        backup_objects: |
          {% set result = [] %}
          {% for obj in all_objects %}
          {%   set _ = result.append({'basename': obj['Key'] | regex_replace('^' ~ prefix ~ '\\d{10}_(.*)\.pgdump', '\\1')} | combine(obj)) %}
          {% endfor %}{{ result }}

The point of the "set _" line is for the side effect of appending revised objects to the "result" list. Otherwise it's pretty straightforward.
Cheers,
--
Todd

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/CAF8BbLZDy-7ym4pcEJSBuCVkZZ2gPzinj8NFTcNnN6WedH%2BfMQ%40mail.gmail.com.

-- 
Todd

[Vladimir Botka]

Create the list of the hashes

bn_regex: '^{{ prefix }}\d{10}_(.*)\.pgdump$'
bn: "{{ all_objects|
map(attribute='Key')|
map('regex_replace', bn_regex, '\\1')|
map('community.general.dict_kv', 'basename') }}"

gives

bn:
- basename: dev_wss_db
- basename: dev_wss_db_requests
- basename: dev_bss_service_database
- basename: dev_bss_frontend_db
- basename: dev_mss_db

zip the lists and combine the items

backup_object: "{{ all_objects|zip(bn)|map('combine') }}"


--
Vladimir Botka

[Vladimir Botka]

On Thu, 17 Aug 2023 00:46:08 +0200
Vladimir Botka <vbo...@gmail.com> wrote:

> Create the list of the hashes
>
> bn_regex: '^{{ prefix }}\d{10}_(.*)\.pgdump$'
> bn: "{{ all_objects|
> map(attribute='Key')|
> map('regex_replace', bn_regex, '\\1')|
> map('community.general.dict_kv', 'basename') }}"
>
> gives
>
> bn:
> - basename: dev_wss_db
> - basename: dev_wss_db_requests
> - basename: dev_bss_service_database
> - basename: dev_bss_frontend_db
> - basename: dev_mss_db

There are options. Filter *basename* and get rid of *prefix*

bn_regex: '^\d{10}_(.*)\.pgdump$'

bn: "{{ all_objects|
map(attribute='Key')|

map('basename')|

map('regex_replace', bn_regex, '\\1')|
map('community.general.dict_kv', 'basename') }}"

Create the hash in *regex_replace* and use *from_yaml* instead
of *community.general.dict_kv*

bn: "{{ all_objects|
map(attribute='Key')|

map('basename')|
map('regex_replace', bn_regex, '{basename: \\1}')|
map('from_yaml') }}"

Instead of *regex_replace* use *splitext* and *split*

bn: "{{ all_objects|
map(attribute='Key')|

map('basename')|
map('splitext')|map('first')|
map('split', '_', 1)|map('last')|

map('community.general.dict_kv', 'basename') }}"

--
Vladimir Botka

[Todd Lewis]

Thanks, Vladimir. I had missed the point of "community.general.dict_kv". I had gotten as far as this:

    - set_fact:
        bn: |
         {{ query('ansible.builtin.nested', ['basename'], (all_objects
            | map(attribute='Key')
            | map('regex_replace', '^' ~ prefix ~ '\d{10}_(.*)\.pgdump', '\1'))) }}

which produces:

  bn:
  - - basename
    - dev_wss_db
  - - basename
    - dev_wss_db_requests
  - - basename
    - dev_bss_service_database
  - - basename
    - dev_bss_frontend_db
  - - basename
    - dev_mss_db

But I didn't find a way to map that using "community.general.dict" to create

  bn:
  - basename: dev_wss_db
  - basename: dev_wss_db_requests
  - basename: dev_bss_service_database
  - basename: dev_bss_frontend_db
  - basename: dev_mss_db

This for me is one of the more frustrating things about Jinja pipelines. I keep wishing "map" would take arbitrary expressions rather than the limited set it's stuck with. So you end up with a fleet of one-off filters like "community.general.dict_kv" which does what "community.general.dict" would do if there were an obvious way to turn this:

  - - basename
    - dev_wss_db
  - - basename
    - dev_wss_db_requests
  - - basename
    - dev_bss_service_database
  - - basename
    - dev_bss_frontend_db
  - - basename
    - dev_mss_db

into this:

  - - - basename
      - dev_wss_db
  - - - basename
      - dev_wss_db_requests
  - - - basename
      - dev_bss_service_database
  - - - basename
      - dev_bss_frontend_db
  - - - basename
      - dev_mss_db

i.e. a "deepen" counterpart to "flatten". But that magical incantation has so far eluded me.
--
Todd

[Vladimir Botka]

On Thu, 17 Aug 2023 07:56:58 -0400
Todd Lewis <uto...@gmail.com> wrote:

> bn:
> - - basename
> - dev_wss_db
> - - basename
> - dev_wss_db_requests
> - - basename
> - dev_bss_service_database
> - - basename
> - dev_bss_frontend_db
> - - basename
> - dev_mss_db
>
> But I didn't find a way to map that using "community.general.dict" to create
>
> bn:
> - basename: dev_wss_db
> - basename: dev_wss_db_requests
> - basename: dev_bss_service_database
> - basename: dev_bss_frontend_db
> - basename: dev_mss_db

You can always use brute-force Jinja as the last resort. For example,
given the list

bn_list:
- dev_wss_db
- dev_wss_db_requests
- dev_bss_service_database
- dev_bss_frontend_db
- dev_mss_db

the below Jinja creates the list of the hashes

bn: |
{% filter from_yaml %}
{% for basename in bn_list %}
- basename: {{ basename }}
{% endfor %}
{% endfilter %}

As a side-note, this is equivalent to

bn: "{{ all_objects|
map(attribute='Key')|

map('regex_replace', bn_regex, '{basename: \\1}')|
map('from_yaml') }}"

> This for me is one of the more frustrating things about Jinja
> pipelines. I keep wishing "map" would take arbitrary
> expressions rather than the limited set it's stuck with.

This is very good point. It would be possible to write such a filter.
However, I'm not sure about the security implications.

--
Vladimir Botka

[Brian Coca]

see map/select/reject filters .. they are actually loops and normally
much simpler than using jinja command syntax ( {% %} ).

--
----------
Brian Coca

[Vladimir Botka]

On Thu, 17 Aug 2023 13:11:25 -0400
Brian Coca <bc...@redhat.com> wrote:

> see map/select/reject filters .. they are actually loops and normally
> much simpler than using jinja command syntax ( {% %} ).

Unfortunately, some filters are not *map* friendly. For example, the
filter *product*

list1|product(list2) .............. works fine
list1|zip(list2)|map('product') ... does not work


Details: Given the list

l1:
- dir: /tmp/test/d1
sub_dir: [a, b]
- dir: /tmp/test/d2
sub_dir: [a, b, c]

the goal is to create the list of products

l2:
- /tmp/test/d1/a
- /tmp/test/d1/b
- /tmp/test/d2/a
- /tmp/test/d2/b
- /tmp/test/d2/c

The iteration (the filter *subelements* not used
to demonstrate the functionality of *product*)

- debug:
msg: "{{ [item.0]|product(item.1) }}"
loop: "{{ dirs|zip(sdirs) }}"
vars:
dirs: "{{ l1|map(attribute='dir') }}"
sdirs: "{{ l1|map(attribute='sub_dir') }}"

works as expected. Gives (abridged)

msg:
- - /tmp/test/d1
- a
- - /tmp/test/d1
- b

msg:
- - /tmp/test/d2
- a
- - /tmp/test/d2
- b
- - /tmp/test/d2
- c

But, the filter *product* doesn't work with *map*

dirs: "{{ l1|map(attribute='dir') }}"
sdirs: "{{ l1|map(attribute='sub_dir') }}"
l3: "{{ dirs|zip(sdirs)|map('product') }}"

gives

l3:
- - - /tmp/test/d1
- - - a
- b
- - - /tmp/test/d2
- - - a
- b
- c

This leaves you with Jinja if you want to avoid the loops in tasks

l3: |
{% filter from_yaml %}
{% for i in l1 %}
{% for s in i.sub_dir %}
- {{ i.dir }}/{{ s }}
{% endfor %}

{% endfor %}
{% endfilter %}

--
Vladimir Botka

[Dick Visser]

I ended up with the inline jinja for loop.

Also I used a pipe lookup instead of an s3 list task prior to this one.

So now everything is done in a one single task, without noise, and with the same results.

Thanks everyone, it was really useful and I will keep it in mind for future reference.

Dick

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/20230817202944.3f3512c8%40gmail.com.