pretty printing 'complete' dictionary vars in a file after reading from a list of dicts

[ishan jain]

I have a large number of YAML files prepared manually which have a fixed structure - list of dictionaries. But within different files, sequence of these keys and their values differ. I need to manage them automatically, so i am writing a kind of converter to add marker blocks. But i am stuck at this point:

example input YAML file

#something

keyA:

  A1:

    - bla

    - bla

  A2: bla bla

  A3:

    A3.1: bla

    A3.2: bla

    

keyA:

  C1:

    - bla

  C2: bla bla

  C3:

    C3.1: bla

    

keyB:

  B1:

    - bla

    - bla

  B2: bla bla

  B3:

    B3.1: bla

    B3.2: bla

    B3.3: bla

  B4:

   - bla

   - bla

playbook to load one of the YAML, and try to write the same variables again with markers appended

    - include_vars:

          file: "path to one of the YAML file"

          name: yamlVars

    - template:

         src: updated.yaml.j2

         dest: "path to that same file"

the template

{% for key, value in yamlVars.iteritems() %}

# BEGIN BLOCK {{ key }}

{{ value | to_nice_yaml }}

# END BLOCK {{ key }}

{% endfor %}

Poblem is, always the contents of each key is printed:

(manually prepared output to hide my variables)

actual output

# BEGIN BLOCK keyA

A1:

  - bla

  - bla

A2: bla bla

A3:

  A3.1: bla

  A3.2: bla

# END BLOCK keyA

# BEGIN BLOCK keyC

C1:

  - bla

C2: bla bla

C3:

  C3.1: bla

# END BLOCK keyC

    

# BEGIN BLOCK keyB

B1:

  - bla

  - bla

B2: bla bla

B3:

  B3.1: bla

  B3.2: bla

  B3.3: bla

B4:

 - bla

 - bla

# END BLOCK keyB

expected output

# BEGIN BLOCK keyA

keyA:

  A1:

    - bla

    - bla

  A2: bla bla

  A3:

    A3.1: bla

    A3.2: bla

# END BLOCK keyA

# BEGIN BLOCK keyC

keyC:

  C1:

    - bla

  C2: bla bla

  C3:

    C3.1: bla

# END BLOCK keyC

    

# BEGIN BLOCK keyB

keyB:

  B1:

    - bla

    - bla

  B2: bla bla

  B3:

    B3.1: bla

    B3.2: bla

    B3.3: bla

  B4:

   - bla

   - bla

# END BLOCK keyB

[Matt Martz]

I *think* you want the following:

{% for key, value in yamlVars.iteritems() %}

# BEGIN BLOCK {{ key }}

{{ key }}:

    {{ value | to_nice_yaml | indent }}

# END BLOCK {{ key }}

{% endfor %}

http://jinja.pocoo.org/docs/dev/templates/#indent

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-project+unsubscribe@googlegroups.com.
To post to this group, send email to ansible-project@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/9a06be57-73b2-4459-88f4-9ffb4a8ef4c6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Matt Martz
@sivel
sivel.net

[ishan jain]

Hi Matt,

Yes, you are thinking in the right direction. Unfortunately it is still not working as intended. The indentation generated is crazy.

Basically i want idempotency by maintaining the actual indentation of the complete file always.

To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
To post to this group, send email to ansible...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/9a06be57-73b2-4459-88f4-9ffb4a8ef4c6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Matt Martz]

You are going to have to not use vars files then.  Or you will need to store the data as strings.

If you don't want formatting to change at all, you can't convert from YAML to python dict, to YAML, to jinja2+YAML.  You will need to find a way to join them without going through those steps.

To unsubscribe from this group and stop receiving emails from it, send an email to ansible-project+unsubscribe@googlegroups.com.
To post to this group, send email to ansible-project@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/235c7a6c-27cd-4e39-90c2-454d57c56db1%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[ishan jain]

Hi,

To add markers in all of them, i either have the option of lineinefile with regex or this. Regex for such a case look almost impossible. So i have to make this work.

When i print the complete  yamlVars, it works perfectly fine. Can i not append each key-value pair to a temporary dictionary and print that dictionary ?

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/235c7a6c-27cd-4e39-90c2-454d57c56db1%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Kai Stian Olstad]

On Tuesday, 27 February 2018 16.54.17 CET ishan jain wrote:
> Hi,
>
> To add markers in all of them, i either have the option of lineinefile with
> regex or this. Regex for such a case look almost impossible. So i have to
> make this work.
>

> When i print the complete *yamlVars*, it works perfectly fine. Can i not

> append each key-value pair to a temporary dictionary and print that
> dictionary ?

Have you tried this for 2 space indentation

# BEGIN BLOCK {{ key }}
{{ key }}:

{{ value | to_nice_yaml(indent=2) }}
# END BLOCK {{ key }}

for 4 space you can use this

# BEGIN BLOCK {{ key }}
{{ key }}:

{{ value | to_nice_yaml(indent=4) }}
# END BLOCK {{ key }}

You will never managed to create this with to_nice_yaml

keyA:
A1:
- bla
- bla

Since the correct 2 space indentation is

keyA:
A1:
- bla
- bla

But if you can live with that the code above should work.

Another thing, dictionary is not sorted so you can get different order of the keyX between subsequent runs.

--
Kai Stian Olstad

[ishan jain]

Hi,

More indentation doesn't work. Maybe i should think of some other approach entirely.

Thanks for your help.

BR,

Ishan

[senorsmile]

I have done similar things.  With enough "experimentation", you can get it right.  

Here's an example from a recent project: 

---
all:
  vars:
        randomkey: "1" # set this first to override
    {% for k in inventory_data.key1.key2["key3-with-dashes"] -%}
    {% set v =  inventory_data.key1.key2["key3-with-dashes"][k] %}
    {{ k | indent( width=1 )}}: {% if v is string %}{{ v | to_nice_json }}{% else %}{{ v | to_nice_json(indent=2) | indent( width=10, first=true) }}{% endif %}
    {% endfor %}

Notice three things:

- the initial spaces in front

- the to_nice_json(indent=2)

- the EXTRA indent(width=10, first=true) afterwards

My workaround here is that nested values end up being json instead of pure yaml (yaml is a superset of json). 

This doesn't produce exactly what I would do with hand crafted yaml, but it does produce valid yaml. 

NB: the inventory_data var is passed in to the template like this: 

- name: Create inventory file and populate
  template:
    src: localhost_inventory.yml.j2
    dest: "{{ ansible_dir }}/inventory/localhost.yml"
    mode: '0644'
    owner: root
  vars:
    inventory_data: "{{ lookup('file','./ansible-inventory/settings/static/group1/hosts.yml') | from_yaml }}"