Is it possible to merge dicts in a jinja map file?

[BKeep]

Hi,

I have been working on consolidating some salt formulas I found for chrony and ntp into a single time-services state/formula. I have run into a couple of general gotchas that I seem to have worked out but one thing is still troubling me. Here is my current map.jinja file.

~cat map.jinja
{##
# get the settings for the os_family grain
##}
{% set os_fam_lookup = {
  'RedHat':
  {
    '7':
    {
      'package': 'chrony',
      'service': 'chronyd',
      'config': '/etc/chrony.conf',
      'config_src': 'salt://time-services/files/chrony.conf',
      'options': 'iburst',
      'logdir': '/var/log/chrony',
      'keyfile': '/etc/chrony.keys',
      'driftfile': '/var/lib/chrony/drift',
      'otherparams': [
                      "rtcsync",
                      "makestep 10 3",
                      "stratumweight 0",
                      "bindcmdaddress 127.0.0.1",
                      "bindcmdaddress ':':1",
                      "commandkey 1",
                      "generatecommandkey",
                      "noclientlog",
                      "logchange 0.5",
                     ],
    },
    '6':
    {
      'package': 'ntp',
      'service': 'ntpd',
      'config': '/etc/ntp.conf',
      'config_src': 'salt://time-services/files/ntp.conf',
      'options': 'iburst',
      'logdir': '/var/log/ntpstats',
      'keyfile': '/etc/ntp/keys',
      'driftfile': '/var/lib/ntp/drift',
    },
  },
} %}

{% set os_fam_map = os_fam_lookup.get(grains.os_family, {}) %}
{% set config = os_fam_map.get(grains.osmajorrelease, {}) %}

{% if 'time-server' not in salt['grains.get']('roles') %}
  # Not a time server just function as a client
  {% set ntpconfig =
  {
    'ntpservers': [
                   "timeserver.domain.tld",
                  ],
  } %}
{% else %}
  # Is a local time server
  {% set ntpconfig =
  {
    'ntpservers': [
                  "0.us.pool.ntp.org",
                  "1.us.pool.ntp.org",
                  "2.us.pool.ntp.org",
                  "3.us.pool.ntp.org",
                 ],
    'allow': [
              '10.0.0.0 mask 255.0.0.0 nomodify notrap',
             ],
  } %}
{% endif %}

Using the above map file, can ntpconfig and config be merged into one dict? If so, an example or a link to a document that explains it would be of great benefit. I have searched high and low, along with trying several variations on set and do but my jinja fu is not strong enough.

Regards,
Brandon

[John Hazen]

Hi Brandon-

Jinja is a *subset* of python, but the objects that you create are python objects.  In python, the way to "merge" dicts is dict.update().  But you have to jump through a non-obvious hoop to get to it.

If you try just:

{% config.update(ntpconfig) %}

You'll get an error like "object not found", because there's no way to execute a statement that doesn't start with a jinja-imported value.

The ugly workaround is to access the 'config' object you've created in the context of a 'set' command.  Since we don't care about the return value of the update method, I usually show that by doing:

{% set _throwaway = config.update(ntpconfig) %}

Since jinja doesn't support comprehenions, I've used the same trick to dynamically build data structures:

{% set my_list = [] %}

{% for val in some_iterator %}

  {% if some_test(val) %}

    {% set _throwaway = my_list.append(val) %}

  {% endif %}

{% endfor %}

For dynamic dict-building you need *two* temp variables:

{% set my_dict = {} %}

{% for key,val in some_iterator %}

  {% if some_test(key,val) %}

    {% set _tmpdict = {key: val} %}

    {% set _throwaway = my_dict.update(_tmpdict) %}

  {% endif %}

{% endfor %}

Of course, the easier answer, since a map file is usually just setting data structures, is to just use the python renderer, but mixing that with jinja isn't supported yet in any released version (it did get merged into the develop branch, though).  See this discussion from a month ago:  https://groups.google.com/forum/#!topic/salt-users/XaLvGQ2_RYI

That's probably way more verbose than you needed.  Hope it helps.

-John

--
You received this message because you are subscribed to the Google Groups "Salt-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[BKeep]

Thanks John,

Your level of verbosity was just perfect since it actually clarified how or what jinja is, plus your examples are very helpful. It has taken some time to get ramped up on the whole terminology used within the salt ecosystem because of python, jinja, yaml etc. I have had no prior exposure to working with python/jinja/yaml and find it to be quite fun.

Thanks again for your response and examples, I greatly appreciate it.

[Loren Gordon]

Also, in Jinja, you ought to be able to use "do" to modify an existing variable. I.e.:

{% do config.update(ntpconfig) %}

-Loren

[John Hazen]

Thanks, Loren!  Of course you're right.  I'm a beginner to jinja, too, obviously.

I'm glad I called my way "the ugly workaround" instead of "the right way to do it"!  ;-)

Here's the section of the docs I found:

> The “do” aka expression-statement extension adds a simple do tag to the template 

> engine that works like a variable expression but ignores the return value.

And, I've tested it in my code.  Thanks again, that makes things simpler.

-John

[BKeep]

Hi Loren,

That works as expected too. I actually tried using do prior to posting but was seeing an error. I'm chalking it up to my exceptional typing skills and the lateness of when I was working. The do syntax seems a little cleaner too.

Thanks for your reply.