Syntax to glob-match a key in a dict

[Loren Gordon]

I'm trying to structure a map.jinja file in a manner that's relatively simple to read and maintain, but I'm having a heck of a time figuring out the syntax to glob-match a key in a dict. Here's some sample code with the initialization for what I'm doing...

{% set version_paths = {
    '11.1' : 'files_11_1',
    '11.0' : 'files_11_0',
    '10.0' : 'files_10_0',
    '9.0' : 'files_9_0',
} %}


# this value is for demonstration -- the value is retrieved dynamically in the actual code
{% set app_version = '11.0.9600.17207' %}

Now, I'm trying to match the 'app_version' against the 'version_paths' and set another variable, 'file_path', to the corresponding value but I can't for the life of me figure out the syntax. I can't use 'version_paths[app_version]' to easily retrieve the value from a dict, because the version number doesn't match the key exactly. Also, this example is rather simplified, and I can't always expect to use version numbers -- sometimes I need to use the grain 'osfullname' to get the granularity I need. So I can't just extract the first two tokens from app_version.

With the sample above, the goal would be to set file_path = 'files_11_0'.

In a somewhat-hackish attempt, I tried using a for loop to iterate over the dict, with an if statement and a .startswith() condition but I couldn't get it to work. I'm a bit new to jinja and python, so I'm not sure I'm referencing the variables properly. There is no error when processing the code below, but file_path only ever returns 'nodata':

{% set file_path = 'nodata' %}
{% for ver in version_paths %}
  {% if app_version.startswith(ver) %}
    {% set file_path = versions_paths[ver] %}
  {% endif %}
{% endfor %}

I have been able to get it to do what I want if I restructure it all in a series of if-elif blocks, embedding the information from the dict in the condition statements, but it's rather ugly, harder to read, and less maintainable, so I've been trying to figure out a syntax that would let me pull the value from the dict.

{% if app_version.startswith('11.1') %}
  {% set file_path = 'files_11_1' %}
{% elif app_version.startswith('11.0') %}
  {% set file_path = 'files_11_0' %}
{% elif app_version.startswith('10.0') %}
  {% set file_path = 'files_10_0' %}
{% elif app_version.startswith('9.0') %}
  {% set file_path = 'files_9_0' %}
{% endif %}

If anyone can help get me on the right path, I'd greatly appreciate it. I have the feeling I'm probably making it harder than it is...

Cheers,

-Loren

[Seth House]

The original use-case for using a lookup table was to get conditional
logic out of Jinja. As you pointed out, it tends to be very illegible.
It also is fairly limited in functionality (a design choice in Jinja)
and has restrictive variable scoping rules. The ``{% set %}`` in your
for-loop is defining a _new_ var that is scoped to inside the loop
which is why it's not available outside the loop. (A common hack
around this is to modify an outside dictionary/list var from inside a
loop.) Also there is no matching/globbing support in Jinja.

I would suggest creating the lookup table in the most ideal format for
the data you're trying to match, then writing a custom ``filter_by``
function in Python to filter the information down to what you need.
That keeps the Jinja clean, keeps logic out of Jinja, and lets you
customize how you match. It's easy to do:

1. mkdir -p /srv/salt/_modules
2. Edit /srv/salt/_modules/myutil.py and add a ``filter_by`` function.
3. Sync the module to all minions with ``salt '*' saltutil.sync_modules``.
4. Use it from Jinja with: ``salt['myutil.filter_by'](arg1, arg2)``.

If you want to use globbing you can use the fnmatch module in Python.
Something like:

# This is /srv/salt/_modules/mytil.py
import fnmatch

def filter_by(lookup_table, match):
[...]
for key in lookup_table.keys():
if fnmatch.fnmatch(key, match):
return lookup_table[key]

The above doesn't take ordering or duplicate matches into account. It
might be used like this:

{% set file_path = salt['myutil.filter_by'](version_paths, '11.0*') %}

There are two 'filter_by' functions that ship with Salt if you need
more inspiration:

https://github.com/saltstack/salt/blob/23fbd41/salt/modules/grains.py#L349
https://github.com/saltstack/salt/blob/23fbd41/salt/modules/match.py#L293

Good luck!

[Loren Gordon]

Thanks Seth, the scope of the variable set in the for-loop was another worry I had. I'll test the custom module. How might custom modules feature in a Formula? What I'm doing will (eventually) need to be consumable easily by others, and I was thinking of structuring it as a Formula to get there.

-Loren

[Loren Gordon]

Hrm, I'm using a masterless Windows minion to test. I placed the custom module in salt://_modules and ran ``salt-call.exe --local saltutil.sync_modules``, but the module does not appear to be available. The masterless minion config has been working fine, otherwise.

If, instead, I place the custom module directly in the salt modules directory (C:\salt\salt-2014.1.7.win-amd64\salt-2014.1.7-py2.7.egg\salt), then it is available immediately, even with calling saltutil.sync_modules.

-Loren

[Seth House]

On Mon, Aug 18, 2014 at 5:56 AM, Loren Gordon <lo...@fleet-it.com> wrote:
> How might custom modules feature in a
> Formula?

Custom modules can be included in Formulas by putting the ``_modules``
directory in the repository root (alongside the README). E.g.:

./myformula
|-- README.rst
|-- _modules/
`-- myformula/
`-- init.sls

[Seth House]

On Mon, Aug 18, 2014 at 7:08 AM, Loren Gordon <lo...@fleet-it.com> wrote:
> If, instead, I place the custom module directly in the salt modules
> directory (C:\salt\salt-2014.1.7.win-amd64\salt-2014.1.7-py2.7.egg\salt),
> then it is available immediately, even with calling saltutil.sync_modules.

That works. :)

For masterless mode you can also change the ``module_dirs`` setting in
your minion config.

[Mike Place]

Really interesting discussion.

This problem seems generic enough that it seems as though it would be helpful to have a generalized filtering module shipping with Salt. Any volunteers?

-mp

--
You received this message because you are subscribed to the Google Groups "Salt-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Loren Gordon]

Seth,

Thanks, your filter_by example worked fine, just had to swap the order of the parameters passed to the fnmatch call.  i.e. ``if fnmatch.fnmatch(match, key):``

Any idea why it might not be working when I put the custom module in the _modules directory? Is that supposed to work on a masterless minion? For greater portability, I'd rather use the built-in feature to distribute the module, rather than rely on an external process/person to place the module in the correct directory or set the ``module_dir`` setting in the minion config...

Thanks,

-Loren

[Loren Gordon]

Hi Mike, I'd certainly be interested in having such functionality ship with Salt. I wish I had the python expertise to write the module. I did review the ``match.py`` module that Seth linked earlier, and other than being specific to a minion_id for the match value, the syntax is exactly what I was looking for. Would it make sense to generalize those functions?

-Loren

[Mike Place]

Hi Loren,

I'm actually going to punt over to Seth to an answer for that. I think it would, but he has far more on-the-ground experience with state files than I do.

-mp

[Seth House]

On Mon, Aug 18, 2014 at 11:45 AM, Loren Gordon <lo...@fleet-it.com> wrote:
> Any idea why it might not be working when I put the custom module in the
> _modules directory? Is that supposed to work on a masterless minion?

It should still work with masterless mode. What directory are you
using as your ``file_roots`` directory?

It'll cache the files in the Windows equivalent of
/var/cache/salt/minion/extmods (sorry I don't have a Windows VM at the
ready -- but I can spin one up to verify if you aren't seeing that).

[Seth House]

On Mon, Aug 18, 2014 at 12:09 PM, Loren Gordon <lo...@fleet-it.com> wrote:
> Would it make sense to generalize those functions?

Yes. It would be useful having mutiple filter_by -style functions for
various kinds of lookup tables. Thumbs-up from me.

[Loren Gordon]

I'm using ``c:\salt\file_roots``. It's the same for custom grains in ``_grains``...I was testing that last week.

-Loren