map.jinja woes

[fREW Schmidt]

If I understand correctly, the map.jinja pattern exists to allow pillardata to be shared among pillars.  You could consider it to be sensible OS level defaults too, but as far as I can see it's so that I can have pure data that various pillars read.  Ok.

Now lets say I'm writing a complex pillar and writing it in Jinja is too much work.  Great, I have all these other renderers, I'll use the `#!py` one.  So I write my pillar, but now I can't import from a map.jinja, right?  If there is a way to import from the map.jinja, let me know, because that would be the best end to this issue.

So, in an effort to get around this we've invented a new idiom, which is basically the exact same thing but python:

#!py

import logging
import sys
import os

def run():

    # gross, hardcoded path

    new_path = os.path.join('/vol/saltstack/pillar', os.path.dirname(__sls__))
    if new_path not in sys.path:
        sys.path.append(new_path)

    from map import run as map_run

    return {
        "nested": map_run()
    }

So that pulls data from the map.py in the same directory as the pillar.  It feels like a lot of boilerplate, and maybe it could be reduced I tiny bit, but the worst thing is, now, jinja files can't read the shared data!

Is there a way to do the map.jinja pattern for more than just the jinja renderer?

--

fREW Schmidt
http://blog.afoolishmanifesto.com

[Seth House]

One option is to maintain a map file that is only data, no logic, as
perhaps a straight YAML file. You have to repeat the filter/merge
logic in each but that's typically just ~two lines. E.g.,

# map.yaml
Debian:
pkg: apache2
RedHat:
pkg: httpd

# some.sls
#!py
import yaml
def run():
with open('map.yaml', 'rb') as f:
map = yaml.safe_load(f.read())
apache = __salt__.grains.filter_by(map,
merge=__salt__.pillar.get('apache:lookup'))
return {...}

# someother.sls
{% import_yaml 'map.yaml' as map %}
{% set apache = salt.grains.filter_by(map,
merge=salt.pillar.get('apache:lookup')) %}

The best permanent solution, IMO, is to expose Salt's renderer modules
directly via an execution module wrapper. It wouldn't be much code to
implement. There's a new module named 'slsutil' [1] that would be a
good home for such an addition. I expect the interface would be as
simple as the following.

{% set apache = salt.slsutil.renderer('map.sls') %}

[1] https://github.com/saltstack/salt/blob/develop/salt/modules/slsutil.py

Where the map file could be written using any combination of Salt's
renderer modules in the normal way. The top she-bang respected as
normal. The only contract for use as a map file would be the final
result would need to produce a data structure.

# map.sls
#!jinja|yaml
{% set apache = salt['grains.filter_by']({
...normal jinja map file here...
}, merge=salt.pillar.get('apache:lookup')) %}
{# Output the resulting Jinja data structure in YAML format, per the
she-bang, to make it universally available. #}
{{ apache | yaml() }}

The following would be functionally equivalent to the above, but using
a different render pipe.

# map.sls
#!py
def run():
apache = __salt__.grains.filter_by({
...normal map here but as a python dict...
}, merge=__salt__.pillar.get('apache:lookup'))
return apache

And regardless of the contents of the map file, it would be available
by simply calling the proposed `slsutil.renderer()` function.

If you're interesting in taking a stab at it, I'd love to see it in
core. The following is a working POC.

import salt.template
import salt.loader

def renderer(path_or_string, default_renderer='jinja|yaml'):
renderers = salt.loader.render(__opts__, __salt__)
return salt.template.compile_template(
path_or_string,
renderers,
default_renderer)

See also:

https://github.com/saltstack/salt/blob/develop/salt/template.py
https://github.com/saltstack/salt/blob/develop/salt/utils/templates.py

[Seth House]

I added docs and submitted a pull request here:

https://github.com/saltstack/salt/pull/31812

[Arnold Bechtoldt]

I'm not a fan of map.jinja, it is very complex and confusing for Salt
beginners.

For some time I was using this solution:

*
https://github.com/bechtoldt/saltstack-elasticsearch-formula/blob/c27ff96ffcb0e609b76985e8cbe4e2ac53b4103c/elasticsearch/defaults.yaml#L1
*
https://github.com/bechtoldt/saltstack-elasticsearch-formula/blob/c27ff96ffcb0e609b76985e8cbe4e2ac53b4103c/elasticsearch/init.sls#L3

Then I switched to

*
https://github.com/bechtoldt/saltstack-elasticsearch-formula/blob/master/states/defaults.yaml
*
https://github.com/bechtoldt/saltstack-elasticsearch-formula/blob/master/states/init.sls#L3
*
https://github.com/bechtoldt/salt-modules/blob/master/_modules/formhelper.py#L83

I'm thinking about getting rid of the custom execution module, because
it is preventing users from using the formula.

Though I think #31812 won't address my personal requirements completely.
Unfortunately I don't have enough time to work on this at the moment.


Arnold

--
+arnoldbechtoldt • arnoldB@IRC • bechtoldt@GH • arbe.io

Am 24.02.16 um 23:31 schrieb fREW Schmidt:

> If I understand correctly, the map.jinja pattern exists to allow
> pillardata to be shared among pillars. You could consider it to be
> sensible OS level defaults too, but as far as I can see it's so that I
> can have pure data that various pillars read. Ok.
>
> Now lets say I'm writing a complex pillar and writing it in Jinja is too
> much work. Great, I have all these other renderers, I'll use the `#!py`
> one. So I write my pillar, but now I can't import from a map.jinja,
> right? If there is a way to import from the map.jinja, let me know,

> because that would be the *best *end to this issue.

>
> So, in an effort to get around this we've invented a new idiom, which is
> basically the exact same thing but python:
>
> #!py
>
> import logging
> import sys
> import os
>
> def run():
> # gross, hardcoded path
> new_path = os.path.join('/vol/saltstack/pillar',
> os.path.dirname(__sls__))
> if new_path not in sys.path:
> sys.path.append(new_path)
> from map import run as map_run
>
> return {
> "nested": map_run()
> }
>
> So that pulls data from the map.py in the same directory as the pillar.
> It feels like a lot of boilerplate, and maybe it could be reduced I tiny
> bit, but the worst thing is, now, jinja files can't read the shared data!
>
> Is there a way to do the map.jinja pattern for more than just the jinja
> renderer?
>
> --
> fREW Schmidt
> http://blog.afoolishmanifesto.com
>

> --
> You received this message because you are subscribed to the Google
> Groups "Salt-users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to salt-users+...@googlegroups.com
> <mailto:salt-users+...@googlegroups.com>.
> For more options, visit https://groups.google.com/d/optout.

[Seth House]

Hi, Arnold.

I don't see the distinction between a map file and a defaults file
that gets run through a defaults() function. In both cases you start
with a plain data structure and then filter it and optionally
post-process it (merge pillar overrides, perform more granular
filtering, etc) by calling one or more functions. They feel like
identical workflows to me. Am I overlooking something?

> To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+...@googlegroups.com.