ext_pillar: rubbish documentation, odd implementation

[Ari Maniatis]

So I'm moving some data from states into the pillar where it belongs. In particular DNS zone files and SSL certificates (with private keys) shouldn't be propagated to every minion, so they belong in the pillar where I can control them better.

The problems with using files as a source of pillar data started with the documentation. I first found this page:

       https://docs.saltstack.com/en/latest/ref/pillar/all/salt.pillar.file_tree.html

After several hours of trying different things, I discovered the trick: ext_pillar isn't a special command you can put in an arbitrary pillar file. Instead you have to put this command into to master configuration file. But none of that is mentioned and unless you happen to come across another page in a completely separate part of the docs, you'd never know.

This is a shame because now I can't use the pillar top.sls to control which minions see this pillar data. Instead I need to use nodegroups or hostnames. Why do I need a new way to attach pillar data to the correct minions, even after I have a top.sls I'm happy with?

OK, so I keep trying... several hours later I stumble on the fact that in order to get the changes to be recognised, I not only need

    salt '*' saltutil.refresh_pillar

But I also have to do a complete restart of the master service.

What now? Oh, the minion id reference is completely different to the way pillar normally works. So files for a minion called "colo.server1" isn't found at hosts/colo/server1/files but instead in hosts/colo.server1.files. I only hit upon this by trial and error... hence the hours of effort... because it is not documented anywhere and nothing appears with salt-call at debug log level.


So finally, I have my pillar data, split into two separate folder structures. One for sls files, and another for files I need to import like certificates and zones. I can't easily share data between minions with the flexibility of top.sls and all the targeting that allows.


So, like so much of salt: a great idea, with poor documentation and some odd choices that don't fit into the big picture. Is this just a natural outcome of an open source project which isn't led top-down by a company also offering a commercial product? How can casual users of salt like myself help?

[C. R. Oldham]

On Sat, Feb 11, 2017 at 10:17 PM, Ari Maniatis <aristedes...@gmail.com> wrote:

So I'm moving some data from states into the pillar where it belongs. In particular DNS zone files and SSL certificates (with private keys) shouldn't be propagated to every minion, so they belong in the pillar where I can control them better.

The problems with using files as a source of pillar data started with the documentation. I first found this page:

       https://docs.saltstack.com/en/latest/ref/pillar/all/salt.pillar.file_tree.html

I just submitted a PR (https://github.com/saltstack/salt/pull/39339) to add a link at the top of that page to our external pillar documentation (the document you found).  I can see why that would be confusing. 

This is a shame because now I can't use the pillar top.sls to control which minions see this pillar data. Instead I need to use nodegroups or hostnames. Why do I need a new way to attach pillar data to the correct minions, even after I have a top.sls I'm happy with?

That has to do with the way external pillars work.  When a minion retrieves its pillar data it passes in its ID.  The external pillar module decides if it will return any data for that ID.  

 

OK, so I keep trying... several hours later I stumble on the fact that in order to get the changes to be recognised, I not only need

    salt '*' saltutil.refresh_pillar

This is true. 

But I also have to do a complete restart of the master service.

I think this is only true for nodegroups, not external pillars.

 

[...]

So, like so much of salt: a great idea, with poor documentation and some odd choices that don't fit into the big picture. Is this just a natural outcome of an open source project which isn't led top-down by a company also offering a commercial product?

Though we are a company offering a commercial product we are also quite liberal in the PRs we accept.  The challenge with this is keeping the documentation up to date and coherent.  We acknowledge that this is a problem and continue to work to fix it.

And to be fair, we get compliments as well as complaints about our docs.  Writing docs that work well for users of all backgrounds is an art form, especially in a project as large and flexible as SaltStack.

 

How can casual users of salt like myself help?

If you are familiar with Python it is certainly possible to make file_tree behave the way you want it to.  But even if you don't know Python it's not hard to submit a PR to add information to the documentation that would have helped you.  For those of us that live and breathe Salt every day it's sometimes hard to see what new users (or even just users that are trying a feature for the first time) might miss or find confusing.  We love it when Salt users file doc PRs, because it brings an end-user perspective that we sometimes don't have.

I hear your frustration, but also appreciate that you took time to email the list.  Thanks for the feedback.

--cro

C. R. Oldham

Senior Platform Engineer, Integrations Team Lead, SaltStack

c...@saltstack.com / 801-564-4673

[Johann Kappacher]

Hello,

I share the frustration about some annoying odds with Ari, but I appreciate your reply and the call for PRs.

I promise I will become more active/proactive than reactive. (starting with some reading about the Github/PR policy as mentioned in the docs.)

Thanks for the warm climate and your patience with frustrated usability testers like me! (yes, after a long night session :)

Hannes

[Aristedes Maniatis]

Thanks for your quick and helpful response.

On 12/2/17 5:09pm, C. R. Oldham wrote:

On Sat, Feb 11, 2017 at 10:17 PM, Ari Maniatis <aristedes...@gmail.com> wrote:

This is a shame because now I can't use the pillar top.sls to control which minions see this pillar data. Instead I need to use nodegroups or hostnames. Why do I need a new way to attach pillar data to the correct minions, even after I have a top.sls I'm happy with?

That has to do with the way external pillars work.  When a minion retrieves its pillar data it passes in its ID.  The external pillar module decides if it will return any data for that ID. 

I guess there are technical reasons why it was built the way it was. Unfortunately that means that there is a huge gap in functionality between one line values (stored in yaml) and multiline content like an SSL certificate. Being able nominate a folder of files from the top.sls would be ideal and remove what appears an arbitrary distinction for end users.

 

But I also have to do a complete restart of the master service.

I think this is only true for nodegroups, not external pillars.

I think it was needed for adding the ext_pillar line to the master config.

How can casual users of salt like myself help?

If you are familiar with Python it is certainly possible to make file_tree behave the way you want it to.  But even if you don't know Python it's not hard to submit a PR to add information to the documentation that would have helped you.  For those of us that live and breathe Salt every day it's sometimes hard to see what new users (or even just users that are trying a feature for the first time) might miss or find confusing.  We love it when Salt users file doc PRs, because it brings an end-user perspective that we sometimes don't have.

I hear your frustration, but also appreciate that you took time to email the list.  Thanks for the feedback.

Thank you. I'll feel more comfortable now making my own doc PRs when I hit this sort of thing, knowing that they are encouraged. However I would have know how to create the correct link syntax.

Ari

[Johann Kappacher]

Hello Ari,

I will try to answer as a novice to Jinja/YAML and Salt.

Your basic question was: How to bring file contents into the Pillar System or SLS (without using external pillar). Am I right?

My approach to your problem:

1) You can use Multiline Strings in YAML (look at the "literal block" Pipe Symbol) and you can feed the files with a Jinja Loop Stmt.

2) You can also use lists (list of lines of file X) as a vaule for your pillar attribute.

3) ... or even better: You can write your pillar sls file like this:

/srv/pillar/files_to_pillar.sls:

#!py

def run():

  file = open('/etc/resolv.conf').read()

  return {'mypillar': file}

Good luck!

Hannes

[Johann Kappacher]

Hello,

coding Jinja can be very tricky because of whitespace control of the Jinja-to-YAML output.

For instance, a multiline block in YAML/Jinja:

/srv/pillar/files_to_pillar.sls:

mypillar: |

{%- import_text '/tmp/bigfile.txt' as bigfile %}

{{ bigfile|indent(2,true) }}

ciao

hannes

[Aristedes Maniatis]

Brilliant, thanks for this.

It hadn't even occurred to me that I can embed jinja inside my pillar files in this way. That's certainly a much cleaner approach. I'm going to try something like

{%- for each zone_name in os.listdir(mypath) -%}
{%- import_text mypath + zone_name as zone: -%}
{{ zone_name }}: |
{{ zone|indent(2,true) }}
{% endfor %}

Then I can put this anywhere I want in the pillar structure, control access any way I want, and not worry about special ext_pillar handling.

I'll test the above today.

Thanks
Ari

[Ari Maniatis]

Unfortunately this isn't going to work. I can't find a way to list a directory of files in jinja. "os.listdir(mypath)" cannot work because I cannot import "os".

I looked for ways to hook in my own jinja filter into saltstack, but other than editing the core code in https://github.com/saltstack/salt/blob/2016.11/salt/utils/jinja.py I couldn't find a way. Is there any hook to add in my own filter?

Saltstack kindly gives us import_text, but now we need list_dir. I managed to get ext_pillar working, but unfortunately this approach puts the files in unhelpful places in the pillar structure, all mixed up with any files you import this way.

Does anyone know another way?

[Johann Kappacher]

Hello Ari

you can use all Salt Execution Module Functions in Jinja if you want:

Quick&Dirty:

{% for file in salt.cmd.run('ls /dirname') %}

Or without a shell-out (avoiding spawning a new subshell/process):

{% for file in salt.file.find('/dirname') %}

Maybe, you will optionally need trim() or strip() or other filter/convert/manip. Utils, but this is it!

For instance: (please be careful, the cmd.run(ls DIR) prints out short filenames!)

mypillar: |
{%- for file in salt.cmd.run('ls /srv/pillar').split() %}
  {{file}}
{%- endfor %}

mypillar: |
{%- for file in salt.file.find('/srv/pillar') %}
  {{file}}
{%- endfor %}

[Rémy Dernat]

Hi,

Like Ari, I moved to ext_pillar (git), and it was not easy, because, as you said, the documentation is huge and it can be tough to find what you are looking for, but I would not say it was tricky. In fact, I get a lot of help from IRC, as always. BTW, I would like to thanks all the guys from IRC who helped us all the time (gtmanfred, basepi, babilen, XenophonF, AndreasLutro and all I have forgotten...) !!

For the command 

"salt '*' saltutil.refresh_pillar"

-> I created 2 little script which can be found here : https://gist.github.com/remyd1/2853f489d828ecda52183ca37a813f4e

One can be used to refresh just pillars, the other one should be used if you transfer all your states to a git external storage.

However, I have to warn you, you have to wait a bit for the refresh (locally, I am waiting about a minute generally).

Moreover, if you are using saltstack 2016.11.2 (carbon), you will have some errors with salt-run (but those bugs will be fixed in the next release).

BTW, moving to full external storage with git  will give you more flexibility if you have a multi-masters configuration locally, like me :) 

Kind regards,

Rémy

--
You received this message because you are subscribed to the Google Groups "Salt-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/salt-users/b7211dde-981e-4c62-b20b-25c64d557e4c%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Thomas Phipps]

there is alternatives to the jinja,yaml render. including the python render. however this is a a road that most agree needs to be better documented. as not as many people use alternative renders as use jinja,yaml. 

entree can be found here into the world of renders 

https://docs.saltstack.com/en/latest/ref/renderers/

and the py render documentation is here

https://docs.saltstack.com/en/latest/ref/renderers/all/salt.renderers.py.html

not saying this is the answer to the issue you brought up. just an idea. some people are a lot more comfortable with python then with jinja. 

--

You received this message because you are subscribed to the Google Groups "Salt-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/salt-users/881ac86c-7fff-4142-9170-682d686c729b%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Thomas Phipps

Linux/Openstack/Logstash

(702) 518-0131

[Dimitri Maziuk]

On 02/13/2017 09:48 AM, Thomas Phipps wrote:
...

> some people are a lot more comfortable with python then with jinja.

Well, YAML's barely tolerable on its own and jinja's OK for templating
webpages, but their marriage is... pushing it.

Putting stuff in /srv/salt/pillars/base seems to work for me. As does
'__salt__["saltutil.refresh_pillar"]` in (#!py) state files.

--
Dimitri Maziuk
Programmer/sysadmin
BioMagResBank, UW-Madison -- http://www.bmrb.wisc.edu

[Ari Maniatis]

Hi everyone

Thanks for much for the help on this issue. In the end, I avoided ext_pillar entirely and ended up with a much better and more flexible solution in only four lines. I'd help with some documentation but I have no idea where it fits in the salt site. Perhaps under ext_pillar there should be a section titled "You probably don't want to use this, try this other thing instead."

So, how did I get my zones in the pillar data? Easy:

zones:
{%- for zone_file in salt.file.find('/usr/local/etc/salt/data/pillar/zones/', type='f') %}
  {{ salt.file.basename(zone_file) }}:
    content: |
{%- import_text zone_file as zone %}
{{ zone | indent(6,true) }}
{%- endfor %}

The above code went straight into any one of the pillar files you already use. Use grains, roles, minion id or anything else to target this from top.sls. You can place the end result anywhere in the pillar structure you like. I put mine at the top level under "zones".

What you see here is then a simple find to grab all the files (not the directories since I put the 120 zones files I manage into folders for easier navigation). It then extracts the filename (which for me is the domain of the zone). Next we read the file contents using import_text, one of the salt extensions to jinja. We put that into the variable 'zone'.

Because we are using the pipe "|" symbol to insert multiline text content into YAML, we need to indent the file past the "content:" key. We do that with the indent() jinja filter.


And that's the whole thing! But to get there I needed to understand several things:

1. You can put jinja into pillar files
2. jinja can call any salt module from the "salt" object
3. There is a salt file module which does lots of helpful things
4. Salt extends jinja with the import_text command, even through there is no salt.file.read command
5. ext_pillar... just don't. Maybe there is some useful things you can do reading data from a db or remote git repo, but for folders and files on disk you just don't want it.

[Ari Maniatis]

One more caveat for anyone reading along. If the first line of the zone file starts with a tab, this approach will break. So you might want to convert:

              MX  10 mail.example.com.

into

@           MX 10 mail.example.com.

which is exactly the same but doesn't start with a tab.

Also, I wrote this really simple trick to get an automatically updating serial into the output zone. It just inserts the unix time (seconds since 1970) into the pillar which is then easy to use in your state.

zones:
{%- for zone_file, mtime in salt.file.find('/usr/local/etc/salt/data/pillar/zones/', type='f', print='path,mtime') %}
  {{ salt.file.basename(zone_file) }}:
    serial: {{ mtime }}

    content: |
{%- import_text zone_file as zone %}

{{ zone | trim | indent(6,true) }}
{%- endfor %}

Ari

[Jon Tegner]

Hi! 

Regarding your options, in particular nr 3, a principal (and perhaps stupid) question: When do one really need external pillar? If you can achieve what you want with python, why bother with external pillar?

Thanks!

/jon

--

You received this message because you are subscribed to the Google Groups "Salt-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/salt-users/9843900a-0a7f-4487-9180-425ae97e8596%40googlegroups.com.

[Aristedes Maniatis]

My use case is that YAML is a terrible file format for multi-line text. The indenting causes all sorts of problems with things like SSL certificates.

Other people may want to have data in other places: version control, databases, etc.

Ari

You received this message because you are subscribed to a topic in the Google Groups "Salt-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/salt-users/PaMVmau8D1Y/unsubscribe.
To unsubscribe from this group and all its topics, send an email to salt-users+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/salt-users/CAP5kWAGRPSSvoe4BwUZbxxzCPsfMPmc9yXfhqs3tUFL85iVggw%40mail.gmail.com.

[Jon Tegner]

It is probably something I fail to understand here, but whatever use case you have, it seems to me that most external data has to be "handled" in some way before it can be read in as pillars. What I don't understand is why one should use the "ext_pillar" framework rather than just creating an "ordinary" "#!py"-file under pillar_root (for one thing you don't have to touch the master configuration), since even in this case it should be possible to pull in external data?

/jon

To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/salt-users/9843900a-0a7f-4487-9180-425ae97e8596%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to a topic in the Google Groups "Salt-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/salt-users/PaMVmau8D1Y/unsubscribe.

To unsubscribe from this group and all its topics, send an email to salt-users+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/salt-users/CAP5kWAGRPSSvoe4BwUZbxxzCPsfMPmc9yXfhqs3tUFL85iVggw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Salt-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/salt-users/ac4f135b-8667-27de-84c2-43f438d97129%40gmail.com.

[Aristedes Maniatis]

If you read to the end of this thread (you seem to have picked up an email from February in the middle of the conversation) you'll see how I solved this problem. ext_pillar isn't terribly useful.

Ari

To unsubscribe from this group and all its topics, send an email to salt-users+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/salt-users/CAP5kWAEWHPjj2_mhW6yTskv63gGbGcXBZDQLkW144F9D5sjb0Q%40mail.gmail.com.

[Jon Tegner]

Yes, I saw that. Guess that what I want is just an explanation of when and why one should use ext_pillar.

/jon

To unsubscribe from this group and all its topics, send an email to salt-users+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/salt-users/CAP5kWAEWHPjj2_mhW6yTskv63gGbGcXBZDQLkW144F9D5sjb0Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Salt-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/salt-users/02150a10-ab51-5db1-96ed-49f96789b008%40gmail.com.

[C. R. Oldham]

Most folks use it to connect an external system like a CMDB to Salt so the external system's data is exposed to Pillar.

-- 

C. R. Oldham, Engineer, SaltStack

c...@saltstack.com

To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/salt-users/CAP5kWAFXeguaQseVrUwxO6tD9MTJVRjE6_Ppnb4Te7hS%2B4GFFA%40mail.gmail.com.

[Rémy Dernat]

Hi

I am using it (within MongoDB) with a reactor to update some pillars from my minions. In my use case, I couldn't use Salt Mines, because the reactor is updating another minion which actually contains a kind of database.

For the other pillars I am using gitfs pillars to ease the use of pillars between 3 salt masters.

Best regards,

Remy

PS: the multine issue with YAML is easily solved with `|` character in Jinja syntax.

To view this discussion on the web visit https://groups.google.com/d/msgid/salt-users/CAP5kWAFXeguaQseVrUwxO6tD9MTJVRjE6_Ppnb4Te7hS%2B4GFFA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "Salt-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/salt-users/5B8F07A8-483D-4148-B922-F5E1C96C989E%40saltstack.com.