External/python pillar tops

[Gergely Czuczy]

Hello,

I would like to ask for some advice, hopefully some practical one
regarding how to properly do a python pillar top, where the source of
the data is an external REST API with saltstack 2016.3.1, since it
wasn't quite clear from the documentation. There seem to be many ways,
however the pillar top appears to be a special little snowflake. Also,
the many possible alternative ways are a bit confusing.

So, in a nutshell, I would like to assemble my pillar tops based on a
REST API, using python, and I don't really know what should be an
optimal way of doing it.

Also, if anyone can recall how to query the pillar tops data, please
share it, that always requires quite some googling, docs are also not
clear on that one.

Best regards,
Gergely

[Seth House]

The Pillar top file will follow the same syntax:

#!py
def run():
return {
'base': {
'<tgt1>': ['list', 'of', 'sls', 'files'],
'<tgt2>': ['list', 'of', 'sls', 'files'],
},
}

So you can `import` and do pretty much anything in there so long as
the return dictionary is formatted correctly.

But! That file will get parsed and executed one time for _each_ minion
that is requesting its Pillar so if you're making an HTTP call, you'll
almost definitely want to cache the HTTP response on the filesystem.

You might even want to avoid doing the HTTP call during Pillar
generation altogether by using the scheduler or cron to fetch the data
from the API on a regular interval and put it into a local file that
can simply be imported from Jinja. E.g.

{% import_json '/path/to/cache_file.json' as my_top_data %}

> --
> You received this message because you are subscribed to the Google Groups
> "Salt-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to salt-users+...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Gergely Czuczy]

Thank you, got that.

I have a very simple test top.sls pillar, basically the same as your
example, just <tgt> filled in.

however, pillar.show_top returns nothing. When I'm running salt-master
with -l trace, I'm getting the following output:

[TRACE ] _get_event() waited 5 seconds and received nothing
[TRACE ] IPCServer: Handling connection to address:
[TRACE ] get_event() received = {'tag':
'salt/run/20160824083409127559/new', 'data': {'fun':
'runner.pillar.show_top', 'jid': '20160824083409127559', 'user': 'root',
'_stamp': '2016-08-24T13:34:09.476663'}}
[DEBUG ] Gathering reactors for tag salt/run/20160824083409127559/new
[ERROR ] Exception occurred while handling stream: [Errno 0] Success
[TRACE ] get_event() received = {'tag':
'salt/run/20160824083409127559/ret', 'data': {'jid':
'20160824083409127559', 'return': {}, 'success': True, '_stamp':
'2016-08-24T13:34:09.765474', 'user': 'root', 'fun':
'runner.pillar.show_top'}}
[DEBUG ] Gathering reactors for tag salt/run/20160824083409127559/ret

That error message seems very useless to me. Any idea how could I debug
and fix this? Oh, and it's on a redhat7.

[Seth House]

On Wed, Aug 24, 2016 at 7:36 AM, Gergely Czuczy
<gergely...@harmless.hu> wrote:
> I have a very simple test top.sls pillar, basically the same as your
> example, just <tgt> filled in.
>
> however, pillar.show_top returns nothing. When I'm running salt-master with
> -l trace, I'm getting the following output:

Ah, gotcha. Debugging that can be a little opaque for sure. Since
there isn't a render step (like with Jinja) there's nothing to show in
the Salt log. If you have a traceback or syntax error then Salt should
show you that. But if the return data structure is subtly wrong then
it will probably fail silently.

Your best tools to debug that are print-lining:

#!py
from pprint import pprint
def run():
ret = {
'base': {
'*': ['foo'],
},
}
pprint('XXX: %s' % ret)
return ret

Or using the Python debugger:

#!py
def run():
ret = {
'base': {
'*': ['foo'],
},
}
import pdb; pdb.set_trace()
return ret

Both of which should work with the `pillar.show_top` runner.

[Gergely Czuczy]

Sadly, this yielded nothing.
Currently the top.sls looks like this:
---
#!py

import logging
import logging.handlers
import datetime

from pprint import pprint

def run():
ret = {'base': {

'*': ['foobar']
}}
return ret

if __name__ == '__main__':
pprint(run())
---

When I run it manually I get the following:
# python /srv/salt/salt-modules/top.sls
{'base': {'*': ['foobar']}}

However, both what you suggested yields no results. The following
message is in the error log as it's anyway:
Aug 25 01:56:52 xxx journal: 2016-08-25 01:56:52,422
[salt.transport.ipc][ERROR ][5023] Exception occurred while handling
stream: [Errno 0] Success

When adding pdb, then when manually run, it breaks into pdb, when
calling through salt, nothing happens.

Similar with the pprint way, manually it shows, when running it through
salt, it doesn't show up at all.

On the other hand, I've found the issue. There was a typo in the master
config when specifying the path for pillar_roots.base ... Too simple.

Anyway, I really appreciate all the help I got, especially that
debugging knowledge always comes handy, and I picked a few things up.

[Seth House]

Ouch! That's rough. But I'm glad you found it. :-)