Interpreting and logging what salt changed

[Peter Waller]

Hi All,

How do you keep around logs of what changes salt made to a system?

I have a few issues:

* Finding out what changed retrospectively is hard since the logs don't contain much information, as far as I can tell.

* Sometimes a developer's terminal or tmux scrollback isn't big enough to contain the complete output of a salt '*' state.highstate. That information then seems destroyed.

* Running the highstate on many machines produces a lot of output, where the amount of space consumed by "EVERYTHING IS OK" in green significantly outweighs the blues and the reds indicating "stuff happened". (We have hundreds of states and maybe one of them changes). 

* The order of the output is a random interspersion of green/blue/red, it would be nice if they were ordered (red last so it's obvious). (This would only help marginally when there are many machines anyway).

* Disseminating the (changes in the) state of the system between multiple developers?

* About 1 in ten times the output of `salt 'machine*' state.highstate` is completely empty, and the information about what changed is lost as far as I can tell.

What are other people doing in production? I'm stuck on 0.17 whilst the Ubuntu packages aren't out yet, and I'm also keen to take a look at halite, but I'm under the impression I can't (or shouldn't) use that without a more recent version of salt.

Thanks in advance,

- Peter

[Volker]

On 2/28/14 12:49 PM, Peter Waller wrote:
> Hi All,
>
> How do you keep around logs of what changes salt made to a system?
>

I use a daemon i wrote that hooks into the salt-event system.

https://github.com/felskrone/salt-eventsd

It basically collects all events from the event-bus and writes them
into a different (or a single) backend-worker(s). The events always
contain everything you need: data sent to the minions and everything
the return.

The backend-workers can do anything you want them to: mysql, postgres,
graphite, etc. They are simple python scripts and just have to supply
certain functions to be callable by the daemon. What you do within a
backend-worker, is up to you.

Have a look at the README and installation.txt. If salt-eventsd is
what you're looking for or if its something you can build on, try it out.

If you try it and run into problems, feel free to contact me.

- felskrone

[Peter Waller]

On 28 February 2014 14:01, Volker <g...@schwicking.de> wrote:

I use a daemon i wrote that hooks into the salt-event system.

https://github.com/felskrone/salt-eventsd

Thanks Volker,

That looks like a very nice effort but is there really nothing built into salt which will serve the purpose? It seems like something everyone would need.

[Tim O'Guin]

Hey Peter,

Salt has a built-in job cache that stores the return data for each jid. You can access it via the saltutil module, or via the jobs runner.

http://docs.saltstack.com/topics/jobs/index.html

--
You received this message because you are subscribed to the Google Groups "Salt-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Volker]

On 2/28/14 3:10 PM, Peter Waller wrote:
> On 28 February 2014 14:01, Volker <g...@schwicking.de

I've asked the very same question a while ago and unfortunately there
is nothing im aware of that scales well with several thousand minions
and is able to archive data without putting a burden on the masters
job-cache.

- Volker

[Peter Waller]

Aha, that looks a lot like what I'm looking for, thanks!

list_jobs only seems to go back a short way, about 6 hours. Is there a way to increase this?

[Volker]

On 2/28/14 3:26 PM, Tim O'Guin wrote:
> Hey Peter,
>
> Salt has a built-in job cache that stores the return data for each jid.
> You can access it via the saltutil module, or via the jobs runner.
>

The job-cache does not scale well and is not (yet) able to archive
long-term data. It should only be used for short-term data and only if
you dont have too many minions. 'too many' is of course use-case
dependent. What i can tell is: do not use it if you have several
thousand minions and if you want to archive data for longer than maybe a
week.

Besides these limitations using the job-cache is fine.

sidenote:

If you use the external_job cache, you lose salt-encryption which is not
very desirable.