Nagios + check_mk + Salt Mine

341 views

Skip to first unread message

Dan Garthwaite

unread,

Jul 24, 2013, 7:35:22 PM7/24/13

to salt-...@googlegroups.com

First - The documentation to get Salt Mine up and running are infuriatingly sparse. And googling for 'salt mine' will also drive you crazy. :) From the docs: "To enable the Salt Mine the mine_functions option needs to be applied to a minion." I have never previously seen discussed or documented how one applies an "option" to a minion. Especially via a pillar. Working backwards --and gathering details from the sparse sum of four salt-users threads on Salt Mine-- I got it to work.

Second - We could do away with the custom module if Salt Mine could simply execute commands, and affix a name to this command.

The custom module:

In /srv/salt/_modules/check_mk.py

#!/usr/bin/env python
'''Support for running check_mk_agent over salt'''
import os
import salt.utils
from salt.exceptions import SaltException

def __virtual__():
''' Only load the module if check-mk-agent is installed '''
if os.path.exists('/usr/bin/check_mk_agent'):
return 'check_mk'
return False

def agent():
''' Return the output of check_mk_agent '''
return __salt__['cmd.run']('/usr/bin/check_mk_agent')

In /srv/pillar/top.sls

base:
targetminion:
- mine

In /src/pillar/mine.sls:

mine_functions:
check_mk.agent: []

Command to run on salt master:

salt targetminion saltutil.refresh_pillar

Verify up to this point # Don't proceed until this grep succeeds!

salt targetminion pillar.data mine_functions | grep check_mk.agent

Wait 5 minutes, or call mine.update:

salt targetminion mine.update

Better yet, run it locally on the minion in debug mode:

ssh targetminion salt-call mine.update -l debug

And, finally, see if there is output and clean it up via some regex backflips:

salt lead mine.get lead check_mk.agent --out=raw | sed 's#\\n#\n#g' | sed -e 's/.*<<<check_mk>>>/<<<check_mk>>>/g' -e "s/'}}//g"

Why, oh why, doesn't --out=raw or --out=txt do either raw nor text? Just output a line demarcating the minion id and dump the output.

For me this resulted in:

<<<check_mk>>>

Version: 1.2.0p3

AgentOS: linux

PluginsDirectory: /usr/lib/check_mk_agent/plugins

LocalDirectory: /usr/lib/check_mk_agent/local

AgentDirectory: /etc/check_mk

<<<df>>>

/dev/xvda ext3 10352888 7383952 2444712 76% /

/dev/xvdc ext3 30705676 18325508 12068220 61% /http

<<<nfsmounts>>>

<<<mounts>>>

/dev/root / ext3 rw,relatime,errors=remount-ro,barrier=0,data=writeback 0 0

/dev/xvdc /http ext3 rw,relatime,errors=remount-ro,barrier=0,data=writeback 0 0

<<<ps>>>

(root,2088,588,0.0) init [2]

(root,0,0,0.0) [kthreadd]

(root,0,0,0.0) [ksoftirqd/0]

(root,0,0,0.0) [kworker/0:0]

(root,0,0,0.0) [kworker/0:0H]

(root,0,0,0.0) [kworker/u:0]

(root,0,0,0.0) [kworker/u:0H]

(root,0,0,0.0) [migration/0]

(root,0,0,0.0) [rcu_bh]

(root,0,0,0.0) [rcu_sched]

(root,0,0,0.0) [migration/1]

(root,0,0,0.0) [ksoftirqd/1]

(root,0,0,0.0) [kworker/1:0]

(root,0,0,0.0) [kworker/1:0H]

(root,0,0,0.0) [migration/2]

(root,0,0,0.0) [ksoftirqd/2]

(root,0,0,0.0) [kworker/2:0]

(root,0,0,0.0) [kworker/2:0H]

(root,0,0,0.0) [migration/3]

(root,0,0,0.0) [ksoftirqd/3]

(root,0,0,0.0) [kworker/3:0]

(root,0,0,0.0) [kworker/3:0H]

(root,0,0,0.0) [migration/4]

(root,0,0,0.0) [ksoftirqd/4]

(root,0,0,0.0) [kworker/4:0]

(root,0,0,0.0) [kworker/4:0H]

(root,0,0,0.0) [migration/5]

(root,0,0,0.0) [ksoftirqd/5]

(root,0,0,0.0) [kworker/5:0]

(root,0,0,0.0) [kworker/5:0H]

(root,0,0,0.0) [migration/6]

(root,0,0,0.0) [ksoftirqd/6]

(root,0,0,0.0) [kworker/6:0]

(root,0,0,0.0) [kworker/6:0H]

(root,0,0,0.0) [migration/7]

(root,0,0,0.0) [ksoftirqd/7]

(root,0,0,0.0) [kworker/7:0]

(root,0,0,0.0) [kworker/7:0H]

(root,0,0,0.0) [khelper]

(root,0,0,0.0) [kdevtmpfs]

(root,0,0,0.0) [kworker/u:1]

(root,0,0,0.0) [xenwatch]

(root,0,0,0.0) [xenbus]

(root,0,0,0.0) [bdi-default]

(root,0,0,0.0) [kblockd]

(root,0,0,0.0) [kworker/4:1]

(root,0,0,0.0) [md]

(root,0,0,0.0) [rpciod]

(root,0,0,0.0) [kworker/2:1]

(root,0,0,0.0) [kworker/5:1]

(root,0,0,0.0) [kswapd0]

(root,0,0,0.0) [ksmd]

(root,0,0,0.0) [fsnotify_mark]

(root,0,0,0.0) [ecryptfs-kthrea]

(root,0,0,0.0) [nfsiod]

(root,0,0,0.0) [cifsiod]

(root,0,0,0.0) [jfsIO]

(root,0,0,0.0) [jfsCommit]

(root,0,0,0.0) [jfsSync]

(root,0,0,0.0) [xfsalloc]

(root,0,0,0.0) [xfs_mru_cache]

(root,0,0,0.0) [xfslogd]

(root,0,0,0.0) [glock_workqueue]

(root,0,0,0.0) [delete_workqueu]

(root,0,0,0.0) [gfs_recovery]

(root,0,0,0.0) [crypto]

(root,0,0,0.0) [khvcd]

(root,0,0,0.0) [drbd-reissue]

(root,0,0,0.0) [aoe_tx]

(root,0,0,0.0) [aoe_ktio]

(root,0,0,0.0) [kworker/6:1]

(root,0,0,0.0) [kworker/3:1]

(root,0,0,0.0) [kworker/1:1]

(root,0,0,0.0) [kworker/0:1]

(root,0,0,0.0) [kworker/7:1]

(root,0,0,0.0) [kpsmoused]

(root,0,0,0.0) [deferwq]

(root,0,0,0.0) [kworker/0:1H]

(root,0,0,0.0) [kjournald]

(root,2312,292,0.0) udevd --daemon

(root,2308,200,0.0) udevd --daemon

(root,2308,204,0.0) udevd --daemon

(root,0,0,0.0) [kjournald]

(root,0,0,0.0) [flush-202:0]

(www-data,1960,188,0.0) /usr/sbin/fcgiwrap

(root,74624,17408,0.4) /usr/bin/python /usr/bin/salt-minion

(root,28596,1768,0.0) /usr/sbin/rsyslogd -c5

(root,5032,412,0.0) nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf

(www-data,5356,1868,0.0) nginx: worker process

(root,2388,632,0.0) dhclient -v -pf /var/run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0

(root,11408,5844,0.0) /usr/bin/python /usr/bin/supervisord

(stunnel4,13896,10808,0.0) /usr/bin/stunnel4 stunnel-beaver.conf

(stunnel4,3964,704,0.0) /usr/bin/stunnel4 stunnel-beaver.conf

(stunnel4,3964,716,0.0) /usr/bin/stunnel4 stunnel-beaver.conf

(daemon,2216,344,0.0) /usr/sbin/atd

(root,4428,908,0.0) /usr/sbin/cron

(root,67600,55128,0.8) /opt/beaver/venv/bin/python /opt/beaver/venv/bin/beaver -c /opt/beaver/logs.ini -t redis

(root,25260,11668,0.0) /opt/beaver/venv/bin/python /opt/beaver/venv/bin/beaver -c /opt/beaver/logs.ini -t redis

(105,2632,628,0.0) /usr/bin/dbus-daemon --system

(avahi,3020,1344,0.0) avahi-daemon: running [lead.local]

(avahi,2896,356,0.0) avahi-daemon: chroot helper

(101,6500,808,0.0) /usr/sbin/exim4 -bd -q30m

(root,25548,5464,0.0) /usr/bin/python /usr/bin/fail2ban-server -b -s /var/run/fail2ban/fail2ban.sock

(jenkins,4288,308,0.0) /usr/bin/daemon --name=jenkins --inherit --env=JENKINS_HOME=/var/lib/jenkins --output=/var/log/jenkins/jenkins.log --pidfile=/var/run/jenkins/jenkins.pid -- /usr/bin/java -Xmx256m -jar /usr/share/jenkins/jenkins.war --webroot=/var/cache/jenkins/war --httpPort=8080 --ajp13Port=-1

(jenkins,547868,357556,0.1) /usr/bin/java -Xmx256m -jar /usr/share/jenkins/jenkins.war --webroot=/var/cache/jenkins/war --httpPort=8080 --ajp13Port=-1

(root,5548,980,0.0) /usr/sbin/sshd

(root,0,0,0.0) [kworker/2:1H]

(root,7340,5252,0.0) /usr/sbin/munin-node

(root,1732,472,0.0) /sbin/mingetty hvc0

(root,0,0,0.0) [kworker/4:1H]

(root,8336,2848,0.0) sshd: dan [priv]

(dan,8484,1336,0.0) sshd: dan@pts/1

(dan,9740,6416,0.0) -bash

(dan,4900,964,0.0) screen -dR

(dan,7040,3756,0.0) /bin/bash

(dan,5084,2300,0.0) ssh hafnium

(root,0,0,0.0) [flush-202:32]

(root,4424,1148,0.0) sudo salt-call mine.update -l debug

(root,59848,19544,52.5) /usr/bin/python /usr/bin/salt-call mine.update -l debug

(root,2704,1192,0.0) /bin/bash /usr/bin/check_mk_agent

(root,2404,912,0.0) ps ax -o user,vsz,rss,pcpu,command --columns 10000

(root,1924,468,0.0) sed -e 1d -e s/ *\\([^ ]*\\) *\\([^ ]*\\) *\\([^ ]*\\) *\\([^ ]*\\) */(\\1,\\2,\\3,\\4) /

(ntp,4728,1892,0.0) /usr/sbin/ntpd -p /var/run/ntpd.pid -g -u 104:106

(redis,248548,17880,0.6) /usr/bin/redis-server /etc/redis/redis.conf

(root,419376,255124,1.1) java -server -Xms64m -Xmx128m -Xss256k -cp /opt/logstash/app/ logstash.runner agent -f logstash.conf --grok-patterns-path=/opt/logstash/app/patterns

(dan,8028,4412,0.0) SCREEN -dR

(dan,9900,6544,0.0) /bin/bash

(dan,7056,3820,0.0) /bin/bash

(dan,5220,2300,0.0) ssh ephemeral

(root,0,0,0.0) [kworker/3:1H]

(root,0,0,0.0) [kworker/1:1H]

(root,0,0,0.0) [kworker/6:1H]

(root,0,0,0.0) [kworker/7:1H]

(root,71160,11284,0.0) /usr/bin/python /usr/bin/salt-master -d

(root,26780,16988,0.0) /usr/bin/python /usr/bin/salt-master -d

(root,70132,10084,0.0) /usr/bin/python /usr/bin/salt-master -d

(root,69996,11012,0.0) /usr/bin/python /usr/bin/salt-master -d

(root,134700,26200,0.3) /usr/bin/python /usr/bin/salt-master -d

(root,134248,25580,0.3) /usr/bin/python /usr/bin/salt-master -d

(root,135860,26036,0.3) /usr/bin/python /usr/bin/salt-master -d

(root,135748,26140,0.2) /usr/bin/python /usr/bin/salt-master -d

(root,135196,26408,0.3) /usr/bin/python /usr/bin/salt-master -d

<<<mem>>>

MemTotal: 2062176 kB

MemFree: 50564 kB

Buffers: 298628 kB

Cached: 675908 kB

SwapCached: 1684 kB

Active: 911568 kB

Inactive: 942704 kB

Active(anon): 515168 kB

Inactive(anon): 364592 kB

Active(file): 396400 kB

Inactive(file): 578112 kB

Unevictable: 0 kB

Mlocked: 0 kB

HighTotal: 1357832 kB

HighFree: 6260 kB

LowTotal: 704344 kB

LowFree: 44304 kB

SwapTotal: 1310712 kB

SwapFree: 1306960 kB

Dirty: 184 kB

Writeback: 0 kB

AnonPages: 878652 kB

Mapped: 20744 kB

Shmem: 20 kB

Slab: 95744 kB

SReclaimable: 81548 kB

SUnreclaim: 14196 kB

KernelStack: 2800 kB

PageTables: 4436 kB

NFS_Unstable: 0 kB

Bounce: 0 kB

WritebackTmp: 0 kB

CommitLimit: 2341800 kB

Committed_AS: 1826012 kB

VmallocTotal: 122880 kB

VmallocUsed: 5820 kB

VmallocChunk: 111212 kB

DirectMap4k: 739320 kB

DirectMap2M: 0 kB

<<<cpu>>>

0.16 0.06 0.06 1/343 13543 8

<<<uptime>>>

540599.33 8593458.70

<<<lnx_if:sep(58)>>>

gre0: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

ip6gre0: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

sit0: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

dummy0: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

lo: 1603831401 6369896 0 0 0 0 0 0 1603831401 6369896 0 0 0 0 0 0

eth0: 842049119 7300174 0 0 0 0 0 0 6558431548 7014695 0 0 0 0 0 0

ip6tnl0: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

tunl0: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

teql0: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

<<<tcp_conn_stats>>>

ESTABLISHED 83

TIME_WAIT 21

<<<md>>>

Personalities : [linear] [raid0] [raid1] [raid6] [raid5] [raid4] [multipath]

unused devices: <none>

<<<diskstat>>>

1374708333

202 0 xvda 230523 113252 6200938 582287 1311416 8385959 77581024 6060294 0 387923 6622652

202 16 xvdb 104 488 4736 829 760 308 8544 1911 0 1135 2737

202 32 xvdc 159234 2359 5947250 236593 49823 210220 2080384 90727 0 176874 326575

[dmsetup_info]

No devices found

<<<kernel>>>

1374708333

nr_free_pages 12619

nr_inactive_anon 91148

nr_active_anon 128751

nr_inactive_file 144528

nr_active_file 99100

nr_unevictable 0

nr_mlock 0

nr_anon_pages 219602

nr_mapped 5186

nr_file_pages 244055

nr_dirty 46

nr_writeback 0

nr_slab_reclaimable 20387

nr_slab_unreclaimable 3549

nr_page_table_pages 1109

nr_kernel_stack 350

nr_unstable 0

nr_bounce 0

nr_vmscan_write 1068

nr_vmscan_immediate_reclaim 22

nr_writeback_temp 0

nr_isolated_anon 0

nr_isolated_file 0

nr_shmem 5

nr_dirtied 9670071

nr_written 9559506

nr_anon_transparent_hugepages 0

nr_free_cma 0

nr_dirty_threshold 26661

nr_dirty_background_threshold 13330

pgpgin 6077678

pgpgout 39834976

pswpin 533

pswpout 1068

pgalloc_dma 10632

pgalloc_normal 50676568

pgalloc_high 88989524

pgalloc_movable 0

pgfree 139691067

pgactivate 638442

pgdeactivate 311506

pgfault 384352708

pgmajfault 6152

pgrefill_dma 0

pgrefill_normal 60900

pgrefill_high 259113

pgrefill_movable 0

pgsteal_kswapd_dma 3

pgsteal_kswapd_normal 583372

pgsteal_kswapd_high 749126

pgsteal_kswapd_movable 0

pgsteal_direct_dma 0

pgsteal_direct_normal 17291

pgsteal_direct_high 0

pgsteal_direct_movable 0

pgscan_kswapd_dma 3

pgscan_kswapd_normal 583404

pgscan_kswapd_high 755781

pgscan_kswapd_movable 0

pgscan_direct_dma 0

pgscan_direct_normal 17291

pgscan_direct_high 0

pgscan_direct_movable 0

pgscan_direct_throttle 0

pginodesteal 2910

slabs_scanned 1429504

kswapd_inodesteal 49845

kswapd_low_wmark_hit_quickly 478

kswapd_high_wmark_hit_quickly 1154

pageoutrun 28277

allocstall 361

pgrotated 1095

unevictable_pgs_culled 0

unevictable_pgs_scanned 0

unevictable_pgs_rescued 144

unevictable_pgs_mlocked 144

unevictable_pgs_munlocked 144

unevictable_pgs_cleared 0

unevictable_pgs_stranded 0

cpu 1257841 0 482581 429237466 42808 45 8616 220893 0 0

cpu0 162453 0 76636 53515433 34360 44 4355 44882 0 0

cpu1 172591 0 65466 53652857 3649 0 476 25583 0 0

cpu2 215799 0 79831 53588339 1242 0 601 26772 0 0

cpu3 153634 0 57942 53673436 1031 0 609 25373 0 0

cpu4 142075 0 69805 53676123 745 0 669 26042 0 0

cpu5 146886 0 56767 53691584 679 0 653 24423 0 0

cpu6 130865 0 45303 53716622 519 0 640 24100 0 0

cpu7 133535 0 30828 53723068 579 0 608 23714 0 0

intr 272748080 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 28077877 39831 11522520 2859 0 22951 0 24746002 14937 2983809 2054 0 161765 0 26813434 16005 2767562 1846 0 25209 0 24671954 19463 2988846 2146 0 217642 0 25042135 14563 2423677 1848 0 180457 0 23608755 17002 2064988 1826 0 73600 0 23522679 16917 1829215 1860 0 33122 0 23579179 14941 1745610 2001 0 26129 0 630 92 746559 799 146337 13470905 0 0

ctxt 368987072

btime 1374167734

processes 1397567

procs_running 1

procs_blocked 0

softirq 114276654 2 49141504 6706699 13188855 0 0 13362 21725596 480999 23019637

<<<netctr>>>

1374708333

gre0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

ip6gre0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

sit0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

dummy0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

lo 1603831401 6369896 0 0 0 0 0 0 1603831401 6369896 0 0 0 0 0 0

eth0 842049487 7300178 0 0 0 0 0 0 6558431969 7014697 0 0 0 0 0 0

ip6tnl0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

tunl0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

teql0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

<<<vbox_guest>>>

<<<ntp>>>

- fairy.mattnordh 209.51.161.238 2 u 741 1024 377 19.208 0.816 0.735

+ barricade.rack9 209.51.161.238 2 u 879 1024 377 21.752 -1.653 1.499

+ ntp4.khresear.c 192.5.41.40 2 u 115 1024 377 0.937 -1.566 1.064

* lithium.constan 128.4.1.1 2 u 640 1024 377 2.462 -3.589 0.957

<<<drbd>>>

version: 8.4.2 (api:1/proto:86-101)

built-in

<<<local>>>

Valentin Bud

unread,

Jul 25, 2013, 2:00:23 AM7/25/13

to salt-...@googlegroups.com

Hello Dan,

What's the reasoning behind using the mine from pillar?

Thanks. Greetings,

--
You received this message because you are subscribed to the Google Groups "Salt-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Valentin Bud

http://databus.pro | vale...@databus.pro

Dan Garthwaite

unread,

Jul 25, 2013, 11:15:39 AM7/25/13

to salt-...@googlegroups.com

Great question! There was no reasoning, it was just the first way that worked for me. Where else could we put this? Like I said, the docs are unclear as to how to apply "options" to minions.

Could this be done with grains.setval?

Or I imagine using file.managed on /etc/salt/minion.d/check_mk.conf would be better on all the minions. Would the salt-minion process have to be restarted after that? This way it can be included with the check_mk state files.

Thursdays are busy for me, I may be able to investigate tomorrow.

-dan

--
You received this message because you are subscribed to a topic in the Google Groups "Salt-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/salt-users/N011ydFYU-A/unsubscribe.
To unsubscribe from this group and all its topics, send an email to salt-users+...@googlegroups.com.

Reply all

Reply to author

Forward

0 new messages