salt problems

skat...@gmail.com

unread,

Mar 18, 2016, 10:47:57 AM3/18/16

to security-onion

I deploy sensors and at first, salt work ok. Now, but rarely see salt minions up across all. Some days some work, different minions work on different day. Very random.

Please help, need good procedure for to troubleshoot and I need understand how to restart all salt minion and master and kill old processes.

My questions:
does salt restart every night?

what logs to post here to help?

Here is response to salt ping:
> sudo salt '*' test.ping
sensorA:
True
sensorB:
Minion did not return. [No response]
sensorC:
Minion did not return. [Not connected]
sensorD:
Minion did not return. [Not connected]

thank you much!

Wes

unread,

Mar 18, 2016, 12:08:00 PM3/18/16

to security-onion

Please attach the output of 'sudo sostat-redacted'.

Do you have any firewalls or network devices that could potentially be the cause of intermittent issues?

You could try taking a look here and following some of the suggested steps:
https://groups.google.com/d/msg/security-onion/3QSR7F2Gm6o/AOgyzYQQQOoJ

Thanks,
Wes

s adz

unread,

Mar 18, 2016, 2:02:02 PM3/18/16

to securit...@googlegroups.com

Only switches between sensors.

Thank you for link. I go look now.

s adz

unread,

Mar 18, 2016, 2:13:26 PM3/18/16

to securit...@googlegroups.com

I check and I see sensorA still only server respond to ping. others [not connected] or [no response]

I run: sudo service salt-minion restart

and then all machines [Not connected]

I run: sudo service salt-master restart and A-C respond.

I run: sudo service salt-minion restart on D and it respond.

All fix now. Thank you too much. I wonder but what happens to make happen this outage?

s adz

unread,

Mar 18, 2016, 6:48:45 PM3/18/16

to securit...@googlegroups.com

Now two minions down again.

restart master and all ping good.

here, latest error from var/logs/salt/minion:

Traceback (most recent call last):

File "/usr/bin/salt-call", line 11, in <module>

salt_call()

File "/usr/lib/python2.7/dist-packages/salt/scripts.py", line 227, in salt_call

client.run()

File "/usr/lib/python2.7/dist-packages/salt/cli/call.py", line 69, in run

caller.run()

File "/usr/lib/python2.7/dist-packages/salt/cli/caller.py", line 236, in run

ret = self.call()

File "/usr/lib/python2.7/dist-packages/salt/cli/caller.py", line 138, in call

ret['return'] = func(*args, **kwargs)

File "/usr/lib/python2.7/dist-packages/salt/modules/state.py", line 515, in highstate

st_ = salt.state.HighState(opts, pillar, kwargs.get('__pub_jid'))

File "/usr/lib/python2.7/dist-packages/salt/state.py", line 3026, in __init__

BaseHighState.__init__(self, opts)

File "/usr/lib/python2.7/dist-packages/salt/state.py", line 2203, in __init__

self.opts = self.__gen_opts(opts)

File "/usr/lib/python2.7/dist-packages/salt/state.py", line 2230, in __gen_opts

mopts = self.client.master_opts()

File "/usr/lib/python2.7/dist-packages/salt/fileclient.py", line 1210, in master_opts

return self.channel.send(load)

File "/usr/lib/python2.7/dist-packages/salt/transport/__init__.py", line 314, in send

return self._crypted_transfer(load, tries, timeout)

File "/usr/lib/python2.7/dist-packages/salt/transport/__init__.py", line 302, in _crypted_transfer

return _do_transfer()

File "/usr/lib/python2.7/dist-packages/salt/transport/__init__.py", line 293, in _do_transfer

timeout)

File "/usr/lib/python2.7/dist-packages/salt/payload.py", line 273, in send

'SaltReqTimeoutError: after {0} seconds, ran {1} tries'.format(timeout * tried, tried)

SaltReqTimeoutError: SaltReqTimeoutError: after 180 seconds, ran 3 tries

2016-03-18 22:45:01,974 [salt.config ][ERROR ][92162] Error parsing configuration file: /etc/salt/minion.d/onionsalt.conf - conf should be a document, not <type 'str'>.

Wes

unread,

Mar 21, 2016, 11:50:48 PM3/21/16

to security-onion

Are you still seeing this error? This could be do to malformed YAML:

Ref: http://stackoverflow.com/questions/23151731/salt-stack-error-parsing-configuration-file-etc-salt-master

Thanks,
Wes

skat...@gmail.com

unread,

Mar 24, 2016, 2:26:26 PM3/24/16

to security-onion

Yes, Wes, yaml at cause of error but now fixed and ping giving still random results.

Master not always respond to ping, then respond after 5 minutes I am waiting and try again.

In sostat I see "Processed: Cannot Lost: allocate RX_RING!" problem, maybe relate to master not respond to pings. Maybe server busy? I have much CPU . CPU, no problem. I no understand load problem.

now error in Master:
Unable to connect to the salt master publisher at /var/run/salt/master

and timeout error in minion log.
2016-03-24 18:20:12,938 [salt.payload ][INFO ][97367] SaltReqTimeoutError: after 60 seconds. (Try 5 of 7)

sudo sostat-redacted output:
=========================================================================
Service Status
=========================================================================
Status: securityonion
* SO-user server[ OK ]
Status: HIDS
* ossec_agent (SO-user)[ OK ]
Status: Bro
Getting process status ...
Getting peer status ...
Name Type Host Status Pid Peers Started
manager manager localhost running 9036 5 19 Mar 04:11:49
proxy proxy localhost running 9205 5 19 Mar 04:11:51
SO-server-eth4-1 worker localhost running 9779 2 19 Mar 04:11:52
SO-server-eth4-2 worker localhost running 9780 2 19 Mar 04:11:52
SO-server-eth4-3 worker localhost running 9778 2 19 Mar 04:11:52
SO-server-eth4-4 worker localhost running 9777 2 19 Mar 04:11:52
Status: SO-server-eth4
* netsniff-ng (full packet data)[ OK ]
* pcap_agent (SO-user)[ OK ]
* snort_agent-1 (SO-user)[ OK ]
* snort_agent-2 (SO-user)[ OK ]
* snort_agent-3 (SO-user)[ OK ]
* snort_agent-4 (SO-user)[ OK ]
* snort-1 (alert data)[ OK ]
* snort-2 (alert data)[ OK ]
* snort-3 (alert data)[ OK ]
* snort-4 (alert data)[ OK ]
* barnyard2-1 (spooler, unified2 format)[ OK ]
* barnyard2-2 (spooler, unified2 format)[ OK ]
* barnyard2-3 (spooler, unified2 format)[ OK ]
* barnyard2-4 (spooler, unified2 format)[ OK ]

=========================================================================
Interface Status
=========================================================================
eth3 Link encap:Ethernet HWaddr MM:MM:MM:MM:MM:MM
inet addr:X.X.X.X Bcast:X.X.X.X Mask:X.X.X.X
inet6 addr: X.X.X.X/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:2579704 errors:0 dropped:0 overruns:0 frame:0
TX packets:1887047 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:933761713 (933.7 MB) TX bytes:573208513 (573.2 MB)
Interrupt:103 Memory:92000000-927fffff

eth4 Link encap:Ethernet HWaddr MM:MM:MM:MM:MM:MM
UP BROADCAST RUNNING NOARP PROMISC MULTICAST MTU:1500 Metric:1
RX packets:7741912669 errors:0 dropped:0 overruns:0 frame:0
TX packets:2 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:6671011233946 (6.6 TB) TX bytes:188 (188.0 B)
Interrupt:114 Memory:c9000000-c97fffff

lo Link encap:Local Loopback
inet addr:X.X.X.X Mask:X.X.X.X
inet6 addr: X.X.X.X/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:21157703 errors:0 dropped:0 overruns:0 frame:0
TX packets:21157703 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:77400216414 (77.4 GB) TX bytes:77400216414 (77.4 GB)

=========================================================================
Link Statistics
=========================================================================
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default
link/loopback MM:MM:MM:MM:MM:MM brd MM:MM:MM:MM:MM:MM
RX: bytes packets errors dropped overrun mcast
77400216414 21157703 0 0 0 0
RX errors: length crc frame fifo missed
0 0 0 0 0
TX: bytes packets errors dropped carrier collsns
77400216414 21157703 0 0 0 0
TX errors: aborted fifo window heartbeat
0 0 0 0
2: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether MM:MM:MM:MM:MM:MM brd MM:MM:MM:MM:MM:MM
RX: bytes packets errors dropped overrun mcast
0 0 0 0 0 0
RX errors: length crc frame fifo missed
0 0 0 0 0
TX: bytes packets errors dropped carrier collsns
0 0 0 0 0 0
TX errors: aborted fifo window heartbeat
0 0 0 0
3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether MM:MM:MM:MM:MM:MM brd MM:MM:MM:MM:MM:MM
RX: bytes packets errors dropped overrun mcast
0 0 0 0 0 0
RX errors: length crc frame fifo missed
0 0 0 0 0
TX: bytes packets errors dropped carrier collsns
0 0 0 0 0 0
TX errors: aborted fifo window heartbeat
0 0 0 0
4: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether MM:MM:MM:MM:MM:MM brd MM:MM:MM:MM:MM:MM
RX: bytes packets errors dropped overrun mcast
0 0 0 0 0 0
RX errors: length crc frame fifo missed
0 0 0 0 0
TX: bytes packets errors dropped carrier collsns
0 0 0 0 0 0
TX errors: aborted fifo window heartbeat
0 0 0 0
5: eth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether MM:MM:MM:MM:MM:MM brd MM:MM:MM:MM:MM:MM
RX: bytes packets errors dropped overrun mcast
933761713 2579704 0 0 0 239623
RX errors: length crc frame fifo missed
0 0 0 0 0
TX: bytes packets errors dropped carrier collsns
573208513 1887047 0 0 0 0
TX errors: aborted fifo window heartbeat
0 0 0 0
6: eth4: <BROADCAST,MULTICAST,NOARP,PROMISC,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether MM:MM:MM:MM:MM:MM brd MM:MM:MM:MM:MM:MM
RX: bytes packets errors dropped overrun mcast
6671011233946 7741912669 0 0 0 898616188
RX errors: length crc frame fifo missed
0 0 0 0 0
TX: bytes packets errors dropped carrier collsns
188 2 0 0 0 0
TX errors: aborted fifo window heartbeat
0 0 0 0
7: eth5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether MM:MM:MM:MM:MM:MM brd MM:MM:MM:MM:MM:MM
RX: bytes packets errors dropped overrun mcast
0 0 0 0 0 0
RX errors: length crc frame fifo missed
0 0 0 0 0
TX: bytes packets errors dropped carrier collsns
0 0 0 0 0 0
TX errors: aborted fifo window heartbeat
0 0 0 0

=========================================================================
Disk Usage
=========================================================================
Filesystem Size Used Avail Use% Mounted on
udev 189G 4.0K 189G 1% /dev
tmpfs 38G 1.8M 38G 1% /run
/dev/mapper/securityonion--vg-root 8.3T 6.9T 1.1T 87% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
none 5.0M 0 5.0M 0% /run/lock
none 189G 16K 189G 1% /run/shm
none 100M 4.0K 100M 1% /run/user
/dev/sda2 237M 83M 142M 37% /boot

=========================================================================
Network Sockets
=========================================================================
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
syslog-ng 741 root 25u IPv4 20184993 0t0 TCP *:514 (LISTEN)
syslog-ng 741 root 26u IPv4 20184994 0t0 UDP *:514
avahi-dae 1186 avahi 12u IPv4 40009 0t0 UDP *:5353
avahi-dae 1186 avahi 13u IPv6 40010 0t0 UDP *:5353
avahi-dae 1186 avahi 14u IPv4 40011 0t0 UDP *:49899
avahi-dae 1186 avahi 15u IPv6 40012 0t0 UDP *:54603
sshd 1890 root 3u IPv4 353 0t0 TCP *:ssh_port (LISTEN)
sshd 1890 root 4u IPv6 355 0t0 TCP *:ssh_port (LISTEN)
cups-brow 1920 root 6u IPv6 38155 0t0 TCP [X.X.X.X]:52145->[X.X.X.X]:631 (CLOSE_WAIT)
cups-brow 1920 root 8u IPv4 48131 0t0 UDP *:631
searchd 1937 sphinxsearch 7u IPv4 1695 0t0 TCP *:9306 (LISTEN)
searchd 1937 sphinxsearch 8u IPv4 1696 0t0 TCP *:9312 (LISTEN)
mysqld 1943 mysql 10u IPv4 12485 0t0 TCP X.X.X.X:3306 (LISTEN)
ossec-csy 2017 ossecm 5u IPv4 40073 0t0 UDP X.X.X.X:34781->X.X.X.X:514
cupsd 7100 root 10u IPv6 23862 0t0 TCP [X.X.X.X]:631 (LISTEN)
cupsd 7100 root 11u IPv4 23863 0t0 TCP X.X.X.X:631 (LISTEN)
sshd 7135 root 3u IPv4 46236 0t0 TCP X.X.X.X:ssh_port->X.X.X.X:49762 (ESTABLISHED)
sshd 7140 root 3u IPv4 46240 0t0 TCP X.X.X.X:ssh_port->X.X.X.X:50857 (ESTABLISHED)
sshd 7212 SO-user 3u IPv4 46236 0t0 TCP X.X.X.X:ssh_port->X.X.X.X:49762 (ESTABLISHED)
sshd 7212 SO-user 9u IPv6 15627 0t0 TCP [X.X.X.X]:50002 (LISTEN)
sshd 7212 SO-user 10u IPv4 15628 0t0 TCP X.X.X.X:50002 (LISTEN)
sshd 7223 SO-user 3u IPv4 46240 0t0 TCP X.X.X.X:ssh_port->X.X.X.X:50857 (ESTABLISHED)
sshd 7223 SO-user 9u IPv6 17900 0t0 TCP [X.X.X.X]:50000 (LISTEN)
sshd 7223 SO-user 10u IPv4 17901 0t0 TCP X.X.X.X:50000 (LISTEN)
sshd 7240 root 3u IPv4 49159 0t0 TCP X.X.X.X:ssh_port->X.X.X.X:37008 (ESTABLISHED)
sshd 7286 SO-user 3u IPv4 49159 0t0 TCP X.X.X.X:ssh_port->X.X.X.X:37008 (ESTABLISHED)
sshd 7286 SO-user 9u IPv6 35092 0t0 TCP [X.X.X.X]:50001 (LISTEN)
sshd 7286 SO-user 10u IPv4 35093 0t0 TCP X.X.X.X:50001 (LISTEN)
ntpd 8345 ntp 16u IPv4 52477 0t0 UDP *:123
ntpd 8345 ntp 17u IPv6 52478 0t0 UDP *:123
ntpd 8345 ntp 18u IPv4 52484 0t0 UDP X.X.X.X:123
ntpd 8345 ntp 19u IPv4 52485 0t0 UDP X.X.X.X:123
ntpd 8345 ntp 20u IPv6 52486 0t0 UDP [X.X.X.X]:123
ntpd 8345 ntp 21u IPv6 52487 0t0 UDP [X.X.X.X]:123
/usr/sbin 8640 root 5u IPv6 40554 0t0 TCP *:443 (LISTEN)
/usr/sbin 8640 root 7u IPv6 40558 0t0 TCP *:9876 (LISTEN)
/usr/sbin 8640 root 9u IPv6 40564 0t0 TCP *:3154 (LISTEN)
bro 9036 SO-user 4u IPv4 53440 0t0 UDP X.X.X.X:48533->X.X.X.X:53
bro 9038 SO-user 0u IPv4 32109 0t0 TCP *:47761 (LISTEN)
bro 9038 SO-user 1u IPv6 32110 0t0 TCP *:47761 (LISTEN)
bro 9038 SO-user 2u IPv4 32118 0t0 TCP X.X.X.X:47761->X.X.X.X:56124 (ESTABLISHED)
bro 9038 SO-user 4u IPv4 53440 0t0 UDP X.X.X.X:48533->X.X.X.X:53
bro 9038 SO-user 268u IPv4 32152 0t0 TCP X.X.X.X:47761->X.X.X.X:56126 (ESTABLISHED)
bro 9038 SO-user 273u IPv4 44399 0t0 TCP X.X.X.X:47761->X.X.X.X:56128 (ESTABLISHED)
bro 9038 SO-user 278u IPv4 44402 0t0 TCP X.X.X.X:47761->X.X.X.X:56130 (ESTABLISHED)
bro 9038 SO-user 283u IPv4 44405 0t0 TCP X.X.X.X:47761->X.X.X.X:56132 (ESTABLISHED)
bro 9205 SO-user 4u IPv4 54372 0t0 UDP X.X.X.X:49571->X.X.X.X:53
bro 9207 SO-user 0u IPv4 53452 0t0 TCP X.X.X.X:56124->X.X.X.X:47761 (ESTABLISHED)
bro 9207 SO-user 4u IPv4 54372 0t0 UDP X.X.X.X:49571->X.X.X.X:53
bro 9207 SO-user 266u IPv4 53457 0t0 TCP *:47762 (LISTEN)
bro 9207 SO-user 267u IPv6 53458 0t0 TCP *:47762 (LISTEN)
bro 9207 SO-user 268u IPv4 32149 0t0 TCP X.X.X.X:47762->X.X.X.X:57260 (ESTABLISHED)
bro 9207 SO-user 273u IPv4 32155 0t0 TCP X.X.X.X:47762->X.X.X.X:57262 (ESTABLISHED)
bro 9207 SO-user 278u IPv4 32158 0t0 TCP X.X.X.X:47762->X.X.X.X:57264 (ESTABLISHED)
bro 9207 SO-user 283u IPv4 32161 0t0 TCP X.X.X.X:47762->X.X.X.X:57266 (ESTABLISHED)
bro 9777 SO-user 4u IPv4 36278 0t0 UDP X.X.X.X:60815->X.X.X.X:53
bro 9778 SO-user 4u IPv4 16295 0t0 UDP X.X.X.X:42017->X.X.X.X:53
bro 9779 SO-user 4u IPv4 43190 0t0 UDP X.X.X.X:48243->X.X.X.X:53
bro 9780 SO-user 4u IPv4 12899 0t0 UDP X.X.X.X:50531->X.X.X.X:53
bro 9781 SO-user 0u IPv4 36281 0t0 TCP X.X.X.X:57260->X.X.X.X:47762 (ESTABLISHED)
bro 9781 SO-user 4u IPv4 16295 0t0 UDP X.X.X.X:42017->X.X.X.X:53
bro 9781 SO-user 266u IPv4 36284 0t0 TCP X.X.X.X:56126->X.X.X.X:47761 (ESTABLISHED)
bro 9781 SO-user 271u IPv4 36289 0t0 TCP *:47765 (LISTEN)
bro 9781 SO-user 272u IPv6 36290 0t0 TCP *:47765 (LISTEN)
bro 9782 SO-user 0u IPv4 16303 0t0 TCP X.X.X.X:57262->X.X.X.X:47762 (ESTABLISHED)
bro 9782 SO-user 4u IPv4 12899 0t0 UDP X.X.X.X:50531->X.X.X.X:53
bro 9782 SO-user 266u IPv4 16306 0t0 TCP X.X.X.X:56128->X.X.X.X:47761 (ESTABLISHED)
bro 9782 SO-user 271u IPv4 16311 0t0 TCP *:47764 (LISTEN)
bro 9782 SO-user 272u IPv6 16312 0t0 TCP *:47764 (LISTEN)
bro 9787 SO-user 0u IPv4 40672 0t0 TCP X.X.X.X:57264->X.X.X.X:47762 (ESTABLISHED)
bro 9787 SO-user 4u IPv4 43190 0t0 UDP X.X.X.X:48243->X.X.X.X:53
bro 9787 SO-user 266u IPv4 40675 0t0 TCP X.X.X.X:56130->X.X.X.X:47761 (ESTABLISHED)
bro 9787 SO-user 271u IPv4 40680 0t0 TCP *:47763 (LISTEN)
bro 9787 SO-user 272u IPv6 40681 0t0 TCP *:47763 (LISTEN)
bro 9790 SO-user 0u IPv4 43194 0t0 TCP X.X.X.X:57266->X.X.X.X:47762 (ESTABLISHED)
bro 9790 SO-user 4u IPv4 36278 0t0 UDP X.X.X.X:60815->X.X.X.X:53
bro 9790 SO-user 266u IPv4 43197 0t0 TCP X.X.X.X:56132->X.X.X.X:47761 (ESTABLISHED)
bro 9790 SO-user 271u IPv4 43202 0t0 TCP *:47766 (LISTEN)
bro 9790 SO-user 272u IPv6 43203 0t0 TCP *:47766 (LISTEN)
tclsh 15297 SO-user 13u IPv4 21134880 0t0 TCP *:7734 (LISTEN)
tclsh 15297 SO-user 14u IPv6 21134881 0t0 TCP *:7734 (LISTEN)
tclsh 15297 SO-user 15u IPv4 21134884 0t0 TCP *:7736 (LISTEN)
tclsh 15297 SO-user 16u IPv6 21134885 0t0 TCP *:7736 (LISTEN)
tclsh 15297 SO-user 17u IPv4 23433722 0t0 TCP X.X.X.X:7736->X.X.X.X:44854 (ESTABLISHED)
tclsh 15297 SO-user 18u IPv4 23380791 0t0 TCP X.X.X.X:7736->X.X.X.X:48484 (ESTABLISHED)
tclsh 15297 SO-user 19u IPv4 23380792 0t0 TCP X.X.X.X:7736->X.X.X.X:46101 (ESTABLISHED)
tclsh 15297 SO-user 20u IPv4 23380793 0t0 TCP X.X.X.X:7736->X.X.X.X:34850 (ESTABLISHED)
tclsh 15297 SO-user 21u IPv4 23380794 0t0 TCP X.X.X.X:7736->X.X.X.X:44747 (ESTABLISHED)
tclsh 15297 SO-user 22u IPv4 23264245 0t0 TCP X.X.X.X:7736->X.X.X.X:47851 (ESTABLISHED)
tclsh 15297 SO-user 23u IPv4 23380795 0t0 TCP X.X.X.X:7736->X.X.X.X:42978 (ESTABLISHED)
tclsh 15297 SO-user 24u IPv4 23264248 0t0 TCP X.X.X.X:7736->X.X.X.X:44002 (ESTABLISHED)
tclsh 15297 SO-user 25u IPv4 23264249 0t0 TCP X.X.X.X:7736->X.X.X.X:55841 (ESTABLISHED)
tclsh 15297 SO-user 26u IPv4 23435818 0t0 TCP X.X.X.X:7736->X.X.X.X:45796 (ESTABLISHED)
tclsh 15297 SO-user 27u IPv4 23435819 0t0 TCP X.X.X.X:7736->X.X.X.X:34032 (ESTABLISHED)
tclsh 15297 SO-user 28u IPv4 23415797 0t0 TCP X.X.X.X:7736->X.X.X.X:47890 (ESTABLISHED)
tclsh 15297 SO-user 29u IPv4 23264255 0t0 TCP X.X.X.X:7736->X.X.X.X:57697 (ESTABLISHED)
tclsh 15297 SO-user 30u IPv4 23435820 0t0 TCP X.X.X.X:7736->X.X.X.X:34088 (ESTABLISHED)
tclsh 15297 SO-user 31u IPv4 23264180 0t0 TCP X.X.X.X:7736->X.X.X.X:50376 (ESTABLISHED)
tclsh 15297 SO-user 32u IPv4 23441453 0t0 TCP X.X.X.X:7736->X.X.X.X:44041 (ESTABLISHED)
tclsh 15297 SO-user 33u IPv4 23435821 0t0 TCP X.X.X.X:7736->X.X.X.X:52817 (ESTABLISHED)
tclsh 15297 SO-user 34u IPv4 23380752 0t0 TCP X.X.X.X:7736->X.X.X.X:39758 (ESTABLISHED)
tclsh 15297 SO-user 35u IPv4 23264177 0t0 TCP X.X.X.X:7736->X.X.X.X:33895 (ESTABLISHED)
tclsh 15297 SO-user 36u IPv4 23264181 0t0 TCP X.X.X.X:7736->X.X.X.X:52849 (ESTABLISHED)
tclsh 15297 SO-user 37u IPv4 23264182 0t0 TCP X.X.X.X:7736->X.X.X.X:45432 (ESTABLISHED)
tclsh 15297 SO-user 38u IPv4 23431082 0t0 TCP X.X.X.X:7736->X.X.X.X:38787 (ESTABLISHED)
tclsh 15297 SO-user 39u IPv4 23435822 0t0 TCP X.X.X.X:7736->X.X.X.X:35428 (ESTABLISHED)
tclsh 15297 SO-user 40u IPv4 23264256 0t0 TCP X.X.X.X:7736->X.X.X.X:57540 (ESTABLISHED)
tclsh 15297 SO-user 41u IPv4 21130559 0t0 TCP X.X.X.X:7736->X.X.X.X:56226 (ESTABLISHED)
tclsh 15297 SO-user 42u IPv4 23380796 0t0 TCP X.X.X.X:7736->X.X.X.X:56179 (ESTABLISHED)
tclsh 15297 SO-user 43u IPv4 23380797 0t0 TCP X.X.X.X:7736->X.X.X.X:47739 (ESTABLISHED)
tclsh 15297 SO-user 44u IPv4 23264183 0t0 TCP X.X.X.X:7736->X.X.X.X:49841 (ESTABLISHED)
tclsh 15297 SO-user 45u IPv4 23443458 0t0 TCP X.X.X.X:7736->X.X.X.X:46998 (ESTABLISHED)
tclsh 15297 SO-user 46u IPv4 23401080 0t0 TCP X.X.X.X:7736->X.X.X.X:60231 (ESTABLISHED)
tclsh 15297 SO-user 47u IPv4 23443459 0t0 TCP X.X.X.X:7736->X.X.X.X:55818 (ESTABLISHED)
tclsh 15297 SO-user 48u IPv4 23443460 0t0 TCP X.X.X.X:7736->X.X.X.X:59310 (ESTABLISHED)
tclsh 15297 SO-user 49u IPv4 23443461 0t0 TCP X.X.X.X:7736->X.X.X.X:35974 (ESTABLISHED)
tclsh 15297 SO-user 50u IPv4 23443462 0t0 TCP X.X.X.X:7736->X.X.X.X:42587 (ESTABLISHED)
tclsh 15297 SO-user 51u IPv4 23435823 0t0 TCP X.X.X.X:7736->X.X.X.X:53068 (ESTABLISHED)
tclsh 15297 SO-user 52u IPv4 23435824 0t0 TCP X.X.X.X:7736->X.X.X.X:46067 (ESTABLISHED)
tclsh 15297 SO-user 54u IPv4 23264149 0t0 TCP X.X.X.X:7736->X.X.X.X:46701 (ESTABLISHED)
tclsh 15297 SO-user 55u IPv4 23264172 0t0 TCP X.X.X.X:7736->X.X.X.X:34709 (ESTABLISHED)
tclsh 15297 SO-user 56u IPv4 23264173 0t0 TCP X.X.X.X:7736->X.X.X.X:57928 (ESTABLISHED)
tclsh 15297 SO-user 57u IPv4 23432203 0t0 TCP X.X.X.X:7736->X.X.X.X:53617 (ESTABLISHED)
tclsh 15297 SO-user 58u IPv4 23380790 0t0 TCP X.X.X.X:7736->X.X.X.X:38291 (ESTABLISHED)
/usr/sbin 16898 www-data 5u IPv6 40554 0t0 TCP *:443 (LISTEN)
/usr/sbin 16898 www-data 7u IPv6 40558 0t0 TCP *:9876 (LISTEN)
/usr/sbin 16898 www-data 9u IPv6 40564 0t0 TCP *:3154 (LISTEN)
/usr/sbin 17520 www-data 5u IPv6 40554 0t0 TCP *:443 (LISTEN)
/usr/sbin 17520 www-data 7u IPv6 40558 0t0 TCP *:9876 (LISTEN)
/usr/sbin 17520 www-data 9u IPv6 40564 0t0 TCP *:3154 (LISTEN)
/usr/sbin 17521 www-data 5u IPv6 40554 0t0 TCP *:443 (LISTEN)
/usr/sbin 17521 www-data 7u IPv6 40558 0t0 TCP *:9876 (LISTEN)
/usr/sbin 17521 www-data 9u IPv6 40564 0t0 TCP *:3154 (LISTEN)
/usr/sbin 37561 www-data 5u IPv6 40554 0t0 TCP *:443 (LISTEN)
/usr/sbin 37561 www-data 7u IPv6 40558 0t0 TCP *:9876 (LISTEN)
/usr/sbin 37561 www-data 9u IPv6 40564 0t0 TCP *:3154 (LISTEN)
sshd 40713 root 3u IPv4 24245128 0t0 TCP X.X.X.X:ssh_port->X.X.X.X:56727 (ESTABLISHED)
sshd 40788 root 3u IPv4 24196021 0t0 TCP X.X.X.X:ssh_port->X.X.X.X:56728 (ESTABLISHED)
sshd 41011 SO-user 3u IPv4 24245128 0t0 TCP X.X.X.X:ssh_port->X.X.X.X:56727 (ESTABLISHED)
sshd 41011 SO-user 9u IPv6 24196049 0t0 TCP [X.X.X.X]:6011 (LISTEN)
sshd 41011 SO-user 10u IPv4 24196050 0t0 TCP X.X.X.X:6011 (LISTEN)
sshd 41012 SO-user 3u IPv4 24196021 0t0 TCP X.X.X.X:ssh_port->X.X.X.X:56728 (ESTABLISHED)
sshd 41012 SO-user 9u IPv6 24245182 0t0 TCP [X.X.X.X]:6010 (LISTEN)
sshd 41012 SO-user 10u IPv4 24245183 0t0 TCP X.X.X.X:6010 (LISTEN)
sshd 41137 root 3u IPv4 24250056 0t0 TCP X.X.X.X:ssh_port->X.X.X.X:56729 (ESTABLISHED)
sshd 41175 SO-user 3u IPv4 24250056 0t0 TCP X.X.X.X:ssh_port->X.X.X.X:56729 (ESTABLISHED)
salt-mast 65272 root 12u IPv4 8576973 0t0 TCP *:4505 (LISTEN)
salt-mast 65272 root 14u IPv4 8567166 0t0 TCP X.X.X.X:4505->X.X.X.X:59873 (ESTABLISHED)
salt-mast 65272 root 15u IPv4 8567253 0t0 TCP X.X.X.X:4505->X.X.X.X:48083 (ESTABLISHED)
salt-mast 65272 root 16u IPv4 8567293 0t0 TCP X.X.X.X:4505->X.X.X.X:41597 (ESTABLISHED)
salt-mast 65272 root 17u IPv4 8585612 0t0 TCP X.X.X.X:4505->X.X.X.X:45050 (ESTABLISHED)
salt-mast 65325 root 42u IPv4 8544015 0t0 TCP *:4506 (LISTEN)
salt-mast 65325 root 56u IPv4 24078236 0t0 TCP X.X.X.X:4506->X.X.X.X:49612 (ESTABLISHED)
salt-mast 65325 root 57u IPv4 24539066 0t0 TCP X.X.X.X:4506->X.X.X.X:42350 (ESTABLISHED)
salt-mast 65325 root 61u IPv4 24399149 0t0 TCP X.X.X.X:4506->X.X.X.X:44326 (ESTABLISHED)
salt-mast 65325 root 62u IPv4 24586926 0t0 TCP X.X.X.X:4506->X.X.X.X:49747 (ESTABLISHED)
salt-mast 65325 root 64u IPv4 24571666 0t0 TCP X.X.X.X:4506->X.X.X.X:44395 (ESTABLISHED)
salt-mast 65325 root 67u IPv4 24569721 0t0 TCP X.X.X.X:4506->X.X.X.X:49750 (ESTABLISHED)
salt-mast 65325 root 68u IPv4 24569722 0t0 TCP X.X.X.X:4506->X.X.X.X:39160 (ESTABLISHED)
tclsh 68977 SO-user 3u IPv4 23419425 0t0 TCP X.X.X.X:53617->X.X.X.X:7736 (ESTABLISHED)
barnyard2 69993 SO-user 3u IPv4 23412341 0t0 TCP X.X.X.X:53532->X.X.X.X:8401 (ESTABLISHED)
barnyard2 70033 SO-user 3u IPv4 23404359 0t0 TCP X.X.X.X:41573->X.X.X.X:8402 (ESTABLISHED)
barnyard2 70077 SO-user 3u IPv4 23436550 0t0 TCP X.X.X.X:58018->X.X.X.X:8403 (ESTABLISHED)
barnyard2 70121 SO-user 3u IPv4 23404360 0t0 TCP X.X.X.X:57854->X.X.X.X:8404 (ESTABLISHED)
tclsh 70189 SO-user 3u IPv4 23434771 0t0 TCP X.X.X.X:60231->X.X.X.X:7736 (ESTABLISHED)
tclsh 70494 SO-user 3u IPv4 23338466 0t0 TCP X.X.X.X:55841->X.X.X.X:7736 (ESTABLISHED)
tclsh 70494 SO-user 4u IPv4 23418863 0t0 TCP X.X.X.X:8401 (LISTEN)
tclsh 70494 SO-user 6u IPv4 23400154 0t0 TCP X.X.X.X:8401->X.X.X.X:53532 (ESTABLISHED)
tclsh 70549 SO-user 3u IPv4 23437518 0t0 TCP X.X.X.X:44041->X.X.X.X:7736 (ESTABLISHED)
tclsh 70549 SO-user 4u IPv4 23442810 0t0 TCP X.X.X.X:8402 (LISTEN)
tclsh 70549 SO-user 6u IPv4 23425338 0t0 TCP X.X.X.X:8402->X.X.X.X:41573 (ESTABLISHED)
tclsh 70678 SO-user 3u IPv4 23383625 0t0 TCP X.X.X.X:53068->X.X.X.X:7736 (ESTABLISHED)
tclsh 70678 SO-user 4u IPv4 23432107 0t0 TCP X.X.X.X:8403 (LISTEN)
tclsh 70678 SO-user 6u IPv4 23338615 0t0 TCP X.X.X.X:8403->X.X.X.X:58018 (ESTABLISHED)
tclsh 70728 SO-user 3u IPv4 23441436 0t0 TCP X.X.X.X:46067->X.X.X.X:7736 (ESTABLISHED)
tclsh 70728 SO-user 4u IPv4 23432110 0t0 TCP X.X.X.X:8404 (LISTEN)
tclsh 70728 SO-user 6u IPv4 23425339 0t0 TCP X.X.X.X:8404->X.X.X.X:57854 (ESTABLISHED)
/usr/sbin 73176 www-data 5u IPv6 40554 0t0 TCP *:443 (LISTEN)
/usr/sbin 73176 www-data 7u IPv6 40558 0t0 TCP *:9876 (LISTEN)
/usr/sbin 73176 www-data 9u IPv6 40564 0t0 TCP *:3154 (LISTEN)
salt-mini 80120 root 24u IPv4 8638656 0t0 TCP X.X.X.X:45050->X.X.X.X:4505 (ESTABLISHED)
salt-mini 80120 root 26u IPv4 24374525 0t0 TCP X.X.X.X:44326->X.X.X.X:4506 (ESTABLISHED)
salt-call 84647 root 10u IPv4 24542150 0t0 TCP X.X.X.X:44395->X.X.X.X:4506 (ESTABLISHED)

=========================================================================
IDS Rules Update
=========================================================================
Thu Mar 24 07:01:04 UTC 2016
Backing up current local_rules.xml file.
Cleaning up local_rules.xml backup files older than 30 days.
Backing up current downloaded.rules file before it gets overwritten.
Cleaning up downloaded.rules backup files older than 30 days.
Backing up current local.rules file before it gets overwritten.
Cleaning up local.rules backup files older than 30 days.
Sleeping for 18 minutes to avoid overwhelming rule sites.
Running PulledPork.
http://code.google.com/p/pulledpork/
_____ ____
`----,\ )
`--==\\ / PulledPork v0.7.0 - Swine Flu!
`--==\\/
.-~~~~-.Y|\\_ Copyright (C) 2009-2013 JJ Cummings
@_/ / 66\_ cumm...@gmail.com
| \ \ _(")
\ /-| ||'--' Rules give me wings!
\_\ \_\\
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Checking latest MD5 for emerging.rules.tar.gz....
No Match
Done
Rules tarball download of emerging.rules.tar.gz....
They Match
Done!
Prepping rules from emerging.rules.tar.gz for work....
Done!
Reading rules...
Generating Stub Rules....
Done
Reading rules...
Reading rules...
Modifying Sids....
Done!
Processing /etc/nsm/pulledpork/enablesid.conf....
Modified 0 rules
Done
Processing /etc/nsm/pulledpork/dropsid.conf....
Modified 0 rules
Done
Processing /etc/nsm/pulledpork/disablesid.conf....
Modified 20 rules
Done
Setting Flowbit State....
Enabled 42 flowbits
Done
Writing /etc/nsm/rules/downloaded.rules....
Done
Generating sid-msg.map....
Done
Writing v1 /etc/nsm/rules/sid-msg.map....
Done
Writing /var/log/nsm/sid_changes.log....
Done
Rule Stats...
New:-------16
Deleted:---18
Enabled Rules:----19080
Dropped Rules:----0
Disabled Rules:---4240
Total Rules:------23320
No IP Blacklist Changes
Done
Please review /var/log/nsm/sid_changes.log for additional details
Fly Piggy Fly!
Restarting Barnyard2.
Restarting: SO-server-eth4
* stopping: barnyard2-1 (spooler, unified2 format)[ OK ]
* starting: barnyard2-1 (spooler, unified2 format)[ OK ]
* stopping: barnyard2-2 (spooler, unified2 format)[ OK ]
* starting: barnyard2-2 (spooler, unified2 format)[ OK ]
* stopping: barnyard2-3 (spooler, unified2 format)[ OK ]
* starting: barnyard2-3 (spooler, unified2 format)[ OK ]
* stopping: barnyard2-4 (spooler, unified2 format)[ OK ]
* starting: barnyard2-4 (spooler, unified2 format)[ OK ]
Restarting IDS Engine.
Restarting: SO-server-eth4
* stopping: snort-1 (alert data)[ OK ]
* starting: snort-1 (alert data)[ OK ]
* stopping: snort-2 (alert data)[ OK ]
* starting: snort-2 (alert data)[ OK ]
* stopping: snort-3 (alert data)[ OK ]
* starting: snort-3 (alert data)[ OK ]
* stopping: snort-4 (alert data)[ OK ]
* starting: snort-4 (alert data)[ OK ]

=========================================================================
CPU Usage
=========================================================================
Load average for the last 1, 5, and 15 minutes:
14.57 13.92 13.88
Processing units: 32
If load average is higher than processing units,
then tune until load average is lower than processing units.

top - 17:50:58 up 5 days, 13:40, 3 users, load average: 14.57, 13.92, 13.88
Tasks: 513 total, 15 running, 498 sleeping, 0 stopped, 0 zombie
%Cpu(s): 10.8 us, 12.4 sy, 0.0 ni, 75.8 id, 0.8 wa, 0.0 hi, 0.2 si, 0.0 st
KiB Mem: 39616294+total, 39524723+used, 915712 free, 40840 buffers
KiB Swap: 40254668+total, 28736348+used, 11518318+free. 1777940 cached Mem

%CPU %MEM COMMAND
99.4 13.3 /usr/bin/python /usr/bin/salt-master
98.5 12.7 /usr/bin/python /usr/bin/salt-master
95.7 7.7 /usr/bin/python /usr/bin/salt-master
95.0 0.0 /usr/bin/indexer --config /etc/sphinxsearch/sphinx.conf --rotate temp_2
81.3 0.2 snort -c /etc/nsm/SO-server-eth4/snort.conf -u SO-user -g SO-user -i eth4 -l /nsm/sensor_data/SO-server-eth4/snort-4 --perfmon-file /nsm/sensor_data/SO-server-eth4/snort-4.stats -U
78.4 1.7 /usr/bin/python /usr/bin/salt-master
65.1 13.2 /usr/bin/python /usr/bin/salt-master
57.8 12.8 /usr/bin/python /usr/bin/salt-master
56.2 16.8 /usr/bin/python /usr/bin/salt-master
50.6 0.2 snort -c /etc/nsm/SO-server-eth4/snort.conf -u SO-user -g SO-user -i eth4 -l /nsm/sensor_data/SO-server-eth4/snort-3 --perfmon-file /nsm/sensor_data/SO-server-eth4/snort-3.stats -U
42.1 16.6 /usr/bin/python /usr/bin/salt-master
33.2 0.2 snort -c /etc/nsm/SO-server-eth4/snort.conf -u SO-user -g SO-user -i eth4 -l /nsm/sensor_data/SO-server-eth4/snort-1 --perfmon-file /nsm/sensor_data/SO-server-eth4/snort-1.stats -U
30.6 0.2 snort -c /etc/nsm/SO-server-eth4/snort.conf -u SO-user -g SO-user -i eth4 -l /nsm/sensor_data/SO-server-eth4/snort-2 --perfmon-file /nsm/sensor_data/SO-server-eth4/snort-2.stats -U
15.6 0.3 /opt/bro/bin/bro -i eth4 -U .status -p broctl -p broctl-live -p local -p SO-server-eth4-4 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto
15.2 0.3 /opt/bro/bin/bro -i eth4 -U .status -p broctl -p broctl-live -p local -p SO-server-eth4-2 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto
15.0 0.3 /opt/bro/bin/bro -i eth4 -U .status -p broctl -p broctl-live -p local -p SO-server-eth4-3 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto
14.9 0.3 /opt/bro/bin/bro -i eth4 -U .status -p broctl -p broctl-live -p local -p SO-server-eth4-1 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto
12.1 0.0 /usr/sbin/mysqld
11.7 0.0 [kswapd1]
11.2 0.4 netsniff-ng -i eth4 -o /nsm/sensor_data/SO-server-eth4/dailylogs/2016-03-24/ --user 1001 --group 1001 -s --prefix snort.log. --verbose --ring-size 1600 iB --interval 150 iB --mmap
8.4 0.0 [kswapd0]
2.9 0.0 /opt/bro/bin/bro -U .status -p broctl -p broctl-live -p local -p manager local.bro broctl base/frameworks/cluster local-manager.bro broctl/auto
2.7 0.0 [kworker/u290:0]
2.3 0.0 perl /opt/elsa/web/cron.pl -c /etc/elsa_web.conf
2.2 0.0 [kworker/u290:2]
2.1 0.0 perl /opt/elsa/node/elsa.pl -c /etc/elsa_node.conf
1.9 0.2 /usr/bin/searchd --nodetach
1.8 0.0 /usr/sbin/syslog-ng -p /var/run/syslog-ng.pid
1.5 0.0 /opt/bro/bin/bro -U .status -p broctl -p broctl-live -p local -p proxy local.bro broctl base/frameworks/cluster local-proxy broctl/auto
0.4 0.0 /usr/bin/python /usr/bin/salt-master
0.4 0.0 [kworker/u289:0]
0.4 0.0 [kworker/u289:1]
0.3 0.0 [rcu_sched]
0.3 0.0 [khugepaged]
0.2 0.0 [jbd2/dm-0-8]
0.2 0.0 tclsh /usr/bin/SO-userd -c /etc/nsm/securityonion/SO-userd.conf -a /etc/nsm/securityonion/autocat.conf -g /etc/nsm/securityonion/SO-userd.queries -A /etc/nsm/securityonion/SO-userd.access -C /etc/nsm/securityonion/certs
0.1 0.0 [rcuos/12]
0.1 0.0 [rcuos/21]
0.1 0.0 /var/ossec/bin/ossec-syscheckd
0.1 0.0 /opt/bro/bin/bro -U .status -p broctl -p broctl-live -p local -p manager local.bro broctl base/frameworks/cluster local-manager.bro broctl/auto
0.1 0.0 [kworker/u290:1]
0.1 0.0 /usr/bin/python /usr/bin/salt-call state.highstate
0.1 0.0 sudo sostat-redacted
0.1 0.0 sed -r s/(([0-9a-fA-F]{1,4}:){7,7}[0-9a-fA-F]{1,4}|([0-9a-fA-F]{1,4}:){1,7}:|([0-9a-fA-F]{1,4}:){1,6}:[0-9a-fA-F]{1,4}|([0-9a-fA-F]{1,4}:){1,5}(:[0-9a-fA-F]{1,4}){1,2}|([0-9a-fA-F]{1,4}:){1,4}(:[0-9a-fA-F]{1,4}){1,3}|([0-9a-fA-F]{1,4}:){1,3}(:[0-9a-fA-F]{1,4}){1,4}|([0-9a-fA-F]{1,4}:){1,2}(:[0-9a-fA-F]{1,4}){1,5}|[0-9a-fA-F]{1,4}:((:[0-9a-fA-F]{1,4}){1,6})|:((:[0-9a-fA-F]{1,4}){1,7}|:)|fe80:(:[0-9a-fA-F]{0,4}){0,4}%[0-9a-zA-Z]{1,}|::(ffff(:0{1,4}){0,1}:){0,1}((25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9]).){3,3}(25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])|([0-9a-fA-F]{1,4}:){1,4}:((25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9]).){3,3}(25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9]))\b/X.X.X.X/g
0.0 0.0 /sbin/init
0.0 0.0 [kthreadd]
0.0 0.0 [ksoftirqd/0]
0.0 0.0 [kworker/0:0H]
0.0 0.0 [rcu_bh]
0.0 0.0 [rcuos/0]
0.0 0.0 [rcuob/0]
0.0 0.0 [migration/0]
0.0 0.0 [watchdog/0]
0.0 0.0 [watchdog/1]
0.0 0.0 [migration/1]
0.0 0.0 [ksoftirqd/1]
0.0 0.0 [kworker/1:0H]
0.0 0.0 [rcuos/1]
0.0 0.0 [rcuob/1]
0.0 0.0 [watchdog/2]
0.0 0.0 [migration/2]
0.0 0.0 [ksoftirqd/2]
0.0 0.0 [kworker/2:0H]
0.0 0.0 [rcuos/2]
0.0 0.0 [rcuob/2]
0.0 0.0 [watchdog/3]
0.0 0.0 [migration/3]
0.0 0.0 [ksoftirqd/3]
0.0 0.0 [kworker/3:0H]
0.0 0.0 [rcuos/3]
0.0 0.0 [rcuob/3]
0.0 0.0 [watchdog/4]
0.0 0.0 [migration/4]
0.0 0.0 [ksoftirqd/4]
0.0 0.0 [kworker/4:0H]
0.0 0.0 [rcuos/4]
0.0 0.0 [rcuob/4]
0.0 0.0 [watchdog/5]
0.0 0.0 [migration/5]
0.0 0.0 [ksoftirqd/5]
0.0 0.0 [kworker/5:0H]
0.0 0.0 [rcuos/5]
0.0 0.0 [rcuob/5]
0.0 0.0 [watchdog/6]
0.0 0.0 [migration/6]
0.0 0.0 [ksoftirqd/6]
0.0 0.0 [kworker/6:0H]
0.0 0.0 [rcuos/6]
0.0 0.0 [rcuob/6]
0.0 0.0 [watchdog/7]
0.0 0.0 [migration/7]
0.0 0.0 [ksoftirqd/7]
0.0 0.0 [kworker/7:0H]
0.0 0.0 [rcuos/7]
0.0 0.0 [rcuob/7]
0.0 0.0 [watchdog/8]
0.0 0.0 [migration/8]
0.0 0.0 [ksoftirqd/8]
0.0 0.0 [kworker/8:0H]
0.0 0.0 [rcuos/8]
0.0 0.0 [rcuob/8]
0.0 0.0 [watchdog/9]
0.0 0.0 [migration/9]
0.0 0.0 [ksoftirqd/9]
0.0 0.0 [kworker/9:0H]
0.0 0.0 [rcuos/9]
0.0 0.0 [rcuob/9]
0.0 0.0 [watchdog/10]
0.0 0.0 [migration/10]
0.0 0.0 [ksoftirqd/10]
0.0 0.0 [kworker/10:0H]
0.0 0.0 [rcuos/10]
0.0 0.0 [rcuob/10]
0.0 0.0 [watchdog/11]
0.0 0.0 [migration/11]
0.0 0.0 [ksoftirqd/11]
0.0 0.0 [kworker/11:0H]
0.0 0.0 [rcuos/11]
0.0 0.0 [rcuob/11]
0.0 0.0 [watchdog/12]
0.0 0.0 [migration/12]
0.0 0.0 [ksoftirqd/12]
0.0 0.0 [kworker/12:0H]
0.0 0.0 [rcuob/12]
0.0 0.0 [watchdog/13]
0.0 0.0 [migration/13]
0.0 0.0 [ksoftirqd/13]
0.0 0.0 [kworker/13:0H]
0.0 0.0 [rcuos/13]
0.0 0.0 [rcuob/13]
0.0 0.0 [watchdog/14]
0.0 0.0 [migration/14]
0.0 0.0 [ksoftirqd/14]
0.0 0.0 [kworker/14:0H]
0.0 0.0 [rcuos/14]
0.0 0.0 [rcuob/14]
0.0 0.0 [watchdog/15]
0.0 0.0 [migration/15]
0.0 0.0 [ksoftirqd/15]
0.0 0.0 [kworker/15:0H]
0.0 0.0 [rcuos/15]
0.0 0.0 [rcuob/15]
0.0 0.0 [watchdog/16]
0.0 0.0 [migration/16]
0.0 0.0 [ksoftirqd/16]
0.0 0.0 [kworker/16:0H]
0.0 0.0 [rcuos/16]
0.0 0.0 [rcuob/16]
0.0 0.0 [watchdog/17]
0.0 0.0 [migration/17]
0.0 0.0 [ksoftirqd/17]
0.0 0.0 [kworker/17:0H]
0.0 0.0 [rcuos/17]
0.0 0.0 [rcuob/17]
0.0 0.0 [watchdog/18]
0.0 0.0 [migration/18]
0.0 0.0 [ksoftirqd/18]
0.0 0.0 [kworker/18:0H]
0.0 0.0 [rcuos/18]
0.0 0.0 [rcuob/18]
0.0 0.0 [watchdog/19]
0.0 0.0 [migration/19]
0.0 0.0 [ksoftirqd/19]
0.0 0.0 [kworker/19:0H]
0.0 0.0 [rcuos/19]
0.0 0.0 [rcuob/19]
0.0 0.0 [watchdog/20]
0.0 0.0 [migration/20]
0.0 0.0 [ksoftirqd/20]
0.0 0.0 [kworker/20:0H]
0.0 0.0 [rcuos/20]
0.0 0.0 [rcuob/20]
0.0 0.0 [watchdog/21]
0.0 0.0 [migration/21]
0.0 0.0 [ksoftirqd/21]
0.0 0.0 [kworker/21:0H]
0.0 0.0 [rcuob/21]
0.0 0.0 [watchdog/22]
0.0 0.0 [migration/22]
0.0 0.0 [ksoftirqd/22]
0.0 0.0 [kworker/22:0H]
0.0 0.0 [rcuos/22]
0.0 0.0 [rcuob/22]
0.0 0.0 [watchdog/23]
0.0 0.0 [migration/23]
0.0 0.0 [ksoftirqd/23]
0.0 0.0 [kworker/23:0H]
0.0 0.0 [rcuos/23]
0.0 0.0 [rcuob/23]
0.0 0.0 [watchdog/24]
0.0 0.0 [migration/24]
0.0 0.0 [ksoftirqd/24]
0.0 0.0 [kworker/24:0H]
0.0 0.0 [rcuos/24]
0.0 0.0 [rcuob/24]
0.0 0.0 [watchdog/25]
0.0 0.0 [migration/25]
0.0 0.0 [ksoftirqd/25]
0.0 0.0 [kworker/25:0H]
0.0 0.0 [rcuos/25]
0.0 0.0 [rcuob/25]
0.0 0.0 [watchdog/26]
0.0 0.0 [migration/26]
0.0 0.0 [ksoftirqd/26]
0.0 0.0 [kworker/26:0H]
0.0 0.0 [rcuos/26]
0.0 0.0 [rcuob/26]
0.0 0.0 [watchdog/27]
0.0 0.0 [migration/27]
0.0 0.0 [ksoftirqd/27]
0.0 0.0 [kworker/27:0H]
0.0 0.0 [rcuos/27]
0.0 0.0 [rcuob/27]
0.0 0.0 [watchdog/28]
0.0 0.0 [migration/28]
0.0 0.0 [ksoftirqd/28]
0.0 0.0 [kworker/28:0H]
0.0 0.0 [rcuos/28]
0.0 0.0 [rcuob/28]
0.0 0.0 [watchdog/29]
0.0 0.0 [migration/29]
0.0 0.0 [ksoftirqd/29]
0.0 0.0 [kworker/29:0H]
0.0 0.0 [rcuos/29]
0.0 0.0 [rcuob/29]
0.0 0.0 [watchdog/30]
0.0 0.0 [migration/30]
0.0 0.0 [ksoftirqd/30]
0.0 0.0 [kworker/30:0H]
0.0 0.0 [rcuos/30]
0.0 0.0 [rcuob/30]
0.0 0.0 [watchdog/31]
0.0 0.0 [migration/31]
0.0 0.0 [ksoftirqd/31]
0.0 0.0 [kworker/31:0H]
0.0 0.0 [rcuos/31]
0.0 0.0 [rcuob/31]
0.0 0.0 [khelper]
0.0 0.0 [kdevtmpfs]
0.0 0.0 [netns]
0.0 0.0 [perf]
0.0 0.0 [khungtaskd]
0.0 0.0 [writeback]
0.0 0.0 [ksmd]
0.0 0.0 [crypto]
0.0 0.0 [kintegrityd]
0.0 0.0 [bioset]
0.0 0.0 [kblockd]
0.0 0.0 [ata_sff]
0.0 0.0 [md]
0.0 0.0 [devfreq_wq]
0.0 0.0 [vmstat]
0.0 0.0 [fsnotify_mark]
0.0 0.0 [ecryptfs-kthrea]
0.0 0.0 [kthrotld]
0.0 0.0 [acpi_thermal_pm]
0.0 0.0 [ipv6_addrconf]
0.0 0.0 [deferwq]
0.0 0.0 [charger_manager]
0.0 0.0 [scsi_eh_0]
0.0 0.0 [scsi_tmf_0]
0.0 0.0 [bnx2x]
0.0 0.0 [bnx2x_iov]
0.0 0.0 [kdmflush]
0.0 0.0 [bioset]
0.0 0.0 [kdmflush]
0.0 0.0 [bioset]
0.0 0.0 [scsi_eh_1]
0.0 0.0 [scsi_tmf_1]
0.0 0.0 [usb-storage]
0.0 0.0 [bioset]
0.0 0.0 [ext4-rsv-conver]
0.0 0.0 upstart-udev-bridge --daemon
0.0 0.0 /lib/systemd/systemd-udevd --daemon
0.0 0.0 supervising syslog-ng
0.0 0.0 [ext4-rsv-conver]
0.0 0.0 upstart-file-bridge --daemon
0.0 0.0 [edac-poller]
0.0 0.0 [kvm-irqfd-clean]
0.0 0.0 [kmpathd]
0.0 0.0 [kmpath_handlerd]
0.0 0.0 dbus-daemon --system --fork
0.0 0.0 /lib/systemd/systemd-logind
0.0 0.0 /usr/sbin/bluetoothd
0.0 0.0 avahi-daemon: running [SO-server.local]
0.0 0.0 avahi-daemon: chroot helper
0.0 0.0 [krfcommd]
0.0 0.0 upstart-socket-bridge --daemon
0.0 0.0 /sbin/getty -8 38400 tty4
0.0 0.0 /sbin/getty -8 38400 tty5
0.0 0.0 /usr/bin/python /usr/bin/salt-minion
0.0 0.0 /sbin/getty -8 38400 tty2
0.0 0.0 /sbin/getty -8 38400 tty3
0.0 0.0 /sbin/getty -8 38400 tty6
0.0 0.0 /usr/sbin/irqbalance
0.0 0.0 /usr/sbin/sshd -D
0.0 0.0 su -s /bin/sh -c exec "$0" "$@" sphinxsearch -- /usr/bin/searchd --nodetach
0.0 0.0 cron
0.0 0.0 acpid -c /etc/acpi/events -s /var/run/acpid.socket
0.0 0.0 [kauditd]
0.0 0.0 /usr/sbin/cups-browsed
0.0 0.0 /usr/sbin/kerneloops
0.0 0.0 /var/ossec/bin/ossec-csyslogd
0.0 0.0 /var/ossec/bin/ossec-execd
0.0 0.0 /var/ossec/bin/ossec-analysisd
0.0 0.0 /var/ossec/bin/ossec-logcollector
0.0 0.0 lightdm
0.0 0.0 /usr/bin/X -core :0 -seat seat0 -auth /var/run/lightdm/root/:0 -nolisten tcp vt7 -novtswitch
0.0 0.0 /usr/lib/accountsservice/accounts-daemon
0.0 0.0 /usr/lib/policykit-1/polkitd --no-debug
0.0 0.0 /var/ossec/bin/ossec-monitord
0.0 0.0 lightdm --session-child 16 19
0.0 0.0 /bin/sh /usr/lib/lightdm/lightdm-greeter-session /usr/sbin/lightdm-gtk-greeter
0.0 0.0 //bin/dbus-daemon --fork --print-pid 5 --print-address 7 --session
0.0 0.0 /usr/sbin/lightdm-gtk-greeter
0.0 0.0 /usr/lib/at-spi2-core/at-spi-bus-launcher
0.0 0.0 /bin/dbus-daemon --config-file=/etc/at-spi2/accessibility.conf --nofork --print-address 3
0.0 0.0 /usr/lib/at-spi2-core/at-spi2-registryd --use-gnome-session
0.0 0.0 /usr/lib/gvfs/gvfsd
0.0 0.0 /usr/lib/gvfs/gvfsd-fuse /run/user/112/gvfs -f -o big_writes
0.0 0.0 lightdm --session-child 12 19
0.0 0.0 [kworker/12:1]
0.0 0.0 [kworker/24:0]
0.0 0.0 /usr/sbin/cupsd -f
0.0 0.0 sshd: SO-user [priv]
0.0 0.0 sshd: SO-user [priv]
0.0 0.0 sshd: SO-user
0.0 0.0 sshd: SO-user
0.0 0.0 sshd: SO-user [priv]
0.0 0.0 sshd: SO-user
0.0 0.0 [kworker/27:1]
0.0 0.0 /usr/sbin/ntpd -p /var/run/ntpd.pid -g -u 117:126
0.0 0.0 /usr/sbin/apache2 -k start
0.0 0.0 /sbin/getty -8 38400 tty1
0.0 0.0 /bin/bash /opt/bro/share/broctl/scripts/run-bro -1 -U .status -p broctl -p broctl-live -p local -p manager local.bro broctl base/frameworks/cluster local-manager.bro broctl/auto
0.0 0.0 /bin/bash /opt/bro/share/broctl/scripts/run-bro -1 -U .status -p broctl -p broctl-live -p local -p proxy local.bro broctl base/frameworks/cluster local-proxy broctl/auto
0.0 0.0 /opt/bro/bin/bro -U .status -p broctl -p broctl-live -p local -p proxy local.bro broctl base/frameworks/cluster local-proxy broctl/auto
0.0 0.0 [kworker/29:0]
0.0 0.0 /bin/bash /opt/bro/share/broctl/scripts/run-bro -1 -i eth4 -U .status -p broctl -p broctl-live -p local -p SO-server-eth4-1 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto
0.0 0.0 /bin/bash /opt/bro/share/broctl/scripts/run-bro -1 -i eth4 -U .status -p broctl -p broctl-live -p local -p SO-server-eth4-2 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto
0.0 0.0 /bin/bash /opt/bro/share/broctl/scripts/run-bro -1 -i eth4 -U .status -p broctl -p broctl-live -p local -p SO-server-eth4-3 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto
0.0 0.0 /bin/bash /opt/bro/share/broctl/scripts/run-bro -1 -i eth4 -U .status -p broctl -p broctl-live -p local -p SO-server-eth4-4 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto
0.0 0.2 /opt/bro/bin/bro -i eth4 -U .status -p broctl -p broctl-live -p local -p SO-server-eth4-3 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto
0.0 0.2 /opt/bro/bin/bro -i eth4 -U .status -p broctl -p broctl-live -p local -p SO-server-eth4-2 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto
0.0 0.2 /opt/bro/bin/bro -i eth4 -U .status -p broctl -p broctl-live -p local -p SO-server-eth4-1 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto
0.0 0.2 /opt/bro/bin/bro -i eth4 -U .status -p broctl -p broctl-live -p local -p SO-server-eth4-4 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto
0.0 0.0 [kworker/0:1H]
0.0 0.0 [kworker/8:1H]
0.0 0.0 [kworker/10:1H]
0.0 0.0 [kworker/14:1H]
0.0 0.0 [kworker/24:1H]
0.0 0.0 [kworker/15:0]
0.0 0.0 [kworker/19:1H]
0.0 0.0 [kworker/12:1H]
0.0 0.0 [kworker/8:0]
0.0 0.0 [kworker/2:1H]
0.0 0.0 [kworker/4:1H]
0.0 0.0 su - SO-user -- /usr/bin/SO-userd -c /etc/nsm/securityonion/SO-userd.conf -a /etc/nsm/securityonion/autocat.conf -g /etc/nsm/securityonion/SO-userd.queries -A /etc/nsm/securityonion/SO-userd.access -C /etc/nsm/securityonion/certs
0.0 0.0 tclsh /usr/bin/SO-userd -c /etc/nsm/securityonion/SO-userd.conf -a /etc/nsm/securityonion/autocat.conf -g /etc/nsm/securityonion/SO-userd.queries -A /etc/nsm/securityonion/SO-userd.access -C /etc/nsm/securityonion/certs
0.0 0.0 tclsh /usr/bin/SO-userd -c /etc/nsm/securityonion/SO-userd.conf -a /etc/nsm/securityonion/autocat.conf -g /etc/nsm/securityonion/SO-userd.queries -A /etc/nsm/securityonion/SO-userd.access -C /etc/nsm/securityonion/certs
0.0 0.0 /usr/sbin/apache2 -k start
0.0 0.0 [kworker/6:1H]
0.0 0.0 [kworker/9:0]
0.0 0.0 /usr/sbin/apache2 -k start
0.0 0.0 /usr/sbin/apache2 -k start
0.0 0.0 [kworker/20:1H]
0.0 0.0 [kworker/18:1H]
0.0 0.0 [kworker/24:1]
0.0 0.0 [kworker/8:1]
0.0 0.0 [kworker/9:2]
0.0 0.0 [kworker/30:1H]
0.0 0.0 [kworker/27:2]
0.0 0.0 [kworker/28:1H]
0.0 0.0 [kworker/1:1H]
0.0 0.0 /usr/sbin/apache2 -k start
0.0 0.0 [kworker/18:2]
0.0 0.0 [kworker/5:1H]
0.0 0.0 [kworker/17:1H]
0.0 0.0 [kworker/3:2]
0.0 0.0 sshd: SO-user [priv]
0.0 0.0 sshd: SO-user [priv]
0.0 0.0 sshd: SO-user@pts/4
0.0 0.0 sshd: SO-user@pts/2
0.0 0.0 -bash
0.0 0.0 -bash
0.0 0.0 sshd: SO-user [priv]
0.0 0.0 sshd: SO-user@pts/5
0.0 0.0 -bash
0.0 0.0 [kworker/5:0]
0.0 0.0 [kworker/26:1H]
0.0 0.0 [kworker/11:1H]
0.0 0.0 [kworker/21:1H]
0.0 0.0 [kworker/9:1H]
0.0 0.0 [kworker/20:0]
0.0 0.0 [kworker/23:1H]
0.0 0.0 [kworker/13:1H]
0.0 0.0 [kworker/30:2]
0.0 0.0 [kworker/1:1]
0.0 0.0 [kworker/15:1H]
0.0 0.0 [kworker/31:0]
0.0 0.0 [kworker/28:2]
0.0 0.0 [kworker/3:1H]
0.0 0.0 /usr/bin/python /usr/bin/salt-master
0.0 0.0 [kworker/23:2]
0.0 0.0 [kworker/14:1]
0.0 0.0 [kworker/22:1H]
0.0 0.0 [kworker/21:2]
0.0 0.0 [kworker/6:0]
0.0 0.0 [kworker/28:0]
0.0 0.0 /usr/bin/python /usr/bin/salt-master
0.0 0.0 [kworker/25:1H]
0.0 0.0 [kworker/4:2]
0.0 0.0 /usr/bin/python /usr/bin/salt-master
0.0 0.0 /usr/bin/python /usr/bin/salt-master
0.0 0.0 /usr/bin/python /usr/bin/salt-master
0.0 0.0 /usr/bin/python /usr/bin/salt-master
0.0 0.0 /usr/bin/python /usr/bin/salt-master
0.0 0.0 /usr/bin/python /usr/bin/salt-master
0.0 0.0 /usr/bin/python /usr/bin/salt-master
0.0 0.0 /usr/bin/python /usr/bin/salt-master
0.0 0.0 /usr/bin/python /usr/bin/salt-master
0.0 0.0 [kworker/10:2]
0.0 0.0 [kworker/27:1H]
0.0 0.0 [kworker/16:0]
0.0 0.0 su - SO-user -- /usr/bin/ossec_agent.tcl -o -f /var/ossec/logs/alerts/alerts.log -i X.X.X.X -p 5 -c /etc/nsm/ossec/ossec_agent.conf
0.0 0.0 tclsh /usr/bin/ossec_agent.tcl -o -f /var/ossec/logs/alerts/alerts.log -i X.X.X.X -p 5 -c /etc/nsm/ossec/ossec_agent.conf
0.0 0.0 tail -n 0 -F /var/ossec/logs/alerts/alerts.log
0.0 0.0 barnyard2 -c /etc/nsm/SO-server-eth4/barnyard2-1.conf -u SO-user -g SO-user -d /nsm/sensor_data/SO-server-eth4/snort-1 -f snort.unified2 -w /etc/nsm/SO-server-eth4/barnyard2.waldo-1 -i 1 -U
0.0 0.0 barnyard2 -c /etc/nsm/SO-server-eth4/barnyard2-2.conf -u SO-user -g SO-user -d /nsm/sensor_data/SO-server-eth4/snort-2 -f snort.unified2 -w /etc/nsm/SO-server-eth4/barnyard2.waldo-2 -i 2 -U
0.0 0.0 barnyard2 -c /etc/nsm/SO-server-eth4/barnyard2-3.conf -u SO-user -g SO-user -d /nsm/sensor_data/SO-server-eth4/snort-3 -f snort.unified2 -w /etc/nsm/SO-server-eth4/barnyard2.waldo-3 -i 3 -U
0.0 0.0 barnyard2 -c /etc/nsm/SO-server-eth4/barnyard2-4.conf -u SO-user -g SO-user -d /nsm/sensor_data/SO-server-eth4/snort-4 -f snort.unified2 -w /etc/nsm/SO-server-eth4/barnyard2.waldo-4 -i 4 -U
0.0 0.0 su - SO-user -- /usr/bin/pcap_agent.tcl -c /etc/nsm/SO-server-eth4/pcap_agent.conf
0.0 0.0 tclsh /usr/bin/pcap_agent.tcl -c /etc/nsm/SO-server-eth4/pcap_agent.conf
0.0 0.0 [kworker/4:0]
0.0 0.0 su - SO-user -- /usr/bin/snort_agent.tcl -c /etc/nsm/SO-server-eth4/snort_agent-1.conf
0.0 0.0 tclsh /usr/bin/snort_agent.tcl -c /etc/nsm/SO-server-eth4/snort_agent-1.conf
0.0 0.0 su - SO-user -- /usr/bin/snort_agent.tcl -c /etc/nsm/SO-server-eth4/snort_agent-2.conf
0.0 0.0 tclsh /usr/bin/snort_agent.tcl -c /etc/nsm/SO-server-eth4/snort_agent-2.conf
0.0 0.0 [kworker/7:0]
0.0 0.0 su - SO-user -- /usr/bin/snort_agent.tcl -c /etc/nsm/SO-server-eth4/snort_agent-3.conf
0.0 0.0 tclsh /usr/bin/snort_agent.tcl -c /etc/nsm/SO-server-eth4/snort_agent-3.conf
0.0 0.0 su - SO-user -- /usr/bin/snort_agent.tcl -c /etc/nsm/SO-server-eth4/snort_agent-4.conf
0.0 0.0 tclsh /usr/bin/snort_agent.tcl -c /etc/nsm/SO-server-eth4/snort_agent-4.conf
0.0 0.0 tail -n 1 -f /nsm/sensor_data/SO-server-eth4/snort-1.stats
0.0 0.0 tail -n 1 -f /nsm/sensor_data/SO-server-eth4/snort-2.stats
0.0 0.0 tail -n 1 -f /nsm/sensor_data/SO-server-eth4/snort-3.stats
0.0 0.0 tail -n 1 -f /nsm/sensor_data/SO-server-eth4/snort-4.stats
0.0 0.0 [kworker/1:0]
0.0 0.0 [kworker/2:2]
0.0 0.0 [kworker/29:1]
0.0 0.0 /usr/sbin/apache2 -k start
0.0 0.0 [kworker/30:0]
0.0 0.0 /bin/sh -c sh /opt/elsa/contrib/securityonion/contrib/securityonion-elsa-syslog-ng.sh
0.0 0.0 sh /opt/elsa/contrib/securityonion/contrib/securityonion-elsa-syslog-ng.sh
0.0 0.0 [kworker/16:1H]
0.0 0.0 [kworker/22:1]
0.0 0.0 [kworker/25:2]
0.0 0.0 [kworker/16:1]
0.0 0.0 [kworker/13:0]
0.0 0.0 /usr/bin/python /usr/bin/salt-minion
0.0 0.0 [kworker/u289:2]
0.0 0.0 [kworker/20:1]
0.0 0.0 [kworker/11:1]
0.0 0.0 [kworker/6:1]
0.0 0.0 CRON
0.0 0.0 /bin/sh -c /usr/bin/salt-call state.highstate >/dev/null 2>&1
0.0 0.0 [kworker/14:0]
0.0 0.0 CRON
0.0 0.0 /bin/sh -c sh /opt/elsa/contrib/securityonion/contrib/securityonion-elsa-cron.sh > /dev/null 2>&1
0.0 0.0 sh /opt/elsa/contrib/securityonion/contrib/securityonion-elsa-cron.sh
0.0 0.0 [kworker/15:2]
0.0 0.0 [kworker/30:1]
0.0 0.0 /bin/bash /usr/bin/sostat-redacted
0.0 0.0 /bin/bash /usr/bin/sostat
0.0 0.0 sed -r s/(\b[0-9]{1,3}\.){3}[0-9]{1,3}\b/X.X.X.X/g
0.0 0.0 sed -r s/([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}/MM:MM:MM:MM:MM:MM/g
0.0 0.0 sed -r s/X:ssh_port/X:ssh_port/g
0.0 0.0 sed -r s/\*:ssh_port/*:ssh_port/g
0.0 0.0 sed -r s/SO-server/SO-server/g
0.0 0.0 sed -r s/SO-node|SO-node|SO-node|SO-node/SO-node/g
0.0 0.0 sed -r s/SO-user|SO-user|SO-user|SO-user/SO-user/g
0.0 0.0 [kworker/22:2]
0.0 0.0 ps -eo pcpu,pmem,args --sort -pcpu
0.0 0.0 /usr/bin/python /usr/bin/salt-master
0.0 0.0 [kworker/29:1H]
0.0 0.0 [kworker/0:0]
0.0 0.0 [kworker/13:1]
0.0 0.0 [kworker/21:0]
0.0 0.0 [kworker/31:1H]
0.0 0.0 /usr/bin/python /usr/bin/salt-master
0.0 0.0 [kworker/23:1]
0.0 0.0 [kworker/19:0]
0.0 0.0 [kworker/7:1H]
0.0 0.0 [kworker/11:2]
0.0 0.0 [kworker/17:2]
0.0 0.0 [kworker/7:2]
0.0 0.0 [kworker/19:1]
0.0 0.0 [kworker/26:0]
0.0 0.0 [kworker/5:2]
0.0 0.0 [kworker/12:0]
0.0 0.0 [kworker/3:1]
0.0 0.0 [kworker/31:2]
0.0 0.0 [kworker/18:1]
0.0 0.0 [kworker/22:0]
0.0 0.0 [kworker/u288:0]
0.0 0.0 [kworker/25:0]
0.0 0.0 [kworker/10:0]
0.0 0.0 [kworker/17:0]
0.0 0.0 [kworker/u288:2]
0.0 0.0 [kworker/26:1]
0.0 0.0 [kworker/0:1]
0.0 0.0 [kworker/2:1]

=========================================================================
Packets received during last monitoring interval (600 seconds)
=========================================================================
eth4: 37054966

=========================================================================
Log Archive
=========================================================================
/nsm/sensor_data/SO-server-eth0/dailylogs/ - 0 days
4.0K .

/nsm/sensor_data/SO-server-eth1/dailylogs/ - 0 days
4.0K .

/nsm/sensor_data/SO-server-eth2/dailylogs/ - 0 days
4.0K .

/nsm/sensor_data/SO-server-eth3/dailylogs/ - 0 days
4.0K .

/nsm/sensor_data/SO-server-eth4/dailylogs/ - 3 days
3.2T .
1.2T ./2016-03-22
1.3T ./2016-03-23
846G ./2016-03-24

/nsm/sensor_data/SO-server-eth5/dailylogs/ - 0 days
4.0K .

/nsm/bro/logs/ - 4 days
4.7G .
1.1G ./2016-03-21
1.3G ./2016-03-22
1.5G ./2016-03-23
883M ./2016-03-24
56M ./stats

=========================================================================
Bro netstats
=========================================================================
Average packet loss as percent across all Bro workers: 0.007704

SO-server-eth4-1: 1458841860.063113 recvd=1486711595 dropped=596261 link=1486711595
SO-server-eth4-2: 1458841860.263108 recvd=1560438668 dropped=5 link=1560438668
SO-server-eth4-3: 1458841860.463102 recvd=1671361897 dropped=12 link=1671361897
SO-server-eth4-4: 1458841860.662991 recvd=3021666281 dropped=48 link=3021666281

=========================================================================
IDS Engine (snort) packet drops
=========================================================================
/nsm/sensor_data/SO-server-eth4/snort-1.stats last reported pkt_drop_percent as 0.000
/nsm/sensor_data/SO-server-eth4/snort-2.stats last reported pkt_drop_percent as 0.000
/nsm/sensor_data/SO-server-eth4/snort-3.stats last reported pkt_drop_percent as 0.000
/nsm/sensor_data/SO-server-eth4/snort-4.stats last reported pkt_drop_percent as 78.038

=========================================================================
pf_ring stats
=========================================================================
PF_RING Version : 6.2.0 (unknown)
Total rings : 8

Standard (non DNA/ZC) Options
Ring slots : 131070
Slot version : 16
Capture TX : Yes [RX+TX]
IP Defragment : No
Socket Mode : Standard
Total plugins : 0
Cluster Fragment Queue : 0
Cluster Fragment Discard : 0

/proc/net/pf_ring/70334-eth4.1504
Appl. Name : snort-cluster-55-socket-0
Tot Packets : 109691149
Tot Pkt Lost : 1512460
Reflect: Fwd Errors: 0
Min Num Slots : 131074
Num Free Slots : 131074

/proc/net/pf_ring/70714-eth4.1505
Appl. Name : snort-cluster-55-socket-0
Tot Packets : 121122875
Tot Pkt Lost : 4573927
Reflect: Fwd Errors: 0
Min Num Slots : 131074
Num Free Slots : 131074

/proc/net/pf_ring/70926-eth4.1506
Appl. Name : snort-cluster-55-socket-0
Tot Packets : 210978030
Tot Pkt Lost : 47822759
Reflect: Fwd Errors: 0
Min Num Slots : 131074
Num Free Slots : 0

/proc/net/pf_ring/71104-eth4.1507
Appl. Name : snort-cluster-55-socket-0
Tot Packets : 377703975
Tot Pkt Lost : 182115177
Reflect: Fwd Errors: 0
Min Num Slots : 131074
Num Free Slots : 0

/proc/net/pf_ring/9777-eth4.1
Appl. Name : bro-eth4
Tot Packets : 3021746543
Tot Pkt Lost : 48
TX: Send Errors : 0
Reflect: Fwd Errors: 0
Min Num Slots : 131070
Num Free Slots : 131069

/proc/net/pf_ring/9778-eth4.4
Appl. Name : bro-eth4
Tot Packets : 1671365643
Tot Pkt Lost : 12
TX: Send Errors : 0
Reflect: Fwd Errors: 0
Min Num Slots : 131070
Num Free Slots : 131070

/proc/net/pf_ring/9779-eth4.2
Appl. Name : bro-eth4
Tot Packets : 1487345461
Tot Pkt Lost : 596261
TX: Send Errors : 0
Reflect: Fwd Errors: 0
Min Num Slots : 131070
Num Free Slots : 131070

/proc/net/pf_ring/9780-eth4.3
Appl. Name : bro-eth4
Tot Packets : 1560443439
Tot Pkt Lost : 5
TX: Send Errors : 0
Reflect: Fwd Errors: 0
Min Num Slots : 131070
Num Free Slots : 131070

=========================================================================
Netsniff-NG - Reported Packet Loss (per interval)
=========================================================================
File: /var/log/nsm/SO-server-eth4/netsniff-ng.log Processed: +237576 Lost: -72764
File: /var/log/nsm/SO-server-eth4/netsniff-ng.log.20160318000402 Processed: Cannot Lost: allocate RX_RING!
File: /var/log/nsm/SO-server-eth4/netsniff-ng.log.20160318000902 Processed: Cannot Lost: allocate RX_RING!
File: /var/log/nsm/SO-server-eth4/netsniff-ng.log.20160319000005 Processed: +373301 Lost: -173008
File: /var/log/nsm/SO-server-eth4/netsniff-ng.log.20160319041153 Processed: +195961 Lost: -3576
File: /var/log/nsm/SO-server-eth4/netsniff-ng.log.20160320000004 Processed: +174079 Lost: -285
File: /var/log/nsm/SO-server-eth4/netsniff-ng.log.20160322000401 Processed: Cannot Lost: allocate RX_RING!

=========================================================================
Sguil Uncategorized Events
=========================================================================
COUNT(*)
147038

=========================================================================
Sguil events summary for yesterday
=========================================================================
Totals GenID:SigID Signature
REMOVED
Total
2288109

=========================================================================
Last update
=========================================================================
Start-Date: 2016-03-08 21:43:06
Commandline: apt-get -y dist-upgrade
Install: linux-image-extra-3.19.0-51-generic:amd64 (3.19.0-51.58~14.04.1, automatic), linux-image-3.19.0-51-generic:amd64 (3.19.0-51.58~14.04.1, automatic), linux-headers-3.19.0-51-generic:amd64 (3.19.0-51.58~14.04.1, automatic), linux-headers-3.19.0-51:amd64 (3.19.0-51.58~14.04.1, automatic)
Upgrade: bind9-host:amd64 (9.9.5.dfsg-3ubuntu0.6, 9.9.5.dfsg-3ubuntu0.7), initscripts:amd64 (2.88dsf-41ubuntu6.2, 2.88dsf-41ubuntu6.3), pciutils:amd64 (3.2.1-1ubuntu5, 3.2.1-1ubuntu5.1), securityonion-suricata:amd64 (3.0rc3-1ubuntu1securityonion1, 3.0stable-1ubuntu1securityonion1), liblwres90:amd64 (9.9.5.dfsg-3ubuntu0.6, 9.9.5.dfsg-3ubuntu0.7), kpartx:amd64 (0.4.9-3ubuntu7.7, 0.4.9-3ubuntu7.9), libstdc++-4.8-dev:amd64 (4.8.4-2ubuntu1~14.04, 4.8.4-2ubuntu1~14.04.1), apt:amd64 (1.0.1ubuntu2.10, 1.0.1ubuntu2.11), libasan0:amd64 (4.8.4-2ubuntu1~14.04, 4.8.4-2ubuntu1~14.04.1), libsystemd-login0:amd64 (204-5ubuntu20.15, 204-5ubuntu20.18), libgnutls-openssl27:amd64 (2.12.23-12ubuntu2.4, 2.12.23-12ubuntu2.5), libquadmath0:amd64 (4.8.4-2ubuntu1~14.04, 4.8.4-2ubuntu1~14.04.1), gcc-4.8-base:amd64 (4.8.4-2ubuntu1~14.04, 4.8.4-2ubuntu1~14.04.1), glib-networking-common:amd64 (2.40.0-1, 2.40.0-1ubuntu0.1), multiarch-support:amd64 (2.19-0ubuntu6.6, 2.19-0ubuntu6.7), libdns100:amd64 (9.9.5.dfsg-3ubuntu0.6, 9.9.5.dfsg-3ubuntu0.7), perl:amd64 (5.18.2-2ubuntu1, 5.18.2-2ubuntu1.1), python-samba:amd64 (4.1.6+dfsg-1ubuntu2.14.04.11, 4.1.6+dfsg-1ubuntu2.14.04.13), mysql-client-core-5.5:amd64 (5.5.46-0ubuntu0.14.04.2, 5.5.47-0ubuntu0.14.04.1), libisccfg90:amd64 (9.9.5.dfsg-3ubuntu0.6, 9.9.5.dfsg-3ubuntu0.7), systemd-services:amd64 (204-5ubuntu20.15, 204-5ubuntu20.18), securityonion-pfring-ld:amd64 (20120827-0ubuntu0securityonion8, 20120827-0ubuntu0securityonion9), libssl1.0.0:amd64 (1.0.1f-1ubuntu2.16, 1.0.1f-1ubuntu2.18), perl-base:amd64 (5.18.2-2ubuntu1, 5.18.2-2ubuntu1.1), apt-transport-https:amd64 (1.0.1ubuntu2.10, 1.0.1ubuntu2.11), os-prober:amd64 (1.63ubuntu1, 1.63ubuntu1.1), libgraphite2-3:amd64 (1.2.4-1ubuntu1, 1.2.4-1ubuntu1.1), libbind9-90:amd64 (9.9.5.dfsg-3ubuntu0.6, 9.9.5.dfsg-3ubuntu0.7), securityonion-capme:amd64 (20121213-0ubuntu0securityonion30, 20121213-0ubuntu0securityonion35), cpp-4.8:amd64 (4.8.4-2ubuntu1~14.04, 4.8.4-2ubuntu1~14.04.1), libgomp1:amd64 (4.8.4-2ubuntu1~14.04, 4.8.4-2ubuntu1~14.04.1), libdrm-intel1:amd64 (2.4.60-2~ubuntu14.04.1, 2.4.64-1~ubuntu14.04.1), squashfs-tools:amd64 (4.2+20130409-2, 4.2+20130409-2ubuntu0.14.04.1), libtsan0:amd64 (4.8.4-2ubuntu1~14.04, 4.8.4-2ubuntu1~14.04.1), apt-utils:amd64 (1.0.1ubuntu2.10, 1.0.1ubuntu2.11), cpio:amd64 (2.11+dfsg-1ubuntu1.1, 2.11+dfsg-1ubuntu1.2), ubiquity:amd64 (X.X.X.X, X.X.X.X), fonts-opensymbol:amd64 (102.6+LibO4.2.8-0ubuntu3, 102.6+LibO4.2.8-0ubuntu4), openssh-server:amd64 (6.6p1-2ubuntu2.3, 6.6p1-2ubuntu2.6), libdrm-radeon1:amd64 (2.4.60-2~ubuntu14.04.1, 2.4.64-1~ubuntu14.04.1), securityonion-elsa:amd64 (1205chartsjsd3-1ubuntu1securityonion5, 1205chartsjsd3-1ubuntu1securityonion6), libsystemd-daemon0:amd64 (204-5ubuntu20.15, 204-5ubuntu20.18), libgudev-1.0-0:amd64 (204-5ubuntu20.15, 204-5ubuntu20.18), libc-dev-bin:amd64 (2.19-0ubuntu6.6, 2.19-0ubuntu6.7), securityonion-nsmnow-admin-scripts:amd64 (20120724-0ubuntu0securityonion127, 20120724-0ubuntu0securityonion130), chromium-codecs-ffmpeg-extra:amd64 (48.0.2564.82-0ubuntu0.14.04.1.1108, 48.0.2564.116-0ubuntu0.14.04.1.1111), libpam-systemd:amd64 (204-5ubuntu20.15, 204-5ubuntu20.18), gtk2-engines-pixbuf:amd64 (2.24.23-0ubuntu1.3, 2.24.23-0ubuntu1.4), libc-bin:amd64 (2.19-0ubuntu6.6, 2.19-0ubuntu6.7), libc6:amd64 (2.19-0ubuntu6.6, 2.19-0ubuntu6.7), libnuma1:amd64 (2.0.9~rc5-1ubuntu3.14.04.1, 2.0.9~rc5-1ubuntu3.14.04.2), openssh-sftp-server:amd64 (6.6p1-2ubuntu2.3, 6.6p1-2ubuntu2.6), linux-image-generic-lts-vivid:amd64 (X.X.X.X.32, X.X.X.X.36), python-libxml2:amd64 (2.9.1+dfsg1-3ubuntu4.6, 2.9.1+dfsg1-3ubuntu4.7), libapt-inst1.5:amd64 (1.0.1ubuntu2.10, 1.0.1ubuntu2.11), libpci3:amd64 (3.2.1-1ubuntu5, 3.2.1-1ubuntu5.1), libnss3-1d:amd64 (X.X.X.X-0ubuntu0.14.04.2, 3.21-0ubuntu0.14.04.1), libgtk2.0-bin:amd64 (2.24.23-0ubuntu1.3, 2.24.23-0ubuntu1.4), libgtk2.0-common:amd64 (2.24.23-0ubuntu1.3, 2.24.23-0ubuntu1.4), ubiquity-ubuntu-artwork:amd64 (X.X.X.X, X.X.X.X), libperl5.18:amd64 (5.18.2-2ubuntu1, 5.18.2-2ubuntu1.1), dnsutils:amd64 (9.9.5.dfsg-3ubuntu0.6, 9.9.5.dfsg-3ubuntu0.7), thermald:amd64 (1.4.3-5~14.04.1, 1.4.3-5~14.04.2), udev:amd64 (204-5ubuntu20.15, 204-5ubuntu20.18), base-files:amd64 (7.2ubuntu5.3, 7.2ubuntu5.4), curl:amd64 (7.35.0-1ubuntu2.5, 7.35.0-1ubuntu2.6), ifupdown:amd64 (0.7.47.2ubuntu4.1, 0.7.47.2ubuntu4.4), libpixman-1-0:amd64 (0.30.2-2ubuntu1, 0.30.2-2ubuntu1.1), libatomic1:amd64 (4.8.4-2ubuntu1~14.04, 4.8.4-2ubuntu1~14.04.1), chromium-browser-l10n:amd64 (48.0.2564.82-0ubuntu0.14.04.1.1108, 48.0.2564.116-0ubuntu0.14.04.1.1111), libnss3-nssdb:amd64 (X.X.X.X-0ubuntu0.14.04.2, 3.21-0ubuntu0.14.04.1), libecryptfs0:amd64 (104-0ubuntu1.14.04.3, 104-0ubuntu1.14.04.4), samba-common-bin:amd64 (4.1.6+dfsg-1ubuntu2.14.04.11, 4.1.6+dfsg-1ubuntu2.14.04.13), libxml2:amd64 (2.9.1+dfsg1-3ubuntu4.6, 2.9.1+dfsg1-3ubuntu4.7), ntp:amd64 (4.2.6.p5+dfsg-3ubuntu2.14.04.6, 4.2.6.p5+dfsg-3ubuntu2.14.04.8), g++-4.8:amd64 (4.8.4-2ubuntu1~14.04, 4.8.4-2ubuntu1~14.04.1), libudev1:amd64 (204-5ubuntu20.15, 204-5ubuntu20.18), linux-headers-generic-lts-vivid:amd64 (X.X.X.X.32, X.X.X.X.36), perl-modules:amd64 (5.18.2-2ubuntu1, 5.18.2-2ubuntu1.1), securityonion-elsa-extras:amd64 (20151011-1ubuntu1securityonion26, 20151011-1ubuntu1securityonion27), ubiquity-frontend-debconf:amd64 (X.X.X.X, X.X.X.X), libssl-dev:amd64 (1.0.1f-1ubuntu2.16, 1.0.1f-1ubuntu2.18), samba-libs:amd64 (4.1.6+dfsg-1ubuntu2.14.04.11, 4.1.6+dfsg-1ubuntu2.14.04.13), openssh-client:amd64 (6.6p1-2ubuntu2.3, 6.6p1-2ubuntu2.6), libapt-pkg4.12:amd64 (1.0.1ubuntu2.10, 1.0.1ubuntu2.11), libgcc-4.8-dev:amd64 (4.8.4-2ubuntu1~14.04, 4.8.4-2ubuntu1~14.04.1), securityonion-sostat:amd64 (20120722-0ubuntu0securityonion49, 20120722-0ubuntu0securityonion51), gcc-4.8:amd64 (4.8.4-2ubuntu1~14.04, 4.8.4-2ubuntu1~14.04.1), libjasper1:amd64 (1.900.1-14ubuntu3.2, 1.900.1-14ubuntu3.3), libgtk2.0-0:amd64 (2.24.23-0ubuntu1.3, 2.24.23-0ubuntu1.4), ca-certificates:amd64 (20141019ubuntu0.14.04.1, 20160104ubuntu0.14.04.1), libnss3:amd64 (X.X.X.X-0ubuntu0.14.04.2, 3.21-0ubuntu0.14.04.1), libdrm-nouveau2:amd64 (2.4.60-2~ubuntu14.04.1, 2.4.64-1~ubuntu14.04.1), libmysqlclient18:amd64 (5.5.46-0ubuntu0.14.04.2, 5.5.47-0ubuntu0.14.04.1), linux-generic-lts-vivid:amd64 (X.X.X.X.32, X.X.X.X.36), sysv-rc:amd64 (2.88dsf-41ubuntu6.2, 2.88dsf-41ubuntu6.3), apache2-data:amd64 (2.4.7-1ubuntu4.8, 2.4.7-1ubuntu4.9), libgnutls26:amd64 (2.12.23-12ubuntu2.4, 2.12.23-12ubuntu2.5), smbclient:amd64 (4.1.6+dfsg-1ubuntu2.14.04.11, 4.1.6+dfsg-1ubuntu2.14.04.13), securityonion-pfring-daq:amd64 (20121107-0ubuntu0securityonion10, 20121107-0ubuntu0securityonion12), libcurl3:amd64 (7.35.0-1ubuntu2.5, 7.35.0-1ubuntu2.6), passwd:amd64 (X.X.X.X-1ubuntu9.1, X.X.X.X-1ubuntu9.2), libnettle4:amd64 (2.7.1-1, 2.7.1-1ubuntu0.1), libgfortran3:amd64 (4.8.4-2ubuntu1~14.04, 4.8.4-2ubuntu1~14.04.1), ecryptfs-utils:amd64 (104-0ubuntu1.14.04.3, 104-0ubuntu1.14.04.4), kpartx-boot:amd64 (0.4.9-3ubuntu7.7, 0.4.9-3ubuntu7.9), securityonion-setup:amd64 (20120912-0ubuntu0securityonion190, 20120912-0ubuntu0securityonion194), linux-firmware:amd64 (1.127.19, 1.127.20), pm-utils:amd64 (1.4.1-13ubuntu0.1, 1.4.1-13ubuntu0.2), sysvinit-utils:amd64 (2.88dsf-41ubuntu6.2, 2.88dsf-41ubuntu6.3), libssl-doc:amd64 (1.0.1f-1ubuntu2.16, 1.0.1f-1ubuntu2.18), securityonion-pfring-userland:amd64 (20150921-0ubuntu0securityonion2, 20160204-1ubuntu1securityonion2), login:amd64 (X.X.X.X-1ubuntu9.1, X.X.X.X-1ubuntu9.2), apache2:amd64 (2.4.7-1ubuntu4.8, 2.4.7-1ubuntu4.9), openssl:amd64 (1.0.1f-1ubuntu2.16, 1.0.1f-1ubuntu2.18), libdebian-installer4:amd64 (0.88ubuntu5.1, 0.88ubuntu5.2), securityonion-networkminer:amd64 (20140829-0ubuntu0securityonion3, 20160210-1ubuntu1securityonion1), apache2-bin:amd64 (2.4.7-1ubuntu4.8, 2.4.7-1ubuntu4.9), libwbclient0:amd64 (4.1.6+dfsg-1ubuntu2.14.04.11, 4.1.6+dfsg-1ubuntu2.14.04.13), rsync:amd64 (3.1.0-2ubuntu0.1, 3.1.0-2ubuntu0.2), libc6-dbg:amd64 (2.19-0ubuntu6.6, 2.19-0ubuntu6.7), linux-libc-dev:amd64 (3.13.0-74.118, 3.13.0-79.123), glib-networking-services:amd64 (2.40.0-1, 2.40.0-1ubuntu0.1), samba-common:amd64 (4.1.6+dfsg-1ubuntu2.14.04.11, 4.1.6+dfsg-1ubuntu2.14.04.13), libstdc++6:amd64 (4.8.4-2ubuntu1~14.04, 4.8.4-2ubuntu1~14.04.1), chromium-browser:amd64 (48.0.2564.82-0ubuntu0.14.04.1.1108, 48.0.2564.116-0ubuntu0.14.04.1.1111), libdrm2:amd64 (2.4.60-2~ubuntu14.04.1, 2.4.64-1~ubuntu14.04.1), libpq5:amd64 (9.3.10-0ubuntu0.14.04, 9.3.11-0ubuntu0.14.04), libitm1:amd64 (4.8.4-2ubuntu1~14.04, 4.8.4-2ubuntu1~14.04.1), glib-networking:amd64 (2.40.0-1, 2.40.0-1ubuntu0.1), libisccc90:amd64 (9.9.5.dfsg-3ubuntu0.6, 9.9.5.dfsg-3ubuntu0.7), libgcrypt11:amd64 (1.5.3-2ubuntu4.2, 1.5.3-2ubuntu4.3), libc6-dev:amd64 (2.19-0ubuntu6.6, 2.19-0ubuntu6.7), libsmbclient:amd64 (4.1.6+dfsg-1ubuntu2.14.04.11, 4.1.6+dfsg-1ubuntu2.14.04.13), libisc95:amd64 (9.9.5.dfsg-3ubuntu0.6, 9.9.5.dfsg-3ubuntu0.7), libcurl3-gnutls:amd64 (7.35.0-1ubuntu2.5, 7.35.0-1ubuntu2.6), ntpdate:amd64 (4.2.6.p5+dfsg-3ubuntu2.14.04.6, 4.2.6.p5+dfsg-3ubuntu2.14.04.8)
End-Date: 2016-03-08 21:46:10

Start-Date: 2016-03-08 21:47:05
Commandline: apt-get -y remove --purge linux-image-3.19.0-43-generic linux-headers-3.19.0-43-generic
Purge: linux-image-extra-3.19.0-43-generic:amd64 (3.19.0-43.49~14.04.1), linux-image-3.19.0-43-generic:amd64 (3.19.0-43.49~14.04.1), linux-headers-3.19.0-43-generic:amd64 (3.19.0-43.49~14.04.1)
End-Date: 2016-03-08 21:47:31

=========================================================================
ELSA
=========================================================================
Syslog-ng
Checking for process:
741 /usr/sbin/syslog-ng -p /var/run/syslog-ng.pid
Checking for connection:
Connection to localhost 514 port [tcp/shell] succeeded!

MySQL
Checking for process:
1943 /usr/sbin/mysqld
Checking for connection:
Connection to localhost 3306 port [tcp/mysql] succeeded!

Sphinx
Checking for process:
1893 su -s /bin/sh -c exec "$0" "$@" sphinxsearch -- /usr/bin/searchd --nodetach
1937 /usr/bin/searchd --nodetach
Checking for connection:
Connection to localhost 9306 port [tcp/*] succeeded!

ELSA Buffers in Queue:
3
If this number is consistently higher than 20, please see:
https://github.com/Security-Onion-Solutions/security-onion/wiki/FAQ#why-does-sostat-show-a-high-number-of-elsa-buffers-in-queue

ELSA Directory Sizes:
3.7T /nsm/elsa/data
77M /var/lib/mysql/syslog
5.2M /var/lib/mysql/syslog_data

ELSA Index Date Range
If you don't have at least 2 full days of logs in the Index Date Range,
then you'll need to increase log_size_limit in /etc/elsa_node.conf.
MIN(start) MAX(end)
2016-02-04 05:44:39 2016-03-24 17:50:38

ELSA Log Node SSH Tunnels:
PORT NODE IP/STATUS
50000 SO-node X.X.X.X
50001 SO-node X.X.X.X
50002 SO-node X.X.X.X

Doug Burks

unread,

Mar 25, 2016, 7:14:11 AM3/25/16

to securit...@googlegroups.com

This is a strange issue. You have lots of salt-master processes using
lots of CPU. Have you tried rebooting the master server?

> You received this message because you are subscribed to the Google Groups "security-onion" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.

> To post to this group, send email to securit...@googlegroups.com.
> Visit this group at https://groups.google.com/group/security-onion.
> For more options, visit https://groups.google.com/d/optout.

--
Doug Burks

s adz

unread,

Mar 29, 2016, 8:41:05 AM3/29/16

to securit...@googlegroups.com

Rebooting no help. One server makes 48% packet loss: server4-eth7-4. IDS_LB_PROCS = 4, make 8. Packet loss now on server4-eth7-8 48%!

Next I change /etc/modprobe.d/pf_ring.conf to make bigger, no luck.

server4 not salt master. Related, maybe unrelated.

Wes

unread,

Mar 29, 2016, 12:48:35 PM3/29/16

to security-onion

> > 0.1 0.0 /opt/bro/...

Please attach the sostat-redacted for the affected machine(s).

Thanks,
Wes

s adz

unread,

Apr 11, 2016, 1:18:59 PM4/11/16

to securit...@googlegroups.com

here is sostat-redacted. We have problem with salt master running too many of itself. It bury CPU. I also paste at bottom of this, interesting log message from minion.

=========================================================================

Service Status

=========================================================================

Status: HIDS

* ossec_agent (SO-user)[ OK ]

Status: Bro

Getting process status ...

Getting peer status ...

Name Type Host Status Pid Peers Started

manager manager localhost running 144198 7 28 Mar 20:05:00

proxy proxy localhost running 144291 7 28 Mar 20:05:01

SO-server-eth4-1 worker localhost running 144390 2 28 Mar 20:05:03

SO-server-eth4-2 worker localhost running 144397 2 28 Mar 20:05:03

SO-server-eth5-1 worker localhost running 144400 2 28 Mar 20:05:03

SO-server-eth5-2 worker localhost running 144403 2 28 Mar 20:05:03

SO-server-eth7-1 worker localhost running 144405 2 28 Mar 20:05:03

SO-server-eth7-2 worker localhost running 144407 2 28 Mar 20:05:03

Status: SO-server-eth4

* netsniff-ng (full packet data)[ OK ]

* pcap_agent (SO-user)[ OK ]

* snort_agent-1 (SO-user)[ OK ]

* snort_agent-2 (SO-user)[ OK ]

* snort_agent-3 (SO-user)[ OK ]

* snort_agent-4 (SO-user)[ OK ]

* snort_agent-5 (SO-user)[ OK ]

* snort_agent-6 (SO-user)[ OK ]

* snort_agent-7 (SO-user)[ OK ]

* snort_agent-8 (SO-user)[ OK ]

* snort-1 (alert data)[ OK ]

* snort-2 (alert data)[ OK ]

* snort-3 (alert data)[ OK ]

* snort-4 (alert data)[ OK ]

* snort-5 (alert data)[ OK ]

* snort-6 (alert data)[ OK ]

* snort-7 (alert data)[ OK ]

* snort-8 (alert data)[ OK ]

* barnyard2-1 (spooler, unified2 format)[ OK ]

* barnyard2-2 (spooler, unified2 format)[ OK ]

* barnyard2-3 (spooler, unified2 format)[ OK ]

* barnyard2-4 (spooler, unified2 format)[ OK ]

* barnyard2-5 (spooler, unified2 format)[ OK ]

* barnyard2-6 (spooler, unified2 format)[ OK ]

* barnyard2-7 (spooler, unified2 format)[ OK ]

* barnyard2-8 (spooler, unified2 format)[ OK ]

Status: SO-server-eth5

* netsniff-ng (full packet data)[ OK ]

* pcap_agent (SO-user)[ OK ]

* snort_agent-1 (SO-user)[ OK ]

* snort_agent-2 (SO-user)[ OK ]

* snort_agent-3 (SO-user)[ OK ]

* snort_agent-4 (SO-user)[ OK ]

* snort_agent-5 (SO-user)[ OK ]

* snort_agent-6 (SO-user)[ OK ]

* snort_agent-7 (SO-user)[ OK ]

* snort_agent-8 (SO-user)[ OK ]

* snort-1 (alert data)[ OK ]

* snort-2 (alert data)[ OK ]

* snort-3 (alert data)[ OK ]

* snort-4 (alert data)[ OK ]

* snort-5 (alert data)[ OK ]

* snort-6 (alert data)[ OK ]

* snort-7 (alert data)[ OK ]

* snort-8 (alert data)[ OK ]

* barnyard2-1 (spooler, unified2 format)[ OK ]

* barnyard2-2 (spooler, unified2 format)[ OK ]

* barnyard2-3 (spooler, unified2 format)[ OK ]

* barnyard2-4 (spooler, unified2 format)[ OK ]

* barnyard2-5 (spooler, unified2 format)[ OK ]

* barnyard2-6 (spooler, unified2 format)[ OK ]

* barnyard2-7 (spooler, unified2 format)[ OK ]

* barnyard2-8 (spooler, unified2 format)[ OK ]

Status: SO-server-eth7

* netsniff-ng (full packet data)[ FAIL ]

* pcap_agent (SO-user)[ OK ]

* snort_agent-1 (SO-user)[ OK ]

* snort_agent-2 (SO-user)[ OK ]

* snort_agent-3 (SO-user)[ OK ]

* snort_agent-4 (SO-user)[ OK ]

* snort_agent-5 (SO-user)[ OK ]

* snort_agent-6 (SO-user)[ OK ]

* snort_agent-7 (SO-user)[ OK ]

* snort_agent-8 (SO-user)[ OK ]

* snort-1 (alert data)[ OK ]

* snort-2 (alert data)[ OK ]

* snort-3 (alert data)[ OK ]

* snort-4 (alert data)[ OK ]

* snort-5 (alert data)[ OK ]

* snort-6 (alert data)[ OK ]

* snort-7 (alert data)[ OK ]

* snort-8 (alert data)[ OK ]

* barnyard2-1 (spooler, unified2 format)[ OK ]

* barnyard2-2 (spooler, unified2 format)[ OK ]

* barnyard2-3 (spooler, unified2 format)[ OK ]

* barnyard2-4 (spooler, unified2 format)[ OK ]

* barnyard2-5 (spooler, unified2 format)[ OK ]

* barnyard2-6 (spooler, unified2 format)[ OK ]

* barnyard2-7 (spooler, unified2 format)[ OK ]

* barnyard2-8 (spooler, unified2 format)[ OK ]

=========================================================================

Interface Status

=========================================================================

eth3 Link encap:Ethernet HWaddr MM:MM:MM:MM:MM:MM

inet addr:X.X.X.X Bcast:X.X.X.X Mask:X.X.X.X

inet6 addr: X.X.X.X/64 Scope:Link

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

RX packets:10285530 errors:0 dropped:0 overruns:0 frame:0

TX packets:11933333 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:1197830060 (1.1 GB) TX bytes:9013711928 (9.0 GB)

Interrupt:103 Memory:92000000-927fffff

eth4 Link encap:Ethernet HWaddr MM:MM:MM:MM:MM:MM

UP BROADCAST RUNNING NOARP PROMISC MULTICAST MTU:1500 Metric:1

RX packets:4155764999 errors:2556 dropped:0 overruns:2556 frame:0

TX packets:5 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:4250039941898 (4.2 TB) TX bytes:438 (438.0 B)

Interrupt:114 Memory:97800000-97ffffff

eth5 Link encap:Ethernet HWaddr MM:MM:MM:MM:MM:MM

UP BROADCAST RUNNING NOARP PROMISC MULTICAST MTU:1500 Metric:1

RX packets:17327489923 errors:3091 dropped:0 overruns:3091 frame:0

TX packets:5 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:12952360401225 (12.9 TB) TX bytes:438 (438.0 B)

Interrupt:103 Memory:96800000-96ffffff

eth7 Link encap:Ethernet HWaddr MM:MM:MM:MM:MM:MM

UP BROADCAST RUNNING NOARP PROMISC MULTICAST MTU:1500 Metric:1

RX packets:26586083926 errors:72422 dropped:0 overruns:72210 frame:212

TX packets:4 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:21986531144609 (21.9 TB) TX bytes:356 (356.0 B)

Interrupt:146 Memory:c8000000-c87fffff

lo Link encap:Local Loopback

inet addr:X.X.X.X Mask:X.X.X.X

inet6 addr: X.X.X.X/128 Scope:Host

UP LOOPBACK RUNNING MTU:65536 Metric:1

RX packets:93361009 errors:0 dropped:0 overruns:0 frame:0

TX packets:93361009 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:0

RX bytes:449099270488 (449.0 GB) TX bytes:449099270488 (449.0 GB)

=========================================================================

Link Statistics

=========================================================================

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default

link/loopback MM:MM:MM:MM:MM:MM brd MM:MM:MM:MM:MM:MM

RX: bytes packets errors dropped overrun mcast

449099270488 93361009 0 0 0 0

RX errors: length crc frame fifo missed

0 0 0 0 0

TX: bytes packets errors dropped carrier collsns

449099270488 93361009 0 0 0 0

1197830060 10285530 0 0 0 604019

RX errors: length crc frame fifo missed

0 0 0 0 0

TX: bytes packets errors dropped carrier collsns

9013711928 11933333 0 0 0 0

TX errors: aborted fifo window heartbeat

0 0 0 0

6: eth4: <BROADCAST,MULTICAST,NOARP,PROMISC,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000

link/ether MM:MM:MM:MM:MM:MM brd MM:MM:MM:MM:MM:MM

RX: bytes packets errors dropped overrun mcast

4250039941898 4155764999 2556 0 0 0

RX errors: length crc frame fifo missed

0 0 0 2556 0

TX: bytes packets errors dropped carrier collsns

438 5 0 0 0 0

TX errors: aborted fifo window heartbeat

0 0 0 0

7: eth5: <BROADCAST,MULTICAST,NOARP,PROMISC,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000

link/ether MM:MM:MM:MM:MM:MM brd MM:MM:MM:MM:MM:MM

RX: bytes packets errors dropped overrun mcast

12952360401225 17327489923 3091 0 0 1666319

RX errors: length crc frame fifo missed

0 0 0 3091 0

TX: bytes packets errors dropped carrier collsns

438 5 0 0 0 0

TX errors: aborted fifo window heartbeat

0 0 0 0

8: eth6: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000

link/ether MM:MM:MM:MM:MM:MM brd MM:MM:MM:MM:MM:MM

RX: bytes packets errors dropped overrun mcast

0 0 0 0 0 0

RX errors: length crc frame fifo missed

0 0 0 0 0

TX: bytes packets errors dropped carrier collsns

0 0 0 0 0 0

TX errors: aborted fifo window heartbeat

0 0 0 0

9: eth7: <BROADCAST,MULTICAST,NOARP,PROMISC,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000

link/ether MM:MM:MM:MM:MM:MM brd MM:MM:MM:MM:MM:MM

RX: bytes packets errors dropped overrun mcast

21986531144609 26586083926 72422 0 212 513290313

RX errors: length crc frame fifo missed

0 0 0 72210 0

TX: bytes packets errors dropped carrier collsns

356 4 0 0 0 0

TX errors: aborted fifo window heartbeat

0 0 0 0

=========================================================================

Disk Usage

=========================================================================

Filesystem Size Used Avail Use% Mounted on

udev 189G 4.0K 189G 1% /dev

tmpfs 38G 2.0M 38G 1% /run

/dev/mapper/securityonion--vg-root 8.3T 5.6T 2.4T 71% /

none 4.0K 0 4.0K 0% /sys/fs/cgroup

none 5.0M 0 5.0M 0% /run/lock

none 189G 12K 189G 1% /run/shm

none 100M 4.0K 100M 1% /run/user

/dev/sda2 237M 121M 104M 54% /boot

=========================================================================

Network Sockets

=========================================================================

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME

avahi-dae 1087 avahi 12u IPv4 25718 0t0 UDP *:5353

avahi-dae 1087 avahi 13u IPv6 25719 0t0 UDP *:5353

avahi-dae 1087 avahi 14u IPv4 25720 0t0 UDP *:43186

avahi-dae 1087 avahi 15u IPv6 25721 0t0 UDP *:33148

sshd 1999 root 3u IPv4 21672 0t0 TCP *:ssh_port (LISTEN)

sshd 1999 root 4u IPv6 21674 0t0 TCP *:ssh_port (LISTEN)

cups-brow 2087 root 6u IPv6 42644 0t0 TCP [X.X.X.X]:41863->[X.X.X.X]:631 (CLOSE_WAIT)

cups-brow 2087 root 8u IPv4 42646 0t0 UDP *:631

searchd 2100 sphinxsearch 7u IPv4 414 0t0 TCP *:9306 (LISTEN)

searchd 2100 sphinxsearch 8u IPv4 415 0t0 TCP *:9312 (LISTEN)

mysqld 2130 mysql 10u IPv4 40062 0t0 TCP X.X.X.X:50000 (LISTEN)

ossec-csy 2176 ossecm 5u IPv4 13482 0t0 UDP X.X.X.X:50689->X.X.X.X:514

starman 2391 www-data 5u IPv6 558 0t0 TCP *:3154 (LISTEN)

starman 2403 www-data 5u IPv6 558 0t0 TCP *:3154 (LISTEN)

starman 2404 www-data 5u IPv6 558 0t0 TCP *:3154 (LISTEN)

starman 2405 www-data 5u IPv6 558 0t0 TCP *:3154 (LISTEN)

starman 2406 www-data 5u IPv6 558 0t0 TCP *:3154 (LISTEN)

starman 2406 www-data 14u IPv4 419739 0t0 TCP X.X.X.X:60176->X.X.X.X:3154 (CLOSE_WAIT)

starman 2407 www-data 5u IPv6 558 0t0 TCP *:3154 (LISTEN)

starman 2407 www-data 14u IPv4 446694 0t0 TCP X.X.X.X:60173->X.X.X.X:3154 (CLOSE_WAIT)

cupsd 3912 root 10u IPv6 41189 0t0 TCP [X.X.X.X]:631 (LISTEN)

cupsd 3912 root 11u IPv4 41190 0t0 TCP X.X.X.X:631 (LISTEN)

ssh 6462 root 3u IPv4 45136 0t0 TCP X.X.X.X:57360->X.X.X.X:ssh_port (ESTABLISHED)

ssh 6462 root 4u IPv6 31446 0t0 TCP [X.X.X.X]:3306 (LISTEN)

ssh 6462 root 5u IPv4 31447 0t0 TCP X.X.X.X:3306 (LISTEN)

ntpd 8201 ntp 16u IPv4 14199 0t0 UDP *:123

ntpd 8201 ntp 17u IPv6 14200 0t0 UDP *:123

ntpd 8201 ntp 18u IPv4 14206 0t0 UDP X.X.X.X:123

ntpd 8201 ntp 19u IPv4 14207 0t0 UDP X.X.X.X:123

ntpd 8201 ntp 20u IPv6 14208 0t0 UDP [X.X.X.X]:123

ntpd 8201 ntp 21u IPv6 14209 0t0 UDP [X.X.X.X]:123

salt-mini 12161 root 12u IPv4 50600621 0t0 TCP X.X.X.X:49885->X.X.X.X:4505 (ESTABLISHED)

salt-mini 12161 root 26u IPv4 50659730 0t0 TCP X.X.X.X:55190->X.X.X.X:4506 (ESTABLISHED)

tclsh 32930 SO-user 3u IPv4 48748160 0t0 TCP X.X.X.X:39546->X.X.X.X:7736 (ESTABLISHED)

barnyard2 35358 SO-user 3u IPv4 50034340 0t0 TCP X.X.X.X:56484->X.X.X.X:8502 (ESTABLISHED)

barnyard2 51834 SO-user 3u IPv4 50003595 0t0 TCP X.X.X.X:53608->X.X.X.X:8701 (ESTABLISHED)

barnyard2 51867 SO-user 3u IPv4 49999238 0t0 TCP X.X.X.X:38904->X.X.X.X:8702 (ESTABLISHED)

barnyard2 51898 SO-user 3u IPv4 50002458 0t0 TCP X.X.X.X:55060->X.X.X.X:8703 (ESTABLISHED)

barnyard2 51928 SO-user 3u IPv4 50000369 0t0 TCP X.X.X.X:47354->X.X.X.X:8704 (ESTABLISHED)

barnyard2 51955 SO-user 3u IPv4 50003671 0t0 TCP X.X.X.X:40060->X.X.X.X:8705 (ESTABLISHED)

barnyard2 51981 SO-user 3u IPv4 50000367 0t0 TCP X.X.X.X:53328->X.X.X.X:8706 (ESTABLISHED)

barnyard2 52007 SO-user 3u IPv4 50015239 0t0 TCP X.X.X.X:45947->X.X.X.X:8707 (ESTABLISHED)

sshd 80625 root 3u IPv4 481680 0t0 TCP X.X.X.X:ssh_port->X.X.X.X:47602 (ESTABLISHED)

sshd 80661 SO-user 3u IPv4 481680 0t0 TCP X.X.X.X:ssh_port->X.X.X.X:47602 (ESTABLISHED)

sshd 80661 SO-user 9u IPv6 481703 0t0 TCP [X.X.X.X]:6010 (LISTEN)

sshd 80661 SO-user 10u IPv4 481704 0t0 TCP X.X.X.X:6010 (LISTEN)

barnyard2 107269 SO-user 3u IPv4 50012503 0t0 TCP X.X.X.X:56172->X.X.X.X:8507 (ESTABLISHED)

barnyard2 107327 SO-user 3u IPv4 50004576 0t0 TCP X.X.X.X:40748->X.X.X.X:8508 (ESTABLISHED)

tclsh 114539 SO-user 3u IPv4 49970602 0t0 TCP X.X.X.X:52351->X.X.X.X:7736 (ESTABLISHED)

barnyard2 114714 SO-user 3u IPv4 9679057 0t0 TCP X.X.X.X:55331->X.X.X.X:8503 (CLOSE_WAIT)

barnyard2 114755 SO-user 3u IPv4 9678278 0t0 TCP X.X.X.X:50974->X.X.X.X:8504 (CLOSE_WAIT)

barnyard2 114802 SO-user 3u IPv4 9681112 0t0 TCP X.X.X.X:59061->X.X.X.X:8505 (CLOSE_WAIT)

tclsh 116154 SO-user 3u IPv4 49995161 0t0 TCP X.X.X.X:56955->X.X.X.X:7736 (ESTABLISHED)

tclsh 116523 SO-user 3u IPv4 49964573 0t0 TCP X.X.X.X:52451->X.X.X.X:7736 (ESTABLISHED)

tclsh 116905 SO-user 3u IPv4 49977283 0t0 TCP X.X.X.X:45717->X.X.X.X:7736 (ESTABLISHED)

tclsh 117161 SO-user 3u IPv4 49964608 0t0 TCP X.X.X.X:51146->X.X.X.X:7736 (ESTABLISHED)

tclsh 117161 SO-user 4u IPv4 50001301 0t0 TCP X.X.X.X:8401 (LISTEN)

tclsh 117276 SO-user 3u IPv4 49955781 0t0 TCP X.X.X.X:38093->X.X.X.X:7736 (ESTABLISHED)

tclsh 117276 SO-user 4u IPv4 49996368 0t0 TCP X.X.X.X:8402 (LISTEN)

tclsh 117321 SO-user 3u IPv4 50009151 0t0 TCP X.X.X.X:34904->X.X.X.X:7736 (ESTABLISHED)

tclsh 117321 SO-user 4u IPv4 50004109 0t0 TCP X.X.X.X:8403 (LISTEN)

tclsh 117364 SO-user 3u IPv4 49996392 0t0 TCP X.X.X.X:52470->X.X.X.X:7736 (ESTABLISHED)

tclsh 117364 SO-user 4u IPv4 50004114 0t0 TCP X.X.X.X:8404 (LISTEN)

tclsh 117407 SO-user 3u IPv4 50002289 0t0 TCP X.X.X.X:47489->X.X.X.X:7736 (ESTABLISHED)

tclsh 117407 SO-user 4u IPv4 50002290 0t0 TCP X.X.X.X:8405 (LISTEN)

tclsh 117449 SO-user 3u IPv4 49994296 0t0 TCP X.X.X.X:37026->X.X.X.X:7736 (ESTABLISHED)

tclsh 117449 SO-user 4u IPv4 50001475 0t0 TCP X.X.X.X:8406 (LISTEN)

tclsh 117560 SO-user 3u IPv4 49998458 0t0 TCP X.X.X.X:44513->X.X.X.X:7736 (ESTABLISHED)

tclsh 117560 SO-user 4u IPv4 50002367 0t0 TCP X.X.X.X:8407 (LISTEN)

tclsh 117603 SO-user 3u IPv4 49997466 0t0 TCP X.X.X.X:39657->X.X.X.X:7736 (ESTABLISHED)

tclsh 117603 SO-user 4u IPv4 49996409 0t0 TCP X.X.X.X:8408 (LISTEN)

tclsh 117719 SO-user 3u IPv4 49976032 0t0 TCP X.X.X.X:42354->X.X.X.X:7736 (ESTABLISHED)

tclsh 117719 SO-user 4u IPv4 50001510 0t0 TCP X.X.X.X:8501 (LISTEN)

tclsh 117719 SO-user 6u IPv4 50012639 0t0 TCP X.X.X.X:8501->X.X.X.X:57633 (ESTABLISHED)

tclsh 117761 SO-user 3u IPv4 49996417 0t0 TCP X.X.X.X:34657->X.X.X.X:7736 (ESTABLISHED)

tclsh 117761 SO-user 4u IPv4 50000284 0t0 TCP X.X.X.X:8502 (LISTEN)

tclsh 117761 SO-user 6u IPv4 50042254 0t0 TCP X.X.X.X:8502->X.X.X.X:56484 (ESTABLISHED)

tclsh 117803 SO-user 3u IPv4 50000292 0t0 TCP X.X.X.X:57395->X.X.X.X:7736 (ESTABLISHED)

tclsh 117803 SO-user 4u IPv4 49996418 0t0 TCP X.X.X.X:8503 (LISTEN)

tclsh 117846 SO-user 3u IPv4 49987100 0t0 TCP X.X.X.X:36108->X.X.X.X:7736 (ESTABLISHED)

tclsh 117846 SO-user 4u IPv4 49996421 0t0 TCP X.X.X.X:8504 (LISTEN)

tclsh 117889 SO-user 3u IPv4 49987123 0t0 TCP X.X.X.X:39339->X.X.X.X:7736 (ESTABLISHED)

tclsh 117889 SO-user 4u IPv4 49981410 0t0 TCP X.X.X.X:8505 (LISTEN)

tclsh 117931 SO-user 3u IPv4 49953764 0t0 TCP X.X.X.X:48921->X.X.X.X:7736 (ESTABLISHED)

tclsh 117931 SO-user 4u IPv4 49981413 0t0 TCP X.X.X.X:8506 (LISTEN)

tclsh 117931 SO-user 6u IPv4 50014339 0t0 TCP X.X.X.X:8506->X.X.X.X:42954 (ESTABLISHED)

tclsh 117974 SO-user 3u IPv4 49986244 0t0 TCP X.X.X.X:34521->X.X.X.X:7736 (ESTABLISHED)

tclsh 117974 SO-user 4u IPv4 50014214 0t0 TCP X.X.X.X:8507 (LISTEN)

tclsh 117974 SO-user 6u IPv4 50004385 0t0 TCP X.X.X.X:8507->X.X.X.X:56172 (ESTABLISHED)

tclsh 118016 SO-user 3u IPv4 49998516 0t0 TCP X.X.X.X:57963->X.X.X.X:7736 (ESTABLISHED)

tclsh 118016 SO-user 4u IPv4 49979369 0t0 TCP X.X.X.X:8508 (LISTEN)

tclsh 118016 SO-user 6u IPv4 50000837 0t0 TCP X.X.X.X:8508->X.X.X.X:40748 (ESTABLISHED)

tclsh 118132 SO-user 3u IPv4 50010205 0t0 TCP X.X.X.X:55957->X.X.X.X:7736 (ESTABLISHED)

tclsh 118132 SO-user 4u IPv4 50014221 0t0 TCP X.X.X.X:8701 (LISTEN)

tclsh 118132 SO-user 6u IPv4 50014228 0t0 TCP X.X.X.X:8701->X.X.X.X:53608 (ESTABLISHED)

tclsh 118174 SO-user 3u IPv4 50010228 0t0 TCP X.X.X.X:58467->X.X.X.X:7736 (ESTABLISHED)

tclsh 118174 SO-user 4u IPv4 50014224 0t0 TCP X.X.X.X:8702 (LISTEN)

tclsh 118174 SO-user 6u IPv4 50014227 0t0 TCP X.X.X.X:8702->X.X.X.X:38904 (ESTABLISHED)

tclsh 118217 SO-user 3u IPv4 49997554 0t0 TCP X.X.X.X:48950->X.X.X.X:7736 (ESTABLISHED)

tclsh 118217 SO-user 4u IPv4 50001618 0t0 TCP X.X.X.X:8703 (LISTEN)

tclsh 118217 SO-user 6u IPv4 50003669 0t0 TCP X.X.X.X:8703->X.X.X.X:55060 (ESTABLISHED)

tclsh 118259 SO-user 3u IPv4 49971130 0t0 TCP X.X.X.X:34919->X.X.X.X:7736 (ESTABLISHED)

tclsh 118259 SO-user 4u IPv4 50001621 0t0 TCP X.X.X.X:8704 (LISTEN)

tclsh 118259 SO-user 6u IPv4 50003670 0t0 TCP X.X.X.X:8704->X.X.X.X:47354 (ESTABLISHED)

tclsh 118302 SO-user 3u IPv4 49964895 0t0 TCP X.X.X.X:42924->X.X.X.X:7736 (ESTABLISHED)

tclsh 118302 SO-user 4u IPv4 50000358 0t0 TCP X.X.X.X:8705 (LISTEN)

tclsh 118302 SO-user 6u IPv4 49996430 0t0 TCP X.X.X.X:8705->X.X.X.X:40060 (ESTABLISHED)

tclsh 118345 SO-user 3u IPv4 49994426 0t0 TCP X.X.X.X:48416->X.X.X.X:7736 (ESTABLISHED)

tclsh 118345 SO-user 4u IPv4 50004300 0t0 TCP X.X.X.X:8706 (LISTEN)

tclsh 118345 SO-user 6u IPv4 50012398 0t0 TCP X.X.X.X:8706->X.X.X.X:53328 (ESTABLISHED)

tclsh 118388 SO-user 3u IPv4 50011249 0t0 TCP X.X.X.X:55281->X.X.X.X:7736 (ESTABLISHED)

tclsh 118388 SO-user 4u IPv4 50000364 0t0 TCP X.X.X.X:8707 (LISTEN)

tclsh 118388 SO-user 6u IPv4 50012401 0t0 TCP X.X.X.X:8707->X.X.X.X:45947 (ESTABLISHED)

tclsh 118430 SO-user 3u IPv4 50014253 0t0 TCP X.X.X.X:55657->X.X.X.X:7736 (ESTABLISHED)

tclsh 118430 SO-user 4u IPv4 50005133 0t0 TCP X.X.X.X:8708 (LISTEN)

tclsh 118430 SO-user 6u IPv4 50001696 0t0 TCP X.X.X.X:8708->X.X.X.X:39239 (ESTABLISHED)

sshd 122435 root 3u IPv4 50619109 0t0 TCP X.X.X.X:ssh_port->X.X.X.X:52259 (ESTABLISHED)

sshd 122471 SO-user 3u IPv4 50619109 0t0 TCP X.X.X.X:ssh_port->X.X.X.X:52259 (ESTABLISHED)

barnyard2 127217 SO-user 3u IPv4 50012402 0t0 TCP X.X.X.X:39239->X.X.X.X:8708 (ESTABLISHED)

syslog-ng 135816 root 28u IPv4 48118587 0t0 TCP *:514 (LISTEN)

syslog-ng 135816 root 29u IPv4 48118588 0t0 UDP *:514

bro 144198 SO-user 4u IPv4 765094 0t0 UDP X.X.X.X:58508->X.X.X.X:53

bro 144200 SO-user 0u IPv4 756030 0t0 TCP *:47761 (LISTEN)

bro 144200 SO-user 1u IPv6 756031 0t0 TCP *:47761 (LISTEN)

bro 144200 SO-user 2u IPv4 756137 0t0 TCP X.X.X.X:47761->X.X.X.X:37860 (ESTABLISHED)

bro 144200 SO-user 4u IPv4 765094 0t0 UDP X.X.X.X:58508->X.X.X.X:53

bro 144200 SO-user 268u IPv4 734915 0t0 TCP X.X.X.X:47761->X.X.X.X:37861 (ESTABLISHED)

bro 144200 SO-user 273u IPv4 734918 0t0 TCP X.X.X.X:47761->X.X.X.X:37863 (ESTABLISHED)

bro 144200 SO-user 278u IPv4 766993 0t0 TCP X.X.X.X:47761->X.X.X.X:37865 (ESTABLISHED)

bro 144200 SO-user 283u IPv4 734921 0t0 TCP X.X.X.X:47761->X.X.X.X:37867 (ESTABLISHED)

bro 144200 SO-user 288u IPv4 729041 0t0 TCP X.X.X.X:47761->X.X.X.X:37870 (ESTABLISHED)

bro 144200 SO-user 293u IPv4 765111 0t0 TCP X.X.X.X:47761->X.X.X.X:37871 (ESTABLISHED)

bro 144291 SO-user 4u IPv4 717473 0t0 UDP X.X.X.X:47457->X.X.X.X:53

bro 144298 SO-user 0u IPv4 751986 0t0 TCP X.X.X.X:37860->X.X.X.X:47761 (ESTABLISHED)

bro 144298 SO-user 4u IPv4 717473 0t0 UDP X.X.X.X:47457->X.X.X.X:53

bro 144298 SO-user 266u IPv4 751991 0t0 TCP *:47762 (LISTEN)

bro 144298 SO-user 267u IPv6 751992 0t0 TCP *:47762 (LISTEN)

bro 144298 SO-user 268u IPv4 752029 0t0 TCP X.X.X.X:47762->X.X.X.X:42678 (ESTABLISHED)

bro 144298 SO-user 273u IPv4 757402 0t0 TCP X.X.X.X:47762->X.X.X.X:42680 (ESTABLISHED)

bro 144298 SO-user 278u IPv4 763190 0t0 TCP X.X.X.X:47762->X.X.X.X:42682 (ESTABLISHED)

bro 144298 SO-user 283u IPv4 734924 0t0 TCP X.X.X.X:47762->X.X.X.X:42684 (ESTABLISHED)

bro 144298 SO-user 288u IPv4 734927 0t0 TCP X.X.X.X:47762->X.X.X.X:42685 (ESTABLISHED)

bro 144298 SO-user 293u IPv4 721916 0t0 TCP X.X.X.X:47762->X.X.X.X:42688 (ESTABLISHED)

bro 144390 SO-user 4u IPv4 766053 0t0 UDP X.X.X.X:39657->X.X.X.X:53

bro 144397 SO-user 4u IPv4 756144 0t0 UDP X.X.X.X:60760->X.X.X.X:53

bro 144400 SO-user 4u IPv4 763153 0t0 UDP X.X.X.X:41845->X.X.X.X:53

bro 144403 SO-user 4u IPv4 739164 0t0 UDP X.X.X.X:43910->X.X.X.X:53

bro 144405 SO-user 4u IPv4 748106 0t0 UDP X.X.X.X:60105->X.X.X.X:53

bro 144407 SO-user 4u IPv4 732155 0t0 UDP X.X.X.X:47052->X.X.X.X:53

bro 144460 SO-user 0u IPv4 764088 0t0 TCP X.X.X.X:37861->X.X.X.X:47761 (ESTABLISHED)

bro 144460 SO-user 4u IPv4 739164 0t0 UDP X.X.X.X:43910->X.X.X.X:53

bro 144460 SO-user 266u IPv4 764091 0t0 TCP X.X.X.X:42678->X.X.X.X:47762 (ESTABLISHED)

bro 144460 SO-user 271u IPv4 764096 0t0 TCP *:47766 (LISTEN)

bro 144460 SO-user 272u IPv6 764097 0t0 TCP *:47766 (LISTEN)

bro 144461 SO-user 0u IPv4 766983 0t0 TCP X.X.X.X:37863->X.X.X.X:47761 (ESTABLISHED)

bro 144461 SO-user 4u IPv4 732155 0t0 UDP X.X.X.X:47052->X.X.X.X:53

bro 144461 SO-user 266u IPv4 766986 0t0 TCP X.X.X.X:42680->X.X.X.X:47762 (ESTABLISHED)

bro 144461 SO-user 271u IPv4 766991 0t0 TCP *:47768 (LISTEN)

bro 144461 SO-user 272u IPv6 766992 0t0 TCP *:47768 (LISTEN)

bro 144468 SO-user 0u IPv4 766068 0t0 TCP X.X.X.X:37865->X.X.X.X:47761 (ESTABLISHED)

bro 144468 SO-user 4u IPv4 748106 0t0 UDP X.X.X.X:60105->X.X.X.X:53

bro 144468 SO-user 266u IPv4 766071 0t0 TCP X.X.X.X:42682->X.X.X.X:47762 (ESTABLISHED)

bro 144468 SO-user 271u IPv4 766076 0t0 TCP *:47767 (LISTEN)

bro 144468 SO-user 272u IPv6 766077 0t0 TCP *:47767 (LISTEN)

bro 144486 SO-user 0u IPv4 764098 0t0 TCP X.X.X.X:37867->X.X.X.X:47761 (ESTABLISHED)

bro 144486 SO-user 4u IPv4 756144 0t0 UDP X.X.X.X:60760->X.X.X.X:53

bro 144486 SO-user 266u IPv4 764101 0t0 TCP X.X.X.X:42684->X.X.X.X:47762 (ESTABLISHED)

bro 144486 SO-user 271u IPv4 764106 0t0 TCP *:47764 (LISTEN)

bro 144486 SO-user 272u IPv6 764107 0t0 TCP *:47764 (LISTEN)

bro 144492 SO-user 0u IPv4 721906 0t0 TCP X.X.X.X:42685->X.X.X.X:47762 (ESTABLISHED)

bro 144492 SO-user 4u IPv4 763153 0t0 UDP X.X.X.X:41845->X.X.X.X:53

bro 144492 SO-user 266u IPv4 721909 0t0 TCP X.X.X.X:37870->X.X.X.X:47761 (ESTABLISHED)

bro 144492 SO-user 271u IPv4 721914 0t0 TCP *:47765 (LISTEN)

bro 144492 SO-user 272u IPv6 721915 0t0 TCP *:47765 (LISTEN)

bro 144499 SO-user 0u IPv4 740136 0t0 TCP X.X.X.X:37871->X.X.X.X:47761 (ESTABLISHED)

bro 144499 SO-user 4u IPv4 766053 0t0 UDP X.X.X.X:39657->X.X.X.X:53

bro 144499 SO-user 266u IPv4 740139 0t0 TCP X.X.X.X:42688->X.X.X.X:47762 (ESTABLISHED)

bro 144499 SO-user 271u IPv4 740144 0t0 TCP *:47763 (LISTEN)

bro 144499 SO-user 272u IPv6 740145 0t0 TCP *:47763 (LISTEN)

barnyard2 145055 SO-user 3u IPv4 752079 0t0 TCP X.X.X.X:36061->X.X.X.X:8401 (CLOSE_WAIT)

barnyard2 145071 SO-user 3u IPv4 38808405 0t0 TCP X.X.X.X:38939->X.X.X.X:8402 (CLOSE_WAIT)

barnyard2 145087 SO-user 3u IPv4 771103 0t0 TCP X.X.X.X:50788->X.X.X.X:8403 (CLOSE_WAIT)

barnyard2 145103 SO-user 3u IPv4 25813598 0t0 TCP X.X.X.X:48618->X.X.X.X:8404 (CLOSE_WAIT)

barnyard2 145119 SO-user 3u IPv4 38820277 0t0 TCP X.X.X.X:35627->X.X.X.X:8405 (CLOSE_WAIT)

barnyard2 145136 SO-user 3u IPv4 38815530 0t0 TCP X.X.X.X:37092->X.X.X.X:8406 (CLOSE_WAIT)

barnyard2 145152 SO-user 3u IPv4 763245 0t0 TCP X.X.X.X:58163->X.X.X.X:8407 (CLOSE_WAIT)

barnyard2 145168 SO-user 3u IPv4 38818405 0t0 TCP X.X.X.X:58650->X.X.X.X:8408 (CLOSE_WAIT)

barnyard2 145600 SO-user 3u IPv4 50007495 0t0 TCP X.X.X.X:57633->X.X.X.X:8501 (ESTABLISHED)

barnyard2 145693 SO-user 3u IPv4 50007511 0t0 TCP X.X.X.X:42954->X.X.X.X:8506 (ESTABLISHED)

=========================================================================

CPU Usage

=========================================================================

Load average for the last 1, 5, and 15 minutes:

8.78 9.69 10.56

Processing units: 32

If load average is higher than processing units,

then tune until load average is lower than processing units.

top - 17:05:59 up 14 days, 9 min, 2 users, load average: 8.78, 9.69, 10.56

Tasks: 588 total, 8 running, 580 sleeping, 0 stopped, 0 zombie

%Cpu(s): 19.2 us, 2.4 sy, 0.0 ni, 77.8 id, 0.1 wa, 0.0 hi, 0.4 si, 0.0 st

KiB Mem: 39616294+total, 38768364+used, 8479300 free, 531864 buffers

KiB Swap: 40254668+total, 5600080 used, 39694659+free. 30170022+cached Mem

%CPU %MEM COMMAND

99.9 0.0 /usr/bin/indexer --config /etc/sphinxsearch/sphinx.conf --rotate temp_2

56.1 0.3 snort -c /etc/nsm/SO-server-eth7/snort.conf -u SO-user -g SO-user -i eth7 -l /nsm/sensor_data/SO-server-eth7/snort-8 --perfmon-file /nsm/sensor_data/SO-server-eth7/snort-8.stats -U

45.1 0.3 snort -c /etc/nsm/SO-server-eth7/snort.conf -u SO-user -g SO-user -i eth7 -l /nsm/sensor_data/SO-server-eth7/snort-5 --perfmon-file /nsm/sensor_data/SO-server-eth7/snort-5.stats -U

35.8 0.3 snort -c /etc/nsm/SO-server-eth7/snort.conf -u SO-user -g SO-user -i eth7 -l /nsm/sensor_data/SO-server-eth7/snort-4 --perfmon-file /nsm/sensor_data/SO-server-eth7/snort-4.stats -U

33.1 0.3 snort -c /etc/nsm/SO-server-eth7/snort.conf -u SO-user -g SO-user -i eth7 -l /nsm/sensor_data/SO-server-eth7/snort-1 --perfmon-file /nsm/sensor_data/SO-server-eth7/snort-1.stats -U

30.1 0.3 snort -c /etc/nsm/SO-server-eth7/snort.conf -u SO-user -g SO-user -i eth7 -l /nsm/sensor_data/SO-server-eth7/snort-3 --perfmon-file /nsm/sensor_data/SO-server-eth7/snort-3.stats -U

26.8 0.3 snort -c /etc/nsm/SO-server-eth7/snort.conf -u SO-user -g SO-user -i eth7 -l /nsm/sensor_data/SO-server-eth7/snort-7 --perfmon-file /nsm/sensor_data/SO-server-eth7/snort-7.stats -U

25.2 0.3 snort -c /etc/nsm/SO-server-eth5/snort.conf -u SO-user -g SO-user -i eth5 -l /nsm/sensor_data/SO-server-eth5/snort-8 --perfmon-file /nsm/sensor_data/SO-server-eth5/snort-8.stats -U

21.9 1.1 /opt/bro/bin/bro -i eth7 -U .status -p broctl -p broctl-live -p local -p SO-server-eth7-1 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto

21.7 1.0 /opt/bro/bin/bro -i eth5 -U .status -p broctl -p broctl-live -p local -p SO-server-eth5-1 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto

21.2 1.3 /opt/bro/bin/bro -i eth7 -U .status -p broctl -p broctl-live -p local -p SO-server-eth7-2 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto

21.1 1.0 /opt/bro/bin/bro -i eth5 -U .status -p broctl -p broctl-live -p local -p SO-server-eth5-2 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto

20.6 0.3 snort -c /etc/nsm/SO-server-eth7/snort.conf -u SO-user -g SO-user -i eth7 -l /nsm/sensor_data/SO-server-eth7/snort-2 --perfmon-file /nsm/sensor_data/SO-server-eth7/snort-2.stats -U

20.5 0.3 snort -c /etc/nsm/SO-server-eth5/snort.conf -u SO-user -g SO-user -i eth5 -l /nsm/sensor_data/SO-server-eth5/snort-4 --perfmon-file /nsm/sensor_data/SO-server-eth5/snort-4.stats -U

19.2 0.3 snort -c /etc/nsm/SO-server-eth5/snort.conf -u SO-user -g SO-user -i eth5 -l /nsm/sensor_data/SO-server-eth5/snort-3 --perfmon-file /nsm/sensor_data/SO-server-eth5/snort-3.stats -U

18.1 0.3 snort -c /etc/nsm/SO-server-eth7/snort.conf -u SO-user -g SO-user -i eth7 -l /nsm/sensor_data/SO-server-eth7/snort-6 --perfmon-file /nsm/sensor_data/SO-server-eth7/snort-6.stats -U

17.0 0.3 snort -c /etc/nsm/SO-server-eth5/snort.conf -u SO-user -g SO-user -i eth5 -l /nsm/sensor_data/SO-server-eth5/snort-7 --perfmon-file /nsm/sensor_data/SO-server-eth5/snort-7.stats -U

15.7 1.1 /opt/bro/bin/bro -i eth4 -U .status -p broctl -p broctl-live -p local -p SO-server-eth4-2 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto

15.1 0.9 /opt/bro/bin/bro -i eth4 -U .status -p broctl -p broctl-live -p local -p SO-server-eth4-1 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto

15.1 0.3 snort -c /etc/nsm/SO-server-eth5/snort.conf -u SO-user -g SO-user -i eth5 -l /nsm/sensor_data/SO-server-eth5/snort-2 --perfmon-file /nsm/sensor_data/SO-server-eth5/snort-2.stats -U

15.1 0.3 snort -c /etc/nsm/SO-server-eth5/snort.conf -u SO-user -g SO-user -i eth5 -l /nsm/sensor_data/SO-server-eth5/snort-6 --perfmon-file /nsm/sensor_data/SO-server-eth5/snort-6.stats -U

13.2 0.3 snort -c /etc/nsm/SO-server-eth5/snort.conf -u SO-user -g SO-user -i eth5 -l /nsm/sensor_data/SO-server-eth5/snort-1 --perfmon-file /nsm/sensor_data/SO-server-eth5/snort-1.stats -U

13.1 0.0 /usr/sbin/mysqld

11.9 0.3 snort -c /etc/nsm/SO-server-eth5/snort.conf -u SO-user -g SO-user -i eth5 -l /nsm/sensor_data/SO-server-eth5/snort-5 --perfmon-file /nsm/sensor_data/SO-server-eth5/snort-5.stats -U

10.2 1.0 netsniff-ng -i eth5 -o /nsm/sensor_data/SO-server-eth5/dailylogs/2016-04-11/ --user 1001 --group 1001 -s --prefix snort.log. --verbose --ring-size 4000 iB --interval 150 iB --mmap

6.9 0.0 [kswapd0]

4.3 0.0 /opt/bro/bin/bro -U .status -p broctl -p broctl-live -p local -p manager local.bro broctl base/frameworks/cluster local-manager.bro broctl/auto

3.9 0.0 perl /opt/elsa/node/elsa.pl -c /etc/elsa_node.conf

3.5 0.2 snort -c /etc/nsm/SO-server-eth4/snort.conf -u SO-user -g SO-user -i eth4 -l /nsm/sensor_data/SO-server-eth4/snort-6 --perfmon-file /nsm/sensor_data/SO-server-eth4/snort-6.stats -U

3.4 0.3 snort -c /etc/nsm/SO-server-eth4/snort.conf -u SO-user -g SO-user -i eth4 -l /nsm/sensor_data/SO-server-eth4/snort-8 --perfmon-file /nsm/sensor_data/SO-server-eth4/snort-8.stats -U

3.3 0.3 snort -c /etc/nsm/SO-server-eth4/snort.conf -u SO-user -g SO-user -i eth4 -l /nsm/sensor_data/SO-server-eth4/snort-1 --perfmon-file /nsm/sensor_data/SO-server-eth4/snort-1.stats -U

3.0 0.0 [kswapd1]

2.9 0.3 snort -c /etc/nsm/SO-server-eth4/snort.conf -u SO-user -g SO-user -i eth4 -l /nsm/sensor_data/SO-server-eth4/snort-5 --perfmon-file /nsm/sensor_data/SO-server-eth4/snort-5.stats -U

2.8 1.0 netsniff-ng -i eth4 -o /nsm/sensor_data/SO-server-eth4/dailylogs/2016-04-11/ --user 1001 --group 1001 -s --prefix snort.log. --verbose --ring-size 4000 iB --interval 150 iB --mmap

2.8 0.0 /usr/sbin/syslog-ng -p /var/run/syslog-ng.pid

2.8 0.3 snort -c /etc/nsm/SO-server-eth4/snort.conf -u SO-user -g SO-user -i eth4 -l /nsm/sensor_data/SO-server-eth4/snort-2 --perfmon-file /nsm/sensor_data/SO-server-eth4/snort-2.stats -U

2.7 0.3 snort -c /etc/nsm/SO-server-eth4/snort.conf -u SO-user -g SO-user -i eth4 -l /nsm/sensor_data/SO-server-eth4/snort-3 --perfmon-file /nsm/sensor_data/SO-server-eth4/snort-3.stats -U

2.6 0.3 snort -c /etc/nsm/SO-server-eth4/snort.conf -u SO-user -g SO-user -i eth4 -l /nsm/sensor_data/SO-server-eth4/snort-4 --perfmon-file /nsm/sensor_data/SO-server-eth4/snort-4.stats -U

2.5 0.2 snort -c /etc/nsm/SO-server-eth4/snort.conf -u SO-user -g SO-user -i eth4 -l /nsm/sensor_data/SO-server-eth4/snort-7 --perfmon-file /nsm/sensor_data/SO-server-eth4/snort-7.stats -U

2.4 0.0 htop

1.5 0.9 /usr/bin/searchd --nodetach

1.5 0.0 /opt/bro/bin/bro -U .status -p broctl -p broctl-live -p local -p proxy local.bro broctl base/frameworks/cluster local-proxy broctl/auto

1.3 0.0 perl /opt/elsa/web/cron.pl -c /etc/elsa_web.conf

0.6 0.0 [jbd2/dm-0-8]

0.4 0.0 [kworker/u290:2]

0.3 0.0 [rcu_sched]

0.2 0.0 [khugepaged]

0.2 0.0 /usr/lib/accountsservice/accounts-daemon

0.2 0.0 /opt/bro/bin/bro -U .status -p broctl -p broctl-live -p local -p manager local.bro broctl base/frameworks/cluster local-manager.bro broctl/auto

0.1 0.0 [rcuos/0]

0.0 0.0 /sbin/init

0.0 0.0 [kthreadd]

0.0 0.0 [ksoftirqd/0]

0.0 0.0 [kworker/0:0H]

0.0 0.0 [rcu_bh]

0.0 0.0 [rcuob/0]

0.0 0.0 [rcuos/12]

0.0 0.0 [rcuos/21]

0.0 0.0 [bioset]

0.0 0.0 [ext4-rsv-conver]

0.0 0.0 upstart-udev-bridge --daemon

0.0 0.0 /lib/systemd/systemd-udevd --daemon

0.0 0.0 [ext4-rsv-conver]

0.0 0.0 upstart-file-bridge --daemon

0.0 0.0 dbus-daemon --system --fork

0.0 0.0 [edac-poller]

0.0 0.0 [kworker/21:1H]

0.0 0.0 /usr/sbin/bluetoothd

0.0 0.0 /lib/systemd/systemd-logind

0.0 0.0 [kmpathd]

0.0 0.0 [kmpath_handlerd]

0.0 0.0 avahi-daemon: running [SO-server.local]

0.0 0.0 avahi-daemon: chroot helper

0.0 0.0 [krfcommd]

0.0 0.0 [kvm-irqfd-clean]

0.0 0.0 upstart-socket-bridge --daemon

0.0 0.0 /sbin/getty -8 38400 tty4

0.0 0.0 /sbin/getty -8 38400 tty5

0.0 0.0 /sbin/getty -8 38400 tty2

0.0 0.0 /sbin/getty -8 38400 tty3

0.0 0.0 /sbin/getty -8 38400 tty6

0.0 0.0 /usr/sbin/sshd -D

0.0 0.0 su -s /bin/sh -c exec "$0" "$@" sphinxsearch -- /usr/bin/searchd --nodetach

0.0 0.0 cron

0.0 0.0 acpid -c /etc/acpi/events -s /var/run/acpid.socket

0.0 0.0 [kauditd]

0.0 0.0 /usr/sbin/irqbalance

0.0 0.0 /usr/sbin/cups-browsed

0.0 0.0 /usr/sbin/kerneloops

0.0 0.0 /var/ossec/bin/ossec-csyslogd

0.0 0.0 /var/ossec/bin/ossec-execd

0.0 0.0 /var/ossec/bin/ossec-analysisd

0.0 0.0 /var/ossec/bin/ossec-logcollector

0.0 0.0 lightdm

0.0 0.0 /var/ossec/bin/ossec-syscheckd

0.0 0.0 /var/ossec/bin/ossec-monitord

0.0 0.0 /usr/bin/X -core :0 -seat seat0 -auth /var/run/lightdm/root/:0 -nolisten tcp vt7 -novtswitch

0.0 0.0 /usr/lib/policykit-1/polkitd --no-debug

0.0 0.0 lightdm --session-child 16 19

0.0 0.0 /bin/sh /usr/lib/lightdm/lightdm-greeter-session /usr/sbin/lightdm-gtk-greeter

0.0 0.0 //bin/dbus-daemon --fork --print-pid 5 --print-address 7 --session

0.0 0.0 /usr/sbin/lightdm-gtk-greeter

0.0 0.0 starman master -I/opt/elsa/web/lib --user=www-data --listen :3154 --daemonize --pid /var/run/starman.pid --error-log /var/log/starman.log /opt/elsa/web/lib/Web.psgi

0.0 0.0 starman worker -I/opt/elsa/web/lib --user=www-data --listen :3154 --daemonize --pid /var/run/starman.pid --error-log /var/log/starman.log /opt/elsa/web/lib/Web.psgi

0.0 0.0 /usr/lib/at-spi2-core/at-spi-bus-launcher

0.0 0.0 /bin/dbus-daemon --config-file=/etc/at-spi2/accessibility.conf --nofork --print-address 3

0.0 0.0 /usr/lib/at-spi2-core/at-spi2-registryd --use-gnome-session

0.0 0.0 /usr/lib/gvfs/gvfsd

0.0 0.0 /usr/lib/gvfs/gvfsd-fuse /run/user/112/gvfs -f -o big_writes

0.0 0.0 lightdm --session-child 12 19

0.0 0.0 [kworker/6:1H]

0.0 0.0 /usr/sbin/cupsd -f

0.0 0.0 /usr/lib/autossh/autossh -M 0 -q -N -o ServerAliveInterval 60 -o ServerAliveCountMax 3 -i /root/.ssh/securityonion -L 3306:X.X.X.X:3306 -R 50001:localhost:3154 SO-...@X.X.X.X

0.0 0.0 /usr/bin/ssh -q -N -o ServerAliveInterval 60 -o ServerAliveCountMax 3 -i /root/.ssh/securityonion -L 3306:X.X.X.X:3306 -R 50001:localhost:3154 SO-...@X.X.X.X

0.0 0.0 /usr/sbin/ntpd -p /var/run/ntpd.pid -g -u 117:126

0.0 0.0 /sbin/getty -8 38400 tty1

0.0 0.0 [kworker/10:1H]

0.0 0.0 [kworker/2:1H]

0.0 0.0 [kworker/8:1H]

0.0 0.0 [kworker/12:1H]

0.0 0.0 [kworker/4:1H]

0.0 0.0 [kworker/26:1H]

0.0 0.0 [kworker/12:0]

0.0 0.0 /usr/bin/python /usr/bin/salt-minion

0.0 0.0 [kworker/28:1H]

0.0 0.0 [kworker/0:1H]

0.0 0.0 [kworker/3:1H]

0.0 0.0 [kworker/13:1H]

0.0 0.0 [kworker/1:1H]

0.0 0.0 [kworker/9:1H]

0.0 0.0 [kworker/14:1H]

0.0 0.0 [kworker/7:1H]

0.0 0.0 [kworker/u288:1]

0.0 0.0 [kworker/11:1H]

0.0 0.0 [kworker/27:1H]

0.0 0.0 [kworker/30:1H]

0.0 0.0 [kworker/19:1H]

0.0 0.0 [kworker/17:1H]

0.0 0.0 [kworker/5:1H]

0.0 0.0 [kworker/31:1H]

0.0 0.0 [kworker/20:1H]

0.0 0.0 [kworker/18:1H]

0.0 0.0 [kworker/u288:0]

0.0 0.0 [kworker/7:1]

0.0 0.0 [kworker/25:1H]

0.0 0.0 [kworker/22:1H]

0.0 0.0 [kworker/13:2]

0.0 0.0 [kworker/16:1H]

0.0 0.0 [kworker/25:1]

0.0 0.0 [kworker/24:1H]

0.0 0.0 [kworker/19:2]

0.0 0.0 [kworker/16:1]

0.0 0.0 [kworker/15:1]

0.0 0.0 [kworker/7:2]

0.0 0.0 su - SO-user -- /usr/bin/pcap_agent.tcl -c /etc/nsm/SO-server-eth4/pcap_agent.conf

0.0 0.0 tclsh /usr/bin/pcap_agent.tcl -c /etc/nsm/SO-server-eth4/pcap_agent.conf

0.0 0.0 [kworker/u290:0]

0.0 0.0 barnyard2 -c /etc/nsm/SO-server-eth5/barnyard2-2.conf -u SO-user -g SO-user -d /nsm/sensor_data/SO-server-eth5/snort-2 -f snort.unified2 -w /etc/nsm/SO-server-eth5/barnyard2.waldo-2 -i 2 -U

0.0 0.0 [kworker/29:1H]

0.0 0.0 [kworker/9:0]

0.0 0.0 [kworker/4:2]

0.0 0.0 [kworker/23:2]

0.0 0.0 [kworker/21:2]

0.0 0.0 [kworker/15:1H]

0.0 0.0 barnyard2 -c /etc/nsm/SO-server-eth7/barnyard2-1.conf -u SO-user -g SO-user -d /nsm/sensor_data/SO-server-eth7/snort-1 -f snort.unified2 -w /etc/nsm/SO-server-eth7/barnyard2.waldo-1 -i 1 -U

0.0 0.0 barnyard2 -c /etc/nsm/SO-server-eth7/barnyard2-2.conf -u SO-user -g SO-user -d /nsm/sensor_data/SO-server-eth7/snort-2 -f snort.unified2 -w /etc/nsm/SO-server-eth7/barnyard2.waldo-2 -i 2 -U

0.0 0.0 barnyard2 -c /etc/nsm/SO-server-eth7/barnyard2-3.conf -u SO-user -g SO-user -d /nsm/sensor_data/SO-server-eth7/snort-3 -f snort.unified2 -w /etc/nsm/SO-server-eth7/barnyard2.waldo-3 -i 3 -U

0.0 0.0 barnyard2 -c /etc/nsm/SO-server-eth7/barnyard2-4.conf -u SO-user -g SO-user -d /nsm/sensor_data/SO-server-eth7/snort-4 -f snort.unified2 -w /etc/nsm/SO-server-eth7/barnyard2.waldo-4 -i 4 -U

0.0 0.0 barnyard2 -c /etc/nsm/SO-server-eth7/barnyard2-5.conf -u SO-user -g SO-user -d /nsm/sensor_data/SO-server-eth7/snort-5 -f snort.unified2 -w /etc/nsm/SO-server-eth7/barnyard2.waldo-5 -i 5 -U

0.0 0.0 barnyard2 -c /etc/nsm/SO-server-eth7/barnyard2-6.conf -u SO-user -g SO-user -d /nsm/sensor_data/SO-server-eth7/snort-6 -f snort.unified2 -w /etc/nsm/SO-server-eth7/barnyard2.waldo-6 -i 6 -U

0.0 0.0 barnyard2 -c /etc/nsm/SO-server-eth7/barnyard2-7.conf -u SO-user -g SO-user -d /nsm/sensor_data/SO-server-eth7/snort-7 -f snort.unified2 -w /etc/nsm/SO-server-eth7/barnyard2.waldo-7 -i 7 -U

0.0 0.0 [kworker/4:0]

0.0 0.0 [kworker/21:0]

0.0 0.0 [kworker/6:1]

0.0 0.0 [kworker/12:1]

0.0 0.0 [kworker/2:1]

0.0 0.0 [kworker/14:2]

0.0 0.0 [kworker/0:0]

0.0 0.0 [kworker/18:2]

0.0 0.0 [kworker/23:1H]

0.0 0.0 [kworker/28:0]

0.0 0.0 [kworker/27:0]

0.0 0.0 [kworker/6:0]

0.0 0.0 [kworker/18:0]

0.0 0.0 [kworker/30:0]

0.0 0.0 [kworker/24:0]

0.0 0.0 [kworker/31:0]

0.0 0.0 [kworker/8:2]

0.0 0.0 sshd: SO-user [priv]

0.0 0.0 sshd: SO-user@pts/3

0.0 0.0 -bash

0.0 0.0 [kworker/17:0]

0.0 0.0 [kworker/14:1]

0.0 0.0 [kworker/22:1]

0.0 0.0 [kworker/1:1]

0.0 0.0 [kworker/15:2]

0.0 0.0 [kworker/31:1]

0.0 0.0 [kworker/27:1]

0.0 0.0 [kworker/29:0]

0.0 0.0 [kworker/0:1]

0.0 0.0 [kworker/20:2]

0.0 0.0 /bin/sh -c sh /opt/elsa/contrib/securityonion/contrib/securityonion-elsa-syslog-ng.sh

0.0 0.0 sh /opt/elsa/contrib/securityonion/contrib/securityonion-elsa-syslog-ng.sh

0.0 0.0 [kworker/29:1]

0.0 0.0 barnyard2 -c /etc/nsm/SO-server-eth5/barnyard2-7.conf -u SO-user -g SO-user -d /nsm/sensor_data/SO-server-eth5/snort-7 -f snort.unified2 -w /etc/nsm/SO-server-eth5/barnyard2.waldo-7 -i 7 -U

0.0 0.0 barnyard2 -c /etc/nsm/SO-server-eth5/barnyard2-8.conf -u SO-user -g SO-user -d /nsm/sensor_data/SO-server-eth5/snort-8 -f snort.unified2 -w /etc/nsm/SO-server-eth5/barnyard2.waldo-8 -i 8 -U

0.0 0.0 [kworker/20:0]

0.0 0.0 [kworker/11:2]

0.0 0.0 [kworker/8:1]

0.0 0.0 [kworker/10:1]

0.0 0.0 su - SO-user -- /usr/bin/ossec_agent.tcl -o -f /var/ossec/logs/alerts/alerts.log -i X.X.X.X -p 5 -c /etc/nsm/ossec/ossec_agent.conf

0.0 0.0 tclsh /usr/bin/ossec_agent.tcl -o -f /var/ossec/logs/alerts/alerts.log -i X.X.X.X -p 5 -c /etc/nsm/ossec/ossec_agent.conf

0.0 0.0 tail -n 0 -F /var/ossec/logs/alerts/alerts.log

0.0 0.0 barnyard2 -c /etc/nsm/SO-server-eth5/barnyard2-3.conf -u SO-user -g SO-user -d /nsm/sensor_data/SO-server-eth5/snort-3 -f snort.unified2 -w /etc/nsm/SO-server-eth5/barnyard2.waldo-3 -i 3 -U

0.0 0.0 barnyard2 -c /etc/nsm/SO-server-eth5/barnyard2-4.conf -u SO-user -g SO-user -d /nsm/sensor_data/SO-server-eth5/snort-4 -f snort.unified2 -w /etc/nsm/SO-server-eth5/barnyard2.waldo-4 -i 4 -U

0.0 0.0 barnyard2 -c /etc/nsm/SO-server-eth5/barnyard2-5.conf -u SO-user -g SO-user -d /nsm/sensor_data/SO-server-eth5/snort-5 -f snort.unified2 -w /etc/nsm/SO-server-eth5/barnyard2.waldo-5 -i 5 -U

0.0 0.0 [kworker/3:0]

0.0 0.0 [kworker/u289:1]

0.0 0.0 [kworker/25:2]

0.0 0.0 [kworker/2:0]

0.0 0.0 su - SO-user -- /usr/bin/pcap_agent.tcl -c /etc/nsm/SO-server-eth4/pcap_agent.conf

0.0 0.0 tclsh /usr/bin/pcap_agent.tcl -c /etc/nsm/SO-server-eth4/pcap_agent.conf

0.0 0.0 su - SO-user -- /usr/bin/pcap_agent.tcl -c /etc/nsm/SO-server-eth5/pcap_agent.conf

0.0 0.0 tclsh /usr/bin/pcap_agent.tcl -c /etc/nsm/SO-server-eth5/pcap_agent.conf

0.0 0.0 su - SO-user -- /usr/bin/pcap_agent.tcl -c /etc/nsm/SO-server-eth7/pcap_agent.conf

0.0 0.0 tclsh /usr/bin/pcap_agent.tcl -c /etc/nsm/SO-server-eth7/pcap_agent.conf

0.0 0.0 su - SO-user -- /usr/bin/snort_agent.tcl -c /etc/nsm/SO-server-eth4/snort_agent-1.conf

0.0 0.0 tclsh /usr/bin/snort_agent.tcl -c /etc/nsm/SO-server-eth4/snort_agent-1.conf

0.0 0.0 tail -n 1 -f /nsm/sensor_data/SO-server-eth4/snort-1.stats

0.0 0.0 su - SO-user -- /usr/bin/snort_agent.tcl -c /etc/nsm/SO-server-eth4/snort_agent-2.conf

0.0 0.0 tclsh /usr/bin/snort_agent.tcl -c /etc/nsm/SO-server-eth4/snort_agent-2.conf

0.0 0.0 tail -n 1 -f /nsm/sensor_data/SO-server-eth4/snort-2.stats

0.0 0.0 su - SO-user -- /usr/bin/snort_agent.tcl -c /etc/nsm/SO-server-eth4/snort_agent-3.conf

0.0 0.0 tclsh /usr/bin/snort_agent.tcl -c /etc/nsm/SO-server-eth4/snort_agent-3.conf

0.0 0.0 tail -n 1 -f /nsm/sensor_data/SO-server-eth4/snort-3.stats

0.0 0.0 su - SO-user -- /usr/bin/snort_agent.tcl -c /etc/nsm/SO-server-eth4/snort_agent-4.conf

0.0 0.0 tclsh /usr/bin/snort_agent.tcl -c /etc/nsm/SO-server-eth4/snort_agent-4.conf

0.0 0.0 tail -n 1 -f /nsm/sensor_data/SO-server-eth4/snort-4.stats

0.0 0.0 su - SO-user -- /usr/bin/snort_agent.tcl -c /etc/nsm/SO-server-eth4/snort_agent-5.conf

0.0 0.0 tclsh /usr/bin/snort_agent.tcl -c /etc/nsm/SO-server-eth4/snort_agent-5.conf

0.0 0.0 tail -n 1 -f /nsm/sensor_data/SO-server-eth4/snort-5.stats

0.0 0.0 su - SO-user -- /usr/bin/snort_agent.tcl -c /etc/nsm/SO-server-eth4/snort_agent-6.conf

0.0 0.0 tclsh /usr/bin/snort_agent.tcl -c /etc/nsm/SO-server-eth4/snort_agent-6.conf

0.0 0.0 tail -n 1 -f /nsm/sensor_data/SO-server-eth4/snort-6.stats

0.0 0.0 su - SO-user -- /usr/bin/snort_agent.tcl -c /etc/nsm/SO-server-eth4/snort_agent-7.conf

0.0 0.0 tclsh /usr/bin/snort_agent.tcl -c /etc/nsm/SO-server-eth4/snort_agent-7.conf

0.0 0.0 tail -n 1 -f /nsm/sensor_data/SO-server-eth4/snort-7.stats

0.0 0.0 su - SO-user -- /usr/bin/snort_agent.tcl -c /etc/nsm/SO-server-eth4/snort_agent-8.conf

0.0 0.0 tclsh /usr/bin/snort_agent.tcl -c /etc/nsm/SO-server-eth4/snort_agent-8.conf

0.0 0.0 tail -n 1 -f /nsm/sensor_data/SO-server-eth4/snort-8.stats

0.0 0.0 [kworker/30:2]

0.0 0.0 su - SO-user -- /usr/bin/snort_agent.tcl -c /etc/nsm/SO-server-eth5/snort_agent-1.conf

0.0 0.0 tclsh /usr/bin/snort_agent.tcl -c /etc/nsm/SO-server-eth5/snort_agent-1.conf

0.0 0.0 tail -n 1 -f /nsm/sensor_data/SO-server-eth5/snort-1.stats

0.0 0.0 su - SO-user -- /usr/bin/snort_agent.tcl -c /etc/nsm/SO-server-eth5/snort_agent-2.conf

0.0 0.0 tclsh /usr/bin/snort_agent.tcl -c /etc/nsm/SO-server-eth5/snort_agent-2.conf

0.0 0.0 tail -n 1 -f /nsm/sensor_data/SO-server-eth5/snort-2.stats

0.0 0.0 su - SO-user -- /usr/bin/snort_agent.tcl -c /etc/nsm/SO-server-eth5/snort_agent-3.conf

0.0 0.0 tclsh /usr/bin/snort_agent.tcl -c /etc/nsm/SO-server-eth5/snort_agent-3.conf

0.0 0.0 tail -n 1 -f /nsm/sensor_data/SO-server-eth5/snort-3.stats

0.0 0.0 su - SO-user -- /usr/bin/snort_agent.tcl -c /etc/nsm/SO-server-eth5/snort_agent-4.conf

0.0 0.0 tclsh /usr/bin/snort_agent.tcl -c /etc/nsm/SO-server-eth5/snort_agent-4.conf

0.0 0.0 tail -n 1 -f /nsm/sensor_data/SO-server-eth5/snort-4.stats

0.0 0.0 [kworker/19:0]

0.0 0.0 su - SO-user -- /usr/bin/snort_agent.tcl -c /etc/nsm/SO-server-eth5/snort_agent-5.conf

0.0 0.0 tclsh /usr/bin/snort_agent.tcl -c /etc/nsm/SO-server-eth5/snort_agent-5.conf

0.0 0.0 tail -n 1 -f /nsm/sensor_data/SO-server-eth5/snort-5.stats

0.0 0.0 su - SO-user -- /usr/bin/snort_agent.tcl -c /etc/nsm/SO-server-eth5/snort_agent-6.conf

0.0 0.0 tclsh /usr/bin/snort_agent.tcl -c /etc/nsm/SO-server-eth5/snort_agent-6.conf

0.0 0.0 tail -n 1 -f /nsm/sensor_data/SO-server-eth5/snort-6.stats

0.0 0.0 su - SO-user -- /usr/bin/snort_agent.tcl -c /etc/nsm/SO-server-eth5/snort_agent-7.conf

0.0 0.0 tclsh /usr/bin/snort_agent.tcl -c /etc/nsm/SO-server-eth5/snort_agent-7.conf

0.0 0.0 tail -n 1 -f /nsm/sensor_data/SO-server-eth5/snort-7.stats

0.0 0.0 su - SO-user -- /usr/bin/snort_agent.tcl -c /etc/nsm/SO-server-eth5/snort_agent-8.conf

0.0 0.0 tclsh /usr/bin/snort_agent.tcl -c /etc/nsm/SO-server-eth5/snort_agent-8.conf

0.0 0.0 tail -n 1 -f /nsm/sensor_data/SO-server-eth5/snort-8.stats

0.0 0.0 [kworker/16:2]

0.0 0.0 su - SO-user -- /usr/bin/snort_agent.tcl -c /etc/nsm/SO-server-eth7/snort_agent-1.conf

0.0 0.0 tclsh /usr/bin/snort_agent.tcl -c /etc/nsm/SO-server-eth7/snort_agent-1.conf

0.0 0.0 tail -n 1 -f /nsm/sensor_data/SO-server-eth7/snort-1.stats

0.0 0.0 su - SO-user -- /usr/bin/snort_agent.tcl -c /etc/nsm/SO-server-eth7/snort_agent-2.conf

0.0 0.0 tclsh /usr/bin/snort_agent.tcl -c /etc/nsm/SO-server-eth7/snort_agent-2.conf

0.0 0.0 tail -n 1 -f /nsm/sensor_data/SO-server-eth7/snort-2.stats

0.0 0.0 su - SO-user -- /usr/bin/snort_agent.tcl -c /etc/nsm/SO-server-eth7/snort_agent-3.conf

0.0 0.0 tclsh /usr/bin/snort_agent.tcl -c /etc/nsm/SO-server-eth7/snort_agent-3.conf

0.0 0.0 tail -n 1 -f /nsm/sensor_data/SO-server-eth7/snort-3.stats

0.0 0.0 su - SO-user -- /usr/bin/snort_agent.tcl -c /etc/nsm/SO-server-eth7/snort_agent-4.conf

0.0 0.0 tclsh /usr/bin/snort_agent.tcl -c /etc/nsm/SO-server-eth7/snort_agent-4.conf

0.0 0.0 tail -n 1 -f /nsm/sensor_data/SO-server-eth7/snort-4.stats

0.0 0.0 su - SO-user -- /usr/bin/snort_agent.tcl -c /etc/nsm/SO-server-eth7/snort_agent-5.conf

0.0 0.0 tclsh /usr/bin/snort_agent.tcl -c /etc/nsm/SO-server-eth7/snort_agent-5.conf

0.0 0.0 tail -n 1 -f /nsm/sensor_data/SO-server-eth7/snort-5.stats

0.0 0.0 su - SO-user -- /usr/bin/snort_agent.tcl -c /etc/nsm/SO-server-eth7/snort_agent-6.conf

0.0 0.0 tclsh /usr/bin/snort_agent.tcl -c /etc/nsm/SO-server-eth7/snort_agent-6.conf

0.0 0.0 tail -n 1 -f /nsm/sensor_data/SO-server-eth7/snort-6.stats

0.0 0.0 su - SO-user -- /usr/bin/snort_agent.tcl -c /etc/nsm/SO-server-eth7/snort_agent-7.conf

0.0 0.0 tclsh /usr/bin/snort_agent.tcl -c /etc/nsm/SO-server-eth7/snort_agent-7.conf

0.0 0.0 tail -n 1 -f /nsm/sensor_data/SO-server-eth7/snort-7.stats

0.0 0.0 su - SO-user -- /usr/bin/snort_agent.tcl -c /etc/nsm/SO-server-eth7/snort_agent-8.conf

0.0 0.0 tclsh /usr/bin/snort_agent.tcl -c /etc/nsm/SO-server-eth7/snort_agent-8.conf

0.0 0.0 tail -n 1 -f /nsm/sensor_data/SO-server-eth7/snort-8.stats

0.0 0.0 [kworker/10:0]

0.0 0.0 [kworker/22:2]

0.0 0.0 [kworker/13:1]

0.0 0.0 [kworker/17:1]

0.0 0.0 [kworker/5:2]

0.0 0.0 [kworker/11:0]

0.0 0.0 sshd: SO-user [priv]

0.0 0.0 sshd: SO-user@pts/1

0.0 0.0 -bash

0.0 0.0 [kworker/24:1]

0.0 0.0 [kworker/u289:2]

0.0 0.0 [kworker/1:2]

0.0 0.0 [kworker/26:2]

0.0 0.0 [kworker/9:1]

0.0 0.0 barnyard2 -c /etc/nsm/SO-server-eth7/barnyard2-8.conf -u SO-user -g SO-user -d /nsm/sensor_data/SO-server-eth7/snort-8 -f snort.unified2 -w /etc/nsm/SO-server-eth7/barnyard2.waldo-8 -i 8 -U

0.0 0.0 [kworker/5:1]

0.0 0.0 [kworker/3:1]

0.0 0.0 [kworker/26:0]

0.0 0.0 [kworker/19:1]

0.0 0.0 CRON

0.0 0.0 /bin/sh -c sh /opt/elsa/contrib/securityonion/contrib/securityonion-elsa-cron.sh > /dev/null 2>&1

0.0 0.0 sh /opt/elsa/contrib/securityonion/contrib/securityonion-elsa-cron.sh

0.0 0.0 [kworker/3:2]

0.0 0.0 [kworker/u289:0]

0.0 0.0 sudo sostat-redacted

0.0 0.0 /bin/bash /usr/bin/sostat-redacted

0.0 0.0 /bin/bash /usr/bin/sostat

0.0 0.0 sed -r s/(\b[0-9]{1,3}\.){3}[0-9]{1,3}\b/X.X.X.X/g

0.0 0.0 sed -r s/([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}/MM:MM:MM:MM:MM:MM/g

0.0 0.0 sed -r s/(([0-9a-fA-F]{1,4}:){7,7}[0-9a-fA-F]{1,4}|([0-9a-fA-F]{1,4}:){1,7}:|([0-9a-fA-F]{1,4}:){1,6}:[0-9a-fA-F]{1,4}|([0-9a-fA-F]{1,4}:){1,5}(:[0-9a-fA-F]{1,4}){1,2}|([0-9a-fA-F]{1,4}:){1,4}(:[0-9a-fA-F]{1,4}){1,3}|([0-9a-fA-F]{1,4}:){1,3}(:[0-9a-fA-F]{1,4}){1,4}|([0-9a-fA-F]{1,4}:){1,2}(:[0-9a-fA-F]{1,4}){1,5}|[0-9a-fA-F]{1,4}:((:[0-9a-fA-F]{1,4}){1,6})|:((:[0-9a-fA-F]{1,4}){1,7}|:)|fe80:(:[0-9a-fA-F]{0,4}){0,4}%[0-9a-zA-Z]{1,}|::(ffff(:0{1,4}){0,1}:){0,1}((25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9]).){3,3}(25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])|([0-9a-fA-F]{1,4}:){1,4}:((25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9]).){3,3}(25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9]))\b/X.X.X.X/g

0.0 0.0 sed -r s/X:ssh_port/X:ssh_port/g

0.0 0.0 sed -r s/\*:ssh_port/*:ssh_port/g

0.0 0.0 sed -r s/SO-server/SO-server/g

0.0 0.0 sed -r s/SO-node/SO-node/g

0.0 0.0 sed -r s/SO-user|SO-user|SO-user|SO-user/SO-user/g

0.0 0.0 [kworker/20:1]

0.0 0.0 [kworker/28:1]

0.0 0.0 ps -eo pcpu,pmem,args --sort -pcpu

0.0 0.0 supervising syslog-ng

0.0 0.0 [kworker/23:0]

0.0 0.0 /bin/bash /opt/bro/share/broctl/scripts/run-bro -1 -U .status -p broctl -p broctl-live -p local -p manager local.bro broctl base/frameworks/cluster local-manager.bro broctl/auto

0.0 0.0 /bin/bash /opt/bro/share/broctl/scripts/run-bro -1 -U .status -p broctl -p broctl-live -p local -p proxy local.bro broctl base/frameworks/cluster local-proxy broctl/auto

0.0 0.0 /opt/bro/bin/bro -U .status -p broctl -p broctl-live -p local -p proxy local.bro broctl base/frameworks/cluster local-proxy broctl/auto

0.0 0.0 /bin/bash /opt/bro/share/broctl/scripts/run-bro -1 -i eth4 -U .status -p broctl -p broctl-live -p local -p SO-server-eth4-1 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto

0.0 0.0 /bin/bash /opt/bro/share/broctl/scripts/run-bro -1 -i eth4 -U .status -p broctl -p broctl-live -p local -p SO-server-eth4-2 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto

0.0 0.0 /bin/bash /opt/bro/share/broctl/scripts/run-bro -1 -i eth5 -U .status -p broctl -p broctl-live -p local -p SO-server-eth5-1 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto

0.0 0.0 /bin/bash /opt/bro/share/broctl/scripts/run-bro -1 -i eth5 -U .status -p broctl -p broctl-live -p local -p SO-server-eth5-2 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto

0.0 0.0 /bin/bash /opt/bro/share/broctl/scripts/run-bro -1 -i eth7 -U .status -p broctl -p broctl-live -p local -p SO-server-eth7-1 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto

0.0 0.0 /bin/bash /opt/bro/share/broctl/scripts/run-bro -1 -i eth7 -U .status -p broctl -p broctl-live -p local -p SO-server-eth7-2 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto

0.0 0.8 /opt/bro/bin/bro -i eth5 -U .status -p broctl -p broctl-live -p local -p SO-server-eth5-2 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto

0.0 0.8 /opt/bro/bin/bro -i eth7 -U .status -p broctl -p broctl-live -p local -p SO-server-eth7-2 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto

0.0 0.8 /opt/bro/bin/bro -i eth7 -U .status -p broctl -p broctl-live -p local -p SO-server-eth7-1 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto

0.0 0.8 /opt/bro/bin/bro -i eth4 -U .status -p broctl -p broctl-live -p local -p SO-server-eth4-2 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto

0.0 0.8 /opt/bro/bin/bro -i eth5 -U .status -p broctl -p broctl-live -p local -p SO-server-eth5-1 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto

0.0 0.8 /opt/bro/bin/bro -i eth4 -U .status -p broctl -p broctl-live -p local -p SO-server-eth4-1 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto

0.0 0.0 barnyard2 -c /etc/nsm/SO-server-eth4/barnyard2-1.conf -u SO-user -g SO-user -d /nsm/sensor_data/SO-server-eth4/snort-1 -f snort.unified2 -w /etc/nsm/SO-server-eth4/barnyard2.waldo-1 -i 1 -U

0.0 0.0 barnyard2 -c /etc/nsm/SO-server-eth4/barnyard2-2.conf -u SO-user -g SO-user -d /nsm/sensor_data/SO-server-eth4/snort-2 -f snort.unified2 -w /etc/nsm/SO-server-eth4/barnyard2.waldo-2 -i 2 -U

0.0 0.0 barnyard2 -c /etc/nsm/SO-server-eth4/barnyard2-3.conf -u SO-user -g SO-user -d /nsm/sensor_data/SO-server-eth4/snort-3 -f snort.unified2 -w /etc/nsm/SO-server-eth4/barnyard2.waldo-3 -i 3 -U

0.0 0.0 barnyard2 -c /etc/nsm/SO-server-eth4/barnyard2-4.conf -u SO-user -g SO-user -d /nsm/sensor_data/SO-server-eth4/snort-4 -f snort.unified2 -w /etc/nsm/SO-server-eth4/barnyard2.waldo-4 -i 4 -U

0.0 0.0 barnyard2 -c /etc/nsm/SO-server-eth4/barnyard2-5.conf -u SO-user -g SO-user -d /nsm/sensor_data/SO-server-eth4/snort-5 -f snort.unified2 -w /etc/nsm/SO-server-eth4/barnyard2.waldo-5 -i 5 -U

0.0 0.0 barnyard2 -c /etc/nsm/SO-server-eth4/barnyard2-6.conf -u SO-user -g SO-user -d /nsm/sensor_data/SO-server-eth4/snort-6 -f snort.unified2 -w /etc/nsm/SO-server-eth4/barnyard2.waldo-6 -i 6 -U

0.0 0.0 barnyard2 -c /etc/nsm/SO-server-eth4/barnyard2-7.conf -u SO-user -g SO-user -d /nsm/sensor_data/SO-server-eth4/snort-7 -f snort.unified2 -w /etc/nsm/SO-server-eth4/barnyard2.waldo-7 -i 7 -U

0.0 0.0 barnyard2 -c /etc/nsm/SO-server-eth4/barnyard2-8.conf -u SO-user -g SO-user -d /nsm/sensor_data/SO-server-eth4/snort-8 -f snort.unified2 -w /etc/nsm/SO-server-eth4/barnyard2.waldo-8 -i 8 -U

0.0 0.0 barnyard2 -c /etc/nsm/SO-server-eth5/barnyard2-1.conf -u SO-user -g SO-user -d /nsm/sensor_data/SO-server-eth5/snort-1 -f snort.unified2 -w /etc/nsm/SO-server-eth5/barnyard2.waldo-1 -i 1 -U

0.0 0.0 barnyard2 -c /etc/nsm/SO-server-eth5/barnyard2-6.conf -u SO-user -g SO-user -d /nsm/sensor_data/SO-server-eth5/snort-6 -f snort.unified2 -w /etc/nsm/SO-server-eth5/barnyard2.waldo-6 -i 6 -U

=========================================================================

Packets received during last monitoring interval (600 seconds)

=========================================================================

eth4: 4535684

eth5: 13891473

eth7: 22410993

=========================================================================

Log Archive

=========================================================================

/nsm/sensor_data/SO-server-eth0/dailylogs/ - 0 days

4.0K .

/nsm/sensor_data/SO-server-eth1/dailylogs/ - 0 days

4.0K .

/nsm/sensor_data/SO-server-eth2/dailylogs/ - 0 days

4.0K .

/nsm/sensor_data/SO-server-eth3/dailylogs/ - 0 days

4.0K .

/nsm/sensor_data/SO-server-eth4/dailylogs/ - 2 days

285G .

99G ./2016-04-10

187G ./2016-04-11

/nsm/sensor_data/SO-server-eth5/dailylogs/ - 2 days

856G .

311G ./2016-04-10

546G ./2016-04-11

/nsm/sensor_data/SO-server-eth6/dailylogs/ - 0 days

4.0K .

/nsm/sensor_data/SO-server-eth7/dailylogs/ - 2 days

647G .

647G ./2016-04-10

4.0K ./2016-04-11

/nsm/bro/logs/ - 3 days

4.6G .

1.5G ./2016-04-09

1.4G ./2016-04-10

1.8G ./2016-04-11

102M ./stats

=========================================================================

Bro netstats

=========================================================================

Average packet loss as percent across all Bro workers: 0.000003

SO-server-eth4-1: 1460394360.170027 recvd=2019835728 dropped=0 link=2019835728

SO-server-eth4-2: 1460394360.369981 recvd=2054522688 dropped=1 link=2054522688

SO-server-eth5-1: 1460394360.569926 recvd=209991139 dropped=99 link=209991139

SO-server-eth5-2: 1460394360.769934 recvd=3920419521 dropped=57 link=3920419521

SO-server-eth7-1: 1460394360.974224 recvd=994462716 dropped=132 link=994462716

SO-server-eth7-2: 1460394361.174005 recvd=3621353418 dropped=148 link=3621353418

=========================================================================

IDS Engine (snort) packet drops

=========================================================================

/nsm/sensor_data/SO-server-eth4/snort-1.stats last reported pkt_drop_percent as 0.000

/nsm/sensor_data/SO-server-eth4/snort-2.stats last reported pkt_drop_percent as 0.000

/nsm/sensor_data/SO-server-eth4/snort-3.stats last reported pkt_drop_percent as 0.000

/nsm/sensor_data/SO-server-eth4/snort-4.stats last reported pkt_drop_percent as 0.000

/nsm/sensor_data/SO-server-eth4/snort-5.stats last reported pkt_drop_percent as 0.000

/nsm/sensor_data/SO-server-eth4/snort-6.stats last reported pkt_drop_percent as 0.000

/nsm/sensor_data/SO-server-eth4/snort-7.stats last reported pkt_drop_percent as 0.000

/nsm/sensor_data/SO-server-eth4/snort-8.stats last reported pkt_drop_percent as 0.000

/nsm/sensor_data/SO-server-eth5/snort-1.stats last reported pkt_drop_percent as 0.000

/nsm/sensor_data/SO-server-eth5/snort-2.stats last reported pkt_drop_percent as 0.000

/nsm/sensor_data/SO-server-eth5/snort-3.stats last reported pkt_drop_percent as 0.000

/nsm/sensor_data/SO-server-eth5/snort-4.stats last reported pkt_drop_percent as 0.000

/nsm/sensor_data/SO-server-eth5/snort-5.stats last reported pkt_drop_percent as 0.000

/nsm/sensor_data/SO-server-eth5/snort-6.stats last reported pkt_drop_percent as 0.000

/nsm/sensor_data/SO-server-eth5/snort-7.stats last reported pkt_drop_percent as 0.000

/nsm/sensor_data/SO-server-eth5/snort-8.stats last reported pkt_drop_percent as 0.000

/nsm/sensor_data/SO-server-eth7/snort-1.stats last reported pkt_drop_percent as 0.000

/nsm/sensor_data/SO-server-eth7/snort-2.stats last reported pkt_drop_percent as 0.000

/nsm/sensor_data/SO-server-eth7/snort-3.stats last reported pkt_drop_percent as 0.000

/nsm/sensor_data/SO-server-eth7/snort-4.stats last reported pkt_drop_percent as 0.000

/nsm/sensor_data/SO-server-eth7/snort-5.stats last reported pkt_drop_percent as 0.000

/nsm/sensor_data/SO-server-eth7/snort-6.stats last reported pkt_drop_percent as 0.000

/nsm/sensor_data/SO-server-eth7/snort-7.stats last reported pkt_drop_percent as 0.000

/nsm/sensor_data/SO-server-eth7/snort-8.stats last reported pkt_drop_percent as 0.000

=========================================================================

pf_ring stats

=========================================================================

PF_RING Version : 6.2.0 (unknown)

Total rings : 30

Standard (non DNA/ZC) Options

Ring slots : 393204

Slot version : 16

Capture TX : Yes [RX+TX]

IP Defragment : No

Socket Mode : Standard

Total plugins : 0

Cluster Fragment Queue : 6021

Cluster Fragment Discard : 0

/proc/net/pf_ring/144390-eth4.251

Appl. Name : bro-eth4

Tot Packets : 2019841641

Tot Pkt Lost : 0

TX: Send Errors : 0

Reflect: Fwd Errors: 0

Min Num Slots : 393204

Num Free Slots : 393204

/proc/net/pf_ring/144397-eth4.255

Appl. Name : bro-eth4

Tot Packets : 2054531600

Tot Pkt Lost : 1

TX: Send Errors : 0

Reflect: Fwd Errors: 0

Min Num Slots : 393204

Num Free Slots : 393204

/proc/net/pf_ring/144400-eth5.254

Appl. Name : bro-eth5

Tot Packets : 8799940857

Tot Pkt Lost : 99

TX: Send Errors : 0

Reflect: Fwd Errors: 0

Min Num Slots : 393204

Num Free Slots : 393204

/proc/net/pf_ring/144403-eth5.250

Appl. Name : bro-eth5

Tot Packets : 8215394119

Tot Pkt Lost : 57

TX: Send Errors : 0

Reflect: Fwd Errors: 0

Min Num Slots : 393204

Num Free Slots : 393203

/proc/net/pf_ring/144405-eth7.253

Appl. Name : bro-eth7

Tot Packets : 13879385954

Tot Pkt Lost : 132

TX: Send Errors : 0

Reflect: Fwd Errors: 0

Min Num Slots : 393204

Num Free Slots : 393204

/proc/net/pf_ring/144407-eth7.252

Appl. Name : bro-eth7

Tot Packets : 12211302807

Tot Pkt Lost : 148

TX: Send Errors : 0

Reflect: Fwd Errors: 0

Min Num Slots : 393204

Num Free Slots : 393204

/proc/net/pf_ring/144930-eth4.259

Appl. Name : snort-cluster-55-socket-0

Tot Packets : 500774418

Tot Pkt Lost : 0

Reflect: Fwd Errors: 0

Min Num Slots : 393206

Num Free Slots : 393124

/proc/net/pf_ring/144945-eth4.260

Appl. Name : snort-cluster-55-socket-0

Tot Packets : 490543953

Tot Pkt Lost : 2102017

Reflect: Fwd Errors: 0

Min Num Slots : 393206

Num Free Slots : 393098

/proc/net/pf_ring/144961-eth4.261

Appl. Name : snort-cluster-55-socket-0

Tot Packets : 577054013

Tot Pkt Lost : 74072

Reflect: Fwd Errors: 0

Min Num Slots : 393206

Num Free Slots : 393181

/proc/net/pf_ring/144976-eth4.263

Appl. Name : snort-cluster-55-socket-0

Tot Packets : 497436395

Tot Pkt Lost : 0

Reflect: Fwd Errors: 0

Min Num Slots : 393206

Num Free Slots : 393090

/proc/net/pf_ring/144992-eth4.262

Appl. Name : snort-cluster-55-socket-0

Tot Packets : 528935990

Tot Pkt Lost : 0

Reflect: Fwd Errors: 0

Min Num Slots : 393206

Num Free Slots : 393106

/proc/net/pf_ring/145007-eth4.264

Appl. Name : snort-cluster-55-socket-0

Tot Packets : 492442838

Tot Pkt Lost : 0

Reflect: Fwd Errors: 0

Min Num Slots : 393206

Num Free Slots : 393177

/proc/net/pf_ring/145022-eth4.265

Appl. Name : snort-cluster-55-socket-0

Tot Packets : 479070579

Tot Pkt Lost : 0

Reflect: Fwd Errors: 0

Min Num Slots : 393206

Num Free Slots : 393154

/proc/net/pf_ring/145038-eth4.266

Appl. Name : snort-cluster-55-socket-0

Tot Packets : 507755007

Tot Pkt Lost : 0

Reflect: Fwd Errors: 0

Min Num Slots : 393206

Num Free Slots : 393160

/proc/net/pf_ring/145437-eth5.268

Appl. Name : snort-cluster-56-socket-0

Tot Packets : 1832849461

Tot Pkt Lost : 40389609

Reflect: Fwd Errors: 0

Min Num Slots : 393206

Num Free Slots : 393112

/proc/net/pf_ring/145452-eth5.267

Appl. Name : snort-cluster-56-socket-0

Tot Packets : 2089080631

Tot Pkt Lost : 203301060

Reflect: Fwd Errors: 0

Min Num Slots : 393206

Num Free Slots : 393105

/proc/net/pf_ring/145467-eth5.269

Appl. Name : snort-cluster-56-socket-0

Tot Packets : 2342849081

Tot Pkt Lost : 186855053

Reflect: Fwd Errors: 0

Min Num Slots : 393206

Num Free Slots : 393131

/proc/net/pf_ring/145483-eth5.270

Appl. Name : snort-cluster-56-socket-0

Tot Packets : 2224718419

Tot Pkt Lost : 94667376

Reflect: Fwd Errors: 0

Min Num Slots : 393206

Num Free Slots : 393128

/proc/net/pf_ring/145498-eth5.271

Appl. Name : snort-cluster-56-socket-0

Tot Packets : 1779826057

Tot Pkt Lost : 78313973

Reflect: Fwd Errors: 0

Min Num Slots : 393206

Num Free Slots : 393157

/proc/net/pf_ring/145515-eth5.272

Appl. Name : snort-cluster-56-socket-0

Tot Packets : 1866426629

Tot Pkt Lost : 116231895

Reflect: Fwd Errors: 0

Min Num Slots : 393206

Num Free Slots : 393125

/proc/net/pf_ring/145562-eth5.273

Appl. Name : snort-cluster-56-socket-0

Tot Packets : 2002625504

Tot Pkt Lost : 86381070

Reflect: Fwd Errors: 0

Min Num Slots : 393206

Num Free Slots : 393147

/proc/net/pf_ring/145582-eth5.274

Appl. Name : snort-cluster-56-socket-0

Tot Packets : 2875056627

Tot Pkt Lost : 714545355

Reflect: Fwd Errors: 0

Min Num Slots : 393206

Num Free Slots : 393135

/proc/net/pf_ring/145982-eth7.275

Appl. Name : snort-cluster-58-socket-0

Tot Packets : 2797609108

Tot Pkt Lost : 228536257

Reflect: Fwd Errors: 0

Min Num Slots : 393206

Num Free Slots : 393198

/proc/net/pf_ring/146018-eth7.276

Appl. Name : snort-cluster-58-socket-0

Tot Packets : 2511659066

Tot Pkt Lost : 138895146

Reflect: Fwd Errors: 0

Min Num Slots : 393206

Num Free Slots : 387975

/proc/net/pf_ring/146086-eth7.277

Appl. Name : snort-cluster-58-socket-0

Tot Packets : 2999193520

Tot Pkt Lost : 349070775

Reflect: Fwd Errors: 0

Min Num Slots : 393206

Num Free Slots : 393206

/proc/net/pf_ring/146149-eth7.278

Appl. Name : snort-cluster-58-socket-0

Tot Packets : 3785308513

Tot Pkt Lost : 520963007

Reflect: Fwd Errors: 0

Min Num Slots : 393206

Num Free Slots : 393108

/proc/net/pf_ring/146217-eth7.279

Appl. Name : snort-cluster-58-socket-0

Tot Packets : 3476973052

Tot Pkt Lost : 278398787

Reflect: Fwd Errors: 0

Min Num Slots : 393206

Num Free Slots : 393116

/proc/net/pf_ring/146289-eth7.280

Appl. Name : snort-cluster-58-socket-0

Tot Packets : 2425931248

Tot Pkt Lost : 229776876

Reflect: Fwd Errors: 0

Min Num Slots : 393206

Num Free Slots : 393190

/proc/net/pf_ring/146348-eth7.281

Appl. Name : snort-cluster-58-socket-0

Tot Packets : 2932742762

Tot Pkt Lost : 163760155

Reflect: Fwd Errors: 0

Min Num Slots : 393206

Num Free Slots : 393092

/proc/net/pf_ring/146420-eth7.282

Appl. Name : snort-cluster-58-socket-0

Tot Packets : 5155118732

Tot Pkt Lost : 1524194054

Reflect: Fwd Errors: 0

Min Num Slots : 393206

Num Free Slots : 393189

=========================================================================

Netsniff-NG - Reported Packet Loss (per interval)

=========================================================================

File: /var/log/nsm/SO-server-eth4/netsniff-ng.log Processed: +161004 Lost: -15550

File: /var/log/nsm/SO-server-eth4/netsniff-ng.log.20160408000004 Processed: +152902 Lost: -50136

File: /var/log/nsm/SO-server-eth4/netsniff-ng.log.20160408000401 Processed: Cannot Lost: allocate RX_RING!

File: /var/log/nsm/SO-server-eth4/netsniff-ng.log.20160408000901 Processed: Cannot Lost: allocate RX_RING!

File: /var/log/nsm/SO-server-eth4/netsniff-ng.log.20160409000004 Processed: +172176 Lost: -450297

File: /var/log/nsm/SO-server-eth4/netsniff-ng.log.20160409000401 Processed: Cannot Lost: allocate RX_RING!

File: /var/log/nsm/SO-server-eth4/netsniff-ng.log.20160409000902 Processed: Cannot Lost: allocate RX_RING!

File: /var/log/nsm/SO-server-eth4/netsniff-ng.log.20160410000005 Processed: +230047 Lost: -316856

File: /var/log/nsm/SO-server-eth4/netsniff-ng.log.20160410000402 Processed: Cannot Lost: allocate RX_RING!

File: /var/log/nsm/SO-server-eth4/netsniff-ng.log.20160410000902 Processed: Cannot Lost: allocate RX_RING!

File: /var/log/nsm/SO-server-eth4/netsniff-ng.log.20160411000004 Processed: +176168 Lost: -214904

File: /var/log/nsm/SO-server-eth5/netsniff-ng.log Processed: +213263 Lost: -6239

File: /var/log/nsm/SO-server-eth5/netsniff-ng.log.20160402000005 Processed: +205546 Lost: -14553

File: /var/log/nsm/SO-server-eth5/netsniff-ng.log.20160402000402 Processed: Cannot Lost: allocate RX_RING!

File: /var/log/nsm/SO-server-eth5/netsniff-ng.log.20160404000006 Processed: +280675 Lost: -22059

File: /var/log/nsm/SO-server-eth5/netsniff-ng.log.20160405000006 Processed: +299625 Lost: -33468

File: /var/log/nsm/SO-server-eth5/netsniff-ng.log.20160407000006 Processed: +361835 Lost: -18801

File: /var/log/nsm/SO-server-eth5/netsniff-ng.log.20160407000402 Processed: Cannot Lost: allocate RX_RING!

File: /var/log/nsm/SO-server-eth5/netsniff-ng.log.20160408000005 Processed: +311103 Lost: -16558

File: /var/log/nsm/SO-server-eth5/netsniff-ng.log.20160409000005 Processed: +261335 Lost: -17789

File: /var/log/nsm/SO-server-eth5/netsniff-ng.log.20160410000006 Processed: +329835 Lost: -18764

File: /var/log/nsm/SO-server-eth5/netsniff-ng.log.20160411000005 Processed: +286168 Lost: -1226

File: /var/log/nsm/SO-server-eth7/netsniff-ng.log Processed: Cannot Lost: allocate RX_RING!

File: /var/log/nsm/SO-server-eth7/netsniff-ng.log.20160402000007 Processed: +219020 Lost: -21190

File: /var/log/nsm/SO-server-eth7/netsniff-ng.log.20160403000007 Processed: +171941 Lost: -4948

File: /var/log/nsm/SO-server-eth7/netsniff-ng.log.20160403000403 Processed: Cannot Lost: allocate RX_RING!

File: /var/log/nsm/SO-server-eth7/netsniff-ng.log.20160404000008 Processed: +307454 Lost: -21479

File: /var/log/nsm/SO-server-eth7/netsniff-ng.log.20160405000007 Processed: +201305 Lost: -30492

File: /var/log/nsm/SO-server-eth7/netsniff-ng.log.20160406000007 Processed: +196617 Lost: -27334

File: /var/log/nsm/SO-server-eth7/netsniff-ng.log.20160408000007 Processed: +140958 Lost: -28809

File: /var/log/nsm/SO-server-eth7/netsniff-ng.log.20160409000006 Processed: +210290 Lost: -8189

File: /var/log/nsm/SO-server-eth7/netsniff-ng.log.20160410000007 Processed: +279606 Lost: -12309

File: /var/log/nsm/SO-server-eth7/netsniff-ng.log.20160411000007 Processed: +206792 Lost: -6782

=========================================================================

Last update

=========================================================================

ELSA

=========================================================================

Syslog-ng

Checking for process:

135816 /usr/sbin/syslog-ng -p /var/run/syslog-ng.pid

Checking for connection:

Connection to localhost 514 port [tcp/shell] succeeded!

MySQL

Checking for process:

2130 /usr/sbin/mysqld

Checking for connection:

Connection to localhost 50000 port [tcp/*] succeeded!

Sphinx

Checking for process:

2015 su -s /bin/sh -c exec "$0" "$@" sphinxsearch -- /usr/bin/searchd --nodetach

2100 /usr/bin/searchd --nodetach

Checking for connection:

Connection to localhost 9306 port [tcp/*] succeeded!

ELSA Buffers in Queue:

3

If this number is consistently higher than 20, please see:

https://github.com/Security-Onion-Solutions/security-onion/wiki/FAQ#why-does-sostat-show-a-high-number-of-elsa-buffers-in-queue

ELSA Directory Sizes:

3.8T /nsm/elsa/data

98M /var/lib/mysql/syslog

8.4M /var/lib/mysql/syslog_data

ELSA Index Date Range

If you don't have at least 2 full days of logs in the Index Date Range,

then you'll need to increase log_size_limit in /etc/elsa_node.conf.

MIN(start) MAX(end)

2016-03-14 22:26:05 2016-04-11 17:05:47

autossh

Checking for process:

6461 /usr/lib/autossh/autossh -M 0 -q -N -o ServerAliveInterval 60 -o ServerAliveCountMax 3 -i /root/.ssh/securityonion -L 3306:X.X.X.X:3306 -R 50001:localhost:3154 SO-...@X.X.X.X

Checking APIKEY:

APIKEY matches server.

starman

Checking for processes:

2391 starman master -I/opt/elsa/web/lib --user=www-data --listen :3154 --daemonize --pid /var/run/starman.pid --error-log /var/log/starman.log /opt/elsa/web/lib/Web.psgi

2403 starman worker -I/opt/elsa/web/lib --user=www-data --listen :3154 --daemonize --pid /var/run/starman.pid --error-log /var/log/starman.log /opt/elsa/web/lib/Web.psgi

2404 starman worker -I/opt/elsa/web/lib --user=www-data --listen :3154 --daemonize --pid /var/run/starman.pid --error-log /var/log/starman.log /opt/elsa/web/lib/Web.psgi

2405 starman worker -I/opt/elsa/web/lib --user=www-data --listen :3154 --daemonize --pid /var/run/starman.pid --error-log /var/log/starman.log /opt/elsa/web/lib/Web.psgi

2406 starman worker -I/opt/elsa/web/lib --user=www-data --listen :3154 --daemonize --pid /var/run/starman.pid --error-log /var/log/starman.log /opt/elsa/web/lib/Web.psgi

2407 starman worker -I/opt/elsa/web/lib --user=www-data --listen :3154 --daemonize --pid /var/run/starman.pid --error-log /var/log/starman.log /opt/elsa/web/lib/Web.psgi

MINION LOG MESSAGES:

2016-04-11 16:48:02,051 [salt.log.setup ][ERROR ][60881] An un-handled exception was caught by salt's global exception handler:

SaltReqTimeoutError: SaltReqTimeoutError: after 180 seconds, ran 3 tries

Traceback (most recent call last):

File "/usr/bin/salt-call", line 11, in <module>

salt_call()

File "/usr/lib/python2.7/dist-packages/salt/scripts.py", line 227, in salt_call

client.run()

File "/usr/lib/python2.7/dist-packages/salt/cli/call.py", line 69, in run

caller.run()

File "/usr/lib/python2.7/dist-packages/salt/cli/caller.py", line 236, in run

ret = self.call()

File "/usr/lib/python2.7/dist-packages/salt/cli/caller.py", line 138, in call

ret['return'] = func(*args, **kwargs)

File "/usr/lib/python2.7/dist-packages/salt/modules/state.py", line 515, in highstate

st_ = salt.state.HighState(opts, pillar, kwargs.get('__pub_jid'))

File "/usr/lib/python2.7/dist-packages/salt/state.py", line 3026, in __init__

BaseHighState.__init__(self, opts)

File "/usr/lib/python2.7/dist-packages/salt/state.py", line 2205, in __init__

self.avail = self.__gather_avail()

File "/usr/lib/python2.7/dist-packages/salt/state.py", line 2215, in __gather_avail

avail[saltenv] = self.client.list_states(saltenv)

File "/usr/lib/python2.7/dist-packages/salt/fileclient.py", line 411, in list_states

for path in self.file_list(saltenv):

File "/usr/lib/python2.7/dist-packages/salt/fileclient.py", line 1096, in file_list

return self.channel.send(load)

File "/usr/lib/python2.7/dist-packages/salt/transport/__init__.py", line 314, in send

return self._crypted_transfer(load, tries, timeout)

File "/usr/lib/python2.7/dist-packages/salt/transport/__init__.py", line 302, in _crypted_transfer

return _do_transfer()

File "/usr/lib/python2.7/dist-packages/salt/transport/__init__.py", line 293, in _do_transfer

timeout)

File "/usr/lib/python2.7/dist-packages/salt/payload.py", line 273, in send

'SaltReqTimeoutError: after {0} seconds, ran {1} tries'.format(timeout * tried, tried)

SaltReqTimeoutError: SaltReqTimeoutError: after 180 seconds, ran 3 tries

2016-04-11 17:01:01,974 [salt.payload ][INFO ][66500] SaltReqTimeoutError: after 60 seconds. (Try 1 of 3)

Wes

unread,

Apr 11, 2016, 8:40:12 PM4/11/16

to security-onion

> syslo..

A short term fix could be to try setting the number of worker threads in the salt-master config (/etc/salt/master) to "1":

See: https://github.com/saltstack/salt/issues/5376

Thanks,
Wes

s adz

unread,

Apr 11, 2016, 11:16:35 PM4/11/16

to securit...@googlegroups.com

I find today someone setup hosts file on all servers. Now when ssh to any server as root, it asks me for password. I try ping and server said

"No minions matched the target. No command was sent, no jid was assigned." Salt use ssh, yes? hosts file possible or not? Maybe this new thread, but salt still broke, perhaps broke even more after this.

Wes

unread,

Apr 12, 2016, 2:56:37 PM4/12/16

to security-onion

I'm not sure the host file would affect this, because I think salt stores its own association for the host, but possibly.

https://docs.saltstack.com/en/latest/ref/states/all/salt.states.host.html

What was the exact syntax of the command you issued for the test.ping? you should at least be able to ping the minion on the master.

Ex. sudo salt "*" test.ping

Thanks,
Wes

s adz

unread,

Apr 13, 2016, 11:40:00 AM4/13/16

to securit...@googlegroups.com

sudo salt '*' test.ping

salt-master CPU at 100% constant. Ping work until process hit 100%.

original problem persists:

>sudo salt '*' test.ping

sensorA:
True
sensorB:
Minion did not return. [No response]
sensorC:
Minion did not return. [Not connected]
sensorD:
Minion did not return. [Not connected]

When restarted, ping good for a few minutes, then master CPU reach 100% and we have timeouts.

Wes

unread,

Apr 13, 2016, 6:10:41 PM4/13/16

to security-onion

> > > tclsh 117761 SO-user 4u IPv4 50000284 0t0 TC...

Again, did you try modifying the number of worker threads in /etc/salt/master?

Thanks,
Wes

s adz

unread,

Apr 14, 2016, 11:10:22 AM4/14/16

to securit...@googlegroups.com

Yes, I set /etc/salt/master to "worker_threads: 1"

top 3:

PID User PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

25350 sguil 20 0 1099352 996756 222272 R 100.1 0.3 433:48.74 snort

101866 root 20 0 7657352 6.339g 7520 R 100.1 1.7 16:19.61 salt-master

119920 root 20 0 533740 309428 7080 R 100.0 0.1 0:17.18 indexer

When I ping:

"Salt request timed out. The master is not responding. If this error persists after verifying the master is up, worker_threads may need to be increased."

Please help me, how do I see how master stuck? Is master stuck somewhere?

Doug Burks

unread,

Apr 16, 2016, 2:41:37 PM4/16/16

to securit...@googlegroups.com

You could try running the following command on the master server and
your sensors to see if it shows any errors or unexpected delays:
sudo salt-call state.highstate

> You received this message because you are subscribed to the Google Groups
> "security-onion" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to security-onio...@googlegroups.com.
> To post to this group, send email to securit...@googlegroups.com.
> Visit this group at https://groups.google.com/group/security-onion.
> For more options, visit https://groups.google.com/d/optout.

--
Doug Burks

s adz

unread,

Apr 18, 2016, 5:10:24 PM4/18/16

to securit...@googlegroups.com

I would like to remove salt completely and reinstall? Nothing seems to helping. highstate command output:

[INFO ] SaltReqTimeoutError: after 60 seconds. (Try 1 of 7)

Wes

unread,

Apr 18, 2016, 5:25:10 PM4/18/16

to security-onion

> ...

You could try running apt-get remove for the securityonion-onionsalt package (and reinstalling), but your installation may still retain configuration files.

I would have a look here to get a better idea of the configuration involved:

https://github.com/Security-Onion-Solutions/security-onion/wiki/Salt

Thanks,
Wes

s adz

unread,

Apr 19, 2016, 4:16:18 PM4/19/16

to securit...@googlegroups.com

I run htop:

101857 root 20 0 149M 1080 0 S 0.0 0.0 0:00.00 ├─ /usr/bin/python /usr/bin/salt-master -d

101863 root 20 0 149M 5560 2932 S 0.0 0.0 0:00.04 │ ├─ /usr/bin/python /usr/bin/salt-master -d

110317 root 20 0 881M 30432 6376 S 0.0 0.0 0:00.84 │ │ ├─ /usr/bin/python /usr/bin/salt-master -d

110411 root 20 0 881M 30432 6376 S 0.0 0.0 0:00.00 │ │ │ ├─ /usr/bin/python /usr/bin/salt-master -d

110410 root 20 0 881M 30432 6376 S 0.0 0.0 0:00.00 │ │ │ ├─ /usr/bin/python /usr/bin/salt-master -d

110391 root 20 0 881M 30432 6376 S 0.0 0.0 0:00.00 │ │ │ ├─ /usr/bin/python /usr/bin/salt-master -d

110390 root 20 0 881M 30432 6376 S 0.0 0.0 0:00.00 │ │ │ ├─ /usr/bin/python /usr/bin/salt-master -d

110389 root 20 0 881M 30432 6376 S 0.0 0.0 0:00.00 │ │ │ ├─ /usr/bin/python /usr/bin/salt-master -d

110388 root 20 0 881M 30432 6376 S 0.0 0.0 0:00.00 │ │ │ ├─ /usr/bin/python /usr/bin/salt-master -d

110369 root 20 0 881M 30432 6376 S 0.0 0.0 0:00.00 │ │ │ ├─ /usr/bin/python /usr/bin/salt-master -d

110368 root 20 0 881M 30432 6376 S 0.0 0.0 0:00.00 │ │ │ ├─ /usr/bin/python /usr/bin/salt-master -d

110367 root 20 0 881M 30432 6376 S 0.0 0.0 0:00.00 │ │ │ ├─ /usr/bin/python /usr/bin/salt-master -d

110366 root 20 0 881M 30432 6376 S 0.0 0.0 0:00.00 │ │ │ ├─ /usr/bin/python /usr/bin/salt-master -d

110347 root 20 0 881M 30432 6376 S 0.0 0.0 0:00.00 │ │ │ ├─ /usr/bin/python /usr/bin/salt-master -d

110346 root 20 0 881M 30432 6376 S 0.0 0.0 0:00.00 │ │ │ ├─ /usr/bin/python /usr/bin/salt-master -d

110345 root 20 0 881M 30432 6376 S 0.0 0.0 0:00.00 │ │ │ ├─ /usr/bin/python /usr/bin/salt-master -d

110344 root 20 0 881M 30432 6376 S 0.0 0.0 0:00.00 │ │ │ ├─ /usr/bin/python /usr/bin/salt-master -d

110321 root 20 0 881M 30432 6376 S 0.0 0.0 0:00.00 │ │ │ ├─ /usr/bin/python /usr/bin/salt-master -d

110320 root 20 0 881M 30432 6376 S 0.0 0.0 0:00.00 │ │ │ ├─ /usr/bin/python /usr/bin/salt-master -d

110319 root 20 0 881M 30432 6376 S 0.0 0.0 0:00.00 │ │ │ ├─ /usr/bin/python /usr/bin/salt-master -d

110318 root 20 0 881M 30432 6376 S 0.0 0.0 0:00.00 │ │ │ └─ /usr/bin/python /usr/bin/salt-master -d

101867 root 20 0 229M 1896 484 S 0.0 0.0 0:01.48 │ │ └─ /usr/bin/python /usr/bin/salt-master -d

101871 root 20 0 229M 1896 484 S 0.0 0.0 0:01.06 │ │ ├─ /usr/bin/python /usr/bin/salt-master -d

101870 root 20 0 229M 1896 484 S 0.0 0.0 0:00.00 │ │ └─ /usr/bin/python /usr/bin/salt-master -d

101860 root 20 0 229M 3560 1748 S 0.0 0.0 0:00.04 │ ├─ /usr/bin/python /usr/bin/salt-master -d

101865 root 20 0 229M 3560 1748 S 0.0 0.0 0:00.03 │ │ ├─ /usr/bin/python /usr/bin/salt-master -d

101864 root 20 0 229M 3560 1748 S 0.0 0.0 0:00.00 │ │ └─ /usr/bin/python /usr/bin/salt-master -d

101859 root 20 0 229M 964 0 S 0.0 0.0 0:00.00 │ ├─ /usr/bin/python /usr/bin/salt-master -d

101862 root 20 0 229M 964 0 S 0.0 0.0 0:00.00 │ │ ├─ /usr/bin/python /usr/bin/salt-master -d

101861 root 20 0 229M 964 0 S 0.0 0.0 0:00.00 │ │ └─ /usr/bin/python /usr/bin/salt-master -d

101858 root 20 0 267M 25820 3764 S 0.0 0.0 47:34.25 │ └─ /usr/bin/python /usr/bin/salt-master -d

101895 root 20 0 267M 25820 3764 S 0.0 0.0 0:00.00 │ ├─ /usr/bin/python /usr/bin/salt-master -d

101894 root 20 0 267M 25820 3764 S 0.0 0.0 0:00.00 │ └─ /usr/bin/python /usr/bin/salt-master -d

so then I strace:

sudo strace -f -p 35916 -o salt.strace.txt

here is sample (many more in large output like this, just 5 second run):

35939 epoll_wait(81, <unfinished ...>

35938 epoll_wait(79, <unfinished ...>

35919 epoll_wait(72, <unfinished ...>

35918 epoll_wait(70, <unfinished ...>

35917 epoll_wait(63, <unfinished ...>

35897 epoll_wait(54, <unfinished ...>

35916 epoll_wait(61, <unfinished ...>

35896 epoll_wait(52, <unfinished ...>

35894 epoll_wait(43, <unfinished ...>

35895 epoll_wait(45, <unfinished ...>

35875 epoll_wait(36, <unfinished ...>

35874 epoll_wait(34, <unfinished ...>

35873 epoll_wait(27, <unfinished ...>

35872 epoll_wait(25, <unfinished ...>

35851 epoll_wait(18, <unfinished ...>

35850 epoll_wait(16, <unfinished ...>

35849 epoll_wait(9, <unfinished ...>

35848 epoll_wait(7, <unfinished ...>

35847 lstat("/opt/onionsalt/salt/sensor/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/rules/rules/rules/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/Kijiueth4-1/rules/rules/Kijiueth4-1/rules/rules/Kijiueth4-1/rules/rules/bro/protocols/mysql/software.bro", {st_mode=S_IFREG|0644, st_size=410, ...}) = 0

35847 openat(AT_FDCWD, "/opt/onionsalt/salt/sensor/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/rules/rules/rules/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/Kijiueth4-1/rules/rules/Kijiueth4-1/rules/rules/Kijiueth4-1/rules/rules/bro/protocols/http", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 84

35847 getdents(84, /* 10 entries */, 32768) = 384

35847 getdents(84, /* 0 entries */, 32768) = 0

35847 close(84) = 0

35847 stat("/opt/onionsalt/salt/sensor/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/rules/rules/rules/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/Kijiueth4-1/rules/rules/Kijiueth4-1/rules/rules/Kijiueth4-1/rules/rules/bro/protocols/http/var-extraction-uri.bro", {st_mode=S_IFREG|0644, st_size=446, ...}) = 0

35847 stat("/opt/onionsalt/salt/sensor/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/rules/rules/rules/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/Kijiueth4-1/rules/rules/Kijiueth4-1/rules/rules/Kijiueth4-1/rules/rules/bro/protocols/http/detect-webapps.bro", {st_mode=S_IFREG|0644, st_size=1678, ...}) = 0

35847 stat("/opt/onionsalt/salt/sensor/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/rules/rules/rules/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/Kijiueth4-1/rules/rules/Kijiueth4-1/rules/rules/Kijiueth4-1/rules/rules/bro/protocols/http/software.bro", {st_mode=S_IFREG|0644, st_size=1300, ...}) = 0

35847 stat("/opt/onionsalt/salt/sensor/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/rules/rules/rules/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/Kijiueth4-1/rules/rules/Kijiueth4-1/rules/rules/Kijiueth4-1/rules/rules/bro/protocols/http/var-extraction-cookies.bro", {st_mode=S_IFREG|0644, st_size=459, ...}) = 0

35847 stat("/opt/onionsalt/salt/sensor/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/rules/rules/rules/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/Kijiueth4-1/rules/rules/Kijiueth4-1/rules/rules/Kijiueth4-1/rules/rules/bro/protocols/http/software-browser-plugins.bro", {st_mode=S_IFREG|0644, st_size=1627, ...}) = 0

35847 stat("/opt/onionsalt/salt/sensor/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/rules/rules/rules/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/Kijiueth4-1/rules/rules/Kijiueth4-1/rules/rules/Kijiueth4-1/rules/rules/bro/protocols/http/detect-webapps.sig", {st_mode=S_IFREG|0644, st_size=2443, ...}) = 0

35847 stat("/opt/onionsalt/salt/sensor/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/rules/rules/rules/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/Kijiueth4-1/rules/rules/Kijiueth4-1/rules/rules/Kijiueth4-1/rules/rules/bro/protocols/http/header-names.bro", {st_mode=S_IFREG|0644, st_size=1437, ...}) = 0

35847 stat("/opt/onionsalt/salt/sensor/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/rules/rules/rules/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/Kijiueth4-1/rules/rules/Kijiueth4-1/rules/rules/Kijiueth4-1/rules/rules/bro/protocols/http/detect-sqli.bro", {st_mode=S_IFREG|0644, st_size=5978, ...}) = 0

35847 lstat("/opt/onionsalt/salt/sensor/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/rules/rules/rules/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/Kijiueth4-1/rules/rules/Kijiueth4-1/rules/rules/Kijiueth4-1/rules/rules/bro/protocols/http/var-extraction-uri.bro", {st_mode=S_IFREG|0644, st_size=446, ...}) = 0

35847 lstat("/opt/onionsalt/salt/sensor/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/rules/rules/rules/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/Kijiueth4-1/rules/rules/Kijiueth4-1/rules/rules/Kijiueth4-1/rules/rules/bro/protocols/http/detect-webapps.bro", {st_mode=S_IFREG|0644, st_size=1678, ...}) = 0

35847 lstat("/opt/onionsalt/salt/sensor/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/rules/rules/rules/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/Kijiueth4-1/rules/rules/Kijiueth4-1/rules/rules/Kijiueth4-1/rules/rules/bro/protocols/http/software.bro", {st_mode=S_IFREG|0644, st_size=1300, ...}) = 0

35847 lstat("/opt/onionsalt/salt/sensor/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/rules/rules/rules/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/Kijiueth4-1/rules/rules/Kijiueth4-1/rules/rules/Kijiueth4-1/rules/rules/bro/protocols/http/var-extraction-cookies.bro", {st_mode=S_IFREG|0644, st_size=459, ...}) = 0

35847 lstat("/opt/onionsalt/salt/sensor/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/rules/rules/rules/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/Kijiueth4-1/rules/rules/Kijiueth4-1/rules/rules/Kijiueth4-1/rules/rules/bro/protocols/http/software-browser-plugins.bro", {st_mode=S_IFREG|0644, st_size=1627, ...}) = 0

35847 lstat("/opt/onionsalt/salt/sensor/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/rules/rules/rules/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/Kijiueth4-1/rules/rules/Kijiueth4-1/rules/rules/Kijiueth4-1/rules/rules/bro/protocols/http/detect-webapps.sig", {st_mode=S_IFREG|0644, st_size=2443, ...}) = 0

35847 lstat("/opt/onionsalt/salt/sensor/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/rules/rules/rules/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/Kijiueth4-1/rules/rules/Kijiueth4-1/rules/rules/Kijiueth4-1/rules/rules/bro/protocols/http/header-names.bro", {st_mode=S_IFREG|0644, st_size=1437, ...}) = 0

35847 lstat("/opt/onionsalt/salt/sensor/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/rules/rules/rules/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/Kijiueth4-1/rules/rules/Kijiueth4-1/rules/rules/Kijiueth4-1/rules/rules/bro/protocols/http/detect-sqli.bro", {st_mode=S_IFREG|0644, st_size=5978, ...}) = 0

35847 openat(AT_FDCWD, "/opt/onionsalt/salt/sensor/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/rules/rules/rules/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/Kijiueth4-1/rules/rules/Kijiueth4-1/rules/rules/Kijiueth4-1/rules/rules/bro/protocols/rdp", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 84

35847 getdents(84, /* 3 entries */, 32768) = 88

35847 getdents(84, /* 0 entries */, 32768) = 0

35847 close(84) = 0

35847 stat("/opt/onionsalt/salt/sensor/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/rules/rules/rules/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/Kijiueth4-1/rules/rules/Kijiueth4-1/rules/rules/Kijiueth4-1/rules/rules/bro/protocols/rdp/indicate_ssl.bro", {st_mode=S_IFREG|0644, st_size=396, ...}) = 0

35847 lstat("/opt/onionsalt/salt/sensor/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/rules/rules/rules/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/Kijiueth4-1/rules/rules/Kijiueth4-1/rules/rules/Kijiueth4-1/rules/rules/bro/protocols/rdp/indicate_ssl.bro", {st_mode=S_IFREG|0644, st_size=396, ...}) = 0

35847 openat(AT_FDCWD, "/opt/onionsalt/salt/sensor/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/rules/rules/rules/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/Kijiueth4-1/rules/rules/Kijiueth4-1/rules/rules/Kijiueth4-1/rules/rules/bro/protocols/smtp", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 84

35847 getdents(84, /* 6 entries */, 32768) = 208

35847 getdents(84, /* 0 entries */, 32768) = 0

35847 close(84) = 0

35847 stat("/opt/onionsalt/salt/sensor/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/rules/rules/rules/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/Kijiueth4-1/rules/rules/Kijiueth4-1/rules/rules/Kijiueth4-1/rules/rules/bro/protocols/smtp/blocklists.bro", {st_mode=S_IFREG|0644, st_size=1849, ...}) = 0

35847 stat("/opt/onionsalt/salt/sensor/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/rules/rules/rules/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/Kijiueth4-1/rules/rules/Kijiueth4-1/rules/rules/Kijiueth4-1/rules/rules/bro/protocols/smtp/software.bro", {st_mode=S_IFREG|0644, st_size=2837, ...}) = 0

35847 stat("/opt/onionsalt/salt/sensor/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/rules/rules/rules/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/Kijiueth4-1/rules/rules/Kijiueth4-1/rules/rules/Kijiueth4-1/rules/rules/bro/protocols/smtp/entities-excerpt.bro", {st_mode=S_IFREG|0644, st_size=872, ...}) = 0

35847 stat("/opt/onionsalt/salt/sensor/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/rules/rules/rules/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/Kijiueth4-1/rules/rules/Kijiueth4-1/rules/rules/Kijiueth4-1/rules/rules/bro/protocols/smtp/detect-suspicious-orig.bro", {st_mode=S_IFREG|0644, st_size=1396, ...}) = 0

35847 lstat("/opt/onionsalt/salt/sensor/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/rules/rules/rules/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/Kijiueth4-1/rules/rules/Kijiueth4-1/rules/rules/Kijiueth4-1/rules/rules/bro/protocols/smtp/blocklists.bro", {st_mode=S_IFREG|0644, st_size=1849, ...}) = 0

35847 lstat("/opt/onionsalt/salt/sensor/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/rules/rules/rules/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/Kijiueth4-1/rules/rules/Kijiueth4-1/rules/rules/Kijiueth4-1/rules/rules/bro/protocols/smtp/software.bro", {st_mode=S_IFREG|0644, st_size=2837, ...}) = 0

35847 lstat("/opt/onionsalt/salt/sensor/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/rules/rules/rules/rules/Kijiueth4-1/Kijiueth4-1/Kijiueth4-1/rules/Kijiueth4-1/rules/rules/Kijiueth4-1/rules/rules/Kijiueth4-1/rules/rules/bro/protocols/smtp/entities-excerpt.bro", {st_mode=S_IFREG|0644, st_size=872, ...}) = 0

Any more understanding helps. Thank you.

Wes

unread,

Apr 19, 2016, 4:33:41 PM4/19/16

to security-onion

> ...

Do you see a large number of salt-minion processes on the affected sensor(s)?

Also for the master, does "sudo salt-key -L" product any accepted keys?

Thanks,
Wes

s adz

unread,

Apr 19, 2016, 4:44:17 PM4/19/16

to securit...@googlegroups.com

Accepted Keys:

ServerA

ServerB

ServerC (Kijiu)

ServerD (master and minion)

Denied Keys:

Unaccepted Keys:

Rejected Keys:

htop result on Kijiu:

1930 root 20 0 73724 11820 7052 S 0.0 0.0 0:00.07 ├─ /usr/bin/python /usr/bin/salt-minion

2183 root 20 0 578M 18756 8572 S 0.0 0.0 3:57.02 │ └─ /usr/bin/python /usr/bin/salt-minion

11193 root 20 0 578M 18756 8572 S 0.0 0.0 0:00.13 │ ├─ /usr/bin/python /usr/bin/salt-minion

11192 root 20 0 578M 18756 8572 S 0.0 0.0 0:00.00 │ ├─ /usr/bin/python /usr/bin/salt-minion

11171 root 20 0 578M 18756 8572 S 0.0 0.0 0:00.11 │ ├─ /usr/bin/python /usr/bin/salt-minion

11170 root 20 0 578M 18756 8572 S 0.0 0.0 0:00.00 │ ├─ /usr/bin/python /usr/bin/salt-minion

2184 root 20 0 578M 18756 8572 S 0.0 0.0 0:05.31 │ └─ /usr/bin/python /usr/bin/salt-minion

Wes

unread,

Apr 19, 2016, 5:56:01 PM4/19/16

to security-onion

> > >...

What happens when you stop the salt-minion on the sensors? Do the number of salt-master instances (on the master) decrease?

Thanks,
Wes

s adz

unread,

Apr 20, 2016, 12:28:55 PM4/20/16

to securit...@googlegroups.com

This odd thing happen when I try stop the salt-minion on kijiu:

>sudo salt-minion stop

[ERROR ] Attempt to authenticate with the salt master failed

[WARNING ] ** Restarting minion **

[ERROR ] Attempt to authenticate with the salt master failed

[WARNING ] ** Restarting minion **

[ERROR ] Attempt to authenticate with the salt master failed

[WARNING ] ** Restarting minion **

[ERROR ] Attempt to authenticate with the salt master failed

[WARNING ] ** Restarting minion **

Wes

unread,

Apr 20, 2016, 12:37:22 PM4/20/16

to security-onion

> ...

Yes, but did the number of salt-master instances on the master decrease?

Thanks,
Wes

s adz

unread,

Apr 20, 2016, 12:54:46 PM4/20/16

to securit...@googlegroups.com

No. Salt-minion never stop. It keeps to try to talk to master. maybe run kill spid?

Wes

unread,

Apr 20, 2016, 2:17:34 PM4/20/16

to security-onion

It's worth a shot :)

Thanks,
Wes

s adz

unread,

Apr 20, 2016, 5:26:20 PM4/20/16

to securit...@googlegroups.com

I kill two processes:

root 131001 130992 0 21:00 ? 00:00:00 /bin/sh -c /usr/bin/salt-call state.highstate >/dev/null 2>&1

root 131010 131001 0 21:00 ? 00:00:00 /usr/bin/python /usr/bin/salt-call state.highstate

No change on master.

now, we find salt-master eat 512 GB RAM on server, so I take it to trash too:

kill -9 multipled pids on master, finally, salt all dead, on all servers.

Question: how to start over? salt put jobs in queue on master? on minions? Where is queue? I need to kill queue then try restart salt?

thanks!

Wes

unread,

Apr 20, 2016, 8:10:33 PM4/20/16

to security-onion

I would try performing the reverse of the following steps to remove the pre-existing salt configuration, and then performing the steps again (in the correct order) to re-configure salt-master and the minions:

https://github.com/Security-Onion-Solutions/security-onion/wiki/Salt#salting-an-existing-deployment

Thanks,
Wes

s adz

unread,

Apr 21, 2016, 4:41:09 PM4/21/16

to securit...@googlegroups.com

Ok, I remove. Now, I like to install. Can one server be both master and minion?

Wes

unread,

Apr 21, 2016, 4:53:26 PM4/21/16

to security-onion

The master server will have both a salt-master and salt-minion.

Thanks,
Wes

Reply all

Reply to author

Forward