OSSEC dedicated server and ELSA DB size

[Brian Kellogg]

I have a test OSSEC server with 5TB of disk dedicated to ELSA. That disk slice never uses more than 52% of space. I'm not sure why. I have the below set in elsa_node.conf:

"log_size_limit" : 4556000000000 #4.5TB

I do have archive log percentage set to 0.

We will be buying a server with over 100TB of space dedicated to ELSA and just want to ensure I understand what I'm doing wrong. Is there another process cleaning up space even though it is only 52% used? Thanks

[Doug Burks]

Hi Brian,

Just to confirm, is the disk space totally dedicated to ELSA? Nothing
else is using it, like full packet capture?

Please go ahead and send sostat-redacted output.

> --
> You received this message because you are subscribed to the Google Groups "security-onion" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
> To post to this group, send email to securit...@googlegroups.com.
> Visit this group at http://groups.google.com/group/security-onion.
> For more options, visit https://groups.google.com/d/optout.



--
Doug Burks
Need Security Onion Training or Commercial Support?
http://securityonionsolutions.com

[Brian Kellogg]

Yes, completely dedicated to ELSA.

I played with the num_indexes this morning thinking I may not be getting the minimum number of logs/second. I'll send the redacted sostat shortly.

[Brian Kellogg]

I have three very busy firewalls logging to this OSSEC only sensor along with some servers, so I wouldn't think the minimum logs/second would be an issue.


warning: new bro version detected (run the broctl "restart --clean" or "install" command)
error: cannot acquire lock: [Errno 13] Permission denied: '/nsm/bro/spool/lock.27257'
grep: /nsm/sensor_data/*/snort-*.stats: No such file or directory
=========================================================================
Service Status
=========================================================================
Status: HIDS
* ossec_agent (SO-user)[ OK ]
Status: SO-server-eth0

=========================================================================
Interface Status
=========================================================================
eth0 Link encap:Ethernet HWaddr MM:MM:MM:MM:MM:MM
inet addr:X.X.X.X Bcast:X.X.X.X Mask:X.X.X.X
inet6 addr: X.X.X.X/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:11502630 errors:0 dropped:0 overruns:0 frame:0
TX packets:31803 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:3128398499 (3.1 GB) TX bytes:100231638 (100.2 MB)

lo Link encap:Local Loopback
inet addr:X.X.X.X Mask:X.X.X.X
inet6 addr: X.X.X.X/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:117058 errors:0 dropped:0 overruns:0 frame:0
TX packets:117058 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:393771379 (393.7 MB) TX bytes:393771379 (393.7 MB)


=========================================================================
Link Statistics
=========================================================================
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
link/loopback MM:MM:MM:MM:MM:MM brd MM:MM:MM:MM:MM:MM
RX: bytes packets errors dropped overrun mcast
393771379 117058 0 0 0 0
RX errors: length crc frame fifo missed
0 0 0 0 0
TX: bytes packets errors dropped carrier collsns
393771379 117058 0 0 0 0
TX errors: aborted fifo window heartbeat
0 0 0 0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UNKNOWN qlen 1000
link/ether MM:MM:MM:MM:MM:MM brd MM:MM:MM:MM:MM:MM
RX: bytes packets errors dropped overrun mcast
3128398785 11502631 0 0 0 22
RX errors: length crc frame fifo missed
0 0 0 0 0
TX: bytes packets errors dropped carrier collsns
100231638 31803 0 0 0 0
TX errors: aborted fifo window heartbeat
0 0 0 0

=========================================================================
Disk Usage
=========================================================================
Filesystem Size Used Avail Use% Mounted on
/dev/sda2 84G 6.8G 73G 9% /
udev 16G 4.0K 16G 1% /dev
tmpfs 3.2G 740K 3.2G 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 16G 0 16G 0% /run/shm
/dev/sdc1 1008G 118G 840G 13% /var/ossec
/dev/sdb1 5.0T 2.4T 2.4T 51% /nsm

=========================================================================
Network Sockets
=========================================================================
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
sshd 1519 root 3r IPv4 9485 0t0 TCP *:ssh_port (LISTEN)
sshd 1519 root 4u IPv6 9487 0t0 TCP *:ssh_port (LISTEN)
avahi-dae 1567 avahi 12u IPv4 1752 0t0 UDP *:5353
avahi-dae 1567 avahi 13u IPv6 1753 0t0 UDP *:5353
avahi-dae 1567 avahi 14u IPv4 1754 0t0 UDP *:44917
avahi-dae 1567 avahi 15u IPv6 1755 0t0 UDP *:41552
cupsd 1573 root 8u IPv6 9520 0t0 TCP [X.X.X.X]:631 (LISTEN)
cupsd 1573 root 9u IPv4 9521 0t0 TCP X.X.X.X:631 (LISTEN)
salt-mini 1646 root 10u IPv4 11438 0t0 TCP X.X.X.X:40549->X.X.X.X:4506 (ESTABLISHED)
salt-mini 1646 root 21u IPv4 9774 0t0 TCP X.X.X.X:38633->X.X.X.X:4505 (ESTABLISHED)
syslog-ng 1655 root 9u IPv4 9565 0t0 TCP *:514 (LISTEN)
syslog-ng 1655 root 10u IPv4 9566 0t0 UDP *:514
mysqld 1799 mysql 10u IPv4 11451 0t0 TCP X.X.X.X:50000 (LISTEN)
searchd 1840 sphinxsearch 7u IPv4 9157 0t0 TCP *:9306 (LISTEN)
searchd 1840 sphinxsearch 8u IPv4 9158 0t0 TCP *:9312 (LISTEN)
ossec-csy 1942 ossecm 5u IPv4 9655 0t0 UDP X.X.X.X:50716->X.X.X.X:514
ossec-rem 2003 ossecr 4u IPv4 11322 0t0 UDP *:1514
ossec-rem 2004 ossecr 4u IPv4 9717 0t0 UDP *:5555
starman 3117 www-data 5u IPv6 11949 0t0 TCP *:3154 (LISTEN)
starman 3119 www-data 5u IPv6 11949 0t0 TCP *:3154 (LISTEN)
starman 3119 www-data 17u IPv4 214704 0t0 TCP X.X.X.X:43301->X.X.X.X:3154 (CLOSE_WAIT)
starman 3120 www-data 5u IPv6 11949 0t0 TCP *:3154 (LISTEN)
starman 3120 www-data 17u IPv4 214563 0t0 TCP X.X.X.X:43218->X.X.X.X:3154 (CLOSE_WAIT)
starman 3121 www-data 5u IPv6 11949 0t0 TCP *:3154 (LISTEN)
starman 3121 www-data 17u IPv4 216111 0t0 TCP X.X.X.X:43269->X.X.X.X:3154 (CLOSE_WAIT)
starman 3122 www-data 5u IPv6 11949 0t0 TCP *:3154 (LISTEN)
starman 3122 www-data 17u IPv4 215181 0t0 TCP X.X.X.X:43320->X.X.X.X:3154 (CLOSE_WAIT)
starman 3123 www-data 5u IPv6 11949 0t0 TCP *:3154 (LISTEN)
starman 3123 www-data 19u IPv4 213597 0t0 TCP X.X.X.X:43123->X.X.X.X:3154 (CLOSE_WAIT)
ntpd 3151 ntp 16u IPv4 10058 0t0 UDP *:123
ntpd 3151 ntp 17u IPv6 10059 0t0 UDP *:123
ntpd 3151 ntp 18u IPv4 10065 0t0 UDP X.X.X.X:123
ntpd 3151 ntp 19u IPv4 10066 0t0 UDP X.X.X.X:123
ntpd 3151 ntp 20u IPv6 10067 0t0 UDP [X.X.X.X]:123
ntpd 3151 ntp 21u IPv6 10068 0t0 UDP [X.X.X.X]:123
/usr/sbin 3216 root 4u IPv4 14414 0t0 TCP *:443 (LISTEN)
/usr/sbin 3216 root 5u IPv4 14417 0t0 TCP *:9876 (LISTEN)
/usr/sbin 3216 root 6u IPv4 14419 0t0 TCP *:444 (LISTEN)
/usr/sbin 3271 www-data 4u IPv4 14414 0t0 TCP *:443 (LISTEN)
/usr/sbin 3271 www-data 5u IPv4 14417 0t0 TCP *:9876 (LISTEN)
/usr/sbin 3271 www-data 6u IPv4 14419 0t0 TCP *:444 (LISTEN)
/usr/sbin 3272 www-data 4u IPv4 14414 0t0 TCP *:443 (LISTEN)
/usr/sbin 3272 www-data 5u IPv4 14417 0t0 TCP *:9876 (LISTEN)
/usr/sbin 3272 www-data 6u IPv4 14419 0t0 TCP *:444 (LISTEN)
/usr/sbin 3273 www-data 4u IPv4 14414 0t0 TCP *:443 (LISTEN)
/usr/sbin 3273 www-data 5u IPv4 14417 0t0 TCP *:9876 (LISTEN)
/usr/sbin 3273 www-data 6u IPv4 14419 0t0 TCP *:444 (LISTEN)
/usr/sbin 3274 www-data 4u IPv4 14414 0t0 TCP *:443 (LISTEN)
/usr/sbin 3274 www-data 5u IPv4 14417 0t0 TCP *:9876 (LISTEN)
/usr/sbin 3274 www-data 6u IPv4 14419 0t0 TCP *:444 (LISTEN)
/usr/sbin 3276 www-data 4u IPv4 14414 0t0 TCP *:443 (LISTEN)
/usr/sbin 3276 www-data 5u IPv4 14417 0t0 TCP *:9876 (LISTEN)
/usr/sbin 3276 www-data 6u IPv4 14419 0t0 TCP *:444 (LISTEN)
ssh 5618 root 3r IPv4 22928 0t0 TCP X.X.X.X:53562->X.X.X.X:ssh_port (ESTABLISHED)
ssh 5618 root 4u IPv6 22940 0t0 TCP [X.X.X.X]:3306 (LISTEN)
ssh 5618 root 5u IPv4 22941 0t0 TCP X.X.X.X:3306 (LISTEN)
tclsh 5690 SO-user 3u IPv4 20201 0t0 TCP X.X.X.X:45516->X.X.X.X:7736 (ESTABLISHED)
sshd 7744 root 3r IPv4 26522 0t0 TCP X.X.X.X:ssh_port->X.X.X.X:1166 (ESTABLISHED)
sshd 7912 SO-user 3u IPv4 26522 0t0 TCP X.X.X.X:ssh_port->X.X.X.X:1166 (ESTABLISHED)

=========================================================================
CPU Usage
=========================================================================
top - 19:09:41 up 4:24, 1 user, load average: 0.33, 0.29, 0.25
Tasks: 182 total, 1 running, 181 sleeping, 0 stopped, 0 zombie
Cpu(s): 4.3%us, 0.8%sy, 0.0%ni, 94.7%id, 0.2%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 32950856k total, 11549320k used, 21401536k free, 211116k buffers
Swap: 32099612k total, 0k used, 32099612k free, 8777560k cached

%CPU %MEM COMMAND
3.1 0.0 /var/ossec/bin/ossec-analysisd
2.3 0.1 perl /opt/elsa/node/elsa.pl -c /etc/elsa_node.conf
1.5 0.0 /var/ossec/bin/ossec-remoted
1.0 0.2 /usr/sbin/mysqld
0.7 0.4 /usr/sbin/syslog-ng -p /var/run/syslog-ng.pid
0.7 0.0 /var/ossec/bin/ossec-remoted
0.2 4.4 /usr/bin/searchd --nodetach
0.1 0.0 /usr/sbin/lightdm-gtk-greeter
0.0 0.0 /var/ossec/bin/ossec-syscheckd
0.0 0.1 /usr/bin/python /usr/bin/salt-minion
0.0 0.0 /usr/bin/X :0 -auth /var/run/lightdm/root/:0 -nolisten tcp vt7 -novtswitch -background none
0.0 0.3 starman worker -I/opt/elsa/web/lib --user=www-data --listen :3154 --daemonize --pid /var/run/starman.pid --error-log /var/log/starman.log /opt/elsa/web/lib/Web.psgi
0.0 0.3 starman worker -I/opt/elsa/web/lib --user=www-data --listen :3154 --daemonize --pid /var/run/starman.pid --error-log /var/log/starman.log /opt/elsa/web/lib/Web.psgi
0.0 0.3 starman worker -I/opt/elsa/web/lib --user=www-data --listen :3154 --daemonize --pid /var/run/starman.pid --error-log /var/log/starman.log /opt/elsa/web/lib/Web.psgi
0.0 0.3 starman worker -I/opt/elsa/web/lib --user=www-data --listen :3154 --daemonize --pid /var/run/starman.pid --error-log /var/log/starman.log /opt/elsa/web/lib/Web.psgi
0.0 0.3 starman worker -I/opt/elsa/web/lib --user=www-data --listen :3154 --daemonize --pid /var/run/starman.pid --error-log /var/log/starman.log /opt/elsa/web/lib/Web.psgi
0.0 0.0 [flush-8:16]
0.0 0.0 /usr/bin/ssh -q -N -o ServerAliveInterval 60 -o ServerAliveCountMax 3 -i /root/.ssh/securityonion -L 3306:X.X.X.X:3306 -R 50011:localhost:3154 SO-...@X.X.X.X
0.0 0.0 /usr/sbin/irqbalance
0.0 0.0 [kworker/3:0]
0.0 0.0 /sbin/init
0.0 0.0 [jbd2/sdb1-8]
0.0 0.0 [flush-8:32]
0.0 0.0 [jbd2/sda2-8]
0.0 0.0 /usr/sbin/ntpd -p /var/run/ntpd.pid -g -u 118:126
0.0 0.0 [jbd2/sdc1-8]
0.0 0.0 tclsh /usr/bin/ossec_agent.tcl -o -f /var/ossec/logs/alerts/alerts.log -i X.X.X.X -p 5 -c /etc/nsm/ossec/ossec_agent.conf
0.0 0.0 /usr/sbin/apache2 -k start
0.0 0.0 -bash
0.0 0.0 starman master -I/opt/elsa/web/lib --user=www-data --listen :3154 --daemonize --pid /var/run/starman.pid --error-log /var/log/starman.log /opt/elsa/web/lib/Web.psgi
0.0 0.0 [kworker/0:0]
0.0 0.0 [kworker/1:0]
0.0 0.0 /var/ossec/bin/ossec-logcollector
0.0 0.0 sshd: SO-user@pts/0
0.0 0.0 [flush-8:0]
0.0 0.0 [kworker/2:1]
0.0 0.0 /usr/lib/accountsservice/accounts-daemon
0.0 0.0 [kworker/u:30]
0.0 0.0 [kworker/0:2]
0.0 0.0 cron
0.0 0.0 [kworker/1:2]
0.0 0.0 [ksoftirqd/0]
0.0 0.0 PassengerHelperAgent
0.0 0.0 /var/ossec/bin/ossec-csyslogd
0.0 0.0 [ksoftirqd/3]
0.0 0.0 upstart-udev-bridge --daemon
0.0 0.0 /var/ossec/bin/ossec-maild
0.0 0.0 [kworker/3:1]
0.0 0.0 [ksoftirqd/1]
0.0 0.0 [migration/2]
0.0 0.0 /sbin/udevd --daemon
0.0 0.0 [migration/0]
0.0 0.0 [watchdog/0]
0.0 0.0 [migration/1]
0.0 0.0 [ksoftirqd/2]
0.0 0.0 [migration/3]
0.0 0.0 sshd: SO-user [priv]
0.0 0.0 [watchdog/1]
0.0 0.0 [watchdog/2]
0.0 0.0 [watchdog/3]
0.0 0.0 dbus-daemon --system --fork --activation=upstart
0.0 0.0 Passenger spawn server
0.0 0.0 /usr/sbin/console-kit-daemon --no-daemon
0.0 0.0 /usr/lib/policykit-1/polkitd --no-debug
0.0 0.0 PassengerLoggingAgent
0.0 0.0 /usr/sbin/apache2 -k start
0.0 0.0 [sync_supers]
0.0 0.0 [scsi_eh_31]
0.0 0.0 /usr/lib/upower/upowerd
0.0 0.0 [kworker/2:2]
0.0 0.0 avahi-daemon: running [SO-server.local]
0.0 0.0 lightdm
0.0 0.0 /var/ossec/bin/ossec-monitord
0.0 0.0 [kthreadd]
0.0 0.0 [cpuset]
0.0 0.0 [khelper]
0.0 0.0 [kdevtmpfs]
0.0 0.0 [netns]
0.0 0.0 [bdi-default]
0.0 0.0 [kintegrityd]
0.0 0.0 [kblockd]
0.0 0.0 [ata_sff]
0.0 0.0 [khubd]
0.0 0.0 [md]
0.0 0.0 [khungtaskd]
0.0 0.0 [kswapd0]
0.0 0.0 [ksmd]
0.0 0.0 [khugepaged]
0.0 0.0 [fsnotify_mark]
0.0 0.0 [ecryptfs-kthrea]
0.0 0.0 [crypto]
0.0 0.0 [kthrotld]
0.0 0.0 [scsi_eh_0]
0.0 0.0 [scsi_eh_1]
0.0 0.0 [scsi_eh_2]
0.0 0.0 [scsi_eh_3]
0.0 0.0 [scsi_eh_4]
0.0 0.0 [scsi_eh_5]
0.0 0.0 [scsi_eh_6]
0.0 0.0 [scsi_eh_7]
0.0 0.0 [scsi_eh_8]
0.0 0.0 [scsi_eh_9]
0.0 0.0 [scsi_eh_10]
0.0 0.0 [scsi_eh_11]
0.0 0.0 [scsi_eh_12]
0.0 0.0 [scsi_eh_13]
0.0 0.0 [scsi_eh_14]
0.0 0.0 [scsi_eh_15]
0.0 0.0 [scsi_eh_16]
0.0 0.0 [scsi_eh_17]
0.0 0.0 [scsi_eh_18]
0.0 0.0 [scsi_eh_19]
0.0 0.0 [scsi_eh_20]
0.0 0.0 [scsi_eh_21]
0.0 0.0 [scsi_eh_22]
0.0 0.0 [scsi_eh_23]
0.0 0.0 [scsi_eh_24]
0.0 0.0 [scsi_eh_25]
0.0 0.0 [scsi_eh_26]
0.0 0.0 [scsi_eh_27]
0.0 0.0 [scsi_eh_28]
0.0 0.0 [scsi_eh_29]
0.0 0.0 [scsi_eh_30]
0.0 0.0 [kworker/u:31]
0.0 0.0 [devfreq_wq]
0.0 0.0 [ttm_swap]
0.0 0.0 [mpt_poll_0]
0.0 0.0 [mpt/0]
0.0 0.0 [scsi_eh_32]
0.0 0.0 [ext4-dio-unwrit]
0.0 0.0 [ext4-dio-unwrit]
0.0 0.0 /sbin/udevd --daemon
0.0 0.0 /sbin/udevd --daemon
0.0 0.0 [kmpathd]
0.0 0.0 [kmpath_handlerd]
0.0 0.0 [ext4-dio-unwrit]
0.0 0.0 [kpsmoused]
0.0 0.0 upstart-socket-bridge --daemon
0.0 0.0 /usr/sbin/sshd -D
0.0 0.0 /usr/sbin/bluetoothd
0.0 0.0 [krfcommd]
0.0 0.0 avahi-daemon: chroot helper
0.0 0.0 /usr/sbin/cupsd -F
0.0 0.0 /sbin/getty -8 38400 tty4
0.0 0.0 /sbin/getty -8 38400 tty5
0.0 0.0 supervising syslog-ng
0.0 0.0 /sbin/getty -8 38400 tty2
0.0 0.0 /sbin/getty -8 38400 tty3
0.0 0.0 /sbin/getty -8 38400 tty6
0.0 0.0 acpid -c /etc/acpi/events -s /var/run/acpid.socket
0.0 0.0 atd
0.0 0.0 su -s /bin/sh -c exec "$0" "$@" sphinxsearch -- /usr/bin/searchd --nodetach
0.0 0.0 lightdm --session-child 16 19
0.0 0.0 /var/ossec/bin/ossec-execd
0.0 0.0 /bin/sh /usr/lib/lightdm/lightdm-greeter-session /usr/sbin/lightdm-gtk-greeter
0.0 0.0 //bin/dbus-daemon --fork --print-pid 5 --print-address 7 --session
0.0 0.0 /usr/lib/gvfs/gvfsd
0.0 0.0 /usr/lib/gvfs//gvfs-fuse-daemon -f /var/lib/lightdm/.gvfs
0.0 0.0 lightdm --session-child 12 19
0.0 0.0 /bin/sh -c perl /opt/elsa/node/elsa.pl -c /etc/elsa_node.conf
0.0 0.0 PassengerWatchdog
0.0 0.0 /usr/sbin/apache2 -k start
0.0 0.0 /usr/sbin/apache2 -k start
0.0 0.0 /usr/sbin/apache2 -k start
0.0 0.0 /usr/sbin/apache2 -k start
0.0 0.0 /sbin/getty -8 38400 tty1
0.0 0.0 /usr/lib/autossh/autossh -M 0 -q -N -o ServerAliveInterval 60 -o ServerAliveCountMax 3 -i /root/.ssh/securityonion -L 3306:X.X.X.X:3306 -R 50011:localhost:3154 SO-...@X.X.X.X
0.0 0.0 su - SO-user -- /usr/bin/ossec_agent.tcl -o -f /var/ossec/logs/alerts/alerts.log -i X.X.X.X -p 5 -c /etc/nsm/ossec/ossec_agent.conf
0.0 0.0 tail -n 0 -F /var/ossec/logs/alerts/alerts.log
0.0 0.0 sudo sostat-redacted
0.0 0.0 /bin/bash /usr/bin/sostat-redacted
0.0 0.0 /bin/bash /usr/bin/sostat
0.0 0.0 sed -r s/(\b[0-9]{1,3}\.){3}[0-9]{1,3}\b/X.X.X.X/g
0.0 0.0 sed -r s/([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}/MM:MM:MM:MM:MM:MM/g
0.0 0.0 sed -r s/(([0-9a-fA-F]{1,4}:){7,7}[0-9a-fA-F]{1,4}|([0-9a-fA-F]{1,4}:){1,7}:|([0-9a-fA-F]{1,4}:){1,6}:[0-9a-fA-F]{1,4}|([0-9a-fA-F]{1,4}:){1,5}(:[0-9a-fA-F]{1,4}){1,2}|([0-9a-fA-F]{1,4}:){1,4}(:[0-9a-fA-F]{1,4}){1,3}|([0-9a-fA-F]{1,4}:){1,3}(:[0-9a-fA-F]{1,4}){1,4}|([0-9a-fA-F]{1,4}:){1,2}(:[0-9a-fA-F]{1,4}){1,5}|[0-9a-fA-F]{1,4}:((:[0-9a-fA-F]{1,4}){1,6})|:((:[0-9a-fA-F]{1,4}){1,7}|:)|fe80:(:[0-9a-fA-F]{0,4}){0,4}%[0-9a-zA-Z]{1,}|::(ffff(:0{1,4}){0,1}:){0,1}((25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9]).){3,3}(25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])|([0-9a-fA-F]{1,4}:){1,4}:((25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9]).){3,3}(25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9]))\b/X.X.X.X/g
0.0 0.0 sed -r s/X:ssh_port/X:ssh_port/g
0.0 0.0 sed -r s/\*:ssh_port/*:ssh_port/g
0.0 0.0 sed -r s/SO-server/SO-server/g
0.0 0.0 sed -r s/SO-node/SO-node/g
0.0 0.0 sed -r s/SO-user|SO-user/SO-user/g
0.0 0.0 ps -eo pcpu,pmem,args --sort -pcpu

=========================================================================
Packets received during last monitoring interval (600 seconds)
=========================================================================
eth0: 355847

=========================================================================
Log Archive
=========================================================================
/nsm/sensor_data/SO-server-eth0/dailylogs/ - 19 days
80K .
4.0K ./2014-09-08
4.0K ./2014-09-10
4.0K ./2014-09-13
4.0K ./2014-09-15
4.0K ./2014-09-23
4.0K ./2014-10-09
4.0K ./2014-10-16
4.0K ./2014-10-30
4.0K ./2014-11-03
4.0K ./2014-11-24
4.0K ./2014-12-03
4.0K ./2014-12-13
4.0K ./2014-12-28
4.0K ./2015-01-13
4.0K ./2015-01-23
4.0K ./2015-01-30
4.0K ./2015-02-04
4.0K ./2015-02-05
4.0K ./2015-02-09

/nsm/bro/logs/ - 0 days
528K .
524K ./stats

=========================================================================
IDS Engine (snort) packet drops
=========================================================================
ERROR: No stats found in /nsm/sensor_data/*/snort-*.stats

=========================================================================
pf_ring stats
=========================================================================
PF_RING Version : 6.0.2 ($Revision: $)
Total rings : 0

Standard (non DNA) Options
Ring slots : 4096
Slot version : 16
Capture TX : Yes [RX+TX]
IP Defragment : No
Socket Mode : Standard
Transparent mode : Yes [mode 0]
Total plugins : 0
Cluster Fragment Queue : 0
Cluster Fragment Discard : 0

=========================================================================
ELSA
=========================================================================
Syslog-ng
Checking for process:
1654 supervising syslog-ng
1655 /usr/sbin/syslog-ng -p /var/run/syslog-ng.pid
Checking for connection:
Connection to localhost 514 port [tcp/shell] succeeded!

MySQL
Checking for process:
1799 /usr/sbin/mysqld
Checking for connection:
Connection to localhost 50000 port [tcp/*] succeeded!

Sphinx
Checking for process:
1715 su -s /bin/sh -c exec "$0" "$@" sphinxsearch -- /usr/bin/searchd --nodetach
Checking for connection:
Connection to localhost 9306 port [tcp/*] succeeded!

ELSA Buffers in Queue:
-rw-r--r-- 1 root root 13658589 Feb 9 19:09 /nsm/elsa/data/elsa/tmp/buffers/1423508928.79072
-rw-r--r-- 1 root root 19 Feb 9 19:09 /nsm/elsa/data/elsa/tmp/buffers/host_stats.tsv
-rw-r--r-- 1 root root 5969685 Feb 9 14:45 /nsm/elsa/data/elsa/tmp/buffers/1423493073.92419

ELSA Directory Sizes:
2.4T /nsm/elsa/data
69M /var/lib/mysql/syslog
2.5M /var/lib/mysql/syslog_data

ELSA Index Date Range:
MIN(start) MAX(end)
2014-11-26 18:44:46 2015-02-09 19:08:48

autossh
Checking for process:
5616 /usr/lib/autossh/autossh -M 0 -q -N -o ServerAliveInterval 60 -o ServerAliveCountMax 3 -i /root/.ssh/securityonion -L 3306:X.X.X.X:3306 -R 50011:localhost:3154 SO-...@X.X.X.X

Checking APIKEY:
APIKEY matches server.

starman
Checking for processes:
3117 starman master -I/opt/elsa/web/lib --user=www-data --listen :3154 --daemonize --pid /var/run/starman.pid --error-log /var/log/starman.log /opt/elsa/web/lib/Web.psgi
3119 starman worker -I/opt/elsa/web/lib --user=www-data --listen :3154 --daemonize --pid /var/run/starman.pid --error-log /var/log/starman.log /opt/elsa/web/lib/Web.psgi
3120 starman worker -I/opt/elsa/web/lib --user=www-data --listen :3154 --daemonize --pid /var/run/starman.pid --error-log /var/log/starman.log /opt/elsa/web/lib/Web.psgi
3121 starman worker -I/opt/elsa/web/lib --user=www-data --listen :3154 --daemonize --pid /var/run/starman.pid --error-log /var/log/starman.log /opt/elsa/web/lib/Web.psgi
3122 starman worker -I/opt/elsa/web/lib --user=www-data --listen :3154 --daemonize --pid /var/run/starman.pid --error-log /var/log/starman.log /opt/elsa/web/lib/Web.psgi
3123 starman worker -I/opt/elsa/web/lib --user=www-data --listen :3154 --daemoniz

[Doug Burks]

Have you looked at /nsm/elsa/data/elsa/log/node.log for any additional clues?

[Brian Kellogg]

Below are some log entries that I found:

* ERROR [2015/02/17 00:01:03] /opt/elsa/web/cron.pl (81) main:: 27717 [undef]
Error: DBD::mysql::st execute failed: Deadlock found when trying to get lock; try restarting transaction at /opt/elsa/web/../node//Indexer.pm line 1877.
* ERROR [2015/02/17 00:01:03] /opt/elsa/web/../node//Indexer.pm (2304) Indexer::_drop_indexes 27178 [undef]
Unknown index 8

* TRACE [2015/02/18 13:57:29] /opt/elsa/node/elsa.pl (275) main::__ANON__ 21666 [undef]
ALARM

* ERROR [2015/02/18 13:58:29] /opt/elsa/web/../node//Indexer.pm (3028) Indexer::_get_index_schema 17139 [undef]
Unable to get index schema for index temp_7, got output: Sphinx 2.0.7-id64-dev (rel20-r3736)
Copyright (c) 2001-2012, Andrew Aksyonoff
Copyright (c) 2008-2012, Sphinx Technologies Inc (http://sphinxsearch.com)

using config file '/etc/sphinxsearch/sphinx.conf'...
dumping header for index 'temp_7'...
dumping header file '/nsm/elsa/data/sphinx/temp_7.sph'...
FATAL: failed to load header: failed to open /nsm/elsa/data/sphinx/temp_7.sph: No such file or directory.
* INFO [2015/02/18 13:58:29] /opt/elsa/web/../node//Indexer.pm (2223) Indexer::_sphinx_index 17139 [undef]


When I check sostat every day I see the ELSA index MIN(start) date move several hours ahead so something is pruning the Index. Not sure what. I do not have a retention period set in elsa_node.conf. I do have the percentage for archives set to 0.

[Doug Burks]

I wonder if there are any non-error log entries in
/nsm/elsa/data/elsa/log/node.log that might have additional clues.

[Brian Kellogg]

yeah, my thoughts too, I'm still looking through it as I have time.

[Brian Kellogg]

* WARN [2015/02/17 00:12:12] /opt/elsa/web/../node//Indexer.pm (792) Indexer::_check_consolidate 32293 [undef]
All permanent indexes used and none to consolidate, we will have to overwrite a permanent index.

* WARN [2015/02/17 00:34:11] /opt/elsa/web/../node//Indexer.pm (792) Indexer::_check_consolidate 6157 [undef]
All permanent indexes used and none to consolidate, we will have to overwrite a permanent index.


Not sure I'm understanding what these are telling me. I do see several of these. From what I read on another forum contrary to the comment in elsa_node.conf sphinx does not actually lose logs.

[Brian Kellogg]

Am I understanding this correctly that Sphinx only keeps 200 indexes on disk and then overwrites them?

If I am then I can increase that num_indexes field to 400, but that won't buy me the kind of index timeframe I'm looking for. I read on the Sphinx group that Sphinx really isn't tested with any setting above 400.

I want to keep up to 365 days of indexes on a very large partition. Still trying to understand how the num_indexes, temp indexes, and everything else work together.

Ideally I want all my OSSEC agents reporting to one server and have all the logs indexed by ELSA and quickly searchable for up to 365 days of history. I could use archives, but I've been in situations where I need to quickly access historical information from a year ago and would rather not.

[Brian Kellogg]

Ok, I think it may finally be sinking in.

https://groups.google.com/forum/#!searchin/enterprise-log-search-and-archive/%22allowed_temp_percent%22/enterprise-log-search-and-archive/auUSYj77ctw/mzF-YqVa5KMJ

The above is a good write up on the tuning options of ELSA. I think if I up the allowed_temp_percent and up the perm_index_size that that will solve both problems I am having. One, this is a low volume log box evidently thus it is overwriting indexes on disk early and two I have more disk space than 200 x 10 million logs can fit. So I need more index space in my perm indexes.