Additional info: the syslog file shows things humming along nicely and then suddenly a row of
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
and it freezes. I thought it may be due to the open-vm-tools drivers because that was referenced in one of the lines directly before the freeze, so I reinstalled SO without the open-vm-tools drivers, and it froze again the next day. I thought it may be because of the type of virtual NIC I chose when creating the VM, so I rebuilt it again and instead of the VMXNET3 card type, I chose the E1000 card type. This also made no difference with or without the vm-tools drivers. SO froze the next day again. I thought maybe some update was causing it, so I reinstalled again and did not run any updates. The build was straight from the ISO with no updates. This ran a week with no issues, but today it's frozen twice. I've got full packet capturing turned off because it was consuming too much disk space, but all the other services offered in the setup are enabled. The VM has 16 Gigs RAM, 1 CPU which hovers at 10%, 60 Gig drive which is only 40% full, and 3 Nics.
Anyone experienced this? Ideas?
--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onion+unsubscribe@googlegroups.com.
To post to this group, send email to security-onion@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.
Kris,
Are you behind a proxy?
https://github.com/Security-Onion-Solutions/security-onion/wiki/Proxy
Could there be any firewall rules blocking the update traffic?
Thanks,
Wes
Update: I ran 'soup' to get the latest updates and it came back with many 'Failed to fetch' ubuntu dists errors. I tried running 'apt-get update' and it threw the same errors. I checked my DNS in /etc/resolv.conf and they're correct, and pinging the internet works, and browsing websites works.
--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onion+unsubscribe@googlegroups.com.
To post to this group, send email to security-onion@googlegroups.com.
Have you tried a fresh copy of the ISO? Also, did you verify the previous download?
Thanks,
Wes
Here is the last few syslog lines before it froze the last time.
---------------------------------------
Sep 16 18:54:01 OnionServer CRON[13870]: (root) CMD (/usr/sbin/nsm_sensor_clean -y >> /var/log/nsm/sensor-clean.log 2>&1)
Sep 16 18:54:31 OnionServer vmsvc[5829]: [ warning] [guestinfo] Failed to get vmstats.
Sep 16 18:55:01 OnionServer CRON[14279]: (root) CMD (find /var/www/so/capme/pcap/*.pcap -mmin +5 -delete >/dev/null 2>&1)
Sep 16 18:55:01 OnionServer CRON[14280]: (root) CMD (sh /opt/elsa/contrib/securityonion/contrib/securityonion-elsa-cron.sh > /dev/null 2>&1)
Sep 16 18:55:01 OnionServer CRON[14281]: (root) CMD ([ -d /var/lib/mysql/securityonion_db/ ] && /usr/bin/php -e /var/www/so/squert/.inc/ip2c.php 1 > /dev/null 2>&1)
Sep 16 18:55:01 OnionServer CRON[14282]: (root) CMD (/usr/sbin/nsm_sensor_clean -y >> /var/log/nsm/sensor-clean.log 2>&1)
Sep 16 18:55:01 OnionServer CRON[14286]: (root) CMD (/usr/sbin/so-bro-cron >> /var/log/nsm/so-bro-cron.log 2>&1)
Sep 16 18:55:01 OnionServer vmsvc[5829]: [ warning] [guestinfo] Failed to get vmstats.
Sep 16 18:55:31 OnionServer vmsvc[5829]: [ warning] [guestinfo] Failed to get vmstats.
Sep 16 18:56:01 OnionServer CRON[14549]: (root) CMD (sh /opt/elsa/contrib/securityonion/contrib/securityonion-elsa-cron.sh > /dev/null 2>&1)
Sep 16 18:56:01 OnionServer CRON[14550]: (root) CMD (find /var/www/so/capme/pcap/*.pcap -mmin +5 -delete >/dev/null 2>&1)
Sep 16 18:56:01 OnionServer CRON[14551]: (root) CMD (/usr/sbin/nsm_sensor_clean -y >> /var/log/nsm/sensor-clean.log 2>&1)
Sep 16 18:56:01 OnionServer vmsvc[5829]: [ warning] [guestinfo] Failed to get vmstats.
Sep 16 18:56:31 OnionServer vmsvc[5829]: [ warning] [guestinfo] Failed to get vmstats.
Sep 16 18:57:01 OnionServer CRON[14961]: (root) CMD (find /var/www/so/capme/pcap/*.pcap -mmin +5 -delete >/dev/null 2>&1)
Sep 16 18:57:01 OnionServer CRON[14962]: (root) CMD (sh /opt/elsa/contrib/securityonion/contrib/securityonion-elsa-cron.sh > /dev/null 2>&1)
Sep 16 18:57:01 OnionServer CRON[14963]: (root) CMD (/usr/sbin/nsm_sensor_clean -y >> /var/log/nsm/sensor-clean.log 2>&1)
Sep 16 18:57:01 OnionServer vmsvc[5829]: [ warning] [guestinfo] Failed to get vmstats.
Sep 16 18:57:31 OnionServer vmsvc[5829]: [ warning] [guestinfo] Failed to get vmstats.
Sep 16 18:58:01 OnionServer vmsvc[5829]: [ warning] [guestinfo] Failed to get vmstats.
Sep 16 18:58:01 OnionServer CRON[15017]: (root) CMD (find /var/www/so/capme/pcap/*.pcap -mmin +5 -delete >/dev/null 2>&1)
Sep 16 18:58:01 OnionServer CRON[15016]: (root) CMD (sh /opt/elsa/contrib/securityonion/contrib/securityonion-elsa-cron.sh > /dev/null 2>&1)
Sep 16 18:58:01 OnionServer CRON[15018]: (root) CMD (/usr/sbin/nsm_sensor_clean -y >> /var/log/nsm/sensor-clean.log 2>&1)
Sep 16 18:58:31 OnionServer vmsvc[5829]: [ warning] [guestinfo] Failed to get vmstats.
Sep 16 18:59:01 OnionServer CRON[15132]: (root) CMD ( ( date ; /usr/sbin/nsm_server_ps-restart --if-stale ; /usr/sbin/nsm_sensor_ps-restart --if-stale) >> /var/log/nsm/watchdog.log)
Sep 16 18:59:01 OnionServer CRON[15131]: (root) CMD (find /var/www/so/capme/pcap/*.pcap -mmin +5 -delete >/dev/null 2>&1)
Sep 16 18:59:01 OnionServer CRON[15133]: (root) CMD (sh /opt/elsa/contrib/securityonion/contrib/securityonion-elsa-cron.sh > /dev/null 2>&1)
Sep 16 18:59:01 OnionServer CRON[15134]: (root) CMD (/usr/sbin/nsm_sensor_clean -y >> /var/log/nsm/sensor-clean.log 2>&1)
Sep 16 18:59:01 OnionServer vmsvc[5829]: [ warning] [guestinfo] Failed to get vmstats.
Sep 16 18:59:31 OnionServer vmsvc[5829]: [ warning] [guestinfo] Failed to get vmstats.
Maybe try an approach similar to the following?
Thanks,
Wes
--------------------------------
Sep 19 13:10:01 OnionServer CRON[5297]: (root) CMD (sh /opt/elsa/contrib/securityonion/contrib/securityonion-elsa-cron.sh > /dev/null 2>&1)
Sep 19 13:10:01 OnionServer CRON[5296]: (root) CMD (find /var/www/so/capme/pcap/*.pcap -mmin +5 -delete >/dev/null 2>&1)
Sep 19 13:10:01 OnionServer CRON[5298]: (root) CMD ([ -d /var/lib/mysql/securityonion_db/ ] && /usr/bin/php -e /var/www/so/squert/.inc/ip2c.php 1 > /dev/null 2>&1)
Sep 19 13:10:01 OnionServer CRON[5299]: (root) CMD (/usr/sbin/nsm_sensor_clean -y >> /var/log/nsm/sensor-clean.log 2>&1)
Sep 19 13:10:01 OnionServer CRON[5303]: (root) CMD (/usr/sbin/so-bro-cron >> /var/log/nsm/so-bro-cron.log 2>&1)
Sep 19 13:11:01 OnionServer CRON[5636]: (root) CMD (sh /opt/elsa/contrib/securityonion/contrib/securityonion-elsa-cron.sh > /dev/null 2>&1)
Sep 19 13:11:01 OnionServer CRON[5637]: (root) CMD (find /var/www/so/capme/pcap/*.pcap -mmin +5 -delete >/dev/null 2>&1)
Sep 19 13:11:01 OnionServer CRON[5638]: (root) CMD (/usr/sbin/nsm_sensor_clean -y >> /var/log/nsm/sensor-clean.log 2>&1)
Sep 19 13:12:01 OnionServer CRON[5816]: (root) CMD (find /var/www/so/capme/pcap/*.pcap -mmin +5 -delete >/dev/null 2>&1)
Sep 19 13:12:01 OnionServer CRON[5817]: (root) CMD (sh /opt/elsa/contrib/securityonion/contrib/securityonion-elsa-cron.sh > /dev/null 2>&1)
Sep 19 13:12:01 OnionServer CRON[5818]: (root) CMD (/usr/sbin/nsm_sensor_clean -y >> /var/log/nsm/sensor-clean.log 2>&1)
Sep 19 13:13:01 OnionServer CRON[5999]: (root) CMD (sh /opt/elsa/contrib/securityonion/contrib/securityonion-elsa-cron.sh > /dev/null 2>&1)
Sep 19 13:13:01 OnionServer CRON[6000]: (root) CMD (find /var/www/so/capme/pcap/*.pcap -mmin +5 -delete >/dev/null 2>&1)
Sep 19 13:13:01 OnionServer CRON[6003]: (root) CMD (/usr/sbin/nsm_sensor_clean -y >> /var/log/nsm/sensor-clean.log 2>&1)
Sep 19 13:14:02 OnionServer CRON[6231]: (root) CMD (find /var/www/so/capme/pcap/*.pcap -mmin +5 -delete >/dev/null 2>&1)
Sep 19 13:14:02 OnionServer CRON[6232]: (root) CMD ( ( date ; /usr/sbin/nsm_server_ps-restart --if-stale ; /usr/sbin/nsm_sensor_ps-restart --if-stale) >> /var/log/nsm/watchdog.log)
Sep 19 13:14:02 OnionServer CRON[6233]: (root) CMD (sh /opt/elsa/contrib/securityonion/contrib/securityonion-elsa-cron.sh > /dev/null 2>&1)
Sep 19 13:14:02 OnionServer CRON[6243]: (root) CMD (/usr/sbin/nsm_sensor_clean -y >> /var/log/nsm/sensor-clean.log 2>&1)
Kris,
Please provide the output pf sostat-redacted, attaching as a text file, or using a service like Pastebin.com
Thanks,
Wes
Kris,
Have you considered disabling the following services and/or increasing the available number of CPU cores?
* prads
* sancp_agent
* pads_agent
* http_agent
It seems like you are trying to run this all off of a single core--is this true?
If so, you will definitely want to cut back on the number of services enabled and/or the number of interfaces you are attempting to monitor.
https://github.com/Security-Onion-Solutions/security-onion/wiki/Best-Practices
Thanks,
Wes
-----------------------------------
Sep 26 01:57:01 OnionServer CRON[29407]: (root) CMD (find /var/www/so/capme/pcap/*.pcap -mmin +5 -delete >/dev/null 2>&1)
Sep 26 01:57:01 OnionServer CRON[29408]: (root) CMD (sh /opt/elsa/contrib/securityonion/contrib/securityonion-elsa-cron.sh > /dev/null 2>&1)
Sep 26 01:57:01 OnionServer CRON[29409]: (root) CMD (/usr/sbin/nsm_sensor_clean -y >> /var/log/nsm/sensor-clean.log 2>&1)
Sep 26 01:58:01 OnionServer CRON[29516]: (root) CMD (find /var/www/so/capme/pcap/*.pcap -mmin +5 -delete >/dev/null 2>&1)
Sep 26 01:58:01 OnionServer CRON[29517]: (root) CMD (sh /opt/elsa/contrib/securityonion/contrib/securityonion-elsa-cron.sh > /dev/null 2>&1)
Sep 26 01:58:01 OnionServer CRON[29518]: (root) CMD (/usr/sbin/nsm_sensor_clean -y >> /var/log/nsm/sensor-clean.log 2>&1)
Sep 26 01:59:01 OnionServer CRON[29574]: (root) CMD (find /var/www/so/capme/pcap/*.pcap -mmin +5 -delete >/dev/null 2>&1)
Sep 26 01:59:01 OnionServer CRON[29575]: (root) CMD ( ( date ; /usr/sbin/nsm_server_ps-restart --if-stale ; /usr/sbin/nsm_sensor_ps-restart --if-stale) >> /var/log/nsm/watchdog.log)
Sep 26 01:59:01 OnionServer CRON[29576]: (root) CMD (sh /opt/elsa/contrib/securityonion/contrib/securityonion-elsa-cron.sh > /dev/null 2>&1)
Sep 26 01:59:01 OnionServer CRON[29577]: (root) CMD (/usr/sbin/nsm_sensor_clean -y >> /var/log/nsm/sensor-clean.log 2>&1)
Kris,
Could you provide a fresh copy of sostat-redacted?
Thanks,
Wes
Kris,
The average load looks a little high:
=========================
CPU Usage
=========================================================================
Load average for the last 1, 5, and 15 minutes:
6.41 5.28 2.74
Processing units: 1
If load average is higher than processing units,
then tune until load average is lower than processing units.
Could this be related?
I would try adding another core or two to see if that helps any and/or try disabling some more IDS rules -- looks like you have over 27k enabled.
Typically for the cores, if you have netsniff-ng installed, you would want a core for (netsniff-ng)each monitoring interface, a core for each IDS process on each interface, and a single core for the OS.
You may want to also try decreasing either the number of interfaces you are monitoring, or decreasing the number of IDS processes.
Thanks,
Wes
See attached output from sostat-redacted
--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onion+unsubscribe@googlegroups.com.
You shouldn't need to configure anything additional.
Is there any configs that I need to edit after adding an additional core so the load gets distributed by SO properly?
--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onion+unsubscribe@googlegroups.com.
To post to this group, send email to security-onion@googlegroups.com.
I downloaded the latest ISO and reinstalled. It's been stable and running smooth for 2 weeks now. Here's the differences between the unstable VM and the new stable VM.
Unstable:
'VM Tools' installed
'XRDP' and 'XFCE4' installed
Stable:
No 'VM Tools' installed
No 'XRDP' or 'XFCE4' installed
I'll post again if the new VM freezes.