ELSA stopped gathering new data possibly after upgrade
Danie de Jager
I upgraded to the latest build using uBuntu 14.04 and noticed only now that since then ELSA stopped gathering new information. When I change the date range all the latest dates are greyed out for the last couple of months. Sguil works fine.
I'm not sure where to start troubleshooting. The Sphinxsearch service is up on my master and sensor. There are no log files in /var/log/sphinxsearch/ on my master and sensor.
From sostat:
=========================================================================
ELSA
=========================================================================
Syslog-ng
Checking for process:
2078 /usr/sbin/syslog-ng -p /var/run/syslog-ng.pid
Checking for connection:
nc: connect to localhost port 514 (tcp) failed: Connection refused
MySQL
Checking for process:
2065 /usr/sbin/mysqld
Checking for connection:
Connection to localhost 3306 port [tcp/mysql] succeeded!
Sphinx
Checking for process:
1981 su -s /bin/sh -c exec "$0" "$@" sphinxsearch -- /usr/bin/searchd --nodetach
2179 /usr/bin/searchd --nodetach
Checking for connection:
Connection to localhost 9306 port [tcp/*] succeeded!
ELSA Buffers in Queue:
6
If this number is consistently higher than 20, please see:
https://github.com/Security-Onion-Solutions/security-onion/wiki/FAQ#why-does-sostat-show-a-high-number-of-elsa-buffers-in-queue
ELSA Directory Sizes:
3.4G /nsm/elsa/data
89M /var/lib/mysql/syslog
32K /var/lib/mysql/syslog_data
ELSA Index Date Range
If you don't have at least 2 full days of logs in the Index Date Range,
then you'll need to increase log_size_limit in /etc/elsa_node.conf.
+---------------------+---------------------+
| MIN(start) | MAX(end) |
+---------------------+---------------------+
| 2015-05-28 08:36:41 | 2016-02-11 07:09:05 |
+---------------------+---------------------+
ELSA Log Node SSH Tunnels:
PORT NODE IP/STATUS
50000 seconion-sensor1 10.2.37.204
Wes
Danie,
Did you follow ALL of the steps here, making sure to not replace the syslog-ng.conf file?
https://github.com/Security-Onion-Solutions/security-onion/wiki/Upgrading-from-12.04-to-14.04
Have you tried restarting syslog-ng or rebooting?
Also check /nsm/elsa/data/elsa/log/, especially node.log, which you could try tailing (tail -f) when restarting syslog-ng.
How does your syslog-ng.conf (/etc/syslog-ng/syslog-ng.conf) compare to /opt/elsa/contrib/securityonion/contrib/securityonion-syslog-ng.conf?
Also, please include the full output of sostat-redacted:
Thanks,
Wes
Danie de Jager
> Danie,
>
> Did you follow ALL of the steps here, making sure to not replace the syslog-ng.conf file?
> https://github.com/Security-Onion-Solutions/security-onion/wiki/Upgrading-from-12.04-to-14.04
>
> Have you tried restarting syslog-ng or rebooting?
>
> Also check /nsm/elsa/data/elsa/log/, especially node.log, which you could try tailing (tail -f) when restarting syslog-ng.
>
> How does your syslog-ng.conf (/etc/syslog-ng/syslog-ng.conf) compare to /opt/elsa/contrib/securityonion/contrib/securityonion-syslog-ng.conf?
>
> Also, please include the full output of sostat-redacted:
>
> https://github.com/Security-Onion-Solutions/security-onion/wiki/MailingLists#include-sostat-redacted-output
>
> Thanks,
> Wes
Thanks Wes,
I followed the steps. I don't use syslog unless SO uses syslog for some if its processes I wouldn't touch the configs. Doing a diff on the 2 files shows a big difference. Regrettably I took over this install from someone else who previously went live with the POC so luckily not a lot of tuning was done but the system is dog slow.
Danie
Danie de Jager
Here is the redacted output on pastebin:
http://pastebin.com/6RQi5Tx8
Wes
Danie,
First, I would try backing up the current syslog-ng.conf file you have in /etc/syslog-ng.conf and replace it with the one in /opt/elsa/contrib/securityonion/contrib/securityonion-syslog-ng.conf, then restart syslog-ng.
Not related to this particular issue, I would also log into Sguil and categorize your uncategorized events. Failure to do so could result in poor performance and/or database table corruption.
It also looks like you are running PRADS/pads_agent on your sensor. You may want to disable this for better performance on your sensor.
Thanks,
Wes
Danie de Jager
Thanks Wes,
When we disable PRADS the guys doing the investigation are missing the events around the incident so we enabled it again.
I will do as you suggested with Syslog.
Is there a schematic to show how the different systems combine to make SO? If ELSA is not getting new data what can cause that?
Danie
Danie de Jager
#head /etc/syslog-ng/syslog-ng.conf
@version: 3.5
@include "scl.conf"
@include "`scl-root`/system/tty10.conf"
# Syslog-ng configuration file, compatible with default Debian syslogd
# installation.
# First, set some global options.
options { chain_hostnames(off); flush_lines(0); use_dns(no); use_fqdn(no);
owner("root"); group("adm"); perm(0640); stats_freq(0);
----------------------------------------------------------------------------
#head /opt/elsa/contrib/securityonion/contrib/securityo
@version: 3.5
source s_syslog { unix-dgram("/dev/log"); };
source s_network {
tcp();
udp();
};
parser p_db {
db-parser(file("/opt/elsa/node/conf/patterndb.xml"));
daniej@seconion-master:~$ head /etc/syslog-ng/syslog-ng.conf
Wes
Danie,
Those (the includes in the /etc/syslog-ng) are not required. Have you replaced the file (using the one from /opt/elsa) and tried restarting syslog-ng?
Thanks,
Wes
Wes
Bro offers much of the same information as PRADS viewable through ELSA.
You can have a look here to get an idea of the general SO architecture:
https://github.com/Security-Onion-Solutions/security-onion/wiki/Architecture
From my experience, I have found that if ELSA is not getting new data, it could be because:
-There is an issue with syslog-ng (bad config, etc.)
-There is an issue with Sphinx indexing the buffers
-Corrupt MySQL database (elsa_web, syslog_data)
-Overwhelmed machine (not enough resources, or other issues cause resource hogging)
These are just a few things I have noticed. You can learn more about ELSA specifically, here:
https://github.com/mcholste/elsa/wiki/Documentation
Thanks,
Wes
Danie de Jager
|
Danie de Jager Striata Global Security Engineer |
| Office: +27 11 5309600 |
| Striata on: Twitter | LinkedIn | Facebook | www.striata.com |
--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to a topic in the Google Groups "security-onion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/security-onion/-z38LqYP8xs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.
Danie de Jager
|
Danie de Jager Striata Global Security Engineer |
| Office: +27 11 5309600 |
| Striata on: Twitter | LinkedIn | Facebook | www.striata.com |
Wes
It pulls in the information, formats/filters, as necessary and ships the data off to ELSA (Sphinx for indexing, MySQL for overall storage).
Does that help clarify?
Thanks,
Wes
Danie de Jager
|
Danie de Jager Striata Global Security Engineer |
| Office: +27 11 5309600 |
| Striata on: Twitter | LinkedIn | Facebook | www.striata.com |
Wes
tail -f /nsm/elsa/data/elsa/log/node.log
See if that produces a useful error or clue.
Also, try rebooting and see if that helps.
Thanks,
Wes
Danie de Jager
|
Danie de Jager Striata Global Security Engineer |
| Office: +27 11 5309600 |
| Striata on: Twitter | LinkedIn | Facebook | www.striata.com |
Wes
Please provide the latest sostat-redacted output.
Thanks,
Wes
Danie de Jager
Wes
Danie,
Please provide the output of the following:
sudo lsof -nP -i | grep 514
and the output of the following:
sudo dpkg -l | grep syslog-ng
Thanks,
Wes
Danie de Jager
|
Danie de Jager Striata Global Security Engineer |
| Office: +27 11 5309600 |
| Striata on: Twitter | LinkedIn | Facebook | www.striata.com |
Wes
If you have the ability, it may be best (and quicker) to compare a test VM (syslog-ng.conf, and other files) running Security Onion 14.04 to your current machine.
Thanks,
Wes
Danie de Jager
|
Danie de Jager Striata Global Security Engineer |
| Office: +27 11 5309600 |
| Striata on: Twitter | LinkedIn | Facebook | www.striata.com |
Wes
Sensors store most of their own data (with the exception of alert data) in their own ELSA database, so they should have their own index/date range. After running sostat on the sensor, what is the index range displayed?
Please include the sostat-redacted output for the sensor.
Thanks,
Wes
Danie de Jager
|
Danie de Jager Striata Global Security Engineer |
| Office: +27 11 5309600 |
| Striata on: Twitter | LinkedIn | Facebook | www.striata.com |
Danie de Jager
|
Danie de Jager Striata Global Security Engineer |
| Office: +27 11 5309600 |
| Striata on: Twitter | LinkedIn | Facebook | www.striata.com |
Danie de Jager
|
Danie de Jager Striata Global Security Engineer |
| Office: +27 11 5309600 |
| Striata on: Twitter | LinkedIn | Facebook | www.striata.com |
Wes
> Further the previous mail I found:
>
> #tail /var/log/nsm/sensor-clean.log
> OOPS: The sensor "" does not exist!
> OOPS: The sensor "" does not exist!
>
>
> Is that normal on a master instance?
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> Danie de Jager
>
> Striata Global Security Engineer
>
>
>
> Office: +27 11 5309600
>
>
>
> Striata on: Twitter | LinkedIn | Facebook | www.striata.com
>
>
>
> On 11 May 2016 at 08:53, Danie de Jager <danie....@striata.com> wrote:
>
>
> Hi Wes,
>
>
> I can confirm that syslog-ng is now running and after 24 hours ELSA's interface is now updating to the latest date but there is no information in any of the reports.
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
What happens if you query from the 10th (2016-05-10 00:00:00) onward, or the 11th ?
Try looking in:
/nsm/elsa/data/elsa/log/node.log
and
/nsm/elsa/data/elsa/log/web.log
for clues.
Please attach the most recent output of sostat-redacted for the master and the sensor.
The content you mentioned within the sensor-clean.log is normal for the master, as this job is used to purge data on sensors.
Thanks,
Wes
Danie de Jager
|
Danie de Jager Striata Global Security Engineer |
| Office: +27 11 5309600 |
| Striata on: Twitter | LinkedIn | Facebook | www.striata.com |
Wes
Try running the same query appending "nobatch:1" if you do not want it to batch.
Ex. class=BRO_CONN "-" groupby:srcip nobatch:1
Have you tried to see what happens when you query starting from a more recent date, such as the 10th or 11th?
Again, please attach the sostat-redacted output for both the sensor and master.
Thanks,
Wes
Danie de Jager
|
Danie de Jager Striata Global Security Engineer |
| Office: +27 11 5309600 |
| Striata on: Twitter | LinkedIn | Facebook | www.striata.com |
Danie de Jager
|
Danie de Jager Striata Global Security Engineer |
| Office: +27 11 5309600 |
| Striata on: Twitter | LinkedIn | Facebook | www.striata.com |
Wes
Danie,
It looks like, from your sensor's sostat output, that syslog-ng is not running as it should (on the sensor). I would recommend replacing the file on the sensor as you did with the master. Let's try that first and go from there.
Thanks,
Wes
Danie de Jager
|
Danie de Jager Striata Global Security Engineer |
| Office: +27 11 5309600 |
| Striata on: Twitter | LinkedIn | Facebook | www.striata.com |
Danie,
It looks like, from your sensor's sostat output, that syslog-ng is not running as it should (on the sensor). I would recommend replacing the file on the sensor as you did with the master. Let's try that first and go from there.
Thanks,
Wes
Danie de Jager
|
Danie de Jager Striata Global Security Engineer |
| Office: +27 11 5309600 |
| Striata on: Twitter | LinkedIn | Facebook | www.striata.com |
Wes
Thanks,
Wes