sosetup -f sosetup.conf errors out on configuring elsa

[James Gordon]

Hello all,

We recently deployed a small group of Security Onion boxes in our environment in a server / sensor deployment. We've had some issues with ELSA since day one and recently decided to re-run sosetup on all the sensors to get ELSA working again. Our sensors (and our console) do not have internet access - our console will eventually, but there are no plans to connect the sensors to the internet. Attempting to run sosetup (utilizing a configuration file) results in hanging at "Configuring Elsa" and eventually errors out. After this, all services start but ELSA still does not function. I have read that this could be a result of the sensors not being able to download new GeoIP rules, but I have not stumbled on a workaround for this while running sosetup using a config file. However the error messages we are getting indicate to me that sosetup is failing to edit the elsa_web.conf on the server. The user given to sosetup has passwordless sudo access on the server. If this is an issue of downloading geopIP rules, what is the best method to disable attempts to download rules while running sosetup utilizing a configuration file, or is there another problem going on here??

Here are the errors we are getting, and the configuration file I am using as well. The elsa_web.conf file on the sensor also appears suspicious to me, and I can upload that if it may be helpful.

Thanks in advance for any help on this issue!


root@sensor:/etc# sosetup -f /home/user/sosetup.conf

Security Onion Setup

Ready to configure system using parameters in /home/user/sosetup.conf.

WARNING! Continuing will destroy any existing data/config.
Are you sure you want to continue?
Type yes to continue or anything else to exit.
yes

Please wait while...
setting OS timezone to UTC...
setting OSSEC timezone to UTC...
restarting OSSEC...
stopping all NSM services...
creating Sguil sensor(s)...
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
creating Sguil sensor: sensor-eth0...
creating Sguil sensor: sensor-eth1...
configuring /etc/nsm/securityonion.conf...
stopping and disabling Apache...
configuring salt...
starting all Security Onion services...
configuring ELSA...
/usr/lib/ruby/1.9.1/json/common.rb:148:in `parse': 746: unexpected token at '{ "apikeys": { "elsa":

"7f5d85fd4e361258dd643a09314e356e" }, "version": { "Author": "mcholste", "Date": "2014-07-17 15:12:58 -0700 (Thu,

17 Jul 2014)", "Rev": "1205", "Sphinx": "Sphinx 2.1.9" }, "peers": { "127.0.0.1": { "url":

"http://127.0.0.1:3154/", "username": "elsa", "apikey": "7f5d85fd4e361258dd643a09314e356e" }, },

"admin_email_address": "root@localhost", "connectors": { }, "dashboards": { }, "datasources": { }, "transforms": {

"whois": { "known_subnets": { "10.0.0.0": { "end": "10.255.255.255", "org": "MyOrg"

}, "192.168.0.0": { "end": "192.168.255.255", "org": "MyOrg" }, "172.16.0.0": {

"end": "172.31.255.255", "org": "MyOrg" } }, "known_orgs": { "MyOrg": {

"name": "MyOrg", "org": "MyOrg", "descr": "MyOrg", "cc": "US", "country": "United

States", "city": "Anytown", "state": "Somestate" } } }, "parse": { "tld": [

{ "field": "domain", "pattern": "\\.([a-zA-Z]+)$", "extractions": [ "tld"

] }, { "field": "site", "pattern": "\\.([a-zA-Z]+)$", "extractions": [

"tld" ] }, { "field": "uri", "pattern": "\\.([a-zA-Z]+)(:|/|$)",

"extractions": [ "tld" ] } ], "url": [ { "field": "uri",

"pattern": "(?:(?<proto>[a-zA-Z]+)://)?(?:(?<username>[^/]+):(?<password>[^/]+)@)?(?<domain>\\d{1,3}\\.\\d{1,3}\\.\\d

{1,3}\\.\\d{1,3}|[^/]+\\.(?<tld>[a-zA-Z]+))(?::(?<port>\\d+))?(?<resource>/[^?]*)?(?:\\?(?<query_string>.*))?$",

"extractions": [ "proto", "username", "password", "domain", "tld",

"port", "resource", "query_string" ] } ], "mimetype": [ {

"field": "msg", "pattern": "[\"'\\(\\[\\s\\|;:](?<mime>(?<type>application|audio|chemical|image|

message|model|multipart|text|video)/(?<subtype>[\\w-_]+))[\"'\\)\\]\\s\\|;:]", "extractions": [

"mime", "type", "subtype" ] } ] } }, "plugins": { "SNORT":

"Info::Snort", "WINDOWS": "Info::Windows", "URL": "Info::Url", "BRO_NOTICE": "Info::Bro" }, "info": {

"snort": { "url_templates": [ "http://doc.emergingthreats.net/bin/view/Main/%d" ] }, "url": {

"url_templates": [ "http://whois.domaintools.com/%s" ] }, "windows": { "url_templates": [

"http://www.ultimatewindowssecurity.com/securitylog/encyclopedia/event.aspx?eventid=%d" ] } },

"max_concurrent_archive_queries": 4, "schedule_interval": 60, "node_info_cache_timeout": 60, "email": {

"display_address": "norepl...@example.com", "base_url": "http://elsa/", "subject": "ELSA Alert" }, "link_key":

"secret", "yui": { "local": "inc" }, "data_db": { "db": "syslog", "username": "elsa", "password": "biglog"

}, "meta_db": { "dsn": "dbi:mysql:database=elsa_web", "username": "elsa", "password": "biglog" }, "auth": {

"method": "security_onion" }, "admin_groups": [ "system", "admin" ], "auth_db": { "dsn":

"dbi:mysql:database=securityonion_db", "username": "root", "password": "", "auth_statement": "SELECT PASSWORD

(password) FROM user_info WHERE username=?", "email_statement": "SELECT email FROM user_info WHERE username=?" },

"peer_id_multiplier": 1000000000000, "query_timeout": 55, "pcap_url": "/capme", "logdir": "/nsm/elsa/data/elsa/log",

"buffer_dir": "/nsm/elsa/data/elsa/tmp/buffers", "debug_level": "TRACE", "default_start_time_offset": 2, "livetail": {

"poll_interval": 5, "time_limit": 3600 }}' (JSON::ParserError)
from /usr/lib/ruby/1.9.1/json/common.rb:148:in `parse'
from /usr/bin/securityonion_elsa_register.rb:214:in `parse_conf'
from /usr/bin/securityonion_elsa_register.rb:348:in `<main>'

Security Onion Setup is now complete!



FWIW I've seen similiar threads where the issue appears to be permissions on the server's elsa_web.conf - permissions appear to be correct on ours.

user@server:/etc$ ls -lah elsa_web.conf
-rw-rw-r-- 1 root securityonion 4.1K Aug 30 14:39 elsa_web.conf

[Philip Plantamura]

Has elsa_web.conf on the server been manually edited previously?

There may be an extraneous comma in the peers section after the localhost peer:

"apikey": "7f5d85fd4e361258dd643a09314e356e" }, <-- this comma

Of course, ensure you backup the elsa_web.conf before modifying it.

Phil

> --
> Follow Security Onion on Twitter!
> https://twitter.com/securityonion
> ---
> You received this message because you are subscribed to the Google Groups "security-onion" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
> To post to this group, send email to securit...@googlegroups.com.
> Visit this group at https://groups.google.com/group/security-onion.
> For more options, visit https://groups.google.com/d/optout.

[James Gordon]

Thanks Phil! The file was manually edited a little while back in an attempt at adding ELSA apikeys. Removing that comma seems to have done the trick - sosetup ran and completed and I see the node and logs in the top right of ELSA.

Unfortunately, now ELSA is batching every query run against it and never returns any results. ELSA's sostat on the sensor and server seem to check out to me, so I'm not sure where to begin looking for this one.

sostat on the server:
Syslog-ng
Checking for process:
22801 /usr/sbin/syslog-ng -p /var/run/syslog-ng.pid
Checking for connection:
Connection to localhost 514 port [tcp/shell] succeeded!

MySQL
Checking for process:
22513 /usr/sbin/mysqld
Checking for connection:
Connection to localhost 3306 port [tcp/mysql] succeeded!

Sphinx
Checking for process:
18629 su -s /bin/sh -c exec "$0" "$@" sphinxsearch -- /usr/bin/searchd --nodetach
18631 /usr/bin/searchd --nodetach
Checking for connection:
Connection to localhost 9306 port [tcp/*] succeeded!

ELSA Buffers in Queue:
2
If this number is consistently higher than 20, please see:
https://github.com/Security-Onion-Solutions/security-onion/wiki/FAQ#why-does-sostat-show-a-high-number-of-elsa-buffers-in-queue

ELSA Directory Sizes:
2.4G /nsm/elsa/data
53M /var/lib/mysql/syslog
32K /var/lib/mysql/syslog_data

ELSA Index Date Range
If you don't have at least 2 full days of logs in the Index Date Range,
then you'll need to increase log_size_limit in /etc/elsa_node.conf.
MIN(start) MAX(end)
2016-04-22 23:50:11 2016-08-30 17:31:55

ELSA Log Node SSH Tunnels:
PORT NODE IP/STATUS
50000 SO-node X.X.X.X


sostat on the sensor:
Syslog-ng
Checking for process:
22941 /usr/sbin/syslog-ng -p /var/run/syslog-ng.pid
Checking for connection:
Connection to localhost 514 port [tcp/shell] succeeded!

MySQL
Checking for process:
7861 /usr/sbin/mysqld
Checking for connection:
Connection to localhost 50000 port [tcp/*] succeeded!

Sphinx
Checking for process:
11399 su -s /bin/sh -c exec "$0" "$@" sphinxsearch -- /usr/bin/searchd --nodetach
11405 /usr/bin/searchd --nodetach
Checking for connection:
Connection to localhost 9306 port [tcp/*] succeeded!

ELSA Buffers in Queue:
3
If this number is consistently higher than 20, please see:
https://github.com/Security-Onion-Solutions/security-onion/wiki/FAQ#why-does-sostat-show-a-high-number-of-elsa-buffers-in-queue

ELSA Directory Sizes:
66G /nsm/elsa/data
97M /var/lib/mysql/syslog
720K /var/lib/mysql/syslog_data

ELSA Index Date Range
If you don't have at least 2 full days of logs in the Index Date Range,
then you'll need to increase log_size_limit in /etc/elsa_node.conf.
MIN(start) MAX(end)
2016-08-30 13:16:48 2016-08-30 17:30:25

autossh
Checking for process:
13529 /usr/lib/autossh/autossh -M 0 -q -N -o ServerAliveInterval 60 -o ServerAliveCountMax 3 -i /root/.ssh/securityonion -L 3306:X.X.X.X:3306 -R :localhost:3154 SO-user@soflokydlnetfm00
13703 /usr/lib/autossh/autossh -M 0 -q -N -o ServerAliveInterval 60 -o ServerAliveCountMax 3 -i /root/.ssh/securityonion -L 3306:X.X.X.X:3306 -R 50000:localhost:3154 SO-user@soflokydlnetfm00
18987 /usr/lib/autossh/autossh -M 0 -q -N -o ServerAliveInterval 60 -o ServerAliveCountMax 3 -i /root/.ssh/securityonion -L 3306:X.X.X.X:3306 -R :localhost:3154 SO-user@soflokydlnetfm00

Checking APIKEY:
APIKEY matches server.

starman
Checking for processes:
13690 starman master -I/opt/elsa/web/lib --user=www-data --listen :3154 --daemonize --pid /var/run/starman.pid --error-log /var/log/starman.log /opt/elsa/web/lib/Web.psgi
13692 starman worker -I/opt/elsa/web/lib --user=www-data --listen :3154 --daemonize --pid /var/run/starman.pid --error-log /var/log/starman.log /opt/elsa/web/lib/Web.psgi
13693 starman worker -I/opt/elsa/web/lib --user=www-data --listen :3154 --daemonize --pid /var/run/starman.pid --error-log /var/log/starman.log /opt/elsa/web/lib/Web.psgi
13694 starman worker -I/opt/elsa/web/lib --user=www-data --listen :3154 --daemonize --pid /var/run/starman.pid --error-log /var/log/starman.log /opt/elsa/web/lib/Web.psgi
13695 starman worker -I/opt/elsa/web/lib --user=www-data --listen :3154 --daemonize --pid /var/run/starman.pid --error-log /var/log/starman.log /opt/elsa/web/lib/Web.psgi
13696 starman worker -I/opt/elsa/web/lib --user=www-data --listen :3154 --daemonize --pid /var/run/starman.pid --error-log /var/log/starman.log /opt/elsa/web/lib/Web.psgi

Thanks,

James Gordon

[Philip Plantamura]

Hi, James, try rebooting that sensor.
Any clues in /nsm/elsa/data/elsa/log/ ?
Sorry to be short.
Phil

If the problem persists

[Philip Plantamura]

If the problem persists, submit a full sostat-redacted.
Phil

[Wes Lambert]

James,

What date are you trying to start searching from (in ELSA)?  If you are trying to search beyond the index for the sensors (into the archive (s)), it may be the reason for the queries batching.  By default, ELSA is set to use two days for the search range.  What happens if you just choose today, or run a query with "nobatch:1" appended?

Ex. class=BRO_HTTP nobatch:1

Thanks,
Wes

>>> > an email to security-onion+unsubscribe@googlegroups.com.
>>> > To post to this group, send email to security-onion@googlegroups.com.

>>> > Visit this group at https://groups.google.com/group/security-onion.
>>> > For more options, visit https://groups.google.com/d/optout.
>>
>> --
>> Follow Security Onion on Twitter!
>> https://twitter.com/securityonion
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "security-onion" group.
>> To unsubscribe from this group and stop receiving emails from it, send an

>> email to security-onion+unsubscribe@googlegroups.com.
>> To post to this group, send email to security-onion@googlegroups.com.

>> Visit this group at https://groups.google.com/group/security-onion.
>> For more options, visit https://groups.google.com/d/optout.

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to security-onion+unsubscribe@googlegroups.com.
To post to this group, send email to security-onion@googlegroups.com.

[James Gordon]

Phil, Wes, thanks for the suggestions.

Phil, I bounced the sensor but the issue still persists. I've attached a full sostat-redacted from the sensor. Nothing stands out to me in /nsm/elsa/data/elsa/logs, but I'm not sure what to look for. Here's a tail on the node.log:
using config file '/etc/sphinxsearch/sphinx.conf'...
indexing index 'temp_77'...
collected 180693 docs, 84.6 MB
sorted 10.0 Mhits, 100.0% done
total 180693 docs, 84554182 bytes
total 5.768 sec, 14658857 bytes/sec, 31326.10 docs/sec
total 5 reads, 0.029 sec, 16588.5 kb/call avg, 5.9 msec/call avg
total 79 writes, 0.089 sec, 1996.1 kb/call avg, 1.1 msec/call avg
WARNING: failed to scanf pid from pid_file '/var/run/sphinxsearch/searchd.pid'.
WARNING: indices NOT rotated.
* TRACE [2016/08/30 19:34:13] /opt/elsa/web/../node/Indexer.pm (2205) Indexer::_sphinx_index 15465 [undef]
ran cmd: /usr/bin/indexer --config /etc/sphinxsearch/sphinx.conf --rotate temp_77 2>&1
* INFO [2016/08/30 19:34:13] /opt/elsa/web/../node/Indexer.pm (2234) Indexer::_sphinx_index 15465 [undef]
Indexed temp_77 with 180693 rows in 5.81636 seconds (31066.30520 rows/sec)
* TRACE [2016/08/30 19:34:13] /opt/elsa/web/../node/Indexer.pm (3035) Indexer::_get_index_schema 15465 [undef]
Getting index schema with command: /usr/bin/indextool --config /etc/sphinxsearch/sphinx.conf --dumpheader temp_77 2>&1
* TRACE [2016/08/30 19:34:13] /opt/elsa/web/../node/Indexer.pm (1889) Indexer::_index_records 15465 [undef]
Unlocked indexes between 8333886811 and 8334067503
* ERROR [2016/08/30 19:34:13] /opt/elsa/web/cron.pl (96) main:: 15465 [undef]
Error: DBI connect('host=127.0.0.1;port=9306','',...) failed: Can't connect to MySQL server on '127.0.0.1' (111) at /opt/elsa/web/../node/Indexer.pm line 2809.
* TRACE [2016/08/30 19:34:13] /opt/elsa/web/cron.pl (99) main:: 15465 [undef]
cron.pl finished.


Wes, I'll be honest in that I'm not familiar with using ELSA. Since we deployed Security Onion we've been utilizing bro logs from the CLI, so I'm much more comfortable there. To validate that ELSA was working I was only clicking on some of the default queries on the right hand side - so DNS top IP's, HTTP top IP's, etc. The searches default to a date string: NaN-NaN-NaN NaN:NaN:NaN.

I appended nobatch:1 to one of those default queries and it never returned anything (but didn't batch the query, either). Then I changed the date to this morning, and appended nobatch:1 and have not yet gotten any results - I ran this about 5 minutes ago.

Thanks!

James Gordon

> >>> > an email to security-onio...@googlegroups.com.
>
> >>> > To post to this group, send email to securit...@googlegroups.com.

>
> >>> > Visit this group at https://groups.google.com/group/security-onion.
>
> >>> > For more options, visit https://groups.google.com/d/optout.
>
> >>
>
> >> --
>
> >> Follow Security Onion on Twitter!
>
> >> https://twitter.com/securityonion
>
> >> ---
>
> >> You received this message because you are subscribed to the Google Groups
>
> >> "security-onion" group.
>
> >> To unsubscribe from this group and stop receiving emails from it, send an
>

> >> email to security-onio...@googlegroups.com.
>
> >> To post to this group, send email to securit...@googlegroups.com.

>
> >> Visit this group at https://groups.google.com/group/security-onion.
>
> >> For more options, visit https://groups.google.com/d/optout.
>
>
>
> --
>
> Follow Security Onion on Twitter!
>
> https://twitter.com/securityonion
>
> ---
>
> You received this message because you are subscribed to the Google Groups "security-onion" group.
>

> To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
>
> To post to this group, send email to securit...@googlegroups.com.

[Wes]

From your sostat:

It looks like Snort is not working as it should (likely due to PF_RING):

* snort-1 (alert data)[ FAIL ]
* stale PID file found, process will be restarted at the next 5-minute interval!
* snort-2 (alert data)[ FAIL ]
* stale PID file found, process will be restarted at the next 5-minute interval!
* snort-3 (alert data)[ FAIL ]
* stale PID file found, process will be restarted at the next 5-minute interval!
* snort-4 (alert data)[ FAIL ]
* stale PID file found, process will be restarted at the next 5-minute interval!
* snort-5 (alert data)[ FAIL ]
* stale PID file found, process will be restarted at the next 5-minute interval!

Out of curiosity, when is the last time you ran soup on this machine?

Ex. sudo soup -y

It appears to be using the old sostat format and the PF_RING version appears to be older (6.2.0). It should be 6.4.1.

You could also try checking /var/log/nsm/[hostname-interface]/snortu-*.log
to identify the issue for Snort (replacing hostname-interface with the name of your sensor and sensor interface, and the asterisk with the appropriate number):

sudo tail -100 /var/log/nsm/[hostname-intferface]/snortu-*.log

--------

Sphinx
Checking for process:
Checking for connection:
nc: connect to localhost port 9306 (tcp) failed: Connection refused

ELSA Buffers in Queue:
14

If this number is consistently higher than 20, please see:
https://github.com/Security-Onion-Solutions/security-onion/wiki/FAQ#why-does-sostat-show-a-high-number-of-elsa-buffers-in-queue

It looks like Sphinx has an issue as well. As a result, ELSA buffers look as thought they are piling up.

Try the following for Sphinx:

-Rename the files in the /var/lib/sphinxsearch/data/ directory (watch out for line-wrapping):

sudo mv /var/lib/sphinxsearch/data/binlog.001 /var/lib/sphinxsearch/data/binlog.001.bak
sudo mv /var/lib/sphinxsearch/data/binlog.lock /var/lib/sphinxsearch/data/binlog.lock.bak
sudo mv /var/lib/sphinxsearch/data/binlog.meta /var/lib/sphinxsearch/data/binlog.meta.bak

..and restart the machine.

Thanks,
Wes