Only one node showing in Elsa

[Elec]

I hope this is not a duplicate issue but I have searched and searched and cannot figure out what is going on. I have one Analyst setup on a VM and a sensor setup on a server. Squert and Sguil are both running excellent but Elsa is only seeing one node. I am seeing this error on the sensor in /nsm/elsa/data/elsa/log/web.log:

* ERROR [2016-05-01 07:35:02] /opt/elsa/web/lib/Utils.pm (142) Utils::_dbh_error_handler 11789 [undef]
DBI connect('database=elsa_web','elsa',...) failed: Access denied for user 'elsa'@'localhost' (using password: YES) QUERY:

[Wes]

Elec,

Please provide the output of sostat-redacted:
https://github.com/Security-Onion-Solutions/security-onion/wiki/MailingLists#include-sostat-redacted-output

Thanks,
Wes

[Elec]

Wes,

Please see attached. Thanks so much!

[Wes]

On Tuesday, May 3, 2016 at 12:38:06 PM UTC-4, Elec wrote:
> Wes,
>
> Please see attached. Thanks so much!

Elec,

Are you able to manually SSH to the server using the sensor account?

Is the sensor defined correctly in the server's /etc/elsa_web.conf?

Does the APIKEY for the sensor match (from it's elsa_web.conf to that of the master's)?

Try taking a look in the server's log files (/nsm/elsa/data/elsa/log/*) to see if you notice any errors that may be related to your issue.

I would also try logging into Sguil and categorizing some events, as you currently have quite a few uncategorized events:

Ex. 70561

Failure to do so could cause performance impact and potential database table corruption to securityonion_db (Sguil's DB).

Also, please provide the output of sostat-redacted for the sensor.

Finally, did you happen to install the sensor/master using the ISO or via PPA?

If you did install via PPA, did you make sure to perform the following step?

echo "debconf debconf/frontend select noninteractive" | sudo debconf-set-selections

Thanks,
Wes

[Elec]

Wes,

I am able to ssh from the sensor to the server.

The sensor does seem to be defined correctly in the server's /etc/elsa_web.conf

Both APIKEYs do indeed match. I attached a screenshot of both API Keys (Elsa_web_conf_sensor.jpg and Elsa_web_conf_server.jpg)

The only log file that I am receiving an error in is /nsm/elsa/data/elsa/log/node.log and the errors are attached in a screenshot (Elsa_node_log_error.jpg).

I am working on categorizing events more. They are already tuned ALOT by changing the definition of $EXTERNAL_NET from "any" to "!$HOME_NET".

I tried doing the same thing with sostat for the sensor as I did with the server and it keeps getting hung on "Checking APIKEY". I waited for an hour and it still would not complete so I ran "sudo sostat" without piping it out and it completed (see attached screenshots; Elsa_error_log.jpg and Elsa_error_log2.jpg). As you can see, it says "APIKEY not found on master server". Also as you can see on the Starman processes, it looks like it is writing to an error log. So I also attached the screenshot for that log (starman_log.jpg). I am not sure what it means by "Couldn't unlink "/var/run/starman.pid" [Permission Denied]", but I thought it was interesting.

[Wes]

Elec,

It looks like there is no entry for the sensor in the server's elsa_web.conf.

The APIKEY that you thought matched was only referring to

You will need to have something like the following:

},
"peers": {
"127.0.0.1": {
"url": "http://127.0.0.1:3154/",
"username": "elsa",
"apikey": "yyyyyyyyyyyyyyyyyyyyy"
},
"sensor1": {
"url": "http://127.0.0.1:50000/",
"username": "elsa",
"apikey": "xxxxxxxxxxxxxxxxxxxxx"
}
}

Above, the server has it's own APIKEY (127.0.0.1) and is referencing the sensor and its APIKEY (sensor1)

In the sensor's elsa_web.conf, you would have something like:

},
"peers": {
"127.0.0.1": {
"url": "http://127.0.0.1:3154/",
"username": "elsa",
"apikey": "xxxxxxxxxxxxxxxxxxxxxx"
}
},

This defines the sensor's APIKEY.

Try modifying your configuration as above and restarting services to see if it helps.

Also, you may want to change your APIKEYs now. Generally, you do not want to expose this information externally, as it is essentially a password for each of your machines to talk to one another.

Thanks,
Wes

[Elec]

Okay,

I have added the sensor and changed my APIKEYs and restarted services and it is now recognizing both nodes. However, it is showing "2 node(s) with undefined logs indexed and undefined archived" and the from time is "1969-12-29 19:00:00"

[Wes Lambert]

Before we try anything else, please reboot the server and wait for all services to start. Then reboot the sensor.

Thanks,
Wes

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

[Elec]

Wes,

I did as you said and there is no change. I am still getting "2 node(s) with undefined logs indexed and undefined archived" and the from time is still "1969-12-29 19:00:00".

[Wes]

Elec,

Try running through some of the steps Doug has described here to see if it helps:

https://groups.google.com/d/msg/security-onion/8_ZwTnY0Law/PhZjv0sKJPoJ

Thanks,
Wes

[Elec]

Wes,

I do not have any 50### ports listening on my master server. So I tried running "sudo pkill -USR1 autossh" on my sensor to restart the autossh tunnel and then re-ran the lsof command on the server and still nothing. Port 50000 is listening on the sensor but that is it.

The weird thing is that now Elsa is showing 2 nodes and 83000+ logs but when I query anything, I get no results and the earliest archive date matches the start date of the server's Index Date Range instead of the sensor's (screenshot attached).

When doing a sostat on the sensor and server (screenshots attached), I noticed that the date ranges for elsa index date ranges are different. Also, on the server, it is showing onionsensor1 on port 50000 as being "disconnected" in regards to Log Node SSH Tunnels.

[Elec]

Also, sostat for the sensor is still hanging on Checking APIKEY

[Wes]

Elec,

I would again ensure the APIKEY for the sensor defined in the sensor's elsa_web.conf matches the entry defined in the server's elsa_web.conf.

Could you please post the resultant elsa_web.conf files, redacting sensitive information as necessary?

Also, please post the output of sudo lsof -nP -i |grep "127.0.0.1:50... (LISTEN)".

Have you again tried rebooting the sensor/server, waiting for the server to completely start, then restarting the sensor?

Thanks,
Wes

[Elec]

Following your steps earlier, I believe the API Keys to be correct. I have verified that the "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" keys do match.

Attached you will find the elsa_web.conf files for both server and sensor. One thing I did notice that is missing is the "Nodes" section. I am not sure if that is an update or if that is my problem?

I also attached screenshots of the lsof command output.

I have restarted the master, ran 'sudo service nsm status' which came back as failed. Then waited a few minutes and re-ran the status which came back as ok (just to verify that everything is running). Then I restarted the sensor.

Still no luck. I appreciate your patience!

[Wes]

Elec,

The "nodes" portion you were referring to is from an older version, I believe.

Out of curiosity, what value do you have for ELSA under the ELSA stanza in /etc/nsm/securityonion.conf (on the sensor)?

Ex.
# ELSA
ELSA=

Please attach new sostat-redacted output for the server and sensor.

Also try this on the following, copying output:

sensor--> pgrep -lf autossh
sudo pkill -USR1 autossh



server--> nc localhost 50000
nc localhost 50001

Thanks,
Wes

[Elec]

ELSA = YES

I have attached both sostat-redacted results but again, the sostat-redacted for the sensor keeps hanging after prompting for the master server's password so it is incomplete.

There is no output on the sensor for pgrep -lf autossh even after running sudo pkill -USR1 autossh.

Also there is no output on the server when running nc localhost 50000 and nc localhost 50001.

[Wes]

Elec,

Since you setup the sensor, did the nodes ever display correctly? Does/did the sensor account on the server have sudo privileges when you ran setup?

Do you have a logfile for elsa registration on the sensor, in /var/log/nsm/so-elsa/? If so, does it offer any additional insight?

Please attach /var/log/nsm/sosetup.log (for the sensor) as well, redacting sensitive information as necessary.

It may be beneficial to look through the sosetup script for the specific steps to configure/register an ELSA log node during sensor setup:

Ex.
if [ $SERVER -ne 1 ] && [ "$ELSA" = "YES" ]; then
# Register the log node and restart the server.
SSH_CMD="/usr/bin/securityonion_elsa_register.rb --register --peer-name `hostname` --force"
ELSA_REGISTER_RESPONSE=`ssh -i $KEY $SSH_USERNAME@$SERVERNAME $SSH_CMD`
ELSA_PORT=`echo $ELSA_REGISTER_RESPONSE | cut -d',' -f1`
ELSA_APIKEY=`echo $ELSA_REGISTER_RESPONSE | cut -d',' -f2`
# Copy the ELSA port back into the SSH config file for future use.
echo "ELSA_PORT=$ELSA_PORT" >> $SSH_CONF
# Update the local ELSA API key
/usr/bin/securityonion_elsa_register.rb --update-apikey $ELSA_APIKEY
# Since the securityonion service started before we the ELSA ports
# were determined, we need to tear down the SSH tunnel and restart it.
# Kill autossh with SIGINT
if pgrep autossh>/dev/null; then
kill -SIGINT `pgrep autossh`
# Restart the autossh tunnel
/usr/bin/autossh -M 0 -f -q -N -o "ServerAliveInterval 60" -o "ServerAliveCountMax 3" -i "$KEY" -L 3306:127.0.0.1:3306 -R $ELSA_PORT:localhost:3154 $SSH_USERNAME@$SERVERNAME
fi
if [ "$UPDATE_ELSA_SERVER" = "YES" ]; then
# Instruct the server to restart apache2.
ssh -i "$KEY" -t $SSH_USERNAME@$SERVERNAME service apache2 restart >> $LOG 2>&1
fi
fi

Also, take a look here:
https://groups.google.com/d/msg/security-onion/UpGTdKbuBLY/EucDwqeTHEsJ

...and maybe try following the steps here:
https://groups.google.com/d/msg/security-onion/UpGTdKbuBLY/9xEjKm64zaQJ

Thanks,
Wes