Elsa doesn't seem to be working

[packetsmacker]

I can't seem to get Elsa to work. Maybe a user error but I just get a white page. Doesn't matter what I click on in the left window I get zero results. How do I start troubleshooting this?

[Doug Burks]

Hi packetsmacker,

Please run the following command:

sudo sostat-redacted

There will be a lot of output, so you may need to increase your
terminal's scroll buffer OR redirect the output of the command to a
file:

sudo sostat-redacted > sostat-redacted.txt 2>&1

sostat-redacted will automatically redact any IPv4/IPv6/MAC addresses,
but there may be additional sensitive info that you still need to
redact manually.

Attach the output to your email in plain text format (.txt) OR use a
service like http://pastebin.com.

On Tue, Nov 17, 2015 at 3:19 PM, packetsmacker <ott....@gmail.com> wrote:
> I can't seem to get Elsa to work. Maybe a user error but I just get a white page. Doesn't matter what I click on in the left window I get zero results. How do I start troubleshooting this?
>

> --
> You received this message because you are subscribed to the Google Groups "security-onion" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
> To post to this group, send email to securit...@googlegroups.com.
> Visit this group at http://groups.google.com/group/security-onion.
> For more options, visit https://groups.google.com/d/optout.



--
Doug Burks
Need Security Onion Training or Commercial Support?
http://securityonionsolutions.com

[Wes]

packetsmacker,

Try taking a look at the ELSA logs, in /nsm/elsa/data/elsa/log/.

Also, have you tried restarting apache2?

sudo service apache2 restart

Have you tried rebooting the machine?

Thanks,
Wes

[packetsmacker]

I had looked at that log. I ran the sostat cmd and found this in the log.

ELSA Buffers in Queue:
31542
If this number is consistently higher than 20, please see:
https://github.com/Security-Onion-Solutions/security-onion/wiki/FAQ#why-does-sostat-show-a-high-number-of-elsa-buffers-in-queue

Went to the link and followed the suggestions.

ran this sudo grep syslogs_archive_1 /nsm/elsa/data/elsa/log/node.log

Got a bunch of hit then ran sudo securityonion-elsa-reset-archive

Didn't fix the white page issue

Ran the sosat and got the same results for Elsa buffers. I am going to upload the sostat in a minute.

[packetsmacker]

I forgot to add this.

I looked a the logs and I couldn't make sense of it.

I restarted apache and ran soup.

[Wes]

packetsmacker,

You could try deleting the buffers from /nsm/elsa/data/elsa/tmp/buffers/ if you don't need them--then restart apache2 and see if ELSA will process the buffers appropriately.

Also, try running mysqlcheck -A and see if you get any errors.

Is Sphinx successfully running?

Have you had any ungraceful shutdowns or disk space issues lately?

Thanks,
Wes

[packetsmacker]

mysqlcheck -A has been running for about 4 hours so far no errors.

Sphinx is running

No ungraceful shutdowns and disk space is good.

Some back ground might be helpful. I think you helped me with this post https://groups.google.com/forum/#!searchin/security-onion/packetsmacker/security-onion/uQhInQ4221Y/GX0S0fFODQAJ

To summarize that post the box didn't have enough hardware given the rules i needed to run at that point on the network. We moved it to a different location to focus on a segment of the network. Now the load is good. I wonder if there is a issue with the databases due to the large amount of traffic it was getting from its original location.

Would it be worth starting over? I don't really want to run the setup again because we have a far amount of custom config. I know I could back it up and fix it but it seems like it would be faster if I could just blow away the data in the databases. Is that a good idea? How would I do it? Would that fix the problem? If it didn't fix it at lest we could rule that out.

[Wes]

packetsmacker,

You could try running mysqlcheck for only the ELSA-related DBs:

"mysqlcheck --databases elsa_web -u root"

"mysqlcheck --databases syslog -u root"

"mysqlcheck --databases syslog_data -u root"

I suppose you could try altering the log_size_limit setting in /etc/elsa_node.conf to purge old records, then restart syslog-ng--but I'm not sure this would fix your issue. I believe your best bet is to check the databases, repair them if needed, ensure Sphinx is working appropriately, ensure there are no issues with Apache, and then if none of that works or gives you a clue, cut your losses and backup what ever config/data you need, and re-run setup--this is probably the safest and most consistent approach.

You could also try the manual alternative to the elsa-reset-archive script:

# Become root
sudo -i
# Stop services
service nsm stop
service syslog-ng stop
# Cleanup database tables and entries
mysql -uroot syslog_data -e "DROP TABLE syslog_data.syslogs_archive_1"

mysql -uroot syslog_data -e "DELETE FROM syslog.tables WHERE table_name='syslog_data.syslogs_archive_1'"
# Cleanup database files
rm /nsm/elsa/data/elsa/mysql/syslogs_archive_1*
rm /var/lib/mysql/syslog_data/syslogs_archive_1*
# Restart services
service mysql restart
service syslog-ng restart
service nsm start

Thanks,
Wes

[packetsmacker]

Deleting the buffers seemed to have fixed Elsa. My load has been really high since the mysqlcheck finished. Its at 31 right now. I am guessing I need to run setup again. Something just isn't right.