Elsa "Invalid query result"

Devinder kang

unread,

Aug 31, 2016, 6:09:00 PM8/31/16

to security-onion

Hello Everyone,

I'm new to security onion, I have recently install verison 14.04. It was working fine for couple of days. Today I was trying to apply BPF at /etc/nsm/hostname-interface/bpf.conf . I don't know what happened after that elsa is throwing error "Invalid query result".

sudo service nsm status

"stale PID file found, process will be restarted at the next 5-minute interval!"

I have also attached squild.log

where I'm getting below error.

mysqlsel/db server: Error writing file '/tmp/MYbA0OPh' (Errcode: 28)
while executing
"mysqlsel $MAIN_DB_SOCKETID $tmpQry -list"
invoked from within
"if { $mergeTableListArray(event) != "" } {

# Get the archived alerts
LogMessage "Querying DB for archived events..."
set MAJOR_MYSQL_VERS..."
(file "/usr/bin/sguild" line 737)
2016-08-31 15:43:44 pid(15198) Unknown command received from sguild:

Thanks,

Devinder

sostat-redacted.txt

squild.txt

Wes

unread,

Aug 31, 2016, 6:40:16 PM8/31/16

to security-onion

This is probably why you are experiencing the error with the query:

Sphinx
Checking for process:
Checking for connection:
nc: connect to localhost port 9306 (tcp) failed: Connection refused

Try renaming the files in /var/lib/sphinxsearch/data/

and restarting to see if that resolves that particular issue for you.

Also try checking /var/log/nsm/[hostname-interface]/snortu-*.log for clues as to why Snort is experiencing an issue--you may find that this is due to your BPF change(s). If so, clear out the out the file and see if that helps with your issue.

You may also want to consider implementing autocats to help with managing your uncategorized Sguil events:

https://github.com/Security-Onion-Solutions/security-onion/wiki/ManagingAlerts
https://github.com/Security-Onion-Solutions/security-onion/wiki/ManagingAlerts#autocategorize-events

Consider the following to clear out some of the uncategorized events:

-Lower the number for DAYSTOKEEP in /etc/nsm/securityonion.conf (maybe "1")
-Run "sudo sguil-db-purge"

Last, you may want to tune better tune PF_RING to avoid dropping so many packets:

https://github.com/Security-Onion-Solutions/security-onion/wiki/PF_RING#tuning
https://github.com/Security-Onion-Solutions/security-onion/wiki/PF_RING#slots

Hope this helps.

Thanks,
Wes

Devinder kang

unread,

Sep 1, 2016, 2:42:03 PM9/1/16

to security-onion

Hi Wes,

Thanks for the quick reply. I did followed your instructions.

Renamed the two files in /var/lib/sphinxsearch/data/

I have cleared everything in /etc/nsm/server-eth1$ sudo vi bpf.conf

Implement autocats to manage uncategorized Sguil events.

Lower the number for DAYSTOKEEP in /etc/nsm/securityonion.conf (to "1")
Ran "sudo sguil-db-purge" ,which did purged the database.

PF_Ring was already tunned:
Number of bro worker = 3
Number of squil agents = 3.

Did increased the PF_Ring slot size from default to 8000.

Reboot the vm, now in Elsa "Invalid query result" is okay, but now when I submit the query I get zero result. Doesn't matter what category I select.

and when I run below command, I still get below error.

sudo service nsm status

"stale PID file found, process will be restarted at the next 5-minute interval!"

I 'm not sniffing much traffic, but I don't know why ELSA buffer is too high.

I have assigned 8 GB RAM, 4 cores cpu, and i'm only sniffing 3 ports, which has almost zero traffic.

I have again attached sostat-reducted and copy of snortu-*.log

Please let me know what else needs to be done.

Thanks again
Devinder

sostat-reducted.txt

snortu-3.txt

Wes Lambert

unread,

Sep 1, 2016, 3:47:12 PM9/1/16

to securit...@googlegroups.com

This is probably why you are getting the stale pid message(from snortu log):

pfring DAQ configured to passive.
Acquiring network traffic from "eth1".
ERROR: Can't set DAQ BPF filter to '
' ()!
Fatal Error, Quitting...

It doesn't like your bpf.conf. I would double check the syntax of it or clear it out and restart services to see if it helps with that issue. For ELSA you could simply remove all of the buffers from
/nsm/elsa/data/elsa/tmp/buffers/ or try running:

sudo securityonion-elsa-reset

Thanks,
Wes

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onion+unsubscribe@googlegroups.com.
To post to this group, send email to security-onion@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

Devinder kang

unread,

Sep 1, 2016, 4:19:07 PM9/1/16

to security-onion

Wes,

There is nothing in /etc/nsm/server-eth1$ sudo vi bpf.conf
Can you please provide and command to reset bpf.conf

I did reset elsa buffer.

Thanks again.
Devinder

Wes

unread,

Sep 1, 2016, 5:33:04 PM9/1/16

to security-onion

Devinder,

Do you still get the same message at the end of snortu-*.log?

You may want to try renaming/backing up the current file and creating a new file titled bpf.conf.

Also see:
https://github.com/Security-Onion-Solutions/security-onion/wiki/BPF

Thanks,
Wes

Devinder kang

unread,

Sep 6, 2016, 3:07:28 PM9/6/16

to security-onion

Hey Wes,

Sorry about the late reply caz i was off because of long weekend.

I renamed and created new bpf.conf at following location /etc/nsm/server-eth1$/bpf.conf
Reboot computer couple of time.
Now ELSA is working fine, but snortu is still failing with below msg.

snort-1 (alert data) [ FAIL ]
* stale PID file found, process will be restarted at the next 5-minute interval!

I have also changed number of bro workers and snort from 3 to 1 at below location.

Edit --- etc/nsm/$HOSTNAME-$INTERFACE/sensor.conf
Edit --- /opt/bro/etc/node.cfg

This time no error in snortu*-log
I have attached the copy of snortu-log and sostat redacted output.

Please help me to sort this out.

Thanks,
Devinder

snortu-1.log.20160906182532.txt

sostat-redacted-6sept16.txt

Wes

unread,

Sep 6, 2016, 5:05:40 PM9/6/16

to security-onion

Devinder,

Are you consistently receiving the stale PID error message or is it every once in a while?

I have experienced this from time to time, but it should not be doing it upon every restart of the service.

Thanks,
Wes

Devinder kang

unread,

Sep 6, 2016, 5:08:46 PM9/6/16

to security-onion

Yes when i restart my system everything works fine. but when i run sudo service nsm restart after couple of minutes, it failed again.

Thanks,

Devinder

Devinder kang

unread,

Sep 11, 2016, 1:23:38 PM9/11/16

to security-onion

So Wes, am I assume below error message is normal in this situation or is their is a way to clear it up ?

snort-1 (alert data) [ FAIL ]
* stale PID file found, process will be restarted at the next 5-minute interval!

Thanks,
devinder

Wes

unread,

Sep 11, 2016, 8:52:52 PM9/11/16

to security-onion

Devinder,

Are there any clues in your current snortu-1.log? The service will normally restart itself if this error occurs (which shouldn't be often under normal operation), but it should not continually display the same issue.

Thanks,
Wes

Reply all

Reply to author

Forward