ELSA error on new node install
900 views
Skip to first unread message
Michael Bower
May 10, 2013, 12:26:34 PM5/10/13
to securit...@googlegroups.com
I just installed a new sensor, setup went well. I get the following error testing out if ELSA can see the new node.
No nodes available at /opt/elsa/web/lib/API.pm line 1938.
I have checked and searchd is running.
Scott Runnels
May 10, 2013, 1:15:46 PM5/10/13
to securit...@googlegroups.com
Hi Michael,
Was this a new sensor added to an existing setup? Can you check the entries under "nodes" in /etc/elsa_web.conf do your ELSA enabled sensors all appear there?
Thanks,
Scott
Scott Runnels
--
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.
Visit this group at http://groups.google.com/group/security-onion?hl=en-US.
For more options, visit https://groups.google.com/groups/opt_out.
Michael Bower
May 14, 2013, 2:31:05 PM5/14/13
to securit...@googlegroups.com
This is a new sensor added to an existing setup. I fixed the initial issue, I was querying for the wrong node number. It lead to another issue I have run into before where I have some elsa nodes drop off after adding a sensor. Looking at the nodes, not all of them appear.
"nodes": {
"127.0.0.1": {
"db": "syslog",
"username": "elsa",
"password": "biglog",
"port": 3306,
"sphinx_port": 9306
},
"elsa_node_001": {
"db": "syslog",
"username": "elsa",
"password": "biglog",
"port": 50002,
"sphinx_port": 50003
},
"elsa_node_002": {
"db": "syslog",
"username": "elsa",
"password": "biglog",
"port": 50004,
"sphinx_port": 50005
},
"elsa_node_003": {
"db": "syslog",
"username": "elsa",
"password": "biglog",
"port": 50006,
"sphinx_port": 50007
},
"elsa_node_004": {
"db": "syslog",
"username": "elsa",
"password": "biglog",
"port": 50010,
"sphinx_port": 50011
},
"elsa_node_005": {
"db": "syslog",
"username": "elsa",
"password": "biglog",
"port": 50012,
"sphinx_port": 50013
},
"elsa_node_006": {
"db": "syslog",
"username": "elsa",
"password": "biglog",
"port": 50014,
"sphinx_port": 50015
}
--
You received this message because you are subscribed to a topic in the Google Groups "security-onion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/security-onion/8_ZwTnY0Law/unsubscribe?hl=en-US.
To unsubscribe from this group and all its topics, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.
Visit this group at http://groups.google.com/group/security-onion?hl=en-US.
For more options, visit https://groups.google.com/groups/opt_out.
Mike
Doug Burks
May 15, 2013, 7:01:50 AM5/15/13
to securit...@googlegroups.com
Hi Michael,
Please connect to each of your sensors and run the following command:
pgrep -lf autossh
You should be able to look at the output and see the port numbers used
in the -R options. Then update your /etc/elsa_web.conf accordingly
and restart apache:
sudo service apache2 restart
Hope that helps!
Thanks,
Doug
Doug Burks
http://securityonion.blogspot.com
Please connect to each of your sensors and run the following command:
pgrep -lf autossh
You should be able to look at the output and see the port numbers used
in the -R options. Then update your /etc/elsa_web.conf accordingly
and restart apache:
sudo service apache2 restart
Hope that helps!
Thanks,
Doug
http://securityonion.blogspot.com
Michael Bower
May 15, 2013, 9:10:58 AM5/15/13
to securit...@googlegroups.com
Im getting the following, I am reindexing to see if that works:
Warnings: node elsa_node_001 got error $VAR1 = undef; , node elsa_node_001 got error $VAR1 = undef; , node elsa_node_001 got error $VAR1 = undef;
Michael Bower
May 15, 2013, 9:26:04 AM5/15/13
to securit...@googlegroups.com
Looks like that failed, I get:
WARNING: failed to scanf pid from pid_file '/var/run/sphinxsearch/searchd.pid'.
WARNING: indices NOT rotated.
--
Mike
Scott Runnels
May 15, 2013, 9:36:54 AM5/15/13
to securit...@googlegroups.com
make sure you go over the commands Doug sent. My guess would be that the port numbers are off.
They should start at 50,000 and increment in groups of two. So if you have 6 nodes (not counting the master server) you should have:
50000 and 50001
50002 and 50003
50004 and 50005
50006 and 50007
50008 and 50009
50010 and 50011
Yours are fairly spread out and if you've rebooted or restarted the securtyonion upstart job it might have configured as the wrong port.
v/r
Scott Runnels
Michael Bower
May 15, 2013, 10:52:19 AM5/15/13
to securit...@googlegroups.com
I ran the commands on the one sensor that Im having an issue with. I got the following on the sensor:
6878 /usr/lib/autossh/autossh -M 0 -q -N -o ServerAliveInterval 60 -o ServerAliveCountMax 3 -i /root/.ssh/securityonion -L 3306:127.0.0.1:3306 -R 50000:localhost:50000 -R 50001:localhost:9306 [redacted]@[redacted]
I changed the elsa_web.conf file to match.
"127.0.0.1": {
"db": "syslog",
"username": "elsa",
"password": "biglog",
"port": 3306,
"sphinx_port": 9306
},
"elsa_node_001": {
"db": "syslog",
"username": "elsa",
"password": "biglog",
"port": 50000,
"sphinx_port": 50001
},
"elsa_node_002": {
"db": "syslog",
"username": "elsa",
"password": "biglog",
"port": 50002,
"sphinx_port": 50003
},
"elsa_node_003": {
"db": "syslog",
"username": "elsa",
"password": "biglog",
"port": 50004,
"sphinx_port": 50005
},
"elsa_node_004": {
"db": "syslog",
"username": "elsa",
"password": "biglog",
"port": 50006,
"sphinx_port": 50007
},
"elsa_node_005": {
"db": "syslog",
"username": "elsa",
"password": "biglog",
"port": 50010,
"sphinx_port": 50011
},
"elsa_node_006": {
"db": "syslog",
"username": "elsa",
"password": "biglog",
"port": 50012,
"sphinx_port": 50013
},
"elsa_node_007": {
"db": "syslog",
"username": "elsa",
"password": "biglog",
"port": 50014,
"sphinx_port": 50015
},
"elsa_node_008": {
"db": "syslog",
"username": "elsa",
"password": "biglog",
"port": 50016,
"sphinx_port": 50017
}
I do have another sensor that is down, which accounts for the 50008 and 50009 missing.
I get the $VAR1 = undef error running the following query:
class=BRO_CONN node=elsa_node_001
Michael Bower
May 16, 2013, 11:16:06 AM5/16/13
to securit...@googlegroups.com
No, not yet.
On Thu, May 16, 2013 at 10:06 AM, Sean Engelbrecht <sean.eng...@gmail.com> wrote:
Michael,
did you manage to get this resolved ?
I am running into the same issue on a new install... with 2 nodes. If I don't specify a node I get the following:
Warnings: node elsa_node_002 got error $VAR1 = undef; , node 127.0.0.1 got error $VAR1 = undef; , node elsa_node_001 got error $VAR1 = undef;
If I duplicate your query I get the same result.
The queries do not return warnings if I run them against the archive data.
Mike
Michael Bower
May 17, 2013, 7:45:17 PM5/17/13
to securit...@googlegroups.com
Ugh, now its showing me I have "9 nodes with undefined logs indexed and undefined archived". The From field is all jacked too, showing "1969-12-30 00:00:00.000".
--
Mike
Michael Bower
May 17, 2013, 10:15:56 PM5/17/13
to securit...@googlegroups.com
At this point, would a reinstall the server then rerunning sosetup at the sensors be the better option? Will that kill the data on the sensor? I ask because I need to do some digging around in ELSA and of course it chooses to not work when I need it.
--
Mike
Doug Burks
May 17, 2013, 11:38:31 PM5/17/13
to securit...@googlegroups.com
On Fri, May 17, 2013 at 10:15 PM, Michael Bower <mbo...@gmail.com> wrote:
> At this point, would a reinstall the server then rerunning sosetup at the
> sensors be the better option? Will that kill the data on the sensor? I ask
> because I need to do some digging around in ELSA and of course it chooses to
> not work when I need it.
Yes, re-running sosetup will delete all NSM data/configuration, so
> At this point, would a reinstall the server then rerunning sosetup at the
> sensors be the better option? Will that kill the data on the sensor? I ask
> because I need to do some digging around in ELSA and of course it chooses to
> not work when I need it.
let's see if we can nurse it back to health without resorting to that.
On the server, please run the following and include the output in your reply:
sudo lsof -nP -i |grep "127.0.0.1:50... (LISTEN)"
You should see a pair of ports for each of the sensors. If you don't,
then log into each of the sensors and restart the autossh tunnel with
the following command:
sudo pkill -USR1 autossh
Then re-run the lsof command above on the server and see if you have a
port pair for each of the sensors.
If you used different ssh accounts for each sensor, you should be able
to use that output to correlate which forwarded ports belong to which
sensor.
Once all port pairs are up on the server, try connecting to them with
netcat to see if a service is listening:
nc localhost 50000
nc localhost 50001
etc...
Even-numbered ports should be mysql and odd-numbered ports should be
sphinx. If any services aren't responding, log into the proper sensor
and (re)start them.
Once all ports are responding with the proper service banners, verify
that /etc/elsa_web.conf has been updated with the proper port pairs
and restart Apache.
If ELSA still isn't working properly, reboot all boxes starting with
the ELSA web server and wait until each box has fully booted before
moving onto the next. (We're trying to ensure that the port pairs are
in a consistent and contiguous state.) Then repeat all the above
steps.
If that still doesn't help, provide more detailed output and we'll
troubleshoot more.
Michael Bower
May 18, 2013, 12:18:23 AM5/18/13
to securit...@googlegroups.com
Autossh is up:
sshd 1317 [redacted] 9u IPv4 9902 0t0 TCP 127.0.0.1:50008 (LISTEN)
sshd 1317 [redacted] 11u IPv4 9906 0t0 TCP 127.0.0.1:50009 (LISTEN)
sshd 1897 [redacted] 9u IPv4 10873 0t0 TCP 127.0.0.1:50006 (LISTEN)
sshd 1897 [redacted] 11u IPv4 10877 0t0 TCP 127.0.0.1:50007 (LISTEN)
sshd 1936 [redacted] 9u IPv4 11536 0t0 TCP 127.0.0.1:50016 (LISTEN)
sshd 1936 [redacted] 11u IPv4 11540 0t0 TCP 127.0.0.1:50017 (LISTEN)
sshd 2121 [redacted] 9u IPv4 11066 0t0 TCP 127.0.0.1:50012 (LISTEN)
sshd 2121 [redacted] 11u IPv4 11070 0t0 TCP 127.0.0.1:50013 (LISTEN)
sshd 2142 [redacted] 9u IPv4 11081 0t0 TCP 127.0.0.1:50010 (LISTEN)
sshd 2142 [redacted] 11u IPv4 11085 0t0 TCP 127.0.0.1:50011 (LISTEN)
sshd 2304 [redacted] 9u IPv4 11158 0t0 TCP 127.0.0.1:50002 (LISTEN)
sshd 2304 [redacted] 11u IPv4 11162 0t0 TCP 127.0.0.1:50003 (LISTEN)
sshd 2322 [redacted] 9u IPv4 11170 0t0 TCP 127.0.0.1:50014 (LISTEN)
sshd 2322 [redacted] 11u IPv4 11174 0t0 TCP 127.0.0.1:50015 (LISTEN)
sshd 23250 [redacted] 9u IPv4 199763 0t0 TCP 127.0.0.1:50000 (LISTEN)
sshd 23250 [redacted] 11u IPv4 199767 0t0 TCP 127.0.0.1:50001 (LISTEN)
Services are responding:
nc localhost 50001
J
2.0.7-id64-dev (rel20-r373?!
nc localhost 50000
[
5.5.31-0ubuntu0.12.04.1HZ/y6B&6|??0v&V^VtdxN\fmysql_native_password^C
Elsa_web.conf has all the proper port pairs (I had to add 50000 and 50001 back in after adding a new sensor, that process had dorked it up):
"nodes": {
"127.0.0.1": {
"db": "syslog",
"username": "elsa",
"password": "biglog",
"port": 3306,
"sphinx_port": 9306
},
"elsa_node_001": {
"db": "syslog",
"username": "elsa",
"password": "biglog",
What additional detail are you needing? Rebooting the server, then following it up with rebooting each sensor individually didn't help.
--
You received this message because you are subscribed to a topic in the Google Groups "security-onion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/security-onion/8_ZwTnY0Law/unsubscribe?hl=en-US.
To unsubscribe from this group and all its topics, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.
Visit this group at http://groups.google.com/group/security-onion?hl=en-US.
For more options, visit https://groups.google.com/groups/opt_out.
Mike
Doug Burks
May 18, 2013, 12:42:55 AM5/18/13
to securit...@googlegroups.com
On Sat, May 18, 2013 at 12:18 AM, Michael Bower <mbo...@gmail.com> wrote:
<snip>
<snip>
> What additional detail are you needing? Rebooting the server, then following
> it up with rebooting each sensor individually didn't help.
What error(s) are you getting in the ELSA web interface now?
Do you get the same error(s) on every single query?
What happens if you query an individual node?
Are there any additional clues in the ELSA logs?
Have you verified that all of the elsa_node_0?? entries are in
/etc/hosts and are properly resolving to 127.0.0.1?
Thanks,
<snip>
> Services are responding:
>
> nc localhost 50001
> J
> 2.0.7-id64-dev (rel20-r373?!
>
> nc localhost 50000
> [
> 5.5.31-0ubuntu0.12.04.1HZ/y6B&6|??0v&V^VtdxN\fmysql_native_password^C
Did you just test those two, or did you test all the ports?
>
> nc localhost 50001
> J
> 2.0.7-id64-dev (rel20-r373?!
>
> nc localhost 50000
> [
> 5.5.31-0ubuntu0.12.04.1HZ/y6B&6|??0v&V^VtdxN\fmysql_native_password^C
<snip>
> What additional detail are you needing? Rebooting the server, then following
> it up with rebooting each sensor individually didn't help.
Do you get the same error(s) on every single query?
What happens if you query an individual node?
Are there any additional clues in the ELSA logs?
Have you verified that all of the elsa_node_0?? entries are in
/etc/hosts and are properly resolving to 127.0.0.1?
Thanks,
Michael Bower
May 18, 2013, 7:21:21 AM5/18/13
to securit...@googlegroups.com
I tested all ports, they all returned the same values
Screenshots attached are what Im seeing. Nothing shows up under "Add Term" and "Report On". The error in the screenshot is what I get running any query, even trying to run it on single nodes.
I found the following in the error.log file:
Use of uninitialized value $rv in concatenation (.) or string at /opt/elsa/web/lib/Utils.pm line 201.
Use of uninitialized value in numeric gt (>) at /opt/elsa/web/lib/Query.pm line 798.
Use of uninitialized value in numeric gt (>) at /opt/elsa/web/lib/Query.pm line 807.
Use of uninitialized value in numeric gt (>) at /opt/elsa/web/lib/Query.pm line 807.
Use of uninitialized value in numeric gt (>) at /opt/elsa/web/lib/Query.pm line 809.
Use of uninitialized value in numeric gt (>) at /opt/elsa/web/lib/Query.pm line 809.
Use of uninitialized value $new_max in concatenation (.) or string at /opt/elsa/web/lib/Query.pm line 813.
It looks like all but the most recent one is listed in /etc/hosts:
127.0.0.1 elsa_node_001 elsa_node_002 elsa_node_003 elsa_node_004 elsa_node_005 elsa_node_006 elsa_node_007
--
You received this message because you are subscribed to a topic in the Google Groups "security-onion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/security-onion/8_ZwTnY0Law/unsubscribe?hl=en-US.
To unsubscribe from this group and all its topics, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.
Visit this group at http://groups.google.com/group/security-onion?hl=en-US.
For more options, visit https://groups.google.com/groups/opt_out.
Mike
Doug Burks
May 18, 2013, 11:35:28 AM5/18/13
to securit...@googlegroups.com
OK, that's a different error than you had previously reported, so that's interesting.
Have you tried adding elsa_node_008 to /etc/hosts?
Doug
--
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.
Visit this group at http://groups.google.com/group/security-onion?hl=en-US.
For more options, visit https://groups.google.com/groups/opt_out.
Doug Burks
May 20, 2013, 6:22:08 AM5/20/13
to securit...@googlegroups.com
Perhaps you could try commenting out all elsa_nodes but localhost from /etc/elsa_web.conf (and restart apache) to see if you get the same behavior with just one node:
"nodes": {
"127.0.0.1": {
"db": "syslog",
"username": "elsa",
"password": "biglog",
"port": 3306,
"sphinx_port": 9306
}
Then test each of the elsa_nodes one at a time to see if you can pinpoint which node it's coming from.
Martin, have you ever seen that error message that Mike is receiving? Any ideas?
Thanks,
Doug
Martin, have you ever seen that error message that Mike is receiving? Any ideas?
Thanks,
Doug
On Sun, May 19, 2013 at 9:14 AM, Michael Bower <mbo...@gmail.com> wrote:
>
> Yes, I did. Then restarted apache, same result.
>
>
> On Sat, May 18, 2013 at 11:35 AM, Doug Burks <doug....@gmail.com> wrote:
>>
>> OK, that's a different error than you had previously reported, so that's interesting.
>>
>> Have you tried adding elsa_node_008 to /etc/hosts?
>>
>> Doug
>>
>>
>> On Sat, May 18, 2013 at 7:21 AM, Michael Bower <mbo...@gmail.com> wrote:
>>>
>>> I tested all ports, they all returned the same values
>>>
>>> Screenshots attached are what Im seeing. Nothing shows up under "Add Term" and "Report On". The error in the screenshot is what I get running any query, even trying to run it on single nodes.
>>>
>>> I found the following in the error.log file:
>>>
>>> Use of uninitialized value $rv in concatenation (.) or string at /opt/elsa/web/lib/Utils.pm line 201.
>>> Use of uninitialized value in numeric gt (>) at /opt/elsa/web/lib/Query.pm line 798.
>>> Use of uninitialized value in numeric gt (>) at /opt/elsa/web/lib/Query.pm line 807.
>>> Use of uninitialized value in numeric gt (>) at /opt/elsa/web/lib/Query.pm line 807.
>>> Use of uninitialized value in numeric gt (>) at /opt/elsa/web/lib/Query.pm line 809.
>>> Use of uninitialized value in numeric gt (>) at /opt/elsa/web/lib/Query.pm line 809.
>>> Use of uninitialized value $new_max in concatenation (.) or string at /opt/elsa/web/lib/Query.pm line 813.
>>>
>>> It looks like all but the most recent one is listed in /etc/hosts:
>>>
>>> 127.0.0.1 elsa_node_001 elsa_node_002 elsa_node_003 elsa_node_004 elsa_node_005 elsa_node_006 elsa_node_007
>>>
>>>
>>>
>>>
>>>
>>>
Martin Holste
May 20, 2013, 6:14:46 PM5/20/13
to security-onion
On my to-do list is to clean up horrible error messages like that one. The problem is that the actual error is the error in the log just before that message. I have made some headway in the latest ELSA version to that end, but for this, you'd need to look at the /nsm/data/elsa/log/web.log file to see what the actual error was. Reproduce the error, go to the end of the web.log file with "less" and use ? to search backwards for \* ERROR to see each individual error. The error just before the one that made it to the web console will be the one you're looking for. Apologies again for the mess with error reporting.
Michael Bower
May 29, 2013, 11:58:48 AM5/29/13
to securit...@googlegroups.com
Sorry its taken so long for me to get back to this, vacation on the beach. I tried just a simple query for site: www.google.com and the following showed up at the end of the web.log file. I didn't have the file in the location you specified, I actually found it here: /nsm/elsa/data/elsa/log/web.log
* DEBUG [2013/05/29 15:48:46] /opt/elsa/web/lib/Query.pm (763) Query::_parse_query 22170 [undef]
query_term_count: 0, num_added_terms: 0
* ERROR [2013/05/29 15:48:46] /opt/elsa/web/lib/Web/Query.pm (52) Web::Query::call 22170 [undef]
All query terms were stripped based on permissions or they were too common at /opt/elsa/web/lib/Query.pm line 766.
* ERROR [2013/05/29 15:49:03] /opt/elsa/web/lib/AsyncDB.pm (99) AsyncDB::__ANON__ 22277 [undef]
Can't connect to MySQL server on 'elsa_node_003' (111) at /usr/share/perl5/AnyEvent/DBI.pm line 87.
at /opt/elsa/web/lib/AsyncDB.pm 107
* ERROR [2013/05/29 15:49:03] /opt/elsa/web/lib/Utils.pm (201) Utils::__ANON__ 22277 [undef]
No indexes for node elsa_node_003, rv:
* ERROR [2013/05/29 15:49:03] /opt/elsa/web/lib/AsyncDB.pm (99) AsyncDB::__ANON__ 22277 [undef]
no database connection at /opt/elsa/web/lib/AsyncDB.pm 149
* ERROR [2013/05/29 15:49:03] /opt/elsa/web/lib/AsyncDB.pm (154) AsyncDB::multi_query 22277 [undef]
Got error no database connection at /opt/elsa/web/lib/AsyncDB.pm line 101.
* ERROR [2013/05/29 15:49:04] /opt/elsa/web/lib/Utils.pm (201) Utils::__ANON__ 22277 [undef]
No indexes for node elsa_node_003, rv:
* ERROR [2013/05/29 15:49:04] /opt/elsa/web/lib/AsyncDB.pm (99) AsyncDB::__ANON__ 22277 [undef]
no database connection at /opt/elsa/web/lib/AsyncDB.pm 149
* ERROR [2013/05/29 15:49:04] /opt/elsa/web/lib/AsyncDB.pm (154) AsyncDB::multi_query 22277 [undef]
Got error no database connection at /opt/elsa/web/lib/AsyncDB.pm line 101.
* ERROR [2013/05/29 15:49:05] /opt/elsa/web/lib/Utils.pm (201) Utils::__ANON__ 22277 [undef]
No indexes for node elsa_node_003, rv:
* ERROR [2013/05/29 15:49:05] /opt/elsa/web/lib/AsyncDB.pm (99) AsyncDB::__ANON__ 22277 [undef]
no database connection at /opt/elsa/web/lib/AsyncDB.pm 149
* ERROR [2013/05/29 15:49:05] /opt/elsa/web/lib/AsyncDB.pm (154) AsyncDB::multi_query 22277 [undef]
Got error no database connection at /opt/elsa/web/lib/AsyncDB.pm line 101.
* ERROR [2013/05/29 15:49:06] /opt/elsa/web/lib/Utils.pm (227) Utils::__ANON__ 22277 [undef]
No tables for node elsa_node_003
* ERROR [2013/05/29 15:49:06] /opt/elsa/web/lib/AsyncDB.pm (99) AsyncDB::__ANON__ 22277 [undef]
no database connection at /opt/elsa/web/lib/AsyncDB.pm 149
* ERROR [2013/05/29 15:49:06] /opt/elsa/web/lib/AsyncDB.pm (154) AsyncDB::multi_query 22277 [undef]
Got error no database connection at /opt/elsa/web/lib/AsyncDB.pm line 101.
* ERROR [2013/05/29 15:49:07] /opt/elsa/web/lib/Utils.pm (227) Utils::__ANON__ 22277 [undef]
No tables for node elsa_node_003
* ERROR [2013/05/29 15:49:07] /opt/elsa/web/lib/AsyncDB.pm (99) AsyncDB::__ANON__ 22277 [undef]
no database connection at /opt/elsa/web/lib/AsyncDB.pm 149
* ERROR [2013/05/29 15:49:07] /opt/elsa/web/lib/AsyncDB.pm (154) AsyncDB::multi_query 22277 [undef]
Got error no database connection at /opt/elsa/web/lib/AsyncDB.pm line 101.
* ERROR [2013/05/29 15:49:08] /opt/elsa/web/lib/Utils.pm (227) Utils::__ANON__ 22277 [undef]
No tables for node elsa_node_003
* ERROR [2013/05/29 15:49:08] /opt/elsa/web/lib/AsyncDB.pm (99) AsyncDB::__ANON__ 22277 [undef]
no database connection at /opt/elsa/web/lib/AsyncDB.pm 149
* ERROR [2013/05/29 15:49:08] /opt/elsa/web/lib/AsyncDB.pm (154) AsyncDB::multi_query 22277 [undef]
Got error no database connection at /opt/elsa/web/lib/AsyncDB.pm line 101.
* ERROR [2013/05/29 15:49:09] /opt/elsa/web/lib/Utils.pm (247) Utils::__ANON__ 22277 [undef]
No classes for node elsa_node_003
* ERROR [2013/05/29 15:49:09] /opt/elsa/web/lib/AsyncDB.pm (99) AsyncDB::__ANON__ 22277 [undef]
no database connection at /opt/elsa/web/lib/AsyncDB.pm 149
* ERROR [2013/05/29 15:49:09] /opt/elsa/web/lib/AsyncDB.pm (154) AsyncDB::multi_query 22277 [undef]
Got error no database connection at /opt/elsa/web/lib/AsyncDB.pm line 101.
* ERROR [2013/05/29 15:49:10] /opt/elsa/web/lib/Utils.pm (247) Utils::__ANON__ 22277 [undef]
No classes for node elsa_node_003
* ERROR [2013/05/29 15:49:10] /opt/elsa/web/lib/AsyncDB.pm (99) AsyncDB::__ANON__ 22277 [undef]
no database connection at /opt/elsa/web/lib/AsyncDB.pm 149
* ERROR [2013/05/29 15:49:10] /opt/elsa/web/lib/AsyncDB.pm (154) AsyncDB::multi_query 22277 [undef]
Got error no database connection at /opt/elsa/web/lib/AsyncDB.pm line 101.
* ERROR [2013/05/29 15:49:11] /opt/elsa/web/lib/Utils.pm (247) Utils::__ANON__ 22277 [undef]
No classes for node elsa_node_003
* ERROR [2013/05/29 15:49:11] /opt/elsa/web/lib/AsyncDB.pm (99) AsyncDB::__ANON__ 22277 [undef]
no database connection at /opt/elsa/web/lib/AsyncDB.pm 149
* ERROR [2013/05/29 15:49:11] /opt/elsa/web/lib/AsyncDB.pm (154) AsyncDB::multi_query 22277 [undef]
Got error no database connection at /opt/elsa/web/lib/AsyncDB.pm line 101.
* ERROR [2013/05/29 15:49:12] /opt/elsa/web/lib/Utils.pm (282) Utils::__ANON__ 22277 [undef]
No fields for node elsa_node_003
* ERROR [2013/05/29 15:49:12] /opt/elsa/web/lib/AsyncDB.pm (99) AsyncDB::__ANON__ 22277 [undef]
no database connection at /opt/elsa/web/lib/AsyncDB.pm 149
* ERROR [2013/05/29 15:49:12] /opt/elsa/web/lib/AsyncDB.pm (154) AsyncDB::multi_query 22277 [undef]
Got error no database connection at /opt/elsa/web/lib/AsyncDB.pm line 101.
* ERROR [2013/05/29 15:49:13] /opt/elsa/web/lib/Utils.pm (282) Utils::__ANON__ 22277 [undef]
No fields for node elsa_node_003
* ERROR [2013/05/29 15:49:13] /opt/elsa/web/lib/AsyncDB.pm (99) AsyncDB::__ANON__ 22277 [undef]
no database connection at /opt/elsa/web/lib/AsyncDB.pm 149
* ERROR [2013/05/29 15:49:13] /opt/elsa/web/lib/AsyncDB.pm (154) AsyncDB::multi_query 22277 [undef]
Got error no database connection at /opt/elsa/web/lib/AsyncDB.pm line 101.
* ERROR [2013/05/29 15:49:14] /opt/elsa/web/lib/Utils.pm (282) Utils::__ANON__ 22277 [undef]
No fields for node elsa_node_003
* ERROR [2013/05/29 15:49:14] /opt/elsa/web/lib/AsyncDB.pm (99) AsyncDB::__ANON__ 22277 [undef]
no database connection at /opt/elsa/web/lib/AsyncDB.pm 149
* ERROR [2013/05/29 15:49:14] /opt/elsa/web/lib/AsyncDB.pm (154) AsyncDB::multi_query 22277 [undef]
Got error no database connection at /opt/elsa/web/lib/AsyncDB.pm line 101.
Doug Burks
May 30, 2013, 8:14:38 PM5/30/13
to securit...@googlegroups.com
On Wed, May 29, 2013 at 11:58 AM, Michael Bower <mbo...@gmail.com> wrote:
> Sorry its taken so long for me to get back to this, vacation on the beach.
> I tried just a simple query for site: www.google.com and the following
> showed up at the end of the web.log file. I didn't have the file in the
> location you specified, I actually found it here:
> /nsm/elsa/data/elsa/log/web.log
>
>
> * DEBUG [2013/05/29 15:48:46] /opt/elsa/web/lib/Query.pm (763)
> Query::_parse_query 22170 [undef]
> query_term_count: 0, num_added_terms: 0
> * ERROR [2013/05/29 15:48:46] /opt/elsa/web/lib/Web/Query.pm (52)
> Web::Query::call 22170 [undef]
> All query terms were stripped based on permissions or they were too common
> at /opt/elsa/web/lib/Query.pm line 766.
> * ERROR [2013/05/29 15:49:03] /opt/elsa/web/lib/AsyncDB.pm (99)
> AsyncDB::__ANON__ 22277 [undef]
> Can't connect to MySQL server on 'elsa_node_003' (111) at
> /usr/share/perl5/AnyEvent/DBI.pm line 87.
> at /opt/elsa/web/lib/AsyncDB.pm 107
> * ERROR [2013/05/29 15:49:03] /opt/elsa/web/lib/Utils.pm (201)
> Utils::__ANON__ 22277 [undef]
> No indexes for node elsa_node_003, rv:
1. On your master server, ensure elsa_node_003 is in /etc/hosts.
> Sorry its taken so long for me to get back to this, vacation on the beach.
> I tried just a simple query for site: www.google.com and the following
> showed up at the end of the web.log file. I didn't have the file in the
> location you specified, I actually found it here:
> /nsm/elsa/data/elsa/log/web.log
>
>
> * DEBUG [2013/05/29 15:48:46] /opt/elsa/web/lib/Query.pm (763)
> Query::_parse_query 22170 [undef]
> query_term_count: 0, num_added_terms: 0
> * ERROR [2013/05/29 15:48:46] /opt/elsa/web/lib/Web/Query.pm (52)
> Web::Query::call 22170 [undef]
> All query terms were stripped based on permissions or they were too common
> at /opt/elsa/web/lib/Query.pm line 766.
> * ERROR [2013/05/29 15:49:03] /opt/elsa/web/lib/AsyncDB.pm (99)
> AsyncDB::__ANON__ 22277 [undef]
> Can't connect to MySQL server on 'elsa_node_003' (111) at
> /usr/share/perl5/AnyEvent/DBI.pm line 87.
> at /opt/elsa/web/lib/AsyncDB.pm 107
> * ERROR [2013/05/29 15:49:03] /opt/elsa/web/lib/Utils.pm (201)
> Utils::__ANON__ 22277 [undef]
> No indexes for node elsa_node_003, rv:
2. elsa_node_003 should be using ports 50004 and 50005, so what is
the output of the following?
nc elsa_node_003 50004
nc elsa_node_003 50005
3. If you don't get output for either of these, then you'll need to
log into elsa_node_003 and (re)start MySQL and/or sphinx.
Michael Bower
May 31, 2013, 4:57:04 PM5/31/13
to securit...@googlegroups.com
I am getting this error in the web.log file.
* ERROR [2013/05/31 20:25:46] /opt/elsa/web/lib/Utils.pm (101) Utils::_dbh_error_handler 1653 [undef]
DBD::mysql::st execute failed: Lost connection to MySQL server during query QUERY: UPDATE query_log SET num_results=?, milliseconds=? WHERE qid=?
--
You received this message because you are subscribed to a topic in the Google Groups "security-onion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/security-onion/8_ZwTnY0Law/unsubscribe?hl=en-US.
To unsubscribe from this group and all its topics, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.
Visit this group at http://groups.google.com/group/security-onion?hl=en-US.
For more options, visit https://groups.google.com/groups/opt_out.
Mike
Michael Bower
May 31, 2013, 4:59:24 PM5/31/13
to securit...@googlegroups.com
I should say that Im getting that error after rebuilding the server and readding the sensors.
--
Mike
Michael Bower
Jun 2, 2013, 10:07:13 AM6/2/13
to securit...@googlegroups.com
Some additional info, it seems when I specify a node in the query, it works. If I don't, I get the following in the web.log.
* ERROR [2013/06/02 13:57:16] /opt/elsa/web/lib/Utils.pm (101) Utils::_dbh_error_handler 8714 [undef]
DBD::mysql::st execute failed: Lost connection to MySQL server during query QUERY: UPDATE query_log SET num_results=?, milliseconds=? WHERE qid=?
* ERROR [2013/06/02 13:57:16] /opt/elsa/web/lib/Web/Query.pm (52) Web::Query::call 8714 [undef]
DBD::mysql::st execute failed: Lost connection to MySQL server during query QUERY: UPDATE query_log SET num_results=?, milliseconds=? WHERE qid=? at /opt/elsa/web/lib/Utils.pm line 106
Utils::_dbh_error_handler('DBD::mysql::st execute failed: Lost connection to MySQL serve...', 'DBI::st=HASH(0x7f3a6bbdd2a8)', undef) called at /opt/elsa/web/lib/Query.pm line 281
Query::_set_time_taken('Query=HASH(0x7f3a6b9f85c0)', 567) called at accessor Query::time_taken (defined at /opt/elsa/web/lib/Query.pm line 70) line 10
Query::time_taken('Query=HASH(0x7f3a6b9f85c0)', 567) called at /opt/elsa/web/lib/API.pm line 1742
API::query('API=HASH(0x7f3a697dc008)', 'HASH(0x7f3a68fc3a00)') called at /opt/elsa/web/lib/Web/Query.pm line 45
eval {...} called at /opt/elsa/web/lib/Web/Query.pm line 43
Web::Query::call('Web::Query=HASH(0x7f3a6b93d278)', 'HASH(0x7f3a69df6e78)') called at /usr/share/perl5/Plack/Component.pm line 39
Plack::Component::__ANON__('HASH(0x7f3a69df6e78)') called at /usr/share/perl5/Plack/App/URLMap.pm line 71
Plack::App::URLMap::call('Plack::App::URLMap=HASH(0x7f3a69df7028)', 'HASH(0x7f3a69df6e78)') called at /usr/share/perl5/Plack/Component.pm line 39
Plack::Component::__ANON__('HASH(0x7f3a69df6e78)') called at /usr/share/perl5/Plack/Middleware/Auth/Basic.pm line 30
Plack::Middleware::Auth::Basic::call('Plack::Middleware::Auth::Basic=HASH(0x7f3a6b942620)', 'HASH(0x7f3a69df6e78)') called at /usr/share/perl5/Plack/Component.pm line 39
Plack::Component::__ANON__('HASH(0x7f3a69df6e78)') called at /usr/share/perl5/Plack/Middleware/Conditional.pm line 19
Plack::Middleware::Conditional::call('Plack::Middleware::Conditional=HASH(0x7f3a6b93fe80)', 'HASH(0x7f3a69df6e78)') called at /usr/share/perl5/Plack/Component.pm line 39
Plack::Component::__ANON__('HASH(0x7f3a69df6e78)') called at /usr/share/perl5/Plack/Middleware/Session.pm line 52
Plack::Middleware::Session::call('Plack::Middleware::Session=HASH(0x7f3a6b942860)', 'HASH(0x7f3a69df6e78)') called at /usr/share/perl5/Plack/Component.pm line 39
Plack::Component::__ANON__('HASH(0x7f3a69df6e78)') called at /usr/share/perl5/Plack/Middleware/CrossOrigin.pm line 115
Plack::Middleware::CrossOrigin::call('Plack::Middleware::CrossOrigin=HASH(0x7f3a6b97f220)', 'HASH(0x7f3a69df6e78)') called at /usr/share/perl5/Plack/Component.pm line 39
Plack::Component::__ANON__('HASH(0x7f3a69df6e78)') called at /usr/share/perl5/Plack/Middleware/Static.pm line 18
Plack::Middleware::Static::call('Plack::Middleware::Static=HASH(0x7f3a6b97f550)', 'HASH(0x7f3a69df6e78)') called at /usr/share/perl5/Plack/Component.pm line 39
Plack::Component::__ANON__('HASH(0x7f3a69df6e78)') called at /usr/share/perl5/Plack/Middleware/NoMultipleSlashes.pm line 14
Plack::Middleware::NoMultipleSlashes::call('Plack::Middleware::NoMultipleSlashes=HASH(0x7f3a6b97f658)', 'HASH(0x7f3a69df6e78)') called at /usr/share/perl5/Plack/Component.pm line 39
Plack::Component::__ANON__('HASH(0x7f3a69df6e78)') called at /usr/share/perl5/Plack/Handler/Apache2.pm line 66
Plack::Handler::Apache2::call_app('Plack::Handler::Apache2', 'Apache2::RequestRec=SCALAR(0x7f3a660f0988)', 'CODE(0x7f3a6b9511d0)') called at /usr/share/perl5/Plack/Handler/Apache2.pm line 91
Plack::Handler::Apache2::handler('Apache2::RequestRec=SCALAR(0x7f3a660f0988)') called at -e line 0
eval {...} called at -e line 0
--
Mike
Doug Burks
Jun 3, 2013, 8:12:45 PM6/3/13
to securit...@googlegroups.com
Have you tried every node individually? Does it always succeed when
you just query one node, or is there one node in particular that's
failing?
Doug
> You received this message because you are subscribed to the Google Groups
> "security-onion" group.
> To unsubscribe from this group and stop receiving emails from it, send an
http://securityonion.blogspot.com
you just query one node, or is there one node in particular that's
failing?
Doug
> "security-onion" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to security-onio...@googlegroups.com.
> To post to this group, send email to securit...@googlegroups.com.
> Visit this group at http://groups.google.com/group/security-onion?hl=en-US.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
--
Doug Burks
> To post to this group, send email to securit...@googlegroups.com.
> Visit this group at http://groups.google.com/group/security-onion?hl=en-US.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
--
http://securityonion.blogspot.com
Michael Bower
Jun 3, 2013, 8:24:24 PM6/3/13
to securit...@googlegroups.com
It completes on all nodes individually.
Doug Burks
Jun 4, 2013, 6:43:16 AM6/4/13
to securit...@googlegroups.com
As I suggested to Brian in the other thread:
It's possible that you have more securityonion_db tables than your
open-files-limit and so, even though Sguil isn't producing an "out of
resources" error, it could be spending a lot of MySQL time just
opening tables and therefore creating a timeout condition when ELSA
tries to query MySQL.
How many tables are in the securityonion_db database on your master server?
What is your MySQL open-files limit set to?
Thanks,
Doug
It's possible that you have more securityonion_db tables than your
open-files-limit and so, even though Sguil isn't producing an "out of
resources" error, it could be spending a lot of MySQL time just
opening tables and therefore creating a timeout condition when ELSA
tries to query MySQL.
How many tables are in the securityonion_db database on your master server?
What is your MySQL open-files limit set to?
Thanks,
Doug
Michael Bower
Jun 4, 2013, 8:44:46 AM6/4/13
to securit...@googlegroups.com
My open_files_limit is below:
open-files-limit = 4096
DB tables and the query I used:
mysql> SELECT IFNULL(table_schema,'Total') "Database",TableCount
-> FROM (SELECT COUNT(1) TableCount,table_schema
-> FROM information_schema.tables
-> WHERE table_schema NOT IN ('information_schema','mysql')
-> GROUP BY table_schema WITH ROLLUP) A;
+--------------------+------------+
| Database | TableCount |
+--------------------+------------+
| elsa_web | 18 |
| performance_schema | 17 |
| securityonion_db | 421 |
| snorby | 31 |
| syslog | 19 |
| syslog_data | 2 |
| Total | 508 |
+--------------------+------------+
Doug Burks
Jun 4, 2013, 9:47:34 AM6/4/13
to securit...@googlegroups.com
Please watch your MySQL processes while performing an ELSA query and
see if you see anything that could be slowing MySQL down:
watch "mysql -uroot -e 'show full processlist' |grep -v 'Sleep'"
Also, are any of your sensors on the other side of slow network links
that might trigger the timeout when all nodes are queried at once?
Thanks,
Doug
see if you see anything that could be slowing MySQL down:
watch "mysql -uroot -e 'show full processlist' |grep -v 'Sleep'"
Also, are any of your sensors on the other side of slow network links
that might trigger the timeout when all nodes are queried at once?
Thanks,
Doug
Michael Bower
Jun 4, 2013, 10:10:01 AM6/4/13
to securit...@googlegroups.com
I see the following, nothing else pops up when I run the query.
Every 2.0s: mysql -uroot -e 'show full processlist' |grep -v 'Sleep' Tue Jun 4 14:02:16 2013
Id User Host db Command Time State Info
27085 sguil localhost securityonion_db Query 0 query end INSERT INTO history (sid, cid, uid, timestamp, status, comment) VALUES ( 17, 245429, 1, '2
013-06-04 14:02:16', 1, 'Auto Update')
32463 root localhost NULL Query 0 NULL show full processlist
All are at least on a T1 MPLS line. Is there a way via the elsa_node.conf to change the timeout?
Doug Burks
Jun 4, 2013, 10:20:01 AM6/4/13
to securit...@googlegroups.com
Are you running the http_agent in its default configuration (send
*all* Bro HTTP logs to the Sguil database)? Perhaps that's keeping
Sguil/MySQL busy. If you're running ELSA, you really don't need the
HTTP logs in the Sguil database, so try disabling http_agent on all
your sensors:
# Terminate the running http_agent
sudo nsm_sensor_ps-stop --only-http-agent
# Disable http_agent
sudo chmod 0 /usr/bin/http_agent.tcl
Thanks,
Doug
*all* Bro HTTP logs to the Sguil database)? Perhaps that's keeping
Sguil/MySQL busy. If you're running ELSA, you really don't need the
HTTP logs in the Sguil database, so try disabling http_agent on all
your sensors:
# Terminate the running http_agent
sudo nsm_sensor_ps-stop --only-http-agent
# Disable http_agent
sudo chmod 0 /usr/bin/http_agent.tcl
Thanks,
Doug
Michael Bower
Jun 4, 2013, 11:56:18 AM6/4/13
to securit...@googlegroups.com
Ugh, thought we had a winner.
Same thing. This is now what is showing with the watch command:
Every 2.0s: mysql -uroot -e 'show full processlist' |grep -v 'Sleep' Tue Jun 4 15:55:22 2013
Id User Host db Command Time State Info
491 root localhost NULL Query 0 NULL show full processlist
Reply all
Reply to author
Forward
0 new messages