ossec-dbd(5203): ERROR: 'MySQL server has gone away'
Kai-Uwe...@t-systems.com
I have installed the ossec-Server v2.4.1 on SLES 11 SP1 with MySQL-Support, also a few agents on some test systems.
Everything is working fine, but from time to time I get error messages from ossec-dbd and the database connection is broken.
The MySQL-Logfiles are clean, the MySQL-Server is in the same machine like the ossec-server.
If I restart the ossec-server with /var/ossec/bin/ossec-control restart then everything works for a few days until the ossec-dbd error reoccurs.
Any ideas?
Output from /var/ossec/logs/ossec.log:
2010/07/17 08:57:11 ossec-dbd(5203): ERROR: Error executing query 'SELECT id FROM location WHERE name = 'sles11-sp1-vm1-k
us->ossec-monitord' AND server_id = '1' LIMIT 1'. Error: 'MySQL server has gone away'.
2010/07/17 08:57:11 ossec-dbd(5209): INFO: Closing connection to database.
2010/07/17 08:57:11 ossec-dbd(5210): INFO: Attempting to reconnect to database.
2010/07/17 08:57:11 ossec-dbd(5202): ERROR: Error connecting to database 'localhost'(ossec): ERROR: Unknown MySQL server
host 'localhost' (3).
2010/07/17 08:57:13 ossec-dbd(5210): INFO: Attempting to reconnect to database.
2010/07/17 08:57:13 ossec-dbd(5202): ERROR: Error connecting to database 'localhost'(ossec): ERROR: Unknown MySQL server
host 'localhost' (3).
2010/07/17 08:57:17 ossec-dbd(5210): INFO: Attempting to reconnect to database.
2010/07/17 08:57:17 ossec-dbd(5202): ERROR: Error connecting to database 'localhost'(ossec): ERROR: Unknown MySQL server
host 'localhost' (3).
2010/07/17 08:57:25 ossec-dbd(5210): INFO: Attempting to reconnect to database.
2010/07/17 08:57:25 ossec-dbd(5202): ERROR: Error connecting to database 'localhost'(ossec): ERROR: Unknown MySQL server
host 'localhost' (3).
2010/07/17 08:57:41 ossec-dbd(5210): INFO: Attempting to reconnect to database.
2010/07/17 08:57:41 ossec-dbd(5202): ERROR: Error connecting to database 'localhost'(ossec): ERROR: Unknown MySQL server
host 'localhost' (3).
2010/07/17 08:58:13 ossec-dbd(5210): INFO: Attempting to reconnect to database.
2010/07/17 08:58:13 ossec-dbd(5202): ERROR: Error connecting to database 'localhost'(ossec): ERROR: Unknown MySQL server
host 'localhost' (3).
2010/07/17 08:59:17 ossec-dbd(5210): INFO: Attempting to reconnect to database.
2010/07/17 08:59:17 ossec-dbd(5202): ERROR: Error connecting to database 'localhost'(ossec): ERROR: Unknown MySQL server
host 'localhost' (3).
2010/07/17 09:01:25 ossec-dbd(5210): INFO: Attempting to reconnect to database.
2010/07/17 09:01:25 ossec-dbd(5202): ERROR: Error connecting to database 'localhost'(ossec): ERROR: Unknown MySQL server
host 'localhost' (3).
2010/07/17 09:05:41 ossec-dbd(5210): INFO: Attempting to reconnect to database.
2010/07/17 09:05:41 ossec-dbd(5202): ERROR: Error connecting to database 'localhost'(ossec): ERROR: Unknown MySQL server
host 'localhost' (3).
2010/07/17 09:14:13 ossec-dbd(5210): INFO: Attempting to reconnect to database.
2010/07/17 09:14:13 ossec-dbd(5202): ERROR: Error connecting to database 'localhost'(ossec): ERROR: Unknown MySQL server
host 'localhost' (3).
2010/07/17 09:31:17 ossec-dbd(5210): INFO: Attempting to reconnect to database.
2010/07/17 09:31:17 ossec-dbd(5202): ERROR: Error connecting to database 'localhost'(ossec): ERROR: Unknown MySQL server
host 'localhost' (3).
2010/07/17 10:05:25 ossec-dbd(5208): ERROR: Multiple database errors. Exiting.
Thanks and regards,
Kai-Uwe
dan (ddp)
I couldn't tell you what's going on (you could try running the
ossec-dbd in debug mode, but it may not help). I'd try using
"127.0.0.1" instead of 'localhost' for the database in ossec.conf.
Kai-Uwe...@t-systems.com
2010/07/20 03:51:28 ossec-dbd(5203): ERROR: Error executing query 'SELECT id FROM location WHERE name = 'sles11-sp1-vm1-kus->ossec-monitord' AND server_id = '1' LIMIT 1'. Error: 'MySQL server has gone away'.
Running var/ossec/bin/ossec-control status shows the following
sles11-sp1-vm1-kus:/var/ossec # /var/ossec/bin/ossec-control status
ossec-monitord is running...
ossec-logcollector is running...
ossec-remoted is running...
ossec-syscheckd is running...
ossec-analysisd is running...
ossec-maild not running...
ossec-execd not running...
ossec-dbd: Process 28624 not used by ossec, removing ..
ossec-dbd not running...
So it seems that the ossec-dbd is dying for whatever reason after some runtime!?
I have added dbd.debug=2 in the internal_options.conf file and also changed the hostname from localhost to 127.0.0.1 in the ossec.conf file and restarted ossec afterwards.
If I get any new results I will share with you.
Regards,
Kai-Uwe
Jason 'XenoPhage' Frisvold
Hash: SHA1
On Jul 19, 2010, at 3:24 AM, <Kai-Uwe...@t-systems.com> <Kai-Uwe...@t-systems.com> wrote:
> Hi all,
>
> I have installed the ossec-Server v2.4.1 on SLES 11 SP1 with MySQL-Support, also a few agents on some test systems.
> Everything is working fine, but from time to time I get error messages from ossec-dbd and the database connection is broken.
> The MySQL-Logfiles are clean, the MySQL-Server is in the same machine like the ossec-server.
> If I restart the ossec-server with /var/ossec/bin/ossec-control restart then everything works for a few days until the ossec-dbd error reoccurs.
>
> Any ideas?
>
> Output from /var/ossec/logs/ossec.log:
>
> 2010/07/17 08:57:11 ossec-dbd(5203): ERROR: Error executing query 'SELECT id FROM location WHERE name = 'sles11-sp1-vm1-k
> us->ossec-monitord' AND server_id = '1' LIMIT 1'. Error: 'MySQL server has gone away'.
> 2010/07/17 08:57:11 ossec-dbd(5209): INFO: Closing connection to database.
Check your MySQL configuration. You may be running into a connection limit problem. I believe the default limit is 100.
If it's not a connection limit issue, it may be something else, but likely MySQL related. Check the .err file in your MySQL directory for more hints at what the problem may be...
> Thanks and regards,
> Kai-Uwe
- ---------------------------
Jason 'XenoPhage' Frisvold
xeno...@godshell.com
- ---------------------------
"Any sufficiently advanced magic is indistinguishable from technology."
- - Niven's Inverse of Clarke's Third Law
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.14 (Darwin)
iEYEARECAAYFAkxFEuwACgkQ8CjzPZyTUTTP5wCfSuBrOrPyju+AKYSxtj3eT0Z+
KCgAnjQ1RHWowxF90CBFABMh0MAmGwki
=CzoZ
-----END PGP SIGNATURE-----
Kai-Uwe...@t-systems.com
i can try using 127.0.0.1 instead localhost, but I doubt that it make any difference (the /etc/hosts contains a line localhost 127.0.0.1)
How to run ossec-dbd in debug mode, I can't find any debug option for ossec-dbd in /var/ossec/etc/internal_options.conf
Maybe it is dbd.debug=2 ???
Looking at the history of this mailing list I found a few other people having this kind of issue with ossec-dbd, but nobody has a solution yet.
Regards,
dan (ddp)
know if there's a setting to enable that.
Kai-Uwe...@t-systems.com
i do not see any error in the MySQL-logs.
Today the problem reoccurs, ossec-dbd was still running but lost connection to the DB.
Error message was the same:
2010/07/22 10:08:59 ossec-dbd(5203): ERROR: Error executing query 'SELECT id FROM location WHERE name = 'sles11
-sp1-vm1-kus->/var/log/messages' AND server_id = '1' LIMIT 1'. Error: 'MySQL server has gone away'.
The MySQL-DB was still up and running, also a restart of the MySQL-DB does not helped.
I was able to look with strace to the already running ossec-dbd:
ossecm 459 1 0 Jul20 ? 00:00:02 /var/ossec/bin/ossec-dbd
root 18432 18224 0 10:11 pts/0 00:00:00 grep dbd
sles11-sp1-vm1-kus:~ # strace -p 459
Process 459 attached - interrupt to quit
select(0, NULL, NULL, NULL, {0, 815112}) = 0 (Timeout)
read(5, "", 4096) = 0
select(0, NULL, NULL, NULL, {5, 0}) = 0 (Timeout)
stat("/etc/localtime", {st_mode=S_IFREG|0555, st_size=2309, ...}) = 0
read(5, "", 4096) = 0
select(0, NULL, NULL, NULL, {5, 0}) = 0 (Timeout)
read(5, "", 4096) = 0
select(0, NULL, NULL, NULL, {5, 0}) = 0 (Timeout)
read(5, "", 4096) = 0
select(0, NULL, NULL, NULL, {5, 0}) = 0 (Timeout)
read(5, "", 4096) = 0
select(0, NULL, NULL, NULL, {5, 0}) = 0 (Timeout)
read(5, "", 4096) = 0
select(0, NULL, NULL, NULL, {5, 0}) = 0 (Timeout)
stat("/etc/localtime", {st_mode=S_IFREG|0555, st_size=2309, ...}) = 0
read(5, "", 4096) = 0
select(0, NULL, NULL, NULL, {5, 0}) = 0 (Timeout)
read(5, "", 4096) = 0
select(0, NULL, NULL, NULL, {5, 0}) = 0 (Timeout)
read(5, "** Alert 1279786370.333: - syslo"..., 4096) = 335
poll([{fd=4, events=POLLIN|POLLPRI}], 1, 0) = 0 (Timeout)
write(4, "i\0\0\0\3SELECT id FROM location WHE"..., 109) = 109
read(4, "\1\0\0\1\1/\0\0\2\3def\5ossec\10location\10loc"..., 16384) = 80
poll([{fd=4, events=POLLIN|POLLPRI}], 1, 0) = 0 (Timeout)
write(4, "\316\0\0\0\3INSERT INTO data(id, server"..., 210) = 210
read(4, "\7\0\0\1\0\1\0\2\0\0\0", 16384) = 11
poll([{fd=4, events=POLLIN|POLLPRI}], 1, 0) = 0 (Timeout)
write(4, "\201\0\0\0\3INSERT INTO alert(id,server"..., 133) = 133
read(4, "\7\0\0\1\0\1\0\2\0\0\0", 16384) = 11
stat("/etc/localtime", {st_mode=S_IFREG|0555, st_size=2309, ...}) = 0
read(5, "", 4096) = 0
select(0, NULL, NULL, NULL, {5, 0}) = 0 (Timeout)
read(5, "", 4096) = 0
select(0, NULL, NULL, NULL, {5, 0}) = 0 (Timeout)
read(5, "", 4096) = 0
select(0, NULL, NULL, NULL, {5, 0}) = 0 (Timeout)
read(5, "", 4096) = 0
select(0, NULL, NULL, NULL, {5, 0}) = 0 (Timeout)
read(5, "", 4096) = 0
select(0, NULL, NULL, NULL, {5, 0}) = 0 (Timeout)
stat("/etc/localtime", {st_mode=S_IFREG|0555, st_size=2309, ...}) = 0
read(5, "", 4096) = 0
select(0, NULL, NULL, NULL, {5, 0}) = 0 (Timeout)
read(5, "", 4096) = 0
select(0, NULL, NULL, NULL, {5, 0}) = 0 (Timeout)
read(5, "", 4096) = 0
select(0, NULL, NULL, NULL, {5, 0}) = 0 (Timeout)
read(5, "", 4096) = 0
select(0, NULL, NULL, NULL, {5, 0}) = 0 (Timeout)
read(5, "", 4096) = 0
select(0, NULL, NULL, NULL, {5, 0}) = 0 (Timeout)
stat("/etc/localtime", {st_mode=S_IFREG|0555, st_size=2309, ...}) = 0
read(5, "", 4096) = 0
select(0, NULL, NULL, NULL, {5, 0}) = 0 (Timeout)
read(5, "", 4096) = 0
select(0, NULL, NULL, NULL, {5, 0}) = 0 (Timeout)
read(5, "", 4096) = 0
select(0, NULL, NULL, NULL, {5, 0}) = 0 (Timeout)
read(5, "", 4096) = 0
select(0, NULL, NULL, NULL, {5, 0}) = 0 (Timeout)
read(5, "", 4096) = 0
select(0, NULL, NULL, NULL, {5, 0}) = 0 (Timeout)
stat("/etc/localtime", {st_mode=S_IFREG|0555, st_size=2309, ...}) = 0
read(5, "", 4096) = 0
select(0, NULL, NULL, NULL, {5, 0}) = 0 (Timeout)
read(5, "", 4096) = 0
Seems to be a timeout problem with ossec-dbd.
After restarting ossec everything is working again.
Any ideas?
Regards,
Kai-Uwe
Kai-Uwe...@t-systems.com
http://www.atomicorp.com/wiki/index.php/ASL_Troubleshooting#OSSEC-dbd_Reports:_Lost_connection_to_MySQL_server_during_query
After setting the timeouts to 24 hours (86400 seconds) for my test environment the problems with the "MySQL server has gone away" messages are disappeared and the ossec server is running fine now.
/etc/my.cnf
wait_timeout = 86400
interactive_timeout = 86400
Regards,
Kai-Uwe
Michael Barrett
I have removed several sensors from the server using manage_agent and they still appear on the web ui
Any suggestions on how to remove these ghosts?
____________________________________________
Michael Barrett | Information Security Analyst - Lead | Mortgage Guaranty Insurance Corporation
270 E. Kilbourn Ave. | Milwaukee, WI 53202 USA | ( 1.414.347.6271 | 7 1.888.601.4440 | * michael...@MGIC.com
“Accomplishing the impossible means only
that your boss will add it to your regular duties” Doug Larson
This message is intended for use only by the person(s) addressed above
and may contain privileged and confidential information. Disclosure or
use of this message by any other person is strictly prohibited. If this
message is received in error, please notify the sender immediately and
delete this message.
dan (ddp)
Check ossec/queue/agent-info for files named after the deleted agents.
Carter, Dennis A
Michael,
You may have to CD to /var/ossec/queue/rids. In the RIDS file you see the numbers that correspond to the agents. Check and see if the numbers for the agents you removed are still listed in the RIDS file if so delete then out. You can also try restarting the HTTPD service on your system so the web UI will get a current update of agents from OSSEC.
Hope this helps
Michael Barrett
There are files in that directory, do I delete the files if the agents no longer exist?
Michael Barrett | Information Security Analyst - Lead | Mortgage Guaranty Insurance Corporation
270 E. Kilbourn Ave. | Milwaukee, WI 53202 USA | ( 1.414.347.6271 | 7 1.888.601.4440 | * michael...@MGIC.com
“Accomplishing the impossible means only
that your boss will add it to your regular duties” Doug Larson
This message is intended for use only by the person(s) addressed above
and may contain privileged and confidential information. Disclosure or
use of this message by any other person is strictly prohibited. If this
message is received in error, please notify the sender immediately and
delete this message.
| From: | "dan (ddp)" <ddp...@gmail.com> |
| To: | ossec...@googlegroups.com |
| Date: | 08/17/2010 08:59 AM |
| Subject: | Re: [ossec-list] sensors showing up in UI after removed |
| Sent by: | ossec...@googlegroups.com |
dan (ddp)
<Michael...@mgic.com> wrote:
>
> There are files in that directory, do I delete the files if the agents no longer exist?
> ____________________________________________
> Michael Barrett | Information Security Analyst - Lead | Mortgage Guaranty Insurance Corporation
> 270 E. Kilbourn Ave. | Milwaukee, WI 53202 USA | ( 1.414.347.6271 | 7 1.888.601.4440 | * michael...@MGIC.com
>
Yes, you should be able to delete the files associated with agents
that no longer exist. It might be safest to shutdown the ossec server,
move the files to a temporary directory and start the ossec server
back up. After verifying that everything seems to be working, you can
then delete the moved files.
Carter, Dennis A
Michael,
If you have files that correspond to the agents you deleted then yes deleted those files.