Unexplained crash with PXDBC 5.5.28 ?

42 views
Skip to first unread message

Laurent MINOST

unread,
Jan 31, 2013, 7:17:33 AM1/31/13
to codersh...@googlegroups.com
Hi,

We had an unexplained crash today with one PXDBC node 5.5.28, the mysqld process crashed and then the server cannot be restarted without crashing and crashing after each restart, I don't have any idea what is the cause of this crash since we restarted the server this morning (around 9.30am) and everything was fine, the crash occured at 11.02am as you can see in the full error-log provided in attachement.
After this crash, we tried to restart the server 3 or 4 times but it crashed after a few seconds online, finally we tried to disabled wsrep_provider and set it to 'none', then the server finally stayed online and checked for crashed table, so I don't really know if wsrep_provider change was the solution to make it restarting without crash or if it was a coincidence ? For the moment, we ran a full mysqlcheck on all databases to see if everything is OK and it's still runing, so the server is still running with wsrep_provider = none for the moment.
 
Furthermore, it is needed to indicate that this server is :
1. not a production one but development environnement hosting approximatly 130 databases for a dataset of ~50GB.
2. a single node only (wsrep_provider enabled but gcomm://) and there is no other nodes, we use it only to have the same environnment as our future production environnment.

Is there a way to get some more informations on the cause if possible by analyzing the informations provided by mysqld when it crashed ? If yes how to do this please ?

Thanks for your help and informations.

Regards,

Laurent

Laurent MINOST

unread,
Jan 31, 2013, 7:20:37 AM1/31/13
to codersh...@googlegroups.com
Attachement added as Google Groups returned an error 340 if I tried to attach it on the first post ...
mysql-error.zip

Alex Yurchenko

unread,
Jan 31, 2013, 11:18:44 AM1/31/13
to codersh...@googlegroups.com
Hi Laurent,

https://bugs.launchpad.net/codership-mysql/+bug/1078346

Percona has just made a new PXC release, you should upgrade to it.

Regards,
Alex
--
Alexey Yurchenko,
Codership Oy, www.codership.com
Skype: alexey.yurchenko, Phone: +358-400-516-011

Laurent MINOST

unread,
Feb 1, 2013, 5:00:42 AM2/1/13
to codersh...@googlegroups.com
Hi Alex,

Thanks a lot for your reply and informations.
I've done the upgrade so we hope to not more have this problem again but ... it seems there is some problems with this Percona upgrade 5.5.29 as I had some weird errors during apt upgrade on debian :

130131 21:28:07 Percona XtraDB (http://www.percona.com) 1.1.8-rel29.1 started; log sequence number 68570443738
ERROR: 1064  You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'ALTER TABLE user ADD column Show_v$
130131 21:28:07 [ERROR] Aborting
...

130131 21:28:25 Percona XtraDB (http://www.percona.com) 1.1.8-rel29.1 started; log sequence number 68570443738
ERROR: 1050  Table 'plugin' already exists
130131 21:28:25 [ERROR] Aborting
...

It did not prevent apt to finish the package installation but the --configure phase of dpkg does not seem completed and so apt is now always complaining about this and trying to stop/start the server to complete the upgrade ... will report this on Percona groups, this is problematic but did not prevent us to restart the server

Moreover, we had a more important problem this morning a few minutes ago, the server got its load average increasing and increasing and was unavailable for communications for a moment, on the mysql error log , we are having some very strange messages that can be seen on the log attached at the end of the file.

Do you have an idea of what can have caused this please ?

Regards,

Laurent
mysql-error_2.zip

Alex Yurchenko

unread,
Feb 1, 2013, 5:21:37 AM2/1/13
to codersh...@googlegroups.com
We'll have a look, but at first glance this it unrelated to wsrep
patch. However it may be related to Percona's.

Burkhard Ott

unread,
Feb 1, 2013, 5:15:01 AM2/1/13
to Laurent MINOST, codersh...@googlegroups.com
Sorry if I interfere, I had the same stack trace after the update
today.

I a few tests it turned out it only happens with the galera plaugin, so
the same MySQL DB version with row based replication (mysql
replication) doesn't show that problem.

I resolved it by removing all databases on one node, resynced and
the error disappeared.

Maybe it helps.

cheers

Henrik Ingo

unread,
Feb 3, 2013, 12:16:04 PM2/3/13
to Laurent MINOST, codersh...@googlegroups.com
On Fri, Feb 1, 2013 at 12:00 PM, Laurent MINOST <lol...@gmail.com> wrote:
> ERROR: 1050 Table 'plugin' already exists

I don't know why specifically this would happen in PXC 5.5.29 but this
error message comes from the mysql_upgrade script. You could file it
as a bug on percona-xtradb-cluster in launchpad.

Still, if you just force a new SST to happen you will get a new copy
of the database from the other nodes, so whatever reason caused the
above error will not matter anymore.

henrik

--
henri...@avoinelama.fi
+358-40-8211286 skype: henrik.ingo irc: hingo
www.openlife.cc

My LinkedIn profile: http://www.linkedin.com/profile/view?id=9522559

Laurent MINOST

unread,
Feb 4, 2013, 4:52:00 AM2/4/13
to codersh...@googlegroups.com, Laurent MINOST, henri...@avoinelama.fi
Hi Henrik,

Concerning the errors about table plugin already existing and alter which is truncated, I think the problem is OS related due to the fact these machines were upgraded recently to Debian Wheezy and not using Squeeze anymore, tried on another Debian Wheezy server and the upgrade produced the same error. I do not have anymore Debian Squeeze server to upgrade to check/confirm that the error is not present.

The bug entry has been filled by Percona member on their launchpad : https://bugs.launchpad.net/percona-xtradb-cluster/+bug/1112363

Concerning the tip/trick to force a new SST, problem in my case is that this server is a standalone node not member of a cluster only there to have the same MySQL base as our future production cluster so I do not have any more members to sync from for an SST :p Hope it will be OK and nothing important has been lost during the upgrade process ...

Regards,

Laurent

Henrik Ingo

unread,
Feb 4, 2013, 6:47:37 AM2/4/13
to Laurent MINOST, codersh...@googlegroups.com
Aha, ok.

Your database is fine, this is related to the system tables in
mysql.*. In fact it seems it is trying to run mysql_upgrade on a
database that already has been upgraded before.

Once I have fixed a similar error simply be deleting (or renaming,
safer) mysql.plugin table - then the upgrade script creates it back
and is happy. This is a bit adventurous of course :-)

henrik
> --
> You received this message because you are subscribed to the Google Groups
> "codership" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to codership-tea...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
Reply all
Reply to author
Forward
0 new messages