Percona XtraDB Cluster 5.5.30-23.7.4-405.squeeze assertion failure issue

[Ilias Bertsimas]

Hello,

We upgraded to the latest PXC version  5.5.30-23.7.4-405.squeeze  in debian 6.0 throught the Percona apt repository and it was running fine for 5 days now.

Just out of the blue it crashed today.

Here is some of the error log:

130506 18:17:20  InnoDB: Assertion failure in thread 140601459480320 in file buf0buf.ic line 1277

InnoDB: Failing assertion: block->page.buf_fix_count > 0

InnoDB: We intentionally generate a memory trap.

InnoDB: Submit a detailed bug report to http://bugs.mysql.com.

InnoDB: If you get repeated assertion failures or crashes, even

InnoDB: immediately after the mysqld startup, there may be

InnoDB: corruption in the InnoDB tablespace. Please refer to

InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html

InnoDB: about forcing recovery.

16:17:20 UTC - mysqld got signal 6 ;

This could be because you hit a bug. It is also possible that this binary

or one of the libraries it was linked against is corrupt, improperly built,

or misconfigured. This error can also be caused by malfunctioning hardware.

We will try our best to scrape up some info that will hopefully help

diagnose the problem, but since we have already crashed,

something is definitely wrong and this may fail.

Please help us make Percona Server better by reporting any

bugs at http://bugs.percona.com/

Thread pointer: 0x7fdaf21701e0

Attempting backtrace. You can use the following information to find out

where mysqld died. If you see no messages after this, something went

terribly wrong...

stack_bottom = 7fe0540c0e78 thread_stack 0x40000

/usr/sbin/mysqld(my_print_stacktrace+0x35)[0x7ed245]

/usr/sbin/mysqld(handle_fatal_signal+0x4b4)[0x6ba864]

/lib/libpthread.so.0(+0xeff0)[0x7fe4993c0ff0]

/lib/libc.so.6(gsignal+0x35)[0x7fe497fc61b5]

/lib/libc.so.6(abort+0x180)[0x7fe497fc8fc0]

/usr/sbin/mysqld[0x9251b2]

/usr/sbin/mysqld[0x92548e]

/usr/sbin/mysqld[0x846984]

/usr/sbin/mysqld[0x816954]

/usr/sbin/mysqld(_ZN12ha_partition21handle_unordered_nextEPhb+0x50)[0x9aa840]

/usr/sbin/mysqld[0x5c6db1]

/usr/sbin/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0x79)[0x5d0099]

/usr/sbin/mysqld[0x5d0a67]

/usr/sbin/mysqld(_ZN4JOIN4execEv+0x4b5)[0x5e56f5]

/usr/sbin/mysqld(_Z12mysql_selectP3THDPPP4ItemP10TABLE_LISTjR4ListIS1_ES2_jP8st_orderSB_S2_SB_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x12c)[0x5e766c]

/usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1cd)[0x5e812d]

/usr/sbin/mysqld[0x5a0782]

/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x3415)[0x5a7a65]

/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x343)[0x5a9d33]

/usr/sbin/mysqld[0x5aadd2]

/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x1a92)[0x5acf72]

/usr/sbin/mysqld(_Z10do_commandP3THD+0x167)[0x5ad567]

/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x14f)[0x64b4cf]

/usr/sbin/mysqld(handle_one_connection+0x51)[0x64b6b1]

/lib/libpthread.so.0(+0x68ca)[0x7fe4993b88ca]

/lib/libc.so.6(clone+0x6d)[0x7fe498063b6d]

Trying to get some variables.

Some pointers may be invalid and cause the dump to abort.

Query (7fdad00d33c0): is an invalid pointer

Connection ID (thread ID): 1593192

Status: NOT_KILLED

[Raghavendra D Prabhu]

Hi Illias,

>--
>You received this message because you are subscribed to the Google Groups "Percona Discussion" group.
>To unsubscribe from this group and stop receiving emails from it, send an email to percona-discuss...@googlegroups.com.
>To post to this group, send email to percona-d...@googlegroups.com.
>For more options, visit https://groups.google.com/groups/opt_out.
>
>

From the assertion it looks like -
http://bugs.mysql.com/bug.php?id=26081 - are you on AMD? Also, can
you provide details on your platform - OS (kernel version) and hardware. Also, look at
http://timetobleed.com/mysql-doesnt-always-suck-this-time-its-amd/


Regards,
--
Raghavendra Prabhu
Percona LLC. - http://www.percona.com / Blog: http://www.mysqlperformanceblog.com/
Skype: percona.raghavendrap
GPG: 0xD72BE977
Contact: http://wnohang.net/contact

Make plans to attend Percona Live London MySQL Conference
2013.<http://www.percona.com/live/london-2013/>

[Ilias Bertsimas]

Hi Raghavendra,

We have no amd cpus Intel(R) Xeon(R) CPU           E5630  @ 2.53GHz.

Our platfrom is Debian 6.0 squeeze with kernel 3.2.35-xxxx-std-ipv6-64 #1 SMP Thu Jan 31 14:09:17 CET 2013 x86_64 GNU/Linux.

Kind Regards,

Ilias.

>To unsubscribe from this group and stop receiving emails from it, send an email to percona-discussion+unsub...@googlegroups.com.

[Ilias Bertsimas]

It happened again today:

130507 12:05:28  InnoDB: Assertion failure in thread 140011477292800 in file buf0buf.ic line 1277

InnoDB: Failing assertion: block->page.buf_fix_count > 0

InnoDB: We intentionally generate a memory trap.

InnoDB: Submit a detailed bug report to http://bugs.mysql.com.

InnoDB: If you get repeated assertion failures or crashes, even

InnoDB: immediately after the mysqld startup, there may be

InnoDB: corruption in the InnoDB tablespace. Please refer to

InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html

InnoDB: about forcing recovery.

10:05:28 UTC - mysqld got signal 6 ;

This could be because you hit a bug. It is also possible that this binary

or one of the libraries it was linked against is corrupt, improperly built,

or misconfigured. This error can also be caused by malfunctioning hardware.

We will try our best to scrape up some info that will hopefully help

diagnose the problem, but since we have already crashed,

something is definitely wrong and this may fail.

Please help us make Percona Server better by reporting any

bugs at http://bugs.percona.com/

Thread pointer: 0x7f56a2102600

Attempting backtrace. You can use the following information to find out

where mysqld died. If you see no messages after this, something went

terribly wrong...

stack_bottom = 7f56f65e4e78 thread_stack 0x40000

/usr/sbin/mysqld(my_print_stacktrace+0x35)[0x7ed245]

/usr/sbin/mysqld(handle_fatal_signal+0x4b4)[0x6ba864]

/lib/libpthread.so.0(+0xeff0)[0x7f60261d9ff0]

/lib/libc.so.6(gsignal+0x35)[0x7f6024ddf1b5]

/lib/libc.so.6(abort+0x180)[0x7f6024de1fc0]

/usr/sbin/mysqld[0x9251b2]

/usr/sbin/mysqld[0x92548e]

/usr/sbin/mysqld[0x846984]

/usr/sbin/mysqld[0x816954]

/usr/sbin/mysqld(_ZN12ha_partition21handle_unordered_nextEPhb+0x50)[0x9aa840]

/usr/sbin/mysqld[0x5c6db1]

/usr/sbin/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0x79)[0x5d0099]

/usr/sbin/mysqld[0x5d0a67]

/usr/sbin/mysqld(_ZN4JOIN4execEv+0x4b5)[0x5e56f5]

/usr/sbin/mysqld(_Z12mysql_selectP3THDPPP4ItemP10TABLE_LISTjR4ListIS1_ES2_jP8st_orderSB_S2_SB_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x12c)[0x5e766c]

/usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1cd)[0x5e812d]

/usr/sbin/mysqld[0x5a0782]

/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x3415)[0x5a7a65]

/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x343)[0x5a9d33]

/usr/sbin/mysqld[0x5aadd2]

/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x1a92)[0x5acf72]

/usr/sbin/mysqld(_Z10do_commandP3THD+0x167)[0x5ad567]

/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x14f)[0x64b4cf]

/usr/sbin/mysqld(handle_one_connection+0x51)[0x64b6b1]

/lib/libpthread.so.0(+0x68ca)[0x7f60261d18ca]

/lib/libc.so.6(clone+0x6d)[0x7f6024e7cb6d]

Starting up again the node seems to be working recovering wsrep positioon and taking an IST.

[Raghavendra D Prabhu]

Hi Ilias,

Can you report this bug PXC bug tracker? This looks like a bug in
Percona Server or MYSQL. I will check on this and update it
there.

>
>Starting up again the node seems to be working recovering wsrep positioon
>and taking an IST.
>
>On Tuesday, May 7, 2013 11:22:58 AM UTC+3, Ilias Bertsimas wrote:
>>
>> Hi Raghavendra,
>>
>> We have no amd cpus Intel(R) Xeon(R) CPU E5630 @ 2.53GHz.
>> Our platfrom is Debian 6.0 squeeze with kernel 3.2.35-xxxx-std-ipv6-64 #1
>> SMP Thu Jan 31 14:09:17 CET 2013 x86_64 GNU/Linux.
>>
>>
>> Kind Regards,
>> Ilias.
>>

>> *
>> *

>>> an email to percona-discuss...@googlegroups.com.

>>> >To post to this group, send email to percona-d...@googlegroups.com.
>>> >For more options, visit https://groups.google.com/groups/opt_out.
>>> >
>>> >
>>>
>>> From the assertion it looks like -
>>> http://bugs.mysql.com/bug.php?id=26081 - are you on AMD? Also, can
>>> you provide details on your platform - OS (kernel version) and hardware.
>>> Also, look at
>>> http://timetobleed.com/mysql-doesnt-always-suck-this-time-its-amd/
>>>
>>>
>>> Regards,
>>> --
>>> Raghavendra Prabhu
>>> Percona LLC. - http://www.percona.com / Blog:
>>> http://www.mysqlperformanceblog.com/
>>> Skype: percona.raghavendrap
>>> GPG: 0xD72BE977
>>> Contact: http://wnohang.net/contact
>>>
>>> Make plans to attend Percona Live London MySQL Conference
>>> 2013.<http://www.percona.com/live/london-2013/>
>>>
>>
>

>--
>You received this message because you are subscribed to the Google Groups "Percona Discussion" group.

>To unsubscribe from this group and stop receiving emails from it, send an email to percona-discuss...@googlegroups.com.

>To post to this group, send email to percona-d...@googlegroups.com.
>For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Ilias Bertsimas]

Hi Raghavendra,

I just reported here: https://bugs.launchpad.net/percona-xtradb-cluster/+bug/1178575

Kind Regards,

Ilias.

>>> an email to percona-discussion+unsub...@googlegroups.com.

>>> >To post to this group, send email to percona-d...@googlegroups.com.
>>> >For more options, visit https://groups.google.com/groups/opt_out.
>>> >
>>> >
>>>
>>>  From the assertion it looks like -
>>> http://bugs.mysql.com/bug.php?id=26081 - are you on AMD? Also, can
>>> you provide details on your platform - OS (kernel version) and hardware.
>>> Also, look at
>>> http://timetobleed.com/mysql-doesnt-always-suck-this-time-its-amd/
>>>
>>>
>>> Regards,
>>> --
>>> Raghavendra Prabhu
>>> Percona LLC. - http://www.percona.com / Blog:
>>> http://www.mysqlperformanceblog.com/
>>> Skype: percona.raghavendrap
>>> GPG: 0xD72BE977
>>> Contact: http://wnohang.net/contact
>>>
>>> Make plans to attend Percona Live London MySQL Conference
>>> 2013.<http://www.percona.com/live/london-2013/>
>>>
>>
>
>--
>You received this message because you are subscribed to the Google Groups "Percona Discussion" group.

>To unsubscribe from this group and stop receiving emails from it, send an email to percona-discussion+unsub...@googlegroups.com.

[Amol]

hi IIias and Gea-Suan,

So when it crashed what were the steps you took to recover?

did all the 3 nodes crash?

or just 1 node crashed and started fine after SST?

Just trying to learn more from your experiences, as i am running 5.5.30 on 5 clusters now...on ubuntu 10.04

[Laurent MINOST]

Hi all,

We are also running 5.5.30 for only a few hours now (upgraded from 5.5.29) and I'm seeing this topic (maybe a little late because I would probably not upgraded then ...), hope that we will not faced the same situation ...

Like Amol, I would be also very interested to get more informations about the "how to recover" process and how the problem occurs ? Were you able to find a similar context when the two crashes occured ? (same queries, hour, load, high i/O ... etc ) 

Thanks by advance for your replies.
Regards,

Laurent

[Ilias Bertsimas]

Hi guys,

As it is an InnoDB assertion failure it does not affect the whole cluster at least you will not lose all nodes at the same time. I lost one node one day and the other the next.

You can recover it by simply starting mysql in the node normally. In my case 2 out of the 3 times wsrep managed to recover a position and just performed an IST joining the cluster afterwards. In the worst case it will perform an SST.

Kind Regards,

Ilias.

[Laurent MINOST]

Hi Ilias,

Thks for these informations, at least we have some contents to react if the problem occurs, will follow this topic with attention to see when the bug will be identified by Percona !

Really hope to have a release soon where I will install it to fix old bugs but that no "major" new ones are coming with :p because this is my real feeling for the last week releases ...

I'm even interrogated/afraid myself sometimes, against the last releases, and wondered if the product is really "production ready" sometimes ... ? Do we have some users/companies real usage cases/articles in production for PXDBC somewhere please ? (I realise this is out of topic here ... sorry)

Wait and see ...

Best regards,

Laurent

[Laurent MINOST]

Hi Ilias,

I saw (through mail notification yesterday) that you managed to log queries concerned by the assertion failure, this is a good news but this morning I wanted to check the bug status and it seems unavailable at https://bugs.launchpad.net/bugs/1178575

Do you know if this one was deleted/moved ? Or maybe is it a launchpad temporary problem ?

Regards,

Laurent

Le vendredi 10 mai 2013 17:03:25 UTC+2, Ilias Bertsimas a écrit :

[Raghavendra D Prabhu]

Hi Illias/Laurent,

* On Thu, Jun 20, 2013 at 01:34:34AM -0700, Laurent MINOST <lol...@gmail.com> wrote:
>Hi Ilias,
>
>I saw (through mail notification yesterday) that you managed to log queries
>concerned by the assertion failure, this is a good news but this morning I
>wanted to check the bug status and it seems unavailable at
>https://bugs.launchpad.net/bugs/1178575

It was moved to private (so that schema etc. can be shared).

Anyways, PXC 5.5.31 will be released on Monday. Currently we
have RPMs/DEBs in experimental repo for testing. You can test
with it to see if you are still seeing this issue.

>--
>You received this message because you are subscribed to the Google Groups "Percona Discussion" group.

>To unsubscribe from this group and stop receiving emails from it, send an email to percona-discuss...@googlegroups.com.

[Raghavendra D Prabhu]

* On Fri, Jun 21, 2013 at 02:11:49PM +0530, Raghavendra D Prabhu <raghavend...@percona.com> wrote:
>Hi Illias/Laurent,
>
>* On Thu, Jun 20, 2013 at 01:34:34AM -0700, Laurent MINOST <lol...@gmail.com> wrote:
>>Hi Ilias,
>>
>>I saw (through mail notification yesterday) that you managed to log queries
>>concerned by the assertion failure, this is a good news but this morning I
>>wanted to check the bug status and it seems unavailable at
>>https://bugs.launchpad.net/bugs/1178575
>
>It was moved to private (so that schema etc. can be shared).
>
>Anyways, PXC 5.5.31 will be released on Monday. Currently we have
>RPMs/DEBs in experimental repo for testing. You can test with it to
>see if you are still seeing this issue.

Only the RPMs are in experimental at the moment. The debs are a
bit old (and not the latest), but these should be pushed soon as
well.

[Laurent MINOST]

Hi Raghavendra,

Thanks for this reply, I cannot reproduced the problem by myself because I do not have any test case/queries to reproduce it every time, but I'm interested in testing this pre-release on one of my test cluster so If I find something unclear, I can report it to you if you're OK ? Could you please indicates where I can find test packages please ? (I need binary one for this test cluster)

Regards,

Laurent 

>To unsubscribe from this group and stop receiving emails from it, send an email to percona-discussion+unsub...@googlegroups.com.

[Raghavendra D Prabhu]

Hi Minost,

* On Fri, Jun 21, 2013 at 05:05:56AM -0700, Laurent MINOST <lol...@gmail.com> wrote:
>Hi Raghavendra,
>
>Thanks for this reply, I cannot reproduced the problem by myself because I
>do not have any test case/queries to reproduce it every time, but I'm
>interested in testing this pre-release on one of my test cluster so If I
>find something unclear, I can report it to you if you're OK ? Could you
>please indicates where I can find test packages please ? (I need binary one
>for this test cluster)

PXC 5.5.31 has been released - http://www.mysqlperformanceblog.com/2013/06/25/percona-xtradb-cluster-5-5-31-23-7-5-now-available/

It is available in normal repo now.

>
>Regards,
>
>Laurent
>
>Le vendredi 21 juin 2013 10:41:49 UTC+2, Raghavendra Prabhu a écrit :
>>
>> Hi Illias/Laurent,
>>
>> * On Thu, Jun 20, 2013 at 01:34:34AM -0700, Laurent MINOST <

>> email to percona-discuss...@googlegroups.com <javascript:>.
>> >To post to this group, send email to percona-d...@googlegroups.com<javascript:>.

>>
>> >For more options, visit https://groups.google.com/groups/opt_out.
>> >
>> >
>>
>>
>>
>>
>> Regards,
>> --
>> Raghavendra Prabhu
>> Percona LLC. - http://www.percona.com / Blog:
>> http://www.mysqlperformanceblog.com/
>> Skype: percona.raghavendrap
>> GPG: 0xD72BE977
>> Contact: http://wnohang.net/contact
>>
>> Make plans to attend Percona Live London MySQL Conference
>> 2013.<http://www.percona.com/live/london-2013/>
>>
>

>--
>You received this message because you are subscribed to the Google Groups "Percona Discussion" group.

>To unsubscribe from this group and stop receiving emails from it, send an email to percona-discuss...@googlegroups.com.

[Laurent MINOST]

Hi,

Yes thanks, I've just downloaded the i686 version from http://www.percona.com/redir/downloads/Percona-XtraDB-Cluster/5.5.31-23.7.5/binary/linux/i686/Percona-XtraDB-Cluster-5.5.31-23.7.5.438.Linux.i686.tar.gz

but :

130625 11:19:53 mysqld_safe WSREP: Recovered position 00000000-0000-0000-0000-000000000000:-1

130625 11:19:53 [Note] WSREP: wsrep_start_position var submitted: '00000000-0000-0000-0000-000000000000:-1'

130625 11:19:53 [Note] WSREP: Read nil XID from storage engines, skipping position init

130625 11:19:53 [Note] WSREP: wsrep_load(): loading provider library '/opt/mysql-galera/lib/libgalera_smm.so'

130625 11:19:53 [ERROR] WSREP: wsrep_load(): dlopen(): /opt/mysql-galera/lib/libgalera_smm.so: wrong ELF class: ELFCLASS64

130625 11:19:53 [ERROR] WSREP: wsrep_load(/opt/mysql-galera/lib/libgalera_smm.so) failed: Invalid argument (22). Reverting to no provider.

# file /opt/mysql-galera/lib/libgalera_smm.so

/opt/mysql-galera/lib/libgalera_smm.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped

# md5sum /opt/mysql-galera/lib/libgalera_smm.so

3c476a105874541b0ee37e761b6420cd  /opt/mysql-galera/lib/libgalera_smm.so

It seems we have a 64 bits binary for the galera lib in the i686 package :(

At your disposal if you need any more informations !

Regards,

Laurent

>> email to percona-discussion+unsub...@googlegroups.com <javascript:>.

>> >To post to this group, send email to percona-d...@googlegroups.com<javascript:>.
>>
>> >For more options, visit https://groups.google.com/groups/opt_out.
>> >
>> >
>>
>>
>>
>>
>> Regards,
>> --
>> Raghavendra Prabhu
>> Percona LLC. - http://www.percona.com / Blog:
>> http://www.mysqlperformanceblog.com/
>> Skype: percona.raghavendrap
>> GPG: 0xD72BE977
>> Contact: http://wnohang.net/contact
>>
>> Make plans to attend Percona Live London MySQL Conference
>> 2013.<http://www.percona.com/live/london-2013/>
>>
>
>--
>You received this message because you are subscribed to the Google Groups "Percona Discussion" group.

>To unsubscribe from this group and stop receiving emails from it, send an email to percona-discussion+unsub...@googlegroups.com.

[Laurent MINOST]

Hi,

To keep intact the initial problematic of this thread, I've finally created a new thread for the binary package problem of the new XtraDB Cluster version : https://groups.google.com/forum/?fromgroups=#!topic/percona-discussion/YHNq0YwzkZY

Regards,

Laurent

[Ilias Bertsimas]

Hi guys,

Any update on the bug report with the info I provided ?

>To unsubscribe from this group and stop receiving emails from it, send an email to percona-discussion+unsub...@googlegroups.com.