upgrade to percona 5.5 crash

[David Stainton]

Hi,

I ran into a weird bug after upgrading to percona 5.5;

The master runs Percona 5.1.57-rel12.8-log

The slave runs Percona 5.1.56-rel12.7 - I did an in-place upgrade to 55-5.5.23-rel25.3.240

as described here: http://www.percona.com/doc/percona-server/5.5/upgrading_guide_51_55.html

Now this isn't really good. Next time I'll install the exact same version of percona on the slave as is on the master.

After the upgrade I noticed replication stopped and there was this error in the log:

120517 23:34:26  InnoDB: Assertion failure in thread 1692027200 in file row0mysql.c line 1475

InnoDB: Failing assertion: node->pcur->rel_pos == BTR_PCUR_ON

InnoDB: We intentionally generate a memory trap.

InnoDB: Submit a detailed bug report to http://bugs.mysql.com.

InnoDB: If you get repeated assertion failures or crashes, even

InnoDB: immediately after the mysqld startup, there may be

InnoDB: corruption in the InnoDB tablespace. Please refer to

InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html

InnoDB: about forcing recovery.

23:34:26 UTC - mysqld got signal 6 ;

This could be because you hit a bug. It is also possible that this binary

or one of the libraries it was linked against is corrupt, improperly built,

or misconfigured. This error can also be caused by malfunctioning hardware.

We will try our best to scrape up some info that will hopefully help

diagnose the problem, but since we have already crashed,

something is definitely wrong and this may fail.

key_buffer_size=134217728

read_buffer_size=1048576

max_used_connections=2

max_threads=10000

thread_count=1

connection_count=1

It is possible that mysqld could use up to

key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 30973572 K  bytes of memory

Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x2ab29a8acc30

Attempting backtrace. You can use the following information to find out

where mysqld died. If you see no messages after this, something went

terribly wrong...

stack_bottom = 64da40f8 thread_stack 0x40000

/usr/sbin/mysqld(my_print_stacktrace+0x35)[0x7a3b95]

/usr/sbin/mysqld(handle_fatal_signal+0x4a4)[0x67f474]

/lib64/libpthread.so.0[0x370680ebe0]

/lib64/libc.so.6(gsignal+0x35)[0x3706030285]

/lib64/libc.so.6(abort+0x110)[0x3706031d30]

/usr/sbin/mysqld[0x7cf28f]

/usr/sbin/mysqld[0x7b372f]

/usr/sbin/mysqld(_ZN7handler13ha_delete_rowEPKh+0x5f)[0x684a5f]

/usr/sbin/mysqld(_ZN12ha_partition10delete_rowEPKh+0x7a)[0x95481a]

/usr/sbin/mysqld(_ZN7handler13ha_delete_rowEPKh+0x5f)[0x684a5f]

/usr/sbin/mysqld(_Z12mysql_deleteP3THDP10TABLE_LISTP4ItemP10SQL_I_ListI8st_orderEyy+0xab5)[0x760b05]

/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x2e75)[0x5880b5]

/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x333)[0x589d53]

/usr/sbin/mysqld(_ZN15Query_log_event14do_apply_eventEPK14Relay_log_infoPKcj+0xcc8)[0x725468]

/usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14Relay_log_info+0x125)[0x528275]

/usr/sbin/mysqld[0x52cd39]

/usr/sbin/mysqld(handle_slave_sql+0x919)[0x52e109]

/lib64/libpthread.so.0[0x370680677d]

/lib64/libc.so.6(clone+0x6d)[0x37060d325d]

Trying to get some variables.

Some pointers may be invalid and cause the dump to abort.

Query (2ab29a8b7ee4): is an invalid pointer

Connection ID (thread ID): 9

Status: NOT_KILLED

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains

information that should help you find out what is causing the crash.

120517 23:34:26 mysqld_safe Number of processes running now: 0

120517 23:34:26 mysqld_safe mysqld restarted

120517 23:34:26 [Warning] The syntax '--log-slow-queries' is deprecated and will be removed in a future release. Please use '--slow-query-log'/'--slow-query-log-file' instead.

120517 23:34:26 [Note] Flashcache bypass: disabled

120517 23:34:26 [Note] Flashcache setup error is : ioctl failed

120517 23:34:26 [Note] Plugin 'FEDERATED' is disabled.

120517 23:34:26 InnoDB: The InnoDB memory heap is disabled

120517 23:34:26 InnoDB: Mutexes and rw_locks use GCC atomic builtins

120517 23:34:26 InnoDB: Compressed tables use zlib 1.2.3

120517 23:34:26 InnoDB: Using Linux native AIO

120517 23:34:26 InnoDB: Initializing buffer pool, size = 30.0G

120517 23:34:28 InnoDB: Completed initialization of buffer pool

120517 23:34:28 InnoDB: highest supported file format is Barracuda.

InnoDB: The log sequence number in ibdata files does not match

InnoDB: the log sequence number in the ib_logfiles!

120517 23:34:28  InnoDB: Database was not shut down normally!

InnoDB: Starting crash recovery.

InnoDB: Reading tablespace information from the .ibd files...

InnoDB: Restoring possible half-written data pages from the doublewrite

InnoDB: buffer...

InnoDB: In a MySQL replication slave the last master binlog file

InnoDB: position 0 959094133, file name dog-db01-bin.001516

InnoDB: and relay log file

InnoDB: position 0 4300897, file name /disk1/mysql/dog-db01-relay.000007

InnoDB: Last MySQL binlog file position 0 135782318, file name /var/log/mysql/dog-db01-bin.001515

120517 23:34:30  InnoDB: Waiting for the background threads to start

120517 23:34:31 Percona XtraDB (http://www.percona.com) 1.1.8-rel25.3 started; log sequence number 2188901348217

120517 23:34:31 [Warning] 'user' entry 'root@fu' ignored in --skip-name-resolve mode.

120517 23:34:31 [Warning] 'user' entry '@fu' ignored in --skip-name-resolve mode.

120517 23:34:31 [Note] Event Scheduler: Loaded 0 events

120517 23:34:31 [Note] /usr/sbin/mysqld: ready for connections.

Version: '5.5.23-55-log'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Percona Server (GPL), Release rel25.3, Revision 240

What do you think happened here?

I even tried restarting mysql with a smaller buffer pool... but no there is something else going on.

I think I'll try it again without the mismatched 5.1 versions...

Cheers,

David

[Laurynas Biveinis]

Hello David -

> 120517 23:34:26  InnoDB: Assertion failure in thread 1692027200 in file
> row0mysql.c line 1475
> InnoDB: Failing assertion: node->pcur->rel_pos == BTR_PCUR_ON

> stack_bottom = 64da40f8 thread_stack 0x40000
> /usr/sbin/mysqld(my_print_stacktrace+0x35)[0x7a3b95]
> /usr/sbin/mysqld(handle_fatal_signal+0x4a4)[0x67f474]
> /lib64/libpthread.so.0[0x370680ebe0]
> /lib64/libc.so.6(gsignal+0x35)[0x3706030285]
> /lib64/libc.so.6(abort+0x110)[0x3706031d30]
> /usr/sbin/mysqld[0x7cf28f]
> /usr/sbin/mysqld[0x7b372f]
> /usr/sbin/mysqld(_ZN7handler13ha_delete_rowEPKh+0x5f)[0x684a5f]
> /usr/sbin/mysqld(_ZN12ha_partition10delete_rowEPKh+0x7a)[0x95481a]
> /usr/sbin/mysqld(_ZN7handler13ha_delete_rowEPKh+0x5f)[0x684a5f]
> /usr/sbin/mysqld(_Z12mysql_deleteP3THDP10TABLE_LISTP4ItemP10SQL_I_ListI8st_orderEyy+0xab5)[0x760b05]
> /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x2e75)[0x5880b5]
> /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x333)[0x589d53]
> /usr/sbin/mysqld(_ZN15Query_log_event14do_apply_eventEPK14Relay_log_infoPKcj+0xcc8)[0x725468]
> /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14Relay_log_info+0x125)[0x528275]
> /usr/sbin/mysqld[0x52cd39]
> /usr/sbin/mysqld(handle_slave_sql+0x919)[0x52e109]
> /lib64/libpthread.so.0[0x370680677d]
> /lib64/libc.so.6(clone+0x6d)[0x37060d325d]

It is likely that you have encountered a bug in partitioning. Is it
possible for you to test with the Oracle MySQL server? Can you try to
provide a minimum test case that crashes the slave and file a bug?
Looking at other partitioning bugs I see that CREATE TABLE and any
subsequent DDL statements for the partitioned table with some subset
of data should be enough.

Here are some other partitioning bugs that are fixed by now but have
similar symptoms:
http://bugs.mysql.com/bug.php?id=42849
http://bugs.mysql.com/bug.php?id=55944

--
Laurynas
www.percona.com

[David Stainton]

Hi,

Partitioning?

I'm just trying to do an in-place upgrade.

This here bug causes replication to stop... but doesn't actually crash mysql.

I will try this again later...

David

--
Laurynas
www.percona.com

--
You received this message because you are subscribed to the Google Groups "Percona Discussion" group.
To post to this group, send email to percona-d...@googlegroups.com.
To unsubscribe from this group, send email to percona-discuss...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/percona-discussion?hl=en.

[Laurynas Biveinis]

On Fri, May 18, 2012 at 10:13 PM, David Stainton <dstain...@gmail.com> wrote:
> Partitioning?
> I'm just trying to do an in-place upgrade.

The stacktrace shows that the partitioned table handler is being
called, that's why I mentioned partitioning. Are you saying that you
don't actually have any partitioned tables?

Thanks,
--
Laurynas
www.percona.com