MariaDB galera 10.1.17 replication crashes

162 views
Skip to first unread message

Michael Mittentag

unread,
Sep 14, 2016, 10:05:55 AM9/14/16
to codership
Hi,

I currently have a production cluster MariaDB 5.5.34 with galera ( wsrep_23.7.6). 

I setup a new MariaDB cluster with 10.1.17, and made one of the nodes in this new cluster a slave of one of the nodes from the MariaDB galera 5.5.34 cluster.

Every time I start up the slave on the server crashes with the following:



2016-09-14  9:30:21 139628806363904 [Note] WSREP: ready state reached

2016-09-14  9:30:21 139628806363904 [Note] Slave SQL thread initialized, starting replication in log 'standbydb0-bin-log.002736' at position 1049018668, relay log '/var/lib/mysqllogs/slave-relay-log.000002' position: 1107844

2016-09-14 09:30:21 7efddd675b00  InnoDB: Assertion failure in thread 139628806363904 in file row0ins.cc line 283

InnoDB: Failing assertion: *cursor->index->name == TEMP_INDEX_PREFIX

InnoDB: We intentionally generate a memory trap.

InnoDB: Submit a detailed bug report to http://bugs.mysql.com.

InnoDB: If you get repeated assertion failures or crashes, even

InnoDB: immediately after the mysqld startup, there may be

InnoDB: corruption in the InnoDB tablespace. Please refer to

InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html

InnoDB: about forcing recovery.

160914  9:30:21 [ERROR] mysqld got signal 6 ;

This could be because you hit a bug. It is also possible that this binary

or one of the libraries it was linked against is corrupt, improperly built,

or misconfigured. This error can also be caused by malfunctioning hardware.

 

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

 

We will try our best to scrape up some info that will hopefully help

diagnose the problem, but since we have already crashed,

something is definitely wrong and this may fail.

 

Server version: 10.1.17-MariaDB

key_buffer_size=67108864

read_buffer_size=1048576

max_used_connections=1

max_threads=2002

thread_count=18

It is possible that mysqld could use up to

key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 4206730 K  bytes of memory

Hope that's ok; if not, decrease some variables in the equation.

 

Thread pointer: 0x0x7efe1882b008

Attempting backtrace. You can use the following information to find out

where mysqld died. If you see no messages after this, something went

terribly wrong...

stack_bottom = 0x7efddd6748d0 thread_stack 0x48400

2016-09-14  9:30:21 139628806667008 [Note] Slave I/O thread: connected to master '',replication started in log 'standbydb0-bin-log.002737' at position 771685374

/usr/sbin/mysqld(my_print_stacktrace+0x2b)[0x7f3f0838b16b]

/usr/sbin/mysqld(handle_fatal_signal+0x475)[0x7f3f07ee7375]

/lib64/libpthread.so.0(+0xf100)[0x7f3f074eb100]

/lib64/libc.so.6(gsignal+0x37)[0x7f3f058535f7]

/lib64/libc.so.6(abort+0x148)[0x7f3f05854ce8]

mysys/stacktrace.c:268(my_print_stacktrace)[0x7f3f080e8a0d]

sql/signal_handler.cc:166(handle_fatal_signal)[0x7f3f080eac54]

row/row0ins.cc:3101(row_ins_index_entry)[0x7f3f080eb018]

row/row0mysql.cc:1377(row_insert_for_mysql(unsigned char*, row_prebuilt_t*))[0x7f3f080f7f40]

handler/ha_innodb.cc:8740(ha_innobase::write_row(unsigned char*))[0x7f3f0804d76c]

sql/handler.cc:5891(handler::ha_write_row(unsigned char*))[0x7f3f07ef0df7]

sql/log_event.cc:11585(Rows_log_event::write_row(rpl_group_info*, bool))[0x7f3f07fb5a58]

sql/log_event.cc:11770(Write_rows_log_event::do_exec_row(rpl_group_info*))[0x7f3f07fb5e2d]

sql/log_event.cc:10063(Rows_log_event::do_apply_event(rpl_group_info*))[0x7f3f07fa8a9f]

sql/log_event.h:1344(Log_event::apply_event(rpl_group_info*))[0x7f3f07d072e1]

sql/slave.cc:3380(apply_event_and_update_pos(Log_event*, THD*, rpl_group_info*, rpl_parallel_thread*))[0x7f3f07cfd28b]

sql/slave.cc:3707(exec_relay_log_event)[0x7f3f07d0085a]

/lib64/libpthread.so.0(+0x7dc5)[0x7f3f074e3dc5]

/lib64/libc.so.6(clone+0x6d)[0x7f3f05914bdd]

 

Trying to get some variables.

Some pointers may be invalid and cause the dump to abort.

Query (0x0): is an invalid pointer

Connection ID (thread ID): 23

Status: NOT_KILLED

 

Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=off

 

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains

information that should help you find out what is causing the crash.

 

We think the query pointer is invalid, but we will try to print it anyway.

Query:

 



any ideas?  is this a known issue or bug?



Thanks,

Michael


hunter86bg

unread,
Sep 14, 2016, 10:23:48 AM9/14/16
to codership
Are you sure that MariaDB 5.5.34 is compatible with MariaDB  10.1.17 ?  I lack experience with MariaDB, but I doubt that what you are doing is even possible.
In your shoes, I would create a clean dump from one of the nodes and then import it in the new cluster (freshly installed).

Michael Mittentag

unread,
Sep 14, 2016, 12:58:11 PM9/14/16
to codership
It should, since when you go to upgrade you always want to upgrade the slaves first, then the master, since if you have a large environment you cannot upgrade everything at once.  The only difference here is that this a master / slave within 2 separate galera clusters

Jörg Brühe

unread,
Sep 15, 2016, 3:35:56 AM9/15/16
to codersh...@googlegroups.com
Hi all!

On 14.09.2016 18:58, Michael Mittentag wrote (re-ordered):
> On Wednesday, September 14, 2016 at 10:23:48 AM UTC-4, hunter86bg wrote:
>
> Are you sure that MariaDB 5.5.34 is compatible with MariaDB 10.1.17
> ? I lack experience with MariaDB, but I doubt that what you are
> doing is even possible.
> In your shoes, I would create a clean dump from one of the nodes and
> then import it in the new cluster (freshly installed).
>
> It should, since when you go to upgrade you always want to upgrade the
> slaves first, then the master, since if you have a large environment you
> cannot upgrade everything at once. The only difference here is that
> this a master / slave within 2 separate galera clusters

Michael, you are missing the fact that there was/is a MariaDB 10.0.
Replication is not specified to work across a version gap, only from one
release family to the next: 5.5 -> 10.0 *or* 10.0 -> 10.1

Replicating from 5.5 directly to 10.1 is violating the specification, as
you can read here:
http://dev.mysql.com/doc/refman/5.5/en/replication-compatibility.html

Take care!
Jörg

--
Joerg Bruehe, Senior MySQL Support Engineer, joerg....@fromdual.com
FromDual GmbH, Rebenweg 6, CH - 8610 Uster; phone +41 44 500 58 26
Geschäftsführer: Oliver Sennhauser
Handelsregister-Eintrag: CH-020.4.044.539-3

Michael Mittentag

unread,
Sep 16, 2016, 11:40:21 AM9/16/16
to codership
Thanks you are correct, I realized that afterwards, I launched a 10.0.27 cluster and was able to replicate from my current production 5.5 to this new one and replication works without any issue.

I guess I will need to first upgrade to 10.0, and then later go 10.1

-Michael
Reply all
Reply to author
Forward
0 new messages