Node Crash on 10.1.17

30 views
Skip to first unread message

Wilson Lo

unread,
Oct 10, 2016, 1:35:15 AM10/10/16
to codership
Hi, We have formed a three node cluster, and recently experienced node crash on two nodes:

We are running galera 10.1.17. We have spotted log as follow:

Oct  9 06:28:28 ip-10-12-3-22 mysqld: 161009  6:28:28 [ERROR] mysqld got signal 11 ;
Oct  9 06:28:28 ip-10-12-3-22 mysqld: This could be because you hit a bug. It is also possible that this binary
Oct  9 06:28:28 ip-10-12-3-22 mysqld: or one of the libraries it was linked against is corrupt, improperly built,
Oct  9 06:28:28 ip-10-12-3-22 mysqld: or misconfigured. This error can also be caused by malfunctioning hardware.
Oct  9 06:28:28 ip-10-12-3-22 mysqld: 
Oct  9 06:28:28 ip-10-12-3-22 mysqld: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
Oct  9 06:28:28 ip-10-12-3-22 mysqld: 
Oct  9 06:28:28 ip-10-12-3-22 mysqld: We will try our best to scrape up some info that will hopefully help
Oct  9 06:28:28 ip-10-12-3-22 mysqld: diagnose the problem, but since we have already crashed,
Oct  9 06:28:28 ip-10-12-3-22 mysqld: something is definitely wrong and this may fail.
Oct  9 06:28:28 ip-10-12-3-22 mysqld: 
Oct  9 06:28:28 ip-10-12-3-22 mysqld: Server version: 10.1.17-MariaDB-1~trusty
Oct  9 06:28:28 ip-10-12-3-22 mysqld: key_buffer_size=134217728
Oct  9 06:28:28 ip-10-12-3-22 mysqld: read_buffer_size=2097152
Oct  9 06:28:28 ip-10-12-3-22 mysqld: max_used_connections=2
Oct  9 06:28:28 ip-10-12-3-22 mysqld: max_threads=102
Oct  9 06:28:28 ip-10-12-3-22 mysqld: thread_count=4
Oct  9 06:28:28 ip-10-12-3-22 mysqld: It is possible that mysqld could use up to
Oct  9 06:28:28 ip-10-12-3-22 mysqld: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 759834 K  bytes of memory
Oct  9 06:28:28 ip-10-12-3-22 mysqld: Hope that's ok; if not, decrease some variables in the equation.
Oct  9 06:28:28 ip-10-12-3-22 mysqld: 
Oct  9 06:28:28 ip-10-12-3-22 mysqld: Thread pointer: 0x0x7fd40b77f008
Oct  9 06:28:28 ip-10-12-3-22 mysqld: Attempting backtrace. You can use the following information to find out
Oct  9 06:28:28 ip-10-12-3-22 mysqld: where mysqld died. If you see no messages after this, something went
Oct  9 06:28:28 ip-10-12-3-22 mysqld: terribly wrong...
Oct  9 06:28:28 ip-10-12-3-22 mysqld: stack_bottom = 0x7fd4523051f0 thread_stack 0x48400
Oct  9 06:28:28 ip-10-12-3-22 mysqld: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x7fd451d8985e]
Oct  9 06:28:28 ip-10-12-3-22 mysqld: /usr/sbin/mysqld(handle_fatal_signal+0x2d5)[0x7fd4518b04c5]
Oct  9 06:28:28 ip-10-12-3-22 mysqld: /lib/x86_64-linux-gnu/libpthread.so.0(+0x10330)[0x7fd44fe05330]
Oct  9 06:28:28 ip-10-12-3-22 mysqld: /usr/sbin/mysqld(_ZN13MYSQL_BIN_LOG21do_checkpoint_requestEm+0x9d)[0x7fd4519723cd]
Oct  9 06:28:28 ip-10-12-3-22 mysqld: /usr/sbin/mysqld(_ZN13MYSQL_BIN_LOG20checkpoint_and_purgeEm+0x11)[0x7fd451972461]
Oct  9 06:28:28 ip-10-12-3-22 mysqld: /usr/sbin/mysqld(_ZN13MYSQL_BIN_LOG16rotate_and_purgeEb+0xc2)[0x7fd451974ad2]
Oct  9 06:28:28 ip-10-12-3-22 mysqld: /usr/sbin/mysqld(_Z20reload_acl_and_cacheP3THDyP10TABLE_LISTPi+0x130)[0x7fd451818fc0]
Oct  9 06:28:28 ip-10-12-3-22 mysqld: /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x1314)[0x7fd45172c3b4]
Oct  9 06:28:28 ip-10-12-3-22 mysqld: /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x1314)[0x7fd45172c3b4]
Oct  9 06:28:28 ip-10-12-3-22 mysqld: /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x26d)[0x7fd4517348cd]
Oct  9 06:28:28 ip-10-12-3-22 mysqld: /usr/sbin/mysqld(+0x4360d9)[0x7fd4517350d9]
Oct  9 06:28:28 ip-10-12-3-22 mysqld: /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x1e2b)[0x7fd4517375db]
Oct  9 06:28:28 ip-10-12-3-22 mysqld: /usr/sbin/mysqld(_Z10do_commandP3THD+0x169)[0x7fd4517383c9]
Oct  9 06:28:28 ip-10-12-3-22 mysqld: /usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x18a)[0x7fd4517fec3a]
Oct  9 06:28:28 ip-10-12-3-22 mysqld: /usr/sbin/mysqld(handle_one_connection+0x40)[0x7fd4517fee10]
Oct  9 06:28:28 ip-10-12-3-22 mysqld: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8184)[0x7fd44fdfd184]
Oct  9 06:28:28 ip-10-12-3-22 mysqld: /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fd44f31c37d]
Oct  9 06:28:28 ip-10-12-3-22 mysqld: 
Oct  9 06:28:28 ip-10-12-3-22 mysqld: Trying to get some variables.
Oct  9 06:28:28 ip-10-12-3-22 mysqld: Some pointers may be invalid and cause the dump to abort.
Oct  9 06:28:28 ip-10-12-3-22 mysqld: Query (0x7fd43f743020): is an invalid pointer
Oct  9 06:28:28 ip-10-12-3-22 mysqld: Connection ID (thread ID): 5495
Oct  9 06:28:28 ip-10-12-3-22 mysqld: Status: NOT_KILLED
Oct  9 06:28:28 ip-10-12-3-22 mysqld: 
Oct  9 06:28:28 ip-10-12-3-22 mysqld: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=off
Oct  9 06:28:28 ip-10-12-3-22 mysqld: 
Oct  9 06:28:28 ip-10-12-3-22 mysqld: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
Oct  9 06:28:28 ip-10-12-3-22 mysqld: information that should help you find out what is causing the crash.
Oct  9 06:28:28 ip-10-12-3-22 mysqld: 
Oct  9 06:28:28 ip-10-12-3-22 mysqld: We think the query pointer is invalid, but we will try to print it anyway.
Oct  9 06:28:28 ip-10-12-3-22 mysqld: Query: flush logs
Oct  9 06:28:28 ip-10-12-3-22 mysqld: 
Oct  9 06:28:28 ip-10-12-3-22 mysqld_safe: Number of processes running now: 0
 
I am just wondering if this is some bug that we just hit?

regards,
Wilson

Philip Stoev

unread,
Oct 10, 2016, 2:17:47 AM10/10/16
to Wilson Lo, codersh...@googlegroups.com
Hello,

It seems that you have run into a MariaDB-specific issue. You may wish to
check out the following ticket:

https://jira.mariadb.org/browse/MDEV-9510

and the tickets linked to it. It seems that running FLUSH LOGS is what
caused the crash, so please see if you can avoid running this statement.

Philip Stoev
--
You received this message because you are subscribed to the Google Groups
"codership" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to codership-tea...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Reply all
Reply to author
Forward
0 new messages