Fail to join cluster at the end of SST

461 views
Skip to first unread message

Pierre Schwartz

unread,
Jan 17, 2018, 4:26:54 AM1/17/18
to codership
Hello,
I am facing a new problem when a galera node is trying to join a running cluster. The data are successfully transfered into the new node but at the end of the IST I've got the following error.
This is not a problem of available memory. It seems it encountered a null pointer exception in a thread, anyone has an idea about it ?

2018-01-17 10:16:27 139922367309568 [Note] WSREP: Shifting JOINER -> OPEN (TO: 132014456)
2018-01-17 10:16:27 139922367309568 [Note] WSREP: Received self-leave message.
2018-01-17 10:16:27 139922367309568 [Note] WSREP: Flow-control interval: [0, 0]
2018-01-17 10:16:27 139922367309568 [Note] WSREP: Received SELF-LEAVE. Closing connection.
2018-01-17 10:16:27 139922367309568 [Note] WSREP: Shifting OPEN -> CLOSED (TO: 132014456)
2018-01-17 10:16:27 139922367309568 [Note] WSREP: RECV thread exiting 0: Success
2018-01-17 10:16:27 139922933397248 [Note] WSREP: recv_thread() joined.
2018-01-17 10:16:27 139922933397248 [Note] WSREP: Closing replication queue.
2018-01-17 10:16:27 139922933397248 [Note] WSREP: Closing slave action queue.
2018-01-17 10:16:27 139922934231616 [Note] WSREP: Signalling provider to continue.
2018-01-17 10:16:27 139922934231616 [Note] WSREP: Read WSREPXid from InnoDB:  51993a17-cb9f-11e7-ba08-5a9b1bd63677:131851967
2018-01-17 10:16:27 139922934231616 [Note] WSREP: SST received: 51993a17-cb9f-11e7-ba08-5a9b1bd63677:131851967
2018-01-17 10:16:27 139922934231616 [ERROR] WSREP: Trying to launch slave threads before creating connection at 'gcomm://192.168.0.130,192.168.0.131,192.168.0.135'
mysqld: /build/mariadb-10.1-5BEZAc/mariadb-10.1-10.1.26/sql/wsrep_thd.cc:447: void wsrep_create_appliers(long int): Assertion `0' failed.
180117 10:16:27 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Server version: 10.1.26-MariaDB-0+deb9u1
key_buffer_size=16777216
read_buffer_size=131072
max_used_connections=0
max_threads=153
thread_count=2
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 352445 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x30000
2018-01-17 10:16:27 139922934015744 [ERROR] WSREP: gcs/src/gcs.cpp:_join():811: Sending JOIN failed: -103 (Software caused connection abort).
2018-01-17 10:16:27 139922934015744 [ERROR] WSREP: Failed to JOIN the cluster after SST
2018-01-17 10:16:27 139922425427712 [Warning] WSREP: Failed to report last committed 131824999, -77 (File descriptor in bad state)
2018-01-17 10:16:27 139922425427712 [Warning] WSREP: Failed to report last committed 131825126, -77 (File descriptor in bad state)
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x56511945e36e]
2018-01-17 10:16:27 139922425427712 [Warning] WSREP: Failed to report last committed 131825254, -77 (File descriptor in bad state)
2018-01-17 10:16:27 139922425427712 [Warning] WSREP: Failed to report last committed 131825383, -77 (File descriptor in bad state)
/usr/sbin/mysqld(handle_fatal_signal+0x3bd)[0x565118fa3a4d]
2018-01-17 10:16:27 139922425427712 [Warning] WSREP: Failed to report last committed 131825511, -77 (File descriptor in bad state)
/lib/x86_64-linux-gnu/libpthread.so.0(+0x110c0)[0x7f42589120c0]
2018-01-17 10:16:27 139922425427712 [Warning] WSREP: Failed to report last committed 131825633, -77 (File descriptor in bad state)
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcf)[0x7f425720cfcf]
2018-01-17 10:16:27 139922425427712 [Warning] WSREP: Failed to report last committed 131825767, -77 (File descriptor in bad state)
2018-01-17 10:16:27 139922425427712 [Warning] WSREP: Failed to report last committed 131825893, -77 (File descriptor in bad state)
2018-01-17 10:16:27 139922425427712 [Warning] WSREP: Failed to report last committed 131826021, -77 (File descriptor in bad state)
2018-01-17 10:16:27 139922425427712 [Warning] WSREP: Failed to report last committed 131826150, -77 (File descriptor in bad state)
/lib/x86_64-linux-gnu/libc.so.6(abort+0x16a)[0x7f425720e3fa]
2018-01-17 10:16:27 139922425427712 [Warning] WSREP: Failed to report last committed 131826279, -77 (File descriptor in bad state)
2018-01-17 10:16:27 139922425427712 [Warning] WSREP: Failed to report last committed 131826406, -77 (File descriptor in bad state)
2018-01-17 10:16:27 139922425427712 [Warning] WSREP: Failed to report last committed 131826535, -77 (File descriptor in bad state)
2018-01-17 10:16:27 139922425427712 [Warning] WSREP: Failed to report last committed 131826663, -77 (File descriptor in bad state)
2018-01-17 10:16:27 139922425427712 [Warning] WSREP: Failed to report last committed 131826787, -77 (File descriptor in bad state)
/lib/x86_64-linux-gnu/libc.so.6(+0x2be37)[0x7f4257205e37]
2018-01-17 10:16:27 139922425427712 [Warning] WSREP: Failed to report last committed 131826919, -77 (File descriptor in bad state)
2018-01-17 10:16:27 139922425427712 [Warning] WSREP: Failed to report last committed 131827047, -77 (File descriptor in bad state)
2018-01-17 10:16:27 139922425427712 [Warning] WSREP: Failed to report last committed 131827174, -77 (File descriptor in bad state)
2018-01-17 10:16:27 139922425427712 [Warning] WSREP: Failed to report last committed 131827302, -77 (File descriptor in bad state)
2018-01-17 10:16:27 139922425427712 [Warning] WSREP: Failed to report last committed 131827431, -77 (File descriptor in bad state)
/lib/x86_64-linux-gnu/libc.so.6(+0x2bee2)[0x7f4257205ee2]
2018-01-17 10:16:27 139922425427712 [Warning] WSREP: Failed to report last committed 131827559, -77 (File descriptor in bad state)
2018-01-17 10:16:27 139922425427712 [Warning] WSREP: Failed to report last committed 131827685, -77 (File descriptor in bad state)
2018-01-17 10:16:27 139922425427712 [Warning] WSREP: Failed to report last committed 131827814, -77 (File descriptor in bad state)
2018-01-17 10:16:27 139922425427712 [Warning] WSREP: Failed to report last committed 131827943, -77 (File descriptor in bad state)
2018-01-17 10:16:27 139922425427712 [Warning] WSREP: Failed to report last committed 131828067, -77 (File descriptor in bad state)
2018-01-17 10:16:27 139922425427712 [Warning] WSREP: Failed to report last committed 131828199, -77 (File descriptor in bad state)
/usr/sbin/mysqld(+0x5523da)[0x565118f533da]
2018-01-17 10:16:27 139922425427712 [Warning] WSREP: Failed to report last committed 131828320, -77 (File descriptor in bad state)
2018-01-17 10:16:27 139922425427712 [Warning] WSREP: Failed to report last committed 131828454, -77 (File descriptor in bad state)
2018-01-17 10:16:27 139922425427712 [Warning] WSREP: Failed to report last committed 131828583, -77 (File descriptor in bad state)
2018-01-17 10:16:27 139922425427712 [Warning] WSREP: Failed to report last committed 131828710, -77 (File descriptor in bad state)
/usr/sbin/mysqld(_Z11mysqld_mainiPPc+0x2494)[0x565118d80214]
2018-01-17 10:16:27 139922425427712 [Warning] WSREP: Failed to report last committed 131828839, -77 (File descriptor in bad state)
2018-01-17 10:16:27 139922425427712 [Warning] WSREP: Failed to report last committed 131828964, -77 (File descriptor in bad state)
2018-01-17 10:16:27 139922425427712 [Warning] WSREP: Failed to report last committed 131829093, -77 (File descriptor in bad state)
2018-01-17 10:16:27 139922425427712 [Warning] WSREP: Failed to report last committed 131829222, -77 (File descriptor in bad state)
2018-01-17 10:16:27 139922425427712 [Warning] WSREP: Failed to report last committed 131829349, -77 (File descriptor in bad state)
2018-01-17 10:16:27 139922425427712 [Warning] WSREP: Failed to report last committed 131829476, -77 (File descriptor in bad state)
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0x7f42571fa2b1]
2018-01-17 10:16:27 139922425427712 [Warning] WSREP: Failed to report last committed 131829606, -77 (File descriptor in bad state)
2018-01-17 10:16:27 139922425427712 [Warning] WSREP: Failed to report last committed 131829735, -77 (File descriptor in bad state)
2018-01-17 10:16:27 139922425427712 [Warning] WSREP: Failed to report last committed 131829861, -77 (File descriptor in bad state)
2018-01-17 10:16:27 139922425427712 [Warning] WSREP: Failed to report last committed 131829989, -77 (File descriptor in bad state)
2018-01-17 10:16:27 139922425427712 [Warning] WSREP: Failed to report last committed 131830118, -77 (File descriptor in bad state)
2018-01-17 10:16:27 139922425427712 [Warning] WSREP: Failed to report last committed 131830247, -77 (File descriptor in bad state)
/usr/sbin/mysqld(_start+0x2a)[0x565118d73c4a]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
Reply all
Reply to author
Forward
0 new messages