IST completed then cluster crashed
76 views
Skip to first unread message
Brandon Vander Meersch
Jun 14, 2016, 5:21:56 PM6/14/16
to codership
Hello,
I have a 3 node cluster,
Right now 2 of the nodes are offline. I rejoined one of the nodes to the bootstrapped node and it started an IST and seemed to complete as seen in the log below. Once it joined mysqld got a signal 11 and then died. This caused a quorum loss and caused the other node to be bootstrapped again.
From the log can you tell why the node failed after the IST and how I can prevent it again?
2016-06-14 14:21:40 17359 [Note] WSREP: IST received: f904a9a7-db79-11e5-ae9e-6ac3a9358431:933795102
2016-06-14 14:21:40 17359 [Note] WSREP: 1.0 (10.0.3.20): State transfer from 0.0 (10.0.3.21) complete.
2016-06-14 14:21:40 17359 [Note] WSREP: Shifting JOINER -> JOINED (TO: 934002747)
2016-06-14 14:25:44 17359 [Note] WSREP: Member 1.0 (10.0.3.20) synced with group.
2016-06-14 14:25:44 17359 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 934029258)
2016-06-14 14:25:44 17359 [Note] WSREP: Synchronized with group, ready for connections
2016-06-14 14:25:44 17359 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2016-06-14 14:25:58 17359 [Warning] WSREP: last inactive check more than PT1.5S ago (PT1.98961S), skipping check
2016-06-14 14:26:01 17359 [Warning] WSREP: last inactive check more than PT1.5S ago (PT1.87283S), skipping check
2016-06-14 14:26:04 17359 [Warning] WSREP: last inactive check more than PT1.5S ago (PT1.83629S), skipping check
2016-06-14 14:26:19 17359 [Warning] WSREP: Failed to report last committed 934030118, -4 (Interrupted system call)
19:26:31 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona XtraDB Cluster better by reporting any
bugs at https://bugs.launchpad.net/percona-xtradb-cluster
key_buffer_size=268435456
read_buffer_size=131072
max_used_connections=343
max_threads=2002
thread_count=307
connection_count=282
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1059535 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x147807990
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
I have a 3 node cluster,
Right now 2 of the nodes are offline. I rejoined one of the nodes to the bootstrapped node and it started an IST and seemed to complete as seen in the log below. Once it joined mysqld got a signal 11 and then died. This caused a quorum loss and caused the other node to be bootstrapped again.
From the log can you tell why the node failed after the IST and how I can prevent it again?
2016-06-14 14:21:40 17359 [Note] WSREP: IST received: f904a9a7-db79-11e5-ae9e-6ac3a9358431:933795102
2016-06-14 14:21:40 17359 [Note] WSREP: 1.0 (10.0.3.20): State transfer from 0.0 (10.0.3.21) complete.
2016-06-14 14:21:40 17359 [Note] WSREP: Shifting JOINER -> JOINED (TO: 934002747)
2016-06-14 14:25:44 17359 [Note] WSREP: Member 1.0 (10.0.3.20) synced with group.
2016-06-14 14:25:44 17359 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 934029258)
2016-06-14 14:25:44 17359 [Note] WSREP: Synchronized with group, ready for connections
2016-06-14 14:25:44 17359 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2016-06-14 14:25:58 17359 [Warning] WSREP: last inactive check more than PT1.5S ago (PT1.98961S), skipping check
2016-06-14 14:26:01 17359 [Warning] WSREP: last inactive check more than PT1.5S ago (PT1.87283S), skipping check
2016-06-14 14:26:04 17359 [Warning] WSREP: last inactive check more than PT1.5S ago (PT1.83629S), skipping check
2016-06-14 14:26:19 17359 [Warning] WSREP: Failed to report last committed 934030118, -4 (Interrupted system call)
19:26:31 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona XtraDB Cluster better by reporting any
bugs at https://bugs.launchpad.net/percona-xtradb-cluster
key_buffer_size=268435456
read_buffer_size=131072
max_used_connections=343
max_threads=2002
thread_count=307
connection_count=282
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1059535 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x147807990
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Brandon Vander Meersch
Jun 14, 2016, 10:21:28 PM6/14/16
to codership
Tried it again and got this error:
2016-06-14 20:58:13 25109 [Note] WSREP: IST received: f904a9a7-db79-11e5-ae9e-6a c3a9358431:936038530
2016-06-14 20:58:13 25109 [Note] WSREP: 0.0 (10.0.3.20): State transfer from 1.0 (10.0.3.21) complete.
2016-06-14 20:58:13 25109 [Note] WSREP: Shifting JOINER -> JOINED (TO: 936327934 )
2016-06-14 21:03:44 25109 [Note] WSREP: Member 0.0 (10.0.3.20) synced with group .
2016-06-14 21:03:44 25109 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 936358072 )
2016-06-14 21:03:45 25109 [Note] WSREP: Synchronized with group, ready for conne ctions
2016-06-14 21:03:45 25109 [Note] WSREP: wsrep_notify_cmd is not defined, skippin g notification.
02:04:31 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona XtraDB Cluster better by reporting any
bugs at https://bugs.launchpad.net/percona-xtradb-cluster
key_buffer_size=268435456
read_buffer_size=131072
max_used_connections=449
max_threads=2002
thread_count=311
connection_count=286
max_threads=2002
thread_count=311
connection_count=286
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1059535 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x14727f330
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f438c2f5d38 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x35)[0x8fd375]
/usr/sbin/mysqld(handle_fatal_signal+0x4b4)[0x666264]
/lib64/libpthread.so.0[0x3058e0f790]
[0x7f3f04015ea0]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7f3f04004bf0): is an invalid pointer
Connection ID (thread ID): 4947
Status: NOT_KILLED
You may download the Percona XtraDB Cluster operations manual by visiting
http://www.percona.com/software/percona-xtradb-cluster/. You may find informatio n
in the manual which will help you identify the cause of the crash.
160614 21:04:33 mysqld_safe Number of processes running now: 0
160614 21:04:33 mysqld_safe WSREP: not restarting wsrep node automatically
160614 21:04:33 mysqld_safe mysqld from pid file /mysql/mysqld.pid ended
/usr/sbin/mysqld(my_print_stacktrace+0x35)[0x8fd375]
/usr/sbin/mysqld(handle_fatal_signal+0x4b4)[0x666264]
/lib64/libpthread.so.0[0x3058e0f790]
[0x7f3f04015ea0]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7f3f04004bf0): is an invalid pointer
Connection ID (thread ID): 4947
Status: NOT_KILLED
You may download the Percona XtraDB Cluster operations manual by visiting
http://www.percona.com/software/percona-xtradb-cluster/. You may find informatio n
in the manual which will help you identify the cause of the crash.
160614 21:04:33 mysqld_safe Number of processes running now: 0
160614 21:04:33 mysqld_safe WSREP: not restarting wsrep node automatically
160614 21:04:33 mysqld_safe mysqld from pid file /mysql/mysqld.pid ended
Umarzuki Mochlis
Jun 14, 2016, 10:35:50 PM6/14/16
to Brandon Vander Meersch, codership
I had something similar like this before
What I did I renamed datadir folder and make a new empty one owned by
mysql user and group
After I killed all mysql services, then I started mysql service
> --
> You received this message because you are subscribed to the Google Groups
> "codership" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to codership-tea...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
What I did I renamed datadir folder and make a new empty one owned by
mysql user and group
After I killed all mysql services, then I started mysql service
> You received this message because you are subscribed to the Google Groups
> "codership" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to codership-tea...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
James Wang
Jun 15, 2016, 5:25:50 AM6/15/16
to codership
I had similar experience before and my solution was to remove all data (under datadir) on the troubled node and do a SST
Brandon Vander Meersch
Jun 15, 2016, 8:52:14 AM6/15/16
to codership
That's what I was afraid of. I was trying to avoid an SST.
Think this may be a bug?
Brandon Vander Meersch
Jun 15, 2016, 2:01:13 PM6/15/16
to codership
I tried to restore and IST on a different server and it worked without issue.
The signal 11 typically indicates a segfault. so I am reinstalling mysql and wsrep etc and then attempting again.
The signal 11 typically indicates a segfault. so I am reinstalling mysql and wsrep etc and then attempting again.
Reply all
Reply to author
Forward
0 new messages