Nodes are getting shutdown
瀏覽次數：134 次
跳到第一則未讀訊息
Munazir
2015年6月30日 上午8:17:262015/6/30
收件者：codersh...@googlegroups.com
Hi,
I have 3 nodes multi-master cluster. My nodes are getting shutdown one after one with below error.
All 3 nodes getting error with sessions table.. below show table create for sessions
Please suggest what wrong it have so that we can solve this issue.
+----------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| sessions | CREATE TABLE `sessions` (
`id` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
`payload` longtext COLLATE utf8_unicode_ci NOT NULL,
`last_activity` int(11) NOT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `sessions_id_unique` (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci |
+----------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
Regards
Munazir
I have 3 nodes multi-master cluster. My nodes are getting shutdown one after one with below error.
*** Priority TRANSACTION:
TRANSACTION 839167, ACTIVE 0 sec inserting
mysql tables in use 1, locked 1
1 lock struct(s), heap size 360, 0 row lock(s)
MySQL thread id 2, OS thread handle 0x7f4aa8bc0700, query id 884596 Write_rows_log_event::write_row(370539)
*** Victim TRANSACTION:
TRANSACTION 839166, ACTIVE 0 sec, thread declared inside InnoDB 4545
mysql tables in use 1, locked 1
25 lock struct(s), heap size 2936, 473 row lock(s), undo log entries 6
MySQL thread id 65126, OS thread handle 0x7f4aa01c9700, query id 884594 10.60.32.81 dlv_admin query end
delete from `sessions` where `last_activity` <= '1435663956'
*** WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 100 page no 118 n bits 96 index `PRIMARY` of table `mol_dlvdb`.`sessions` trx id 839166 lock_mode X
Record lock, heap no 1 PHYSICAL RECORD: n_fields 1; compact format; info bits 0
0: len 8; hex 73757072656d756d; asc supremum;;
Record lock, heap no 2 PHYSICAL RECORD: n_fields 5; compact format; info bits 0
0: len 30; hex 346433306662346464356332373636313865393931373634386365323261; asc 4d30fb4dd5c276618e9917648ce22a; (total 40 bytes);
1: len 6; hex 0000000cc12d; asc -;;
2: len 7; hex 0b000002180bfe; asc ;;
3: len 30; hex 59546f344f6e747a4f6a5936496c39306232746c62694937637a6f304d44; asc YTo4OntzOjY6Il90b2tlbiI7czo0MD; (total 1132 bytes);
4: len 4; hex d5928098; asc ;;
Record lock, heap no 3 PHYSICAL RECORD: n_fields 5; compact format; info bits 0
0: len 30; hex 346433373237383732623665343166376430626533336336383065306636; asc 4d3727872b6e41f7d0be33c680e0f6; (total 40 bytes);
1: len 6; hex 0000000cbe05; asc ;;
2: len 7; hex 070000023b233e; asc ;#>;;
All 3 nodes getting error with sessions table.. below show table create for sessions
Please suggest what wrong it have so that we can solve this issue.
+----------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| sessions | CREATE TABLE `sessions` (
`id` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
`payload` longtext COLLATE utf8_unicode_ci NOT NULL,
`last_activity` int(11) NOT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `sessions_id_unique` (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci |
+----------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
Regards
Munazir
Philip Stoev
2015年6月30日 上午8:33:012015/6/30
收件者：Munazir、codersh...@googlegroups.com
Hello,
The log lines you provided would explain why individual transactions failed,
but not why the entire node would go down.
Can you please provide at least the 100 lines at the very end of the log?
Philip Stoev
--
You received this message because you are subscribed to the Google Groups
"codership" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to codership-tea...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
The log lines you provided would explain why individual transactions failed,
but not why the entire node would go down.
Can you please provide at least the 100 lines at the very end of the log?
Philip Stoev
You received this message because you are subscribed to the Google Groups
"codership" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to codership-tea...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Munazir
2015年6月30日 上午9:24:152015/6/30
收件者：codersh...@googlegroups.com、mdmu...@gmail.com
Hi Philips,
Here is logs, again one node get shutdown
*** Priority TRANSACTION:
TRANSACTION 867087, ACTIVE 0 sec starting index read
Here is logs, again one node get shutdown
*** Priority TRANSACTION:
TRANSACTION 867087, ACTIVE 0 sec starting index read
mysql tables in use 1, locked 1
1 lock struct(s), heap size 360, 0 row lock(s)
MySQL thread id 1, OS thread handle 0x7ff7603a2700, query id 49175 System lock
*** Victim TRANSACTION:
TRANSACTION 867082, ACTIVE 0 sec, thread declared inside InnoDB 4629
*** Victim TRANSACTION:
TRANSACTION 867082, ACTIVE 0 sec, thread declared inside InnoDB 4629
mysql tables in use 1, locked 1
19 lock struct(s), heap size 2936, 361 row lock(s), undo log entries 28
MySQL thread id 3081, OS thread handle 0x7ff7600e3700, query id 49173 10.60.32.81 dlv_admin query end
delete from `sessions` where `last_activity` <= '1435667077'
MySQL thread id 3081, OS thread handle 0x7ff7600e3700, query id 49173 10.60.32.81 dlv_admin query end
delete from `sessions` where `last_activity` <= '1435667077'
*** WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 100 page no 113 n bits 88 index `PRIMARY` of table `mol_dlvdb`.`sessions` trx id 867082 lock_mode X
Record lock, heap no 1 PHYSICAL RECORD: n_fields 1; compact format; info bits 0
0: len 8; hex 73757072656d756d; asc supremum;;
Record lock, heap no 3 PHYSICAL RECORD: n_fields 5; compact format; info bits 0
0: len 30; hex 383930653765316331663461666633633431393266333536626338343164; asc 890e7e1c1f4aff3c4192f356bc841d; (total 40 bytes);
1: len 6; hex 0000000d2aaa; asc * ;;
2: len 7; hex 97000001580110; asc X ;;
3: len 30; hex 59546f304f6e747a4f6a5936496c39306232746c62694937637a6f304d44; asc YTo0OntzOjY6Il90b2tlbiI7czo0MD; (total 352 bytes);
4: len 4; hex d5928bc6; asc ;;
Record lock, heap no 4 PHYSICAL RECORD: n_fields 5; compact format; info bits 0
0: len 30; hex 386133316435353531383630313432636363396632636332646666376331; asc 8a31d5551860142ccc9f2cc2dff7c1; (total 40 bytes);
1: len 6; hex 0000000d3779; asc 7y;;
2: len 7; hex 7000000212089b; asc p ;;
3: len 30; hex 59546f344f6e747a4f6a5936496c39306232746c62694937637a6f304d44; asc YTo4OntzOjY6Il90b2tlbiI7czo0MD; (total 1176 bytes);
4: len 4; hex d5928d74; asc t;;
Record lock, heap no 5 PHYSICAL RECORD: n_fields 5; compact format; info bits 0
0: len 30; hex 386261633166323266636561386231313331333838363032343162353136; asc 8bac1f22fcea8b113138860241b516; (total 40 bytes);
1: len 6; hex 0000000d36f7; asc 6 ;;
2: len 7; hex b2000001600110; asc ` ;;
3: len 30; hex 59546f304f6e747a4f6a5936496c39306232746c62694937637a6f304d44; asc YTo0OntzOjY6Il90b2tlbiI7czo0MD; (total 352 bytes);
4: len 4; hex d5928d5a; asc Z;;
Record lock, heap no 7 PHYSICAL RECORD: n_fields 5; compact format; info bits 0
0: len 30; hex 386434636366373534363764346139393639663935356531393662373836; asc 8d4ccf75467d4a9969f955e196b786; (total 40 bytes);
1: len 6; hex 0000000d2d9a; asc - ;;
2: len 7; hex f4000001410110; asc A ;;
3: len 30; hex 59546f304f6e747a4f6a5936496c39306232746c62694937637a6f304d44; asc YTo0OntzOjY6Il90b2tlbiI7czo0MD; (total 352 bytes);
4: len 4; hex d5928c3b; asc ;;;
Record lock, heap no 8 PHYSICAL RECORD: n_fields 5; compact format; info bits 0
0: len 30; hex 386561313836343333373336316535333233373130643739353739366636; asc 8ea1864337361e5323710d795796f6; (total 40 bytes);
1: len 6; hex 0000000d261d; asc & ;;
2: len 7; hex 36000002270e6e; asc 6 ' n;;
3: len 30; hex 59546f304f6e747a4f6a5936496c39306232746c62694937637a6f304d44; asc YTo0OntzOjY6Il90b2tlbiI7czo0MD; (total 372 bytes);
4: len 4; hex d5928b2c; asc ,;;
Record lock, heap no 9 PHYSICAL RECORD: n_fields 5; compact format; info bits 0
0: len 30; hex 386564336163313333356361643666336334626239353166656165373333; asc 8ed3ac1335cad6f3c4bb951feae733; (total 40 bytes);
1: len 6; hex 0000000d24e2; asc $ ;;
2: len 7; hex b7000001c60110; asc ;;
3: len 30; hex 59546f304f6e747a4f6a5936496c39306232746c62694937637a6f304d44; asc YTo0OntzOjY6Il90b2tlbiI7czo0MD; (total 352 bytes);
4: len 4; hex d5928b08; asc ;;
Record lock, heap no 10 PHYSICAL RECORD: n_fields 5; compact format; info bits 0
0: len 30; hex 386635386236316635666435643331343236613532356366393639646431; asc 8f58b61f5fd5d31426a525cf969dd1; (total 40 bytes);
1: len 6; hex 0000000d2836; asc (6;;
2: len 7; hex 90000001530110; asc S ;;
3: len 30; hex 59546f304f6e747a4f6a5936496c39306232746c62694937637a6f304d44; asc YTo0OntzOjY6Il90b2tlbiI7czo0MD; (total 352 bytes);
4: len 4; hex d5928b7d; asc };;
Record lock, heap no 12 PHYSICAL RECORD: n_fields 5; compact format; info bits 0
0: len 30; hex 393061643734323933316165343533343764666364303430363061376437; asc 90ad742931ae45347dfcd04060a7d7; (total 40 bytes);
1: len 6; hex 0000000d2334; asc #4;;
2: len 7; hex 7d0000023d0c8c; asc } = ;;
3: len 30; hex 59546f314f6e747a4f6a5936496c39306232746c62694937637a6f304d44; asc YTo1OntzOjY6Il90b2tlbiI7czo0MD; (total 960 bytes);
4: len 4; hex d5928ad5; asc ;;
Record lock, heap no 14 PHYSICAL RECORD: n_fields 5; compact format; info bits 0
0: len 30; hex 393138393837393830333534646234383166343833643961363632633065; asc 918987980354db481f483d9a662c0e; (total 40 bytes);
1: len 6; hex 0000000d34cc; asc 4 ;;
2: len 7; hex ae0000015e0110; asc ^ ;;
3: len 30; hex 59546f304f6e747a4f6a5936496c39306232746c62694937637a6f304d44; asc YTo0OntzOjY6Il90b2tlbiI7czo0MD; (total 352 bytes);
4: len 4; hex d5928d0a; asc ;;
Record lock, heap no 15 PHYSICAL RECORD: n_fields 5; compact format; info bits 0
0: len 30; hex 393231363063663131353739343233353461663763663566663831343761; asc 92160cf1157942354af7cf5ff8147a; (total 40 bytes);
1: len 6; hex 0000000d27bf; asc ' ;;
2: len 7; hex e2000001880110; asc ;;
3: len 30; hex 59546f314f6e747a4f6a5936496c39306232746c62694937637a6f304d44; asc YTo1OntzOjY6Il90b2tlbiI7czo0MD; (total 1052 bytes);
4: len 4; hex d5928b69; asc i;;
Record lock, heap no 18 PHYSICAL RECORD: n_fields 5; compact format; info bits 0
0: len 30; hex 393036363864306332303739346465396638323063363664373761376436; asc 90668d0c20794de9f820c66d77a7d6; (total 40 bytes);
1: len 6; hex 0000000d395d; asc 9];;
2: len 7; hex 3e000001e80535; asc > 5;;
2015-06-30 15:39:37 4205 [Note] WSREP: cluster conflict due to high priority abort for threads:
2015-06-30 15:39:37 4205 [Note] WSREP: Winning thread:
THD: 1, mode: applier, state: executing, conflict: no conflict, seqno: 380188
SQL: (null)
2015-06-30 15:39:37 4205 [Note] WSREP: Victim thread:
THD: 3081, mode: local, state: committing, conflict: no conflict, seqno: 380189
SQL: delete from `sessions` where `last_activity` <= '1435667077'
2015-06-30 15:39:37 4205 [Note] WSREP: BF kill (1, seqno: 380188), victim: (3081) trx: 867082
2015-06-30 15:39:37 4205 [Note] WSREP: Aborting query: delete from `sessions` where `last_activity` <= '1435667077'
2015-06-30 15:39:37 4205 [Note] WSREP: kill trx QUERY_COMMITTING for 867082
2015-06-30 15:39:37 4205 [Note] WSREP: thd 3081 seqno 380189 BF aborted by provider, will replay
2015-06-30 15:39:37 4205 [Note] WSREP: replaying increased: 1, thd: 3081
2015-06-30 15:39:37 4205 [Note] WSREP: commit failed for reason: 4 3081 delete from `sessions` where `last_activity` <= '1435667077'
2015-06-30 15:39:37 4205 [Note] WSREP: conflict state: 4
2015-06-30 15:39:37 4205 [Note] WSREP: cleanup transaction for LOCAL_STATE: delete from `sessions` where `last_activity` <= '1435667077'
2015-06-30 15:39:37 4205 [Note] WSREP: replay trx: delete from `sessions` where `last_activity` <= '1435667077' -1
2015-06-30 15:39:37 4205 [Warning] WSREP: BF applier failed to open_and_lock_tables: 1615, fatal: 0 wsrep = (exec_mode: 1 conflict_state: 5 seqno: 380189)
2015-06-30 15:39:37 4205 [Warning] WSREP: RBR event 3 Delete_rows apply warning: 1615, 380189
2015-06-30 15:39:37 4205 [Warning] WSREP: Failed to apply app buffer: seqno: 380189, status: 1
at galera/src/trx_handle.cpp:apply():351
Retrying 2th time
2015-06-30 15:39:37 4205 [Warning] WSREP: BF applier failed to open_and_lock_tables: 1615, fatal: 0 wsrep = (exec_mode: 1 conflict_state: 5 seqno: 380189)
2015-06-30 15:39:37 4205 [Warning] WSREP: RBR event 3 Delete_rows apply warning: 1615, 380189
2015-06-30 15:39:37 4205 [Warning] WSREP: Failed to apply app buffer: seqno: 380189, status: 1
at galera/src/trx_handle.cpp:apply():351
Retrying 3th time
2015-06-30 15:39:37 4205 [Warning] WSREP: BF applier failed to open_and_lock_tables: 1615, fatal: 0 wsrep = (exec_mode: 1 conflict_state: 5 seqno: 380189)
2015-06-30 15:39:37 4205 [Warning] WSREP: RBR event 3 Delete_rows apply warning: 1615, 380189
2015-06-30 15:39:37 4205 [Warning] WSREP: Failed to apply app buffer: seqno: 380189, status: 1
at galera/src/trx_handle.cpp:apply():351
Retrying 4th time
2015-06-30 15:39:37 4205 [Warning] WSREP: BF applier failed to open_and_lock_tables: 1615, fatal: 0 wsrep = (exec_mode: 1 conflict_state: 5 seqno: 380189)
2015-06-30 15:39:37 4205 [Warning] WSREP: RBR event 3 Delete_rows apply warning: 1615, 380189
2015-06-30 15:39:37 4205 [Warning] WSREP: failed to replay trx: source: d3a77cee-1f1e-11e5-89a4-ab6ae97f3cf8 version: 3 local: 1 state: REPLAYING flags: 1 conn_id: 3081 trx_id: 867082 seqnos (l: 8201, g: 380189, s: 380187, d: 380121, ts: 1041510854992104)
2015-06-30 15:39:37 4205 [Warning] WSREP: Failed to apply trx 380189 4 times
2015-06-30 15:39:37 4205 [ERROR] WSREP: trx_replay failed for: 6, query: void
2015-06-30 15:39:37 4205 [ERROR] Aborting
2015-06-30 15:39:39 4205 [Note] WSREP: waiting for client connections to close: 3
2015-06-30 15:39:39 4205 [Note] WSREP: Closing send monitor...
2015-06-30 15:39:39 4205 [Note] WSREP: Closed send monitor.
2015-06-30 15:39:39 4205 [Note] WSREP: gcomm: terminating thread
2015-06-30 15:39:39 4205 [Note] WSREP: gcomm: joining thread
2015-06-30 15:39:39 4205 [Note] WSREP: gcomm: closing backend
2015-06-30 15:39:39 4205 [Note] WSREP: view(view_id(NON_PRIM,77ca548c,101) memb {
d3a77cee,0
} joined {
} left {
} partitioned {
77ca548c,0
929d019a,0
})
2015-06-30 15:39:39 4205 [Note] WSREP: view((empty))
2015-06-30 15:39:39 4205 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
2015-06-30 15:39:39 4205 [Note] WSREP: gcomm: closed
2015-06-30 15:39:39 4205 [Note] WSREP: Flow-control interval: [16, 16]
2015-06-30 15:39:39 4205 [Note] WSREP: Received NON-PRIMARY.
2015-06-30 15:39:39 4205 [Note] WSREP: Shifting SYNCED -> OPEN (TO: 380197)
2015-06-30 15:39:39 4205 [Note] WSREP: Received self-leave message.
2015-06-30 15:39:39 4205 [Note] WSREP: Flow-control interval: [0, 0]
2015-06-30 15:39:39 4205 [Note] WSREP: Received SELF-LEAVE. Closing connection.
2015-06-30 15:39:39 4205 [Note] WSREP: Shifting OPEN -> CLOSED (TO: 380197)
2015-06-30 15:39:39 4205 [Note] WSREP: RECV thread exiting 0: Success
2015-06-30 15:39:39 4205 [Note] WSREP: recv_thread() joined.
2015-06-30 15:39:39 4205 [Note] WSREP: Closing replication queue.
2015-06-30 15:39:39 4205 [Note] WSREP: Closing slave action queue.
2015-06-30 15:39:39 4205 [Note] WSREP: Service disconnected.
2015-06-30 15:39:39 4205 [Note] WSREP: closing wsrep thread 1
2015-06-30 15:39:39 4205 [Note] WSREP: closing wsrep thread 2
2015-06-30 15:39:39 4205 [Note] WSREP: WSREP rollback thread wakes for signal
2015-06-30 15:39:39 4205 [Note] WSREP: WSREP rollback thread has empty abort queue
2015-06-30 15:39:39 4205 [Note] WSREP: rollbacker thread exiting
2015-06-30 15:39:39 4205 [Note] WSREP: avoiding thread re-use for applier, thd: 2
2015-06-30 15:39:39 4205 [Note] WSREP: cleanup transaction for LOCAL_STATE: select * from `sessions` where `id` = 'af42b669b6da8d2734f8e174895cced7847b7551' limit 1
2015-06-30 15:39:39 4205 [Note] WSREP: cleanup transaction for LOCAL_STATE: select * from `users` where `users`.`deleted_at` is null and `id_number` = '1039467251' limit 1
2015-06-30 15:39:39 4205 [Note] WSREP: cleanup transaction for LOCAL_STATE: select count(*) as aggregate from `visa_requests` where `visa_requests`.`deleted_at` is null and `user_id` = '7915'
2015-06-30 15:39:39 4205 [Note] WSREP: cleanup transaction for LOCAL_STATE: select * from `visa_requests` where `visa_requests`.`deleted_at` is null and `user_id` = '7915' limit 15 offset 0
2015-06-30 15:39:39 4205 [Note] WSREP: commit failed for reason: 3 3083 update `sessions` set `payload` = 'YTo4OntzOjY6Il90b2tlbiI7czo0MDoiaGFKNk5rZE1RdTI4SkFvaU9QZlpvVEd6WUVYZzFxckt0TDNLdVVwaSI7czo5OiJfcHJldmlvdXMiO2E6MTp7czozOiJ1cmwiO3M6NDA6Imh0dHA6Ly92aXNhLm11c2FuZWQuZ292LnNhL3Zpc2FfcmVxdWVzdHMiO31zOjU6ImZsYXNoIjthOjI6e3M6Mzoib2xkIjthOjA6e31zOjM6Im5ldyI7YTowOnt9fXM6MTI6InJlZ2lzdHJhdGlvbiI7YTozOntzOjQ6ImRhdGEiO2E6Njp7czo2OiJtb2JpbGUiO3M6MTA6IjA1NTU1ODg5NjgiO3M6NToiZW1haWwiO3M6MjA6InNtb3dzZG4xMTFAZ21haWwuY29tIjtzOjk6ImlkX251bWJlciI7czoxMDoiMTAzOTQ2NzI1MSI7czo4OiJwYXNzd29yZCI7czoxMToiU21vNDg3NE1vaEAiO3M6NDoibmFtZSI7czo0NToi2YXYrdmF2K8g2LnZhNmJINio2YYg2K/ZhNmK2YUg2KLZhCDYstmK2K/Yp9mGIjtzOjEwOiJiaXJ0aF9kYXRlIjtzOjEwOiIxNDA0LTA5LTIyIjt9czo0OiJjb2RlIjtzOjY6IjAxOTUzNCI7czoxNToiY29kZV9leHBpcmF0aW9uIjtPOjEzOiJDYXJib25cQ2FyYm9uIjozOntzOjQ6ImRhdGUiO3M6MjY6IjIwMTUtMDYtMzAgMTU6NDM6MzMuMDAwMDAwIjtzOjEzOiJ0aW1lem9uZV90eXBlIjtpOjM7czo4OiJ0aW1lem9uZSI7czoxMToiQXNpYS9SaXlhZGgiO319czo3OiJtZXNzYWdlIjthOjA6e31zOjM4OiJsb2dpbl84MmU1ZDJjNTZiZGQwODExMzE4ZjBjZjA3
2015-06-30 15:39:39 4205 [Note] WSREP: conflict state: 0
2015-06-30 15:39:39 4205 [Note] WSREP: cluster conflict due to certification failure for threads:
2015-06-30 15:39:39 4205 [Note] WSREP: Victim thread:
THD: 3083, mode: local, state: executing, conflict: cert failure, seqno: -1
SQL: update `sessions` set `payload` = 'YTo4OntzOjY6Il90b2tlbiI7czo0MDoiaGFKNk5rZE1RdTI4SkFvaU9QZlpvVEd6WUVYZzFxckt0TDNLdVVwaSI7czo5OiJfcHJldmlvdXMiO2E6MTp7czozOiJ1cmwiO3M6NDA6Imh0dHA6Ly92aXNhLm11c2FuZWQuZ292LnNhL3Zpc2FfcmVxdWVzdHMiO31zOjU6ImZsYXNoIjthOjI6e3M6Mzoib2xkIjthOjA6e31zOjM6Im5ldyI7YTowOnt9fXM6MTI6InJlZ2lzdHJhdGlvbiI7YTozOntzOjQ6ImRhdGEiO2E6Njp7czo2OiJtb2JpbGUiO3M6MTA6IjA1NTU1ODg5NjgiO3M6NToiZW1haWwiO3M6MjA6InNtb3dzZG4xMTFAZ21haWwuY29tIjtzOjk6ImlkX251bWJlciI7czoxMDoiMTAzOTQ2NzI1MSI7czo4OiJwYXNzd29yZCI7czoxMToiU21vNDg3NE1vaEAiO3M6NDoibmFtZSI7czo0NToi2YXYrdmF2K8g2LnZhNmJINio2YYg2K/ZhNmK2YUg2KLZhCDYstmK2K/Yp9mGIjtzOjEwOiJiaXJ0aF9kYXRlIjtzOjEwOiIxNDA0LTA5LTIyIjt9czo0OiJjb2RlIjtzOjY6IjAxOTUzNCI7czoxNToiY29kZV9leHBpcmF0aW9uIjtPOjEzOiJDYXJib25cQ2FyYm9uIjozOntzOjQ6ImRhdGUiO3M6MjY6IjIwMTUtMDYtMzAgMTU6NDM6MzMuMDAwMDAwIjtzOjEzOiJ0aW1lem9uZV90eXBlIjtpOjM7czo4OiJ0aW1lem9uZSI7czoxMToiQXNpYS9SaXlhZGgiO319czo3Oi
2015-06-30 15:39:39 4205 [Note] WSREP: cleanup transaction for LOCAL_STATE: update `sessions` set `payload` = 'YTo4OntzOjY6Il90b2tlbiI7czo0MDoiaGFKNk5rZE1RdTI4SkFvaU9QZlpvVEd6WUVYZzFxckt0TDNLdVVwaSI7czo5OiJfcHJldmlvdXMiO2E6MTp7czozOiJ1cmwiO3M6NDA6Imh0dHA6Ly92aXNhLm11c2FuZWQuZ292LnNhL3Zpc2FfcmVxdWVzdHMiO31zOjU6ImZsYXNoIjthOjI6e3M6Mzoib2xkIjthOjA6e31zOjM6Im5ldyI7YTowOnt9fXM6MTI6InJlZ2lzdHJhdGlvbiI7YTozOntzOjQ6ImRhdGEiO2E6Njp7czo2OiJtb2JpbGUiO3M6MTA6IjA1NTU1ODg5NjgiO3M6NToiZW1haWwiO3M6MjA6InNtb3dzZG4xMTFAZ21haWwuY29tIjtzOjk6ImlkX251bWJlciI7czoxMDoiMTAzOTQ2NzI1MSI7czo4OiJwYXNzd29yZCI7czoxMToiU21vNDg3NE1vaEAiO3M6NDoibmFtZSI7czo0NToi2YXYrdmF2K8g2LnZhNmJINio2YYg2K/ZhNmK2YUg2KLZhCDYstmK2K/Yp9mGIjtzOjEwOiJiaXJ0aF9kYXRlIjtzOjEwOiIxNDA0LTA5LTIyIjt9czo0OiJjb2RlIjtzOjY6IjAxOTUzNCI7czoxNToiY29kZV9leHBpcmF0aW9uIjtPOjEzOiJDYXJib25cQ2FyYm9uIjozOntzOjQ6ImRhdGUiO3M6MjY6IjIwMTUtMDYtMzAgMTU6NDM6MzMuMDAwMDAwIjtzOjEzOiJ0aW1lem9uZV90eXBlIjtpOjM7czo4OiJ0aW1lem9uZSI7czoxMToiQXNpYS9SaXlhZGgiO319czo3OiJtZXNzYWdlIjthOjA6e31zOjM4OiJsb2dpbl84MmU1ZDJjNTZiZGQwODExMzE4ZjBj
2015-06-30 15:39:39 4205 [Note] WSREP: PS execute fail for CERT_FAILURE: thd: 3083 err: 1213
2015-06-30 15:39:40 4205 [Note] WSREP: Some threads may fail to exit.
2015-06-30 15:39:40 4205 [Note] Binlog end
2015-06-30 15:39:40 4205 [Note] Shutting down plugin 'partition'
2015-06-30 15:39:40 4205 [Note] Shutting down plugin 'BLACKHOLE'
2015-06-30 15:39:40 4205 [Note] Shutting down plugin 'ARCHIVE'
2015-06-30 15:39:40 4205 [Note] Shutting down plugin 'PERFORMANCE_SCHEMA'
2015-06-30 15:39:40 4205 [Note] Shutting down plugin 'INNODB_SYS_DATAFILES'
2015-06-30 15:39:40 4205 [Note] Shutting down plugin 'INNODB_SYS_TABLESPACES'
2015-06-30 15:39:40 4205 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN_COLS'
2015-06-30 15:39:40 4205 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN'
2015-06-30 15:39:40 4205 [Note] Shutting down plugin 'INNODB_SYS_FIELDS'
2015-06-30 15:39:40 4205 [Note] Shutting down plugin 'INNODB_SYS_COLUMNS'
2015-06-30 15:39:40 4205 [Note] Shutting down plugin 'INNODB_SYS_INDEXES'
2015-06-30 15:39:40 4205 [Note] Shutting down plugin 'INNODB_SYS_TABLESTATS'
2015-06-30 15:39:40 4205 [Note] Shutting down plugin 'INNODB_SYS_TABLES'
2015-06-30 15:39:40 4205 [Note] Shutting down plugin 'INNODB_FT_INDEX_TABLE'
2015-06-30 15:39:40 4205 [Note] Shutting down plugin 'INNODB_FT_INDEX_CACHE'
2015-06-30 15:39:40 4205 [Note] Shutting down plugin 'INNODB_FT_CONFIG'
2015-06-30 15:39:40 4205 [Note] Shutting down plugin 'INNODB_FT_BEING_DELETED'
2015-06-30 15:39:40 4205 [Note] Shutting down plugin 'INNODB_FT_DELETED'
2015-06-30 15:39:40 4205 [Note] Shutting down plugin 'INNODB_FT_DEFAULT_STOPWORD'
2015-06-30 15:39:40 4205 [Note] Shutting down plugin 'INNODB_METRICS'
2015-06-30 15:39:40 4205 [Note] Shutting down plugin 'INNODB_BUFFER_POOL_STATS'
2015-06-30 15:39:40 4205 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE_LRU'
2015-06-30 15:39:40 4205 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE'
2015-06-30 15:39:40 4205 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX_RESET'
2015-06-30 15:39:40 4205 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX'
2015-06-30 15:39:40 4205 [Note] Shutting down plugin 'INNODB_CMPMEM_RESET'
2015-06-30 15:39:40 4205 [Note] Shutting down plugin 'INNODB_CMPMEM'
2015-06-30 15:39:40 4205 [Note] Shutting down plugin 'INNODB_CMP_RESET'
2015-06-30 15:39:40 4205 [Note] Shutting down plugin 'INNODB_CMP'
2015-06-30 15:39:40 4205 [Note] Shutting down plugin 'INNODB_LOCK_WAITS'
2015-06-30 15:39:40 4205 [Note] Shutting down plugin 'INNODB_LOCKS'
2015-06-30 15:39:40 4205 [Note] Shutting down plugin 'INNODB_TRX'
2015-06-30 15:39:40 4205 [Note] Shutting down plugin 'InnoDB'
2015-06-30 15:39:40 4205 [Note] InnoDB: FTS optimize thread exiting.
2015-06-30 15:39:40 4205 [Note] InnoDB: Starting shutdown...
12:39:42 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
key_buffer_size=268435456
read_buffer_size=104857600
max_used_connections=4
max_threads=500
thread_count=3
connection_count=3
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 51725237 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x900ba50
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7ff760de3df8 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x35)[0x8fc4a5]
/usr/sbin/mysqld(handle_fatal_signal+0x494)[0x67d854]
/lib64/libpthread.so.0[0x391ae0f710]
/usr/sbin/mysqld[0x69bd9b]
/usr/sbin/mysqld[0x69cace]
/usr/sbin/mysqld[0x69cdb6]
/usr/sbin/mysqld[0x68f874]
/usr/sbin/mysqld(_Z16acl_authenticateP3THDj+0x1f7)[0x6a87b7]
/usr/sbin/mysqld[0x6cc66a]
/usr/sbin/mysqld(_Z16login_connectionP3THD+0x45)[0x6cc895]
/usr/sbin/mysqld(_Z22thd_prepare_connectionP3THD+0x24)[0x6cc924]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x139)[0x6cd619]
/usr/sbin/mysqld(handle_one_connection+0x47)[0x6cd7d7]
/usr/sbin/mysqld(pfs_spawn_thread+0x12a)[0xb1fe7a]
/lib64/libpthread.so.0[0x391ae079d1]
/lib64/libc.so.6(clone+0x6d)[0x391aae88fd]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0): is an invalid pointer
Connection ID (thread ID): 3085
Status: NOT_KILLED
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
150630 15:39:42 mysqld_safe Number of processes running now: 0
150630 15:39:42 mysqld_safe WSREP: not restarting wsrep node automatically
150630 15:39:42 mysqld_safe mysqld from pid file /var/lib/mysql/mysql.pid ended
150630 16:08:10 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150630 16:08:10 mysqld_safe WSREP: Running position recovery with --log_error='/var/lib/mysql/wsrep_recovery.NOB7qO' --pid-file='/var/lib/mysql/MOL-DLV-MySQLND3-recover.pid'
2015-06-30 16:08:10 0 [Warning] Using unique option prefix key_buffer instead of key_buffer_size is deprecated and will be removed in a future release. Please use the full name instead.
2015-06-30 16:08:10 0 [Warning] Using unique option prefix thread_cache instead of thread_cache_size is deprecated and will be removed in a future release. Please use the full name instead.
2015-06-30 16:08:10 0 [Warning] option 'thread_cache_size': unsigned value 268435456 adjusted to 16384
2015-06-30 16:08:10 0 [Warning] 'THREAD_CONCURRENCY' is deprecated and will be removed in a future release.
2015-06-30 16:08:10 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
150630 16:08:13 mysqld_safe WSREP: Recovered position c8e1f7ef-0ce4-11e5-89a6-f3af952dc915:380188
2015-06-30 16:08:13 0 [Warning] Using unique option prefix key_buffer instead of key_buffer_size is deprecated and will be removed in a future release. Please use the full name instead.
2015-06-30 16:08:13 0 [Warning] Using unique option prefix thread_cache instead of thread_cache_size is deprecated and will be removed in a future release. Please use the full name instead.
2015-06-30 16:08:13 16955 [Note] WSREP: CRC-32C: using hardware acceleration.
2015-06-30 16:08:13 16955 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1
2015-06-30 16:08:13 16955 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 10.60.32.85; base_port = 4567; cert.log_conflicts = no; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 1G; gcache.size = 300M; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false; pc
2015-06-30 16:08:13 16955 [Note] WSREP: Service thread queue flushed.
2015-06-30 16:08:13 16955 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1
2015-06-30 16:08:13 16955 [Note] WSREP: wsrep_sst_grab()
2015-06-30 16:08:13 16955 [Note] WSREP: Start replication
2015-06-30 16:08:13 16955 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1
2015-06-30 16:08:13 16955 [Note] WSREP: protonet asio version 0
2015-06-30 16:08:13 16955 [Note] WSREP: Using CRC-32C for message checksums.
2015-06-30 16:08:13 16955 [Note] WSREP: backend: asio
2015-06-30 16:08:13 16955 [Warning] WSREP: access file(/var/lib/mysql//gvwstate.dat) failed(No such file or directory)
2015-06-30 16:08:13 16955 [Note] WSREP: restore pc from disk failed
2015-06-30 16:08:13 16955 [Note] WSREP: GMCast version 0
2015-06-30 16:08:13 16955 [Note] WSREP: (10bd9501, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
2015-06-30 16:08:13 16955 [Note] WSREP: (10bd9501, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
2015-06-30 16:08:13 16955 [Note] WSREP: EVS version 0
2015-06-30 16:08:13 16955 [Note] WSREP: gcomm: connecting to group 'MOL-DLV-GCMySQL', peer '10.60.32.83:,10.60.32.84:,10.60.32.85:'
2015-06-30 16:08:13 16955 [Warning] WSREP: (10bd9501, 'tcp://0.0.0.0:4567') address 'tcp://10.60.32.85:4567' points to own listening address, blacklisting
2015-06-30 16:08:13 16955 [Note] WSREP: (10bd9501, 'tcp://0.0.0.0:4567') address 'tcp://10.60.32.85:4567' pointing to uuid 10bd9501 is blacklisted, skipping
2015-06-30 16:08:13 16955 [Note] WSREP: (10bd9501, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:
2015-06-30 16:08:13 16955 [Note] WSREP: (10bd9501, 'tcp://0.0.0.0:4567') address 'tcp://10.60.32.85:4567' pointing to uuid 10bd9501 is blacklisted, skipping
Just now i restart mysql.
From last 6 hours all 3 nodes are shutdown for almost 3 times each one by one. One time node 1 shutdown, one time node 3 and one time node 2.
Now 3 times node 3 shutdown.
Regards
Munazir
1: len 6; hex 0000000d2aaa; asc * ;;
2: len 7; hex 97000001580110; asc X ;;
3: len 30; hex 59546f304f6e747a4f6a5936496c39306232746c62694937637a6f304d44; asc YTo0OntzOjY6Il90b2tlbiI7czo0MD; (total 352 bytes);
4: len 4; hex d5928bc6; asc ;;
Record lock, heap no 4 PHYSICAL RECORD: n_fields 5; compact format; info bits 0
0: len 30; hex 386133316435353531383630313432636363396632636332646666376331; asc 8a31d5551860142ccc9f2cc2dff7c1; (total 40 bytes);
1: len 6; hex 0000000d3779; asc 7y;;
2: len 7; hex 7000000212089b; asc p ;;
3: len 30; hex 59546f344f6e747a4f6a5936496c39306232746c62694937637a6f304d44; asc YTo4OntzOjY6Il90b2tlbiI7czo0MD; (total 1176 bytes);
4: len 4; hex d5928d74; asc t;;
Record lock, heap no 5 PHYSICAL RECORD: n_fields 5; compact format; info bits 0
0: len 30; hex 386261633166323266636561386231313331333838363032343162353136; asc 8bac1f22fcea8b113138860241b516; (total 40 bytes);
1: len 6; hex 0000000d36f7; asc 6 ;;
2: len 7; hex b2000001600110; asc ` ;;
3: len 30; hex 59546f304f6e747a4f6a5936496c39306232746c62694937637a6f304d44; asc YTo0OntzOjY6Il90b2tlbiI7czo0MD; (total 352 bytes);
4: len 4; hex d5928d5a; asc Z;;
Record lock, heap no 7 PHYSICAL RECORD: n_fields 5; compact format; info bits 0
0: len 30; hex 386434636366373534363764346139393639663935356531393662373836; asc 8d4ccf75467d4a9969f955e196b786; (total 40 bytes);
1: len 6; hex 0000000d2d9a; asc - ;;
2: len 7; hex f4000001410110; asc A ;;
3: len 30; hex 59546f304f6e747a4f6a5936496c39306232746c62694937637a6f304d44; asc YTo0OntzOjY6Il90b2tlbiI7czo0MD; (total 352 bytes);
4: len 4; hex d5928c3b; asc ;;;
Record lock, heap no 8 PHYSICAL RECORD: n_fields 5; compact format; info bits 0
0: len 30; hex 386561313836343333373336316535333233373130643739353739366636; asc 8ea1864337361e5323710d795796f6; (total 40 bytes);
1: len 6; hex 0000000d261d; asc & ;;
2: len 7; hex 36000002270e6e; asc 6 ' n;;
3: len 30; hex 59546f304f6e747a4f6a5936496c39306232746c62694937637a6f304d44; asc YTo0OntzOjY6Il90b2tlbiI7czo0MD; (total 372 bytes);
4: len 4; hex d5928b2c; asc ,;;
Record lock, heap no 9 PHYSICAL RECORD: n_fields 5; compact format; info bits 0
0: len 30; hex 386564336163313333356361643666336334626239353166656165373333; asc 8ed3ac1335cad6f3c4bb951feae733; (total 40 bytes);
1: len 6; hex 0000000d24e2; asc $ ;;
2: len 7; hex b7000001c60110; asc ;;
3: len 30; hex 59546f304f6e747a4f6a5936496c39306232746c62694937637a6f304d44; asc YTo0OntzOjY6Il90b2tlbiI7czo0MD; (total 352 bytes);
4: len 4; hex d5928b08; asc ;;
Record lock, heap no 10 PHYSICAL RECORD: n_fields 5; compact format; info bits 0
0: len 30; hex 386635386236316635666435643331343236613532356366393639646431; asc 8f58b61f5fd5d31426a525cf969dd1; (total 40 bytes);
1: len 6; hex 0000000d2836; asc (6;;
2: len 7; hex 90000001530110; asc S ;;
3: len 30; hex 59546f304f6e747a4f6a5936496c39306232746c62694937637a6f304d44; asc YTo0OntzOjY6Il90b2tlbiI7czo0MD; (total 352 bytes);
4: len 4; hex d5928b7d; asc };;
Record lock, heap no 12 PHYSICAL RECORD: n_fields 5; compact format; info bits 0
0: len 30; hex 393061643734323933316165343533343764666364303430363061376437; asc 90ad742931ae45347dfcd04060a7d7; (total 40 bytes);
1: len 6; hex 0000000d2334; asc #4;;
2: len 7; hex 7d0000023d0c8c; asc } = ;;
3: len 30; hex 59546f314f6e747a4f6a5936496c39306232746c62694937637a6f304d44; asc YTo1OntzOjY6Il90b2tlbiI7czo0MD; (total 960 bytes);
4: len 4; hex d5928ad5; asc ;;
Record lock, heap no 14 PHYSICAL RECORD: n_fields 5; compact format; info bits 0
0: len 30; hex 393138393837393830333534646234383166343833643961363632633065; asc 918987980354db481f483d9a662c0e; (total 40 bytes);
1: len 6; hex 0000000d34cc; asc 4 ;;
2: len 7; hex ae0000015e0110; asc ^ ;;
3: len 30; hex 59546f304f6e747a4f6a5936496c39306232746c62694937637a6f304d44; asc YTo0OntzOjY6Il90b2tlbiI7czo0MD; (total 352 bytes);
4: len 4; hex d5928d0a; asc ;;
Record lock, heap no 15 PHYSICAL RECORD: n_fields 5; compact format; info bits 0
0: len 30; hex 393231363063663131353739343233353461663763663566663831343761; asc 92160cf1157942354af7cf5ff8147a; (total 40 bytes);
1: len 6; hex 0000000d27bf; asc ' ;;
2: len 7; hex e2000001880110; asc ;;
3: len 30; hex 59546f314f6e747a4f6a5936496c39306232746c62694937637a6f304d44; asc YTo1OntzOjY6Il90b2tlbiI7czo0MD; (total 1052 bytes);
4: len 4; hex d5928b69; asc i;;
Record lock, heap no 18 PHYSICAL RECORD: n_fields 5; compact format; info bits 0
0: len 30; hex 393036363864306332303739346465396638323063363664373761376436; asc 90668d0c20794de9f820c66d77a7d6; (total 40 bytes);
1: len 6; hex 0000000d395d; asc 9];;
2: len 7; hex 3e000001e80535; asc > 5;;
2015-06-30 15:39:37 4205 [Note] WSREP: cluster conflict due to high priority abort for threads:
2015-06-30 15:39:37 4205 [Note] WSREP: Winning thread:
THD: 1, mode: applier, state: executing, conflict: no conflict, seqno: 380188
SQL: (null)
2015-06-30 15:39:37 4205 [Note] WSREP: Victim thread:
THD: 3081, mode: local, state: committing, conflict: no conflict, seqno: 380189
SQL: delete from `sessions` where `last_activity` <= '1435667077'
2015-06-30 15:39:37 4205 [Note] WSREP: BF kill (1, seqno: 380188), victim: (3081) trx: 867082
2015-06-30 15:39:37 4205 [Note] WSREP: Aborting query: delete from `sessions` where `last_activity` <= '1435667077'
2015-06-30 15:39:37 4205 [Note] WSREP: kill trx QUERY_COMMITTING for 867082
2015-06-30 15:39:37 4205 [Note] WSREP: thd 3081 seqno 380189 BF aborted by provider, will replay
2015-06-30 15:39:37 4205 [Note] WSREP: replaying increased: 1, thd: 3081
2015-06-30 15:39:37 4205 [Note] WSREP: commit failed for reason: 4 3081 delete from `sessions` where `last_activity` <= '1435667077'
2015-06-30 15:39:37 4205 [Note] WSREP: conflict state: 4
2015-06-30 15:39:37 4205 [Note] WSREP: cleanup transaction for LOCAL_STATE: delete from `sessions` where `last_activity` <= '1435667077'
2015-06-30 15:39:37 4205 [Note] WSREP: replay trx: delete from `sessions` where `last_activity` <= '1435667077' -1
2015-06-30 15:39:37 4205 [Warning] WSREP: BF applier failed to open_and_lock_tables: 1615, fatal: 0 wsrep = (exec_mode: 1 conflict_state: 5 seqno: 380189)
2015-06-30 15:39:37 4205 [Warning] WSREP: RBR event 3 Delete_rows apply warning: 1615, 380189
2015-06-30 15:39:37 4205 [Warning] WSREP: Failed to apply app buffer: seqno: 380189, status: 1
at galera/src/trx_handle.cpp:apply():351
Retrying 2th time
2015-06-30 15:39:37 4205 [Warning] WSREP: BF applier failed to open_and_lock_tables: 1615, fatal: 0 wsrep = (exec_mode: 1 conflict_state: 5 seqno: 380189)
2015-06-30 15:39:37 4205 [Warning] WSREP: RBR event 3 Delete_rows apply warning: 1615, 380189
2015-06-30 15:39:37 4205 [Warning] WSREP: Failed to apply app buffer: seqno: 380189, status: 1
at galera/src/trx_handle.cpp:apply():351
Retrying 3th time
2015-06-30 15:39:37 4205 [Warning] WSREP: BF applier failed to open_and_lock_tables: 1615, fatal: 0 wsrep = (exec_mode: 1 conflict_state: 5 seqno: 380189)
2015-06-30 15:39:37 4205 [Warning] WSREP: RBR event 3 Delete_rows apply warning: 1615, 380189
2015-06-30 15:39:37 4205 [Warning] WSREP: Failed to apply app buffer: seqno: 380189, status: 1
at galera/src/trx_handle.cpp:apply():351
Retrying 4th time
2015-06-30 15:39:37 4205 [Warning] WSREP: BF applier failed to open_and_lock_tables: 1615, fatal: 0 wsrep = (exec_mode: 1 conflict_state: 5 seqno: 380189)
2015-06-30 15:39:37 4205 [Warning] WSREP: RBR event 3 Delete_rows apply warning: 1615, 380189
2015-06-30 15:39:37 4205 [Warning] WSREP: failed to replay trx: source: d3a77cee-1f1e-11e5-89a4-ab6ae97f3cf8 version: 3 local: 1 state: REPLAYING flags: 1 conn_id: 3081 trx_id: 867082 seqnos (l: 8201, g: 380189, s: 380187, d: 380121, ts: 1041510854992104)
2015-06-30 15:39:37 4205 [Warning] WSREP: Failed to apply trx 380189 4 times
2015-06-30 15:39:37 4205 [ERROR] WSREP: trx_replay failed for: 6, query: void
2015-06-30 15:39:37 4205 [ERROR] Aborting
2015-06-30 15:39:39 4205 [Note] WSREP: waiting for client connections to close: 3
2015-06-30 15:39:39 4205 [Note] WSREP: Closing send monitor...
2015-06-30 15:39:39 4205 [Note] WSREP: Closed send monitor.
2015-06-30 15:39:39 4205 [Note] WSREP: gcomm: terminating thread
2015-06-30 15:39:39 4205 [Note] WSREP: gcomm: joining thread
2015-06-30 15:39:39 4205 [Note] WSREP: gcomm: closing backend
2015-06-30 15:39:39 4205 [Note] WSREP: view(view_id(NON_PRIM,77ca548c,101) memb {
d3a77cee,0
} joined {
} left {
} partitioned {
77ca548c,0
929d019a,0
})
2015-06-30 15:39:39 4205 [Note] WSREP: view((empty))
2015-06-30 15:39:39 4205 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
2015-06-30 15:39:39 4205 [Note] WSREP: gcomm: closed
2015-06-30 15:39:39 4205 [Note] WSREP: Flow-control interval: [16, 16]
2015-06-30 15:39:39 4205 [Note] WSREP: Received NON-PRIMARY.
2015-06-30 15:39:39 4205 [Note] WSREP: Shifting SYNCED -> OPEN (TO: 380197)
2015-06-30 15:39:39 4205 [Note] WSREP: Received self-leave message.
2015-06-30 15:39:39 4205 [Note] WSREP: Flow-control interval: [0, 0]
2015-06-30 15:39:39 4205 [Note] WSREP: Received SELF-LEAVE. Closing connection.
2015-06-30 15:39:39 4205 [Note] WSREP: Shifting OPEN -> CLOSED (TO: 380197)
2015-06-30 15:39:39 4205 [Note] WSREP: RECV thread exiting 0: Success
2015-06-30 15:39:39 4205 [Note] WSREP: recv_thread() joined.
2015-06-30 15:39:39 4205 [Note] WSREP: Closing replication queue.
2015-06-30 15:39:39 4205 [Note] WSREP: Closing slave action queue.
2015-06-30 15:39:39 4205 [Note] WSREP: Service disconnected.
2015-06-30 15:39:39 4205 [Note] WSREP: closing wsrep thread 1
2015-06-30 15:39:39 4205 [Note] WSREP: closing wsrep thread 2
2015-06-30 15:39:39 4205 [Note] WSREP: WSREP rollback thread wakes for signal
2015-06-30 15:39:39 4205 [Note] WSREP: WSREP rollback thread has empty abort queue
2015-06-30 15:39:39 4205 [Note] WSREP: rollbacker thread exiting
2015-06-30 15:39:39 4205 [Note] WSREP: avoiding thread re-use for applier, thd: 2
2015-06-30 15:39:39 4205 [Note] WSREP: cleanup transaction for LOCAL_STATE: select * from `sessions` where `id` = 'af42b669b6da8d2734f8e174895cced7847b7551' limit 1
2015-06-30 15:39:39 4205 [Note] WSREP: cleanup transaction for LOCAL_STATE: select * from `users` where `users`.`deleted_at` is null and `id_number` = '1039467251' limit 1
2015-06-30 15:39:39 4205 [Note] WSREP: cleanup transaction for LOCAL_STATE: select count(*) as aggregate from `visa_requests` where `visa_requests`.`deleted_at` is null and `user_id` = '7915'
2015-06-30 15:39:39 4205 [Note] WSREP: cleanup transaction for LOCAL_STATE: select * from `visa_requests` where `visa_requests`.`deleted_at` is null and `user_id` = '7915' limit 15 offset 0
2015-06-30 15:39:39 4205 [Note] WSREP: commit failed for reason: 3 3083 update `sessions` set `payload` = 'YTo4OntzOjY6Il90b2tlbiI7czo0MDoiaGFKNk5rZE1RdTI4SkFvaU9QZlpvVEd6WUVYZzFxckt0TDNLdVVwaSI7czo5OiJfcHJldmlvdXMiO2E6MTp7czozOiJ1cmwiO3M6NDA6Imh0dHA6Ly92aXNhLm11c2FuZWQuZ292LnNhL3Zpc2FfcmVxdWVzdHMiO31zOjU6ImZsYXNoIjthOjI6e3M6Mzoib2xkIjthOjA6e31zOjM6Im5ldyI7YTowOnt9fXM6MTI6InJlZ2lzdHJhdGlvbiI7YTozOntzOjQ6ImRhdGEiO2E6Njp7czo2OiJtb2JpbGUiO3M6MTA6IjA1NTU1ODg5NjgiO3M6NToiZW1haWwiO3M6MjA6InNtb3dzZG4xMTFAZ21haWwuY29tIjtzOjk6ImlkX251bWJlciI7czoxMDoiMTAzOTQ2NzI1MSI7czo4OiJwYXNzd29yZCI7czoxMToiU21vNDg3NE1vaEAiO3M6NDoibmFtZSI7czo0NToi2YXYrdmF2K8g2LnZhNmJINio2YYg2K/ZhNmK2YUg2KLZhCDYstmK2K/Yp9mGIjtzOjEwOiJiaXJ0aF9kYXRlIjtzOjEwOiIxNDA0LTA5LTIyIjt9czo0OiJjb2RlIjtzOjY6IjAxOTUzNCI7czoxNToiY29kZV9leHBpcmF0aW9uIjtPOjEzOiJDYXJib25cQ2FyYm9uIjozOntzOjQ6ImRhdGUiO3M6MjY6IjIwMTUtMDYtMzAgMTU6NDM6MzMuMDAwMDAwIjtzOjEzOiJ0aW1lem9uZV90eXBlIjtpOjM7czo4OiJ0aW1lem9uZSI7czoxMToiQXNpYS9SaXlhZGgiO319czo3OiJtZXNzYWdlIjthOjA6e31zOjM4OiJsb2dpbl84MmU1ZDJjNTZiZGQwODExMzE4ZjBjZjA3
2015-06-30 15:39:39 4205 [Note] WSREP: conflict state: 0
2015-06-30 15:39:39 4205 [Note] WSREP: cluster conflict due to certification failure for threads:
2015-06-30 15:39:39 4205 [Note] WSREP: Victim thread:
THD: 3083, mode: local, state: executing, conflict: cert failure, seqno: -1
SQL: update `sessions` set `payload` = 'YTo4OntzOjY6Il90b2tlbiI7czo0MDoiaGFKNk5rZE1RdTI4SkFvaU9QZlpvVEd6WUVYZzFxckt0TDNLdVVwaSI7czo5OiJfcHJldmlvdXMiO2E6MTp7czozOiJ1cmwiO3M6NDA6Imh0dHA6Ly92aXNhLm11c2FuZWQuZ292LnNhL3Zpc2FfcmVxdWVzdHMiO31zOjU6ImZsYXNoIjthOjI6e3M6Mzoib2xkIjthOjA6e31zOjM6Im5ldyI7YTowOnt9fXM6MTI6InJlZ2lzdHJhdGlvbiI7YTozOntzOjQ6ImRhdGEiO2E6Njp7czo2OiJtb2JpbGUiO3M6MTA6IjA1NTU1ODg5NjgiO3M6NToiZW1haWwiO3M6MjA6InNtb3dzZG4xMTFAZ21haWwuY29tIjtzOjk6ImlkX251bWJlciI7czoxMDoiMTAzOTQ2NzI1MSI7czo4OiJwYXNzd29yZCI7czoxMToiU21vNDg3NE1vaEAiO3M6NDoibmFtZSI7czo0NToi2YXYrdmF2K8g2LnZhNmJINio2YYg2K/ZhNmK2YUg2KLZhCDYstmK2K/Yp9mGIjtzOjEwOiJiaXJ0aF9kYXRlIjtzOjEwOiIxNDA0LTA5LTIyIjt9czo0OiJjb2RlIjtzOjY6IjAxOTUzNCI7czoxNToiY29kZV9leHBpcmF0aW9uIjtPOjEzOiJDYXJib25cQ2FyYm9uIjozOntzOjQ6ImRhdGUiO3M6MjY6IjIwMTUtMDYtMzAgMTU6NDM6MzMuMDAwMDAwIjtzOjEzOiJ0aW1lem9uZV90eXBlIjtpOjM7czo4OiJ0aW1lem9uZSI7czoxMToiQXNpYS9SaXlhZGgiO319czo3Oi
2015-06-30 15:39:39 4205 [Note] WSREP: cleanup transaction for LOCAL_STATE: update `sessions` set `payload` = 'YTo4OntzOjY6Il90b2tlbiI7czo0MDoiaGFKNk5rZE1RdTI4SkFvaU9QZlpvVEd6WUVYZzFxckt0TDNLdVVwaSI7czo5OiJfcHJldmlvdXMiO2E6MTp7czozOiJ1cmwiO3M6NDA6Imh0dHA6Ly92aXNhLm11c2FuZWQuZ292LnNhL3Zpc2FfcmVxdWVzdHMiO31zOjU6ImZsYXNoIjthOjI6e3M6Mzoib2xkIjthOjA6e31zOjM6Im5ldyI7YTowOnt9fXM6MTI6InJlZ2lzdHJhdGlvbiI7YTozOntzOjQ6ImRhdGEiO2E6Njp7czo2OiJtb2JpbGUiO3M6MTA6IjA1NTU1ODg5NjgiO3M6NToiZW1haWwiO3M6MjA6InNtb3dzZG4xMTFAZ21haWwuY29tIjtzOjk6ImlkX251bWJlciI7czoxMDoiMTAzOTQ2NzI1MSI7czo4OiJwYXNzd29yZCI7czoxMToiU21vNDg3NE1vaEAiO3M6NDoibmFtZSI7czo0NToi2YXYrdmF2K8g2LnZhNmJINio2YYg2K/ZhNmK2YUg2KLZhCDYstmK2K/Yp9mGIjtzOjEwOiJiaXJ0aF9kYXRlIjtzOjEwOiIxNDA0LTA5LTIyIjt9czo0OiJjb2RlIjtzOjY6IjAxOTUzNCI7czoxNToiY29kZV9leHBpcmF0aW9uIjtPOjEzOiJDYXJib25cQ2FyYm9uIjozOntzOjQ6ImRhdGUiO3M6MjY6IjIwMTUtMDYtMzAgMTU6NDM6MzMuMDAwMDAwIjtzOjEzOiJ0aW1lem9uZV90eXBlIjtpOjM7czo4OiJ0aW1lem9uZSI7czoxMToiQXNpYS9SaXlhZGgiO319czo3OiJtZXNzYWdlIjthOjA6e31zOjM4OiJsb2dpbl84MmU1ZDJjNTZiZGQwODExMzE4ZjBj
2015-06-30 15:39:39 4205 [Note] WSREP: PS execute fail for CERT_FAILURE: thd: 3083 err: 1213
2015-06-30 15:39:40 4205 [Note] WSREP: Some threads may fail to exit.
2015-06-30 15:39:40 4205 [Note] Binlog end
2015-06-30 15:39:40 4205 [Note] Shutting down plugin 'partition'
2015-06-30 15:39:40 4205 [Note] Shutting down plugin 'BLACKHOLE'
2015-06-30 15:39:40 4205 [Note] Shutting down plugin 'ARCHIVE'
2015-06-30 15:39:40 4205 [Note] Shutting down plugin 'PERFORMANCE_SCHEMA'
2015-06-30 15:39:40 4205 [Note] Shutting down plugin 'INNODB_SYS_DATAFILES'
2015-06-30 15:39:40 4205 [Note] Shutting down plugin 'INNODB_SYS_TABLESPACES'
2015-06-30 15:39:40 4205 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN_COLS'
2015-06-30 15:39:40 4205 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN'
2015-06-30 15:39:40 4205 [Note] Shutting down plugin 'INNODB_SYS_FIELDS'
2015-06-30 15:39:40 4205 [Note] Shutting down plugin 'INNODB_SYS_COLUMNS'
2015-06-30 15:39:40 4205 [Note] Shutting down plugin 'INNODB_SYS_INDEXES'
2015-06-30 15:39:40 4205 [Note] Shutting down plugin 'INNODB_SYS_TABLESTATS'
2015-06-30 15:39:40 4205 [Note] Shutting down plugin 'INNODB_SYS_TABLES'
2015-06-30 15:39:40 4205 [Note] Shutting down plugin 'INNODB_FT_INDEX_TABLE'
2015-06-30 15:39:40 4205 [Note] Shutting down plugin 'INNODB_FT_INDEX_CACHE'
2015-06-30 15:39:40 4205 [Note] Shutting down plugin 'INNODB_FT_CONFIG'
2015-06-30 15:39:40 4205 [Note] Shutting down plugin 'INNODB_FT_BEING_DELETED'
2015-06-30 15:39:40 4205 [Note] Shutting down plugin 'INNODB_FT_DELETED'
2015-06-30 15:39:40 4205 [Note] Shutting down plugin 'INNODB_FT_DEFAULT_STOPWORD'
2015-06-30 15:39:40 4205 [Note] Shutting down plugin 'INNODB_METRICS'
2015-06-30 15:39:40 4205 [Note] Shutting down plugin 'INNODB_BUFFER_POOL_STATS'
2015-06-30 15:39:40 4205 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE_LRU'
2015-06-30 15:39:40 4205 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE'
2015-06-30 15:39:40 4205 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX_RESET'
2015-06-30 15:39:40 4205 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX'
2015-06-30 15:39:40 4205 [Note] Shutting down plugin 'INNODB_CMPMEM_RESET'
2015-06-30 15:39:40 4205 [Note] Shutting down plugin 'INNODB_CMPMEM'
2015-06-30 15:39:40 4205 [Note] Shutting down plugin 'INNODB_CMP_RESET'
2015-06-30 15:39:40 4205 [Note] Shutting down plugin 'INNODB_CMP'
2015-06-30 15:39:40 4205 [Note] Shutting down plugin 'INNODB_LOCK_WAITS'
2015-06-30 15:39:40 4205 [Note] Shutting down plugin 'INNODB_LOCKS'
2015-06-30 15:39:40 4205 [Note] Shutting down plugin 'INNODB_TRX'
2015-06-30 15:39:40 4205 [Note] Shutting down plugin 'InnoDB'
2015-06-30 15:39:40 4205 [Note] InnoDB: FTS optimize thread exiting.
2015-06-30 15:39:40 4205 [Note] InnoDB: Starting shutdown...
12:39:42 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
key_buffer_size=268435456
read_buffer_size=104857600
max_used_connections=4
max_threads=500
thread_count=3
connection_count=3
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 51725237 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x900ba50
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7ff760de3df8 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x35)[0x8fc4a5]
/usr/sbin/mysqld(handle_fatal_signal+0x494)[0x67d854]
/lib64/libpthread.so.0[0x391ae0f710]
/usr/sbin/mysqld[0x69bd9b]
/usr/sbin/mysqld[0x69cace]
/usr/sbin/mysqld[0x69cdb6]
/usr/sbin/mysqld[0x68f874]
/usr/sbin/mysqld(_Z16acl_authenticateP3THDj+0x1f7)[0x6a87b7]
/usr/sbin/mysqld[0x6cc66a]
/usr/sbin/mysqld(_Z16login_connectionP3THD+0x45)[0x6cc895]
/usr/sbin/mysqld(_Z22thd_prepare_connectionP3THD+0x24)[0x6cc924]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x139)[0x6cd619]
/usr/sbin/mysqld(handle_one_connection+0x47)[0x6cd7d7]
/usr/sbin/mysqld(pfs_spawn_thread+0x12a)[0xb1fe7a]
/lib64/libpthread.so.0[0x391ae079d1]
/lib64/libc.so.6(clone+0x6d)[0x391aae88fd]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0): is an invalid pointer
Connection ID (thread ID): 3085
Status: NOT_KILLED
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
150630 15:39:42 mysqld_safe Number of processes running now: 0
150630 15:39:42 mysqld_safe WSREP: not restarting wsrep node automatically
150630 15:39:42 mysqld_safe mysqld from pid file /var/lib/mysql/mysql.pid ended
150630 16:08:10 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150630 16:08:10 mysqld_safe WSREP: Running position recovery with --log_error='/var/lib/mysql/wsrep_recovery.NOB7qO' --pid-file='/var/lib/mysql/MOL-DLV-MySQLND3-recover.pid'
2015-06-30 16:08:10 0 [Warning] Using unique option prefix key_buffer instead of key_buffer_size is deprecated and will be removed in a future release. Please use the full name instead.
2015-06-30 16:08:10 0 [Warning] Using unique option prefix thread_cache instead of thread_cache_size is deprecated and will be removed in a future release. Please use the full name instead.
2015-06-30 16:08:10 0 [Warning] option 'thread_cache_size': unsigned value 268435456 adjusted to 16384
2015-06-30 16:08:10 0 [Warning] 'THREAD_CONCURRENCY' is deprecated and will be removed in a future release.
2015-06-30 16:08:10 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
150630 16:08:13 mysqld_safe WSREP: Recovered position c8e1f7ef-0ce4-11e5-89a6-f3af952dc915:380188
2015-06-30 16:08:13 0 [Warning] Using unique option prefix key_buffer instead of key_buffer_size is deprecated and will be removed in a future release. Please use the full name instead.
2015-06-30 16:08:13 0 [Warning] Using unique option prefix thread_cache instead of thread_cache_size is deprecated and will be removed in a future release. Please use the full name instead.
2015-06-30 16:08:13 16955 [Note] WSREP: CRC-32C: using hardware acceleration.
2015-06-30 16:08:13 16955 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1
2015-06-30 16:08:13 16955 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 10.60.32.85; base_port = 4567; cert.log_conflicts = no; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 1G; gcache.size = 300M; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false; pc
2015-06-30 16:08:13 16955 [Note] WSREP: Service thread queue flushed.
2015-06-30 16:08:13 16955 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1
2015-06-30 16:08:13 16955 [Note] WSREP: wsrep_sst_grab()
2015-06-30 16:08:13 16955 [Note] WSREP: Start replication
2015-06-30 16:08:13 16955 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1
2015-06-30 16:08:13 16955 [Note] WSREP: protonet asio version 0
2015-06-30 16:08:13 16955 [Note] WSREP: Using CRC-32C for message checksums.
2015-06-30 16:08:13 16955 [Note] WSREP: backend: asio
2015-06-30 16:08:13 16955 [Warning] WSREP: access file(/var/lib/mysql//gvwstate.dat) failed(No such file or directory)
2015-06-30 16:08:13 16955 [Note] WSREP: restore pc from disk failed
2015-06-30 16:08:13 16955 [Note] WSREP: GMCast version 0
2015-06-30 16:08:13 16955 [Note] WSREP: (10bd9501, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
2015-06-30 16:08:13 16955 [Note] WSREP: (10bd9501, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
2015-06-30 16:08:13 16955 [Note] WSREP: EVS version 0
2015-06-30 16:08:13 16955 [Note] WSREP: gcomm: connecting to group 'MOL-DLV-GCMySQL', peer '10.60.32.83:,10.60.32.84:,10.60.32.85:'
2015-06-30 16:08:13 16955 [Warning] WSREP: (10bd9501, 'tcp://0.0.0.0:4567') address 'tcp://10.60.32.85:4567' points to own listening address, blacklisting
2015-06-30 16:08:13 16955 [Note] WSREP: (10bd9501, 'tcp://0.0.0.0:4567') address 'tcp://10.60.32.85:4567' pointing to uuid 10bd9501 is blacklisted, skipping
2015-06-30 16:08:13 16955 [Note] WSREP: (10bd9501, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:
2015-06-30 16:08:13 16955 [Note] WSREP: (10bd9501, 'tcp://0.0.0.0:4567') address 'tcp://10.60.32.85:4567' pointing to uuid 10bd9501 is blacklisted, skipping
Just now i restart mysql.
From last 6 hours all 3 nodes are shutdown for almost 3 times each one by one. One time node 1 shutdown, one time node 3 and one time node 2.
Now 3 times node 3 shutdown.
Regards
Munazir
Philip Stoev
2015年6月30日 上午10:25:392015/6/30
收件者：Munazir、codersh...@googlegroups.com
Hello,
What Galera Cluster flavor and version are you running?
The issue you are observing has been reported before. You can find the bug
reports by googling for "BF applier failed to open_and_lock_tables: 1615".
One workaround you can try is to drastically increase the size of the table
cache:
table_open_cache 128=>16384
table_definition_cache 1024=>16384
You can try to do this on one of the nodes and see if this particular node
then stops crashing before changing the settings on the other nodes.
What Galera Cluster flavor and version are you running?
The issue you are observing has been reported before. You can find the bug
reports by googling for "BF applier failed to open_and_lock_tables: 1615".
One workaround you can try is to drastically increase the size of the table
cache:
table_open_cache 128=>16384
table_definition_cache 1024=>16384
You can try to do this on one of the nodes and see if this particular node
then stops crashing before changing the settings on the other nodes.
Munazir
2015年6月30日 上午10:55:562015/6/30
收件者：codersh...@googlegroups.com、mdmu...@gmail.com
Dear Philips,
We are using
wsrep_provider_version | 3.10(r8182fa6)
MySQmysql-wsrep-server-5.6-5.6.23-25.10.el6.x86_64
We are using
wsrep_provider_version | 3.10(r8182fa6)
MySQmysql-wsrep-server-5.6-5.6.23-25.10.el6.x86_64
galera-3-25.3.10-2.el6.x86_64
I will increase both parameters and update.
Well this table has unique key and primary, with this any issue.
On this table we are getting Deadlock error as this table using for maintain sessions.
Regards
Munazir
I will increase both parameters and update.
Well this table has unique key and primary, with this any issue.
On this table we are getting Deadlock error as this table using for maintain sessions.
Regards
Munazir
Philip Stoev
2015年7月1日 凌晨1:20:542015/7/1
收件者：Munazir、codersh...@googlegroups.com
Hello,
The deadlock errors you are getting are separate from the issue of nodes
going down.
Deadlock errors are inevitable in multi-master setups and the most universal
solution is to have the application detect the error and retry the
transaction. If the transaction that deadlocks is a single-statement, that
is, autocommit, transaction, you can try increasing wsrep_retry_autocommit
to 2 or 3 and see if it helps reduce the number of deadlocks. You can also
configure your application or proxy so that all statements pertaining to
your "hot" table are only sent to a single master.
The deadlock errors you are getting are separate from the issue of nodes
going down.
Deadlock errors are inevitable in multi-master setups and the most universal
solution is to have the application detect the error and retry the
transaction. If the transaction that deadlocks is a single-statement, that
is, autocommit, transaction, you can try increasing wsrep_retry_autocommit
to 2 or 3 and see if it helps reduce the number of deadlocks. You can also
configure your application or proxy so that all statements pertaining to
your "hot" table are only sent to a single master.
回覆所有人
回覆作者
轉寄
0 則新訊息