Hello,
We are using a percona cluster with 3 nodes and today at the same time 2 nodes go down.
In the log I only see that they is an active transaction going in ROLL BACK and after a report about the crash.
any idea ?
BR
Laurent
---TRANSACTION 126594032, ACTIVE 356 sec inserting
mysql tables in use 1, locked 1
ROLLING BACK 1 lock struct(s), heap size 1136, 1 row lock(s)
MySQL thread id 3857859, OS thread handle 139660501247744, query id 116528964 185.190.125.3 user_kam update
insert into `location` (`username`,`contact`,`expires`,`q`,`callid`,`cseq`,`flags`,`cflags`,`user_agent`,`received`,`socket`,`last_modified`,`ruid`,`reg_id`,`server_id`,`connection_id`,`keepalive`,`partition` ) values ('90795831979','sip:90795...@192.168.178.55:5060','2018-06-10 21:00:25',0.000000 ,'164f0905963e13bb',567427461,0,128,'OpenScape Business M5T SIP Stack/4.2.20.35','sip:89.245.11.190:62620','udp:185.190.125.3:5060','2018-06-10 20:57:25','uloc-5aead4bb-9f5-680fb2',0,0,-1,0,12)
---TRANSACTION 126594031, ACTIVE 364 sec inserting
mysql tables in use 1, locked 1
ROLLING BACK 1 lock struct(s), heap size 1136, 1 row lock(s)
MySQL thread id 3857848, OS thread handle 139660503111424, query id 116528738 185.190.125.3 user_kam update
insert into `location` (`username`,`contact`,`expires`,`q`,`callid`,`cseq`,`flags`,`cflags`,`user_agent`,`received`,`socket`,`last_modified`,`ruid`,`reg_id`,`server_id`,`connection_id`,`keepalive`,`partition` ) values ('90795831979','sip:90795...@192.168.178.55:5060','2018-06-10 21:00:25',0.000000 ,'164f0905963e13bb',567427461,0,128,'OpenScape Business M5T SIP Stack/4.2.20.35','sip:89.245.11.190:62620','udp:185.190.125.3:5060','2018-06-10 20:57:25','uloc-5aead4bb-9f5-680fb2',0,0,-1,0,12)
---TRANSACTION 126594030, ACTIVE (PREPARED) 388 sec
2 lock struct(s), heap size 1136, 1 row lock(s), undo log entries 1
MySQL thread id 1, OS thread handle 139680704780032, query id 116528471 wsrep: committing write set (50490120)
---TRANSACTION 126594028, ACTIVE (PREPARED) 820 sec
2 lock struct(s), heap size 1136, 1 row lock(s), undo log entries 1
MySQL thread id 6, OS thread handle 139680270214912, query id 116524487 wsrep: committing write set (50490119)
--------
FILE I/O
--------
I/O thread 0 state: waiting for completed aio requests (insert buffer thread)
I/O thread 1 state: waiting for completed aio requests (log thread)
I/O thread 2 state: waiting for completed aio requests (read thread)
I/O thread 3 state: waiting for completed aio requests (read thread)
I/O thread 4 state: waiting for completed aio requests (read thread)
I/O thread 5 state: waiting for completed aio requests (read thread)
I/O thread 6 state: waiting for completed aio requests (write thread)
I/O thread 7 state: waiting for completed aio requests (write thread)
I/O thread 8 state: waiting for completed aio requests (write thread)
I/O thread 9 state: waiting for completed aio requests (write thread)
Pending normal aio reads: [0, 0, 0, 0] , aio writes: [0, 0, 0, 0] ,
ibuf aio reads:, log i/o's:, sync i/o's:
Pending flushes (fsync) log: 1; buffer pool: 0
2096 OS file reads, 41744672 OS file writes, 29691175 OS fsyncs
0 pending preads, 1 pending pwrites
0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s
-------------------------------------
INSERT BUFFER AND ADAPTIVE HASH INDEX
-------------------------------------
Ibuf: size 1, free list len 0, seg size 2, 7 merges
merged operations:
insert 153, delete mark 6, delete 4
discarded operations:
insert 0, delete mark 0, delete 0
Hash table size 4730347, node heap has 0 buffer(s)
Hash table size 4730347, node heap has 79 buffer(s)
Hash table size 4730347, node heap has 77 buffer(s)
Hash table size 4730347, node heap has 1 buffer(s)
Hash table size 4730347, node heap has 2 buffer(s)
Hash table size 4730347, node heap has 30 buffer(s)
Hash table size 4730347, node heap has 80 buffer(s)
Hash table size 4730347, node heap has 75 buffer(s)
7.50 hash searches/s, 19.44 non-hash searches/s
---
LOG
---
Log sequence number 47691824014
Log flushed up to 47691821593
Pages flushed up to 47690668247
Last checkpoint at 47690668247
Max checkpoint age 80826164
Checkpoint age target 78300347
Modified age 1155767
Checkpoint age 1155767
1 pending log flushes, 0 pending chkp writes
26789311 log i/o's done, 0.00 log i/o's/second
----------------------
BUFFER POOL AND MEMORY
----------------------
Total large memory allocated 17884512256
Dictionary memory allocated 805996
Internal hash tables (constant factor + variable factor)
Adaptive hash index 308411968 (302742208 + 5669760)
Page hash 2213368 (buffer pool 0 only)
Dictionary cache 76491548 (75685552 + 805996)
File system 882048 (812272 + 69776)
Lock system 42547032 (42500456 + 46576)
Recovery system 0 (0 + 0)
Buffer pool size 1048448
Buffer pool size, bytes 17177772032
Free buffers 1045683
Database pages 2421
Old database pages 0
Modified db pages 85
Pending reads 0
Pending writes: LRU 0, flush list 3, single page 0
Pages made young 0, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 1990, created 431, written 13599090
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 2421, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
----------------------
INDIVIDUAL BUFFER POOL INFO
----------------------
---BUFFER POOL 0
Buffer pool size 131056
Buffer pool size, bytes 2147221504
Free buffers 130617
Database pages 394
Old database pages 0
Modified db pages 5
Pending reads 0
Pending writes: LRU 0, flush list 3, single page 0
Pages made young 0, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 314, created 80, written 2054376
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 394, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 1
Buffer pool size 131056
Buffer pool size, bytes 2147221504
Free buffers 130846
Database pages 171
Old database pages 0
Modified db pages 0
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 0, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 161, created 10, written 197501
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 171, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 2
Buffer pool size 131056
Buffer pool size, bytes 2147221504
Free buffers 130851
Database pages 164
Old database pages 0
Modified db pages 0
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 0, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 155, created 9, written 195693
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 164, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 3
Buffer pool size 131056
Buffer pool size, bytes 2147221504
Free buffers 130664
Database pages 351
Old database pages 0
Modified db pages 13
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 0, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 345, created 6, written 1746420
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 351, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 4
Buffer pool size 131056
Buffer pool size, bytes 2147221504
Free buffers 130673
Database pages 340
Old database pages 0
Modified db pages 5
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 0, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 296, created 44, written 1208702
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 340, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 5
Buffer pool size 131056
Buffer pool size, bytes 2147221504
Free buffers 130719
Database pages 291
Old database pages 0
Modified db pages 3
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 0, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 230, created 61, written 1665040
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 291, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 6
Buffer pool size 131056
Buffer pool size, bytes 2147221504
Free buffers 130632
Database pages 378
Old database pages 0
Modified db pages 8
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 0, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 236, created 142, written 4455875
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 378, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 7
Buffer pool size 131056
Buffer pool size, bytes 2147221504
Free buffers 130681
Database pages 332
Old database pages 0
Modified db pages 51
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 0, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 253, created 79, written 2075483
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 332, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
--------------
ROW OPERATIONS
--------------
0 queries inside InnoDB, 0 queries in queue
0 read views open inside InnoDB
4 RW transactions active inside InnoDB
Process ID=29903, Main thread ID=139661988427520, state: enforcing dict cache limit
Number of rows inserted 206637369, updated 29445495, deleted 276672, read 1301611325
0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 676.83 reads/s
----------------------------
END OF INNODB MONITOR OUTPUT
============================
InnoDB: ###### Diagnostic info printed to the standard error stream
2018-06-10T19:05:35.303135Z 0 [ERROR] [FATAL] InnoDB: Semaphore wait has lasted > 600 seconds. We intentionally crash the server because it appears to be hung.
2018-06-10 21:05:35 0x7f0597b58700 InnoDB: Assertion failure in thread 139661996820224 in file ut0ut.cc line 917
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
19:05:35 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.
Please help us make Percona XtraDB Cluster better by reporting any
bugs at https://bugs.launchpad.net/percona-xtradb-cluster
key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=67
max_threads=1001
thread_count=74
connection_count=65
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 406815 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x3b)[0xf3e34b]
/usr/sbin/mysqld(handle_fatal_signal+0x471)[0x7ae1d1]
/lib64/libpthread.so.0(+0xf5e0)[0x7f0a0a48e5e0]
/lib64/libc.so.6(gsignal+0x37)[0x7f0a085971f7]
/lib64/libc.so.6(abort+0x148)[0x7f0a085988e8]
/usr/sbin/mysqld[0x77775a]
/usr/sbin/mysqld(_ZN2ib5fatalD1Ev+0xfd)[0x11169bd]
/usr/sbin/mysqld(srv_error_monitor_thread+0xaf9)[0x10b1099]
/lib64/libpthread.so.0(+0x7e25)[0x7f0a0a486e25]
/lib64/libc.so.6(clone+0x6d)[0x7f0a0865a34d]
You may download the Percona XtraDB Cluster operations manual by visiting
http://www.percona.com/software/percona-xtradb-cluster/. You may find information
in the manual which will help you identify the cause of the crash.
Regards,
Noor Syamiera
--
You received this message because you are subscribed to the Google Groups "Percona Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to percona-discussion+unsub...@googlegroups.com.
To post to this group, send email to percona-discussion@googlegroups.com.
Visit this group at https://groups.google.com/group/percona-discussion.
To view this discussion on the web visit https://groups.google.com/d/msgid/percona-discussion/e3ac1d37-fb47-40a3-a084-5c0cdc615057%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.