crash of 2 nodes

50 views
Skip to first unread message

Laurent Schweizer

unread,
Jun 12, 2018, 9:39:50 AM6/12/18
to Percona Discussion

Hello,

 

We are using a percona cluster with 3 nodes and today at the same time 2 nodes go down.

 

In the log I only see that they is an active transaction  going in ROLL BACK and after a report about the crash.


any idea ?


BR

Laurent

 

---TRANSACTION 126594032, ACTIVE 356 sec inserting

mysql tables in use 1, locked 1

ROLLING BACK 1 lock struct(s), heap size 1136, 1 row lock(s)

MySQL thread id 3857859, OS thread handle 139660501247744, query id 116528964 185.190.125.3 user_kam update

insert into `location` (`username`,`contact`,`expires`,`q`,`callid`,`cseq`,`flags`,`cflags`,`user_agent`,`received`,`socket`,`last_modified`,`ruid`,`reg_id`,`server_id`,`connection_id`,`keepalive`,`partition` ) values ('90795831979','sip:90795...@192.168.178.55:5060','2018-06-10 21:00:25',0.000000  ,'164f0905963e13bb',567427461,0,128,'OpenScape Business M5T SIP Stack/4.2.20.35','sip:89.245.11.190:62620','udp:185.190.125.3:5060','2018-06-10 20:57:25','uloc-5aead4bb-9f5-680fb2',0,0,-1,0,12)

---TRANSACTION 126594031, ACTIVE 364 sec inserting

mysql tables in use 1, locked 1

ROLLING BACK 1 lock struct(s), heap size 1136, 1 row lock(s)

MySQL thread id 3857848, OS thread handle 139660503111424, query id 116528738 185.190.125.3 user_kam update

insert into `location` (`username`,`contact`,`expires`,`q`,`callid`,`cseq`,`flags`,`cflags`,`user_agent`,`received`,`socket`,`last_modified`,`ruid`,`reg_id`,`server_id`,`connection_id`,`keepalive`,`partition` ) values ('90795831979','sip:90795...@192.168.178.55:5060','2018-06-10 21:00:25',0.000000  ,'164f0905963e13bb',567427461,0,128,'OpenScape Business M5T SIP Stack/4.2.20.35','sip:89.245.11.190:62620','udp:185.190.125.3:5060','2018-06-10 20:57:25','uloc-5aead4bb-9f5-680fb2',0,0,-1,0,12)

---TRANSACTION 126594030, ACTIVE (PREPARED) 388 sec

2 lock struct(s), heap size 1136, 1 row lock(s), undo log entries 1

MySQL thread id 1, OS thread handle 139680704780032, query id 116528471 wsrep: committing write set (50490120)

---TRANSACTION 126594028, ACTIVE (PREPARED) 820 sec

2 lock struct(s), heap size 1136, 1 row lock(s), undo log entries 1

MySQL thread id 6, OS thread handle 139680270214912, query id 116524487 wsrep: committing write set (50490119)

--------

FILE I/O

--------

I/O thread 0 state: waiting for completed aio requests (insert buffer thread)

I/O thread 1 state: waiting for completed aio requests (log thread)

I/O thread 2 state: waiting for completed aio requests (read thread)

I/O thread 3 state: waiting for completed aio requests (read thread)

I/O thread 4 state: waiting for completed aio requests (read thread)

I/O thread 5 state: waiting for completed aio requests (read thread)

I/O thread 6 state: waiting for completed aio requests (write thread)

I/O thread 7 state: waiting for completed aio requests (write thread)

I/O thread 8 state: waiting for completed aio requests (write thread)

I/O thread 9 state: waiting for completed aio requests (write thread)

Pending normal aio reads: [0, 0, 0, 0] , aio writes: [0, 0, 0, 0] ,

 ibuf aio reads:, log i/o's:, sync i/o's:

Pending flushes (fsync) log: 1; buffer pool: 0

2096 OS file reads, 41744672 OS file writes, 29691175 OS fsyncs

0 pending preads, 1 pending pwrites

0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s

-------------------------------------

INSERT BUFFER AND ADAPTIVE HASH INDEX

-------------------------------------

Ibuf: size 1, free list len 0, seg size 2, 7 merges

merged operations:

 insert 153, delete mark 6, delete 4

discarded operations:

 insert 0, delete mark 0, delete 0

Hash table size 4730347, node heap has 0 buffer(s)

Hash table size 4730347, node heap has 79 buffer(s)

Hash table size 4730347, node heap has 77 buffer(s)

Hash table size 4730347, node heap has 1 buffer(s)

Hash table size 4730347, node heap has 2 buffer(s)

Hash table size 4730347, node heap has 30 buffer(s)

Hash table size 4730347, node heap has 80 buffer(s)

Hash table size 4730347, node heap has 75 buffer(s)

7.50 hash searches/s, 19.44 non-hash searches/s

---

LOG

---

Log sequence number 47691824014

Log flushed up to   47691821593

Pages flushed up to 47690668247

Last checkpoint at  47690668247

Max checkpoint age    80826164

Checkpoint age target 78300347

Modified age          1155767

Checkpoint age        1155767

1 pending log flushes, 0 pending chkp writes

26789311 log i/o's done, 0.00 log i/o's/second

----------------------

BUFFER POOL AND MEMORY

----------------------

Total large memory allocated 17884512256

Dictionary memory allocated 805996

Internal hash tables (constant factor + variable factor)

    Adaptive hash index 308411968         (302742208 + 5669760)

    Page hash           2213368 (buffer pool 0 only)

    Dictionary cache    76491548 (75685552 + 805996)

    File system         882048          (812272 + 69776)

    Lock system         42547032    (42500456 + 46576)

    Recovery system     0 (0 + 0)

Buffer pool size   1048448

Buffer pool size, bytes 17177772032

Free buffers       1045683

Database pages     2421

Old database pages 0

Modified db pages  85

Pending reads      0

Pending writes: LRU 0, flush list 3, single page 0

Pages made young 0, not young 0

0.00 youngs/s, 0.00 non-youngs/s

Pages read 1990, created 431, written 13599090

0.00 reads/s, 0.00 creates/s, 0.00 writes/s

Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000

Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s

LRU len: 2421, unzip_LRU len: 0

I/O sum[0]:cur[0], unzip sum[0]:cur[0]

----------------------

INDIVIDUAL BUFFER POOL INFO

----------------------

---BUFFER POOL 0

Buffer pool size   131056

Buffer pool size, bytes 2147221504

Free buffers       130617

Database pages     394

Old database pages 0

Modified db pages  5

Pending reads      0

Pending writes: LRU 0, flush list 3, single page 0

Pages made young 0, not young 0

0.00 youngs/s, 0.00 non-youngs/s

Pages read 314, created 80, written 2054376

0.00 reads/s, 0.00 creates/s, 0.00 writes/s

Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000

Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s

LRU len: 394, unzip_LRU len: 0

I/O sum[0]:cur[0], unzip sum[0]:cur[0]

---BUFFER POOL 1

Buffer pool size   131056

Buffer pool size, bytes 2147221504

Free buffers       130846

Database pages     171

Old database pages 0

Modified db pages  0

Pending reads      0

Pending writes: LRU 0, flush list 0, single page 0

Pages made young 0, not young 0

0.00 youngs/s, 0.00 non-youngs/s

Pages read 161, created 10, written 197501

0.00 reads/s, 0.00 creates/s, 0.00 writes/s

Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000

Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s

LRU len: 171, unzip_LRU len: 0

I/O sum[0]:cur[0], unzip sum[0]:cur[0]

---BUFFER POOL 2

Buffer pool size   131056

Buffer pool size, bytes 2147221504

Free buffers       130851

Database pages     164

Old database pages 0

Modified db pages  0

Pending reads      0

Pending writes: LRU 0, flush list 0, single page 0

Pages made young 0, not young 0

0.00 youngs/s, 0.00 non-youngs/s

Pages read 155, created 9, written 195693

0.00 reads/s, 0.00 creates/s, 0.00 writes/s

Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000

Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s

LRU len: 164, unzip_LRU len: 0

I/O sum[0]:cur[0], unzip sum[0]:cur[0]

---BUFFER POOL 3

Buffer pool size   131056

Buffer pool size, bytes 2147221504

Free buffers       130664

Database pages     351

Old database pages 0

Modified db pages  13

Pending reads      0

Pending writes: LRU 0, flush list 0, single page 0

Pages made young 0, not young 0

0.00 youngs/s, 0.00 non-youngs/s

Pages read 345, created 6, written 1746420

0.00 reads/s, 0.00 creates/s, 0.00 writes/s

Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000

Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s

LRU len: 351, unzip_LRU len: 0

I/O sum[0]:cur[0], unzip sum[0]:cur[0]

---BUFFER POOL 4

Buffer pool size   131056

Buffer pool size, bytes 2147221504

Free buffers       130673

Database pages     340

Old database pages 0

Modified db pages  5

Pending reads      0

Pending writes: LRU 0, flush list 0, single page 0

Pages made young 0, not young 0

0.00 youngs/s, 0.00 non-youngs/s

Pages read 296, created 44, written 1208702

0.00 reads/s, 0.00 creates/s, 0.00 writes/s

Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000

Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s

LRU len: 340, unzip_LRU len: 0

I/O sum[0]:cur[0], unzip sum[0]:cur[0]

---BUFFER POOL 5

Buffer pool size   131056

Buffer pool size, bytes 2147221504

Free buffers       130719

Database pages     291

Old database pages 0

Modified db pages  3

Pending reads      0

Pending writes: LRU 0, flush list 0, single page 0

Pages made young 0, not young 0

0.00 youngs/s, 0.00 non-youngs/s

Pages read 230, created 61, written 1665040

0.00 reads/s, 0.00 creates/s, 0.00 writes/s

Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000

Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s

LRU len: 291, unzip_LRU len: 0

I/O sum[0]:cur[0], unzip sum[0]:cur[0]

---BUFFER POOL 6

Buffer pool size   131056

Buffer pool size, bytes 2147221504

Free buffers       130632

Database pages     378

Old database pages 0

Modified db pages  8

Pending reads      0

Pending writes: LRU 0, flush list 0, single page 0

Pages made young 0, not young 0

0.00 youngs/s, 0.00 non-youngs/s

Pages read 236, created 142, written 4455875

0.00 reads/s, 0.00 creates/s, 0.00 writes/s

Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000

Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s

LRU len: 378, unzip_LRU len: 0

I/O sum[0]:cur[0], unzip sum[0]:cur[0]

---BUFFER POOL 7

Buffer pool size   131056

Buffer pool size, bytes 2147221504

Free buffers       130681

Database pages     332

Old database pages 0

Modified db pages  51

Pending reads      0

Pending writes: LRU 0, flush list 0, single page 0

Pages made young 0, not young 0

0.00 youngs/s, 0.00 non-youngs/s

Pages read 253, created 79, written 2075483

0.00 reads/s, 0.00 creates/s, 0.00 writes/s

Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000

Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s

LRU len: 332, unzip_LRU len: 0

I/O sum[0]:cur[0], unzip sum[0]:cur[0]

--------------

ROW OPERATIONS

--------------

0 queries inside InnoDB, 0 queries in queue

0 read views open inside InnoDB

4 RW transactions active inside InnoDB

Process ID=29903, Main thread ID=139661988427520, state: enforcing dict cache limit

Number of rows inserted 206637369, updated 29445495, deleted 276672, read 1301611325

0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 676.83 reads/s

----------------------------

END OF INNODB MONITOR OUTPUT

============================

InnoDB: ###### Diagnostic info printed to the standard error stream

2018-06-10T19:05:35.303135Z 0 [ERROR] [FATAL] InnoDB: Semaphore wait has lasted > 600 seconds. We intentionally crash the server because it appears to be hung.

2018-06-10 21:05:35 0x7f0597b58700  InnoDB: Assertion failure in thread 139661996820224 in file ut0ut.cc line 917

InnoDB: We intentionally generate a memory trap.

InnoDB: Submit a detailed bug report to http://bugs.mysql.com.

InnoDB: If you get repeated assertion failures or crashes, even

InnoDB: immediately after the mysqld startup, there may be

InnoDB: corruption in the InnoDB tablespace. Please refer to

InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html

InnoDB: about forcing recovery.

19:05:35 UTC - mysqld got signal 6 ;

This could be because you hit a bug. It is also possible that this binary

or one of the libraries it was linked against is corrupt, improperly built,

or misconfigured. This error can also be caused by malfunctioning hardware.

Attempting to collect some information that could help diagnose the problem.

As this is a crash and something is definitely wrong, the information

collection process might fail.

Please help us make Percona XtraDB Cluster better by reporting any

bugs at https://bugs.launchpad.net/percona-xtradb-cluster

 

key_buffer_size=8388608

read_buffer_size=131072

max_used_connections=67

max_threads=1001

thread_count=74

connection_count=65

It is possible that mysqld could use up to

key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 406815 K  bytes of memory

Hope that's ok; if not, decrease some variables in the equation.

 

Thread pointer: 0x0

Attempting backtrace. You can use the following information to find out

where mysqld died. If you see no messages after this, something went

terribly wrong...

stack_bottom = 0 thread_stack 0x40000

/usr/sbin/mysqld(my_print_stacktrace+0x3b)[0xf3e34b]

/usr/sbin/mysqld(handle_fatal_signal+0x471)[0x7ae1d1]

/lib64/libpthread.so.0(+0xf5e0)[0x7f0a0a48e5e0]

/lib64/libc.so.6(gsignal+0x37)[0x7f0a085971f7]

/lib64/libc.so.6(abort+0x148)[0x7f0a085988e8]

/usr/sbin/mysqld[0x77775a]

/usr/sbin/mysqld(_ZN2ib5fatalD1Ev+0xfd)[0x11169bd]

/usr/sbin/mysqld(srv_error_monitor_thread+0xaf9)[0x10b1099]

/lib64/libpthread.so.0(+0x7e25)[0x7f0a0a486e25]

/lib64/libc.so.6(clone+0x6d)[0x7f0a0865a34d]

You may download the Percona XtraDB Cluster operations manual by visiting

http://www.percona.com/software/percona-xtradb-cluster/. You may find information

in the manual which will help you identify the cause of the crash.

 

Syamiera Ahmad Sobri

unread,
Jun 12, 2018, 9:47:00 AM6/12/18
to percona-d...@googlegroups.com
Please stop all node. And start new one using galera new cluster command

Regards,

Noor Syamiera

Genusis Consultancy Sdn. Bhd.
Tel: 603-7876 0145

--
You received this message because you are subscribed to the Google Groups "Percona Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to percona-discussion+unsub...@googlegroups.com.
To post to this group, send email to percona-discussion@googlegroups.com.
Visit this group at https://groups.google.com/group/percona-discussion.
To view this discussion on the web visit https://groups.google.com/d/msgid/percona-discussion/e3ac1d37-fb47-40a3-a084-5c0cdc615057%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Reply all
Reply to author
Forward
0 new messages