Nodes "closing tables" forever

181 views
Skip to first unread message

Florent B

unread,
Jun 12, 2015, 6:08:54 AM6/12/15
to codersh...@googlegroups.com
Hi everyone,

On an up-to-date MariaDB/Galera 3 nodes cluster, I have a problem with
nodes hanging at "closing tables".

It occurs on specific tables (not all), after write queries. All other
tables are running fine.

Sometimes "closing tables" processes occurs on the node where the write
query occured, sometimes on the others...

The only solution is to restart nodes, but it comes back later.

There is enough disk space, no IO wait ...

I tried OPTIMIZE TABLE but got the same issue.

How can I debug this ? What configuration value do you need ?

Thank you.

Florent

Florent B

unread,
Jun 12, 2015, 6:42:47 AM6/12/15
to codersh...@googlegroups.com
Here is SHOW ENGINE INNODB STATUS on a node having 4 "closing tables"
frozen :

| InnoDB | |
=====================================
2015-06-12 12:40:38 7f53b6a91700 INNODB MONITOR OUTPUT
=====================================
Per second averages calculated from the last 31 seconds
-----------------
BACKGROUND THREAD
-----------------
srv_master_thread loops: 10 srv_active, 0 srv_shutdown, 581 srv_idle
srv_master_thread log flush and writes: 591
----------
SEMAPHORES
----------
OS WAIT ARRAY INFO: reservation count 71
OS WAIT ARRAY INFO: signal count 71
Mutex spin waits 181, rounds 1458, OS waits 43
RW-shared spins 22, rounds 660, OS waits 22
RW-excl spins 0, rounds 180, OS waits 6
Spin rounds per wait: 8.06 mutex, 30.00 RW-shared, 180.00 RW-excl
------------
TRANSACTIONS
------------
Trx id counter 4036651
Purge done for trx's n:o < 4036651 undo n:o < 0 state: running but idle
History list length 297
LIST OF TRANSACTIONS FOR EACH SESSION:
---TRANSACTION 0, not started
MySQL thread id 39, OS thread handle 0x7f53b6a91700, query id 728
10.111.111.134 root init
SHOW ENGINE INNODB STATUS
---TRANSACTION 0, not started
MySQL thread id 121, OS thread handle 0x7f53b6a48700, query id 685
10.111.17.170 root cleaning up
---TRANSACTION 4036380, not started
MySQL thread id 4, OS thread handle 0x7f53b6ada700, query id 259 closing
tables
---TRANSACTION 4036378, not started
MySQL thread id 6, OS thread handle 0x7f53b6b6c700, query id 258 closing
tables
---TRANSACTION 4036376, not started
MySQL thread id 5, OS thread handle 0x7f53b6b23700, query id 257 closing
tables
---TRANSACTION 4036382, not started
MySQL thread id 2, OS thread handle 0x7f541d7b4700, query id 260 closing
tables
---TRANSACTION 4036355, not started
MySQL thread id 3, OS thread handle 0x7f53b6bfe700, query id 0 Waiting
for background binlog tasks
--------
FILE I/O
--------
I/O thread 0 state: waiting for completed aio requests (insert buffer
thread)
I/O thread 1 state: waiting for completed aio requests (log thread)
I/O thread 2 state: waiting for completed aio requests (read thread)
I/O thread 3 state: waiting for completed aio requests (read thread)
I/O thread 4 state: waiting for completed aio requests (read thread)
I/O thread 5 state: waiting for completed aio requests (read thread)
I/O thread 6 state: waiting for completed aio requests (write thread)
I/O thread 7 state: waiting for completed aio requests (write thread)
I/O thread 8 state: waiting for completed aio requests (write thread)
I/O thread 9 state: waiting for completed aio requests (write thread)
Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 0 [0, 0, 0, 0] ,
ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0
Pending flushes (fsync) log: 0; buffer pool: 0
740 OS file reads, 303 OS file writes, 89 OS fsyncs
0.23 reads/s, 311296 avg bytes/read, 4.90 writes/s, 2.00 fsyncs/s
-------------------------------------
INSERT BUFFER AND ADAPTIVE HASH INDEX
-------------------------------------
Ibuf: size 1, free list len 53, seg size 55, 0 merges
merged operations:
insert 0, delete mark 0, delete 0
discarded operations:
insert 0, delete mark 0, delete 0
0.00 hash searches/s, 7.45 non-hash searches/s
---
LOG
---
Log sequence number 4324724161
Log flushed up to 4324724161
Pages flushed up to 4324724161
Last checkpoint at 4324724161
Max checkpoint age 108005254
Checkpoint age target 104630090
Modified age 0
Checkpoint age 0
0 pending log writes, 0 pending chkp writes
101 log i/o's done, 0.74 log i/o's/second
----------------------
BUFFER POOL AND MEMORY
----------------------
Total memory allocated 1098907648; in additional pool allocated 0
Total memory allocated by read views 200
Internal hash tables (constant factor + variable factor)
Adaptive hash index 18958896 (18921928 + 36968)
Page hash 139112 (buffer pool 0 only)
Dictionary cache 4809664 (4426736 + 382928)
File system 957736 (812272 + 145464)
Lock system 2659696 (2657176 + 2520)
Recovery system 0 (0 + 0)
Dictionary memory allocated 382928
Buffer pool size 65528
Buffer pool size, bytes 1073610752
Free buffers 64802
Database pages 724
Old database pages 0
Modified db pages 0
Percent of dirty pages(LRU & free pages): 0.000
Max dirty pages percent: 0.001
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 0, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 685, created 39, written 170
0.16 reads/s, 1.26 creates/s, 2.97 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read
ahead 0.00/s
LRU len: 724, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
----------------------
INDIVIDUAL BUFFER POOL INFO
----------------------
---BUFFER POOL 0
Buffer pool size 8191
Buffer pool size, bytes 134201344
Free buffers 8075
Database pages 115
Old database pages 0
Modified db pages 0
Percent of dirty pages(LRU & free pages): 0.000
Max dirty pages percent: 0.001
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 0, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 115, created 0, written 36
0.00 reads/s, 0.00 creates/s, 0.39 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read
ahead 0.00/s
LRU len: 115, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 1
Buffer pool size 8191
Buffer pool size, bytes 134201344
Free buffers 8129
Database pages 61
Old database pages 0
Modified db pages 0
Percent of dirty pages(LRU & free pages): 0.000
Max dirty pages percent: 0.001
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 0, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 61, created 0, written 21
0.06 reads/s, 0.00 creates/s, 0.29 writes/s
Buffer pool hit rate 986 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read
ahead 0.00/s
LRU len: 61, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 2
Buffer pool size 8191
Buffer pool size, bytes 134201344
Free buffers 8146
Database pages 45
Old database pages 0
Modified db pages 0
Percent of dirty pages(LRU & free pages): 0.000
Max dirty pages percent: 0.001
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 0, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 45, created 0, written 6
0.00 reads/s, 0.00 creates/s, 0.06 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read
ahead 0.00/s
LRU len: 45, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 3
Buffer pool size 8191
Buffer pool size, bytes 134201344
Free buffers 8104
Database pages 87
Old database pages 0
Modified db pages 0
Percent of dirty pages(LRU & free pages): 0.000
Max dirty pages percent: 0.001
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 0, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 87, created 0, written 18
0.00 reads/s, 0.00 creates/s, 0.03 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read
ahead 0.00/s
LRU len: 87, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 4
Buffer pool size 8191
Buffer pool size, bytes 134201344
Free buffers 8102
Database pages 89
Old database pages 0
Modified db pages 0
Percent of dirty pages(LRU & free pages): 0.000
Max dirty pages percent: 0.001
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 0, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 89, created 0, written 13
0.00 reads/s, 0.00 creates/s, 0.26 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read
ahead 0.00/s
LRU len: 89, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 5
Buffer pool size 8191
Buffer pool size, bytes 134201344
Free buffers 8097
Database pages 94
Old database pages 0
Modified db pages 0
Percent of dirty pages(LRU & free pages): 0.000
Max dirty pages percent: 0.001
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 0, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 94, created 0, written 10
0.00 reads/s, 0.00 creates/s, 0.19 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read
ahead 0.00/s
LRU len: 94, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 6
Buffer pool size 8191
Buffer pool size, bytes 134201344
Free buffers 8065
Database pages 126
Old database pages 0
Modified db pages 0
Percent of dirty pages(LRU & free pages): 0.000
Max dirty pages percent: 0.001
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 0, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 126, created 0, written 10
0.06 reads/s, 0.00 creates/s, 0.06 writes/s
Buffer pool hit rate 996 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read
ahead 0.00/s
LRU len: 126, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 7
Buffer pool size 8191
Buffer pool size, bytes 134201344
Free buffers 8084
Database pages 107
Old database pages 0
Modified db pages 0
Percent of dirty pages(LRU & free pages): 0.000
Max dirty pages percent: 0.001
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 0, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 68, created 39, written 56
0.03 reads/s, 1.26 creates/s, 1.68 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read
ahead 0.00/s
LRU len: 107, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
--------------
ROW OPERATIONS
--------------
0 queries inside InnoDB, 0 queries in queue
0 read views open inside InnoDB
0 RW transactions active inside InnoDB
0 RO transactions active inside InnoDB
0 out of 1000 descriptors used
Main thread process no. 6270, id 139997717968640, state: sleeping
Number of rows inserted 0, updated 30, deleted 0, read 16427
0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 24.93 reads/s
Number of system rows inserted 0, updated 0, deleted 0, read 0
0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s
----------------------------
END OF INNODB MONITOR OUTPUT
============================

Daniel Black

unread,
Jun 15, 2015, 3:52:33 AM6/15/15
to codersh...@googlegroups.com

Florent B

unread,
Jun 16, 2015, 3:24:40 AM6/16/15
to codersh...@googlegroups.com
Thank you ;)
--
You received this message because you are subscribed to the Google Groups "codership" group.
To unsubscribe from this group and stop receiving emails from it, send an email to codership-tea...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Reply all
Reply to author
Forward
0 new messages