Hanging Galera Cluster, looking for the cause

3,069 views
Skip to first unread message

Jan Kirchhoff

unread,
May 31, 2013, 4:29:13 AM5/31/13
to codersh...@googlegroups.com
Hi,

we have been testing MariaDB 5.5.29 with Galera
(wsrep_provider_version 23.2.4(r147)) for a few months now and have had
a freeze of the cluster for the second time now. I didn't report the
first hang about 6 weeks ago as I wanted to be sure it wasn't our fault,
but I can't find any type of network issue or such in our monitoring
system and nothing in the logs that could be related to this problem:

We have 2 database servers running Ubuntu 12.10 with Kernel 3.5.0
(names: db8, db9), with a bunch of 15k-disks, some SSDs, 128GB RAM and a
test database of about 600-700GB. The amount of binlogs written every
day is around 500GB, so you get an idea of the workload.
db8 is a slave (traditional replication) of our main database. We have a
garbd running on another server as a third galera instance.

Everything runs smooth for some weeks, but since this morning at 3:50am
db9 seems to be frozen. The server itself looks good, mysqld is taking
about 50% of one CPU core, everything else looks normal. Nothing in the
logs. Connecting to mysql works, but any query (like "show status" or
normal selects) hang forever. Only "show engine innodb status" returns.
There are almost 3000 processes that piled up like this:

---TRANSACTION 2F89D7CB, not started
MySQL thread id 113575, OS thread handle 0x7fab31821700, query id
45249946 cl4.logical-line.de 192.168.0.16 stats_pflege sleeping

but none of them ever gets any further. I can issue a "kill 113575" and
the prompt returns, but nothing happens.

This is what db8's global status says:
wsrep_local_state_uuid 41d7259d-7200-11e2-0800-943dd0e2f659
wsrep_protocol_version 4
wsrep_last_committed 179518511
wsrep_replicated 53590833
wsrep_replicated_bytes 14768686288759
wsrep_received 894995
wsrep_received_bytes 7161712
wsrep_local_commits 53583808
wsrep_local_cert_failures 60
wsrep_local_bf_aborts 0
wsrep_local_replays 0
wsrep_local_send_queue 8
wsrep_local_send_queue_avg 0.000000
wsrep_local_recv_queue 0
wsrep_local_recv_queue_avg 0.000000
wsrep_flow_control_paused 1.000000
wsrep_flow_control_sent 0
wsrep_flow_control_recv 0
wsrep_cert_deps_distance 73.229730
wsrep_apply_oooe 0.000000
wsrep_apply_oool 0.000000
wsrep_apply_window 0.000000
wsrep_commit_oooe 0.000000
wsrep_commit_oool 0.000000
wsrep_commit_window 0.000000
wsrep_local_state 4
wsrep_local_state_comment Synced
wsrep_cert_index_size 25606
wsrep_causal_reads 0
wsrep_incoming_addresses 192.168.10.30:3306,192.168.10.31:3306,
wsrep_cluster_conf_id 15
wsrep_cluster_size 3
wsrep_cluster_state_uuid 41d7259d-7200-11e2-0800-943dd0e2f659
wsrep_cluster_status Primary
wsrep_connected ON
wsrep_local_index 0
wsrep_provider_name Galera
wsrep_provider_vendor Codership Oy <in...@codership.com>
wsrep_provider_version 23.2.4(r147)
wsrep_ready ON




Here is the innodb status from db9 (the one that hangs). This is the
only output I could get out of db9 at all:

Type Name Status
InnoDB
=====================================
130531 10:02:08 INNODB MONITOR OUTPUT
=====================================
Per second averages calculated from the last 18 seconds
-----------------
BACKGROUND THREAD
-----------------
srv_master_thread loops: 907033 1_second, 906789 sleeps, 89831
10_second, 22491 background, 22489 flush
srv_master_thread log flush and writes: 478222
----------
SEMAPHORES
----------
OS WAIT ARRAY INFO: reservation count 17017782, signal count 287266881
Mutex spin waits 98936299, rounds 91970366, OS waits 868715
RW-shared spins 139527705, rounds 455067309, OS waits 6488041
RW-excl spins 58122849, rounds 842712424, OS waits 8565312
Spin rounds per wait: 0.93 mutex, 3.26 RW-shared, 14.50 RW-excl
--------
FILE I/O
--------
I/O thread 0 state: waiting for completed aio requests (insert buffer
thread)
I/O thread 1 state: waiting for completed aio requests (log thread)
I/O thread 2 state: waiting for completed aio requests (read thread)
I/O thread 3 state: waiting for completed aio requests (read thread)
I/O thread 4 state: waiting for completed aio requests (read thread)
I/O thread 5 state: waiting for completed aio requests (read thread)
I/O thread 6 state: waiting for completed aio requests (write thread)
I/O thread 7 state: waiting for completed aio requests (write thread)
I/O thread 8 state: waiting for completed aio requests (write thread)
I/O thread 9 state: waiting for completed aio requests (write thread)
Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 0 [0, 0, 0, 0] ,
ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0
Pending flushes (fsync) log: 0; buffer pool: 0
49405970 OS file reads, 277420410 OS file writes, 6653801 OS fsyncs
1 pending preads, 0 pending pwrites
49.44 reads/s, 16384 avg bytes/read, 0.61 writes/s, 0.61 fsyncs/s
-------------------------------------
INSERT BUFFER AND ADAPTIVE HASH INDEX
-------------------------------------
Ibuf: size 1, free list len 698, seg size 700, 300226 merges
merged operations:
insert 410879, delete mark 8032461, delete 82075
discarded operations:
insert 0, delete mark 0, delete 0
Hash table size 212497717, node heap has 450852 buffer(s)
16846.79 hash searches/s, 77045.28 non-hash searches/s
---
LOG
---
Log sequence number 18522888544071
Log flushed up to 18522888544071
Last checkpoint at 18522888544051
Max checkpoint age 2035833201
Checkpoint age target 1972213414
Modified age 20
Checkpoint age 20
0 pending log writes, 0 pending chkp writes
6895012 log i/o's done, 0.39 log i/o's/second
----------------------
BUFFER POOL AND MEMORY
----------------------
Total memory allocated 110310195200; in additional pool allocated 0
Internal hash tables (constant factor + variable factor)
Adaptive hash index 9086745120 \t(1699981736 + 7386763384)
Page hash 106249768 (buffer pool 0 only)
Dictionary cache 426512815 \t(424997488 + 1515327)
File system 429392 \t(82672 + 346720)
Lock system 296805680 \t(295674296 + 1131384)
Recovery system 0 \t(0 + 0)
Dictionary memory allocated 1515327
Buffer pool size 6553599
Buffer pool size, bytes 107374166016
Free buffers 0
Database pages 6102747
Old database pages 2252752
Modified db pages 1
Pending reads 1
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 48135625, not young 0
64.83 youngs/s, 0.00 non-youngs/s
Pages read 49405737, created 7213276, written 266046626
49.44 reads/s, 0.00 creates/s, 0.11 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 40.11/s, Random read
ahead 0.00/s
LRU len: 6102747, unzip_LRU len: 0
I/O sum[2662]:cur[22], unzip sum[0]:cur[0]
--------------
ROW OPERATIONS
--------------
1 queries inside InnoDB, 0 queries in queue
2 read views open inside InnoDB
---OLDEST VIEW---
Normal read view
Read view low limit trx n:o 2F754555
Read view up limit trx id 2F754555
Read view low limit trx id 2F754555
Read view individually stored trx ids:
-----------------
Main thread process no. 15888, id 140381271201536, state: sleeping
Number of rows inserted 46059494, updated 2501543763, deleted 28801859,
read 17046037029
0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 48965.50 reads/s
------------
TRANSACTIONS
------------
Trx id counter 2F89DD34
Purge done for trx's n:o < 2F754554 undo n:o < 0
History list length 140
LIST OF TRANSACTIONS FOR EACH SESSION:
---TRANSACTION 0, not started
MySQL thread id 113586, OS thread handle 0x7fab314fe700, query id
45250432 localhost root sleeping
SHOW engine INNODB STATUS
---TRANSACTION 2F89DC7B, not started
MySQL thread id 113583, OS thread handle 0x7fab315d9700, query id
45250367 cl4.logical-line.de 192.168.0.16 stats_pflege sleeping
---TRANSACTION 2F89DB4F, not started
MySQL thread id 113582, OS thread handle 0x7fab31622700, query id
45250261 cl4.logical-line.de 192.168.0.16 stats_pflege sleeping
---TRANSACTION 2F89DA23, not started
MySQL thread id 113578, OS thread handle 0x7fab31746700, query id
45250159 cl4.logical-line.de 192.168.0.16 stats_pflege sleeping
---TRANSACTION 0, not started
<many many rows like these, around 3000 although max_connections is 300?! >

As soon as I trigger the restart of mysql on db9 (service mysql
restart), I see normal log entries like this:
May 31 10:16:59 db9 mysqld: 130531 10:16:59 [Note] WSREP: Stop replication
May 31 10:16:59 db9 mysqld: 130531 10:16:59 [Note] WSREP: Closing send
monitor...
May 31 10:16:59 db9 mysqld: 130531 10:16:59 [Note] WSREP: Closed send
monitor.
May 31 10:16:59 db9 mysqld: 130531 10:16:59 [Note] WSREP: gcomm:
terminating thread

and db8 is accepting inserts again and catches up with replication.

meanwhile, db9 says:
May 31 10:17:02 db9 mysqld: 130531 10:17:02 [Note] WSREP: killing local
connection: 113529
May 31 10:17:02 db9 mysqld: 130531 10:17:02 [Note] WSREP: killing local
connection: 113199
May 31 10:17:02 db9 mysqld: 130531 10:17:02 [Note] WSREP: killing local
connection: 110221

and nothing happens any more. No CPU and no IO load coming from the
mysqld processes.
After 5 minutes I killed the mysqld with "killall -9 mysqld" and then
started it again the normal way.
db9 syncs via IST and everything is fine again. But I have no idea what
caused all this. According to a colleague killing mysql on db9 was the
thing that helped 6 weeks ago, too.

Thanks for any pointers where we should look further.

Jan Kirchhoff

Dustin Marquess

unread,
May 31, 2013, 1:26:46 PM5/31/13
to Jan Kirchhoff, codersh...@googlegroups.com
I'm seeing this also using MariaDB 5.5.29, PXC 5.5.30, and PXC 5.5.29.  PXC 5.5.24 so far is working okay for me, and I plan on trying 5.5.27 & 5.5.28 to try and pinpoint where the problem started.



Jan Kirchhoff

--
You received this message because you are subscribed to the Google Groups "codership" group.
To unsubscribe from this group and stop receiving emails from it, send an email to codership-tea...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.



Jan Kirchhoff

unread,
Jun 1, 2013, 6:31:06 AM6/1/13
to codersh...@googlegroups.com, maria-de...@lists.launchpad.net
I have seen exactly the same issue again, second day in a row. This time
at 7:30 in the morning and again, no logs or anything that would give me
an idea of what could have caused this.
"killall -9 mysqld" and "service mysql start" on the hanging node is the
only solution I have so far...

I am crossposting this to maria-developers, maybe somebody on that list
has an idea where to start?

Jan

Ilias Bertsimas

unread,
Jun 1, 2013, 1:43:56 PM6/1/13
to codersh...@googlegroups.com, maria-de...@lists.launchpad.net
Hello,

From what I can see from your logs:

wsrep_flow_control_paused       1.000000 

This mean your cluster has paused replication as some of it's nodes have exceeded the wsrep_local_recv_queue fc_limit.

If you could provide the wsrep status variables from all the cluster nodes when the issue occurs it would help possibly pinpoint the cause.

In my experience I have seen such issue with large transactions that insert/update/delete multiple rows which are really bad for synchronous row based replication,
This can also occur when a node has a deadlock inside InnoDB and can't process transactions any more.

Kind Regards,
Ilias.

Jay Janssen

unread,
Jun 3, 2013, 8:20:40 AM6/3/13
to codersh...@googlegroups.com

On Jun 1, 2013, at 1:43 PM, Ilias Bertsimas <awar...@gmail.com> wrote:

In my experience I have seen such issue with large transactions that insert/update/delete multiple rows which are really bad for synchronous row based replication,
This can also occur when a node has a deadlock inside InnoDB and can't process transactions any more.

FC pause can also be caused by anything on any Synced node taking a FLUSH TABLES WITH READ LOCK -- hypothetically like a backup job that kicks off at 7:30 in the morning?



Jay Janssen, MySQL Consulting Lead, Percona

Jan Kirchhoff

unread,
Jun 3, 2013, 9:46:13 AM6/3/13
to codersh...@googlegroups.com
Am 03.06.2013 14:20, schrieb Jay Janssen:
>
> On Jun 1, 2013, at 1:43 PM, Ilias Bertsimas <awar...@gmail.com
> <mailto:awar...@gmail.com>> wrote:
>
>> In my experience I have seen such issue with large transactions that
>> insert/update/delete multiple rows which are really bad for
>> synchronous row based replication,
>> This can also occur when a node has a deadlock inside InnoDB and can't
>> process transactions any more.
>
> FC pause can also be caused by anything on any Synced node taking a
> FLUSH TABLES WITH READ LOCK -- hypothetically like a backup job that
> kicks off at 7:30 in the morning?
>
>

Ilias, you asked:
"If you could provide the wsrep status variables from all the cluster
nodes when the issue occurs it would help possibly pinpoint the cause."

I can't because it's only 2 nodes (db8/db9) plus the garbd, and the
hanging node (db9) does not respond to any mysql query but "show engine
innodb status". A "show status like "wsrep%"" hangs forevery.

But our monitoring system collects all these information once a minute.
This are the last numbers reported before the 2 freezes last week:

2013-06-01 07:30:09
+----------------------------+--------------+
| wsrep_apply_oooe | 0.148679 |
| wsrep_apply_oool | 0 |
| wsrep_apply_window | 2.303396 |
| wsrep_causal_reads | 0 |
| wsrep_cert_deps_distance | 74 |
| wsrep_cert_index_size | 12039 |
| wsrep_cluster_conf_id | 17 |
| wsrep_cluster_size | 3 |
| wsrep_cluster_status | 1 |
| wsrep_commit_oooe | 0 |
| wsrep_commit_oool | 0 |
| wsrep_commit_window | 2.178868 |
| wsrep_connected | 1 |
| wsrep_flow_control_paused | 0 |
| wsrep_flow_control_recv | 0 |
| wsrep_flow_control_sent | 0 |
| wsrep_last_committed | 180873289 |
| wsrep_local_bf_aborts | 0 |
| wsrep_local_cert_failures | 0 |
| wsrep_local_commits | 0 |
| wsrep_local_index | 0 |
| wsrep_local_recv_queue | 0 |
| wsrep_local_recv_queue_avg | 0.028086 |
| wsrep_local_replays | 0 |
| wsrep_local_send_queue | 0 |
| wsrep_local_send_queue_avg | 0 |
| wsrep_local_state | 4 |
| wsrep_protocol_version | 4 |
| wsrep_ready | 1 |
| wsrep_received | 1371093 |
| wsrep_received_bytes | 354899735755 |
| wsrep_replicated | 0 |
| wsrep_replicated_bytes | 0 |
+----------------------------+--------------+

2013-05-31 03:50:09
+----------------------------+---------------+
| subitem | last |
+----------------------------+---------------+
| wsrep_apply_oooe | 0.592308 |
| wsrep_apply_oool | 0 |
| wsrep_apply_window | 10.984615 |
| wsrep_causal_reads | 0 |
| wsrep_cert_deps_distance | 8.754717 |
| wsrep_cert_index_size | 17302 |
| wsrep_cluster_conf_id | 15 |
| wsrep_cluster_size | 3 |
| wsrep_cluster_status | 1 |
| wsrep_commit_oooe | 0 |
| wsrep_commit_oool | 0 |
| wsrep_commit_window | 13.058824 |
| wsrep_connected | 1 |
| wsrep_flow_control_paused | 0.490027 |
| wsrep_flow_control_recv | 0 |
| wsrep_flow_control_sent | 0 |
| wsrep_last_committed | 179518388 |
| wsrep_local_bf_aborts | 0 |
| wsrep_local_cert_failures | 0 |
| wsrep_local_commits | 0 |
| wsrep_local_index | 1 |
| wsrep_local_recv_queue | 0 |
| wsrep_local_recv_queue_avg | 0.038462 |
| wsrep_local_replays | 0 |
| wsrep_local_send_queue | 0 |
| wsrep_local_send_queue_avg | 0 |
| wsrep_local_state | 4 |
| wsrep_protocol_version | 4 |
| wsrep_ready | 1 |
| wsrep_received | 6515460 |
| wsrep_received_bytes | 1725743892430 |
| wsrep_replicated | 0 |
| wsrep_replicated_bytes | 0 |
+----------------------------+---------------+


And the same reports for db8, the other node in the cluster at about the
same moment or right before the freeze of db9:
2013-06-01 07:30:18
+----------------------------+----------------+
| wsrep_apply_oooe | 0.001606 |
| wsrep_apply_oool | 0 |
| wsrep_apply_window | 1.001606 |
| wsrep_causal_reads | 0 |
| wsrep_cert_deps_distance | 31.877193 |
| wsrep_cert_index_size | 14631 |
| wsrep_cluster_conf_id | 17 |
| wsrep_cluster_size | 3 |
| wsrep_cluster_status | 1 |
| wsrep_commit_oooe | 0 |
| wsrep_commit_oool | 0 |
| wsrep_commit_window | 1 |
| wsrep_connected | 1 |
| wsrep_flow_control_paused | 0 |
| wsrep_flow_control_recv | 0 |
| wsrep_flow_control_sent | 0 |
| wsrep_last_committed | 180873400 |
| wsrep_local_bf_aborts | 0 |
| wsrep_local_cert_failures | 60 |
| wsrep_local_commits | 54938540 |
| wsrep_local_index | 1 |
| wsrep_local_recv_queue | 0 |
| wsrep_local_recv_queue_avg | 0 |
| wsrep_local_replays | 0 |
| wsrep_local_send_queue | 0 |
| wsrep_local_send_queue_avg | 0.001408 |
| wsrep_local_state | 4 |
| wsrep_protocol_version | 4 |
| wsrep_ready | 1 |
| wsrep_received | 917059 |
| wsrep_received_bytes | 7338735 |
| wsrep_replicated | 54945722 |
| wsrep_replicated_bytes | 15125028570872 |
+----------------------------+----------------+

and this is db8 around 8 minutes after db9 froze:
2013-06-01 07:38:18
+----------------------------+----------------+
| wsrep_apply_oooe | 0 |
| wsrep_apply_oool | 0 |
| wsrep_apply_window | 0 |
| wsrep_causal_reads | 0 |
| wsrep_cert_deps_distance | 48.768293 |
| wsrep_cert_index_size | 20833 |
| wsrep_cluster_conf_id | 17 |
| wsrep_cluster_size | 3 |
| wsrep_cluster_status | 1 |
| wsrep_commit_oooe | 0 |
| wsrep_commit_oool | 0 |
| wsrep_commit_window | 0 |
| wsrep_connected | 1 |
| wsrep_flow_control_paused | 1 |
| wsrep_flow_control_recv | 0 |
| wsrep_flow_control_sent | 0 |
| wsrep_last_committed | 180873681 |
| wsrep_local_bf_aborts | 0 |
| wsrep_local_cert_failures | 60 |
| wsrep_local_commits | 54938821 |
| wsrep_local_index | 1 |
| wsrep_local_recv_queue | 0 |
| wsrep_local_recv_queue_avg | 0 |
| wsrep_local_replays | 0 |
| wsrep_local_send_queue | 7 |
| wsrep_local_send_queue_avg | 0 |
| wsrep_local_state | 4 |
| wsrep_protocol_version | 4 |
| wsrep_ready | 1 |
| wsrep_received | 917065 |
| wsrep_received_bytes | 7338783 |
| wsrep_replicated | 54946003 |
| wsrep_replicated_bytes | 15125039941061 |
+----------------------------+----------------+



db9@ 2013-05-31 03:50:18
+----------------------------+----------------+
| wsrep_apply_oooe | 0.158879 |
| wsrep_apply_oool | 0 |
| wsrep_apply_window | 1.205607 |
| wsrep_causal_reads | 0 |
| wsrep_cert_deps_distance | 11.133333 |
| wsrep_cert_index_size | 30936 |
| wsrep_cluster_conf_id | 15 |
| wsrep_cluster_size | 3 |
| wsrep_cluster_status | 1 |
| wsrep_commit_oooe | 0 |
| wsrep_commit_oool | 0 |
| wsrep_commit_window | 1.046729 |
| wsrep_connected | 1 |
| wsrep_flow_control_paused | 0.346329 |
| wsrep_flow_control_recv | 0 |
| wsrep_flow_control_sent | 0 |
| wsrep_last_committed | 179518395 |
| wsrep_local_bf_aborts | 0 |
| wsrep_local_cert_failures | 60 |
| wsrep_local_commits | 53583692 |
| wsrep_local_index | 0 |
| wsrep_local_recv_queue | 0 |
| wsrep_local_recv_queue_avg | 0 |
| wsrep_local_replays | 0 |
| wsrep_local_send_queue | 0 |
| wsrep_local_send_queue_avg | 0.082645 |
| wsrep_local_state | 4 |
| wsrep_protocol_version | 4 |
| wsrep_ready | 1 |
| wsrep_received | 894990 |
| wsrep_received_bytes | 7161672 |
| wsrep_replicated | 53590717 |
| wsrep_replicated_bytes | 14768677263714 |
+----------------------------+----------------+

The hanging node is running "empty". It is only used to run some simple
selects on historical tables (using the primary key, these return within
subseconds). No locks, no backup jobs. No backups or locks on db8 either.

The writes all go to the other node (db8) in the cluster.
There are around 25GB per day coming via traditional replication (from
our main production database), max_query_size is 16MB, but the
inserts/updates are all in the area of up to 5MB. Most of them come
between midnight and 8am.
99% of all other writes are 600GB of INSERT ... ON DUPLICATE KEY UPDATEs
on 4-5 tables. These queries are run on db8, they are 99.99% updates.
Query-Size 4MB.

This whole setup was running fine for 2-3 months until db9 was suddenly
frozen 6 weeks ago. A colleague got it up and running again by killing
and restarting mysqld on db9. Now it froze again on friday and saturday.
I couldn't find any other solution but kill mysqld on db9. I don't see
anything has changed regarding the type/size of queries, the workload or
anything else. Actually, we're just having this cluster running without
much load as we didn't really get the time to go into stress testing yet.



Jan

Ilias Bertsimas

unread,
Jun 3, 2013, 12:32:15 PM6/3/13
to codersh...@googlegroups.com
Hi,

I can see large commit times at some point on those logs:
| wsrep_commit_window        |     13.058824 | 
| wsrep_connected            |             1 | 
| wsrep_flow_control_paused  |      0.490027 | 

The logs do not seems to be exactly on the time it is happening. Also flow_control_paused is greater than 0 but not 1.0 yet.
Maybe you need to install pt-stalk and try to capture more data like mysql processlist when it happens.

Does it happen randomly or on specific times ?

Also 
 wsrep_cert_index_size      |          30936 | 

seems quite high... maybe a bug ?


Regards,
Ilias.

Jan Kirchhoff

unread,
Jun 4, 2013, 3:34:57 AM6/4/13
to codersh...@googlegroups.com
Hi,
Am 03.06.2013 18:32, schrieb Ilias Bertsimas:
> Hi,
>
> I can see large commit times at some point on those logs:
> | wsrep_commit_window | 13.058824 |
> | wsrep_connected | 1 |
> | wsrep_flow_control_paused | 0.490027 |
>
> The logs do not seems to be exactly on the time it is happening. Also
> flow_control_paused is greater than 0 but not 1.0 yet.
> Maybe you need to install pt-stalk and try to capture more data like
> mysql processlist when it happens.

no, as I wrote these are the last numbers I have before it happened. The
database would not answer on "show global status" once it hangs.
I'll run pt-stalk next time it happens. But I have no idea if this will
be - maybe tomorrow and maybe next month. This makes it so difficult to
figure put what is causing this.
I already posted the innodb status from when it happened.

regarding wsrep_commit_window: This looks normal to me, we set
wsrep_slave_threads=40 some time ago (some time between the first and
the second crash) and observed a big increase in throughput on our
workload. I think its value was 8 or 10 before.

> Does it happen randomly or on specific times ?

it happend 3 times so far, randomly, but during the nights (around 3am,
3:50 am, 7:30am). Not during the daytime when we have the heavy load on
very few tables but in the time area where lots of end-of-day-data is
archived in big tables (coming via the traditional replication).

>
> Also
> wsrep_cert_index_size | 30936 |
>
> seems quite high... maybe a bug ?
>

what does that number mean? It's a little difficult to find decent
documentation on the web :-(

thanks for your input!
Jan

>
> Regards,
> Ilias.
>
>
> On Monday, June 3, 2013 2:46:13 PM UTC+1, Jan Kirchhoff wrote:
>
> Am 03.06.2013 14:20, schrieb Jay Janssen:
> >
> > On Jun 1, 2013, at 1:43 PM, Ilias Bertsimas <awar...@gmail.com
> <javascript:>

Ilias Bertsimas

unread,
Jun 4, 2013, 7:52:59 AM6/4/13
to codersh...@googlegroups.com
Hi,

Any chance getting show status like 'wsrep%' only ?
I believe pt-stalk might be able to get more info up to the moment it locks up. You can use wsrep_flow_control_paused as a trigger.

So the "master" db8 node gets input from async mysql replication as well ?
Maybe a big transaction that comes through there in the form of a statement could cause this and modify a lot of rows.

I am not 100% sure myself about wsrep_cert_index_size  but from what I have seen seems to be an index of the transactions across the cluster that the node uses to certify new transactions against for conflicts.

Kind Regards,
Ilias. 

Teemu Ollakka

unread,
Jun 4, 2013, 8:07:51 AM6/4/13
to codersh...@googlegroups.com

Perhaps the best way to get more info in this kind of case is to print stack traces using gdb:

gdb --batch -ex "thr apply all bt" -p <mysqld_pid> /usr/sbin/mysqld

Beware that stack traces may contain readable query strings, so better to review them before posting.

- Teemu

Alex Yurchenko

unread,
Jun 4, 2013, 1:08:04 PM6/4/13
to codersh...@googlegroups.com
On 2013-06-04 10:34, Jan Kirchhoff wrote:
>> Also
>> wsrep_cert_index_size | 30936 |
>>
>> seems quite high... maybe a bug ?
>>
>
> what does that number mean? It's a little difficult to find decent
> documentation on the web :-(

It means the number of entries in certification index - a tad high, but
really nothing. in conjunction with a reasonably high dependency window
that means that a lot of writing is going on in the system in parallel.
So far it does not look to be galera related, so yes, the next step is
gdb stack dump. Nothing revealing in the logs.

Jan Kirchhoff

unread,
Jun 6, 2013, 7:04:16 AM6/6/13
to codersh...@googlegroups.com
Alex,

can I find more documentation on all these variables somewhere?
Everything I find on the web (including the codership website) is a
little too brief in my opinion. You can get everything up and running
pretty quick, but once I try to do some tuning or understand problems, I
am a little lost...

I feel bad saying that, but I hope the system is freezing again soon,
it has been running fine for the last few days. I'll get a gdb stack
dump and whatevery information I can gather with ps-stalk as soon as it
happends again (hopefully soon...).

Thanks for the input so far.

Jan

Jan Kirchhoff

unread,
Jun 10, 2013, 8:21:35 AM6/10/13
to codersh...@googlegroups.com
We've had the freeze again this morning at 7:08am:

This is the output of gdb:

[New LWP 31658]
[New LWP 4954]
[New LWP 3578]
[New LWP 3577]
[New LWP 2267]
[New LWP 2266]
[New LWP 29647]
[New LWP 4205]
[New LWP 4189]
[New LWP 4188]
[New LWP 27534]
[New LWP 27515]
[New LWP 27514]
[New LWP 27513]
[New LWP 27512]
[New LWP 27511]
[New LWP 27510]
[New LWP 27509]
[New LWP 27508]
[New LWP 27507]
[New LWP 27506]
[New LWP 27505]
[New LWP 27504]
[New LWP 27503]
[New LWP 27502]
[New LWP 27501]
[New LWP 27500]
[New LWP 27499]
[New LWP 27498]
[New LWP 27497]
[New LWP 27496]
[New LWP 27495]
[New LWP 27494]
[New LWP 27493]
[New LWP 27492]
[New LWP 27491]
[New LWP 27490]
[New LWP 27489]
[New LWP 27488]
[New LWP 27487]
[New LWP 27486]
[New LWP 27485]
[New LWP 27484]
[New LWP 27483]
[New LWP 27482]
[New LWP 27481]
[New LWP 27480]
[New LWP 27479]
[New LWP 27478]
[New LWP 27477]
[New LWP 27476]
[New LWP 27475]
[New LWP 27474]
[New LWP 27473]
[New LWP 27472]
[New LWP 27471]
[New LWP 27470]
[New LWP 27469]
[New LWP 27408]
[New LWP 27407]
[New LWP 27406]
[New LWP 27405]
[New LWP 27404]
[New LWP 27403]
[New LWP 27402]
[New LWP 27401]
[New LWP 27400]
[New LWP 27399]
[New LWP 27343]
[New LWP 27277]
[New LWP 27276]
[New LWP 27275]
[New LWP 27274]
[New LWP 27273]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007fb6c3b8f303 in poll () from /lib/x86_64-linux-gnu/libc.so.6

Thread 75 (Thread 0x7fb6c3890700 (LWP 27273)):
#0 0x00007fb6c4470d84 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fb6c25ba1b1 in galera::ServiceThd::thd_func(void*) () from
/usr/lib/galera/libgalera_smm.so
#2 0x00007fb6c446ce9a in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#3 0x00007fb6c3b9acbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#4 0x0000000000000000 in ?? ()

Thread 74 (Thread 0x7fb43bffe700 (LWP 27274)):
#0 0x00007fb6c3b9b353 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fb6c258c4d9 in asio::detail::epoll_reactor::run(bool,
asio::detail::op_queue<asio::detail::task_io_service_operation>&) ()
from /usr/lib/galera/libgalera_smm.so
#2 0x00007fb6c259136f in
asio::detail::task_io_service::run(asio::error_code&) () from
/usr/lib/galera/libgalera_smm.so
#3 0x00007fb6c2587d87 in
gcomm::AsioProtonet::event_loop(gu::datetime::Period const&) () from
/usr/lib/galera/libgalera_smm.so
#4 0x00007fb6c25a3808 in GCommConn::run() () from
/usr/lib/galera/libgalera_smm.so
#5 0x00007fb6c25a75c9 in GCommConn::run_fn(void*) () from
/usr/lib/galera/libgalera_smm.so
#6 0x00007fb6c446ce9a in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#7 0x00007fb6c3b9acbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#8 0x0000000000000000 in ?? ()

Thread 73 (Thread 0x7fb43b7fd700 (LWP 27275)):
#0 0x00007fb6c4470d84 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fb6c25a2f47 in ?? () from /usr/lib/galera/libgalera_smm.so
#2 0x00007fb6c2599135 in gcs_core_recv () from
/usr/lib/galera/libgalera_smm.so
#3 0x00007fb6c259f181 in ?? () from /usr/lib/galera/libgalera_smm.so
#4 0x00007fb6c446ce9a in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#5 0x00007fb6c3b9acbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 72 (Thread 0x7fb43affc700 (LWP 27276)):
#0 0x00007fb6c4470d84 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fb6c5ab21eb in wsrep_rollback_process(THD*) ()
#2 0x00007fb6c5a2d71b in start_wsrep_THD ()
#3 0x00007fb6c446ce9a in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007fb6c3b9acbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#5 0x0000000000000000 in ?? ()

Thread 71 (Thread 0x7fb43afb3700 (LWP 27277)):
#0 0x00007fb6c4470d84 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fb6c25e9eb0 in
galera::Monitor<galera::ReplicatorSMM::LocalOrder>::enter(galera::ReplicatorSMM::LocalOrder&)
() from /usr/lib/galera/libgalera_smm.so
#2 0x00007fb6c25e5a9f in
galera::ReplicatorSMM::cert(galera::TrxHandle*) () from
/usr/lib/galera/libgalera_smm.so
#3 0x00007fb6c25e77b1 in galera::ReplicatorSMM::process_trx(void*,
galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.so
#4 0x00007fb6c25be124 in galera::GcsActionSource::dispatch(void*,
gcs_action const&) () from /usr/lib/galera/libgalera_smm.so
#5 0x00007fb6c25be39a in galera::GcsActionSource::process(void*) ()
from /usr/lib/galera/libgalera_smm.so
#6 0x00007fb6c25df175 in galera::ReplicatorSMM::async_recv(void*) ()
from /usr/lib/galera/libgalera_smm.so
#7 0x00007fb6c25f6f93 in galera_recv () from
/usr/lib/galera/libgalera_smm.so
#8 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*) ()
#9 0x00007fb6c5a2d71b in start_wsrep_THD ()
#10 0x00007fb6c446ce9a in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#11 0x00007fb6c3b9acbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#12 0x0000000000000000 in ?? ()

Thread 70 (Thread 0x7fb43af6a700 (LWP 27343)):
#0 0x00007fb6c44710fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fb6c5d60152 in ?? ()
#2 0x00007fb6c5d58237 in ?? ()
#3 0x00007fb6c446ce9a in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007fb6c3b9acbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#5 0x0000000000000000 in ?? ()

Thread 69 (Thread 0x7f99e0f8a700 (LWP 27399)):
#0 0x00007fb6c5f27124 in __io_getevents_0_4 ()
#1 0x00007fb6c5eff694 in ?? ()
#2 0x00007fb6c5eaed1a in ?? ()
#3 0x00007fb6c5e199a8 in ?? ()
#4 0x00007fb6c446ce9a in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#5 0x00007fb6c3b9acbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 68 (Thread 0x7f99e0789700 (LWP 27400)):
#0 0x00007fb6c5f27124 in __io_getevents_0_4 ()
#1 0x00007fb6c5eff694 in ?? ()
#2 0x00007fb6c5eaed1a in ?? ()
#3 0x00007fb6c5e199a8 in ?? ()
#4 0x00007fb6c446ce9a in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#5 0x00007fb6c3b9acbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 67 (Thread 0x7f99dff88700 (LWP 27401)):
#0 0x00007fb6c5f27124 in __io_getevents_0_4 ()
#1 0x00007fb6c5eff694 in ?? ()
#2 0x00007fb6c5eaed1a in ?? ()
#3 0x00007fb6c5e199a8 in ?? ()
#4 0x00007fb6c446ce9a in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#5 0x00007fb6c3b9acbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 66 (Thread 0x7f99df787700 (LWP 27402)):
#0 0x00007fb6c5f27124 in __io_getevents_0_4 ()
#1 0x00007fb6c5eff694 in ?? ()
#2 0x00007fb6c5eaed1a in ?? ()
#3 0x00007fb6c5e199a8 in ?? ()
#4 0x00007fb6c446ce9a in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#5 0x00007fb6c3b9acbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 65 (Thread 0x7f99def86700 (LWP 27403)):
#0 0x00007fb6c5f27124 in __io_getevents_0_4 ()
#1 0x00007fb6c5eff694 in ?? ()
#2 0x00007fb6c5eaed1a in ?? ()
#3 0x00007fb6c5e199a8 in ?? ()
#4 0x00007fb6c446ce9a in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#5 0x00007fb6c3b9acbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 64 (Thread 0x7f99de785700 (LWP 27404)):
#0 0x00007fb6c5f27124 in __io_getevents_0_4 ()
#1 0x00007fb6c5eff694 in ?? ()
#2 0x00007fb6c5eaed1a in ?? ()
#3 0x00007fb6c5e199a8 in ?? ()
#4 0x00007fb6c446ce9a in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#5 0x00007fb6c3b9acbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 63 (Thread 0x7f99ddf84700 (LWP 27405)):
#0 0x00007fb6c5f27124 in __io_getevents_0_4 ()
#1 0x00007fb6c5eff694 in ?? ()
#2 0x00007fb6c5eaed1a in ?? ()
#3 0x00007fb6c5e199a8 in ?? ()
#4 0x00007fb6c446ce9a in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#5 0x00007fb6c3b9acbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 62 (Thread 0x7f99dd783700 (LWP 27406)):
#0 0x00007fb6c5f27124 in __io_getevents_0_4 ()
#1 0x00007fb6c5eff694 in ?? ()
#2 0x00007fb6c5eaed1a in ?? ()
#3 0x00007fb6c5e199a8 in ?? ()
#4 0x00007fb6c446ce9a in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#5 0x00007fb6c3b9acbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 61 (Thread 0x7f99dcf82700 (LWP 27407)):
#0 0x00007fb6c5f27124 in __io_getevents_0_4 ()
#1 0x00007fb6c5eff694 in ?? ()
#2 0x00007fb6c5eaed1a in ?? ()
#3 0x00007fb6c5e199a8 in ?? ()
#4 0x00007fb6c446ce9a in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#5 0x00007fb6c3b9acbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 60 (Thread 0x7f99dc781700 (LWP 27408)):
#0 0x00007fb6c5f27124 in __io_getevents_0_4 ()
#1 0x00007fb6c5eff694 in ?? ()
#2 0x00007fb6c5eaed1a in ?? ()
#3 0x00007fb6c5e199a8 in ?? ()
#4 0x00007fb6c446ce9a in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#5 0x00007fb6c3b9acbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 59 (Thread 0x7f99dbf80700 (LWP 27469)):
#0 0x00007fb6c44710fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fb6c5f015eb in ?? ()
#2 0x00007fb6c5e16678 in ?? ()
#3 0x00007fb6c446ce9a in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007fb6c3b9acbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#5 0x0000000000000000 in ?? ()

Thread 58 (Thread 0x7f99db77f700 (LWP 27470)):
#0 0x00007fb6c44710fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fb6c5f015eb in ?? ()
#2 0x00007fb6c5e16ceb in ?? ()
#3 0x00007fb6c446ce9a in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007fb6c3b9acbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#5 0x0000000000000000 in ?? ()

Thread 57 (Thread 0x7f99daf7e700 (LWP 27471)):
#0 0x00007fb6c44710fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fb6c5f015eb in ?? ()
#2 0x00007fb6c5e1635b in ?? ()
#3 0x00007fb6c446ce9a in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007fb6c3b9acbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#5 0x0000000000000000 in ?? ()

Thread 56 (Thread 0x7f99da77d700 (LWP 27472)):
#0 0x00007fb6c44710fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fb6c5f015eb in ?? ()
#2 0x00007fb6c5e17083 in ?? ()
#3 0x00007fb6c446ce9a in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007fb6c3b9acbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#5 0x0000000000000000 in ?? ()

Thread 55 (Thread 0x7f99d9f7c700 (LWP 27473)):
#0 0x00007fb6c4470d84 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fb6c5f014e1 in ?? ()
#2 0x00007fb6c5e17ba2 in ?? ()
#3 0x00007fb6c446ce9a in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007fb6c3b9acbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#5 0x0000000000000000 in ?? ()

Thread 54 (Thread 0x7f99d977b700 (LWP 27474)):
#0 0x00007fb6c4470d84 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fb6c5f014e1 in ?? ()
#2 0x00007fb6c5e195e6 in ?? ()
#3 0x00007fb6c446ce9a in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007fb6c3b9acbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#5 0x0000000000000000 in ?? ()

Thread 53 (Thread 0x7fb6c5572700 (LWP 27475)):
#0 0x00007fb6c44749f7 in do_sigwait () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fb6c4474a79 in sigwait () from
/lib/x86_64-linux-gnu/libpthread.so.0
#2 0x00007fb6c5a2b56f in signal_hand ()
#3 0x00007fb6c446ce9a in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007fb6c3b9acbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#5 0x0000000000000000 in ?? ()

Thread 52 (Thread 0x7fb6c0065700 (LWP 27476)):
#0 0x00007fb6c447389c in __lll_lock_wait () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fb6c446f065 in _L_lock_858 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#2 0x00007fb6c446eeba in pthread_mutex_lock () from
/lib/x86_64-linux-gnu/libpthread.so.0
#3 0x00007fb6c24d6457 in gu::Lock::Lock(gu::Mutex const&) () from
/usr/lib/galera/libgalera_smm.so
#4 0x00007fb6c25b2c91 in
galera::Certification::set_trx_committed(galera::TrxHandle*) () from
/usr/lib/galera/libgalera_smm.so
#5 0x00007fb6c25e72c2 in galera::ReplicatorSMM::apply_trx(void*,
galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.so
#6 0x00007fb6c25e77c5 in galera::ReplicatorSMM::process_trx(void*,
galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.so
#7 0x00007fb6c25be124 in galera::GcsActionSource::dispatch(void*,
gcs_action const&) () from /usr/lib/galera/libgalera_smm.so
#8 0x00007fb6c25be39a in galera::GcsActionSource::process(void*) ()
from /usr/lib/galera/libgalera_smm.so
#9 0x00007fb6c25df175 in galera::ReplicatorSMM::async_recv(void*) ()
from /usr/lib/galera/libgalera_smm.so
#10 0x00007fb6c25f6f93 in galera_recv () from
/usr/lib/galera/libgalera_smm.so
#11 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*) ()
#12 0x00007fb6c5a2d71b in start_wsrep_THD ()
#13 0x00007fb6c446ce9a in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#14 0x00007fb6c3b9acbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#15 0x0000000000000000 in ?? ()

Thread 51 (Thread 0x7fb4381d3700 (LWP 27477)):
#0 0x00007fb6c447389c in __lll_lock_wait () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fb6c446f065 in _L_lock_858 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#2 0x00007fb6c446eeba in pthread_mutex_lock () from
/lib/x86_64-linux-gnu/libpthread.so.0
#3 0x00007fb6c24d6457 in gu::Lock::Lock(gu::Mutex const&) () from
/usr/lib/galera/libgalera_smm.so
#4 0x00007fb6c25b2c91 in
galera::Certification::set_trx_committed(galera::TrxHandle*) () from
/usr/lib/galera/libgalera_smm.so
#5 0x00007fb6c25e72c2 in galera::ReplicatorSMM::apply_trx(void*,
galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.so
#6 0x00007fb6c25e77c5 in galera::ReplicatorSMM::process_trx(void*,
galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.so
#7 0x00007fb6c25be124 in galera::GcsActionSource::dispatch(void*,
gcs_action const&) () from /usr/lib/galera/libgalera_smm.so
#8 0x00007fb6c25be39a in galera::GcsActionSource::process(void*) ()
from /usr/lib/galera/libgalera_smm.so
#9 0x00007fb6c25df175 in galera::ReplicatorSMM::async_recv(void*) ()
from /usr/lib/galera/libgalera_smm.so
#10 0x00007fb6c25f6f93 in galera_recv () from
/usr/lib/galera/libgalera_smm.so
#11 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*) ()
#12 0x00007fb6c5a2d71b in start_wsrep_THD ()
#13 0x00007fb6c446ce9a in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#14 0x00007fb6c3b9acbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#15 0x0000000000000000 in ?? ()

Thread 50 (Thread 0x7fb43818a700 (LWP 27478)):
#0 0x00007fb6c447389c in __lll_lock_wait () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fb6c446f065 in _L_lock_858 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#2 0x00007fb6c446eeba in pthread_mutex_lock () from
/lib/x86_64-linux-gnu/libpthread.so.0
#3 0x00007fb6c24d6457 in gu::Lock::Lock(gu::Mutex const&) () from
/usr/lib/galera/libgalera_smm.so
#4 0x00007fb6c25b2c91 in
galera::Certification::set_trx_committed(galera::TrxHandle*) () from
/usr/lib/galera/libgalera_smm.so
#5 0x00007fb6c25e72c2 in galera::ReplicatorSMM::apply_trx(void*,
galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.so
#6 0x00007fb6c25e77c5 in galera::ReplicatorSMM::process_trx(void*,
galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.so
#7 0x00007fb6c25be124 in galera::GcsActionSource::dispatch(void*,
gcs_action const&) () from /usr/lib/galera/libgalera_smm.so
#8 0x00007fb6c25be39a in galera::GcsActionSource::process(void*) ()
from /usr/lib/galera/libgalera_smm.so
#9 0x00007fb6c25df175 in galera::ReplicatorSMM::async_recv(void*) ()
from /usr/lib/galera/libgalera_smm.so
#10 0x00007fb6c25f6f93 in galera_recv () from
/usr/lib/galera/libgalera_smm.so
#11 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*) ()
#12 0x00007fb6c5a2d71b in start_wsrep_THD ()
#13 0x00007fb6c446ce9a in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#14 0x00007fb6c3b9acbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#15 0x0000000000000000 in ?? ()

Thread 49 (Thread 0x7fb438141700 (LWP 27479)):
#0 0x00007fb6c447389c in __lll_lock_wait () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fb6c446f065 in _L_lock_858 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#2 0x00007fb6c446eeba in pthread_mutex_lock () from
/lib/x86_64-linux-gnu/libpthread.so.0
#3 0x00007fb6c24d6457 in gu::Lock::Lock(gu::Mutex const&) () from
/usr/lib/galera/libgalera_smm.so
#4 0x00007fb6c25b2c91 in
galera::Certification::set_trx_committed(galera::TrxHandle*) () from
/usr/lib/galera/libgalera_smm.so
#5 0x00007fb6c25e72c2 in galera::ReplicatorSMM::apply_trx(void*,
galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.so
#6 0x00007fb6c25e77c5 in galera::ReplicatorSMM::process_trx(void*,
galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.so
#7 0x00007fb6c25be124 in galera::GcsActionSource::dispatch(void*,
gcs_action const&) () from /usr/lib/galera/libgalera_smm.so
#8 0x00007fb6c25be39a in galera::GcsActionSource::process(void*) ()
from /usr/lib/galera/libgalera_smm.so
#9 0x00007fb6c25df175 in galera::ReplicatorSMM::async_recv(void*) ()
from /usr/lib/galera/libgalera_smm.so
#10 0x00007fb6c25f6f93 in galera_recv () from
/usr/lib/galera/libgalera_smm.so
#11 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*) ()
#12 0x00007fb6c5a2d71b in start_wsrep_THD ()
#13 0x00007fb6c446ce9a in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#14 0x00007fb6c3b9acbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#15 0x0000000000000000 in ?? ()

Thread 48 (Thread 0x7fb4380f8700 (LWP 27480)):
#0 0x00007fb6c447389c in __lll_lock_wait () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fb6c446f065 in _L_lock_858 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#2 0x00007fb6c446eeba in pthread_mutex_lock () from
/lib/x86_64-linux-gnu/libpthread.so.0
#3 0x00007fb6c24d6457 in gu::Lock::Lock(gu::Mutex const&) () from
/usr/lib/galera/libgalera_smm.so
#4 0x00007fb6c25b2c91 in
galera::Certification::set_trx_committed(galera::TrxHandle*) () from
/usr/lib/galera/libgalera_smm.so
#5 0x00007fb6c25e72c2 in galera::ReplicatorSMM::apply_trx(void*,
galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.so
#6 0x00007fb6c25e77c5 in galera::ReplicatorSMM::process_trx(void*,
galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.so
#7 0x00007fb6c25be124 in galera::GcsActionSource::dispatch(void*,
gcs_action const&) () from /usr/lib/galera/libgalera_smm.so
#8 0x00007fb6c25be39a in galera::GcsActionSource::process(void*) ()
from /usr/lib/galera/libgalera_smm.so
#9 0x00007fb6c25df175 in galera::ReplicatorSMM::async_recv(void*) ()
from /usr/lib/galera/libgalera_smm.so
#10 0x00007fb6c25f6f93 in galera_recv () from
/usr/lib/galera/libgalera_smm.so
#11 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*) ()
#12 0x00007fb6c5a2d71b in start_wsrep_THD ()
#13 0x00007fb6c446ce9a in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#14 0x00007fb6c3b9acbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#15 0x0000000000000000 in ?? ()

Thread 47 (Thread 0x7fb4380af700 (LWP 27481)):
#0 0x00007fb6c447389c in __lll_lock_wait () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fb6c446f065 in _L_lock_858 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#2 0x00007fb6c446eeba in pthread_mutex_lock () from
/lib/x86_64-linux-gnu/libpthread.so.0
#3 0x00007fb6c24d6457 in gu::Lock::Lock(gu::Mutex const&) () from
/usr/lib/galera/libgalera_smm.so
#4 0x00007fb6c25b2c91 in
galera::Certification::set_trx_committed(galera::TrxHandle*) () from
/usr/lib/galera/libgalera_smm.so
#5 0x00007fb6c25e72c2 in galera::ReplicatorSMM::apply_trx(void*,
galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.so
#6 0x00007fb6c25e77c5 in galera::ReplicatorSMM::process_trx(void*,
galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.so
#7 0x00007fb6c25be124 in galera::GcsActionSource::dispatch(void*,
gcs_action const&) () from /usr/lib/galera/libgalera_smm.so
#8 0x00007fb6c25be39a in galera::GcsActionSource::process(void*) ()
from /usr/lib/galera/libgalera_smm.so
#9 0x00007fb6c25df175 in galera::ReplicatorSMM::async_recv(void*) ()
from /usr/lib/galera/libgalera_smm.so
#10 0x00007fb6c25f6f93 in galera_recv () from
/usr/lib/galera/libgalera_smm.so
#11 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*) ()
#12 0x00007fb6c5a2d71b in start_wsrep_THD ()
#13 0x00007fb6c446ce9a in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#14 0x00007fb6c3b9acbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#15 0x0000000000000000 in ?? ()

Thread 46 (Thread 0x7fb438066700 (LWP 27482)):
#0 0x00007fb6c4470d84 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fb6c25e9eb0 in
galera::Monitor<galera::ReplicatorSMM::LocalOrder>::enter(galera::ReplicatorSMM::LocalOrder&)
() from /usr/lib/galera/libgalera_smm.so
#2 0x00007fb6c25e5a9f in
galera::ReplicatorSMM::cert(galera::TrxHandle*) () from
/usr/lib/galera/libgalera_smm.so
#3 0x00007fb6c25e77b1 in galera::ReplicatorSMM::process_trx(void*,
galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.so
#4 0x00007fb6c25be124 in galera::GcsActionSource::dispatch(void*,
gcs_action const&) () from /usr/lib/galera/libgalera_smm.so
#5 0x00007fb6c25be39a in galera::GcsActionSource::process(void*) ()
from /usr/lib/galera/libgalera_smm.so
#6 0x00007fb6c25df175 in galera::ReplicatorSMM::async_recv(void*) ()
from /usr/lib/galera/libgalera_smm.so
#7 0x00007fb6c25f6f93 in galera_recv () from
/usr/lib/galera/libgalera_smm.so
#8 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*) ()
#9 0x00007fb6c5a2d71b in start_wsrep_THD ()
#10 0x00007fb6c446ce9a in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#11 0x00007fb6c3b9acbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#12 0x0000000000000000 in ?? ()

Thread 45 (Thread 0x7f99d8f7a700 (LWP 27483)):
#0 0x00007fb6c4470d84 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fb6c25e9eb0 in
galera::Monitor<galera::ReplicatorSMM::LocalOrder>::enter(galera::ReplicatorSMM::LocalOrder&)
() from /usr/lib/galera/libgalera_smm.so
#2 0x00007fb6c25e5a9f in
galera::ReplicatorSMM::cert(galera::TrxHandle*) () from
/usr/lib/galera/libgalera_smm.so
#3 0x00007fb6c25e77b1 in galera::ReplicatorSMM::process_trx(void*,
galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.so
#4 0x00007fb6c25be124 in galera::GcsActionSource::dispatch(void*,
gcs_action const&) () from /usr/lib/galera/libgalera_smm.so
#5 0x00007fb6c25be39a in galera::GcsActionSource::process(void*) ()
from /usr/lib/galera/libgalera_smm.so
#6 0x00007fb6c25df175 in galera::ReplicatorSMM::async_recv(void*) ()
from /usr/lib/galera/libgalera_smm.so
#7 0x00007fb6c25f6f93 in galera_recv () from
/usr/lib/galera/libgalera_smm.so
#8 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*) ()
#9 0x00007fb6c5a2d71b in start_wsrep_THD ()
#10 0x00007fb6c446ce9a in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#11 0x00007fb6c3b9acbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#12 0x0000000000000000 in ?? ()

Thread 44 (Thread 0x7f99d8f31700 (LWP 27484)):
#0 0x00007fb6c4470d84 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fb6c25e9eb0 in
galera::Monitor<galera::ReplicatorSMM::LocalOrder>::enter(galera::ReplicatorSMM::LocalOrder&)
() from /usr/lib/galera/libgalera_smm.so
#2 0x00007fb6c25e5a9f in
galera::ReplicatorSMM::cert(galera::TrxHandle*) () from
/usr/lib/galera/libgalera_smm.so
#3 0x00007fb6c25e77b1 in galera::ReplicatorSMM::process_trx(void*,
galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.so
#4 0x00007fb6c25be124 in galera::GcsActionSource::dispatch(void*,
gcs_action const&) () from /usr/lib/galera/libgalera_smm.so
#5 0x00007fb6c25be39a in galera::GcsActionSource::process(void*) ()
from /usr/lib/galera/libgalera_smm.so
#6 0x00007fb6c25df175 in galera::ReplicatorSMM::async_recv(void*) ()
from /usr/lib/galera/libgalera_smm.so
#7 0x00007fb6c25f6f93 in galera_recv () from
/usr/lib/galera/libgalera_smm.so
#8 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*) ()
#9 0x00007fb6c5a2d71b in start_wsrep_THD ()
#10 0x00007fb6c446ce9a in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#11 0x00007fb6c3b9acbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#12 0x0000000000000000 in ?? ()

Thread 43 (Thread 0x7f99d8ee8700 (LWP 27485)):
#0 0x00007fb6c447389c in __lll_lock_wait () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fb6c446f065 in _L_lock_858 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#2 0x00007fb6c446eeba in pthread_mutex_lock () from
/lib/x86_64-linux-gnu/libpthread.so.0
#3 0x00007fb6c24d6457 in gu::Lock::Lock(gu::Mutex const&) () from
/usr/lib/galera/libgalera_smm.so
#4 0x00007fb6c25b2c91 in
galera::Certification::set_trx_committed(galera::TrxHandle*) () from
/usr/lib/galera/libgalera_smm.so
#5 0x00007fb6c25e72c2 in galera::ReplicatorSMM::apply_trx(void*,
galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.so
#6 0x00007fb6c25e77c5 in galera::ReplicatorSMM::process_trx(void*,
galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.so
#7 0x00007fb6c25be124 in galera::GcsActionSource::dispatch(void*,
gcs_action const&) () from /usr/lib/galera/libgalera_smm.so
#8 0x00007fb6c25be39a in galera::GcsActionSource::process(void*) ()
from /usr/lib/galera/libgalera_smm.so
#9 0x00007fb6c25df175 in galera::ReplicatorSMM::async_recv(void*) ()
from /usr/lib/galera/libgalera_smm.so
#10 0x00007fb6c25f6f93 in galera_recv () from
/usr/lib/galera/libgalera_smm.so
#11 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*) ()
#12 0x00007fb6c5a2d71b in start_wsrep_THD ()
#13 0x00007fb6c446ce9a in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#14 0x00007fb6c3b9acbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#15 0x0000000000000000 in ?? ()

Thread 42 (Thread 0x7f99d8e9f700 (LWP 27486)):
#0 0x00007fb6c447389c in __lll_lock_wait () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fb6c446f065 in _L_lock_858 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#2 0x00007fb6c446eeba in pthread_mutex_lock () from
/lib/x86_64-linux-gnu/libpthread.so.0
#3 0x00007fb6c24d6457 in gu::Lock::Lock(gu::Mutex const&) () from
/usr/lib/galera/libgalera_smm.so
#4 0x00007fb6c25b2c91 in
galera::Certification::set_trx_committed(galera::TrxHandle*) () from
/usr/lib/galera/libgalera_smm.so
#5 0x00007fb6c25e72c2 in galera::ReplicatorSMM::apply_trx(void*,
galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.so
#6 0x00007fb6c25e77c5 in galera::ReplicatorSMM::process_trx(void*,
galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.so
#7 0x00007fb6c25be124 in galera::GcsActionSource::dispatch(void*,
gcs_action const&) () from /usr/lib/galera/libgalera_smm.so
#8 0x00007fb6c25be39a in galera::GcsActionSource::process(void*) ()
from /usr/lib/galera/libgalera_smm.so
#9 0x00007fb6c25df175 in galera::ReplicatorSMM::async_recv(void*) ()
from /usr/lib/galera/libgalera_smm.so
#10 0x00007fb6c25f6f93 in galera_recv () from
/usr/lib/galera/libgalera_smm.so
#11 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*) ()
#12 0x00007fb6c5a2d71b in start_wsrep_THD ()
#13 0x00007fb6c446ce9a in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#14 0x00007fb6c3b9acbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#15 0x0000000000000000 in ?? ()

Thread 41 (Thread 0x7f99d8e56700 (LWP 27487)):
#0 0x00007fb6c4470d84 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fb6c25e9eb0 in
galera::Monitor<galera::ReplicatorSMM::LocalOrder>::enter(galera::ReplicatorSMM::LocalOrder&)
() from /usr/lib/galera/libgalera_smm.so
#2 0x00007fb6c25e5a9f in
galera::ReplicatorSMM::cert(galera::TrxHandle*) () from
/usr/lib/galera/libgalera_smm.so
#3 0x00007fb6c25e77b1 in galera::ReplicatorSMM::process_trx(void*,
galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.so
#4 0x00007fb6c25be124 in galera::GcsActionSource::dispatch(void*,
gcs_action const&) () from /usr/lib/galera/libgalera_smm.so
#5 0x00007fb6c25be39a in galera::GcsActionSource::process(void*) ()
from /usr/lib/galera/libgalera_smm.so
#6 0x00007fb6c25df175 in galera::ReplicatorSMM::async_recv(void*) ()
from /usr/lib/galera/libgalera_smm.so
#7 0x00007fb6c25f6f93 in galera_recv () from
/usr/lib/galera/libgalera_smm.so
#8 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*) ()
#9 0x00007fb6c5a2d71b in start_wsrep_THD ()
#10 0x00007fb6c446ce9a in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#11 0x00007fb6c3b9acbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#12 0x0000000000000000 in ?? ()

Thread 40 (Thread 0x7f99d8e0d700 (LWP 27488)):
#0 0x00007fb6c447389c in __lll_lock_wait () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fb6c446f065 in _L_lock_858 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#2 0x00007fb6c446eeba in pthread_mutex_lock () from
/lib/x86_64-linux-gnu/libpthread.so.0
#3 0x00007fb6c24d6457 in gu::Lock::Lock(gu::Mutex const&) () from
/usr/lib/galera/libgalera_smm.so
#4 0x00007fb6c25b2c91 in
galera::Certification::set_trx_committed(galera::TrxHandle*) () from
/usr/lib/galera/libgalera_smm.so
#5 0x00007fb6c25e72c2 in galera::ReplicatorSMM::apply_trx(void*,
galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.so
#6 0x00007fb6c25e77c5 in galera::ReplicatorSMM::process_trx(void*,
galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.so
#7 0x00007fb6c25be124 in galera::GcsActionSource::dispatch(void*,
gcs_action const&) () from /usr/lib/galera/libgalera_smm.so
#8 0x00007fb6c25be39a in galera::GcsActionSource::process(void*) ()
from /usr/lib/galera/libgalera_smm.so
#9 0x00007fb6c25df175 in galera::ReplicatorSMM::async_recv(void*) ()
from /usr/lib/galera/libgalera_smm.so
#10 0x00007fb6c25f6f93 in galera_recv () from
/usr/lib/galera/libgalera_smm.so
#11 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*) ()
#12 0x00007fb6c5a2d71b in start_wsrep_THD ()
#13 0x00007fb6c446ce9a in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#14 0x00007fb6c3b9acbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#15 0x0000000000000000 in ?? ()

Thread 39 (Thread 0x7f99d8dc4700 (LWP 27489)):
#0 0x00007fb6c447389c in __lll_lock_wait () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fb6c446f065 in _L_lock_858 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#2 0x00007fb6c446eeba in pthread_mutex_lock () from
/lib/x86_64-linux-gnu/libpthread.so.0
#3 0x00007fb6c24d6457 in gu::Lock::Lock(gu::Mutex const&) () from
/usr/lib/galera/libgalera_smm.so
#4 0x00007fb6c25b2c91 in
galera::Certification::set_trx_committed(galera::TrxHandle*) () from
/usr/lib/galera/libgalera_smm.so
#5 0x00007fb6c25e72c2 in galera::ReplicatorSMM::apply_trx(void*,
galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.so
#6 0x00007fb6c25e77c5 in galera::ReplicatorSMM::process_trx(void*,
galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.so
#7 0x00007fb6c25be124 in galera::GcsActionSource::dispatch(void*,
gcs_action const&) () from /usr/lib/galera/libgalera_smm.so
#8 0x00007fb6c25be39a in galera::GcsActionSource::process(void*) ()
from /usr/lib/galera/libgalera_smm.so
#9 0x00007fb6c25df175 in galera::ReplicatorSMM::async_recv(void*) ()
from /usr/lib/galera/libgalera_smm.so
#10 0x00007fb6c25f6f93 in galera_recv () from
/usr/lib/galera/libgalera_smm.so
#11 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*) ()
#12 0x00007fb6c5a2d71b in start_wsrep_THD ()
#13 0x00007fb6c446ce9a in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#14 0x00007fb6c3b9acbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#15 0x0000000000000000 in ?? ()

Thread 38 (Thread 0x7f99d8d7b700 (LWP 27490)):
#0 0x00007fb6c4470d84 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fb6c25e9eb0 in
galera::Monitor<galera::ReplicatorSMM::LocalOrder>::enter(galera::ReplicatorSMM::LocalOrder&)
() from /usr/lib/galera/libgalera_smm.so
#2 0x00007fb6c25e5a9f in
galera::ReplicatorSMM::cert(galera::TrxHandle*) () from
/usr/lib/galera/libgalera_smm.so
#3 0x00007fb6c25e77b1 in galera::ReplicatorSMM::process_trx(void*,
galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.so
#4 0x00007fb6c25be124 in galera::GcsActionSource::dispatch(void*,
gcs_action const&) () from /usr/lib/galera/libgalera_smm.so
#5 0x00007fb6c25be39a in galera::GcsActionSource::process(void*) ()
from /usr/lib/galera/libgalera_smm.so
#6 0x00007fb6c25df175 in galera::ReplicatorSMM::async_recv(void*) ()
from /usr/lib/galera/libgalera_smm.so
#7 0x00007fb6c25f6f93 in galera_recv () from
/usr/lib/galera/libgalera_smm.so
#8 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*) ()
#9 0x00007fb6c5a2d71b in start_wsrep_THD ()
#10 0x00007fb6c446ce9a in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#11 0x00007fb6c3b9acbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#12 0x0000000000000000 in ?? ()

Thread 37 (Thread 0x7f99d8d32700 (LWP 27491)):
#0 0x00007fb6c447389c in __lll_lock_wait () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fb6c446f065 in _L_lock_858 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#2 0x00007fb6c446eeba in pthread_mutex_lock () from
/lib/x86_64-linux-gnu/libpthread.so.0
#3 0x00007fb6c24d6457 in gu::Lock::Lock(gu::Mutex const&) () from
/usr/lib/galera/libgalera_smm.so
#4 0x00007fb6c25b2c91 in
galera::Certification::set_trx_committed(galera::TrxHandle*) () from
/usr/lib/galera/libgalera_smm.so
#5 0x00007fb6c25e72c2 in galera::ReplicatorSMM::apply_trx(void*,
galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.so
#6 0x00007fb6c25e77c5 in galera::ReplicatorSMM::process_trx(void*,
galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.so
#7 0x00007fb6c25be124 in galera::GcsActionSource::dispatch(void*,
gcs_action const&) () from /usr/lib/galera/libgalera_smm.so
#8 0x00007fb6c25be39a in galera::GcsActionSource::process(void*) ()
from /usr/lib/galera/libgalera_smm.so
#9 0x00007fb6c25df175 in galera::ReplicatorSMM::async_recv(void*) ()
from /usr/lib/galera/libgalera_smm.so
#10 0x00007fb6c25f6f93 in galera_recv () from
/usr/lib/galera/libgalera_smm.so
#11 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*) ()
#12 0x00007fb6c5a2d71b in start_wsrep_THD ()
#13 0x00007fb6c446ce9a in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#14 0x00007fb6c3b9acbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#15 0x0000000000000000 in ?? ()

Thread 36 (Thread 0x7f99d8ce9700 (LWP 27492)):
#0 0x00007fb6c447389c in __lll_lock_wait () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fb6c446f065 in _L_lock_858 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#2 0x00007fb6c446eeba in pthread_mutex_lock () from
/lib/x86_64-linux-gnu/libpthread.so.0
#3 0x00007fb6c24d6457 in gu::Lock::Lock(gu::Mutex const&) () from
/usr/lib/galera/libgalera_smm.so
#4 0x00007fb6c25b2c91 in
galera::Certification::set_trx_committed(galera::TrxHandle*) () from
/usr/lib/galera/libgalera_smm.so
#5 0x00007fb6c25e72c2 in galera::ReplicatorSMM::apply_trx(void*,
galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.so
#6 0x00007fb6c25e77c5 in galera::ReplicatorSMM::process_trx(void*,
galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.so
#7 0x00007fb6c25be124 in galera::GcsActionSource::dispatch(void*,
gcs_action const&) () from /usr/lib/galera/libgalera_smm.so
#8 0x00007fb6c25be39a in galera::GcsActionSource::process(void*) ()
from /usr/lib/galera/libgalera_smm.so
#9 0x00007fb6c25df175 in galera::ReplicatorSMM::async_recv(void*) ()
from /usr/lib/galera/libgalera_smm.so
#10 0x00007fb6c25f6f93 in galera_recv () from
/usr/lib/galera/libgalera_smm.so
#11 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*) ()
#12 0x00007fb6c5a2d71b in start_wsrep_THD ()
#13 0x00007fb6c446ce9a in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#14 0x00007fb6c3b9acbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#15 0x0000000000000000 in ?? ()

Thread 35 (Thread 0x7f99d8ca0700 (LWP 27493)):
#0 0x00007fb6c447389c in __lll_lock_wait () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fb6c446f065 in _L_lock_858 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#2 0x00007fb6c446eeba in pthread_mutex_lock () from
/lib/x86_64-linux-gnu/libpthread.so.0
#3 0x00007fb6c24d6457 in gu::Lock::Lock(gu::Mutex const&) () from
/usr/lib/galera/libgalera_smm.so
#4 0x00007fb6c25b2c91 in
galera::Certification::set_trx_committed(galera::TrxHandle*) () from
/usr/lib/galera/libgalera_smm.so
#5 0x00007fb6c25e72c2 in galera::ReplicatorSMM::apply_trx(void*,
galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.so
#6 0x00007fb6c25e77c5 in galera::ReplicatorSMM::process_trx(void*,
galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.so
#7 0x00007fb6c25be124 in galera::GcsActionSource::dispatch(void*,
gcs_action const&) () from /usr/lib/galera/libgalera_smm.so
#8 0x00007fb6c25be39a in galera::GcsActionSource::process(void*) ()
from /usr/lib/galera/libgalera_smm.so
#9 0x00007fb6c25df175 in galera::ReplicatorSMM::async_recv(void*) ()
from /usr/lib/galera/libgalera_smm.so
#10 0x00007fb6c25f6f93 in galera_recv () from
/usr/lib/galera/libgalera_smm.so
#11 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*) ()
#12 0x00007fb6c5a2d71b in start_wsrep_THD ()
#13 0x00007fb6c446ce9a in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#14 0x00007fb6c3b9acbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#15 0x0000000000000000 in ?? ()

Thread 34 (Thread 0x7f99d8c57700 (LWP 27494)):
#0 0x00007fb6c447389c in __lll_lock_wait () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fb6c446f065 in _L_lock_858 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#2 0x00007fb6c446eeba in pthread_mutex_lock () from
/lib/x86_64-linux-gnu/libpthread.so.0
#3 0x00007fb6c24d6457 in gu::Lock::Lock(gu::Mutex const&) () from
/usr/lib/galera/libgalera_smm.so
#4 0x00007fb6c25b2c91 in
galera::Certification::set_trx_committed(galera::TrxHandle*) () from
/usr/lib/galera/libgalera_smm.so
#5 0x00007fb6c25e72c2 in galera::ReplicatorSMM::apply_trx(void*,
galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.so
#6 0x00007fb6c25e77c5 in galera::ReplicatorSMM::process_trx(void*,
galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.so
#7 0x00007fb6c25be124 in galera::GcsActionSource::dispatch(void*,
gcs_action const&) () from /usr/lib/galera/libgalera_smm.so
#8 0x00007fb6c25be39a in galera::GcsActionSource::process(void*) ()
from /usr/lib/galera/libgalera_smm.so
#9 0x00007fb6c25df175 in galera::ReplicatorSMM::async_recv(void*) ()
from /usr/lib/galera/libgalera_smm.so
#10 0x00007fb6c25f6f93 in galera_recv () from
/usr/lib/galera/libgalera_smm.so
#11 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*) ()
#12 0x00007fb6c5a2d71b in start_wsrep_THD ()
#13 0x00007fb6c446ce9a in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#14 0x00007fb6c3b9acbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#15 0x0000000000000000 in ?? ()

Thread 33 (Thread 0x7f99d8c0e700 (LWP 27495)):
#0 0x00007fb6c447389c in __lll_lock_wait () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fb6c446f065 in _L_lock_858 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#2 0x00007fb6c446eeba in pthread_mutex_lock () from
/lib/x86_64-linux-gnu/libpthread.so.0
#3 0x00007fb6c24d6457 in gu::Lock::Lock(gu::Mutex const&) () from
/usr/lib/galera/libgalera_smm.so
#4 0x00007fb6c25b2c91 in
galera::Certification::set_trx_committed(galera::TrxHandle*) () from
/usr/lib/galera/libgalera_smm.so
#5 0x00007fb6c25e72c2 in galera::ReplicatorSMM::apply_trx(void*,
galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.so
#6 0x00007fb6c25e77c5 in galera::ReplicatorSMM::process_trx(void*,
galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.so
#7 0x00007fb6c25be124 in galera::GcsActionSource::dispatch(void*,
gcs_action const&) () from /usr/lib/galera/libgalera_smm.so
#8 0x00007fb6c25be39a in galera::GcsActionSource::process(void*) ()
from /usr/lib/galera/libgalera_smm.so
#9 0x00007fb6c25df175 in galera::ReplicatorSMM::async_recv(void*) ()
from /usr/lib/galera/libgalera_smm.so
#10 0x00007fb6c25f6f93 in galera_recv () from
/usr/lib/galera/libgalera_smm.so
#11 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*) ()
#12 0x00007fb6c5a2d71b in start_wsrep_THD ()
#13 0x00007fb6c446ce9a in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#14 0x00007fb6c3b9acbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#15 0x0000000000000000 in ?? ()

Thread 32 (Thread 0x7f99d8bc5700 (LWP 27496)):
#0 0x00007fb6c4470d84 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fb6c25e9eb0 in
galera::Monitor<galera::ReplicatorSMM::LocalOrder>::enter(galera::ReplicatorSMM::LocalOrder&)
() from /usr/lib/galera/libgalera_smm.so
#2 0x00007fb6c25e5a9f in
galera::ReplicatorSMM::cert(galera::TrxHandle*) () from
/usr/lib/galera/libgalera_smm.so
#3 0x00007fb6c25e77b1 in galera::ReplicatorSMM::process_trx(void*,
galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.so
#4 0x00007fb6c25be124 in galera::GcsActionSource::dispatch(void*,
gcs_action const&) () from /usr/lib/galera/libgalera_smm.so
#5 0x00007fb6c25be39a in galera::GcsActionSource::process(void*) ()
from /usr/lib/galera/libgalera_smm.so
#6 0x00007fb6c25df175 in galera::ReplicatorSMM::async_recv(void*) ()
from /usr/lib/galera/libgalera_smm.so
#7 0x00007fb6c25f6f93 in galera_recv () from
/usr/lib/galera/libgalera_smm.so
#8 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*) ()
#9 0x00007fb6c5a2d71b in start_wsrep_THD ()
#10 0x00007fb6c446ce9a in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#11 0x00007fb6c3b9acbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#12 0x0000000000000000 in ?? ()

Thread 31 (Thread 0x7f99d8b7c700 (LWP 27497)):
#0 0x00007fb6c447389c in __lll_lock_wait () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fb6c446f065 in _L_lock_858 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#2 0x00007fb6c446eeba in pthread_mutex_lock () from
/lib/x86_64-linux-gnu/libpthread.so.0
#3 0x00007fb6c24d6457 in gu::Lock::Lock(gu::Mutex const&) () from
/usr/lib/galera/libgalera_smm.so
#4 0x00007fb6c25b2c91 in
galera::Certification::set_trx_committed(galera::TrxHandle*) () from
/usr/lib/galera/libgalera_smm.so
#5 0x00007fb6c25e72c2 in galera::ReplicatorSMM::apply_trx(void*,
galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.so
#6 0x00007fb6c25e77c5 in galera::ReplicatorSMM::process_trx(void*,
galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.so
#7 0x00007fb6c25be124 in galera::GcsActionSource::dispatch(void*,
gcs_action const&) () from /usr/lib/galera/libgalera_smm.so
#8 0x00007fb6c25be39a in galera::GcsActionSource::process(void*) ()
from /usr/lib/galera/libgalera_smm.so
#9 0x00007fb6c25df175 in galera::ReplicatorSMM::async_recv(void*) ()
from /usr/lib/galera/libgalera_smm.so
#10 0x00007fb6c25f6f93 in galera_recv () from
/usr/lib/galera/libgalera_smm.so
#11 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*) ()
#12 0x00007fb6c5a2d71b in start_wsrep_THD ()
#13 0x00007fb6c446ce9a in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#14 0x00007fb6c3b9acbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#15 0x0000000000000000 in ?? ()

Thread 30 (Thread 0x7f99d8b33700 (LWP 27498)):
#0 0x00007fb6c4470d84 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fb6c25e9eb0 in
galera::Monitor<galera::ReplicatorSMM::LocalOrder>::enter(galera::ReplicatorSMM::LocalOrder&)
() from /usr/lib/galera/libgalera_smm.so
#2 0x00007fb6c25e5a9f in
galera::ReplicatorSMM::cert(galera::TrxHandle*) () from
/usr/lib/galera/libgalera_smm.so
#3 0x00007fb6c25e77b1 in galera::ReplicatorSMM::process_trx(void*,
galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.so
#4 0x00007fb6c25be124 in galera::GcsActionSource::dispatch(void*,
gcs_action const&) () from /usr/lib/galera/libgalera_smm.so
#5 0x00007fb6c25be39a in galera::GcsActionSource::process(void*) ()
from /usr/lib/galera/libgalera_smm.so
#6 0x00007fb6c25df175 in galera::ReplicatorSMM::async_recv(void*) ()
from /usr/lib/galera/libgalera_smm.so
#7 0x00007fb6c25f6f93 in galera_recv () from
/usr/lib/galera/libgalera_smm.so
#8 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*) ()
#9 0x00007fb6c5a2d71b in start_wsrep_THD ()
#10 0x00007fb6c446ce9a in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#11 0x00007fb6c3b9acbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#12 0x0000000000000000 in ?? ()

Thread 29 (Thread 0x7f99d8aea700 (LWP 27499)):
#0 0x00007fb6c447389c in __lll_lock_wait () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fb6c446f065 in _L_lock_858 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#2 0x00007fb6c446eeba in pthread_mutex_lock () from
/lib/x86_64-linux-gnu/libpthread.so.0
#3 0x00007fb6c24d6457 in gu::Lock::Lock(gu::Mutex const&) () from
/usr/lib/galera/libgalera_smm.so
#4 0x00007fb6c25b2c91 in
galera::Certification::set_trx_committed(galera::TrxHandle*) () from
/usr/lib/galera/libgalera_smm.so
#5 0x00007fb6c25e72c2 in galera::ReplicatorSMM::apply_trx(void*,
galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.so
#6 0x00007fb6c25e77c5 in galera::ReplicatorSMM::process_trx(void*,
galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.so
#7 0x00007fb6c25be124 in galera::GcsActionSource::dispatch(void*,
gcs_action const&) () from /usr/lib/galera/libgalera_smm.so
#8 0x00007fb6c25be39a in galera::GcsActionSource::process(void*) ()
from /usr/lib/galera/libgalera_smm.so
#9 0x00007fb6c25df175 in galera::ReplicatorSMM::async_recv(void*) ()
from /usr/lib/galera/libgalera_smm.so
#10 0x00007fb6c25f6f93 in galera_recv () from
/usr/lib/galera/libgalera_smm.so
#11 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*) ()
#12 0x00007fb6c5a2d71b in start_wsrep_THD ()
#13 0x00007fb6c446ce9a in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#14 0x00007fb6c3b9acbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#15 0x0000000000000000 in ?? ()

Thread 28 (Thread 0x7f995d2ef700 (LWP 27500)):
#0 0x00007fb6c447389c in __lll_lock_wait () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fb6c446f065 in _L_lock_858 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#2 0x00007fb6c446eeba in pthread_mutex_lock () from
/lib/x86_64-linux-gnu/libpthread.so.0
#3 0x00007fb6c24d6457 in gu::Lock::Lock(gu::Mutex const&) () from
/usr/lib/galera/libgalera_smm.so
#4 0x00007fb6c25e7300 in galera::ReplicatorSMM::apply_trx(void*,
galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.so
#5 0x00007fb6c25e77c5 in galera::ReplicatorSMM::process_trx(void*,
galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.so
#6 0x00007fb6c25be124 in galera::GcsActionSource::dispatch(void*,
gcs_action const&) () from /usr/lib/galera/libgalera_smm.so
#7 0x00007fb6c25be39a in galera::GcsActionSource::process(void*) ()
from /usr/lib/galera/libgalera_smm.so
#8 0x00007fb6c25df175 in galera::ReplicatorSMM::async_recv(void*) ()
from /usr/lib/galera/libgalera_smm.so
#9 0x00007fb6c25f6f93 in galera_recv () from
/usr/lib/galera/libgalera_smm.so
#10 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*) ()
#11 0x00007fb6c5a2d71b in start_wsrep_THD ()
#12 0x00007fb6c446ce9a in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#13 0x00007fb6c3b9acbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#14 0x0000000000000000 in ?? ()

Thread 27 (Thread 0x7f99d8aa1700 (LWP 27501)):
#0 0x00007fb6c4470d84 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fb6c25e9eb0 in
galera::Monitor<galera::ReplicatorSMM::LocalOrder>::enter(galera::ReplicatorSMM::LocalOrder&)
() from /usr/lib/galera/libgalera_smm.so
#2 0x00007fb6c25e5a9f in
galera::ReplicatorSMM::cert(galera::TrxHandle*) () from
/usr/lib/galera/libgalera_smm.so
#3 0x00007fb6c25e77b1 in galera::ReplicatorSMM::process_trx(void*,
galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.so
#4 0x00007fb6c25be124 in galera::GcsActionSource::dispatch(void*,
gcs_action const&) () from /usr/lib/galera/libgalera_smm.so
#5 0x00007fb6c25be39a in galera::GcsActionSource::process(void*) ()
from /usr/lib/galera/libgalera_smm.so
#6 0x00007fb6c25df175 in galera::ReplicatorSMM::async_recv(void*) ()
from /usr/lib/galera/libgalera_smm.so
#7 0x00007fb6c25f6f93 in galera_recv () from
/usr/lib/galera/libgalera_smm.so
#8 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*) ()
#9 0x00007fb6c5a2d71b in start_wsrep_THD ()
#10 0x00007fb6c446ce9a in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#11 0x00007fb6c3b9acbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#12 0x0000000000000000 in ?? ()

Thread 26 (Thread 0x7f99d8a58700 (LWP 27502)):
#0 0x00007fb6c4470d84 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fb6c25e9eb0 in
galera::Monitor<galera::ReplicatorSMM::LocalOrder>::enter(galera::ReplicatorSMM::LocalOrder&)
() from /usr/lib/galera/libgalera_smm.so
#2 0x00007fb6c25e5a9f in
galera::ReplicatorSMM::cert(galera::TrxHandle*) () from
/usr/lib/galera/libgalera_smm.so
#3 0x00007fb6c25e77b1 in galera::ReplicatorSMM::process_trx(void*,
galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.so
#4 0x00007fb6c25be124 in galera::GcsActionSource::dispatch(void*,
gcs_action const&) () from /usr/lib/galera/libgalera_smm.so
#5 0x00007fb6c25be39a in galera::GcsActionSource::process(void*) ()
from /usr/lib/galera/libgalera_smm.so
#6 0x00007fb6c25df175 in galera::ReplicatorSMM::async_recv(void*) ()
from /usr/lib/galera/libgalera_smm.so
#7 0x00007fb6c25f6f93 in galera_recv () from
/usr/lib/galera/libgalera_smm.so
#8 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*) ()
#9 0x00007fb6c5a2d71b in start_wsrep_THD ()
#10 0x00007fb6c446ce9a in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#11 0x00007fb6c3b9acbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#12 0x0000000000000000 in ?? ()

Thread 25 (Thread 0x7f99d8a0f700 (LWP 27503)):
#0 0x00007fb6c4470d84 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fb6c25e9eb0 in
galera::Monitor<galera::ReplicatorSMM::LocalOrder>::enter(galera::ReplicatorSMM::LocalOrder&)
() from /usr/lib/galera/libgalera_smm.so
#2 0x00007fb6c25e5a9f in
galera::ReplicatorSMM::cert(galera::TrxHandle*) () from
/usr/lib/galera/libgalera_smm.so
#3 0x00007fb6c25e77b1 in galera::ReplicatorSMM::process_trx(void*,
galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.so
#4 0x00007fb6c25be124 in galera::GcsActionSource::dispatch(void*,
gcs_action const&) () from /usr/lib/galera/libgalera_smm.so
#5 0x00007fb6c25be39a in galera::GcsActionSource::process(void*) ()
from /usr/lib/galera/libgalera_smm.so
#6 0x00007fb6c25df175 in galera::ReplicatorSMM::async_recv(void*) ()
from /usr/lib/galera/libgalera_smm.so
#7 0x00007fb6c25f6f93 in galera_recv () from
/usr/lib/galera/libgalera_smm.so
#8 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*) ()
#9 0x00007fb6c5a2d71b in start_wsrep_THD ()
#10 0x00007fb6c446ce9a in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#11 0x00007fb6c3b9acbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#12 0x0000000000000000 in ?? ()

Thread 24 (Thread 0x7f99d89c6700 (LWP 27504)):
#0 0x00007fb6c447389c in __lll_lock_wait () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fb6c446f065 in _L_lock_858 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#2 0x00007fb6c446eeba in pthread_mutex_lock () from
/lib/x86_64-linux-gnu/libpthread.so.0
#3 0x00007fb6c24d6457 in gu::Lock::Lock(gu::Mutex const&) () from
/usr/lib/galera/libgalera_smm.so
#4 0x00007fb6c25b2c91 in
galera::Certification::set_trx_committed(galera::TrxHandle*) () from
/usr/lib/galera/libgalera_smm.so
#5 0x00007fb6c25e72c2 in galera::ReplicatorSMM::apply_trx(void*,
galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.so
#6 0x00007fb6c25e77c5 in galera::ReplicatorSMM::process_trx(void*,
galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.so
#7 0x00007fb6c25be124 in galera::GcsActionSource::dispatch(void*,
gcs_action const&) () from /usr/lib/galera/libgalera_smm.so
#8 0x00007fb6c25be39a in galera::GcsActionSource::process(void*) ()
from /usr/lib/galera/libgalera_smm.so
#9 0x00007fb6c25df175 in galera::ReplicatorSMM::async_recv(void*) ()
from /usr/lib/galera/libgalera_smm.so
#10 0x00007fb6c25f6f93 in galera_recv () from
/usr/lib/galera/libgalera_smm.so
#11 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*) ()
#12 0x00007fb6c5a2d71b in start_wsrep_THD ()
#13 0x00007fb6c446ce9a in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#14 0x00007fb6c3b9acbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#15 0x0000000000000000 in ?? ()

Thread 23 (Thread 0x7f99d897d700 (LWP 27505)):
#0 0x00007fb6c4470d84 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fb6c25e9eb0 in
galera::Monitor<galera::ReplicatorSMM::LocalOrder>::enter(galera::ReplicatorSMM::LocalOrder&)
() from /usr/lib/galera/libgalera_smm.so
#2 0x00007fb6c25e5a9f in
galera::ReplicatorSMM::cert(galera::TrxHandle*) () from
/usr/lib/galera/libgalera_smm.so
#3 0x00007fb6c25e77b1 in galera::ReplicatorSMM::process_trx(void*,
galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.so
#4 0x00007fb6c25be124 in galera::GcsActionSource::dispatch(void*,
gcs_action const&) () from /usr/lib/galera/libgalera_smm.so
#5 0x00007fb6c25be39a in galera::GcsActionSource::process(void*) ()
from /usr/lib/galera/libgalera_smm.so
#6 0x00007fb6c25df175 in galera::ReplicatorSMM::async_recv(void*) ()
from /usr/lib/galera/libgalera_smm.so
#7 0x00007fb6c25f6f93 in galera_recv () from
/usr/lib/galera/libgalera_smm.so
#8 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*) ()
#9 0x00007fb6c5a2d71b in start_wsrep_THD ()
#10 0x00007fb6c446ce9a in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#11 0x00007fb6c3b9acbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#12 0x0000000000000000 in ?? ()

Thread 22 (Thread 0x7f99d8934700 (LWP 27506)):
#0 0x00007fb6c447389c in __lll_lock_wait () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fb6c446f065 in _L_lock_858 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#2 0x00007fb6c446eeba in pthread_mutex_lock () from
/lib/x86_64-linux-gnu/libpthread.so.0
#3 0x00007fb6c24d6457 in gu::Lock::Lock(gu::Mutex const&) () from
/usr/lib/galera/libgalera_smm.so
#4 0x00007fb6c25b2c91 in
galera::Certification::set_trx_committed(galera::TrxHandle*) () from
/usr/lib/galera/libgalera_smm.so
#5 0x00007fb6c25e72c2 in galera::ReplicatorSMM::apply_trx(void*,
galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.so
#6 0x00007fb6c25e77c5 in galera::ReplicatorSMM::process_trx(void*,
galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.so
#7 0x00007fb6c25be124 in galera::GcsActionSource::dispatch(void*,
gcs_action const&) () from /usr/lib/galera/libgalera_smm.so
#8 0x00007fb6c25be39a in galera::GcsActionSource::process(void*) ()
from /usr/lib/galera/libgalera_smm.so
#9 0x00007fb6c25df175 in galera::ReplicatorSMM::async_recv(void*) ()
from /usr/lib/galera/libgalera_smm.so
#10 0x00007fb6c25f6f93 in galera_recv () from
/usr/lib/galera/libgalera_smm.so
#11 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*) ()
#12 0x00007fb6c5a2d71b in start_wsrep_THD ()
#13 0x00007fb6c446ce9a in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#14 0x00007fb6c3b9acbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#15 0x0000000000000000 in ?? ()

Thread 21 (Thread 0x7f99d88eb700 (LWP 27507)):
#0 0x00007fb6c447389c in __lll_lock_wait () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fb6c446f065 in _L_lock_858 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#2 0x00007fb6c446eeba in pthread_mutex_lock () from
/lib/x86_64-linux-gnu/libpthread.so.0
#3 0x00007fb6c25b9c4d in
galera::Certification::PurgeAndDiscard::operator()(std::pair<long const,
galera::TrxHandle*>&) const () from /usr/lib/galera/libgalera_smm.so
#4 0x00007fb6c25b429c in galera::Certification::purge_trxs_upto_(long)
() from /usr/lib/galera/libgalera_smm.so
#5 0x00007fb6c25de292 in
galera::ReplicatorSMM::process_commit_cut(long, long) () from
/usr/lib/galera/libgalera_smm.so
#6 0x00007fb6c25be161 in galera::GcsActionSource::dispatch(void*,
gcs_action const&) () from /usr/lib/galera/libgalera_smm.so
#7 0x00007fb6c25be39a in galera::GcsActionSource::process(void*) ()
from /usr/lib/galera/libgalera_smm.so
#8 0x00007fb6c25df175 in galera::ReplicatorSMM::async_recv(void*) ()
from /usr/lib/galera/libgalera_smm.so
#9 0x00007fb6c25f6f93 in galera_recv () from
/usr/lib/galera/libgalera_smm.so
#10 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*) ()
#11 0x00007fb6c5a2d71b in start_wsrep_THD ()
#12 0x00007fb6c446ce9a in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#13 0x00007fb6c3b9acbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#14 0x0000000000000000 in ?? ()

Thread 20 (Thread 0x7f99d88a2700 (LWP 27508)):
#0 0x00007fb6c447389c in __lll_lock_wait () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fb6c446f065 in _L_lock_858 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#2 0x00007fb6c446eeba in pthread_mutex_lock () from
/lib/x86_64-linux-gnu/libpthread.so.0
#3 0x00007fb6c24d6457 in gu::Lock::Lock(gu::Mutex const&) () from
/usr/lib/galera/libgalera_smm.so
#4 0x00007fb6c25b2c91 in
galera::Certification::set_trx_committed(galera::TrxHandle*) () from
/usr/lib/galera/libgalera_smm.so
#5 0x00007fb6c25e72c2 in galera::ReplicatorSMM::apply_trx(void*,
galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.so
#6 0x00007fb6c25e77c5 in galera::ReplicatorSMM::process_trx(void*,
galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.so
#7 0x00007fb6c25be124 in galera::GcsActionSource::dispatch(void*,
gcs_action const&) () from /usr/lib/galera/libgalera_smm.so
#8 0x00007fb6c25be39a in galera::GcsActionSource::process(void*) ()
from /usr/lib/galera/libgalera_smm.so
#9 0x00007fb6c25df175 in galera::ReplicatorSMM::async_recv(void*) ()
from /usr/lib/galera/libgalera_smm.so
#10 0x00007fb6c25f6f93 in galera_recv () from
/usr/lib/galera/libgalera_smm.so
#11 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*) ()
#12 0x00007fb6c5a2d71b in start_wsrep_THD ()
#13 0x00007fb6c446ce9a in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#14 0x00007fb6c3b9acbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#15 0x0000000000000000 in ?? ()

Thread 19 (Thread 0x7f99d8859700 (LWP 27509)):
#0 0x00007fb6c447389c in __lll_lock_wait () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fb6c446f065 in _L_lock_858 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#2 0x00007fb6c446eeba in pthread_mutex_lock () from
/lib/x86_64-linux-gnu/libpthread.so.0
#3 0x00007fb6c24d6457 in gu::Lock::Lock(gu::Mutex const&) () from
/usr/lib/galera/libgalera_smm.so
#4 0x00007fb6c25b2c91 in
galera::Certification::set_trx_committed(galera::TrxHandle*) () from
/usr/lib/galera/libgalera_smm.so
#5 0x00007fb6c25e72c2 in galera::ReplicatorSMM::apply_trx(void*,
galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.so
#6 0x00007fb6c25e77c5 in galera::ReplicatorSMM::process_trx(void*,
galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.so
#7 0x00007fb6c25be124 in galera::GcsActionSource::dispatch(void*,
gcs_action const&) () from /usr/lib/galera/libgalera_smm.so
#8 0x00007fb6c25be39a in galera::GcsActionSource::process(void*) ()
from /usr/lib/galera/libgalera_smm.so
#9 0x00007fb6c25df175 in galera::ReplicatorSMM::async_recv(void*) ()
from /usr/lib/galera/libgalera_smm.so
#10 0x00007fb6c25f6f93 in galera_recv () from
/usr/lib/galera/libgalera_smm.so
#11 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*) ()
#12 0x00007fb6c5a2d71b in start_wsrep_THD ()
#13 0x00007fb6c446ce9a in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#14 0x00007fb6c3b9acbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#15 0x0000000000000000 in ?? ()

Thread 18 (Thread 0x7f99d8810700 (LWP 27510)):
#0 0x00007fb6c447389c in __lll_lock_wait () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fb6c446f065 in _L_lock_858 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#2 0x00007fb6c446eeba in pthread_mutex_lock () from
/lib/x86_64-linux-gnu/libpthread.so.0
#3 0x00007fb6c24d6457 in gu::Lock::Lock(gu::Mutex const&) () from
/usr/lib/galera/libgalera_smm.so
#4 0x00007fb6c25b2c91 in
galera::Certification::set_trx_committed(galera::TrxHandle*) () from
/usr/lib/galera/libgalera_smm.so
#5 0x00007fb6c25e72c2 in galera::ReplicatorSMM::apply_trx(void*,
galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.so
#6 0x00007fb6c25e77c5 in galera::ReplicatorSMM::process_trx(void*,
galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.so
#7 0x00007fb6c25be124 in galera::GcsActionSource::dispatch(void*,
gcs_action const&) () from /usr/lib/galera/libgalera_smm.so
#8 0x00007fb6c25be39a in galera::GcsActionSource::process(void*) ()
from /usr/lib/galera/libgalera_smm.so
#9 0x00007fb6c25df175 in galera::ReplicatorSMM::async_recv(void*) ()
from /usr/lib/galera/libgalera_smm.so
#10 0x00007fb6c25f6f93 in galera_recv () from
/usr/lib/galera/libgalera_smm.so
#11 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*) ()
#12 0x00007fb6c5a2d71b in start_wsrep_THD ()
#13 0x00007fb6c446ce9a in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#14 0x00007fb6c3b9acbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#15 0x0000000000000000 in ?? ()

Thread 17 (Thread 0x7f99d87c7700 (LWP 27511)):
#0 0x00007fb6c4470d84 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fb6c25e9eb0 in
galera::Monitor<galera::ReplicatorSMM::LocalOrder>::enter(galera::ReplicatorSMM::LocalOrder&)
() from /usr/lib/galera/libgalera_smm.so
#2 0x00007fb6c25e5a9f in
galera::ReplicatorSMM::cert(galera::TrxHandle*) () from
/usr/lib/galera/libgalera_smm.so
#3 0x00007fb6c25e77b1 in galera::ReplicatorSMM::process_trx(void*,
galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.so
#4 0x00007fb6c25be124 in galera::GcsActionSource::dispatch(void*,
gcs_action const&) () from /usr/lib/galera/libgalera_smm.so
#5 0x00007fb6c25be39a in galera::GcsActionSource::process(void*) ()
from /usr/lib/galera/libgalera_smm.so
#6 0x00007fb6c25df175 in galera::ReplicatorSMM::async_recv(void*) ()
from /usr/lib/galera/libgalera_smm.so
#7 0x00007fb6c25f6f93 in galera_recv () from
/usr/lib/galera/libgalera_smm.so
#8 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*) ()
#9 0x00007fb6c5a2d71b in start_wsrep_THD ()
#10 0x00007fb6c446ce9a in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#11 0x00007fb6c3b9acbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#12 0x0000000000000000 in ?? ()

Thread 16 (Thread 0x7f99d877e700 (LWP 27512)):
#0 0x00007fb6c4470d84 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fb6c25e9eb0 in
galera::Monitor<galera::ReplicatorSMM::LocalOrder>::enter(galera::ReplicatorSMM::LocalOrder&)
() from /usr/lib/galera/libgalera_smm.so
#2 0x00007fb6c25e5a9f in
galera::ReplicatorSMM::cert(galera::TrxHandle*) () from
/usr/lib/galera/libgalera_smm.so
#3 0x00007fb6c25e77b1 in galera::ReplicatorSMM::process_trx(void*,
galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.so
#4 0x00007fb6c25be124 in galera::GcsActionSource::dispatch(void*,
gcs_action const&) () from /usr/lib/galera/libgalera_smm.so
#5 0x00007fb6c25be39a in galera::GcsActionSource::process(void*) ()
from /usr/lib/galera/libgalera_smm.so
#6 0x00007fb6c25df175 in galera::ReplicatorSMM::async_recv(void*) ()
from /usr/lib/galera/libgalera_smm.so
#7 0x00007fb6c25f6f93 in galera_recv () from
/usr/lib/galera/libgalera_smm.so
#8 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*) ()
#9 0x00007fb6c5a2d71b in start_wsrep_THD ()
#10 0x00007fb6c446ce9a in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#11 0x00007fb6c3b9acbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#12 0x0000000000000000 in ?? ()

Thread 15 (Thread 0x7f99d8735700 (LWP 27513)):
#0 0x00007fb6c447389c in __lll_lock_wait () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fb6c446f065 in _L_lock_858 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#2 0x00007fb6c446eeba in pthread_mutex_lock () from
/lib/x86_64-linux-gnu/libpthread.so.0
#3 0x00007fb6c24d6457 in gu::Lock::Lock(gu::Mutex const&) () from
/usr/lib/galera/libgalera_smm.so
#4 0x00007fb6c25b2c91 in
galera::Certification::set_trx_committed(galera::TrxHandle*) () from
/usr/lib/galera/libgalera_smm.so
#5 0x00007fb6c25e72c2 in galera::ReplicatorSMM::apply_trx(void*,
galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.so
#6 0x00007fb6c25e77c5 in galera::ReplicatorSMM::process_trx(void*,
galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.so
#7 0x00007fb6c25be124 in galera::GcsActionSource::dispatch(void*,
gcs_action const&) () from /usr/lib/galera/libgalera_smm.so
#8 0x00007fb6c25be39a in galera::GcsActionSource::process(void*) ()
from /usr/lib/galera/libgalera_smm.so
#9 0x00007fb6c25df175 in galera::ReplicatorSMM::async_recv(void*) ()
from /usr/lib/galera/libgalera_smm.so
#10 0x00007fb6c25f6f93 in galera_recv () from
/usr/lib/galera/libgalera_smm.so
#11 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*) ()
#12 0x00007fb6c5a2d71b in start_wsrep_THD ()
#13 0x00007fb6c446ce9a in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#14 0x00007fb6c3b9acbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#15 0x0000000000000000 in ?? ()

Thread 14 (Thread 0x7f99d86ec700 (LWP 27514)):
#0 0x00007fb6c447389c in __lll_lock_wait () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fb6c446f065 in _L_lock_858 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#2 0x00007fb6c446eeba in pthread_mutex_lock () from
/lib/x86_64-linux-gnu/libpthread.so.0
#3 0x00007fb6c24d6457 in gu::Lock::Lock(gu::Mutex const&) () from
/usr/lib/galera/libgalera_smm.so
#4 0x00007fb6c25b2c91 in
galera::Certification::set_trx_committed(galera::TrxHandle*) () from
/usr/lib/galera/libgalera_smm.so
#5 0x00007fb6c25e72c2 in galera::ReplicatorSMM::apply_trx(void*,
galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.so
#6 0x00007fb6c25e77c5 in galera::ReplicatorSMM::process_trx(void*,
galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.so
#7 0x00007fb6c25be124 in galera::GcsActionSource::dispatch(void*,
gcs_action const&) () from /usr/lib/galera/libgalera_smm.so
#8 0x00007fb6c25be39a in galera::GcsActionSource::process(void*) ()
from /usr/lib/galera/libgalera_smm.so
#9 0x00007fb6c25df175 in galera::ReplicatorSMM::async_recv(void*) ()
from /usr/lib/galera/libgalera_smm.so
#10 0x00007fb6c25f6f93 in galera_recv () from
/usr/lib/galera/libgalera_smm.so
#11 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*) ()
#12 0x00007fb6c5a2d71b in start_wsrep_THD ()
#13 0x00007fb6c446ce9a in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#14 0x00007fb6c3b9acbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#15 0x0000000000000000 in ?? ()

Thread 13 (Thread 0x7f99d86a3700 (LWP 27515)):
#0 0x00007fb6c447389c in __lll_lock_wait () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fb6c446f09b in _L_lock_1006 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#2 0x00007fb6c446f01c in pthread_mutex_lock () from
/lib/x86_64-linux-gnu/libpthread.so.0
#3 0x00007fb6c5fc50d7 in my_pthread_fastmutex_lock ()
#4 0x00007fb6c5a2ad48 in ?? ()
#5 0x00007fb6c5a2d9ad in unlink_thd(THD*) ()
#6 0x00007fb6c5a2da84 in one_thread_per_connection_end(THD*, bool) ()
#7 0x00007fb6c5b6bed3 in do_handle_one_connection(THD*) ()
#8 0x00007fb6c5b6c181 in handle_one_connection ()
#9 0x00007fb6c446ce9a in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007fb6c3b9acbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#11 0x0000000000000000 in ?? ()

Thread 12 (Thread 0x7f99d8236700 (LWP 27534)):
#0 0x00007fb6c447389c in __lll_lock_wait () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fb6c446f09b in _L_lock_1006 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#2 0x00007fb6c446f01c in pthread_mutex_lock () from
/lib/x86_64-linux-gnu/libpthread.so.0
#3 0x00007fb6c5fc50d7 in my_pthread_fastmutex_lock ()
#4 0x00007fb6c5a2ad48 in ?? ()
#5 0x00007fb6c5a2d9ad in unlink_thd(THD*) ()
#6 0x00007fb6c5a2da84 in one_thread_per_connection_end(THD*, bool) ()
#7 0x00007fb6c5b6bed3 in do_handle_one_connection(THD*) ()
#8 0x00007fb6c5b6c181 in handle_one_connection ()
#9 0x00007fb6c446ce9a in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007fb6c3b9acbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#11 0x0000000000000000 in ?? ()

Thread 11 (Thread 0x7fb6c0654700 (LWP 4188)):
#0 0x00007fb6c447389c in __lll_lock_wait () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fb6c446f09b in _L_lock_1006 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#2 0x00007fb6c446f01c in pthread_mutex_lock () from
/lib/x86_64-linux-gnu/libpthread.so.0
#3 0x00007fb6c5fc50d7 in my_pthread_fastmutex_lock ()
#4 0x00007fb6c5a2ad48 in ?? ()
#5 0x00007fb6c5a2d9ad in unlink_thd(THD*) ()
#6 0x00007fb6c5a2da84 in one_thread_per_connection_end(THD*, bool) ()
#7 0x00007fb6c5b6bed3 in do_handle_one_connection(THD*) ()
#8 0x00007fb6c5b6c181 in handle_one_connection ()
#9 0x00007fb6c446ce9a in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007fb6c3b9acbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#11 0x0000000000000000 in ?? ()

Thread 10 (Thread 0x7fb6c060b700 (LWP 4189)):
#0 0x00007fb6c447389c in __lll_lock_wait () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fb6c446f09b in _L_lock_1006 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#2 0x00007fb6c446f01c in pthread_mutex_lock () from
/lib/x86_64-linux-gnu/libpthread.so.0
#3 0x00007fb6c5fc50d7 in my_pthread_fastmutex_lock ()
#4 0x00007fb6c5a2ad48 in ?? ()
#5 0x00007fb6c5a2d9ad in unlink_thd(THD*) ()
#6 0x00007fb6c5a2da84 in one_thread_per_connection_end(THD*, bool) ()
#7 0x00007fb6c5b6bed3 in do_handle_one_connection(THD*) ()
#8 0x00007fb6c5b6c181 in handle_one_connection ()
#9 0x00007fb6c446ce9a in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007fb6c3b9acbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#11 0x0000000000000000 in ?? ()

Thread 9 (Thread 0x7fb6c05c2700 (LWP 4205)):
#0 0x00007fb6c447389c in __lll_lock_wait () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fb6c446f09b in _L_lock_1006 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#2 0x00007fb6c446f01c in pthread_mutex_lock () from
/lib/x86_64-linux-gnu/libpthread.so.0
#3 0x00007fb6c5fc50d7 in my_pthread_fastmutex_lock ()
#4 0x00007fb6c5a2ad48 in ?? ()
#5 0x00007fb6c5a2d9ad in unlink_thd(THD*) ()
#6 0x00007fb6c5a2da84 in one_thread_per_connection_end(THD*, bool) ()
#7 0x00007fb6c5b6bed3 in do_handle_one_connection(THD*) ()
#8 0x00007fb6c5b6c181 in handle_one_connection ()
#9 0x00007fb6c446ce9a in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007fb6c3b9acbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#11 0x0000000000000000 in ?? ()

Thread 8 (Thread 0x7f99d81ed700 (LWP 29647)):
#0 0x00007fb6c447389c in __lll_lock_wait () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fb6c446f09b in _L_lock_1006 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#2 0x00007fb6c446f01c in pthread_mutex_lock () from
/lib/x86_64-linux-gnu/libpthread.so.0
#3 0x00007fb6c5fc50d7 in my_pthread_fastmutex_lock ()
#4 0x00007fb6c5a2ad48 in ?? ()
#5 0x00007fb6c5a2d9ad in unlink_thd(THD*) ()
#6 0x00007fb6c5a2da84 in one_thread_per_connection_end(THD*, bool) ()
#7 0x00007fb6c5b6bed3 in do_handle_one_connection(THD*) ()
#8 0x00007fb6c5b6c181 in handle_one_connection ()
#9 0x00007fb6c446ce9a in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007fb6c3b9acbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#11 0x0000000000000000 in ?? ()

Thread 7 (Thread 0x7f99d81a4700 (LWP 2266)):
#0 0x00007fb6c447389c in __lll_lock_wait () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fb6c446f09b in _L_lock_1006 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#2 0x00007fb6c446f01c in pthread_mutex_lock () from
/lib/x86_64-linux-gnu/libpthread.so.0
#3 0x00007fb6c5fc50d7 in my_pthread_fastmutex_lock ()
#4 0x00007fb6c5a2ad48 in ?? ()
#5 0x00007fb6c5a2d9ad in unlink_thd(THD*) ()
#6 0x00007fb6c5a2da84 in one_thread_per_connection_end(THD*, bool) ()
#7 0x00007fb6c5b6bed3 in do_handle_one_connection(THD*) ()
#8 0x00007fb6c5b6c181 in handle_one_connection ()
#9 0x00007fb6c446ce9a in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007fb6c3b9acbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#11 0x0000000000000000 in ?? ()

Thread 6 (Thread 0x7f99d815b700 (LWP 2267)):
#0 0x00007fb6c447389c in __lll_lock_wait () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fb6c446f09b in _L_lock_1006 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#2 0x00007fb6c446f01c in pthread_mutex_lock () from
/lib/x86_64-linux-gnu/libpthread.so.0
#3 0x00007fb6c5fc50d7 in my_pthread_fastmutex_lock ()
#4 0x00007fb6c5a2ad48 in ?? ()
#5 0x00007fb6c5a2d9ad in unlink_thd(THD*) ()
#6 0x00007fb6c5a2da84 in one_thread_per_connection_end(THD*, bool) ()
#7 0x00007fb6c5b6bed3 in do_handle_one_connection(THD*) ()
#8 0x00007fb6c5b6c181 in handle_one_connection ()
#9 0x00007fb6c446ce9a in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007fb6c3b9acbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#11 0x0000000000000000 in ?? ()

Thread 5 (Thread 0x7f99d8112700 (LWP 3577)):
#0 0x00007fb6c447389c in __lll_lock_wait () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fb6c446f065 in _L_lock_858 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#2 0x00007fb6c446eeba in pthread_mutex_lock () from
/lib/x86_64-linux-gnu/libpthread.so.0
#3 0x00007fb6c24d6457 in gu::Lock::Lock(gu::Mutex const&) () from
/usr/lib/galera/libgalera_smm.so
#4 0x00007fb6c25f38a8 in galera::ReplicatorSMM::stats_get() const ()
from /usr/lib/galera/libgalera_smm.so
#5 0x00007fb6c5bab110 in wsrep_show_status(THD*, st_mysql_show_var*,
char*) ()
#6 0x00007fb6c5b04ebc in ?? ()
#7 0x00007fb6c5b144e1 in fill_status(THD*, TABLE_LIST*, Item*) ()
#8 0x00007fb6c5b194d7 in get_schema_tables_result(JOIN*,
enum_schema_table_state) ()
#9 0x00007fb6c5afef5d in JOIN::exec() ()
#10 0x00007fb6c5afa612 in mysql_select(THD*, Item***, TABLE_LIST*,
unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*,
Item*, st_order*, unsigned long long, select_result*,
st_select_lex_unit*, st_select_lex*) ()
#11 0x00007fb6c5b013a3 in handle_select(THD*, LEX*, select_result*,
unsigned long) ()
#12 0x00007fb6c5aa9fe1 in ?? ()
#13 0x00007fb6c5ab6296 in mysql_execute_command(THD*) ()
#14 0x00007fb6c5ab8fb7 in ?? ()
#15 0x00007fb6c5ab93af in ?? ()
#16 0x00007fb6c5abb02a in dispatch_command(enum_server_command, THD*,
char*, unsigned int) ()
#17 0x00007fb6c5abb789 in do_command(THD*) ()
#18 0x00007fb6c5b6c0db in do_handle_one_connection(THD*) ()
#19 0x00007fb6c5b6c181 in handle_one_connection ()
#20 0x00007fb6c446ce9a in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#21 0x00007fb6c3b9acbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#22 0x0000000000000000 in ?? ()

Thread 4 (Thread 0x7f99d80c9700 (LWP 3578)):
#0 0x00007fb6c447389c in __lll_lock_wait () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fb6c446f09b in _L_lock_1006 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#2 0x00007fb6c446f01c in pthread_mutex_lock () from
/lib/x86_64-linux-gnu/libpthread.so.0
#3 0x00007fb6c5fc50d7 in my_pthread_fastmutex_lock ()
#4 0x00007fb6c5a2ad48 in ?? ()
#5 0x00007fb6c5a2d9ad in unlink_thd(THD*) ()
#6 0x00007fb6c5a2da84 in one_thread_per_connection_end(THD*, bool) ()
#7 0x00007fb6c5b6bed3 in do_handle_one_connection(THD*) ()
#8 0x00007fb6c5b6c181 in handle_one_connection ()
#9 0x00007fb6c446ce9a in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007fb6c3b9acbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#11 0x0000000000000000 in ?? ()

Thread 3 (Thread 0x7f99d8080700 (LWP 4954)):
#0 0x00007fb6c447389c in __lll_lock_wait () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fb6c446f09b in _L_lock_1006 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#2 0x00007fb6c446f01c in pthread_mutex_lock () from
/lib/x86_64-linux-gnu/libpthread.so.0
#3 0x00007fb6c5fc50d7 in my_pthread_fastmutex_lock ()
#4 0x00007fb6c5a2ad48 in ?? ()
#5 0x00007fb6c5a2d9ad in unlink_thd(THD*) ()
#6 0x00007fb6c5a2da84 in one_thread_per_connection_end(THD*, bool) ()
#7 0x00007fb6c5b6bed3 in do_handle_one_connection(THD*) ()
#8 0x00007fb6c5b6c181 in handle_one_connection ()
#9 0x00007fb6c446ce9a in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007fb6c3b9acbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#11 0x0000000000000000 in ?? ()

Thread 2 (Thread 0x7f9977fff700 (LWP 31658)):
#0 0x00007fb6c447389c in __lll_lock_wait () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fb6c446f09b in _L_lock_1006 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#2 0x00007fb6c446f01c in pthread_mutex_lock () from
/lib/x86_64-linux-gnu/libpthread.so.0
#3 0x00007fb6c5fc50d7 in my_pthread_fastmutex_lock ()
#4 0x00007fb6c5a2ad48 in ?? ()
#5 0x00007fb6c5a2d9ad in unlink_thd(THD*) ()
#6 0x00007fb6c5a2da84 in one_thread_per_connection_end(THD*, bool) ()
#7 0x00007fb6c5b6bed3 in do_handle_one_connection(THD*) ()
#8 0x00007fb6c5b6c181 in handle_one_connection ()
#9 0x00007fb6c446ce9a in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007fb6c3b9acbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#11 0x0000000000000000 in ?? ()

Thread 1 (Thread 0x7fb6c5717740 (LWP 27268)):
#0 0x00007fb6c3b8f303 in poll () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fb6c5a30ebb in handle_connections_sockets() ()
#2 0x00007fb6c5a34de0 in mysqld_main(int, char**) ()
#3 0x00007fb6c3ac876d in __libc_start_main () from
/lib/x86_64-linux-gnu/libc.so.6
#4 0x00007fb6c5a28c3d in _start ()


I have had pt-stalk running once every 30 minutes (cron), but it hung
until we killed the mysqld and not give all information during the
freeze of db9:

- log_error and output are empty

- trigger:
2013_06_10_07_30_01 Not stalking; collect triggered immediately
2013_06_10_07_30_01 pt-stalk ran with --function=status
--variable=Threads_running --threshold=25 --match= --cycles=0
--interval=0 --iterations=1 --run-time=30 --sleep=0
--dest=/var/lib/pt-stalk --prefix= --notify-by-email=
--log=/var/log/pt-stalk.log --pid=/var/run/pt-stalk.pid

variables:
Variable_name Value
aria_block_size 8192
aria_checkpoint_interval 30
aria_checkpoint_log_activity 1048576
aria_force_start_after_recovery_failures 0
aria_group_commit none
aria_group_commit_interval 0
aria_log_file_size 1073741824
aria_log_purge_type immediate
aria_max_sort_file_size 9223372036853727232
aria_page_checksum ON
aria_pagecache_age_threshold 300
aria_pagecache_buffer_size 134217728
aria_pagecache_division_limit 100
aria_recover NORMAL
aria_repair_threads 1
aria_sort_buffer_size 134217728
aria_stats_method nulls_unequal
aria_sync_log_dir NEWFILE
aria_used_for_temp_tables ON
auto_increment_increment 3
auto_increment_offset 1
autocommit ON
automatic_sp_privileges ON
back_log 50
basedir /usr
big_tables OFF
binlog_annotate_row_events OFF
binlog_cache_size 32768
binlog_checksum NONE
binlog_direct_non_transactional_updates OFF
binlog_format ROW
binlog_optimize_thread_scheduling ON
binlog_stmt_cache_size 32768
bulk_insert_buffer_size 16777216
character_set_client latin1
character_set_connection latin1
character_set_database latin1
character_set_filesystem binary
character_set_results latin1
character_set_server latin1
character_set_system utf8
character_sets_dir /usr/share/mysql/charsets/
collation_connection latin1_swedish_ci
collation_database latin1_swedish_ci
collation_server latin1_swedish_ci
completion_type NO_CHAIN
concurrent_insert ALWAYS
connect_timeout 5
datadir /var/lib/mysql/
date_format %Y-%m-%d
datetime_format %Y-%m-%d %H:%i:%s
deadlock_search_depth_long 15
deadlock_search_depth_short 4
deadlock_timeout_long 50000000
deadlock_timeout_short 10000
debug_no_thread_alarm OFF
default_storage_engine InnoDB
default_week_format 0
delay_key_write ON
delayed_insert_limit 100
delayed_insert_timeout 300
delayed_queue_size 1000
div_precision_increment 4
engine_condition_pushdown OFF
event_scheduler OFF
expensive_subquery_limit 100
expire_logs_days 10
extra_max_connections 1
extra_port 0
flush OFF
flush_time 0
foreign_key_checks ON
ft_boolean_syntax + -><()~*:""&|
ft_max_word_len 84
ft_min_word_len 4
ft_query_expansion_limit 20
ft_stopword_file (built-in)
general_log OFF
general_log_file db9.log
group_concat_max_len 1024
have_compress YES
have_crypt YES
have_csv YES
have_dynamic_loading YES
have_geometry YES
have_innodb YES
have_ndbcluster NO
have_openssl DISABLED
have_partitioning YES
have_profiling YES
have_query_cache YES
have_rtree_keys YES
have_ssl DISABLED
have_symlink YES
hostname db9
ignore_builtin_innodb OFF
ignore_db_dirs
init_connect
init_file
init_slave
innodb_adaptive_flushing ON
innodb_adaptive_flushing_method estimate
innodb_adaptive_hash_index ON
innodb_adaptive_hash_index_partitions 1
innodb_additional_mem_pool_size 8388608
innodb_autoextend_increment 1000
innodb_autoinc_lock_mode 2
innodb_blocking_buffer_pool_restore OFF
innodb_buffer_pool_instances 1
innodb_buffer_pool_populate OFF
innodb_buffer_pool_restore_at_startup 0
innodb_buffer_pool_shm_checksum ON
innodb_buffer_pool_shm_key 0
innodb_buffer_pool_size 107374182400
innodb_change_buffering all
innodb_changed_pages_limit 1000000
innodb_checkpoint_age_target 0
innodb_checksums ON
innodb_commit_concurrency 0
innodb_concurrency_tickets 500
innodb_corrupt_table_action assert
innodb_data_file_path ibdata1:10M:autoextend
innodb_data_home_dir
innodb_dict_size_limit 0
innodb_disallow_writes OFF
innodb_doublewrite ON
innodb_doublewrite_file
innodb_fake_changes OFF
innodb_fast_checksum OFF
innodb_fast_shutdown 1
innodb_file_format Antelope
innodb_file_format_check ON
innodb_file_format_max Antelope
innodb_file_per_table ON
innodb_flush_log_at_trx_commit 2
innodb_flush_method O_DIRECT
innodb_flush_neighbor_pages area
innodb_force_load_corrupted OFF
innodb_force_recovery 0
innodb_ibuf_accel_rate 100
innodb_ibuf_active_contract 1
innodb_ibuf_max_size 53687074816
innodb_import_table_from_xtrabackup 0
innodb_io_capacity 2000
innodb_kill_idle_transaction 0
innodb_large_prefix OFF
innodb_lazy_drop_table 0
innodb_lock_wait_timeout 120
innodb_locking_fake_changes ON
innodb_locks_unsafe_for_binlog ON
innodb_log_block_size 512
innodb_log_buffer_size 33554432
innodb_log_file_size 1258291200
innodb_log_files_in_group 2
innodb_log_group_home_dir ./
innodb_max_bitmap_file_size 104857600
innodb_max_dirty_pages_pct 75
innodb_max_purge_lag 0
innodb_merge_sort_block_size 1048576
innodb_mirrored_log_groups 1
innodb_old_blocks_pct 37
innodb_old_blocks_time 0
innodb_open_files 1000
innodb_page_size 16384
innodb_purge_batch_size 20
innodb_purge_threads 1
innodb_random_read_ahead OFF
innodb_read_ahead linear
innodb_read_ahead_threshold 56
innodb_read_io_threads 4
innodb_recovery_stats OFF
innodb_recovery_update_relay_log OFF
innodb_replication_delay 0
innodb_rollback_on_timeout OFF
innodb_rollback_segments 128
innodb_show_locks_held 10
innodb_show_verbose_locks 0
innodb_spin_wait_delay 6
innodb_stats_auto_update 1
innodb_stats_method nulls_equal
innodb_stats_on_metadata ON
innodb_stats_sample_pages 8
innodb_stats_update_need_lock 1
innodb_strict_mode OFF
innodb_support_xa ON
innodb_sync_spin_loops 30
innodb_table_locks ON
innodb_thread_concurrency 32
innodb_thread_concurrency_timer_based OFF
innodb_thread_sleep_delay 10000
innodb_track_changed_pages OFF
innodb_use_global_flush_log_at_trx_commit ON
innodb_use_native_aio ON
innodb_use_sys_malloc ON
innodb_use_sys_stats_table OFF
innodb_version 1.1.8-29.3
innodb_write_io_threads 4
interactive_timeout 28800
join_buffer_size 131072
join_buffer_space_limit 2097152
join_cache_level 2
keep_files_on_create OFF
key_buffer_size 134217728
key_cache_age_threshold 300
key_cache_block_size 1024
key_cache_division_limit 100
key_cache_segments 0
large_files_support ON
large_page_size 0
large_pages OFF
lc_messages en_US
lc_messages_dir /usr/share/mysql
lc_time_names en_US
license GPL
local_infile ON
lock_wait_timeout 31536000
locked_in_memory OFF
log OFF
log_bin ON
log_bin_trust_function_creators OFF
log_error
log_output FILE
log_queries_not_using_indexes OFF
log_slave_updates ON
log_slow_filter
admin,filesort,filesort_on_disk,full_join,full_scan,query_cache,query_cache_miss,tmp_table,tmp_table_on_disk
log_slow_queries OFF
log_slow_rate_limit 1
log_slow_verbosity query_plan
log_warnings 2
long_query_time 10.000000
low_priority_updates OFF
lower_case_file_system OFF
lower_case_table_names 0
master_verify_checksum OFF
max_allowed_packet 16777216
max_binlog_cache_size 4294963200
max_binlog_size 314572800
max_binlog_stmt_cache_size 4294963200
max_connect_errors 10
max_connections 300
max_delayed_threads 20
max_error_count 64
max_heap_table_size 33554432
max_insert_delayed_threads 20
max_join_size 18446744073709551615
max_length_for_sort_data 1024
max_long_data_size 16777216
max_prepared_stmt_count 16382
max_relay_log_size 0
max_seeks_for_key 4294967295
max_sort_length 1024
max_sp_recursion_depth 0
max_tmp_tables 32
max_user_connections 0
max_write_lock_count 4294967295
metadata_locks_cache_size 1024
min_examined_row_limit 0
mrr_buffer_size 262144
multi_range_count 256
myisam_block_size 1024
myisam_data_pointer_size 6
myisam_max_sort_file_size 9223372036853727232
myisam_mmap_size 18446744073709551615
myisam_recover_options BACKUP
myisam_repair_threads 1
myisam_sort_buffer_size 536870912
myisam_stats_method nulls_unequal
myisam_use_mmap OFF
net_buffer_length 16384
net_read_timeout 30
net_retry_count 10
net_write_timeout 60
old OFF
old_alter_table OFF
old_passwords OFF
open_files_limit 4407
optimizer_prune_level 1
optimizer_search_depth 62
optimizer_switch
index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=off
performance_schema OFF
performance_schema_events_waits_history_long_size 10000
performance_schema_events_waits_history_size 10
performance_schema_max_cond_classes 80
performance_schema_max_cond_instances 1000
performance_schema_max_file_classes 50
performance_schema_max_file_handles 32768
performance_schema_max_file_instances 10000
performance_schema_max_mutex_classes 200
performance_schema_max_mutex_instances 1000000
performance_schema_max_rwlock_classes 30
performance_schema_max_rwlock_instances 1000000
performance_schema_max_table_handles 100000
performance_schema_max_table_instances 50000
performance_schema_max_thread_classes 50
performance_schema_max_thread_instances 1000
pid_file /var/run/mysqld/mysqld.pid
plugin_dir /usr/lib/mysql/plugin/
plugin_maturity unknown
port 3306
preload_buffer_size 32768
profiling OFF
profiling_history_size 15
progress_report_time 56
protocol_version 10
query_alloc_block_size 8192
query_cache_limit 131072
query_cache_min_res_unit 4096
query_cache_size 0
query_cache_strip_comments OFF
query_cache_type OFF
query_cache_wlock_invalidate OFF
query_prealloc_size 8192
range_alloc_block_size 4096
read_buffer_size 2097152
read_only OFF
read_rnd_buffer_size 1048576
relay_log /var/log/mysql/relay-bin
relay_log_index /var/log/mysql/relay-bin.index
relay_log_info_file /var/log/mysql/relay-bin.info
relay_log_purge ON
relay_log_recovery OFF
relay_log_space_limit 0
replicate_annotate_row_events OFF
replicate_do_db
replicate_do_table
replicate_events_marked_for_skip replicate
replicate_ignore_db
replicate_ignore_table
replicate_wild_do_table
replicate_wild_ignore_table
report_host
report_password
report_port 3306
report_user
rowid_merge_buff_size 8388608
rpl_recovery_rank 0
secure_auth OFF
secure_file_priv
server_id 31
skip_external_locking ON
skip_name_resolve OFF
skip_networking OFF
skip_show_database OFF
slave_compressed_protocol ON
slave_exec_mode STRICT
slave_load_tmpdir /tmp
slave_max_allowed_packet 1073741824
slave_net_timeout 3600
slave_skip_errors OFF
slave_sql_verify_checksum ON
slave_transaction_retries 10
slave_type_conversions
slow_launch_time 2
slow_query_log OFF
slow_query_log_file /var/log/mysql/mariadb-slow.log
socket /var/run/mysqld/mysqld.sock
sort_buffer_size 4194304
sql_auto_is_null OFF
sql_big_selects ON
sql_big_tables OFF
sql_buffer_result OFF
sql_log_bin ON
sql_log_off OFF
sql_low_priority_updates OFF
sql_max_join_size 18446744073709551615
sql_mode
sql_notes ON
sql_quote_show_create ON
sql_safe_updates OFF
sql_select_limit 18446744073709551615
sql_slave_skip_counter 0
sql_warnings OFF
ssl_ca
ssl_capath
ssl_cert
ssl_cipher
ssl_key
storage_engine InnoDB
stored_program_cache 256
sync_binlog 0
sync_frm ON
sync_master_info 0
sync_relay_log 0
sync_relay_log_info 0
system_time_zone CEST
table_definition_cache 512
table_open_cache 2048
thread_cache_size 128
thread_concurrency 10
thread_handling one-thread-per-connection
thread_pool_idle_timeout 60
thread_pool_max_threads 500
thread_pool_oversubscribe 3
thread_pool_size 12
thread_pool_stall_limit 500
thread_stack 294912
time_format %H:%i:%s
time_zone SYSTEM
timed_mutexes OFF
tmp_table_size 33554432
tmpdir /tmp
transaction_alloc_block_size 8192
transaction_prealloc_size 4096
tx_isolation REPEATABLE-READ
unique_checks ON
updatable_views_with_limit YES
userstat OFF
version 5.5.29-MariaDB-mariadb1~quantal-log
version_comment mariadb.org binary distribution, wsrep_23.7.3.rXXXX
version_compile_machine x86_64
version_compile_os debian-linux-gnu
wait_timeout 600
wsrep_OSU_method TOI
wsrep_auto_increment_control ON
wsrep_causal_reads OFF
wsrep_certify_nonPK ON
wsrep_cluster_address gcomm://192.168.10.30,192.168.10.31
wsrep_cluster_name kursdaten
wsrep_convert_LOCK_to_trx OFF
wsrep_data_home_dir /var/lib/mysql/
wsrep_dbug_option
wsrep_debug OFF
wsrep_drupal_282555_workaround OFF
wsrep_forced_binlog_format NONE
wsrep_log_conflicts OFF
wsrep_max_ws_rows 131072
wsrep_max_ws_size 1073741824
wsrep_mysql_replication_bundle 0
wsrep_node_address 192.168.10.31
wsrep_node_incoming_address AUTO
wsrep_node_name db9
wsrep_notify_cmd
wsrep_on ON
wsrep_provider /usr/lib/galera/libgalera_smm.so
wsrep_provider_options base_host = 192.168.10.31; base_port = 4567;
cert.log_conflicts = no; evs.causal_keepalive_period = PT1S;
evs.debug_log_mask = 0x1; evs.inactive_check_period = PT0.5S;
evs.inactive_timeout = PT15S; evs.info_log_mask = 0; evs.install_timeout
= PT15S; evs.join_retrans_period = PT1S; evs.keepalive_period = PT1S;
evs.max_install_timeouts = 1; evs.send_window = 4;
evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S;
evs.use_aggregate = true; evs.user_send_window = 2; evs.version = 0;
evs.view_forget_timeout = PT5M; gcache.dir = /var/lib/mysql/;
gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name =
/var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.size =
10G; gcs.fc_debug = 0; gcs.fc_factor = 1; gcs.fc_limit = 16;
gcs.fc_master_slave = NO; gcs.max_packet_size = 64500; gcs.max_throttle
= 0.25; gcs.recv_q_hard_limit = 9223372036854775807;
gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = NO; gmcast.listen_addr =
tcp://0.0.0.0:4567; gmcast.mcast_addr = ; gmcast.mcast_ttl = 1;
gmcast.peer_timeout = PT3S; gmcast.time_wait = PT5S; gmcast.version = 0;
ist.recv_addr = 192.168.10.31; pc.checksum = true; pc.ignore_quorum =
false; pc.ignore_sb = false; pc.linger = PT20S; pc.npvo = true;
pc.version = 0; pc.weight = 1; protonet.backend = asio; protonet.version
= 0; replicator.causal_read_timeout = PT30S; replicator.commit_order = 3
wsrep_recover OFF
wsrep_replicate_myisam OFF
wsrep_retry_autocommit 1
wsrep_slave_threads 40
wsrep_sst_auth ********
wsrep_sst_donor
wsrep_sst_donor_rejects_queries OFF
wsrep_sst_method xtrabackup
wsrep_sst_receive_address 192.168.10.31
wsrep_start_position 41d7259d-7200-11e2-0800-943dd0e2f659:180873652

pmap:
27268: /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql
--plugin-dir=/usr/lib/mysql/plugin --user=mysql
--pid-file=/var/run/mysqld/mysqld.pid
--socket=/var/run/mysqld/mysqld.sock --port=3306
--wsrep_start_position=41d7259d-7200-11e2-0800-943dd0e2f659:180873652
Address Kbytes RSS Dirty Mode Mapping
00007f9890000000 132 28 28 rw--- [ anon ]
00007f9890021000 65404 0 0 ----- [ anon ]
00007f9894000000 132 20 20 rw--- [ anon ]
00007f9894021000 65404 0 0 ----- [ anon ]
00007f9898000000 132 28 28 rw--- [ anon ]
00007f9898021000 65404 0 0 ----- [ anon ]
00007f989c000000 132 28 28 rw--- [ anon ]
00007f989c021000 65404 0 0 ----- [ anon ]
00007f98a0000000 132 28 28 rw--- [ anon ]
00007f98a0021000 65404 0 0 ----- [ anon ]
00007f98a4000000 132 28 28 rw--- [ anon ]
00007f98a4021000 65404 0 0 ----- [ anon ]
00007f98a8000000 132 28 28 rw--- [ anon ]
00007f98a8021000 65404 0 0 ----- [ anon ]
00007f98ac000000 132 28 28 rw--- [ anon ]
00007f98ac021000 65404 0 0 ----- [ anon ]
00007f98b0000000 132 28 28 rw--- [ anon ]
00007f98b0021000 65404 0 0 ----- [ anon ]
00007f98b4000000 132 28 28 rw--- [ anon ]
00007f98b4021000 65404 0 0 ----- [ anon ]
00007f98b8000000 132 28 28 rw--- [ anon ]
00007f98b8021000 65404 0 0 ----- [ anon ]
00007f98bc000000 132 28 28 rw--- [ anon ]
00007f98bc021000 65404 0 0 ----- [ anon ]
00007f98c0000000 6760 4116 4116 rw--- [ anon ]
00007f98c069a000 58776 0 0 ----- [ anon ]
00007f98c4000000 7444 968 968 rw--- [ anon ]
00007f98c4745000 58092 0 0 ----- [ anon ]
00007f98c8000000 6664 576 576 rw--- [ anon ]
00007f98c8682000 58872 0 0 ----- [ anon ]
00007f98cc000000 6724 2620 2620 rw--- [ anon ]
00007f98cc691000 58812 0 0 ----- [ anon ]
00007f98d0000000 44956 28240 28240 rw--- [ anon ]
00007f98d2be7000 20580 0 0 ----- [ anon ]
00007f98d4000000 6660 132 132 rw--- [ anon ]
00007f98d4681000 58876 0 0 ----- [ anon ]
00007f98d8000000 131072 130760 130760 rw--- [ anon ]
00007f98e0000000 65536 37876 37876 rw--- [ anon ]
00007f98e8000000 7500 2624 2624 rw--- [ anon ]
00007f98e8753000 58036 0 0 ----- [ anon ]
00007f98ec000000 65536 65536 65536 rw--- [ anon ]
00007f98f0000000 7576 3612 3612 rw--- [ anon ]
00007f98f0766000 57960 0 0 ----- [ anon ]
00007f98f4000000 7296 3268 3268 rw--- [ anon ]
00007f98f4720000 58240 0 0 ----- [ anon ]
00007f98f8000000 132 68 68 rw--- [ anon ]
00007f98f8021000 65404 0 0 ----- [ anon ]
00007f98fc000000 7848 3060 3060 rw--- [ anon ]
00007f98fc7aa000 57688 0 0 ----- [ anon ]
00007f9900000000 132 64 64 rw--- [ anon ]
00007f9900021000 65404 0 0 ----- [ anon ]
00007f9904000000 132 60 60 rw--- [ anon ]
00007f9904021000 65404 0 0 ----- [ anon ]
00007f9908000000 132 64 64 rw--- [ anon ]
00007f9908021000 65404 0 0 ----- [ anon ]
00007f990c000000 9972 5240 5228 rw--- [ anon ]
00007f990c9bd000 55564 0 0 ----- [ anon ]
00007f9910000000 10208 6828 6812 rw--- [ anon ]
00007f99109f8000 55328 0 0 ----- [ anon ]
00007f9914000000 132 12 12 rw--- [ anon ]
00007f9914021000 65404 0 0 ----- [ anon ]
00007f9918000000 13180 12040 12028 rw--- [ anon ]
00007f9918cdf000 52356 0 0 ----- [ anon ]
00007f991c000000 20144 19976 19972 rw--- [ anon ]
00007f991d3ac000 45392 0 0 ----- [ anon ]
00007f9920000000 65536 6372 6372 rw--- [ anon ]
00007f9924000000 19752 18708 18704 rw--- [ anon ]
00007f992534a000 45784 0 0 ----- [ anon ]
00007f9928000000 19360 6648 6644 rw--- [ anon ]
00007f99292e8000 46176 0 0 ----- [ anon ]
00007f992c000000 13080 7080 7076 rw--- [ anon ]
00007f992ccc6000 52456 0 0 ----- [ anon ]
00007f9930000000 14612 5540 5540 rw--- [ anon ]
00007f9930e45000 50924 0 0 ----- [ anon ]
00007f9934000000 20048 19856 19840 rw--- [ anon ]
00007f9935394000 45488 0 0 ----- [ anon ]
00007f9938000000 65536 14380 14380 rw--- [ anon ]
00007f993c000000 65536 6076 6076 rw--- [ anon ]
00007f9940000000 13356 12112 12112 rw--- [ anon ]
00007f9940d0b000 52180 0 0 ----- [ anon ]
00007f9944000000 20412 20212 20212 rw--- [ anon ]
00007f99453ef000 45124 0 0 ----- [ anon ]
00007f9948000000 14424 13348 13348 rw--- [ anon ]
00007f9948e16000 51112 0 0 ----- [ anon ]
00007f994c000000 13032 11884 11884 rw--- [ anon ]
00007f994ccba000 52504 0 0 ----- [ anon ]
00007f9950000000 14316 13224 13224 rw--- [ anon ]
00007f9950dfb000 51220 0 0 ----- [ anon ]
00007f9954000000 14348 8760 8760 rw--- [ anon ]
00007f9954e03000 51188 0 0 ----- [ anon ]
00007f9958000000 13488 10336 10336 rw--- [ anon ]
00007f9958d2c000 52048 0 0 ----- [ anon ]
00007f995d2a7000 4 0 0 ----- [ anon ]
00007f995d2a8000 417400 3492 3328 rw--- [ anon ]
00007f9977c94000 4 0 0 ----- [ anon ]
00007f9977c95000 288 20 20 rw--- [ anon ]
00007f9977cdd000 4 0 0 ----- [ anon ]
00007f9977cde000 288 20 20 rw--- [ anon ]
00007f9977d26000 4 0 0 ----- [ anon ]
00007f9977d27000 288 20 20 rw--- [ anon ]
00007f9977d6f000 4 0 0 ----- [ anon ]
00007f9977d70000 288 20 20 rw--- [ anon ]
00007f9977db8000 4 0 0 ----- [ anon ]
00007f9977db9000 288 20 20 rw--- [ anon ]
00007f9977e01000 4 0 0 ----- [ anon ]
00007f9977e02000 288 20 20 rw--- [ anon ]
00007f9977e4a000 4 0 0 ----- [ anon ]
00007f9977e4b000 288 20 20 rw--- [ anon ]
00007f9977e93000 4 0 0 ----- [ anon ]
00007f9977e94000 288 20 20 rw--- [ anon ]
00007f9977edc000 4 0 0 ----- [ anon ]
00007f9977edd000 288 20 20 rw--- [ anon ]
00007f9977f25000 4 0 0 ----- [ anon ]
00007f9977f26000 288 20 20 rw--- [ anon ]
00007f9977f6e000 4 0 0 ----- [ anon ]
00007f9977f6f000 288 20 20 rw--- [ anon ]
00007f9977fb7000 4 0 0 ----- [ anon ]
00007f9977fb8000 288 20 20 rw--- [ anon ]
00007f9978000000 65536 12256 12256 rw--- [ anon ]
00007f997c000000 20156 9228 9224 rw--- [ anon ]
00007f997d3af000 45380 0 0 ----- [ anon ]
00007f9980000000 9652 5216 5208 rw--- [ anon ]
00007f998096d000 55884 0 0 ----- [ anon ]
00007f9984000000 65536 65356 65352 rw--- [ anon ]
00007f9988000000 9332 6464 6464 rw--- [ anon ]
00007f998891d000 56204 0 0 ----- [ anon ]
00007f998c000000 20596 20336 20336 rw--- [ anon ]
00007f998d41d000 44940 0 0 ----- [ anon ]
00007f9990000000 11020 6856 6848 rw--- [ anon ]
00007f9990ac3000 54516 0 0 ----- [ anon ]
00007f9994000000 9592 6232 6228 rw--- [ anon ]
00007f999495e000 55944 0 0 ----- [ anon ]
00007f9998000000 19400 9364 9364 rw--- [ anon ]
00007f99992f2000 46136 0 0 ----- [ anon ]
00007f999c000000 23916 12064 12060 rw--- [ anon ]
00007f999d75b000 41620 0 0 ----- [ anon ]
00007f99a0000000 13172 5352 5352 rw--- [ anon ]
00007f99a0cdd000 52364 0 0 ----- [ anon ]
00007f99a4000000 9920 6292 6292 rw--- [ anon ]
00007f99a49b0000 55616 0 0 ----- [ anon ]
00007f99a8000000 65536 62544 62536 rw--- [ anon ]
00007f99ac000000 11020 6628 6628 rw--- [ anon ]
00007f99acac3000 54516 0 0 ----- [ anon ]
00007f99b0000000 20264 20096 20096 rw--- [ anon ]
00007f99b13ca000 45272 0 0 ----- [ anon ]
00007f99b4000000 20448 20220 20220 rw--- [ anon ]
00007f99b53f8000 45088 0 0 ----- [ anon ]
00007f99b8000000 12732 6832 6824 rw--- [ anon ]
00007f99b8c6f000 52804 0 0 ----- [ anon ]
00007f99bc000000 65536 6992 6976 rw--- [ anon ]
00007f99c0000000 13464 12240 12236 rw--- [ anon ]
00007f99c0d26000 52072 0 0 ----- [ anon ]
00007f99c4000000 16300 15096 15092 rw--- [ anon ]
00007f99c4feb000 49236 0 0 ----- [ anon ]
00007f99c8000000 20144 20016 20016 rw--- [ anon ]
00007f99c93ac000 45392 0 0 ----- [ anon ]
00007f99cc000000 132 0 0 rw--- [ anon ]
00007f99cc021000 65404 0 0 ----- [ anon ]
00007f99d0000000 65536 16536 16536 rw--- [ anon ]
00007f99d4000000 132 92 92 rw--- [ anon ]
00007f99d4021000 65404 0 0 ----- [ anon ]
00007f99d8038000 4 0 0 ----- [ anon ]
00007f99d8039000 288 64 64 rw--- [ anon ]
00007f99d8081000 4 0 0 ----- [ anon ]
00007f99d8082000 288 64 64 rw--- [ anon ]
00007f99d80ca000 4 0 0 ----- [ anon ]
00007f99d80cb000 288 64 64 rw--- [ anon ]
00007f99d8113000 4 0 0 ----- [ anon ]
00007f99d8114000 288 64 64 rw--- [ anon ]
00007f99d815c000 4 0 0 ----- [ anon ]
00007f99d815d000 288 64 64 rw--- [ anon ]
00007f99d81a5000 4 0 0 ----- [ anon ]
00007f99d81a6000 288 64 64 rw--- [ anon ]
00007f99d81ee000 4 0 0 ----- [ anon ]
00007f99d81ef000 288 64 64 rw--- [ anon ]
00007f99d8237000 96 56 0 r-x-- libresolv-2.15.so
00007f99d824f000 2048 0 0 ----- libresolv-2.15.so
00007f99d844f000 4 0 0 r---- libresolv-2.15.so
00007f99d8450000 4 0 0 rw--- libresolv-2.15.so
00007f99d8451000 8 0 0 rw--- [ anon ]
00007f99d8453000 28 24 0 r-x-- libnss_dns-2.15.so
00007f99d845a000 2044 0 0 ----- libnss_dns-2.15.so
00007f99d8659000 4 0 0 r---- libnss_dns-2.15.so
00007f99d865a000 4 0 0 rw--- libnss_dns-2.15.so
00007f99d865b000 4 0 0 ----- [ anon ]
00007f99d865c000 288 64 64 rw--- [ anon ]
00007f99d86a4000 4 0 0 ----- [ anon ]
00007f99d86a5000 288 56 52 rw--- [ anon ]
00007f99d86ed000 4 0 0 ----- [ anon ]
00007f99d86ee000 288 56 52 rw--- [ anon ]
00007f99d8736000 4 0 0 ----- [ anon ]
00007f99d8737000 288 56 52 rw--- [ anon ]
00007f99d877f000 4 0 0 ----- [ anon ]
00007f99d8780000 288 56 52 rw--- [ anon ]
00007f99d87c8000 4 0 0 ----- [ anon ]
00007f99d87c9000 288 56 52 rw--- [ anon ]
00007f99d8811000 4 0 0 ----- [ anon ]
00007f99d8812000 288 56 52 rw--- [ anon ]
00007f99d885a000 4 0 0 ----- [ anon ]
00007f99d885b000 288 56 52 rw--- [ anon ]
00007f99d88a3000 4 0 0 ----- [ anon ]
00007f99d88a4000 288 56 52 rw--- [ anon ]
00007f99d88ec000 4 0 0 ----- [ anon ]
00007f99d88ed000 288 60 56 rw--- [ anon ]
00007f99d8935000 4 0 0 ----- [ anon ]
00007f99d8936000 288 56 52 rw--- [ anon ]
00007f99d897e000 4 0 0 ----- [ anon ]
00007f99d897f000 288 56 52 rw--- [ anon ]
00007f99d89c7000 4 0 0 ----- [ anon ]
00007f99d89c8000 288 56 52 rw--- [ anon ]
00007f99d8a10000 4 0 0 ----- [ anon ]
00007f99d8a11000 288 56 52 rw--- [ anon ]
00007f99d8a59000 4 0 0 ----- [ anon ]
00007f99d8a5a000 288 56 52 rw--- [ anon ]
00007f99d8aa2000 4 0 0 ----- [ anon ]
00007f99d8aa3000 288 56 52 rw--- [ anon ]
00007f99d8aeb000 4 0 0 ----- [ anon ]
00007f99d8aec000 288 60 56 rw--- [ anon ]
00007f99d8b34000 4 0 0 ----- [ anon ]
00007f99d8b35000 288 56 52 rw--- [ anon ]
00007f99d8b7d000 4 0 0 ----- [ anon ]
00007f99d8b7e000 288 56 52 rw--- [ anon ]
00007f99d8bc6000 4 0 0 ----- [ anon ]
00007f99d8bc7000 288 56 52 rw--- [ anon ]
00007f99d8c0f000 4 0 0 ----- [ anon ]
00007f99d8c10000 288 60 56 rw--- [ anon ]
00007f99d8c58000 4 0 0 ----- [ anon ]
00007f99d8c59000 288 56 52 rw--- [ anon ]
00007f99d8ca1000 4 0 0 ----- [ anon ]
00007f99d8ca2000 288 56 52 rw--- [ anon ]
00007f99d8cea000 4 0 0 ----- [ anon ]
00007f99d8ceb000 288 56 52 rw--- [ anon ]
00007f99d8d33000 4 0 0 ----- [ anon ]
00007f99d8d34000 288 56 52 rw--- [ anon ]
00007f99d8d7c000 4 0 0 ----- [ anon ]
00007f99d8d7d000 288 56 52 rw--- [ anon ]
00007f99d8dc5000 4 0 0 ----- [ anon ]
00007f99d8dc6000 288 56 52 rw--- [ anon ]
00007f99d8e0e000 4 0 0 ----- [ anon ]
00007f99d8e0f000 288 56 52 rw--- [ anon ]
00007f99d8e57000 4 0 0 ----- [ anon ]
00007f99d8e58000 288 56 52 rw--- [ anon ]
00007f99d8ea0000 4 0 0 ----- [ anon ]
00007f99d8ea1000 288 56 52 rw--- [ anon ]
00007f99d8ee9000 4 0 0 ----- [ anon ]
00007f99d8eea000 288 56 52 rw--- [ anon ]
00007f99d8f32000 4 0 0 ----- [ anon ]
00007f99d8f33000 288 60 56 rw--- [ anon ]
00007f99d8f7b000 4 0 0 ----- [ anon ]
00007f99d8f7c000 8192 20 20 rw--- [ anon ]
00007f99d977c000 4 0 0 ----- [ anon ]
00007f99d977d000 8192 12 12 rw--- [ anon ]
00007f99d9f7d000 4 0 0 ----- [ anon ]
00007f99d9f7e000 8192 8 8 rw--- [ anon ]
00007f99da77e000 4 0 0 ----- [ anon ]
00007f99da77f000 8192 8 8 rw--- [ anon ]
00007f99daf7f000 4 0 0 ----- [ anon ]
00007f99daf80000 8192 8 8 rw--- [ anon ]
00007f99db780000 4 0 0 ----- [ anon ]
00007f99db781000 8192 8 8 rw--- [ anon ]
00007f99dbf81000 4 0 0 ----- [ anon ]
00007f99dbf82000 8192 8 8 rw--- [ anon ]
00007f99dc782000 4 0 0 ----- [ anon ]
00007f99dc783000 8192 8 8 rw--- [ anon ]
00007f99dcf83000 4 0 0 ----- [ anon ]
00007f99dcf84000 8192 8 8 rw--- [ anon ]
00007f99dd784000 4 0 0 ----- [ anon ]
00007f99dd785000 8192 8 8 rw--- [ anon ]
00007f99ddf85000 4 0 0 ----- [ anon ]
00007f99ddf86000 8192 16 16 rw--- [ anon ]
00007f99de786000 4 0 0 ----- [ anon ]
00007f99

(file seems to be cut off here)

All other files have timestamps from 1.5h later when we killed mysqld.
Let me know if you need more information from some of these files.


Jan

Mikkel Mikjær Christensen

unread,
Jun 10, 2013, 9:56:16 AM6/10/13
to codersh...@googlegroups.com
We have a full-stack cluster running bad-code, and we have had a lot of crashes. What typically happens is that all the apache processes gets stuck i W-mode (sending output) which to me translates to stalled PHP Processes, at the sometime we have a lot of load, replication and/or replication_pause, we have made software to monitor this issue and we made at script that issues a restart of the web server at just the right moment to circumvent the crash, but sometimes it goes so hard down that we have to do them all simultaneous.

The crashes only happens in prime-time, but we have had no luck recreating them outside of primetime.

We got 4 servers, Dual Quadcore 2.0 Ghz xeon with 32MB of ram, 6zx146GB 10.000 RPM SAS.

We have 400 apaches instances on every server, so 160 in all.

We serve some 30 million page view every weekday from 0800 to around 1400 doing around 400req/sec during the peaks and several thousands mysql requests pr second. Al and all it performs okay, but the crashes happens almost exactly every morning at around 9, and nothing in particular starts there. Most of our users log in at 8 and leaves around 1600, we found several bugs in the software which have solved a lot of the issues, but this particular problem is persistent.

Any suggestions as to how i locate, and eliminate this problem?
 
De bedste hilsener / Best Regards
Mikkel Christensen

Mikjaer ApS
Silkeborgvej 128
8000 Århus C


Hosting - High Availability - High Performance Hosting - Webshops
Linux - Unix - BSD - Firewalls


Teemu Ollakka

unread,
Jun 10, 2013, 1:36:29 PM6/10/13
to codersh...@googlegroups.com

Jan,

Thanks for the trace! It revealed deadlock in Galera provider, reported a bug: https://bugs.launchpad.net/galera/+bug/1189526. 

We have a potential fix for this, so it would be great if you or anyone who can reproduce this issue could test if it fixes the problem. The patch is available   in https://code.launchpad.net/~teemu-ollakka/galera/lp1189526

- Teemu
00007f9929...

Jan Kirchhoff

unread,
Jun 10, 2013, 1:46:59 PM6/10/13
to codersh...@googlegroups.com
Teemu,

I use the galera package from mariadb (23.2.4). Can I just exchange the
/usr/lib/galera/libgalera_smm.so on the servers with a self-compiled
version or are there more dependencies or files to exchange?

Jan
> <http://relay-bin.info>
> version_comment mariadb.org <http://mariadb.org> binary
> distribution, wsrep_23.7.3.rXXXX
> version_compile_machine x86_64
> version_compile_os debian-linux-gnu
> wait_timeout 600
> wsrep_OSU_method TOI
> wsrep_auto_increment_control ON
> wsrep_causal_reads OFF
> wsrep_certify_nonPK ON
> wsrep_cluster_address gcomm://192.168.10.30
> <http://192.168.10.30>,192.168.10.31
> <http://gcache.name> =
> /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.size =
> 10G; gcs.fc_debug = 0; gcs.fc_factor = 1; gcs.fc_limit = 16;
> gcs.fc_master_slave = NO; gcs.max_packet_size = 64500; gcs.max_throttle
> = 0.25; gcs.recv_q_hard_limit = 9223372036854775807;
> gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = NO; gmcast.listen_addr =
> tcp://0.0.0.0:4567 <http://0.0.0.0:4567>; gmcast.mcast_addr = ;

Alex Yurchenko

unread,
Jun 10, 2013, 2:42:12 PM6/10/13
to codersh...@googlegroups.com
On 2013-06-10 16:56, Mikkel Mikjær Christensen wrote:
> We have a full-stack cluster running bad-code, and we have had a lot
> of crashes. What typically happens is that all the apache processes
> gets stuck i W-mode (sending output) which to me translates to stalled
> PHP Processes, at the sometime we have a lot of load, replication
> and/or replication_pause, we have made software to monitor this issue

If "replication_pause" means wsrep_flow_control_paused > 0 then it is
very likely that your IO subsystem cannot handle it (IO). So you should
check that it works at full capacity and overall you have an
IO-optimized configuration.

> and we made at script that issues a restart of the web server at just
> the right moment to circumvent the crash, but sometimes it goes so
> hard down that we have to do them all simultaneous.
>
> The crashes only happens in prime-time, but we have had no luck
> recreating them outside of primetime.
>
> We got 4 servers, Dual Quadcore 2.0 Ghz xeon with 32MB of ram,
> 6zx146GB 10.000 RPM SAS.
>
> We have 400 apaches instances on every server, so 160 in all.
>
> We serve some 30 million page view every weekday from 0800 to around
> 1400 doing around 400req/sec during the peaks and several thousands
> mysql requests pr second. Al and all it performs okay, but the crashes
> happens almost exactly every morning at around 9, and nothing in
> particular starts there. Most of our users log in at 8 and leaves
> around 1600, we found several bugs in the software which have solved a
> lot of the issues, but this particular problem is persistent.
>
> Any suggestions as to how i locate, and eliminate this problem?
>
> De bedste hilsener / Best Regards
> Mikkel Christensen
>
> Mikjaer ApS
> Silkeborgvej 128
> 8000 Århus C
>
> Tlf. +45 4440 1337
>
> Hosting - High Availability - High Performance Hosting - Webshops
> Linux - Unix - BSD - Firewalls

--
Alexey Yurchenko,
Codership Oy, www.codership.com
Skype: alexey.yurchenko, Phone: +358-400-516-011

Alex Yurchenko

unread,
Jun 10, 2013, 2:51:37 PM6/10/13
to codersh...@googlegroups.com
On 2013-06-10 20:46, Jan Kirchhoff wrote:
> Teemu,
>
> I use the galera package from mariadb (23.2.4). Can I just exchange
> the
> /usr/lib/galera/libgalera_smm.so on the servers with a self-compiled
> version or are there more dependencies or files to exchange?
>
> Jan

Sure Jan, that was the whole idea ;)

Regards,
Alex

Jan Kirchhoff

unread,
Jun 10, 2013, 3:01:03 PM6/10/13
to codersh...@googlegroups.com
just wanted to make sure...
just dropped in the replacement file on both servers and we'll see what
happens during the next few days :-D
Jan

Dustin Marquess

unread,
Jun 13, 2013, 12:22:46 AM6/13/13
to Teemu Ollakka, codersh...@googlegroups.com
So far, this has COMPLETELY fixed all of the issues I've had with 5.5.29 & 5.5.30 that I was having in my other thread.  THANK YOU!


--

Jan Kirchhoff

unread,
Jun 13, 2013, 4:24:09 AM6/13/13
to codersh...@googlegroups.com
Same for me, no more problems so far.
As we've had a few days but also a few weeks between crashes I feel it
is too early to say everything is fine - but everything looks fine at
least. Thank you! Especially for fixing this within 5 hours after I
posted the stack trace! Great support!
Jan
> "/lib/x86_64-linux-gnu/__libthread_db.so.1".
> 0x00007fb6c3b8f303 in poll () from
> /lib/x86_64-linux-gnu/libc.so.__6
>
> Thread 75 (Thread 0x7fb6c3890700 (LWP 27273)):
> #0 0x00007fb6c4470d84 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #1 0x00007fb6c25ba1b1 in galera::ServiceThd::thd_func(__void*)
> () from
> /usr/lib/galera/libgalera_smm.__so
> #2 0x00007fb6c446ce9a in start_thread () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #3 0x00007fb6c3b9acbd in clone () from
> /lib/x86_64-linux-gnu/libc.so.__6
> #4 0x0000000000000000 in ?? ()
>
> Thread 74 (Thread 0x7fb43bffe700 (LWP 27274)):
> #0 0x00007fb6c3b9b353 in epoll_wait () from
> /lib/x86_64-linux-gnu/libc.so.__6
> #1 0x00007fb6c258c4d9 in asio::detail::epoll_reactor::__run(bool,
> asio::detail::op_queue<asio::__detail::task_io_service___operation>&)
> ()
> from /usr/lib/galera/libgalera_smm.__so
> #2 0x00007fb6c259136f in
> asio::detail::task_io_service:__:run(asio::error_code&) () from
> /usr/lib/galera/libgalera_smm.__so
> #3 0x00007fb6c2587d87 in
> gcomm::AsioProtonet::event___loop(gu::datetime::Period const&)
> () from
> /usr/lib/galera/libgalera_smm.__so
> #4 0x00007fb6c25a3808 in GCommConn::run() () from
> /usr/lib/galera/libgalera_smm.__so
> #5 0x00007fb6c25a75c9 in GCommConn::run_fn(void*) () from
> /usr/lib/galera/libgalera_smm.__so
> #6 0x00007fb6c446ce9a in start_thread () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #7 0x00007fb6c3b9acbd in clone () from
> /lib/x86_64-linux-gnu/libc.so.__6
> #8 0x0000000000000000 in ?? ()
>
> Thread 73 (Thread 0x7fb43b7fd700 (LWP 27275)):
> #0 0x00007fb6c4470d84 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #1 0x00007fb6c25a2f47 in ?? () from
> /usr/lib/galera/libgalera_smm.__so
> #2 0x00007fb6c2599135 in gcs_core_recv () from
> /usr/lib/galera/libgalera_smm.__so
> #3 0x00007fb6c259f181 in ?? () from
> /usr/lib/galera/libgalera_smm.__so
> #4 0x00007fb6c446ce9a in start_thread () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #5 0x00007fb6c3b9acbd in clone () from
> /lib/x86_64-linux-gnu/libc.so.__6
> #6 0x0000000000000000 in ?? ()
>
> Thread 72 (Thread 0x7fb43affc700 (LWP 27276)):
> #0 0x00007fb6c4470d84 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #1 0x00007fb6c5ab21eb in wsrep_rollback_process(THD*) ()
> #2 0x00007fb6c5a2d71b in start_wsrep_THD ()
> #3 0x00007fb6c446ce9a in start_thread () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #4 0x00007fb6c3b9acbd in clone () from
> /lib/x86_64-linux-gnu/libc.so.__6
> #5 0x0000000000000000 in ?? ()
>
> Thread 71 (Thread 0x7fb43afb3700 (LWP 27277)):
> #0 0x00007fb6c4470d84 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #1 0x00007fb6c25e9eb0 in
> galera::Monitor<galera::__ReplicatorSMM::LocalOrder>::__enter(galera::ReplicatorSMM::__LocalOrder&)
>
> () from /usr/lib/galera/libgalera_smm.__so
> #2 0x00007fb6c25e5a9f in
> galera::ReplicatorSMM::cert(__galera::TrxHandle*) () from
> /usr/lib/galera/libgalera_smm.__so
> #3 0x00007fb6c25e77b1 in
> galera::ReplicatorSMM::__process_trx(void*,
> galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.__so
> #4 0x00007fb6c25be124 in
> galera::GcsActionSource::__dispatch(void*,
> gcs_action const&) () from /usr/lib/galera/libgalera_smm.__so
> #5 0x00007fb6c25be39a in
> galera::GcsActionSource::__process(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #6 0x00007fb6c25df175 in
> galera::ReplicatorSMM::async___recv(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #7 0x00007fb6c25f6f93 in galera_recv () from
> /usr/lib/galera/libgalera_smm.__so
> #8 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*__) ()
> #9 0x00007fb6c5a2d71b in start_wsrep_THD ()
> #10 0x00007fb6c446ce9a in start_thread () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #11 0x00007fb6c3b9acbd in clone () from
> /lib/x86_64-linux-gnu/libc.so.__6
> #12 0x0000000000000000 in ?? ()
>
> Thread 70 (Thread 0x7fb43af6a700 (LWP 27343)):
> #0 0x00007fb6c44710fe in pthread_cond_timedwait@@GLIBC___2.3.2
> () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #1 0x00007fb6c5d60152 in ?? ()
> #2 0x00007fb6c5d58237 in ?? ()
> #3 0x00007fb6c446ce9a in start_thread () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #4 0x00007fb6c3b9acbd in clone () from
> /lib/x86_64-linux-gnu/libc.so.__6
> #5 0x0000000000000000 in ?? ()
>
> Thread 69 (Thread 0x7f99e0f8a700 (LWP 27399)):
> #0 0x00007fb6c5f27124 in __io_getevents_0_4 ()
> #1 0x00007fb6c5eff694 in ?? ()
> #2 0x00007fb6c5eaed1a in ?? ()
> #3 0x00007fb6c5e199a8 in ?? ()
> #4 0x00007fb6c446ce9a in start_thread () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #5 0x00007fb6c3b9acbd in clone () from
> /lib/x86_64-linux-gnu/libc.so.__6
> #6 0x0000000000000000 in ?? ()
>
> Thread 68 (Thread 0x7f99e0789700 (LWP 27400)):
> #0 0x00007fb6c5f27124 in __io_getevents_0_4 ()
> #1 0x00007fb6c5eff694 in ?? ()
> #2 0x00007fb6c5eaed1a in ?? ()
> #3 0x00007fb6c5e199a8 in ?? ()
> #4 0x00007fb6c446ce9a in start_thread () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #5 0x00007fb6c3b9acbd in clone () from
> /lib/x86_64-linux-gnu/libc.so.__6
> #6 0x0000000000000000 in ?? ()
>
> Thread 67 (Thread 0x7f99dff88700 (LWP 27401)):
> #0 0x00007fb6c5f27124 in __io_getevents_0_4 ()
> #1 0x00007fb6c5eff694 in ?? ()
> #2 0x00007fb6c5eaed1a in ?? ()
> #3 0x00007fb6c5e199a8 in ?? ()
> #4 0x00007fb6c446ce9a in start_thread () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #5 0x00007fb6c3b9acbd in clone () from
> /lib/x86_64-linux-gnu/libc.so.__6
> #6 0x0000000000000000 in ?? ()
>
> Thread 66 (Thread 0x7f99df787700 (LWP 27402)):
> #0 0x00007fb6c5f27124 in __io_getevents_0_4 ()
> #1 0x00007fb6c5eff694 in ?? ()
> #2 0x00007fb6c5eaed1a in ?? ()
> #3 0x00007fb6c5e199a8 in ?? ()
> #4 0x00007fb6c446ce9a in start_thread () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #5 0x00007fb6c3b9acbd in clone () from
> /lib/x86_64-linux-gnu/libc.so.__6
> #6 0x0000000000000000 in ?? ()
>
> Thread 65 (Thread 0x7f99def86700 (LWP 27403)):
> #0 0x00007fb6c5f27124 in __io_getevents_0_4 ()
> #1 0x00007fb6c5eff694 in ?? ()
> #2 0x00007fb6c5eaed1a in ?? ()
> #3 0x00007fb6c5e199a8 in ?? ()
> #4 0x00007fb6c446ce9a in start_thread () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #5 0x00007fb6c3b9acbd in clone () from
> /lib/x86_64-linux-gnu/libc.so.__6
> #6 0x0000000000000000 in ?? ()
>
> Thread 64 (Thread 0x7f99de785700 (LWP 27404)):
> #0 0x00007fb6c5f27124 in __io_getevents_0_4 ()
> #1 0x00007fb6c5eff694 in ?? ()
> #2 0x00007fb6c5eaed1a in ?? ()
> #3 0x00007fb6c5e199a8 in ?? ()
> #4 0x00007fb6c446ce9a in start_thread () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #5 0x00007fb6c3b9acbd in clone () from
> /lib/x86_64-linux-gnu/libc.so.__6
> #6 0x0000000000000000 in ?? ()
>
> Thread 63 (Thread 0x7f99ddf84700 (LWP 27405)):
> #0 0x00007fb6c5f27124 in __io_getevents_0_4 ()
> #1 0x00007fb6c5eff694 in ?? ()
> #2 0x00007fb6c5eaed1a in ?? ()
> #3 0x00007fb6c5e199a8 in ?? ()
> #4 0x00007fb6c446ce9a in start_thread () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #5 0x00007fb6c3b9acbd in clone () from
> /lib/x86_64-linux-gnu/libc.so.__6
> #6 0x0000000000000000 in ?? ()
>
> Thread 62 (Thread 0x7f99dd783700 (LWP 27406)):
> #0 0x00007fb6c5f27124 in __io_getevents_0_4 ()
> #1 0x00007fb6c5eff694 in ?? ()
> #2 0x00007fb6c5eaed1a in ?? ()
> #3 0x00007fb6c5e199a8 in ?? ()
> #4 0x00007fb6c446ce9a in start_thread () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #5 0x00007fb6c3b9acbd in clone () from
> /lib/x86_64-linux-gnu/libc.so.__6
> #6 0x0000000000000000 in ?? ()
>
> Thread 61 (Thread 0x7f99dcf82700 (LWP 27407)):
> #0 0x00007fb6c5f27124 in __io_getevents_0_4 ()
> #1 0x00007fb6c5eff694 in ?? ()
> #2 0x00007fb6c5eaed1a in ?? ()
> #3 0x00007fb6c5e199a8 in ?? ()
> #4 0x00007fb6c446ce9a in start_thread () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #5 0x00007fb6c3b9acbd in clone () from
> /lib/x86_64-linux-gnu/libc.so.__6
> #6 0x0000000000000000 in ?? ()
>
> Thread 60 (Thread 0x7f99dc781700 (LWP 27408)):
> #0 0x00007fb6c5f27124 in __io_getevents_0_4 ()
> #1 0x00007fb6c5eff694 in ?? ()
> #2 0x00007fb6c5eaed1a in ?? ()
> #3 0x00007fb6c5e199a8 in ?? ()
> #4 0x00007fb6c446ce9a in start_thread () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #5 0x00007fb6c3b9acbd in clone () from
> /lib/x86_64-linux-gnu/libc.so.__6
> #6 0x0000000000000000 in ?? ()
>
> Thread 59 (Thread 0x7f99dbf80700 (LWP 27469)):
> #0 0x00007fb6c44710fe in pthread_cond_timedwait@@GLIBC___2.3.2
> () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #1 0x00007fb6c5f015eb in ?? ()
> #2 0x00007fb6c5e16678 in ?? ()
> #3 0x00007fb6c446ce9a in start_thread () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #4 0x00007fb6c3b9acbd in clone () from
> /lib/x86_64-linux-gnu/libc.so.__6
> #5 0x0000000000000000 in ?? ()
>
> Thread 58 (Thread 0x7f99db77f700 (LWP 27470)):
> #0 0x00007fb6c44710fe in pthread_cond_timedwait@@GLIBC___2.3.2
> () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #1 0x00007fb6c5f015eb in ?? ()
> #2 0x00007fb6c5e16ceb in ?? ()
> #3 0x00007fb6c446ce9a in start_thread () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #4 0x00007fb6c3b9acbd in clone () from
> /lib/x86_64-linux-gnu/libc.so.__6
> #5 0x0000000000000000 in ?? ()
>
> Thread 57 (Thread 0x7f99daf7e700 (LWP 27471)):
> #0 0x00007fb6c44710fe in pthread_cond_timedwait@@GLIBC___2.3.2
> () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #1 0x00007fb6c5f015eb in ?? ()
> #2 0x00007fb6c5e1635b in ?? ()
> #3 0x00007fb6c446ce9a in start_thread () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #4 0x00007fb6c3b9acbd in clone () from
> /lib/x86_64-linux-gnu/libc.so.__6
> #5 0x0000000000000000 in ?? ()
>
> Thread 56 (Thread 0x7f99da77d700 (LWP 27472)):
> #0 0x00007fb6c44710fe in pthread_cond_timedwait@@GLIBC___2.3.2
> () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #1 0x00007fb6c5f015eb in ?? ()
> #2 0x00007fb6c5e17083 in ?? ()
> #3 0x00007fb6c446ce9a in start_thread () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #4 0x00007fb6c3b9acbd in clone () from
> /lib/x86_64-linux-gnu/libc.so.__6
> #5 0x0000000000000000 in ?? ()
>
> Thread 55 (Thread 0x7f99d9f7c700 (LWP 27473)):
> #0 0x00007fb6c4470d84 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #1 0x00007fb6c5f014e1 in ?? ()
> #2 0x00007fb6c5e17ba2 in ?? ()
> #3 0x00007fb6c446ce9a in start_thread () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #4 0x00007fb6c3b9acbd in clone () from
> /lib/x86_64-linux-gnu/libc.so.__6
> #5 0x0000000000000000 in ?? ()
>
> Thread 54 (Thread 0x7f99d977b700 (LWP 27474)):
> #0 0x00007fb6c4470d84 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #1 0x00007fb6c5f014e1 in ?? ()
> #2 0x00007fb6c5e195e6 in ?? ()
> #3 0x00007fb6c446ce9a in start_thread () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #4 0x00007fb6c3b9acbd in clone () from
> /lib/x86_64-linux-gnu/libc.so.__6
> #5 0x0000000000000000 in ?? ()
>
> Thread 53 (Thread 0x7fb6c5572700 (LWP 27475)):
> #0 0x00007fb6c44749f7 in do_sigwait () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #1 0x00007fb6c4474a79 in sigwait () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #2 0x00007fb6c5a2b56f in signal_hand ()
> #3 0x00007fb6c446ce9a in start_thread () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #4 0x00007fb6c3b9acbd in clone () from
> /lib/x86_64-linux-gnu/libc.so.__6
> #5 0x0000000000000000 in ?? ()
>
> Thread 52 (Thread 0x7fb6c0065700 (LWP 27476)):
> #0 0x00007fb6c447389c in __lll_lock_wait () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #1 0x00007fb6c446f065 in _L_lock_858 () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #2 0x00007fb6c446eeba in pthread_mutex_lock () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #3 0x00007fb6c24d6457 in gu::Lock::Lock(gu::Mutex const&) () from
> /usr/lib/galera/libgalera_smm.__so
> #4 0x00007fb6c25b2c91 in
> galera::Certification::set___trx_committed(galera::__TrxHandle*)
> () from
> /usr/lib/galera/libgalera_smm.__so
> #5 0x00007fb6c25e72c2 in galera::ReplicatorSMM::apply___trx(void*,
> galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.__so
> #6 0x00007fb6c25e77c5 in
> galera::ReplicatorSMM::__process_trx(void*,
> galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.__so
> #7 0x00007fb6c25be124 in
> galera::GcsActionSource::__dispatch(void*,
> gcs_action const&) () from /usr/lib/galera/libgalera_smm.__so
> #8 0x00007fb6c25be39a in
> galera::GcsActionSource::__process(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #9 0x00007fb6c25df175 in
> galera::ReplicatorSMM::async___recv(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #10 0x00007fb6c25f6f93 in galera_recv () from
> /usr/lib/galera/libgalera_smm.__so
> #11 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*__) ()
> #12 0x00007fb6c5a2d71b in start_wsrep_THD ()
> #13 0x00007fb6c446ce9a in start_thread () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #14 0x00007fb6c3b9acbd in clone () from
> /lib/x86_64-linux-gnu/libc.so.__6
> #15 0x0000000000000000 in ?? ()
>
> Thread 51 (Thread 0x7fb4381d3700 (LWP 27477)):
> #0 0x00007fb6c447389c in __lll_lock_wait () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #1 0x00007fb6c446f065 in _L_lock_858 () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #2 0x00007fb6c446eeba in pthread_mutex_lock () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #3 0x00007fb6c24d6457 in gu::Lock::Lock(gu::Mutex const&) () from
> /usr/lib/galera/libgalera_smm.__so
> #4 0x00007fb6c25b2c91 in
> galera::Certification::set___trx_committed(galera::__TrxHandle*)
> () from
> /usr/lib/galera/libgalera_smm.__so
> #5 0x00007fb6c25e72c2 in galera::ReplicatorSMM::apply___trx(void*,
> galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.__so
> #6 0x00007fb6c25e77c5 in
> galera::ReplicatorSMM::__process_trx(void*,
> galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.__so
> #7 0x00007fb6c25be124 in
> galera::GcsActionSource::__dispatch(void*,
> gcs_action const&) () from /usr/lib/galera/libgalera_smm.__so
> #8 0x00007fb6c25be39a in
> galera::GcsActionSource::__process(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #9 0x00007fb6c25df175 in
> galera::ReplicatorSMM::async___recv(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #10 0x00007fb6c25f6f93 in galera_recv () from
> /usr/lib/galera/libgalera_smm.__so
> #11 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*__) ()
> #12 0x00007fb6c5a2d71b in start_wsrep_THD ()
> #13 0x00007fb6c446ce9a in start_thread () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #14 0x00007fb6c3b9acbd in clone () from
> /lib/x86_64-linux-gnu/libc.so.__6
> #15 0x0000000000000000 in ?? ()
>
> Thread 50 (Thread 0x7fb43818a700 (LWP 27478)):
> #0 0x00007fb6c447389c in __lll_lock_wait () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #1 0x00007fb6c446f065 in _L_lock_858 () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #2 0x00007fb6c446eeba in pthread_mutex_lock () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #3 0x00007fb6c24d6457 in gu::Lock::Lock(gu::Mutex const&) () from
> /usr/lib/galera/libgalera_smm.__so
> #4 0x00007fb6c25b2c91 in
> galera::Certification::set___trx_committed(galera::__TrxHandle*)
> () from
> /usr/lib/galera/libgalera_smm.__so
> #5 0x00007fb6c25e72c2 in galera::ReplicatorSMM::apply___trx(void*,
> galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.__so
> #6 0x00007fb6c25e77c5 in
> galera::ReplicatorSMM::__process_trx(void*,
> galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.__so
> #7 0x00007fb6c25be124 in
> galera::GcsActionSource::__dispatch(void*,
> gcs_action const&) () from /usr/lib/galera/libgalera_smm.__so
> #8 0x00007fb6c25be39a in
> galera::GcsActionSource::__process(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #9 0x00007fb6c25df175 in
> galera::ReplicatorSMM::async___recv(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #10 0x00007fb6c25f6f93 in galera_recv () from
> /usr/lib/galera/libgalera_smm.__so
> #11 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*__) ()
> #12 0x00007fb6c5a2d71b in start_wsrep_THD ()
> #13 0x00007fb6c446ce9a in start_thread () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #14 0x00007fb6c3b9acbd in clone () from
> /lib/x86_64-linux-gnu/libc.so.__6
> #15 0x0000000000000000 in ?? ()
>
> Thread 49 (Thread 0x7fb438141700 (LWP 27479)):
> #0 0x00007fb6c447389c in __lll_lock_wait () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #1 0x00007fb6c446f065 in _L_lock_858 () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #2 0x00007fb6c446eeba in pthread_mutex_lock () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #3 0x00007fb6c24d6457 in gu::Lock::Lock(gu::Mutex const&) () from
> /usr/lib/galera/libgalera_smm.__so
> #4 0x00007fb6c25b2c91 in
> galera::Certification::set___trx_committed(galera::__TrxHandle*)
> () from
> /usr/lib/galera/libgalera_smm.__so
> #5 0x00007fb6c25e72c2 in galera::ReplicatorSMM::apply___trx(void*,
> galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.__so
> #6 0x00007fb6c25e77c5 in
> galera::ReplicatorSMM::__process_trx(void*,
> galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.__so
> #7 0x00007fb6c25be124 in
> galera::GcsActionSource::__dispatch(void*,
> gcs_action const&) () from /usr/lib/galera/libgalera_smm.__so
> #8 0x00007fb6c25be39a in
> galera::GcsActionSource::__process(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #9 0x00007fb6c25df175 in
> galera::ReplicatorSMM::async___recv(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #10 0x00007fb6c25f6f93 in galera_recv () from
> /usr/lib/galera/libgalera_smm.__so
> #11 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*__) ()
> #12 0x00007fb6c5a2d71b in start_wsrep_THD ()
> #13 0x00007fb6c446ce9a in start_thread () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #14 0x00007fb6c3b9acbd in clone () from
> /lib/x86_64-linux-gnu/libc.so.__6
> #15 0x0000000000000000 in ?? ()
>
> Thread 48 (Thread 0x7fb4380f8700 (LWP 27480)):
> #0 0x00007fb6c447389c in __lll_lock_wait () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #1 0x00007fb6c446f065 in _L_lock_858 () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #2 0x00007fb6c446eeba in pthread_mutex_lock () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #3 0x00007fb6c24d6457 in gu::Lock::Lock(gu::Mutex const&) () from
> /usr/lib/galera/libgalera_smm.__so
> #4 0x00007fb6c25b2c91 in
> galera::Certification::set___trx_committed(galera::__TrxHandle*)
> () from
> /usr/lib/galera/libgalera_smm.__so
> #5 0x00007fb6c25e72c2 in galera::ReplicatorSMM::apply___trx(void*,
> galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.__so
> #6 0x00007fb6c25e77c5 in
> galera::ReplicatorSMM::__process_trx(void*,
> galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.__so
> #7 0x00007fb6c25be124 in
> galera::GcsActionSource::__dispatch(void*,
> gcs_action const&) () from /usr/lib/galera/libgalera_smm.__so
> #8 0x00007fb6c25be39a in
> galera::GcsActionSource::__process(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #9 0x00007fb6c25df175 in
> galera::ReplicatorSMM::async___recv(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #10 0x00007fb6c25f6f93 in galera_recv () from
> /usr/lib/galera/libgalera_smm.__so
> #11 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*__) ()
> #12 0x00007fb6c5a2d71b in start_wsrep_THD ()
> #13 0x00007fb6c446ce9a in start_thread () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #14 0x00007fb6c3b9acbd in clone () from
> /lib/x86_64-linux-gnu/libc.so.__6
> #15 0x0000000000000000 in ?? ()
>
> Thread 47 (Thread 0x7fb4380af700 (LWP 27481)):
> #0 0x00007fb6c447389c in __lll_lock_wait () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #1 0x00007fb6c446f065 in _L_lock_858 () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #2 0x00007fb6c446eeba in pthread_mutex_lock () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #3 0x00007fb6c24d6457 in gu::Lock::Lock(gu::Mutex const&) () from
> /usr/lib/galera/libgalera_smm.__so
> #4 0x00007fb6c25b2c91 in
> galera::Certification::set___trx_committed(galera::__TrxHandle*)
> () from
> /usr/lib/galera/libgalera_smm.__so
> #5 0x00007fb6c25e72c2 in galera::ReplicatorSMM::apply___trx(void*,
> galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.__so
> #6 0x00007fb6c25e77c5 in
> galera::ReplicatorSMM::__process_trx(void*,
> galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.__so
> #7 0x00007fb6c25be124 in
> galera::GcsActionSource::__dispatch(void*,
> gcs_action const&) () from /usr/lib/galera/libgalera_smm.__so
> #8 0x00007fb6c25be39a in
> galera::GcsActionSource::__process(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #9 0x00007fb6c25df175 in
> galera::ReplicatorSMM::async___recv(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #10 0x00007fb6c25f6f93 in galera_recv () from
> /usr/lib/galera/libgalera_smm.__so
> #11 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*__) ()
> #12 0x00007fb6c5a2d71b in start_wsrep_THD ()
> #13 0x00007fb6c446ce9a in start_thread () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #14 0x00007fb6c3b9acbd in clone () from
> /lib/x86_64-linux-gnu/libc.so.__6
> #15 0x0000000000000000 in ?? ()
>
> Thread 46 (Thread 0x7fb438066700 (LWP 27482)):
> #0 0x00007fb6c4470d84 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #1 0x00007fb6c25e9eb0 in
> galera::Monitor<galera::__ReplicatorSMM::LocalOrder>::__enter(galera::ReplicatorSMM::__LocalOrder&)
>
> () from /usr/lib/galera/libgalera_smm.__so
> #2 0x00007fb6c25e5a9f in
> galera::ReplicatorSMM::cert(__galera::TrxHandle*) () from
> /usr/lib/galera/libgalera_smm.__so
> #3 0x00007fb6c25e77b1 in
> galera::ReplicatorSMM::__process_trx(void*,
> galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.__so
> #4 0x00007fb6c25be124 in
> galera::GcsActionSource::__dispatch(void*,
> gcs_action const&) () from /usr/lib/galera/libgalera_smm.__so
> #5 0x00007fb6c25be39a in
> galera::GcsActionSource::__process(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #6 0x00007fb6c25df175 in
> galera::ReplicatorSMM::async___recv(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #7 0x00007fb6c25f6f93 in galera_recv () from
> /usr/lib/galera/libgalera_smm.__so
> #8 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*__) ()
> #9 0x00007fb6c5a2d71b in start_wsrep_THD ()
> #10 0x00007fb6c446ce9a in start_thread () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #11 0x00007fb6c3b9acbd in clone () from
> /lib/x86_64-linux-gnu/libc.so.__6
> #12 0x0000000000000000 in ?? ()
>
> Thread 45 (Thread 0x7f99d8f7a700 (LWP 27483)):
> #0 0x00007fb6c4470d84 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #1 0x00007fb6c25e9eb0 in
> galera::Monitor<galera::__ReplicatorSMM::LocalOrder>::__enter(galera::ReplicatorSMM::__LocalOrder&)
>
> () from /usr/lib/galera/libgalera_smm.__so
> #2 0x00007fb6c25e5a9f in
> galera::ReplicatorSMM::cert(__galera::TrxHandle*) () from
> /usr/lib/galera/libgalera_smm.__so
> #3 0x00007fb6c25e77b1 in
> galera::ReplicatorSMM::__process_trx(void*,
> galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.__so
> #4 0x00007fb6c25be124 in
> galera::GcsActionSource::__dispatch(void*,
> gcs_action const&) () from /usr/lib/galera/libgalera_smm.__so
> #5 0x00007fb6c25be39a in
> galera::GcsActionSource::__process(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #6 0x00007fb6c25df175 in
> galera::ReplicatorSMM::async___recv(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #7 0x00007fb6c25f6f93 in galera_recv () from
> /usr/lib/galera/libgalera_smm.__so
> #8 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*__) ()
> #9 0x00007fb6c5a2d71b in start_wsrep_THD ()
> #10 0x00007fb6c446ce9a in start_thread () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #11 0x00007fb6c3b9acbd in clone () from
> /lib/x86_64-linux-gnu/libc.so.__6
> #12 0x0000000000000000 in ?? ()
>
> Thread 44 (Thread 0x7f99d8f31700 (LWP 27484)):
> #0 0x00007fb6c4470d84 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #1 0x00007fb6c25e9eb0 in
> galera::Monitor<galera::__ReplicatorSMM::LocalOrder>::__enter(galera::ReplicatorSMM::__LocalOrder&)
>
> () from /usr/lib/galera/libgalera_smm.__so
> #2 0x00007fb6c25e5a9f in
> galera::ReplicatorSMM::cert(__galera::TrxHandle*) () from
> /usr/lib/galera/libgalera_smm.__so
> #3 0x00007fb6c25e77b1 in
> galera::ReplicatorSMM::__process_trx(void*,
> galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.__so
> #4 0x00007fb6c25be124 in
> galera::GcsActionSource::__dispatch(void*,
> gcs_action const&) () from /usr/lib/galera/libgalera_smm.__so
> #5 0x00007fb6c25be39a in
> galera::GcsActionSource::__process(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #6 0x00007fb6c25df175 in
> galera::ReplicatorSMM::async___recv(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #7 0x00007fb6c25f6f93 in galera_recv () from
> /usr/lib/galera/libgalera_smm.__so
> #8 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*__) ()
> #9 0x00007fb6c5a2d71b in start_wsrep_THD ()
> #10 0x00007fb6c446ce9a in start_thread () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #11 0x00007fb6c3b9acbd in clone () from
> /lib/x86_64-linux-gnu/libc.so.__6
> #12 0x0000000000000000 in ?? ()
>
> Thread 43 (Thread 0x7f99d8ee8700 (LWP 27485)):
> #0 0x00007fb6c447389c in __lll_lock_wait () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #1 0x00007fb6c446f065 in _L_lock_858 () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #2 0x00007fb6c446eeba in pthread_mutex_lock () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #3 0x00007fb6c24d6457 in gu::Lock::Lock(gu::Mutex const&) () from
> /usr/lib/galera/libgalera_smm.__so
> #4 0x00007fb6c25b2c91 in
> galera::Certification::set___trx_committed(galera::__TrxHandle*)
> () from
> /usr/lib/galera/libgalera_smm.__so
> #5 0x00007fb6c25e72c2 in galera::ReplicatorSMM::apply___trx(void*,
> galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.__so
> #6 0x00007fb6c25e77c5 in
> galera::ReplicatorSMM::__process_trx(void*,
> galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.__so
> #7 0x00007fb6c25be124 in
> galera::GcsActionSource::__dispatch(void*,
> gcs_action const&) () from /usr/lib/galera/libgalera_smm.__so
> #8 0x00007fb6c25be39a in
> galera::GcsActionSource::__process(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #9 0x00007fb6c25df175 in
> galera::ReplicatorSMM::async___recv(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #10 0x00007fb6c25f6f93 in galera_recv () from
> /usr/lib/galera/libgalera_smm.__so
> #11 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*__) ()
> #12 0x00007fb6c5a2d71b in start_wsrep_THD ()
> #13 0x00007fb6c446ce9a in start_thread () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #14 0x00007fb6c3b9acbd in clone () from
> /lib/x86_64-linux-gnu/libc.so.__6
> #15 0x0000000000000000 in ?? ()
>
> Thread 42 (Thread 0x7f99d8e9f700 (LWP 27486)):
> #0 0x00007fb6c447389c in __lll_lock_wait () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #1 0x00007fb6c446f065 in _L_lock_858 () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #2 0x00007fb6c446eeba in pthread_mutex_lock () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #3 0x00007fb6c24d6457 in gu::Lock::Lock(gu::Mutex const&) () from
> /usr/lib/galera/libgalera_smm.__so
> #4 0x00007fb6c25b2c91 in
> galera::Certification::set___trx_committed(galera::__TrxHandle*)
> () from
> /usr/lib/galera/libgalera_smm.__so
> #5 0x00007fb6c25e72c2 in galera::ReplicatorSMM::apply___trx(void*,
> galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.__so
> #6 0x00007fb6c25e77c5 in
> galera::ReplicatorSMM::__process_trx(void*,
> galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.__so
> #7 0x00007fb6c25be124 in
> galera::GcsActionSource::__dispatch(void*,
> gcs_action const&) () from /usr/lib/galera/libgalera_smm.__so
> #8 0x00007fb6c25be39a in
> galera::GcsActionSource::__process(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #9 0x00007fb6c25df175 in
> galera::ReplicatorSMM::async___recv(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #10 0x00007fb6c25f6f93 in galera_recv () from
> /usr/lib/galera/libgalera_smm.__so
> #11 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*__) ()
> #12 0x00007fb6c5a2d71b in start_wsrep_THD ()
> #13 0x00007fb6c446ce9a in start_thread () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #14 0x00007fb6c3b9acbd in clone () from
> /lib/x86_64-linux-gnu/libc.so.__6
> #15 0x0000000000000000 in ?? ()
>
> Thread 41 (Thread 0x7f99d8e56700 (LWP 27487)):
> #0 0x00007fb6c4470d84 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #1 0x00007fb6c25e9eb0 in
> galera::Monitor<galera::__ReplicatorSMM::LocalOrder>::__enter(galera::ReplicatorSMM::__LocalOrder&)
>
> () from /usr/lib/galera/libgalera_smm.__so
> #2 0x00007fb6c25e5a9f in
> galera::ReplicatorSMM::cert(__galera::TrxHandle*) () from
> /usr/lib/galera/libgalera_smm.__so
> #3 0x00007fb6c25e77b1 in
> galera::ReplicatorSMM::__process_trx(void*,
> galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.__so
> #4 0x00007fb6c25be124 in
> galera::GcsActionSource::__dispatch(void*,
> gcs_action const&) () from /usr/lib/galera/libgalera_smm.__so
> #5 0x00007fb6c25be39a in
> galera::GcsActionSource::__process(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #6 0x00007fb6c25df175 in
> galera::ReplicatorSMM::async___recv(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #7 0x00007fb6c25f6f93 in galera_recv () from
> /usr/lib/galera/libgalera_smm.__so
> #8 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*__) ()
> #9 0x00007fb6c5a2d71b in start_wsrep_THD ()
> #10 0x00007fb6c446ce9a in start_thread () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #11 0x00007fb6c3b9acbd in clone () from
> /lib/x86_64-linux-gnu/libc.so.__6
> #12 0x0000000000000000 in ?? ()
>
> Thread 40 (Thread 0x7f99d8e0d700 (LWP 27488)):
> #0 0x00007fb6c447389c in __lll_lock_wait () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #1 0x00007fb6c446f065 in _L_lock_858 () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #2 0x00007fb6c446eeba in pthread_mutex_lock () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #3 0x00007fb6c24d6457 in gu::Lock::Lock(gu::Mutex const&) () from
> /usr/lib/galera/libgalera_smm.__so
> #4 0x00007fb6c25b2c91 in
> galera::Certification::set___trx_committed(galera::__TrxHandle*)
> () from
> /usr/lib/galera/libgalera_smm.__so
> #5 0x00007fb6c25e72c2 in galera::ReplicatorSMM::apply___trx(void*,
> galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.__so
> #6 0x00007fb6c25e77c5 in
> galera::ReplicatorSMM::__process_trx(void*,
> galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.__so
> #7 0x00007fb6c25be124 in
> galera::GcsActionSource::__dispatch(void*,
> gcs_action const&) () from /usr/lib/galera/libgalera_smm.__so
> #8 0x00007fb6c25be39a in
> galera::GcsActionSource::__process(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #9 0x00007fb6c25df175 in
> galera::ReplicatorSMM::async___recv(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #10 0x00007fb6c25f6f93 in galera_recv () from
> /usr/lib/galera/libgalera_smm.__so
> #11 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*__) ()
> #12 0x00007fb6c5a2d71b in start_wsrep_THD ()
> #13 0x00007fb6c446ce9a in start_thread () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #14 0x00007fb6c3b9acbd in clone () from
> /lib/x86_64-linux-gnu/libc.so.__6
> #15 0x0000000000000000 in ?? ()
>
> Thread 39 (Thread 0x7f99d8dc4700 (LWP 27489)):
> #0 0x00007fb6c447389c in __lll_lock_wait () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #1 0x00007fb6c446f065 in _L_lock_858 () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #2 0x00007fb6c446eeba in pthread_mutex_lock () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #3 0x00007fb6c24d6457 in gu::Lock::Lock(gu::Mutex const&) () from
> /usr/lib/galera/libgalera_smm.__so
> #4 0x00007fb6c25b2c91 in
> galera::Certification::set___trx_committed(galera::__TrxHandle*)
> () from
> /usr/lib/galera/libgalera_smm.__so
> #5 0x00007fb6c25e72c2 in galera::ReplicatorSMM::apply___trx(void*,
> galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.__so
> #6 0x00007fb6c25e77c5 in
> galera::ReplicatorSMM::__process_trx(void*,
> galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.__so
> #7 0x00007fb6c25be124 in
> galera::GcsActionSource::__dispatch(void*,
> gcs_action const&) () from /usr/lib/galera/libgalera_smm.__so
> #8 0x00007fb6c25be39a in
> galera::GcsActionSource::__process(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #9 0x00007fb6c25df175 in
> galera::ReplicatorSMM::async___recv(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #10 0x00007fb6c25f6f93 in galera_recv () from
> /usr/lib/galera/libgalera_smm.__so
> #11 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*__) ()
> #12 0x00007fb6c5a2d71b in start_wsrep_THD ()
> #13 0x00007fb6c446ce9a in start_thread () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #14 0x00007fb6c3b9acbd in clone () from
> /lib/x86_64-linux-gnu/libc.so.__6
> #15 0x0000000000000000 in ?? ()
>
> Thread 38 (Thread 0x7f99d8d7b700 (LWP 27490)):
> #0 0x00007fb6c4470d84 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #1 0x00007fb6c25e9eb0 in
> galera::Monitor<galera::__ReplicatorSMM::LocalOrder>::__enter(galera::ReplicatorSMM::__LocalOrder&)
>
> () from /usr/lib/galera/libgalera_smm.__so
> #2 0x00007fb6c25e5a9f in
> galera::ReplicatorSMM::cert(__galera::TrxHandle*) () from
> /usr/lib/galera/libgalera_smm.__so
> #3 0x00007fb6c25e77b1 in
> galera::ReplicatorSMM::__process_trx(void*,
> galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.__so
> #4 0x00007fb6c25be124 in
> galera::GcsActionSource::__dispatch(void*,
> gcs_action const&) () from /usr/lib/galera/libgalera_smm.__so
> #5 0x00007fb6c25be39a in
> galera::GcsActionSource::__process(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #6 0x00007fb6c25df175 in
> galera::ReplicatorSMM::async___recv(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #7 0x00007fb6c25f6f93 in galera_recv () from
> /usr/lib/galera/libgalera_smm.__so
> #8 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*__) ()
> #9 0x00007fb6c5a2d71b in start_wsrep_THD ()
> #10 0x00007fb6c446ce9a in start_thread () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #11 0x00007fb6c3b9acbd in clone () from
> /lib/x86_64-linux-gnu/libc.so.__6
> #12 0x0000000000000000 in ?? ()
>
> Thread 37 (Thread 0x7f99d8d32700 (LWP 27491)):
> #0 0x00007fb6c447389c in __lll_lock_wait () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #1 0x00007fb6c446f065 in _L_lock_858 () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #2 0x00007fb6c446eeba in pthread_mutex_lock () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #3 0x00007fb6c24d6457 in gu::Lock::Lock(gu::Mutex const&) () from
> /usr/lib/galera/libgalera_smm.__so
> #4 0x00007fb6c25b2c91 in
> galera::Certification::set___trx_committed(galera::__TrxHandle*)
> () from
> /usr/lib/galera/libgalera_smm.__so
> #5 0x00007fb6c25e72c2 in galera::ReplicatorSMM::apply___trx(void*,
> galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.__so
> #6 0x00007fb6c25e77c5 in
> galera::ReplicatorSMM::__process_trx(void*,
> galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.__so
> #7 0x00007fb6c25be124 in
> galera::GcsActionSource::__dispatch(void*,
> gcs_action const&) () from /usr/lib/galera/libgalera_smm.__so
> #8 0x00007fb6c25be39a in
> galera::GcsActionSource::__process(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #9 0x00007fb6c25df175 in
> galera::ReplicatorSMM::async___recv(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #10 0x00007fb6c25f6f93 in galera_recv () from
> /usr/lib/galera/libgalera_smm.__so
> #11 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*__) ()
> #12 0x00007fb6c5a2d71b in start_wsrep_THD ()
> #13 0x00007fb6c446ce9a in start_thread () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #14 0x00007fb6c3b9acbd in clone () from
> /lib/x86_64-linux-gnu/libc.so.__6
> #15 0x0000000000000000 in ?? ()
>
> Thread 36 (Thread 0x7f99d8ce9700 (LWP 27492)):
> #0 0x00007fb6c447389c in __lll_lock_wait () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #1 0x00007fb6c446f065 in _L_lock_858 () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #2 0x00007fb6c446eeba in pthread_mutex_lock () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #3 0x00007fb6c24d6457 in gu::Lock::Lock(gu::Mutex const&) () from
> /usr/lib/galera/libgalera_smm.__so
> #4 0x00007fb6c25b2c91 in
> galera::Certification::set___trx_committed(galera::__TrxHandle*)
> () from
> /usr/lib/galera/libgalera_smm.__so
> #5 0x00007fb6c25e72c2 in galera::ReplicatorSMM::apply___trx(void*,
> galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.__so
> #6 0x00007fb6c25e77c5 in
> galera::ReplicatorSMM::__process_trx(void*,
> galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.__so
> #7 0x00007fb6c25be124 in
> galera::GcsActionSource::__dispatch(void*,
> gcs_action const&) () from /usr/lib/galera/libgalera_smm.__so
> #8 0x00007fb6c25be39a in
> galera::GcsActionSource::__process(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #9 0x00007fb6c25df175 in
> galera::ReplicatorSMM::async___recv(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #10 0x00007fb6c25f6f93 in galera_recv () from
> /usr/lib/galera/libgalera_smm.__so
> #11 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*__) ()
> #12 0x00007fb6c5a2d71b in start_wsrep_THD ()
> #13 0x00007fb6c446ce9a in start_thread () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #14 0x00007fb6c3b9acbd in clone () from
> /lib/x86_64-linux-gnu/libc.so.__6
> #15 0x0000000000000000 in ?? ()
>
> Thread 35 (Thread 0x7f99d8ca0700 (LWP 27493)):
> #0 0x00007fb6c447389c in __lll_lock_wait () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #1 0x00007fb6c446f065 in _L_lock_858 () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #2 0x00007fb6c446eeba in pthread_mutex_lock () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #3 0x00007fb6c24d6457 in gu::Lock::Lock(gu::Mutex const&) () from
> /usr/lib/galera/libgalera_smm.__so
> #4 0x00007fb6c25b2c91 in
> galera::Certification::set___trx_committed(galera::__TrxHandle*)
> () from
> /usr/lib/galera/libgalera_smm.__so
> #5 0x00007fb6c25e72c2 in galera::ReplicatorSMM::apply___trx(void*,
> galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.__so
> #6 0x00007fb6c25e77c5 in
> galera::ReplicatorSMM::__process_trx(void*,
> galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.__so
> #7 0x00007fb6c25be124 in
> galera::GcsActionSource::__dispatch(void*,
> gcs_action const&) () from /usr/lib/galera/libgalera_smm.__so
> #8 0x00007fb6c25be39a in
> galera::GcsActionSource::__process(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #9 0x00007fb6c25df175 in
> galera::ReplicatorSMM::async___recv(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #10 0x00007fb6c25f6f93 in galera_recv () from
> /usr/lib/galera/libgalera_smm.__so
> #11 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*__) ()
> #12 0x00007fb6c5a2d71b in start_wsrep_THD ()
> #13 0x00007fb6c446ce9a in start_thread () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #14 0x00007fb6c3b9acbd in clone () from
> /lib/x86_64-linux-gnu/libc.so.__6
> #15 0x0000000000000000 in ?? ()
>
> Thread 34 (Thread 0x7f99d8c57700 (LWP 27494)):
> #0 0x00007fb6c447389c in __lll_lock_wait () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #1 0x00007fb6c446f065 in _L_lock_858 () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #2 0x00007fb6c446eeba in pthread_mutex_lock () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #3 0x00007fb6c24d6457 in gu::Lock::Lock(gu::Mutex const&) () from
> /usr/lib/galera/libgalera_smm.__so
> #4 0x00007fb6c25b2c91 in
> galera::Certification::set___trx_committed(galera::__TrxHandle*)
> () from
> /usr/lib/galera/libgalera_smm.__so
> #5 0x00007fb6c25e72c2 in galera::ReplicatorSMM::apply___trx(void*,
> galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.__so
> #6 0x00007fb6c25e77c5 in
> galera::ReplicatorSMM::__process_trx(void*,
> galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.__so
> #7 0x00007fb6c25be124 in
> galera::GcsActionSource::__dispatch(void*,
> gcs_action const&) () from /usr/lib/galera/libgalera_smm.__so
> #8 0x00007fb6c25be39a in
> galera::GcsActionSource::__process(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #9 0x00007fb6c25df175 in
> galera::ReplicatorSMM::async___recv(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #10 0x00007fb6c25f6f93 in galera_recv () from
> /usr/lib/galera/libgalera_smm.__so
> #11 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*__) ()
> #12 0x00007fb6c5a2d71b in start_wsrep_THD ()
> #13 0x00007fb6c446ce9a in start_thread () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #14 0x00007fb6c3b9acbd in clone () from
> /lib/x86_64-linux-gnu/libc.so.__6
> #15 0x0000000000000000 in ?? ()
>
> Thread 33 (Thread 0x7f99d8c0e700 (LWP 27495)):
> #0 0x00007fb6c447389c in __lll_lock_wait () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #1 0x00007fb6c446f065 in _L_lock_858 () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #2 0x00007fb6c446eeba in pthread_mutex_lock () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #3 0x00007fb6c24d6457 in gu::Lock::Lock(gu::Mutex const&) () from
> /usr/lib/galera/libgalera_smm.__so
> #4 0x00007fb6c25b2c91 in
> galera::Certification::set___trx_committed(galera::__TrxHandle*)
> () from
> /usr/lib/galera/libgalera_smm.__so
> #5 0x00007fb6c25e72c2 in galera::ReplicatorSMM::apply___trx(void*,
> galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.__so
> #6 0x00007fb6c25e77c5 in
> galera::ReplicatorSMM::__process_trx(void*,
> galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.__so
> #7 0x00007fb6c25be124 in
> galera::GcsActionSource::__dispatch(void*,
> gcs_action const&) () from /usr/lib/galera/libgalera_smm.__so
> #8 0x00007fb6c25be39a in
> galera::GcsActionSource::__process(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #9 0x00007fb6c25df175 in
> galera::ReplicatorSMM::async___recv(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #10 0x00007fb6c25f6f93 in galera_recv () from
> /usr/lib/galera/libgalera_smm.__so
> #11 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*__) ()
> #12 0x00007fb6c5a2d71b in start_wsrep_THD ()
> #13 0x00007fb6c446ce9a in start_thread () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #14 0x00007fb6c3b9acbd in clone () from
> /lib/x86_64-linux-gnu/libc.so.__6
> #15 0x0000000000000000 in ?? ()
>
> Thread 32 (Thread 0x7f99d8bc5700 (LWP 27496)):
> #0 0x00007fb6c4470d84 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #1 0x00007fb6c25e9eb0 in
> galera::Monitor<galera::__ReplicatorSMM::LocalOrder>::__enter(galera::ReplicatorSMM::__LocalOrder&)
>
> () from /usr/lib/galera/libgalera_smm.__so
> #2 0x00007fb6c25e5a9f in
> galera::ReplicatorSMM::cert(__galera::TrxHandle*) () from
> /usr/lib/galera/libgalera_smm.__so
> #3 0x00007fb6c25e77b1 in
> galera::ReplicatorSMM::__process_trx(void*,
> galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.__so
> #4 0x00007fb6c25be124 in
> galera::GcsActionSource::__dispatch(void*,
> gcs_action const&) () from /usr/lib/galera/libgalera_smm.__so
> #5 0x00007fb6c25be39a in
> galera::GcsActionSource::__process(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #6 0x00007fb6c25df175 in
> galera::ReplicatorSMM::async___recv(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #7 0x00007fb6c25f6f93 in galera_recv () from
> /usr/lib/galera/libgalera_smm.__so
> #8 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*__) ()
> #9 0x00007fb6c5a2d71b in start_wsrep_THD ()
> #10 0x00007fb6c446ce9a in start_thread () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #11 0x00007fb6c3b9acbd in clone () from
> /lib/x86_64-linux-gnu/libc.so.__6
> #12 0x0000000000000000 in ?? ()
>
> Thread 31 (Thread 0x7f99d8b7c700 (LWP 27497)):
> #0 0x00007fb6c447389c in __lll_lock_wait () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #1 0x00007fb6c446f065 in _L_lock_858 () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #2 0x00007fb6c446eeba in pthread_mutex_lock () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #3 0x00007fb6c24d6457 in gu::Lock::Lock(gu::Mutex const&) () from
> /usr/lib/galera/libgalera_smm.__so
> #4 0x00007fb6c25b2c91 in
> galera::Certification::set___trx_committed(galera::__TrxHandle*)
> () from
> /usr/lib/galera/libgalera_smm.__so
> #5 0x00007fb6c25e72c2 in galera::ReplicatorSMM::apply___trx(void*,
> galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.__so
> #6 0x00007fb6c25e77c5 in
> galera::ReplicatorSMM::__process_trx(void*,
> galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.__so
> #7 0x00007fb6c25be124 in
> galera::GcsActionSource::__dispatch(void*,
> gcs_action const&) () from /usr/lib/galera/libgalera_smm.__so
> #8 0x00007fb6c25be39a in
> galera::GcsActionSource::__process(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #9 0x00007fb6c25df175 in
> galera::ReplicatorSMM::async___recv(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #10 0x00007fb6c25f6f93 in galera_recv () from
> /usr/lib/galera/libgalera_smm.__so
> #11 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*__) ()
> #12 0x00007fb6c5a2d71b in start_wsrep_THD ()
> #13 0x00007fb6c446ce9a in start_thread () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #14 0x00007fb6c3b9acbd in clone () from
> /lib/x86_64-linux-gnu/libc.so.__6
> #15 0x0000000000000000 in ?? ()
>
> Thread 30 (Thread 0x7f99d8b33700 (LWP 27498)):
> #0 0x00007fb6c4470d84 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #1 0x00007fb6c25e9eb0 in
> galera::Monitor<galera::__ReplicatorSMM::LocalOrder>::__enter(galera::ReplicatorSMM::__LocalOrder&)
>
> () from /usr/lib/galera/libgalera_smm.__so
> #2 0x00007fb6c25e5a9f in
> galera::ReplicatorSMM::cert(__galera::TrxHandle*) () from
> /usr/lib/galera/libgalera_smm.__so
> #3 0x00007fb6c25e77b1 in
> galera::ReplicatorSMM::__process_trx(void*,
> galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.__so
> #4 0x00007fb6c25be124 in
> galera::GcsActionSource::__dispatch(void*,
> gcs_action const&) () from /usr/lib/galera/libgalera_smm.__so
> #5 0x00007fb6c25be39a in
> galera::GcsActionSource::__process(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #6 0x00007fb6c25df175 in
> galera::ReplicatorSMM::async___recv(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #7 0x00007fb6c25f6f93 in galera_recv () from
> /usr/lib/galera/libgalera_smm.__so
> #8 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*__) ()
> #9 0x00007fb6c5a2d71b in start_wsrep_THD ()
> #10 0x00007fb6c446ce9a in start_thread () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #11 0x00007fb6c3b9acbd in clone () from
> /lib/x86_64-linux-gnu/libc.so.__6
> #12 0x0000000000000000 in ?? ()
>
> Thread 29 (Thread 0x7f99d8aea700 (LWP 27499)):
> #0 0x00007fb6c447389c in __lll_lock_wait () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #1 0x00007fb6c446f065 in _L_lock_858 () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #2 0x00007fb6c446eeba in pthread_mutex_lock () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #3 0x00007fb6c24d6457 in gu::Lock::Lock(gu::Mutex const&) () from
> /usr/lib/galera/libgalera_smm.__so
> #4 0x00007fb6c25b2c91 in
> galera::Certification::set___trx_committed(galera::__TrxHandle*)
> () from
> /usr/lib/galera/libgalera_smm.__so
> #5 0x00007fb6c25e72c2 in galera::ReplicatorSMM::apply___trx(void*,
> galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.__so
> #6 0x00007fb6c25e77c5 in
> galera::ReplicatorSMM::__process_trx(void*,
> galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.__so
> #7 0x00007fb6c25be124 in
> galera::GcsActionSource::__dispatch(void*,
> gcs_action const&) () from /usr/lib/galera/libgalera_smm.__so
> #8 0x00007fb6c25be39a in
> galera::GcsActionSource::__process(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #9 0x00007fb6c25df175 in
> galera::ReplicatorSMM::async___recv(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #10 0x00007fb6c25f6f93 in galera_recv () from
> /usr/lib/galera/libgalera_smm.__so
> #11 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*__) ()
> #12 0x00007fb6c5a2d71b in start_wsrep_THD ()
> #13 0x00007fb6c446ce9a in start_thread () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #14 0x00007fb6c3b9acbd in clone () from
> /lib/x86_64-linux-gnu/libc.so.__6
> #15 0x0000000000000000 in ?? ()
>
> Thread 28 (Thread 0x7f995d2ef700 (LWP 27500)):
> #0 0x00007fb6c447389c in __lll_lock_wait () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #1 0x00007fb6c446f065 in _L_lock_858 () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #2 0x00007fb6c446eeba in pthread_mutex_lock () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #3 0x00007fb6c24d6457 in gu::Lock::Lock(gu::Mutex const&) () from
> /usr/lib/galera/libgalera_smm.__so
> #4 0x00007fb6c25e7300 in galera::ReplicatorSMM::apply___trx(void*,
> galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.__so
> #5 0x00007fb6c25e77c5 in
> galera::ReplicatorSMM::__process_trx(void*,
> galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.__so
> #6 0x00007fb6c25be124 in
> galera::GcsActionSource::__dispatch(void*,
> gcs_action const&) () from /usr/lib/galera/libgalera_smm.__so
> #7 0x00007fb6c25be39a in
> galera::GcsActionSource::__process(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #8 0x00007fb6c25df175 in
> galera::ReplicatorSMM::async___recv(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #9 0x00007fb6c25f6f93 in galera_recv () from
> /usr/lib/galera/libgalera_smm.__so
> #10 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*__) ()
> #11 0x00007fb6c5a2d71b in start_wsrep_THD ()
> #12 0x00007fb6c446ce9a in start_thread () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #13 0x00007fb6c3b9acbd in clone () from
> /lib/x86_64-linux-gnu/libc.so.__6
> #14 0x0000000000000000 in ?? ()
>
> Thread 27 (Thread 0x7f99d8aa1700 (LWP 27501)):
> #0 0x00007fb6c4470d84 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #1 0x00007fb6c25e9eb0 in
> galera::Monitor<galera::__ReplicatorSMM::LocalOrder>::__enter(galera::ReplicatorSMM::__LocalOrder&)
>
> () from /usr/lib/galera/libgalera_smm.__so
> #2 0x00007fb6c25e5a9f in
> galera::ReplicatorSMM::cert(__galera::TrxHandle*) () from
> /usr/lib/galera/libgalera_smm.__so
> #3 0x00007fb6c25e77b1 in
> galera::ReplicatorSMM::__process_trx(void*,
> galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.__so
> #4 0x00007fb6c25be124 in
> galera::GcsActionSource::__dispatch(void*,
> gcs_action const&) () from /usr/lib/galera/libgalera_smm.__so
> #5 0x00007fb6c25be39a in
> galera::GcsActionSource::__process(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #6 0x00007fb6c25df175 in
> galera::ReplicatorSMM::async___recv(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #7 0x00007fb6c25f6f93 in galera_recv () from
> /usr/lib/galera/libgalera_smm.__so
> #8 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*__) ()
> #9 0x00007fb6c5a2d71b in start_wsrep_THD ()
> #10 0x00007fb6c446ce9a in start_thread () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #11 0x00007fb6c3b9acbd in clone () from
> /lib/x86_64-linux-gnu/libc.so.__6
> #12 0x0000000000000000 in ?? ()
>
> Thread 26 (Thread 0x7f99d8a58700 (LWP 27502)):
> #0 0x00007fb6c4470d84 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #1 0x00007fb6c25e9eb0 in
> galera::Monitor<galera::__ReplicatorSMM::LocalOrder>::__enter(galera::ReplicatorSMM::__LocalOrder&)
>
> () from /usr/lib/galera/libgalera_smm.__so
> #2 0x00007fb6c25e5a9f in
> galera::ReplicatorSMM::cert(__galera::TrxHandle*) () from
> /usr/lib/galera/libgalera_smm.__so
> #3 0x00007fb6c25e77b1 in
> galera::ReplicatorSMM::__process_trx(void*,
> galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.__so
> #4 0x00007fb6c25be124 in
> galera::GcsActionSource::__dispatch(void*,
> gcs_action const&) () from /usr/lib/galera/libgalera_smm.__so
> #5 0x00007fb6c25be39a in
> galera::GcsActionSource::__process(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #6 0x00007fb6c25df175 in
> galera::ReplicatorSMM::async___recv(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #7 0x00007fb6c25f6f93 in galera_recv () from
> /usr/lib/galera/libgalera_smm.__so
> #8 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*__) ()
> #9 0x00007fb6c5a2d71b in start_wsrep_THD ()
> #10 0x00007fb6c446ce9a in start_thread () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #11 0x00007fb6c3b9acbd in clone () from
> /lib/x86_64-linux-gnu/libc.so.__6
> #12 0x0000000000000000 in ?? ()
>
> Thread 25 (Thread 0x7f99d8a0f700 (LWP 27503)):
> #0 0x00007fb6c4470d84 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #1 0x00007fb6c25e9eb0 in
> galera::Monitor<galera::__ReplicatorSMM::LocalOrder>::__enter(galera::ReplicatorSMM::__LocalOrder&)
>
> () from /usr/lib/galera/libgalera_smm.__so
> #2 0x00007fb6c25e5a9f in
> galera::ReplicatorSMM::cert(__galera::TrxHandle*) () from
> /usr/lib/galera/libgalera_smm.__so
> #3 0x00007fb6c25e77b1 in
> galera::ReplicatorSMM::__process_trx(void*,
> galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.__so
> #4 0x00007fb6c25be124 in
> galera::GcsActionSource::__dispatch(void*,
> gcs_action const&) () from /usr/lib/galera/libgalera_smm.__so
> #5 0x00007fb6c25be39a in
> galera::GcsActionSource::__process(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #6 0x00007fb6c25df175 in
> galera::ReplicatorSMM::async___recv(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #7 0x00007fb6c25f6f93 in galera_recv () from
> /usr/lib/galera/libgalera_smm.__so
> #8 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*__) ()
> #9 0x00007fb6c5a2d71b in start_wsrep_THD ()
> #10 0x00007fb6c446ce9a in start_thread () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #11 0x00007fb6c3b9acbd in clone () from
> /lib/x86_64-linux-gnu/libc.so.__6
> #12 0x0000000000000000 in ?? ()
>
> Thread 24 (Thread 0x7f99d89c6700 (LWP 27504)):
> #0 0x00007fb6c447389c in __lll_lock_wait () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #1 0x00007fb6c446f065 in _L_lock_858 () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #2 0x00007fb6c446eeba in pthread_mutex_lock () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #3 0x00007fb6c24d6457 in gu::Lock::Lock(gu::Mutex const&) () from
> /usr/lib/galera/libgalera_smm.__so
> #4 0x00007fb6c25b2c91 in
> galera::Certification::set___trx_committed(galera::__TrxHandle*)
> () from
> /usr/lib/galera/libgalera_smm.__so
> #5 0x00007fb6c25e72c2 in galera::ReplicatorSMM::apply___trx(void*,
> galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.__so
> #6 0x00007fb6c25e77c5 in
> galera::ReplicatorSMM::__process_trx(void*,
> galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.__so
> #7 0x00007fb6c25be124 in
> galera::GcsActionSource::__dispatch(void*,
> gcs_action const&) () from /usr/lib/galera/libgalera_smm.__so
> #8 0x00007fb6c25be39a in
> galera::GcsActionSource::__process(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #9 0x00007fb6c25df175 in
> galera::ReplicatorSMM::async___recv(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #10 0x00007fb6c25f6f93 in galera_recv () from
> /usr/lib/galera/libgalera_smm.__so
> #11 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*__) ()
> #12 0x00007fb6c5a2d71b in start_wsrep_THD ()
> #13 0x00007fb6c446ce9a in start_thread () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #14 0x00007fb6c3b9acbd in clone () from
> /lib/x86_64-linux-gnu/libc.so.__6
> #15 0x0000000000000000 in ?? ()
>
> Thread 23 (Thread 0x7f99d897d700 (LWP 27505)):
> #0 0x00007fb6c4470d84 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #1 0x00007fb6c25e9eb0 in
> galera::Monitor<galera::__ReplicatorSMM::LocalOrder>::__enter(galera::ReplicatorSMM::__LocalOrder&)
>
> () from /usr/lib/galera/libgalera_smm.__so
> #2 0x00007fb6c25e5a9f in
> galera::ReplicatorSMM::cert(__galera::TrxHandle*) () from
> /usr/lib/galera/libgalera_smm.__so
> #3 0x00007fb6c25e77b1 in
> galera::ReplicatorSMM::__process_trx(void*,
> galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.__so
> #4 0x00007fb6c25be124 in
> galera::GcsActionSource::__dispatch(void*,
> gcs_action const&) () from /usr/lib/galera/libgalera_smm.__so
> #5 0x00007fb6c25be39a in
> galera::GcsActionSource::__process(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #6 0x00007fb6c25df175 in
> galera::ReplicatorSMM::async___recv(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #7 0x00007fb6c25f6f93 in galera_recv () from
> /usr/lib/galera/libgalera_smm.__so
> #8 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*__) ()
> #9 0x00007fb6c5a2d71b in start_wsrep_THD ()
> #10 0x00007fb6c446ce9a in start_thread () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #11 0x00007fb6c3b9acbd in clone () from
> /lib/x86_64-linux-gnu/libc.so.__6
> #12 0x0000000000000000 in ?? ()
>
> Thread 22 (Thread 0x7f99d8934700 (LWP 27506)):
> #0 0x00007fb6c447389c in __lll_lock_wait () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #1 0x00007fb6c446f065 in _L_lock_858 () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #2 0x00007fb6c446eeba in pthread_mutex_lock () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #3 0x00007fb6c24d6457 in gu::Lock::Lock(gu::Mutex const&) () from
> /usr/lib/galera/libgalera_smm.__so
> #4 0x00007fb6c25b2c91 in
> galera::Certification::set___trx_committed(galera::__TrxHandle*)
> () from
> /usr/lib/galera/libgalera_smm.__so
> #5 0x00007fb6c25e72c2 in galera::ReplicatorSMM::apply___trx(void*,
> galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.__so
> #6 0x00007fb6c25e77c5 in
> galera::ReplicatorSMM::__process_trx(void*,
> galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.__so
> #7 0x00007fb6c25be124 in
> galera::GcsActionSource::__dispatch(void*,
> gcs_action const&) () from /usr/lib/galera/libgalera_smm.__so
> #8 0x00007fb6c25be39a in
> galera::GcsActionSource::__process(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #9 0x00007fb6c25df175 in
> galera::ReplicatorSMM::async___recv(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #10 0x00007fb6c25f6f93 in galera_recv () from
> /usr/lib/galera/libgalera_smm.__so
> #11 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*__) ()
> #12 0x00007fb6c5a2d71b in start_wsrep_THD ()
> #13 0x00007fb6c446ce9a in start_thread () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #14 0x00007fb6c3b9acbd in clone () from
> /lib/x86_64-linux-gnu/libc.so.__6
> #15 0x0000000000000000 in ?? ()
>
> Thread 21 (Thread 0x7f99d88eb700 (LWP 27507)):
> #0 0x00007fb6c447389c in __lll_lock_wait () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #1 0x00007fb6c446f065 in _L_lock_858 () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #2 0x00007fb6c446eeba in pthread_mutex_lock () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #3 0x00007fb6c25b9c4d in
> galera::Certification::__PurgeAndDiscard::operator()(__std::pair<long
> const,
> galera::TrxHandle*>&) const () from
> /usr/lib/galera/libgalera_smm.__so
> #4 0x00007fb6c25b429c in
> galera::Certification::purge___trxs_upto_(long)
> () from /usr/lib/galera/libgalera_smm.__so
> #5 0x00007fb6c25de292 in
> galera::ReplicatorSMM::__process_commit_cut(long, long) () from
> /usr/lib/galera/libgalera_smm.__so
> #6 0x00007fb6c25be161 in
> galera::GcsActionSource::__dispatch(void*,
> gcs_action const&) () from /usr/lib/galera/libgalera_smm.__so
> #7 0x00007fb6c25be39a in
> galera::GcsActionSource::__process(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #8 0x00007fb6c25df175 in
> galera::ReplicatorSMM::async___recv(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #9 0x00007fb6c25f6f93 in galera_recv () from
> /usr/lib/galera/libgalera_smm.__so
> #10 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*__) ()
> #11 0x00007fb6c5a2d71b in start_wsrep_THD ()
> #12 0x00007fb6c446ce9a in start_thread () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #13 0x00007fb6c3b9acbd in clone () from
> /lib/x86_64-linux-gnu/libc.so.__6
> #14 0x0000000000000000 in ?? ()
>
> Thread 20 (Thread 0x7f99d88a2700 (LWP 27508)):
> #0 0x00007fb6c447389c in __lll_lock_wait () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #1 0x00007fb6c446f065 in _L_lock_858 () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #2 0x00007fb6c446eeba in pthread_mutex_lock () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #3 0x00007fb6c24d6457 in gu::Lock::Lock(gu::Mutex const&) () from
> /usr/lib/galera/libgalera_smm.__so
> #4 0x00007fb6c25b2c91 in
> galera::Certification::set___trx_committed(galera::__TrxHandle*)
> () from
> /usr/lib/galera/libgalera_smm.__so
> #5 0x00007fb6c25e72c2 in galera::ReplicatorSMM::apply___trx(void*,
> galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.__so
> #6 0x00007fb6c25e77c5 in
> galera::ReplicatorSMM::__process_trx(void*,
> galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.__so
> #7 0x00007fb6c25be124 in
> galera::GcsActionSource::__dispatch(void*,
> gcs_action const&) () from /usr/lib/galera/libgalera_smm.__so
> #8 0x00007fb6c25be39a in
> galera::GcsActionSource::__process(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #9 0x00007fb6c25df175 in
> galera::ReplicatorSMM::async___recv(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #10 0x00007fb6c25f6f93 in galera_recv () from
> /usr/lib/galera/libgalera_smm.__so
> #11 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*__) ()
> #12 0x00007fb6c5a2d71b in start_wsrep_THD ()
> #13 0x00007fb6c446ce9a in start_thread () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #14 0x00007fb6c3b9acbd in clone () from
> /lib/x86_64-linux-gnu/libc.so.__6
> #15 0x0000000000000000 in ?? ()
>
> Thread 19 (Thread 0x7f99d8859700 (LWP 27509)):
> #0 0x00007fb6c447389c in __lll_lock_wait () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #1 0x00007fb6c446f065 in _L_lock_858 () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #2 0x00007fb6c446eeba in pthread_mutex_lock () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #3 0x00007fb6c24d6457 in gu::Lock::Lock(gu::Mutex const&) () from
> /usr/lib/galera/libgalera_smm.__so
> #4 0x00007fb6c25b2c91 in
> galera::Certification::set___trx_committed(galera::__TrxHandle*)
> () from
> /usr/lib/galera/libgalera_smm.__so
> #5 0x00007fb6c25e72c2 in galera::ReplicatorSMM::apply___trx(void*,
> galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.__so
> #6 0x00007fb6c25e77c5 in
> galera::ReplicatorSMM::__process_trx(void*,
> galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.__so
> #7 0x00007fb6c25be124 in
> galera::GcsActionSource::__dispatch(void*,
> gcs_action const&) () from /usr/lib/galera/libgalera_smm.__so
> #8 0x00007fb6c25be39a in
> galera::GcsActionSource::__process(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #9 0x00007fb6c25df175 in
> galera::ReplicatorSMM::async___recv(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #10 0x00007fb6c25f6f93 in galera_recv () from
> /usr/lib/galera/libgalera_smm.__so
> #11 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*__) ()
> #12 0x00007fb6c5a2d71b in start_wsrep_THD ()
> #13 0x00007fb6c446ce9a in start_thread () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #14 0x00007fb6c3b9acbd in clone () from
> /lib/x86_64-linux-gnu/libc.so.__6
> #15 0x0000000000000000 in ?? ()
>
> Thread 18 (Thread 0x7f99d8810700 (LWP 27510)):
> #0 0x00007fb6c447389c in __lll_lock_wait () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #1 0x00007fb6c446f065 in _L_lock_858 () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #2 0x00007fb6c446eeba in pthread_mutex_lock () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #3 0x00007fb6c24d6457 in gu::Lock::Lock(gu::Mutex const&) () from
> /usr/lib/galera/libgalera_smm.__so
> #4 0x00007fb6c25b2c91 in
> galera::Certification::set___trx_committed(galera::__TrxHandle*)
> () from
> /usr/lib/galera/libgalera_smm.__so
> #5 0x00007fb6c25e72c2 in galera::ReplicatorSMM::apply___trx(void*,
> galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.__so
> #6 0x00007fb6c25e77c5 in
> galera::ReplicatorSMM::__process_trx(void*,
> galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.__so
> #7 0x00007fb6c25be124 in
> galera::GcsActionSource::__dispatch(void*,
> gcs_action const&) () from /usr/lib/galera/libgalera_smm.__so
> #8 0x00007fb6c25be39a in
> galera::GcsActionSource::__process(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #9 0x00007fb6c25df175 in
> galera::ReplicatorSMM::async___recv(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #10 0x00007fb6c25f6f93 in galera_recv () from
> /usr/lib/galera/libgalera_smm.__so
> #11 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*__) ()
> #12 0x00007fb6c5a2d71b in start_wsrep_THD ()
> #13 0x00007fb6c446ce9a in start_thread () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #14 0x00007fb6c3b9acbd in clone () from
> /lib/x86_64-linux-gnu/libc.so.__6
> #15 0x0000000000000000 in ?? ()
>
> Thread 17 (Thread 0x7f99d87c7700 (LWP 27511)):
> #0 0x00007fb6c4470d84 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #1 0x00007fb6c25e9eb0 in
> galera::Monitor<galera::__ReplicatorSMM::LocalOrder>::__enter(galera::ReplicatorSMM::__LocalOrder&)
>
> () from /usr/lib/galera/libgalera_smm.__so
> #2 0x00007fb6c25e5a9f in
> galera::ReplicatorSMM::cert(__galera::TrxHandle*) () from
> /usr/lib/galera/libgalera_smm.__so
> #3 0x00007fb6c25e77b1 in
> galera::ReplicatorSMM::__process_trx(void*,
> galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.__so
> #4 0x00007fb6c25be124 in
> galera::GcsActionSource::__dispatch(void*,
> gcs_action const&) () from /usr/lib/galera/libgalera_smm.__so
> #5 0x00007fb6c25be39a in
> galera::GcsActionSource::__process(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #6 0x00007fb6c25df175 in
> galera::ReplicatorSMM::async___recv(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #7 0x00007fb6c25f6f93 in galera_recv () from
> /usr/lib/galera/libgalera_smm.__so
> #8 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*__) ()
> #9 0x00007fb6c5a2d71b in start_wsrep_THD ()
> #10 0x00007fb6c446ce9a in start_thread () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #11 0x00007fb6c3b9acbd in clone () from
> /lib/x86_64-linux-gnu/libc.so.__6
> #12 0x0000000000000000 in ?? ()
>
> Thread 16 (Thread 0x7f99d877e700 (LWP 27512)):
> #0 0x00007fb6c4470d84 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #1 0x00007fb6c25e9eb0 in
> galera::Monitor<galera::__ReplicatorSMM::LocalOrder>::__enter(galera::ReplicatorSMM::__LocalOrder&)
>
> () from /usr/lib/galera/libgalera_smm.__so
> #2 0x00007fb6c25e5a9f in
> galera::ReplicatorSMM::cert(__galera::TrxHandle*) () from
> /usr/lib/galera/libgalera_smm.__so
> #3 0x00007fb6c25e77b1 in
> galera::ReplicatorSMM::__process_trx(void*,
> galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.__so
> #4 0x00007fb6c25be124 in
> galera::GcsActionSource::__dispatch(void*,
> gcs_action const&) () from /usr/lib/galera/libgalera_smm.__so
> #5 0x00007fb6c25be39a in
> galera::GcsActionSource::__process(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #6 0x00007fb6c25df175 in
> galera::ReplicatorSMM::async___recv(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #7 0x00007fb6c25f6f93 in galera_recv () from
> /usr/lib/galera/libgalera_smm.__so
> #8 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*__) ()
> #9 0x00007fb6c5a2d71b in start_wsrep_THD ()
> #10 0x00007fb6c446ce9a in start_thread () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #11 0x00007fb6c3b9acbd in clone () from
> /lib/x86_64-linux-gnu/libc.so.__6
> #12 0x0000000000000000 in ?? ()
>
> Thread 15 (Thread 0x7f99d8735700 (LWP 27513)):
> #0 0x00007fb6c447389c in __lll_lock_wait () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #1 0x00007fb6c446f065 in _L_lock_858 () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #2 0x00007fb6c446eeba in pthread_mutex_lock () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #3 0x00007fb6c24d6457 in gu::Lock::Lock(gu::Mutex const&) () from
> /usr/lib/galera/libgalera_smm.__so
> #4 0x00007fb6c25b2c91 in
> galera::Certification::set___trx_committed(galera::__TrxHandle*)
> () from
> /usr/lib/galera/libgalera_smm.__so
> #5 0x00007fb6c25e72c2 in galera::ReplicatorSMM::apply___trx(void*,
> galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.__so
> #6 0x00007fb6c25e77c5 in
> galera::ReplicatorSMM::__process_trx(void*,
> galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.__so
> #7 0x00007fb6c25be124 in
> galera::GcsActionSource::__dispatch(void*,
> gcs_action const&) () from /usr/lib/galera/libgalera_smm.__so
> #8 0x00007fb6c25be39a in
> galera::GcsActionSource::__process(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #9 0x00007fb6c25df175 in
> galera::ReplicatorSMM::async___recv(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #10 0x00007fb6c25f6f93 in galera_recv () from
> /usr/lib/galera/libgalera_smm.__so
> #11 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*__) ()
> #12 0x00007fb6c5a2d71b in start_wsrep_THD ()
> #13 0x00007fb6c446ce9a in start_thread () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #14 0x00007fb6c3b9acbd in clone () from
> /lib/x86_64-linux-gnu/libc.so.__6
> #15 0x0000000000000000 in ?? ()
>
> Thread 14 (Thread 0x7f99d86ec700 (LWP 27514)):
> #0 0x00007fb6c447389c in __lll_lock_wait () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #1 0x00007fb6c446f065 in _L_lock_858 () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #2 0x00007fb6c446eeba in pthread_mutex_lock () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #3 0x00007fb6c24d6457 in gu::Lock::Lock(gu::Mutex const&) () from
> /usr/lib/galera/libgalera_smm.__so
> #4 0x00007fb6c25b2c91 in
> galera::Certification::set___trx_committed(galera::__TrxHandle*)
> () from
> /usr/lib/galera/libgalera_smm.__so
> #5 0x00007fb6c25e72c2 in galera::ReplicatorSMM::apply___trx(void*,
> galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.__so
> #6 0x00007fb6c25e77c5 in
> galera::ReplicatorSMM::__process_trx(void*,
> galera::TrxHandle*) () from /usr/lib/galera/libgalera_smm.__so
> #7 0x00007fb6c25be124 in
> galera::GcsActionSource::__dispatch(void*,
> gcs_action const&) () from /usr/lib/galera/libgalera_smm.__so
> #8 0x00007fb6c25be39a in
> galera::GcsActionSource::__process(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #9 0x00007fb6c25df175 in
> galera::ReplicatorSMM::async___recv(void*) ()
> from /usr/lib/galera/libgalera_smm.__so
> #10 0x00007fb6c25f6f93 in galera_recv () from
> /usr/lib/galera/libgalera_smm.__so
> #11 0x00007fb6c5ab1ab1 in wsrep_replication_process(THD*__) ()
> #12 0x00007fb6c5a2d71b in start_wsrep_THD ()
> #13 0x00007fb6c446ce9a in start_thread () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #14 0x00007fb6c3b9acbd in clone () from
> /lib/x86_64-linux-gnu/libc.so.__6
> #15 0x0000000000000000 in ?? ()
>
> Thread 13 (Thread 0x7f99d86a3700 (LWP 27515)):
> #0 0x00007fb6c447389c in __lll_lock_wait () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #1 0x00007fb6c446f09b in _L_lock_1006 () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #2 0x00007fb6c446f01c in pthread_mutex_lock () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #3 0x00007fb6c5fc50d7 in my_pthread_fastmutex_lock ()
> #4 0x00007fb6c5a2ad48 in ?? ()
> #5 0x00007fb6c5a2d9ad in unlink_thd(THD*) ()
> #6 0x00007fb6c5a2da84 in one_thread_per_connection_end(__THD*,
> bool) ()
> #7 0x00007fb6c5b6bed3 in do_handle_one_connection(THD*) ()
> #8 0x00007fb6c5b6c181 in handle_one_connection ()
> #9 0x00007fb6c446ce9a in start_thread () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #10 0x00007fb6c3b9acbd in clone () from
> /lib/x86_64-linux-gnu/libc.so.__6
> #11 0x0000000000000000 in ?? ()
>
> Thread 12 (Thread 0x7f99d8236700 (LWP 27534)):
> #0 0x00007fb6c447389c in __lll_lock_wait () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #1 0x00007fb6c446f09b in _L_lock_1006 () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #2 0x00007fb6c446f01c in pthread_mutex_lock () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #3 0x00007fb6c5fc50d7 in my_pthread_fastmutex_lock ()
> #4 0x00007fb6c5a2ad48 in ?? ()
> #5 0x00007fb6c5a2d9ad in unlink_thd(THD*) ()
> #6 0x00007fb6c5a2da84 in one_thread_per_connection_end(__THD*,
> bool) ()
> #7 0x00007fb6c5b6bed3 in do_handle_one_connection(THD*) ()
> #8 0x00007fb6c5b6c181 in handle_one_connection ()
> #9 0x00007fb6c446ce9a in start_thread () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #10 0x00007fb6c3b9acbd in clone () from
> /lib/x86_64-linux-gnu/libc.so.__6
> #11 0x0000000000000000 in ?? ()
>
> Thread 11 (Thread 0x7fb6c0654700 (LWP 4188)):
> #0 0x00007fb6c447389c in __lll_lock_wait () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #1 0x00007fb6c446f09b in _L_lock_1006 () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #2 0x00007fb6c446f01c in pthread_mutex_lock () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #3 0x00007fb6c5fc50d7 in my_pthread_fastmutex_lock ()
> #4 0x00007fb6c5a2ad48 in ?? ()
> #5 0x00007fb6c5a2d9ad in unlink_thd(THD*) ()
> #6 0x00007fb6c5a2da84 in one_thread_per_connection_end(__THD*,
> bool) ()
> #7 0x00007fb6c5b6bed3 in do_handle_one_connection(THD*) ()
> #8 0x00007fb6c5b6c181 in handle_one_connection ()
> #9 0x00007fb6c446ce9a in start_thread () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #10 0x00007fb6c3b9acbd in clone () from
> /lib/x86_64-linux-gnu/libc.so.__6
> #11 0x0000000000000000 in ?? ()
>
> Thread 10 (Thread 0x7fb6c060b700 (LWP 4189)):
> #0 0x00007fb6c447389c in __lll_lock_wait () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #1 0x00007fb6c446f09b in _L_lock_1006 () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #2 0x00007fb6c446f01c in pthread_mutex_lock () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #3 0x00007fb6c5fc50d7 in my_pthread_fastmutex_lock ()
> #4 0x00007fb6c5a2ad48 in ?? ()
> #5 0x00007fb6c5a2d9ad in unlink_thd(THD*) ()
> #6 0x00007fb6c5a2da84 in one_thread_per_connection_end(__THD*,
> bool) ()
> #7 0x00007fb6c5b6bed3 in do_handle_one_connection(THD*) ()
> #8 0x00007fb6c5b6c181 in handle_one_connection ()
> #9 0x00007fb6c446ce9a in start_thread () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #10 0x00007fb6c3b9acbd in clone () from
> /lib/x86_64-linux-gnu/libc.so.__6
> #11 0x0000000000000000 in ?? ()
>
> Thread 9 (Thread 0x7fb6c05c2700 (LWP 4205)):
> #0 0x00007fb6c447389c in __lll_lock_wait () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #1 0x00007fb6c446f09b in _L_lock_1006 () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #2 0x00007fb6c446f01c in pthread_mutex_lock () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #3 0x00007fb6c5fc50d7 in my_pthread_fastmutex_lock ()
> #4 0x00007fb6c5a2ad48 in ?? ()
> #5 0x00007fb6c5a2d9ad in unlink_thd(THD*) ()
> #6 0x00007fb6c5a2da84 in one_thread_per_connection_end(__THD*,
> bool) ()
> #7 0x00007fb6c5b6bed3 in do_handle_one_connection(THD*) ()
> #8 0x00007fb6c5b6c181 in handle_one_connection ()
> #9 0x00007fb6c446ce9a in start_thread () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #10 0x00007fb6c3b9acbd in clone () from
> /lib/x86_64-linux-gnu/libc.so.__6
> #11 0x0000000000000000 in ?? ()
>
> Thread 8 (Thread 0x7f99d81ed700 (LWP 29647)):
> #0 0x00007fb6c447389c in __lll_lock_wait () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #1 0x00007fb6c446f09b in _L_lock_1006 () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #2 0x00007fb6c446f01c in pthread_mutex_lock () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #3 0x00007fb6c5fc50d7 in my_pthread_fastmutex_lock ()
> #4 0x00007fb6c5a2ad48 in ?? ()
> #5 0x00007fb6c5a2d9ad in unlink_thd(THD*) ()
> #6 0x00007fb6c5a2da84 in one_thread_per_connection_end(__THD*,
> bool) ()
> #7 0x00007fb6c5b6bed3 in do_handle_one_connection(THD*) ()
> #8 0x00007fb6c5b6c181 in handle_one_connection ()
> #9 0x00007fb6c446ce9a in start_thread () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #10 0x00007fb6c3b9acbd in clone () from
> /lib/x86_64-linux-gnu/libc.so.__6
> #11 0x0000000000000000 in ?? ()
>
> Thread 7 (Thread 0x7f99d81a4700 (LWP 2266)):
> #0 0x00007fb6c447389c in __lll_lock_wait () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #1 0x00007fb6c446f09b in _L_lock_1006 () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #2 0x00007fb6c446f01c in pthread_mutex_lock () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #3 0x00007fb6c5fc50d7 in my_pthread_fastmutex_lock ()
> #4 0x00007fb6c5a2ad48 in ?? ()
> #5 0x00007fb6c5a2d9ad in unlink_thd(THD*) ()
> #6 0x00007fb6c5a2da84 in one_thread_per_connection_end(__THD*,
> bool) ()
> #7 0x00007fb6c5b6bed3 in do_handle_one_connection(THD*) ()
> #8 0x00007fb6c5b6c181 in handle_one_connection ()
> #9 0x00007fb6c446ce9a in start_thread () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #10 0x00007fb6c3b9acbd in clone () from
> /lib/x86_64-linux-gnu/libc.so.__6
> #11 0x0000000000000000 in ?? ()
>
> Thread 6 (Thread 0x7f99d815b700 (LWP 2267)):
> #0 0x00007fb6c447389c in __lll_lock_wait () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #1 0x00007fb6c446f09b in _L_lock_1006 () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #2 0x00007fb6c446f01c in pthread_mutex_lock () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #3 0x00007fb6c5fc50d7 in my_pthread_fastmutex_lock ()
> #4 0x00007fb6c5a2ad48 in ?? ()
> #5 0x00007fb6c5a2d9ad in unlink_thd(THD*) ()
> #6 0x00007fb6c5a2da84 in one_thread_per_connection_end(__THD*,
> bool) ()
> #7 0x00007fb6c5b6bed3 in do_handle_one_connection(THD*) ()
> #8 0x00007fb6c5b6c181 in handle_one_connection ()
> #9 0x00007fb6c446ce9a in start_thread () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #10 0x00007fb6c3b9acbd in clone () from
> /lib/x86_64-linux-gnu/libc.so.__6
> #11 0x0000000000000000 in ?? ()
>
> Thread 5 (Thread 0x7f99d8112700 (LWP 3577)):
> #0 0x00007fb6c447389c in __lll_lock_wait () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #1 0x00007fb6c446f065 in _L_lock_858 () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #2 0x00007fb6c446eeba in pthread_mutex_lock () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #3 0x00007fb6c24d6457 in gu::Lock::Lock(gu::Mutex const&) () from
> /usr/lib/galera/libgalera_smm.__so
> #4 0x00007fb6c25f38a8 in galera::ReplicatorSMM::stats___get()
> const ()
> from /usr/lib/galera/libgalera_smm.__so
> #5 0x00007fb6c5bab110 in wsrep_show_status(THD*,
> st_mysql_show_var*,
> char*) ()
> #6 0x00007fb6c5b04ebc in ?? ()
> #7 0x00007fb6c5b144e1 in fill_status(THD*, TABLE_LIST*, Item*) ()
> #8 0x00007fb6c5b194d7 in get_schema_tables_result(JOIN*__,
> enum_schema_table_state) ()
> #9 0x00007fb6c5afef5d in JOIN::exec() ()
> #10 0x00007fb6c5afa612 in mysql_select(THD*, Item***, TABLE_LIST*,
> unsigned int, List<Item>&, Item*, unsigned int, st_order*,
> st_order*,
> Item*, st_order*, unsigned long long, select_result*,
> st_select_lex_unit*, st_select_lex*) ()
> #11 0x00007fb6c5b013a3 in handle_select(THD*, LEX*, select_result*,
> unsigned long) ()
> #12 0x00007fb6c5aa9fe1 in ?? ()
> #13 0x00007fb6c5ab6296 in mysql_execute_command(THD*) ()
> #14 0x00007fb6c5ab8fb7 in ?? ()
> #15 0x00007fb6c5ab93af in ?? ()
> #16 0x00007fb6c5abb02a in
> dispatch_command(enum_server___command, THD*,
> char*, unsigned int) ()
> #17 0x00007fb6c5abb789 in do_command(THD*) ()
> #18 0x00007fb6c5b6c0db in do_handle_one_connection(THD*) ()
> #19 0x00007fb6c5b6c181 in handle_one_connection ()
> #20 0x00007fb6c446ce9a in start_thread () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #21 0x00007fb6c3b9acbd in clone () from
> /lib/x86_64-linux-gnu/libc.so.__6
> #22 0x0000000000000000 in ?? ()
>
> Thread 4 (Thread 0x7f99d80c9700 (LWP 3578)):
> #0 0x00007fb6c447389c in __lll_lock_wait () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #1 0x00007fb6c446f09b in _L_lock_1006 () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #2 0x00007fb6c446f01c in pthread_mutex_lock () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #3 0x00007fb6c5fc50d7 in my_pthread_fastmutex_lock ()
> #4 0x00007fb6c5a2ad48 in ?? ()
> #5 0x00007fb6c5a2d9ad in unlink_thd(THD*) ()
> #6 0x00007fb6c5a2da84 in one_thread_per_connection_end(__THD*,
> bool) ()
> #7 0x00007fb6c5b6bed3 in do_handle_one_connection(THD*) ()
> #8 0x00007fb6c5b6c181 in handle_one_connection ()
> #9 0x00007fb6c446ce9a in start_thread () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #10 0x00007fb6c3b9acbd in clone () from
> /lib/x86_64-linux-gnu/libc.so.__6
> #11 0x0000000000000000 in ?? ()
>
> Thread 3 (Thread 0x7f99d8080700 (LWP 4954)):
> #0 0x00007fb6c447389c in __lll_lock_wait () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #1 0x00007fb6c446f09b in _L_lock_1006 () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #2 0x00007fb6c446f01c in pthread_mutex_lock () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #3 0x00007fb6c5fc50d7 in my_pthread_fastmutex_lock ()
> #4 0x00007fb6c5a2ad48 in ?? ()
> #5 0x00007fb6c5a2d9ad in unlink_thd(THD*) ()
> #6 0x00007fb6c5a2da84 in one_thread_per_connection_end(__THD*,
> bool) ()
> #7 0x00007fb6c5b6bed3 in do_handle_one_connection(THD*) ()
> #8 0x00007fb6c5b6c181 in handle_one_connection ()
> #9 0x00007fb6c446ce9a in start_thread () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #10 0x00007fb6c3b9acbd in clone () from
> /lib/x86_64-linux-gnu/libc.so.__6
> #11 0x0000000000000000 in ?? ()
>
> Thread 2 (Thread 0x7f9977fff700 (LWP 31658)):
> #0 0x00007fb6c447389c in __lll_lock_wait () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #1 0x00007fb6c446f09b in _L_lock_1006 () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #2 0x00007fb6c446f01c in pthread_mutex_lock () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #3 0x00007fb6c5fc50d7 in my_pthread_fastmutex_lock ()
> #4 0x00007fb6c5a2ad48 in ?? ()
> #5 0x00007fb6c5a2d9ad in unlink_thd(THD*) ()
> #6 0x00007fb6c5a2da84 in one_thread_per_connection_end(__THD*,
> bool) ()
> #7 0x00007fb6c5b6bed3 in do_handle_one_connection(THD*) ()
> #8 0x00007fb6c5b6c181 in handle_one_connection ()
> #9 0x00007fb6c446ce9a in start_thread () from
> /lib/x86_64-linux-gnu/__libpthread.so.0
> #10 0x00007fb6c3b9acbd in clone () from
> /lib/x86_64-linux-gnu/libc.so.__6
> #11 0x0000000000000000 in ?? ()
>
> Thread 1 (Thread 0x7fb6c5717740 (LWP 27268)):
> #0 0x00007fb6c3b8f303 in poll () from
> /lib/x86_64-linux-gnu/libc.so.__6
> #1 0x00007fb6c5a30ebb in handle_connections_sockets() ()
> #2 0x00007fb6c5a34de0 in mysqld_main(int, char**) ()
> #3 0x00007fb6c3ac876d in __libc_start_main () from
> /lib/x86_64-linux-gnu/libc.so.__6
> #4 0x00007fb6c5a28c3d in _start ()
>
>
> I have had pt-stalk running once every 30 minutes (cron), but it
> hung
> until we killed the mysqld and not give all information during the
> freeze of db9:
>
> - log_error and output are empty
>
> - trigger:
> 2013_06_10_07_30_01 Not stalking; collect triggered immediately
> 2013_06_10_07_30_01 pt-stalk ran with --function=status
> --variable=Threads_running --threshold=25 --match= --cycles=0
> --interval=0 --iterations=1 --run-time=30 --sleep=0
> --dest=/var/lib/pt-stalk --prefix= --notify-by-email=
> --log=/var/log/pt-stalk.log --pid=/var/run/pt-stalk.pid
>
> variables:
> Variable_name Value
> aria_block_size 8192
> aria_checkpoint_interval __ 30
> aria_checkpoint_log_activity __ 1048576
> aria_force_start_after___recovery_failures 0
> aria_group_commit none
> aria_group_commit_interval __ 0
> aria_log_file_size __1073741824
> aria_log_purge_type __immediate
> aria_max_sort_file_size __ 9223372036853727232
> aria_page_checksum ON
> aria_pagecache_age_threshold __ 300
> aria_pagecache_buffer_size __ 134217728
> aria_pagecache_division_limit __ 100
> aria_recover NORMAL
> aria_repair_threads 1
> aria_sort_buffer_size __134217728
> aria_stats_method __nulls_unequal
> aria_sync_log_dir __NEWFILE
> aria_used_for_temp_tables __ ON
> auto_increment_increment __ 3
> auto_increment_offset 1
> autocommit ON
> automatic_sp_privileges __ ON
> back_log 50
> basedir /usr
> big_tables OFF
> binlog_annotate_row_events __ OFF
> binlog_cache_size 32768
> binlog_checksum NONE
> binlog_direct_non___transactional_updates __OFF
> binlog_format ROW
> binlog_optimize_thread___scheduling ON
> binlog_stmt_cache_size __32768
> bulk_insert_buffer_size __ 16777216
> character_set_client __latin1
> character_set_connection __ latin1
> character_set_database __latin1
> character_set_filesystem __ binary
> character_set_results __latin1
> character_set_server __latin1
> character_set_system __utf8
> character_sets_dir /__usr/share/mysql/charsets/
> collation_connection __latin1_swedish_ci
> collation_database __latin1_swedish_ci
> collation_server __latin1_swedish_ci
> completion_type NO___CHAIN
> concurrent_insert __ALWAYS
> connect_timeout 5
> datadir /var/lib/mysql/
> date_format %Y-%m-%d
> datetime_format %Y-%m-%__d %H:%i:%s
> deadlock_search_depth_long __ 15
> deadlock_search_depth_short __ 4
> deadlock_timeout_long __50000000
> deadlock_timeout_short __10000
> debug_no_thread_alarm __OFF
> default_storage_engine __InnoDB
> default_week_format 0
> delay_key_write ON
> delayed_insert_limit __100
> delayed_insert_timeout __300
> delayed_queue_size 1000
> div_precision_increment __ 4
> engine_condition_pushdown __ OFF
> event_scheduler OFF
> expensive_subquery_limit __ 100
> expire_logs_days 10
> extra_max_connections 1
> extra_port 0
> flush OFF
> flush_time 0
> foreign_key_checks ON
> ft_boolean_syntax + -><()~*:""&|
> ft_max_word_len 84
> ft_min_word_len 4
> ft_query_expansion_limit __ 20
> ft_stopword_file (__built-in)
> general_log OFF
> general_log_file db9.__log
> group_concat_max_len __1024
> have_compress YES
> have_crypt YES
> have_csv YES
> have_dynamic_loading __YES
> have_geometry YES
> have_innodb YES
> have_ndbcluster NO
> have_openssl DISABLED
> have_partitioning YES
> have_profiling YES
> have_query_cache YES
> have_rtree_keys YES
> have_ssl DISABLED
> have_symlink YES
> hostname db9
> ignore_builtin_innodb __OFF
> ignore_db_dirs
> init_connect
> init_file
> init_slave
> innodb_adaptive_flushing __ ON
> innodb_adaptive_flushing___method estimate
> innodb_adaptive_hash_index __ ON
> innodb_adaptive_hash_index___partitions 1
> innodb_additional_mem_pool___size 8388608
> innodb_autoextend_increment __ 1000
> innodb_autoinc_lock_mode __ 2
> innodb_blocking_buffer_pool___restore OFF
> innodb_buffer_pool_instances __ 1
> innodb_buffer_pool_populate __ OFF
> innodb_buffer_pool_restore_at___startup 0
> innodb_buffer_pool_shm___checksum ON
> innodb_buffer_pool_shm_key __ 0
> innodb_buffer_pool_size __ 107374182400
> innodb_change_buffering __ all
> innodb_changed_pages_limit __ 1000000
> innodb_checkpoint_age_target __ 0
> innodb_checksums ON
> innodb_commit_concurrency __ 0
> innodb_concurrency_tickets __ 500
> innodb_corrupt_table_action __ assert
> innodb_data_file_path __ibdata1:10M:autoextend
> innodb_data_home_dir
> innodb_dict_size_limit __0
> innodb_disallow_writes __OFF
> innodb_doublewrite ON
> innodb_doublewrite_file __
> innodb_fake_changes OFF
> innodb_fast_checksum __OFF
> innodb_fast_shutdown 1
> innodb_file_format __Antelope
> innodb_file_format_check __ ON
> innodb_file_format_max __Antelope
> innodb_file_per_table __ON
> innodb_flush_log_at_trx___commit 2
> innodb_flush_method O___DIRECT
> innodb_flush_neighbor_pages __ area
> innodb_force_load_corrupted __ OFF
> innodb_force_recovery 0
> innodb_ibuf_accel_rate __100
> innodb_ibuf_active_contract __ 1
> innodb_ibuf_max_size __53687074816
> innodb_import_table_from___xtrabackup 0
> innodb_io_capacity 2000
> innodb_kill_idle_transaction __ 0
> innodb_large_prefix OFF
> innodb_lazy_drop_table __0
> innodb_lock_wait_timeout __ 120
> innodb_locking_fake_changes __ ON
> innodb_locks_unsafe_for___binlog ON
> innodb_log_block_size __512
> innodb_log_buffer_size __33554432
> innodb_log_file_size __1258291200
> innodb_log_files_in_group __ 2
> innodb_log_group_home_dir __ ./
> innodb_max_bitmap_file_size __ 104857600
> innodb_max_dirty_pages_pct __ 75
> innodb_max_purge_lag 0
> innodb_merge_sort_block_size __ 1048576
> innodb_mirrored_log_groups __ 1
> innodb_old_blocks_pct __37
> innodb_old_blocks_time __0
> innodb_open_files 1000
> innodb_page_size 16384
> innodb_purge_batch_size __ 20
> innodb_purge_threads 1
> innodb_random_read_ahead __ OFF
> innodb_read_ahead __linear
> innodb_read_ahead_threshold __ 56
> innodb_read_io_threads __4
> innodb_recovery_stats __OFF
> innodb_recovery_update_relay___log OFF
> innodb_replication_delay __ 0
> innodb_rollback_on_timeout __ OFF
> innodb_rollback_segments __ 128
> innodb_show_locks_held __10
> innodb_show_verbose_locks __ 0
> innodb_spin_wait_delay __6
> innodb_stats_auto_update __ 1
> innodb_stats_method __nulls_equal
> innodb_stats_on_metadata __ ON
> innodb_stats_sample_pages __ 8
> innodb_stats_update_need_lock __ 1
> innodb_strict_mode OFF
> innodb_support_xa ON
> innodb_sync_spin_loops __30
> innodb_table_locks ON
> innodb_thread_concurrency __ 32
> innodb_thread_concurrency___timer_based OFF
> innodb_thread_sleep_delay __ 10000
> innodb_track_changed_pages __ OFF
> innodb_use_global_flush_log___at_trx_commit ON
> innodb_use_native_aio __ON
> innodb_use_sys_malloc __ON
> innodb_use_sys_stats_table __ OFF
> innodb_version 1.1.8-__29.3
> innodb_write_io_threads __ 4
> interactive_timeout __28800
> join_buffer_size 131072
> join_buffer_space_limit __ 2097152
> join_cache_level 2
> keep_files_on_create __OFF
> key_buffer_size __134217728
> key_cache_age_threshold __ 300
> key_cache_block_size __1024
> key_cache_division_limit __ 100
> key_cache_segments 0
> large_files_support ON
> large_page_size 0
> large_pages OFF
> lc_messages en_US
> lc_messages_dir /usr/__share/mysql
> lc_time_names en_US
> license GPL
> local_infile ON
> lock_wait_timeout __31536000
> locked_in_memory OFF
> log OFF
> log_bin ON
> log_bin_trust_function___creators OFF
> log_error
> log_output FILE
> log_queries_not_using_indexes __ OFF
> log_slave_updates ON
> log_slow_filter
> admin,filesort,filesort_on___disk,full_join,full_scan,__query_cache,query_cache_miss,__tmp_table,tmp_table_on_disk
>
> log_slow_queries OFF
> log_slow_rate_limit 1
> log_slow_verbosity __query_plan
> log_warnings 2
> long_query_time 10.__000000
> low_priority_updates __OFF
> lower_case_file_system __OFF
> lower_case_table_names __0
> master_verify_checksum __OFF
> max_allowed_packet __16777216
> max_binlog_cache_size __4294963200
> max_binlog_size __314572800
> max_binlog_stmt_cache_size __ 4294963200
> max_connect_errors 10
> max_connections 300
> max_delayed_threads 20
> max_error_count 64
> max_heap_table_size __33554432
> max_insert_delayed_threads __ 20
> max_join_size __18446744073709551615
> max_length_for_sort_data __ 1024
> max_long_data_size __16777216
> max_prepared_stmt_count __ 16382
> max_relay_log_size 0
> max_seeks_for_key __4294967295
> max_sort_length 1024
> max_sp_recursion_depth __0
> max_tmp_tables 32
> max_user_connections 0
> max_write_lock_count __4294967295
> metadata_locks_cache_size __ 1024
> min_examined_row_limit __0
> mrr_buffer_size 262144
> multi_range_count 256
> myisam_block_size 1024
> myisam_data_pointer_size __ 6
> myisam_max_sort_file_size __ 9223372036853727232
> myisam_mmap_size __18446744073709551615
> myisam_recover_options __BACKUP
> myisam_repair_threads 1
> myisam_sort_buffer_size __ 536870912
> myisam_stats_method __nulls_unequal
> myisam_use_mmap OFF
> net_buffer_length 16384
> net_read_timeout 30
> net_retry_count 10
> net_write_timeout 60
> old OFF
> old_alter_table OFF
> old_passwords OFF
> open_files_limit 4407
> optimizer_prune_level 1
> optimizer_search_depth __62
> optimizer_switch
> index_merge=on,index_merge___union=on,index_merge_sort___union=on,index_merge___intersection=on,index_merge___sort_intersection=off,engine___condition_pushdown=off,index___condition_pushdown=on,derived___merge=on,derived_with_keys=on,__firstmatch=on,loosescan=on,__materialization=on,in_to___exists=on,semijoin=on,partial___match_rowid_merge=on,partial___match_table_scan=on,subquery___cache=on,mrr=off,mrr_cost___based=off,mrr_sort_keys=off,__outer_join_with_cache=on,__semijoin_with_cache=on,join___cache_incremental=on,join___cache_hashed=on,join_cache___bka=on,optimize_join_buffer___size=off,table_elimination=on,__extended_keys=off
>
> performance_schema OFF
> performance_schema_events___waits_history_long_size __ 10000
> performance_schema_events___waits_history_size 10
> performance_schema_max_cond___classes 80
> performance_schema_max_cond___instances 1000
> performance_schema_max_file___classes 50
> performance_schema_max_file___handles 32768
> performance_schema_max_file___instances 10000
> performance_schema_max_mutex___classes 200
> performance_schema_max_mutex___instances 1000000
> performance_schema_max_rwlock___classes 30
> performance_schema_max_rwlock___instances 1000000
> performance_schema_max_table___handles 100000
> performance_schema_max_table___instances 50000
> performance_schema_max_thread___classes 50
> performance_schema_max_thread___instances 1000
> pid_file /var/run/__mysqld/mysqld.pid
> plugin_dir /usr/lib/__mysql/plugin/
> plugin_maturity unknown
> port 3306
> preload_buffer_size __32768
> profiling OFF
> profiling_history_size __15
> progress_report_time 56
> protocol_version 10
> query_alloc_block_size __8192
> query_cache_limit __131072
> query_cache_min_res_unit __ 4096
> query_cache_size 0
> query_cache_strip_comments __ OFF
> query_cache_type OFF
> query_cache_wlock_invalidate __ OFF
> query_prealloc_size __8192
> range_alloc_block_size __4096
> read_buffer_size __2097152
> read_only OFF
> read_rnd_buffer_size __1048576
> relay_log /var/log/__mysql/relay-bin
> relay_log_index /var/__log/mysql/relay-bin.index
> relay_log_info_file /__var/log/mysql/relay-bin.info
> <http://relay-bin.info>
> relay_log_purge ON
> relay_log_recovery OFF
> relay_log_space_limit 0
> replicate_annotate_row_events __ OFF
> replicate_do_db
> replicate_do_table
> replicate_events_marked_for___skip replicate
> replicate_ignore_db
> replicate_ignore_table
> replicate_wild_do_table __
> replicate_wild_ignore_table __
> report_host
> report_password
> report_port 3306
> report_user
> rowid_merge_buff_size __8388608
> rpl_recovery_rank 0
> secure_auth OFF
> secure_file_priv
> server_id 31
> skip_external_locking __ON
> skip_name_resolve OFF
> skip_networking OFF
> skip_show_database OFF
> slave_compressed_protocol __ ON
> slave_exec_mode STRICT
> slave_load_tmpdir /tmp
> slave_max_allowed_packet __ 1073741824
> slave_net_timeout 3600
> slave_skip_errors OFF
> slave_sql_verify_checksum __ ON
> slave_transaction_retries __ 10
> slave_type_conversions
> slow_launch_time 2
> slow_query_log OFF
> slow_query_log_file /__var/log/mysql/mariadb-slow.log
> socket /var/run/mysqld/__mysqld.sock
> sort_buffer_size __4194304
> sql_auto_is_null OFF
> sql_big_selects ON
> sql_big_tables OFF
> sql_buffer_result OFF
> sql_log_bin ON
> sql_log_off OFF
> sql_low_priority_updates __ OFF
> sql_max_join_size __18446744073709551615
> sql_mode
> sql_notes ON
> sql_quote_show_create __ON
> sql_safe_updates OFF
> sql_select_limit __18446744073709551615
> sql_slave_skip_counter __0
> sql_warnings OFF
> ssl_ca
> ssl_capath
> ssl_cert
> ssl_cipher
> ssl_key
> storage_engine InnoDB
> stored_program_cache __256
> sync_binlog 0
> sync_frm ON
> sync_master_info 0
> sync_relay_log 0
> sync_relay_log_info 0
> system_time_zone CEST
> table_definition_cache __512
> table_open_cache 2048
> thread_cache_size 128
> thread_concurrency 10
> thread_handling one-__thread-per-connection
> thread_pool_idle_timeout __ 60
> thread_pool_max_threads __ 500
> thread_pool_oversubscribe __ 3
> thread_pool_size 12
> thread_pool_stall_limit __ 500
> thread_stack 294912
> time_format %H:%i:%s
> time_zone SYSTEM
> timed_mutexes OFF
> tmp_table_size 33554432
> tmpdir /tmp
> transaction_alloc_block_size __ 8192
> transaction_prealloc_size __ 4096
> tx_isolation __REPEATABLE-READ
> unique_checks ON
> updatable_views_with_limit __ YES
> userstat OFF
> version 5.5.29-MariaDB-__mariadb1~quantal-log
> version_comment mariadb__.org <http://mariadb.org> binary
> distribution, wsrep_23.7.3.rXXXX
> version_compile_machine __ x86_64
> version_compile_os __debian-linux-gnu
> wait_timeout 600
> wsrep_OSU_method TOI
> wsrep_auto_increment_control __ ON
> wsrep_causal_reads OFF
> wsrep_certify_nonPK ON
> wsrep_cluster_address __gcomm://192.168.10.30
> <http://192.168.10.30>,192.168.__10.31
> wsrep_cluster_name __kursdaten
> wsrep_convert_LOCK_to_trx __ OFF
> wsrep_data_home_dir /__var/lib/mysql/
> wsrep_dbug_option
> wsrep_debug OFF
> wsrep_drupal_282555___workaround OFF
> wsrep_forced_binlog_format __ NONE
> wsrep_log_conflicts OFF
> wsrep_max_ws_rows __131072
> wsrep_max_ws_size __1073741824
> wsrep_mysql_replication___bundle 0
> wsrep_node_address 192.__168.10.31
> wsrep_node_incoming_address __ AUTO
> wsrep_node_name db9
> wsrep_notify_cmd
> wsrep_on ON
> wsrep_provider /usr/__lib/galera/libgalera_smm.so
> wsrep_provider_options __base_host = 192.168.10.31;
> base_port = 4567;
> cert.log_conflicts = no; evs.causal_keepalive_period = PT1S;
> evs.debug_log_mask = 0x1; evs.inactive_check_period = PT0.5S;
> evs.inactive_timeout = PT15S; evs.info_log_mask = 0;
> evs.install_timeout
> = PT15S; evs.join_retrans_period = PT1S; evs.keepalive_period =
> PT1S;
> evs.max_install_timeouts = 1; evs.send_window = 4;
> evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S;
> evs.use_aggregate = true; evs.user_send_window = 2; evs.version
> = 0;
> evs.view_forget_timeout = PT5M; gcache.dir = /var/lib/mysql/;
> gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name
> <http://gcache.name> =
> /var/lib/mysql//galera.cache; gcache.page_size = 128M;
> gcache.size =
> 10G; gcs.fc_debug = 0; gcs.fc_factor = 1; gcs.fc_limit = 16;
> gcs.fc_master_slave = NO; gcs.max_packet_size = 64500;
> gcs.max_throttle
> = 0.25; gcs.recv_q_hard_limit = 9223372036854775807;
> gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = NO;
> gmcast.listen_addr =
> tcp://0.0.0.0:4567 <http://0.0.0.0:4567>; gmcast.mcast_addr = ;
> gmcast.mcast_ttl = 1;
> gmcast.peer_timeout = PT3S; gmcast.time_wait = PT5S;
> gmcast.version = 0;
> ist.recv_addr = 192.168.10.31; pc.checksum = true;
> pc.ignore_quorum =
> false; pc.ignore_sb = false; pc.linger = PT20S; pc.npvo = true;
> pc.version = 0; pc.weight = 1; protonet.backend = asio;
> protonet.version
> = 0; replicator.causal_read_timeout = PT30S;
> replicator.commit_order = 3
> wsrep_recover OFF
> wsrep_replicate_myisam __OFF
> wsrep_retry_autocommit __1
> wsrep_slave_threads 40
> wsrep_sst_auth ********
> wsrep_sst_donor
> wsrep_sst_donor_rejects___queries OFF
> wsrep_sst_method __xtrabackup
> wsrep_sst_receive_address __ 192.168.10.31
> wsrep_start_position __41d7259d-7200-11e2-0800-__943dd0e2f659:180873652
>
>
> pmap:
> 27268: /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql
> --plugin-dir=/usr/lib/mysql/__plugin --user=mysql
> --pid-file=/var/run/mysqld/__mysqld.pid
> --socket=/var/run/mysqld/__mysqld.sock --port=3306
> --wsrep_start_position=__41d7259d-7200-11e2-0800-__943dd0e2f659:180873652
> <mailto:codership-team%2Bunsu...@googlegroups.com>.

Aleksey Sanin

unread,
Jun 13, 2013, 1:06:23 PM6/13/13
to codersh...@googlegroups.com
Any plans for a new Galera release? :)

(Just trying to see if it is worth it to build a custom package
and deal with custom version number which is a pain)


Aleksey
signature.asc
Reply all
Reply to author
Forward
0 new messages