wsrep_sst_mysqldump Table 'mysql.proc' doesn't exist

[Aleksey Chudov]

Hi,

Currently I'm testing MariaDB Galera Cluster 10.0.16. Some times SST failed with the following error in Donor server log file:

150220  2:33:26 [Note] WSREP: Running: 'wsrep_sst_mysqldump --user 'wsrep_sst' --password 'secret' --host '10.172.4.12' --port '3306' --local-port '3306' --socket '/var/lib/mysql/mysql.sock' --defaults-file '/etc/my.cnf' --gtid 'ad19ad16-b88a-11e4-96ec-0fbf0bf58b11:5''
ERROR 1146 (42S02) at line 14: Table 'mysql.proc' doesn't exist
mysqldump: Got errno 32 on write
150220  2:33:26 [ERROR] WSREP: Process completed with error: wsrep_sst_mysqldump --user 'wsrep_sst' --password 'secret' --host '10.172.4.12' --port '3306' --local-port '3306' --socket '/var/lib/mysql/mysql.sock' --defaults-file '/etc/my.cnf' --gtid 'ad19ad16-b88a-11e4-96ec-0fbf0bf58b11:5': 1 (Operation not permitted)
150220  2:33:26 [ERROR] WSREP: Try 1/3: 'wsrep_sst_mysqldump --user 'wsrep_sst' --password 'secret' --host '10.172.4.12' --port '3306' --local-port '3306' --socket '/var/lib/mysql/mysql.sock' --defaults-file '/etc/my.cnf' --gtid 'ad19ad16-b88a-11e4-96ec-0fbf0bf58b11:5'' failed: 1 (Operation not permitted)


I clean installed the database. But just to be sure I removed /var/lib/mysql and executed mysql_install_db && chown mysql.mysql /var/lib/mysql. Problem persists.

So I tried manually execute wsrep scrip and it fails each second time leaving empty mysql database on second server

[root@srv1 ~]# wsrep_sst_mysqldump --user 'wsrep_sst' --password 'secret' --host '10.172.4.12' --port '3306' --local-port '3306' --socket '/var/lib/mysql/mysql.sock' --defaults-file '/etc/my.cnf' --gtid 'ad19ad16-b88a-11e4-96ec-0fbf0bf58b11:5'
ERROR 1146 (42S02) at line 14: Table 'mysql.proc' doesn't exist
mysqldump: Got errno 32 on write
[root@srv1 ~]# wsrep_sst_mysqldump --user 'wsrep_sst' --password 'secret' --host '10.172.4.12' --port '3306' --local-port '3306' --socket '/var/lib/mysql/mysql.sock' --defaults-file '/etc/my.cnf' --gtid 'ad19ad16-b88a-11e4-96ec-0fbf0bf58b11:5'
[root@srv1 ~]# wsrep_sst_mysqldump --user 'wsrep_sst' --password 'secret' --host '10.172.4.12' --port '3306' --local-port '3306' --socket '/var/lib/mysql/mysql.sock' --defaults-file '/etc/my.cnf' --gtid 'ad19ad16-b88a-11e4-96ec-0fbf0bf58b11:5'
ERROR 1146 (42S02) at line 14: Table 'mysql.proc' doesn't exist
mysqldump: Got errno 32 on write

Then I'm tried executing just the mysqldump part of wsrep_sst_mysqldump and it completes without problems.

After some experiments I removed --add-drop-database from mysqldump command line

--- wsrep_sst_mysqldump 2015-02-20 15:30:23.703660502 +0300
+++ -   2015-02-20 15:45:35.352467037 +0300
@@ -129,7 +129,7 @@
 
 # NOTE: we don't use --routines here because we're dumping mysql.proc table
 MYSQLDUMP="mysqldump $AUTH -S$WSREP_SST_OPT_SOCKET \
---add-drop-database --add-drop-table --skip-add-locks --create-options \
+--add-drop-table --skip-add-locks --create-options \
 --disable-keys --extended-insert --skip-lock-tables --quick --set-charset \
 --skip-comments --flush-privileges --all-databases $OPT_GALERA_SST_MODE"
 
Without --add-drop-database option problem goes away.

I'm stuck at this point. Seems like there is some bug in wsrep_sst_mysqldump while removing / restoring mysql database on second server.

Any ideas?

Regards,
Aleksey

[Will Fong]

Hi Aleksey,

> On Feb 20, 2015, at 8:51 PM, Aleksey Chudov <aleksey...@gmail.com> wrote:
>
> Currently I'm testing MariaDB Galera Cluster 10.0.16. Some times SST failed with the following error in Donor server log file:

I haven't been able to reproduce this under 10.0.16. However, this seems to be a known issue in earlier versions and have been fixed: https://mariadb.atlassian.net/browse/MDEV-6098

Can you verify that you are running 10.0.16 on all the nodes?


Thanks,
-will


--
Will Fong, Support Engineer
MariaDB Corporation

[Aleksey Chudov]

Hi Will,

Can you verify that you are running 10.0.16 on all the nodes?

Sure!

# rpm -qa | grep -Ei 'maria|galera'
MariaDB-common-10.0.16-1.el7.centos.x86_64
MariaDB-Galera-server-10.0.16-1.el7.centos.x86_64
galera-25.3.5-1.rhel6.x86_64
MariaDB-shared-10.0.16-1.el7.centos.x86_64
MariaDB-client-10.0.16-1.el7.centos.x86_64

Packages from official repo http://yum.mariadb.org/10.0/centos7-amd64/rpms/

And to be sure
# md5sum usr/bin/wsrep_sst_mysqldump   
bf86e3198195e92f71c0adae0abda308  /usr/bin/wsrep_sst_mysqldump

[Will Fong]

Hi Aleksey,

> On Feb 24, 2015, at 2:01 AM, Aleksey Chudov <aleksey...@gmail.com> wrote:
>
> # rpm -qa | grep -Ei 'maria|galera'
> MariaDB-common-10.0.16-1.el7.centos.x86_64
> MariaDB-Galera-server-10.0.16-1.el7.centos.x86_64
> galera-25.3.5-1.rhel6.x86_64
> MariaDB-shared-10.0.16-1.el7.centos.x86_64
> MariaDB-client-10.0.16-1.el7.centos.x86_64
>
> Packages from official repo http://yum.mariadb.org/10.0/centos7-amd64/rpms/
>
> And to be sure
> # md5sum usr/bin/wsrep_sst_mysqldump
> bf86e3198195e92f71c0adae0abda308 /usr/bin/wsrep_sst_mysqldump
>

Unfortunately I haven't been able to reproduce this issue. Can you provide the exact steps and output that can reproduce this issue?

Have you also tried another SST method, like rsync?

[Aleksey Chudov]

Hi Will,

Unfortunately I haven't been able to reproduce this issue. Can you provide the exact steps and output that can reproduce this issue?

Current cluster already in production state. I will setup next cluster bit latter and provide the exact steps to reproduce.

Have you also tried another SST method, like rsync

Yes, different methods work as expected. But I need mysqldump.
 

Currently I solved problem with the following quick hack:

--- wsrep_sst_mysqldump.orig
+++ wsrep_sst_mysqldump
@@ -128,10 +128,22 @@
 fi

 # NOTE: we don't use --routines here because we're dumping mysql.proc table

-MYSQLDUMP="mysqldump $AUTH -S$WSREP_SST_OPT_SOCKET \
+MYSQLDUMP_MYQSL="mysqldump $AUTH -S$WSREP_SST_OPT_SOCKET \
+--add-drop-table --skip-add-locks --create-options \
+--disable-keys --extended-insert --skip-lock-tables --quick --set-charset \
+--skip-comments --flush-privileges --databases mysql $OPT_GALERA_SST_MODE"
+
+DATABASES=$(mysql $AUTH -S$WSREP_SST_OPT_SOCKET --skip-column-names -e"\
+SHOW DATABASES WHERE \`Database\` NOT IN ('mysql', 'information_schema', \
+'performance_schema')")
+
+if [ -n "$DATABASES" ]
+then
+    MYSQLDUMP_ALL="mysqldump $AUTH -S$WSREP_SST_OPT_SOCKET \
 --add-drop-database --add-drop-table --skip-add-locks --create-options \

 --disable-keys --extended-insert --skip-lock-tables --quick --set-charset \

---skip-comments --flush-privileges --all-databases $OPT_GALERA_SST_MODE"
+--skip-comments --flush-privileges --databases $DATABASES $OPT_GALERA_SST_MODE"
+fi

 # need to disable logging when loading the dump
 # reason is that dump contains ALTER TABLE for log tables, and
@@ -149,7 +161,8 @@
 if [ $WSREP_SST_OPT_BYPASS -eq 0 ]
 then
     (echo $STOP_WSREP && echo $RESET_MASTER) | $MYSQL || true
-    (echo $STOP_WSREP && $MYSQLDUMP && echo $CSV_TABLES_FIX \
+    (echo $STOP_WSREP && $MYSQLDUMP_MYQSL && ${MYSQLDUMP_ALL:-true} \
+    && echo $CSV_TABLES_FIX \
     && echo $RESTORE_GENERAL_LOG && echo $RESTORE_SLOW_QUERY_LOG \
     && echo $SET_START_POSITION \
     || echo "SST failed to complete;") | $MYSQL

Thanks,
Aleksey

[Aleksey Chudov]

Hi,

Currently I'm configuring my second MariaDB Galera Cluster 10.0. The problem is reproduced exactly as on the first cluster. Below are the exact steps to reproduce in my environment.

1. Setup three clean CentOS 7.0.1406 x86_64 servers
Actually our configuration management system performs some initial configuration for all servers after a clean installation. But they certainly are identical.

2. Open tcp ports 3306, 4567, 4568 on firewall between the servers (on every server)

3. Enable SELinux permissive mode (on every server)

setenforce 0

4. Add MariaDB Yum repo according to https://mariadb.com/kb/en/mariadb/yum/ (on every server)

5. Install MariaDB Galera Cluster 10.0

yum update
yum install MariaDB-Galera-server

6. Start MariaDB (on every server)

systemctl start mysql.service
less /var/lib/mysql/$HOSTNAME.err

7. Secure installation (on every server)

# mysql_secure_installation
Set root password? [Y/n] y
Remove anonymous users? [Y/n] y
Disallow root login remotely? [Y/n] y
Remove test database and access to it? [Y/n] y
Reload privilege tables now? [Y/n] y
 
# mysql -p
mysql> DELETE FROM mysql.proxies_priv;
mysql> GRANT ALL PRIVILEGES ON *.* TO wsrep_sst@'%' IDENTIFIED BY 'MySecretPassword' REQUIRE SSL;
 
# rm .mysql_history

8. Stop MariaDB (on every server)
systemctl stop mysql.service

9. Generate and install SSL certificates for MariaDB (on every server)
I'm using local CA.

cp ~/MYCA/certs/MYCA.crt /etc/pki/tls/certs/MYCA.crt
cp ~/MYCA/certs/$HOSTNAME.crt /etc/pki/tls/certs/$HOSTNAME.crt
cp ~/MYCA/private/$HOSTNAME.key /etc/pki/tls/private/$HOSTNAME.key

groupadd --system ssl_keys
usermod -a -G ssl_keys mysql

chmod 0644 /etc/pki/tls/certs/CA.crt /etc/pki/tls/certs/$HOSTNAME.crt
chown root:ssl_keys /etc/pki/tls/private/
chmod 0710 /etc/pki/tls/private/
chown root:ssl_keys /etc/pki/tls/private/$HOSTNAME.key
chmod 0640 /etc/pki/tls/private/$HOSTNAME.key

9. Generate SSL certificate for Galera (on one server)

openssl req -new -x509 -days 36500 -nodes -keyout /etc/my.cnf.d/galera.key -out /etc/my.cnf.d/galera.crt
chown root:mysql /etc/my.cnf.d/galera.*
chmod 0640 /etc/my.cnf.d/galera.*

10. Copy Galera SSL certificate to others servers (do not forget about files permissions)

11. Edit MariaDB client configuration (on every server)

/etc/my.cnf.d/client.cnf

[client]
ssl_ca=/etc/pki/tls/certs/MYCA.crt
ssl_cert=/etc/pki/tls/certs/$HOSTNAME.crt
ssl_key=/etc/pki/tls/private/$HOSTNAME.key

12. Edit MariaDB server configuration (on every server, correct local ip)

/etc/my.cnf.d/server.cnf

[galera]
ssl_ca=/etc/pki/tls/certs/MYCA.crt
ssl_cert=/etc/pki/tls/certs/$HOSTNAME.crt
ssl_key=/etc/pki/tls/private/$HOSTNAME.key
 
bind-address=10.10.10.1
binlog_format=ROW
character_set_server=utf8
default_storage_engine=InnoDB
innodb_autoinc_lock_mode=2
innodb_flush_log_at_trx_commit=0
skip_name_resolve=ON
 
wsrep_cluster_address=gcomm://10.10.10.1,10.10.10.2,10.10.10.3
wsrep_cluster_name=galera
wsrep_debug=ON
wsrep_provider=/usr/lib64/galera/libgalera_smm.so
wsrep_provider_options=gmcast.listen_addr=tcp://10.10.10.1:4567; socket.ssl_cert=/etc/my.cnf.d/galera.crt; socket.ssl_key=/etc/my.cnf.d/galera.key
wsrep_sst_auth=wsrep_sst:MySecretPassword
wsrep_sst_method=mysqldump

chown root:mysql /etc/my.cnf.d/server.cnf
chmod 0640 /etc/my.cnf.d/server.cnf

13. Start first cluster node using init script

/etc/init.d/mysql start --wsrep-new-cluster
less /var/lib/mysql/$HOSTNAME.err

14. Start other cluster nodes

systemctl start mysql.service
less /var/lib/mysql/$HOSTNAME.err

15. Restart first node using systemd

/etc/init.d/mysql stop
systemctl start mysql.service
less /var/lib/mysql/$HOSTNAME.err



After that point try stop one node and edit uuid in /var/lib/mysql/grastate.dat to initiate SST. First start always failed. But stop and second start always successful.


JOINER node log

150316 12:53:53 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150316 12:53:53 mysqld_safe WSREP: Running position recovery with --log_error='/var/lib/mysql/wsrep_recovery.OeAOC3' --pid-file='/var/lib/mysql/srv3-recover.pid'
150316 12:53:56 mysqld_safe WSREP: Recovered position 28a69a2a-c977-11e4-93a8-d2be0a63c45a:0
150316 12:53:56 [Note] WSREP: wsrep_start_position var submitted: '28a69a2a-c977-11e4-93a8-d2be0a63c45a:0'
150316 12:53:56 [Note] InnoDB: Using mutexes to ref count buffer pool pages
150316 12:53:56 [Note] InnoDB: The InnoDB memory heap is disabled
150316 12:53:56 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
150316 12:53:56 [Note] InnoDB: Memory barrier is not used
150316 12:53:56 [Note] InnoDB: Compressed tables use zlib 1.2.7
150316 12:53:56 [Note] InnoDB: Using Linux native AIO
150316 12:53:56 [Note] InnoDB: Not using CPU crc32 instructions
150316 12:53:56 [Note] InnoDB: Initializing buffer pool, size = 128.0M
150316 12:53:56 [Note] InnoDB: Completed initialization of buffer pool
150316 12:53:56 [Note] InnoDB: Highest supported file format is Barracuda.
150316 12:53:56 [Note] InnoDB: 128 rollback segment(s) are active.
150316 12:53:56 [Note] InnoDB: Waiting for purge to start
150316 12:53:56 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.22-71.0 started; log sequence number 1663689
150316 12:53:56 [Note] Plugin 'FEEDBACK' is disabled.
150316 12:53:56 [Note] WSREP: Initial TC log open: dummy
150316 12:53:56 [Note] Server socket created on IP: '10.10.10.3'.
150316 12:53:56 [Note] Event Scheduler: Loaded 0 events
150316 12:53:56 [Note] WSREP: Setting wsrep_ready to 0
150316 12:53:56 [Note] WSREP: Read WSREPXid from InnoDB:  28a69a2a-c977-11e4-93a8-d2be0a63c45a:0
150316 12:53:56 [Note] WSREP: Initial position: 28a69a2a-c977-11e4-93a8-d2be0a63c45a:0
150316 12:53:56 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib64/galera/libgalera_smm.so'
150316 12:53:56 [Note] WSREP: wsrep_load(): Galera 25.3.5(rXXXX) by Codership Oy <in...@codership.com> loaded successfully.
150316 12:53:56 [Note] WSREP: CRC-32C: using "slicing-by-8" algorithm.
150316 12:53:56 [Note] WSREP: Found saved state: 18a69a2a-c977-11e4-93a8-d2be0a63c45a:0
150316 12:53:56 [Note] WSREP: Passing config to GCS: base_host = 10.10.10.3; base_port = 4567; cert.log_conflicts = no; debug = no; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 1; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.listen_addr = tcp://10.10.10.3:4567; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false; pc.npvo = false; pc.version = 0; pc.wait_prim = true; pc.wait
150316 12:53:56 [Note] WSREP: Service thread queue flushed.
150316 12:53:56 [Note] WSREP: Assign initial position for certification: 0, protocol version: -1
150316 12:53:56 [Note] WSREP: Start replication
150316 12:53:56 [Note] WSREP: Setting initial position to 18a69a2a-c977-11e4-93a8-d2be0a63c45a:0
150316 12:53:56 [Note] WSREP: protonet asio version 0
150316 12:53:56 [Note] WSREP: Using CRC-32C (optimized) for message checksums.
150316 12:53:56 [Note] WSREP: initializing ssl context
150316 12:53:56 [Note] WSREP: backend: asio
150316 12:53:56 [Note] WSREP: GMCast version 0
150316 12:53:56 [Note] WSREP: (5c91a5ff-cbc2-11e4-85fe-bf325a2a05b5, 'ssl://10.10.10.3:4567') listening at ssl://10.10.10.3:4567
150316 12:53:56 [Note] WSREP: (5c91a5ff-cbc2-11e4-85fe-bf325a2a05b5, 'ssl://10.10.10.3:4567') multicast: , ttl: 1
150316 12:53:56 [Note] WSREP: EVS version 0
150316 12:53:56 [Note] WSREP: PC version 0
150316 12:53:56 [Note] WSREP: gcomm: connecting to group 'galera', peer '10.10.10.1:,10.10.10.2:,10.10.10.3:'
150316 12:53:56 [Note] WSREP: SSL handshake successful, remote endpoint ssl://10.10.10.1:4567 local endpoint ssl://10.10.10.3:56052 cipher: AES128-SHA compression:
150316 12:53:56 [Note] WSREP: SSL handshake successful, remote endpoint ssl://10.10.10.2:4567 local endpoint ssl://10.10.10.3:52179 cipher: AES128-SHA compression:
150316 12:53:56 [Note] WSREP: declaring 461a091c-c997-11e4-9ea9-dfcbe8fbeb4f stable
150316 12:53:56 [Note] WSREP: declaring 749ebced-ca49-11e4-af4d-272d418ac720 stable
150316 12:53:56 [Note] WSREP: Node 461a091c-c997-11e4-9ea9-dfcbe8fbeb4f state prim
150316 12:53:56 [Note] WSREP: view(view_id(PRIM,461a091c-c997-11e4-9ea9-dfcbe8fbeb4f,15) memb {
        461a091c-c997-11e4-9ea9-dfcbe8fbeb4f,0
        5c91a5ff-cbc2-11e4-85fe-bf325a2a05b5,0
        749ebced-ca49-11e4-af4d-272d418ac720,0
} joined {
} left {
} partitioned {
})
150316 12:53:57 [Note] WSREP: gcomm: connected
150316 12:53:57 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
150316 12:53:57 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
150316 12:53:57 [Note] WSREP: Opened channel 'galera'
150316 12:53:57 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 3
150316 12:53:57 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
150316 12:53:57 [Note] WSREP: STATE EXCHANGE: sent state msg: 5ca89fdb-cbc2-11e4-a9ac-f7d42d652bbe
150316 12:53:57 [Note] WSREP: STATE EXCHANGE: got state msg: 5ca89fdb-cbc2-11e4-a9ac-f7d42d652bbe from 0 (srv2)
150316 12:53:57 [Note] WSREP: STATE EXCHANGE: got state msg: 5ca89fdb-cbc2-11e4-a9ac-f7d42d652bbe from 2 (srv1)
150316 12:53:57 [Note] WSREP: STATE EXCHANGE: got state msg: 5ca89fdb-cbc2-11e4-a9ac-f7d42d652bbe from 1 (srv3)
150316 12:53:57 [Note] WSREP: Quorum results:
        version    = 3,
        component  = PRIMARY,
        conf_id    = 14,
        members    = 2/3 (joined/total),
        act_id     = 0,
        last_appl. = -1,
        protocols  = 0/5/3 (gcs/repl/appl),
        group UUID = 28a69a2a-c977-11e4-93a8-d2be0a63c45a
150316 12:53:57 [Note] WSREP: Flow-control interval: [28, 28]
150316 12:53:57 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 0)
150316 12:53:57 [Note] WSREP: State transfer required:
        Group state: 28a69a2a-c977-11e4-93a8-d2be0a63c45a:0
        Local state: 18a69a2a-c977-11e4-93a8-d2be0a63c45a:0
150316 12:53:57 [Note] WSREP: New cluster view: global state: 28a69a2a-c977-11e4-93a8-d2be0a63c45a:0, view# 15: Primary, number of nodes: 3, my index: 1, protocol version 3
150316 12:53:57 [Warning] WSREP: Gap in state sequence. Need state transfer.
150316 12:53:57 [Note] WSREP: Setting wsrep_ready to 0
150316 12:53:57 [Note] WSREP: [debug]: closing client connections for PRIM
150316 12:53:57 [Note] /usr/sbin/mysqld: ready for connections.
Version: '10.0.16-MariaDB-wsrep'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MariaDB Server, wsrep_25.10.r4144
150316 12:53:59 [Note] WSREP: waiting for client connections to close: 2
150316 12:53:59 [Note] WSREP: Prepared SST request: mysqldump|10.10.10.3:3306
150316 12:53:59 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
150316 12:53:59 [Note] WSREP: REPL Protocols: 5 (3, 1)
150316 12:53:59 [Note] WSREP: Service thread queue flushed.
150316 12:53:59 [Note] WSREP: Assign initial position for certification: 0, protocol version: 3
150316 12:53:59 [Note] WSREP: Service thread queue flushed.
150316 12:53:59 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (18a69a2a-c977-11e4-93a8-d2be0a63c45a) does not match group state UUID (28a69a2a-c977-11e4-93a8-d2be0a63c45a): 1 (Operation not permitted)
         at galera/src/replicator_str.cpp:prepare_for_IST():447. IST will be unavailable.
150316 12:53:59 [Note] WSREP: Member 1.0 (srv3) requested state transfer from '*any*'. Selected 0.0 (srv2)(SYNCED) as donor.
150316 12:53:59 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 0)
150316 12:53:59 [Note] WSREP: Requesting state transfer: success, donor: 0
2015-03-16 12:54:00 7ffdd436f700 InnoDB: Error: Table "mysql"."innodb_table_stats" not found.
150316 12:54:01 [Note] WSREP: Set WSREPXid for InnoDB:  28a69a2a-c977-11e4-93a8-d2be0a63c45a:0
150316 12:54:01 [Note] WSREP: wsrep_start_position var submitted: '28a69a2a-c977-11e4-93a8-d2be0a63c45a:0'
150316 12:54:01 [Note] WSREP: SST received: 28a69a2a-c977-11e4-93a8-d2be0a63c45a:0
150316 12:54:01 [Note] WSREP: 1.0 (srv3): State transfer from 0.0 (srv2) complete.
150316 12:54:01 [Note] WSREP: Shifting JOINER -> JOINED (TO: 0)
150316 12:54:01 [Note] WSREP: 0.0 (srv2): State transfer to 1.0 (srv3) complete.
150316 12:54:01 [Note] WSREP: Member 1.0 (srv3) synced with group.
150316 12:54:01 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 0)
150316 12:54:01 [Note] WSREP: Synchronized with group, ready for connections
150316 12:54:01 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
150316 12:54:01 [Note] WSREP: Member 0.0 (srv2) synced with group.


DONOR node log


150316 12:53:56 [Note] WSREP: SSL handshake successful, remote endpoint ssl://10.10.10.3:52179 local endpoint ssl://10.10.10.2:4567 cipher: AES128-SHA compression:
150316 12:53:56 [Note] WSREP: (461a091c-c997-11e4-9ea9-dfcbe8fbeb4f, 'ssl://10.10.10.2:4567') turning message relay requesting on, nonlive peers: ssl://10.10.10.3:4567
150316 12:53:56 [Note] WSREP: (461a091c-c997-11e4-9ea9-dfcbe8fbeb4f, 'ssl://10.10.10.2:4567') turning message relay requesting off
150316 12:53:56 [Note] WSREP: declaring 5c91a5ff-cbc2-11e4-85fe-bf325a2a05b5 stable
150316 12:53:56 [Note] WSREP: declaring 749ebced-ca49-11e4-af4d-272d418ac720 stable
150316 12:53:56 [Note] WSREP: Node 461a091c-c997-11e4-9ea9-dfcbe8fbeb4f state prim
150316 12:53:56 [Note] WSREP: view(view_id(PRIM,461a091c-c997-11e4-9ea9-dfcbe8fbeb4f,15) memb {
        461a091c-c997-11e4-9ea9-dfcbe8fbeb4f,0
        5c91a5ff-cbc2-11e4-85fe-bf325a2a05b5,0
        749ebced-ca49-11e4-af4d-272d418ac720,0
} joined {
} left {
} partitioned {
})
150316 12:53:56 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 3
150316 12:53:56 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 5ca89fdb-cbc2-11e4-a9ac-f7d42d652bbe
150316 12:53:56 [Note] WSREP: STATE EXCHANGE: sent state msg: 5ca89fdb-cbc2-11e4-a9ac-f7d42d652bbe
150316 12:53:56 [Note] WSREP: STATE EXCHANGE: got state msg: 5ca89fdb-cbc2-11e4-a9ac-f7d42d652bbe from 0 (srv2)
150316 12:53:56 [Note] WSREP: STATE EXCHANGE: got state msg: 5ca89fdb-cbc2-11e4-a9ac-f7d42d652bbe from 2 (srv1)
150316 12:53:57 [Note] WSREP: STATE EXCHANGE: got state msg: 5ca89fdb-cbc2-11e4-a9ac-f7d42d652bbe from 1 (srv3)
150316 12:53:57 [Note] WSREP: Quorum results:
        version    = 3,
        component  = PRIMARY,
        conf_id    = 14,
        members    = 2/3 (joined/total),
        act_id     = 0,
        last_appl. = 0,
        protocols  = 0/5/3 (gcs/repl/appl),
        group UUID = 28a69a2a-c977-11e4-93a8-d2be0a63c45a
150316 12:53:57 [Note] WSREP: Flow-control interval: [28, 28]
150316 12:53:57 [Note] WSREP: New cluster view: global state: 28a69a2a-c977-11e4-93a8-d2be0a63c45a:0, view# 15: Primary, number of nodes: 3, my index: 0, protocol version 3
150316 12:53:57 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
150316 12:53:57 [Note] WSREP: REPL Protocols: 5 (3, 1)
150316 12:53:57 [Note] WSREP: Service thread queue flushed.
150316 12:53:57 [Note] WSREP: Assign initial position for certification: 0, protocol version: 3
150316 12:53:57 [Note] WSREP: Service thread queue flushed.
150316 12:53:59 [Note] WSREP: Member 1.0 (srv3) requested state transfer from '*any*'. Selected 0.0 (srv2)(SYNCED) as donor.
150316 12:53:59 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 0)
150316 12:53:59 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
150316 12:53:59 [Note] WSREP: Running: 'wsrep_sst_mysqldump --user 'wsrep_sst' --password 'MySecretPassword' --host '10.10.10.3' --port '3306' --local-port '3306' --socket '/var/lib/mysql/mysql.sock' --defaults-file '/etc/my.cnf' --gtid '28a69a2a-c977-11e4-93a8-d2be0a63c45a:0''


ERROR 1146 (42S02) at line 14: Table 'mysql.proc' doesn't exist
mysqldump: Got errno 32 on write


150316 12:53:59 [ERROR] WSREP: Process completed with error: wsrep_sst_mysqldump --user 'wsrep_sst' --password 'MySecretPassword' --host '10.10.10.3' --port '3306' --local-port '3306' --socket '/var/lib/mysql/mysql.sock' --defaults-file '/etc/my.cnf' --gtid '28a69a2a-c977-11e4-93a8-d2be0a63c45a:0': 1 (Operation not permitted)
150316 12:53:59 [ERROR] WSREP: Try 1/3: 'wsrep_sst_mysqldump --user 'wsrep_sst' --password 'MySecretPassword' --host '10.10.10.3' --port '3306' --local-port '3306' --socket '/var/lib/mysql/mysql.sock' --defaults-file '/etc/my.cnf' --gtid '28a69a2a-c977-11e4-93a8-d2be0a63c45a:0'' failed: 1 (Operation not permitted)
150316 12:54:01 [Note] WSREP: 1.0 (srv3): State transfer from 0.0 (srv2) complete.
150316 12:54:01 [Note] WSREP: SST script successfully completed.
150316 12:54:01 [Note] WSREP: 0.0 (srv2): State transfer to 1.0 (srv3) complete.
150316 12:54:01 [Note] WSREP: Shifting DONOR/DESYNCED -> JOINED (TO: 0)
150316 12:54:01 [Note] WSREP: Member 1.0 (srv3) synced with group.
150316 12:54:01 [Note] WSREP: Member 0.0 (srv2) synced with group.
150316 12:54:01 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 0)
150316 12:54:01 [Note] WSREP: Synchronized with group, ready for connections
150316 12:54:01 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.


Any ideas?

Regards,
Aleksey