new node performs SST but then does not connect

[Mati]

Hello,

I'm trying to add a new node to my cluster. When I try to add it, it performs the SST, but then refuses to connect. I couldn't understand from the logs what the issue.

I'd appreciate any help.

Technical info: this new node is on GCE with local SSD (which I've mounted with --bind to the /var/lib/mysql). That is the difference from the other nodes. I couldn't see how this might prevent the node from joining.

Here are the relevant logs:

Dec 25 09:53:09 <server-name> mysqld[10792]: 2017-12-25  9:53:09 140183211063040 [Note] WSREP: 4.2 (tuxger-maindb4): State transfer to 1.3 (<server-name>) complete.

Dec 25 09:53:09 <server-name> mysqld[10792]: 2017-12-25  9:53:09 140183211063040 [Note] WSREP: Member 4.2 (tuxger-maindb4) synced with group.

Dec 25 09:53:12 <server-name> -wsrep-sst-joiner: Preparing the backup at /var/lib/mysql//.sst

Dec 25 09:53:12 <server-name> -wsrep-sst-joiner: Evaluating innobackupex --no-version-check  --apply-log $rebuildcmd ${DATA} 2>&1  | logger -p daemon.err -t -innobackupex-apply

Dec 25 09:53:12 <server-name> -innobackupex-apply: 171225 09:53:12 innobackupex: Starting the apply-log operation

Dec 25 09:53:12 <server-name> -innobackupex-apply:

Dec 25 09:53:12 <server-name> -innobackupex-apply: IMPORTANT: Please check that the apply-log run completes successfully.

Dec 25 09:53:12 <server-name> -innobackupex-apply:            At the end of a successful apply-log run innobackupex

Dec 25 09:53:12 <server-name> -innobackupex-apply:            prints "completed OK!".

Dec 25 09:53:12 <server-name> -innobackupex-apply:

Dec 25 09:53:12 <server-name> -innobackupex-apply: innobackupex version 2.4.9 based on MySQL server 5.7.13 Linux (x86_64) (revision id: a467167cdd4)

Dec 25 09:53:12 <server-name> -innobackupex-apply: xtrabackup: cd to /var/lib/mysql/.sst/

Dec 25 09:53:12 <server-name> -innobackupex-apply: xtrabackup: This target seems to be not prepared yet.

Dec 25 09:53:12 <server-name> -innobackupex-apply: InnoDB: Number of pools: 1

Dec 25 09:53:12 <server-name> -innobackupex-apply: xtrabackup: xtrabackup_logfile detected: size=14155776, start_lsn=(443808104561)

Dec 25 09:53:12 <server-name> -innobackupex-apply: xtrabackup: using the following InnoDB configuration for recovery:

Dec 25 09:53:12 <server-name> -innobackupex-apply: xtrabackup:   innodb_data_home_dir = .

Dec 25 09:53:12 <server-name> -innobackupex-apply: xtrabackup:   innodb_data_file_path = ibdata1:12M:autoextend

Dec 25 09:53:12 <server-name> -innobackupex-apply: xtrabackup:   innodb_log_group_home_dir = .

Dec 25 09:53:12 <server-name> -innobackupex-apply: xtrabackup:   innodb_log_files_in_group = 1

Dec 25 09:53:12 <server-name> -innobackupex-apply: xtrabackup:   innodb_log_file_size = 14155776

Dec 25 09:53:12 <server-name> -innobackupex-apply: xtrabackup: using the following InnoDB configuration for recovery:

Dec 25 09:53:12 <server-name> -innobackupex-apply: xtrabackup:   innodb_data_home_dir = .

Dec 25 09:53:12 <server-name> -innobackupex-apply: xtrabackup:   innodb_data_file_path = ibdata1:12M:autoextend

Dec 25 09:53:12 <server-name> -innobackupex-apply: xtrabackup:   innodb_log_group_home_dir = .

Dec 25 09:53:12 <server-name> -innobackupex-apply: xtrabackup:   innodb_log_files_in_group = 1

Dec 25 09:53:12 <server-name> -innobackupex-apply: xtrabackup:   innodb_log_file_size = 14155776

Dec 25 09:53:12 <server-name> -innobackupex-apply: xtrabackup: Starting InnoDB instance for recovery.

Dec 25 09:53:12 <server-name> -innobackupex-apply: xtrabackup: Using 104857600 bytes for buffer pool (set by --use-memory parameter)

Dec 25 09:53:12 <server-name> -innobackupex-apply: InnoDB: PUNCH HOLE support available

Dec 25 09:53:12 <server-name> -innobackupex-apply: InnoDB: Mutexes and rw_locks use GCC atomic builtins

Dec 25 09:53:12 <server-name> -innobackupex-apply: InnoDB: Uses event mutexes

Dec 25 09:53:12 <server-name> -innobackupex-apply: InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier

Dec 25 09:53:12 <server-name> -innobackupex-apply: InnoDB: Compressed tables use zlib 1.2.8

Dec 25 09:53:12 <server-name> -innobackupex-apply: InnoDB: Number of pools: 1

Dec 25 09:53:12 <server-name> -innobackupex-apply: InnoDB: Using CPU crc32 instructions

Dec 25 09:53:12 <server-name> -innobackupex-apply: InnoDB: Initializing buffer pool, total size = 100M, instances = 1, chunk size = 100M

Dec 25 09:53:12 <server-name> -innobackupex-apply: InnoDB: Completed initialization of buffer pool

Dec 25 09:53:12 <server-name> -innobackupex-apply: InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().

Dec 25 09:53:12 <server-name> -innobackupex-apply: InnoDB: Highest supported file format is Barracuda.

Dec 25 09:53:12 <server-name> -innobackupex-apply: InnoDB: Log scan progressed past the checkpoint lsn 443808104561

Dec 25 09:53:12 <server-name> -innobackupex-apply: InnoDB: Doing recovery: scanned up to log sequence number 443813281792 (41%)

Dec 25 09:53:12 <server-name> -innobackupex-apply: InnoDB: Doing recovery: scanned up to log sequence number 443818524672 (82%)

Dec 25 09:53:12 <server-name> -innobackupex-apply: InnoDB: Doing recovery: scanned up to log sequence number 443820668953 (99%)

Dec 25 09:53:12 <server-name> -innobackupex-apply: InnoDB: Database was not shutdown normally!

Dec 25 09:53:12 <server-name> -innobackupex-apply: InnoDB: Starting crash recovery.

Dec 25 09:53:12 <server-name> -innobackupex-apply: InnoDB: Starting an apply batch of log records to the database...

Dec 25 09:53:13 <server-name> -innobackupex-apply: InnoDB: Progress in percent: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99

Dec 25 09:53:13 <server-name> -innobackupex-apply: InnoDB: Apply batch completed

Dec 25 09:53:13 <server-name> -innobackupex-apply: InnoDB: xtrabackup: Last MySQL binlog file position 15601352, file name /var/log/mysql/mariadb-bin.000525

Dec 25 09:53:13 <server-name> -innobackupex-apply: InnoDB: Creating shared tablespace for temporary tables

Dec 25 09:53:13 <server-name> -innobackupex-apply: InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...

Dec 25 09:53:13 <server-name> -innobackupex-apply: InnoDB: File './ibtmp1' size is now 12 MB.

Dec 25 09:53:13 <server-name> -innobackupex-apply: InnoDB: 96 redo rollback segment(s) found. 1 redo rollback segment(s) are active.

Dec 25 09:53:13 <server-name> -innobackupex-apply: InnoDB: 32 non-redo rollback segment(s) are active.

Dec 25 09:53:13 <server-name> -innobackupex-apply: InnoDB: 5.7.13 started; log sequence number 443820668953

Dec 25 09:53:13 <server-name> -innobackupex-apply: InnoDB: xtrabackup: Last MySQL binlog file position 15601352, file name /var/log/mysql/mariadb-bin.000525

Dec 25 09:53:13 <server-name> -innobackupex-apply:

Dec 25 09:53:13 <server-name> -innobackupex-apply: xtrabackup: starting shutdown with innodb_fast_shutdown = 1

Dec 25 09:53:13 <server-name> -innobackupex-apply: InnoDB: FTS optimize thread exiting.

Dec 25 09:53:13 <server-name> -innobackupex-apply: InnoDB: Starting shutdown...

Dec 25 09:53:15 <server-name> -innobackupex-apply: InnoDB: Shutdown completed; log sequence number 443820673173

Dec 25 09:53:15 <server-name> -innobackupex-apply: InnoDB: Number of pools: 1

Dec 25 09:53:15 <server-name> -innobackupex-apply: xtrabackup: using the following InnoDB configuration for recovery:

Dec 25 09:53:15 <server-name> -innobackupex-apply: xtrabackup:   innodb_data_home_dir = .

Dec 25 09:53:15 <server-name> -innobackupex-apply: xtrabackup:   innodb_data_file_path = ibdata1:12M:autoextend

Dec 25 09:53:15 <server-name> -innobackupex-apply: xtrabackup:   innodb_log_group_home_dir = .

Dec 25 09:53:15 <server-name> -innobackupex-apply: xtrabackup:   innodb_log_files_in_group = 2

Dec 25 09:53:15 <server-name> -innobackupex-apply: xtrabackup:   innodb_log_file_size = 104857600

Dec 25 09:53:15 <server-name> -innobackupex-apply: InnoDB: PUNCH HOLE support available

Dec 25 09:53:15 <server-name> -innobackupex-apply: InnoDB: Mutexes and rw_locks use GCC atomic builtins

Dec 25 09:53:15 <server-name> -innobackupex-apply: InnoDB: Uses event mutexes

Dec 25 09:53:15 <server-name> -innobackupex-apply: InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier

Dec 25 09:53:15 <server-name> -innobackupex-apply: InnoDB: Compressed tables use zlib 1.2.8

Dec 25 09:53:15 <server-name> -innobackupex-apply: InnoDB: Number of pools: 1

Dec 25 09:53:15 <server-name> -innobackupex-apply: InnoDB: Using CPU crc32 instructions

Dec 25 09:53:15 <server-name> -innobackupex-apply: InnoDB: Initializing buffer pool, total size = 100M, instances = 1, chunk size = 100M

Dec 25 09:53:15 <server-name> -innobackupex-apply: InnoDB: Completed initialization of buffer pool

Dec 25 09:53:15 <server-name> -innobackupex-apply: InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().

Dec 25 09:53:15 <server-name> -innobackupex-apply: InnoDB: Setting log file ./ib_logfile101 size to 100 MB

Dec 25 09:53:15 <server-name> -innobackupex-apply: InnoDB: Progress in MB:

Dec 25 09:53:15 <server-name> -innobackupex-apply:  100

Dec 25 09:53:15 <server-name> -innobackupex-apply: InnoDB: Setting log file ./ib_logfile1 size to 100 MB

Dec 25 09:53:15 <server-name> -innobackupex-apply: InnoDB: Progress in MB:

Dec 25 09:53:15 <server-name> -innobackupex-apply:  100

Dec 25 09:53:15 <server-name> -innobackupex-apply: InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0

Dec 25 09:53:15 <server-name> -innobackupex-apply: InnoDB: New log files created, LSN=443820673173

Dec 25 09:53:15 <server-name> -innobackupex-apply: InnoDB: Highest supported file format is Barracuda.

Dec 25 09:53:15 <server-name> -innobackupex-apply: InnoDB: Log scan progressed past the checkpoint lsn 443820673548

Dec 25 09:53:15 <server-name> -innobackupex-apply: InnoDB: Doing recovery: scanned up to log sequence number 443820673557 (0%)

Dec 25 09:53:15 <server-name> -innobackupex-apply: InnoDB: Doing recovery: scanned up to log sequence number 443820673557 (0%)

Dec 25 09:53:15 <server-name> -innobackupex-apply: InnoDB: Database was not shutdown normally!

Dec 25 09:53:15 <server-name> -innobackupex-apply: InnoDB: Starting crash recovery.

Dec 25 09:53:15 <server-name> -innobackupex-apply: InnoDB: xtrabackup: Last MySQL binlog file position 15601352, file name /var/log/mysql/mariadb-bin.000525

Dec 25 09:53:16 <server-name> -innobackupex-apply: InnoDB: Removed temporary tablespace data file: "ibtmp1"

Dec 25 09:53:16 <server-name> -innobackupex-apply: InnoDB: Creating shared tablespace for temporary tables

Dec 25 09:53:16 <server-name> -innobackupex-apply: InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...

Dec 25 09:53:16 <server-name> -innobackupex-apply: InnoDB: File './ibtmp1' size is now 12 MB.

Dec 25 09:53:16 <server-name> -innobackupex-apply: InnoDB: 96 redo rollback segment(s) found. 1 redo rollback segment(s) are active.

Dec 25 09:53:16 <server-name> -innobackupex-apply: InnoDB: 32 non-redo rollback segment(s) are active.

Dec 25 09:53:16 <server-name> -innobackupex-apply: InnoDB: 5.7.13 started; log sequence number 443820673557

Dec 25 09:53:16 <server-name> -innobackupex-apply: xtrabackup: starting shutdown with innodb_fast_shutdown = 1

Dec 25 09:53:16 <server-name> -innobackupex-apply: InnoDB: FTS optimize thread exiting.

Dec 25 09:53:16 <server-name> -innobackupex-apply: InnoDB: Starting shutdown...

Dec 25 09:53:17 <server-name> -innobackupex-apply: InnoDB: Shutdown completed; log sequence number 443820673576

Dec 25 09:53:17 <server-name> -innobackupex-apply: 171225 09:53:17 completed OK!

Dec 25 09:53:17 <server-name> -wsrep-sst-joiner: rm: cannot remove '/var/lib/mysql//innobackup.prepare.log': No such file or directory

Dec 25 09:53:17 <server-name> -wsrep-sst-joiner: rm: cannot remove '/var/lib/mysql//innobackup.move.log': No such file or directory

Dec 25 09:53:17 <server-name> -wsrep-sst-joiner: Moving the backup to /var/lib/mysql/

Dec 25 09:53:17 <server-name> -wsrep-sst-joiner: Evaluating innobackupex  --no-version-check  --datadir=/var/lib/mysql/ --move-back --force-non-empty-directories ${DATA} 2>&1 | logger -p daemon.err -t -innobackupex-move

Dec 25 09:53:17 <server-name> -innobackupex-move: 171225 09:53:17 innobackupex: Starting the move-back operation

Dec 25 09:53:17 <server-name> -innobackupex-move:

Dec 25 09:53:17 <server-name> -innobackupex-move: IMPORTANT: Please check that the move-back run completes successfully.

Dec 25 09:53:17 <server-name> -innobackupex-move:            At the end of a successful move-back run innobackupex

Dec 25 09:53:17 <server-name> -innobackupex-move:            prints "completed OK!".

Dec 25 09:53:17 <server-name> -innobackupex-move:

Dec 25 09:53:17 <server-name> -innobackupex-move: innobackupex version 2.4.9 based on MySQL server 5.7.13 Linux (x86_64) (revision id: a467167cdd4)

Dec 25 09:53:17 <server-name> -innobackupex-move: 171225 09:53:17 [01] Moving ib_logfile0 to /var/lib/mysql//ib_logfile0

...

...

...

Dec 25 09:53:17 <server-name> -innobackupex-move: 171225 09:53:17 [01]        ...done

Dec 25 09:53:17 <server-name> -innobackupex-move: 171225 09:53:17 completed OK!

Dec 25 09:53:17 <server-name> -wsrep-sst-joiner: Move successful, removing /var/lib/mysql//.sst

Dec 25 09:53:17 <server-name> -wsrep-sst-joiner: Galera co-ords from recovery: 9ba7937a-8bd6-11e5-9a1b-73a68953642e:32573523

Dec 25 09:53:17 <server-name> -wsrep-sst-joiner: Total time on joiner: 0 seconds

Dec 25 09:53:17 <server-name> -wsrep-sst-joiner: Removing the sst_in_progress file

Dec 25 09:53:17 <server-name> mysqld[10792]: 2017-12-25  9:53:17 140184005916864 [Note] WSREP: SST complete, seqno: 32573523

Dec 25 09:53:17 <server-name> mysqld[10792]: 2017-12-25  9:53:17 140184005916864 [Warning] InnoDB: Using innodb_file_format is deprecated and the parameter may be removed in future releases. See http://dev.mysql.com/doc/refman/5.7/en/innodb-file-format.html

Dec 25 09:53:17 <server-name> mysqld[10792]: 2017-12-25  9:53:17 140184005916864 [Warning] InnoDB: Using innodb_locks_unsafe_for_binlog is DEPRECATED. This option may be removed in future releases. Please use READ COMMITTED transaction isolation level instead; Please refer to http://dev.mysql.com/doc/refman/5.7/en/set-transaction.html

Dec 25 09:53:17 <server-name> mysqld[10792]: 2017-12-25  9:53:17 140184005916864 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins

Dec 25 09:53:17 <server-name> mysqld[10792]: 2017-12-25  9:53:17 140184005916864 [Note] InnoDB: Uses event mutexes

Dec 25 09:53:17 <server-name> mysqld[10792]: 2017-12-25  9:53:17 140184005916864 [Note] InnoDB: Compressed tables use zlib 1.2.8

Dec 25 09:53:17 <server-name> mysqld[10792]: 2017-12-25  9:53:17 140184005916864 [Note] InnoDB: Using Linux native AIO

Dec 25 09:53:17 <server-name> mysqld[10792]: 2017-12-25  9:53:17 140184005916864 [Note] InnoDB: Number of pools: 1

Dec 25 09:53:17 <server-name> mysqld[10792]: 2017-12-25  9:53:17 140184005916864 [Note] InnoDB: Using SSE2 crc32 instructions

Dec 25 09:53:17 <server-name> mysqld[10792]: 2017-12-25  9:53:17 140184005916864 [Note] InnoDB: Initializing buffer pool, total size = 10G, instances = 8, chunk size = 128M

Dec 25 09:53:18 <server-name> mysqld[10792]: 2017-12-25  9:53:18 140184005916864 [Note] InnoDB: Completed initialization of buffer pool

Dec 25 09:53:18 <server-name> mysqld[10792]: 2017-12-25  9:53:18 140170995013376 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().

Dec 25 09:53:18 <server-name> mysqld[10792]: 2017-12-25  9:53:18 140184005916864 [Note] InnoDB: Highest supported file format is Barracuda.

Dec 25 09:53:18 <server-name> mysqld[10792]: 2017-12-25  9:53:18 140184005916864 [Note] InnoDB: 128 out of 128 rollback segments are active.

Dec 25 09:53:18 <server-name> mysqld[10792]: 2017-12-25  9:53:18 140184005916864 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"

Dec 25 09:53:18 <server-name> mysqld[10792]: 2017-12-25  9:53:18 140184005916864 [Note] InnoDB: Creating shared tablespace for temporary tables

Dec 25 09:53:18 <server-name> mysqld[10792]: 2017-12-25  9:53:18 140184005916864 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...

Dec 25 09:53:18 <server-name> mysqld[10792]: 2017-12-25  9:53:18 140184005916864 [Note] InnoDB: File './ibtmp1' size is now 12 MB.

Dec 25 09:53:18 <server-name> mysqld[10792]: 2017-12-25  9:53:18 140184005916864 [Note] InnoDB: 5.7.20 started; log sequence number 443820673576

Dec 25 09:53:18 <server-name> mysqld[10792]: 2017-12-25  9:53:18 140171173402368 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool

Dec 25 09:53:18 <server-name> mysqld[10792]: 2017-12-25  9:53:18 140184005916864 [Note] Plugin 'FEEDBACK' is disabled.

Dec 25 09:53:18 <server-name> mysqld[10792]: 2017-12-25  9:53:18 140184005916864 [Note] Recovering after a crash using /var/log/mysql/mariadb-bin

Dec 25 09:53:18 <server-name> mysqld[10792]: 2017-12-25  9:53:18 140184005916864 [Note] Starting crash recovery...

Dec 25 09:53:18 <server-name> mysqld[10792]: 2017-12-25  9:53:18 140184005916864 [Note] Crash recovery finished.

Dec 25 09:53:18 <server-name> mysqld[10792]: 2017-12-25  9:53:18 140184005916864 [Note] Server socket created on IP: '0.0.0.0'.

Dec 25 09:53:18 <server-name> mysqld[10792]: 2017-12-25  9:53:18 140183840872192 [Note] /usr/sbin/mysqld (unknown): Normal shutdown

Dec 25 09:53:18 <server-name> mysqld[10792]: 2017-12-25  9:53:18 140183840872192 [Note] WSREP: Stop replication

Dec 25 09:53:18 <server-name> mysqld[10792]: 2017-12-25  9:53:18 140183840872192 [Note] WSREP: Closing send monitor...

Dec 25 09:53:18 <server-name> mysqld[10792]: 2017-12-25  9:53:18 140183840872192 [Note] WSREP: Closed send monitor.

Dec 25 09:53:18 <server-name> mysqld[10792]: 2017-12-25  9:53:18 140183840872192 [Note] WSREP: gcomm: terminating thread

Dec 25 09:53:18 <server-name> mysqld[10792]: 2017-12-25  9:53:18 140183840872192 [Note] WSREP: gcomm: joining thread

Dec 25 09:53:18 <server-name> mysqld[10792]: 2017-12-25  9:53:18 140183840872192 [Note] WSREP: gcomm: closing backend

Dec 25 09:53:18 <server-name> mysqld[10792]: 2017-12-25  9:53:18 140184005916864 [Warning] 'user' entry 'root@tux-maindb1' ignored in --skip-name-resolve mode.

Dec 25 09:53:18 <server-name> mysqld[10792]: 2017-12-25  9:53:18 140184005916864 [Warning] 'proxies_priv' entry '@% root@tux-maindb1' ignored in --skip-name-resolve mode.

Dec 25 09:53:18 <server-name> mysqld[10792]: 2017-12-25  9:53:18 140184005916864 [Note] WSREP: Signalling provider to continue.

Dec 25 09:53:18 <server-name> mysqld[10792]: 2017-12-25  9:53:18 140184005916864 [Note] WSREP: SST received: 9ba7937a-8bd6-11e5-9a1b-73a68953642e:32573523

Dec 25 09:53:18 <server-name> mysqld[10792]: 2017-12-25  9:53:18 140184005916864 [Note] Reading of all Master_info entries succeded

Dec 25 09:53:18 <server-name> mysqld[10792]: 2017-12-25  9:53:18 140184005916864 [Note] Added new Master_info '' to hash table

Dec 25 09:53:18 <server-name> mysqld[10792]: 2017-12-25  9:53:18 140184005916864 [Note] /usr/sbin/mysqld: ready for connections.

Dec 25 09:53:18 <server-name> mysqld[10792]: Version: '10.2.11-MariaDB-10.2.11+maria~xenial-log'  socket: '/var/run/mysqld/mysqld.sock'  port: 7171  mariadb.org binary distribution

Dec 25 09:53:18 <server-name> mysqld[10792]: 2017-12-25  9:53:18 140183840872192 [Note] WSREP: view(view_id(NON_PRIM,16098e05,159) memb {

Dec 25 09:53:18 <server-name> mysqld[10792]: #011b3bb7d2a,3

Dec 25 09:53:18 <server-name> mysqld[10792]: } joined {

Dec 25 09:53:18 <server-name> mysqld[10792]: } left {

Dec 25 09:53:18 <server-name> mysqld[10792]: } partitioned {

Dec 25 09:53:18 <server-name> mysqld[10792]: #01116098e05,2

Dec 25 09:53:18 <server-name> mysqld[10792]: #011d05aae8d,1

Dec 25 09:53:18 <server-name> mysqld[10792]: #011ebaf4418,1

Dec 25 09:53:18 <server-name> mysqld[10792]: #011f515f27a,2

Dec 25 09:53:18 <server-name> mysqld[10792]: })

Dec 25 09:53:18 <server-name> mysqld[10792]: 2017-12-25  9:53:18 140183840872192 [Note] WSREP: view((empty))

Dec 25 09:53:18 <server-name> mysqld[10792]: 2017-12-25  9:53:18 140183840872192 [Note] WSREP: gcomm: closed

Dec 25 09:53:18 <server-name> mysqld[10792]: 2017-12-25  9:53:18 140184005072640 [Warning] WSREP: 0x556842beacd8 down context(s) not set

Dec 25 09:53:18 <server-name> mysqld[10792]: 2017-12-25  9:53:18 140184005072640 [Warning] WSREP: Sending JOIN failed: -107 (Transport endpoint is not connected). Will retry in new primary component.

Dec 25 09:53:18 <server-name> mysqld[10792]: 2017-12-25  9:53:18 140183211063040 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1

Dec 25 09:53:18 <server-name> mysqld[10792]: 2017-12-25  9:53:18 140183211063040 [Note] WSREP: Flow-control interval: [16, 16]

Dec 25 09:53:18 <server-name> mysqld[10792]: 2017-12-25  9:53:18 140183211063040 [Note] WSREP: Trying to continue unpaused monitor

Dec 25 09:53:18 <server-name> mysqld[10792]: 2017-12-25  9:53:18 140183211063040 [Note] WSREP: Received NON-PRIMARY.

Dec 25 09:53:18 <server-name> mysqld[10792]: 2017-12-25  9:53:18 140183211063040 [Note] WSREP: Shifting JOINER -> OPEN (TO: 32573574)

Dec 25 09:53:18 <server-name> mysqld[10792]: 2017-12-25  9:53:18 140183211063040 [Note] WSREP: Received self-leave message.

Dec 25 09:53:18 <server-name> mysqld[10792]: 2017-12-25  9:53:18 140183211063040 [Note] WSREP: Flow-control interval: [0, 0]

Dec 25 09:53:18 <server-name> mysqld[10792]: 2017-12-25  9:53:18 140183211063040 [Note] WSREP: Trying to continue unpaused monitor

Dec 25 09:53:18 <server-name> mysqld[10792]: 2017-12-25  9:53:18 140183211063040 [Note] WSREP: Received SELF-LEAVE. Closing connection.

Dec 25 09:53:18 <server-name> mysqld[10792]: 2017-12-25  9:53:18 140183211063040 [Note] WSREP: Shifting OPEN -> CLOSED (TO: 32573574)

Dec 25 09:53:18 <server-name> mysqld[10792]: 2017-12-25  9:53:18 140183211063040 [Note] WSREP: RECV thread exiting 0: Success

Dec 25 09:53:18 <server-name> mysqld[10792]: 2017-12-25  9:53:18 140183840872192 [Note] WSREP: recv_thread() joined.

Dec 25 09:53:18 <server-name> mysqld[10792]: 2017-12-25  9:53:18 140183840872192 [Note] WSREP: Closing replication queue.

Dec 25 09:53:18 <server-name> mysqld[10792]: 2017-12-25  9:53:18 140183840872192 [Note] WSREP: Closing slave action queue.

Dec 25 09:53:18 <server-name> mysqld[10792]: 2017-12-25  9:53:18 140183864448768 [Warning] WSREP: Failed to report last committed 32572332, -77 (File descriptor in bad state)

Dec 25 09:53:18 <server-name> mysqld[10792]: 2017-12-25  9:53:18 140183864448768 [Warning] WSREP: Failed to report last committed 32572459, -77 (File descriptor in bad state)

Dec 25 09:53:18 <server-name> mysqld[10792]: 2017-12-25  9:53:18 140183864448768 [Warning] WSREP: Failed to report last committed 32572588, -77 (File descriptor in bad state)

Dec 25 09:53:18 <server-name> mysqld[10792]: 2017-12-25  9:53:18 140183864448768 [Warning] WSREP: Failed to report last committed 32572715, -77 (File descriptor in bad state)

Dec 25 09:53:18 <server-name> mysqld[10792]: 2017-12-25  9:53:18 140183864448768 [Warning] WSREP: Failed to report last committed 32572844, -77 (File descriptor in bad state)

Dec 25 09:53:18 <server-name> mysqld[10792]: 2017-12-25  9:53:18 140183864448768 [Warning] WSREP: Failed to report last committed 32572972, -77 (File descriptor in bad state)

Dec 25 09:53:18 <server-name> mysqld[10792]: 2017-12-25  9:53:18 140183864448768 [Warning] WSREP: Failed to report last committed 32573100, -77 (File descriptor in bad state)

Dec 25 09:53:18 <server-name> mysqld[10792]: 2017-12-25  9:53:18 140183864448768 [Warning] WSREP: Failed to report last committed 32573228, -77 (File descriptor in bad state)

Dec 25 09:53:18 <server-name> mysqld[10792]: 2017-12-25  9:53:18 140183864448768 [Warning] WSREP: Failed to report last committed 32573356, -77 (File descriptor in bad state)

Dec 25 09:53:18 <server-name> mysqld[10792]: 2017-12-25  9:53:18 140183864448768 [Warning] WSREP: Failed to report last committed 32573484, -77 (File descriptor in bad state)

Dec 25 09:53:18 <server-name> mysqld[10792]: 2017-12-25  9:53:18 140171173402368 [Note] InnoDB: Buffer pool(s) load completed at 171225  9:53:18

Dec 25 09:53:18 <server-name> mysqld[10792]: 2017-12-25  9:53:18 140182767191808 [Note] WSREP: New cluster view: global state: 9ba7937a-8bd6-11e5-9a1b-73a68953642e:32573574, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version 3

Dec 25 09:53:18 <server-name> mysqld[10792]: 2017-12-25  9:53:18 140182767191808 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

Dec 25 09:53:18 <server-name> mysqld[10792]: 2017-12-25  9:53:18 140182767191808 [Note] WSREP: New cluster view: global state: 9ba7937a-8bd6-11e5-9a1b-73a68953642e:32573574, view# -1: non-Primary, number of nodes: 0, my index: -1, protocol version 3

Dec 25 09:53:18 <server-name> mysqld[10792]: 2017-12-25  9:53:18 140182767191808 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

Dec 25 09:53:18 <server-name> mysqld[10792]: 2017-12-25  9:53:18 140182767191808 [Note] WSREP: applier thread exiting (code:0)

Dec 25 09:53:18 <server-name> mysqld[10792]: 2017-12-25  9:53:18 140182767798016 [Note] WSREP: applier thread exiting (code:6)

Dec 25 09:53:18 <server-name> mysqld[10792]: 2017-12-25  9:53:18 140171293857536 [Note] WSREP: applier thread exiting (code:6)

Dec 25 09:53:18 <server-name> mysqld[10792]: 2017-12-25  9:53:18 140182768101120 [Note] WSREP: applier thread exiting (code:6)

Dec 25 09:53:18 <server-name> mysqld[10792]: 2017-12-25  9:53:18 140171294160640 [Note] WSREP: applier thread exiting (code:6)

Dec 25 09:53:18 <server-name> mysqld[10792]: 2017-12-25  9:53:18 140171295069952 [Note] WSREP: applier thread exiting (code:6)

Dec 25 09:53:18 <server-name> mysqld[10792]: 2017-12-25  9:53:18 140183169791744 [Note] WSREP: applier thread exiting (code:6)

Dec 25 09:53:18 <server-name> mysqld[10792]: 2017-12-25  9:53:18 140184005072640 [Note] WSREP: applier thread exiting (code:6)

Dec 25 09:53:18 <server-name> mysqld[10792]: 2017-12-25  9:53:18 140171294766848 [Note] WSREP: applier thread exiting (code:6)

Dec 25 09:53:18 <server-name> mysqld[10792]: 2017-12-25  9:53:18 140171293554432 [Note] WSREP: applier thread exiting (code:6)

Dec 25 09:53:18 <server-name> mysqld[10792]: 2017-12-25  9:53:18 140183840569088 [Note] WSREP: applier thread exiting (code:6)

Dec 25 09:53:18 <server-name> mysqld[10792]: 2017-12-25  9:53:18 140182766888704 [Note] WSREP: applier thread exiting (code:6)

Dec 25 09:53:18 <server-name> mysqld[10792]: 2017-12-25  9:53:18 140171294463744 [Note] WSREP: applier thread exiting (code:6)

Dec 25 09:53:18 <server-name> mysqld[10792]: 2017-12-25  9:53:18 140183169488640 [Note] WSREP: applier thread exiting (code:6)

Dec 25 09:53:18 <server-name> mysqld[10792]: 2017-12-25  9:53:18 140182767494912 [Note] WSREP: applier thread exiting (code:6)

Dec 25 09:53:18 <server-name> mysqld[10792]: 2017-12-25  9:53:18 140171293251328 [Note] WSREP: applier thread exiting (code:6)

Dec 25 09:53:20 <server-name> mysqld[10792]: 2017-12-25  9:53:20 140184005375744 [Note] WSREP: rollbacker thread exiting

Dec 25 09:53:20 <server-name> mysqld[10792]: 2017-12-25  9:53:20 140171215365888 [Note] InnoDB: FTS optimize thread exiting.

Dec 25 09:53:20 <server-name> mysqld[10792]: 2017-12-25  9:53:20 140183840872192 [Note] Event Scheduler: Purging the queue. 0 events

Dec 25 09:53:20 <server-name> mysqld[10792]: 2017-12-25  9:53:20 140183840872192 [Note] WSREP: dtor state: CLOSED

Dec 25 09:53:20 <server-name> mysqld[10792]: 2017-12-25  9:53:20 140183840872192 [Note] WSREP: mon: entered 51 oooe fraction 0 oool fraction 0

Dec 25 09:53:20 <server-name> mysqld[10792]: 2017-12-25  9:53:20 140183840872192 [Note] WSREP: mon: entered 51 oooe fraction 0.784314 oool fraction 0.0980392

Dec 25 09:53:20 <server-name> mysqld[10792]: 2017-12-25  9:53:20 140183840872192 [Note] WSREP: mon: entered 1373 oooe fraction 0 oool fraction 0.000728332

Dec 25 09:53:20 <server-name> mysqld[10792]: 2017-12-25  9:53:20 140183840872192 [Note] WSREP: cert index usage at exit 0

Dec 25 09:53:20 <server-name> mysqld[10792]: 2017-12-25  9:53:20 140183840872192 [Note] WSREP: cert trx map usage at exit 51

Dec 25 09:53:20 <server-name> mysqld[10792]: 2017-12-25  9:53:20 140183840872192 [Note] WSREP: deps set usage at exit 0

Dec 25 09:53:20 <server-name> mysqld[10792]: 2017-12-25  9:53:20 140183840872192 [Note] WSREP: avg deps dist 158.04

Dec 25 09:53:20 <server-name> mysqld[10792]: 2017-12-25  9:53:20 140183840872192 [Note] WSREP: avg cert interval 0.504018

Dec 25 09:53:20 <server-name> mysqld[10792]: 2017-12-25  9:53:20 140183840872192 [Note] WSREP: cert index size 80

Dec 25 09:53:20 <server-name> mysqld[10792]: 2017-12-25  9:53:20 140183864448768 [Note] WSREP: Service thread queue flushed.

Dec 25 09:53:20 <server-name> mysqld[10792]: 2017-12-25  9:53:20 140183840872192 [Note] WSREP: wsdb trx map usage 0 conn query map usage 0

Dec 25 09:53:20 <server-name> mysqld[10792]: 2017-12-25  9:53:20 140183840872192 [Note] WSREP: MemPool(LocalTrxHandle): hit ratio: 0, misses: 0, in use: 0, in pool: 0

Dec 25 09:53:20 <server-name> mysqld[10792]: 2017-12-25  9:53:20 140183840872192 [Note] WSREP: MemPool(SlaveTrxHandle): hit ratio: 0.0248356, misses: 1335, in use: 0, in pool: 1335

Dec 25 09:53:20 <server-name> mysqld[10792]: 2017-12-25  9:53:20 140183840872192 [Note] WSREP: Shifting CLOSED -> DESTROYED (TO: 32573574)

Dec 25 09:53:20 <server-name> mysqld[10792]: 2017-12-25  9:53:20 140183840872192 [Note] WSREP: Flushing memory map to disk...

Dec 25 09:53:20 <server-name> mysqld[10792]: 2017-12-25  9:53:20 140183840872192 [Note] InnoDB: Starting shutdown...

Dec 25 09:53:20 <server-name> mysqld[10792]: 2017-12-25  9:53:20 140171173402368 [Note] InnoDB: Dumping buffer pool(s) to /var/lib/mysql/ib_buffer_pool

Dec 25 09:53:20 <server-name> mysqld[10792]: 2017-12-25  9:53:20 140171173402368 [Note] InnoDB: Buffer pool(s) dump completed at 171225  9:53:20

Dec 25 09:53:22 <server-name> mysqld[10792]: 2017-12-25  9:53:22 140183840872192 [Note] InnoDB: Shutdown completed; log sequence number 443820945157

Dec 25 09:53:22 <server-name> mysqld[10792]: 2017-12-25  9:53:22 140183840872192 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"

Dec 25 09:53:22 <server-name> mysqld[10792]: 2017-12-25  9:53:22 140183840872192 [Note] /usr/sbin/mysqld: Shutdown complete

Kind regards,

Mati Skiba