Cannor start galera mariaDB cluster after all nodes left the cluster gracefully and servers rebooted

[Natalia R]

Hello,

I had a 3-node Galera MariaDB cluster Version: 10.1.22-MariaDB-1~xenial (mariadb.org binary distribution). Ubuntu 16.04 xenial.

I stopped all 3 nodes and powered them down. 

The first server was powered up and the MariaDB Galera cluster tried to start with the command "galera_new_cluster".

Galera refused to start the first node with the error "To force cluster bootstrap with this node, edit the grastate.dat file manually and set safe_to_bootstrap to 1 .”

I edited /var/lub/mysql/grastate.dat and set "safe_to_bootstrap: 1". 

Here is a content of grastate.dat file: 

# GALERA saved state

version: 2.1

uuid:    ad691a16-2f56-11e7-ae6c-726bfa6905f6

seqno:   -1

safe_to_bootstrap: 1

Now the command "galera_new_cluster" produces a timeout error and Galera Cluster does not start. 

Job for mariadb.service failed because a timeout was exceeded. See "systemctl status mariadb.service" and "journalctl -xe" for details.

Command "systemctl status mariadb.service" output:

Б≈▐ mariadb.service - MariaDB database server

   Loaded: loaded (/etc/systemd/system/mariadb.service; enabled; vendor preset: enabled)

  Drop-In: /etc/systemd/system/mariadb.service.d

           Б■■Б■─migrated-from-my.cnf-settings.conf

   Active: failed (Result: timeout) since Thu 2017-08-10 12:58:11 EDT; 1min 21s ago

  Process: 3735 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= ||   VAR=`/usr/bin/galera_recovery`; [ $? -eq 0 ]   && systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1 (code=killed, signal=TERM)

  Process: 3726 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)

  Process: 3722 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)

    Tasks: 0

   Memory: 18.5M

      CPU: 483ms

Aug 10 12:56:41 vaultv-01 systemd[1]: Starting MariaDB database server...

Aug 10 12:56:41 vaultv-01 mysqld[3735]: 2017-08-10 12:56:41 140359793731840 [Note] /usr/sbin/mysqld (mysqld 10.1.22-MariaDB-1~xenial) starting as process 3965 ...

Aug 10 12:58:11 vaultv-01 systemd[1]: mariadb.service: Start-pre operation timed out. Terminating.

Aug 10 12:58:11 vaultv-01 systemd[1]: Failed to start MariaDB database server.

Aug 10 12:58:11 vaultv-01 systemd[1]: mariadb.service: Unit entered failed state.

Aug 10 12:58:11 vaultv-01 systemd[1]: mariadb.service: Failed with result 'timeout'.

Content of galera configuration file /etc/mysql/conf.d/galera.cnf:

[galera]

# * Galera-related settings

# Galera Mandatory settings

binlog_format                  = ROW

default_storage_engine         = InnoDB

innodb_autoinc_lock_mode       = 2

innodb_flush_log_at_trx_commit = 0

# bind-address                   = 10.10.10.101

bind-address                   = 0.0.0.0

# Galera Provider Configuration

wsrep_on       = ON

wsrep_provider = /usr/lib/galera/libgalera_smm.so

# Galera Cluster Configuration

wsrep_cluster_name    = "test_cluster"

# wsrep_cluster_address = "gcomm://10.10.10.101,10.10.10.102,10.10.10.103"

wsrep_cluster_address = "gcomm://"

# Galera gcache

wsrep_provider_options = 'gcache.size=130M'

# Galera Synchronization Configuration

wsrep_sst_method = xtrabackup-v2

wsrep_sst_auth   = user:password

# Galera Node Configuration

wsrep_node_address = "10.10.10.101"

wsrep_node_name    = "vaultv-01"

# Secure Galera Replication traffic

wsrep_provider_options="socket.ssl_key=/etc/mysql/ssl/server-key.pem;socket.ssl_cert=/etc/mysql/ssl/server-cert.pem;socket.ssl_ca=/etc/mysql/ssl/ca-cert.pem"

# Galera Optional setting

#wsrep_slave_threads=1

[sst]

streamfmt    = xbstream

compressor   = 'gzip'

decompressor = 'gzip -dc'

rebuild      = ON

compact      = ON

encrypt      = 4

ssl-ca       = /etc/mysql/ssl/ca-cert.pem

ssl-cert     = /etc/mysql/ssl/server-cert.pem

ssl-key      = /etc/mysql/ssl/server-key.pem

progress     = /var/log/mysql/xtrabackup-progress.log

time         = 1

Please any advise how  to start the first node and the rest of the cluster. 

Thank you.

[STS-1]

I have the same problem on Maria 10.1.30 Galera. How did you resolve this problem?

Thanks!

[et4...@gmail.com]

The same happened to me.  For now, the only way for myself to resolve this problem is as the error mentioned, set the bootstrap=1 then start a new galera_cluster then start mysql (for all nodes)

I edited /var/lub/mysql/grastate.dat and set "safe_to_bootstrap: 1".

then sudo galera_new_cluster
and sudo service mysql start

this then starts my first node, then I have to start the other nodes manually.

This happens each time rebooting the server, but for now this is the only way I found out would solve the problem