I can start MariaDB/Galera manually but it won't start automatically after a reboot

[jb2cool]

I have a MariaDB/Galera cluster running on three RHEL7 nodes and it was running beautifully till some time in November i think when an update (Sorry, i don't have versions but it was 10.1 something) stopped this starting up automatically. I can start the process manually and it starts up just fine but it doesn't start automatically which makes me sad.

Any ideas where i can start with this?

[Alex Evonosky]

I seen this before on Ubuntu, but to prevent this I always add the manual startup in /etc/rc.local script and call this script from a cronjob at reboot.  e.g.:

/etc/crontab:

@reboot root /etc/rc.local

This way, I'm not relying on other processes..  Others may have better ideas, but this worked for me just fine.

On Tue, Jan 24, 2017 at 12:39 PM, jb2cool <jb2...@gmail.com> wrote:

I have a MariaDB/Galera cluster running on three RHEL7 nodes and it was running beautifully till some time in November i think when an update (Sorry, i don't have versions but it was 10.1 something) stopped this starting up automatically. I can start the process manually and it starts up just fine but it doesn't start automatically which makes me sad.

Any ideas where i can start with this?

--
You received this message because you are subscribed to the Google Groups "codership" group.
To unsubscribe from this group and stop receiving emails from it, send an email to codership-team+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jörg Brühe]

Hi!

RHEL 7 uses systemd.
I have not checked MariaDB whether it comes with the necessary files
(service unit definition, ...), let's assume it does, and let's assume
it uses the service name 'mysqld'.

Use "sudo systemctl status mysqld" to check what systemd thinks about
it, the output will (among other things) tell whether it is running and
whether it is "enabled" (for autostart at boot).

Use "sudo systemctl start mysqld" to start the server and "sudo
systemctl stop mysqld" to stop it.
Use "sudo systemctl enable mysqld" to make the server autostart on boot.

HTH,
Jörg

--
Joerg Bruehe, Senior MySQL Support Engineer, joerg....@fromdual.com
FromDual GmbH, Rebenweg 6, CH - 8610 Uster; phone +41 44 500 58 26
Geschäftsführer: Oliver Sennhauser
Handelsregister-Eintrag: CH-020.4.044.539-3

[jb2cool]

Thanks for the reply Jörg. This is enabled and tries to start, it just falls over somewhere.

[##########@########## ~]$ sudo systemctl status mysqld -l
[sudo] password for ##########:
● mariadb.service - MariaDB database server
   Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/mariadb.service.d
           └─migrated-from-my.cnf-settings.conf
   Active: failed (Result: exit-code) since Wed 2017-01-25 13:12:42 GMT; 3min 37s ago
  Process: 500 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= ||   VAR=`/usr/bin/galera_recovery`; [ $? -eq 0 ]   && systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1 (code=exited, status=1/FAILURE)
  Process: 484 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)

Jan 25 13:12:42 ########## sh[500]: WSREP: Failed to start mysqld for wsrep recovery: '2017-01-25 13:12:39 139791298672896 [Note] InnoDB: Using mutexes to ref count buffer pool pages
Jan 25 13:12:42 ########## sh[500]: 2017-01-25 13:12:39 139791298672896 [Note] InnoDB: The InnoDB memory heap is disabled

Jan 25 13:12:42 ########## sh[500]: 2017-01-25 13:12:39 139791298672896 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins

Jan 25 13:12:42 ########## sh[500]: 2017-01-25 13:12:39 139791298672896 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier

Jan 25 13:12:42 ########## sh[500]: 2017-01-25 13:12:39 139791298672896 [Note] InnoDB: Compressed tables use zlib 1.2.7

Jan 25 13:12:42 ########## sh[500]: 2017-01-25 13:12:39 139791298672896 [Note] InnoDB: Using Linux native AIO

Jan 25 13:12:42 ########## sh[500]: 2017-01-25 13:12:39 139791298672896 [Note] InnoDB: Using SSE crc32 instructions

Jan 25 13:12:42 ########## sh[500]: 2017-01-25 13:12:39 139791298672896 [Note] InnoDB: Initializing buffer pool, size = 122.0M

Jan 25 13:12:42 ########## sh[500]: 2017-01-25 13:12:39 139791298672896 [Note] InnoDB: Completed initialization of buffer pool

Jan 25 13:12:42 ########## sh[500]: 2017-01-25 13:12:39 139791298672896 [Note] InnoDB: Highest supported file format is Barracuda.

[Jörg Brühe]

Hi!

On 25.01.2017 14:33, jb2cool wrote:
> Thanks for the reply Jörg. This is enabled and tries to start, it just
> falls over somewhere.

I didn't take that from the original question.

>
> [##########@########## ~]$ sudo systemctl status mysqld -l
> [sudo] password for ##########:
> ● mariadb.service - MariaDB database server
> Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled;
> vendor preset: disabled)
> Drop-In: /etc/systemd/system/mariadb.service.d
> └─migrated-from-my.cnf-settings.conf
> Active: failed (Result: exit-code) since Wed 2017-01-25 13:12:42 GMT;
> 3min 37s ago
> Process: 500 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ]
> && VAR= || VAR=`/usr/bin/galera_recovery`; [ $? -eq 0 ] && systemctl
> set-environment _WSREP_START_POSITION=$VAR || exit 1 (code=exited,
> status=1/FAILURE)
> Process: 484 ExecStartPre=/bin/sh -c systemctl unset-environment
> _WSREP_START_POSITION (code=exited, status=0/SUCCESS)

AIUI, systemd is trying to start the service, and this begins with the
"ExecStartPre" script. Even though I like one-liners for interactive
use, that shell command line would have deserved a script IMO.

"status=1" to me implies the "exit 1" step was taken, which means the
call "systemctl set-environment ..." must have failed.
In your place, I would check what "/usr/bin/galera_recovery" returns and
try whether I could use that in a "set-environment" call manually.

>
> Jan 25 13:12:42 ########## sh[500]: WSREP: Failed to start mysqld for
> wsrep recovery: '2017-01-25 13:12:39 139791298672896 [Note] InnoDB:
> Using mutexes to ref count buffer pool pages
> Jan 25 13:12:42 ########## sh[500]: 2017-01-25 13:12:39 139791298672896
> [Note] InnoDB: The InnoDB memory heap is disabled
>

> [[...]]

This simply looks weird.
If that's the result of "/usr/bin/galera_recovery", I'd guess something
goes terribly wrong.
But if it isn't, why does it appear here?

Sorry, ATM I'm out of ideas.

[Lammert Bies]

I am using a 5 node cluster on Centos 7 machines which is basically the same as RHEL 7 and faced the same type of problem. There are two init scripts which try to start a MariaDB/cluster on this OS. One is called mariadb, and the other is called mysql. The one called mariadb fails to startup a cluster properly, but the other succeeds in my setup.

You may try to issue the following commands and see if you can get the cluster running:

systemctl stop mariadb

systemctl disable mariadb

systemctl enable mysql

systemctl start mysql

This stops and disables any daemon started with the mariadb script and starts the server again with the mysql script.