Timeout problem while starting second node using SST rsync, xtrabackup on WAN environment

[mat...@websolution.it]

Hi,

just installed a 2-nodes cluster to use in stage environment, as i'm not under fiber but i need the customers to see websites quickly in stage. My idea was to have a local database under ADSL where i can communicate while developing in a short time replicated with a fiber one in a remote env.

Maybe my connection is too slow for this system, but i cannot really understand how to proceed...
The first time i connected 2 nodes, everything worked like a charm. I imported 30MB of databases after 2 nodes were up, and the IST was amazingly fast. 
Then i thought to do some disaster recovery tests, and i gracely shutted down both servers.

Now, one of the nodes start perfectly, but i still have on the other node the error

root@ubuntu:~# service mysql start
Job for mariadb.service failed because a timeout was exceeded. See "systemctl status mariadb.service" and "journalctl -xe" for details.

then running the command systemctl status mariadb.service
(please note i replaced my addressed with blablabla)

gen 31 10:33:36 ubuntu mysqld[18699]: 2017-01-31 10:33:36 139695862314752 [Note] WSREP: (54cd2ab9, 'tcp://0.0.0.0:4567') turning message relay requesting off

gen 31 10:35:02 ubuntu systemd[1]: mariadb.service: Start operation timed out. Terminating.

gen 31 10:36:32 ubuntu systemd[1]: mariadb.service: State 'stop-final-sigterm' timed out. Skipping SIGKILL. Entering failed mode.

gen 31 10:36:32 ubuntu systemd[1]: Failed to start MariaDB database server.

gen 31 10:36:32 ubuntu systemd[1]: mariadb.service: Unit entered failed state.

gen 31 10:36:32 ubuntu systemd[1]: mariadb.service: Failed with result 'timeout'.

gen 31 10:37:53 ubuntu rsyncd[18757]: sent 300 bytes  received 79722094 bytes  total size 79702016

gen 31 10:37:53 ubuntu rsyncd[19787]: connect from blablabla (blablabla)

gen 31 10:37:53 ubuntu rsyncd[19787]: rsync to rsync_sst-log_dir/ from blablabla (blablabla)

gen 31 10:37:53 ubuntu rsyncd[19787]: receiving file list

the funny thing is if i launch on the failed node it remains connected to the living node...

root@ubuntu:~# ps aux | grep mysql

mysql    18699  0.1  6.3 375592 64516 ?        Ssl  10:33   0:00 /usr/sbin/mysqld --wsrep_start_position=70dfa51e-e3e2-11e6-b0b2-6beb9eac4f0e:2093

mysql    18709  0.0  0.0   4508   696 ?        S    10:33   0:00 sh -c wsrep_sst_rsync --role 'joiner' --address 'blablabla' --datadir '/var/lib/mysq /'   --parent '18699' --binlog '/var/log/mysql/mariadb-bin'

mysql    18710  0.3  0.3  12708  3152 ?        S    10:33   0:01 /bin/bash -ue /usr//bin/wsrep_sst_rsync --role joiner --address blablabla --datadir /var/lib/mysql/ --parent 18699 --binlog /var/log/mysql/mariadb-bin

mysql    18742  0.0  0.1  14060  1052 ?        S    10:33   0:00 rsync --daemon --no-detach --port 4444 --config /var/lib/mysql//rsync_sst.conf

mysql    19787  0.0  0.2  27412  2884 ?        S    10:37   0:00 rsync --daemon --no-detach --port 4444 --config /var/lib/mysql//rsync_sst.conf

mysql    19788  1.0  0.1  27672  1264 ?        S    10:37   0:01 rsync --daemon --no-detach --port 4444 --config /var/lib/mysql//rsync_sst.conf

mysql    20531  0.0  0.0   7300   652 ?        S    10:40   0:00 sleep 1

root     20533  0.0  0.0  14252  1016 pts/1    S+   10:40   0:00 grep --color=auto mysql

i tried with rsync,xtrabackup and all times the 2nd node fails.

even if i try the sst mysqldump method, the first node continue to pass from SYNCED to DESYNCED and the second node goes down and up with the state JOINING

I'm running

• Ubuntu 16.04
• Ver 15.1 Distrib 10.1.21-MariaDB

Configuration files are the same on both nodes:

(still covered addresses)

root@ubuntu:~# cat /etc/mysql/my.cnf

# MariaDB database server configuration file.

#

# You can copy this file to one of:

# - "/etc/mysql/my.cnf" to set global options,

# - "~/.my.cnf" to set user-specific options.

#

# One can use all long options that the program supports.

# Run program with --help to get a list of available options and with

# --print-defaults to see which it would actually understand and use.

#

# For explanations see

# http://dev.mysql.com/doc/mysql/en/server-system-variables.html

# This will be passed to all mysql clients

# It has been reported that passwords should be enclosed with ticks/quotes

# escpecially if they contain "#" chars...

# Remember to edit /etc/mysql/debian.cnf when changing the socket location.

[client]

port            = 3306

socket          = /var/run/mysqld/mysqld.sock

# Here is entries for some specific programs

# The following values assume you have at least 32M ram

# This was formally known as [safe_mysqld]. Both versions are currently parsed.

[mysqld_safe]

socket          = /var/run/mysqld/mysqld.sock

nice            = 0

[mysqld]

#

# * Basic Settings

#

user            = mysql

pid-file        = /var/run/mysqld/mysqld.pid

socket          = /var/run/mysqld/mysqld.sock

port            = 3306

basedir         = /usr

datadir         = /var/lib/mysql

tmpdir          = /tmp

lc_messages_dir = /usr/share/mysql

lc_messages     = en_US

skip-external-locking

#

# Instead of skip-networking the default is now to listen only on

# localhost which is more compatible and is not less secure.

bind-address            = ::

#

# * Fine Tuning

#

max_connections         = 100

connect_timeout         = 5

wait_timeout            = 600

max_allowed_packet      = 16M

thread_cache_size       = 128

sort_buffer_size        = 4M

bulk_insert_buffer_size = 16M

tmp_table_size          = 32M

max_heap_table_size     = 32M

#

# * MyISAM

#

# This replaces the startup script and checks MyISAM tables if needed

# the first time they are touched. On error, make copy and try a repair.

myisam_recover_options = BACKUP

key_buffer_size         = 128M

#open-files-limit       = 2000

table_open_cache        = 400

myisam_sort_buffer_size = 512M

concurrent_insert       = 2

read_buffer_size        = 2M

read_rnd_buffer_size    = 1M

#

# * Query Cache Configuration

#

# Cache only tiny result sets, so we can fit more in the query cache.

query_cache_limit               = 128K

query_cache_size                = 64M

# for more write intensive setups, set to DEMAND or OFF

#query_cache_type               = DEMAND

#

# * Logging and Replication

#

# Both location gets rotated by the cronjob.

# Be aware that this log type is a performance killer.

# As of 5.1 you can enable the log at runtime!

#general_log_file        = /var/log/mysql/mysql.log

#general_log             = 1

#

# Error logging goes to syslog due to /etc/mysql/conf.d/mysqld_safe_syslog.cnf.

#

# we do want to know about network errors and such

log_warnings            = 2

#

# Enable the slow query log to see queries with especially long duration

#slow_query_log[={0|1}]

slow_query_log_file     = /var/log/mysql/mariadb-slow.log

long_query_time = 10

#log_slow_rate_limit    = 1000

log_slow_verbosity      = query_plan

#log-queries-not-using-indexes

#log_slow_admin_statements

#

# The following can be used as easy to replay backup logs or for replication.

# note: if you are setting up a replication slave, see README.Debian about

#       other settings you may need to change.

#server-id              = 1

#report_host            = master1

#auto_increment_increment = 2

#auto_increment_offset  = 1

log_bin                 = /var/log/mysql/mariadb-bin

log_bin_index           = /var/log/mysql/mariadb-bin.index

# not fab for performance, but safer

#sync_binlog            = 1

expire_logs_days        = 10

max_binlog_size         = 100M

# slaves

#relay_log              = /var/log/mysql/relay-bin

#relay_log_index        = /var/log/mysql/relay-bin.index

#relay_log_info_file    = /var/log/mysql/relay-bin.info

#log_slave_updates

#read_only

#

# If applications support it, this stricter sql_mode prevents some

# mistakes like inserting invalid dates etc.

#sql_mode               = NO_ENGINE_SUBSTITUTION,TRADITIONAL

#

# * InnoDB

#

# InnoDB is enabled by default with a 10MB datafile in /var/lib/mysql/.

# Read the manual for more InnoDB related options. There are many!

default_storage_engine  = InnoDB

# you can't just change log file size, requires special procedure

#innodb_log_file_size   = 50M

innodb_buffer_pool_size = 256M

innodb_log_buffer_size  = 8M

innodb_file_per_table   = 1

innodb_open_files       = 400

innodb_io_capacity      = 400

innodb_flush_method     = O_DIRECT

#

# * Security Features

#

# Read the manual, too, if you want chroot!

# chroot = /var/lib/mysql/

#

# For generating SSL certificates I recommend the OpenSSL GUI "tinyca".

#

# ssl-ca=/etc/mysql/cacert.pem

# ssl-cert=/etc/mysql/server-cert.pem

# ssl-key=/etc/mysql/server-key.pem

#

# * Galera-related settings

#

[galera]

# Mandatory settings

#wsrep_on=ON

#wsrep_provider=

#wsrep_cluster_address=

#binlog_format=row

#default_storage_engine=InnoDB

#innodb_autoinc_lock_mode=2

#

# Allow server to accept connections on all interfaces.

#

#bind-address=0.0.0.0

#

# Optional setting

#wsrep_slave_threads=1

#innodb_flush_log_at_trx_commit=0

binlog_format=ROW

innodb_autoinc_lock_mode=2

default_storage_engine=InnoDB

bind-address=::

wsrep_on=ON

wsrep_provider=/usr/lib/galera/libgalera_smm.so

wsrep_cluster_name="stage_cluster"

wsrep_cluster_address="gcomm://server1,server2"

wsrep_sst_method=rsync

#wsrep_sst_method=xtrabackup

#wsrep_sst_auth=ws-stage-admin:48ePwdh9km2NFABt

#datadir=/var/lib/mysql/

wsrep_node_address="server1"

wsrep_node_name="stage_server_local"

[mysqldump]

quick

quote-names

max_allowed_packet      = 16M

[mysql]

#no-auto-rehash # faster start of mysql but no tab completion

[isamchk]

key_buffer              = 16M

#

# * IMPORTANT: Additional settings that can override those from this file!

#   The files must end with '.cnf', otherwise they'll be ignored.

#

!includedir /etc/mysql/conf.d/

Could you help me please?

[Philip Stoev]

Hello,

Please use journalctl and or check in /var/log/mysql for more log lines from
mysqld.

There will probably be additional error messages there that will clarify the
situation.

Can it be that one or both of your nodes are behind NAT or a firewall? Each
node must be able to open TCP connections to the other node and the other
way around. Do the nodes have separate public and private IPs for each node?

Philip Stoev

--
You received this message because you are subscribed to the Google Groups
"codership" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to codership-tea...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Lammert Bies]

This could be a firewall issue with your ADSL connection. In order for a cluster to properly startup, a number of TCP ports must allow connections initiated from both servers. With ADSL connections it is often possible to initiate outbound connections, but inbound connections are blocked by default.

You should check the following ports: 3306, 4444, 4567, 4568

[mat...@websolution.it]

Oh, pardon...

first node:

root@ubuntu:~# iptables -L

Chain INPUT (policy ACCEPT)

target     prot opt source               destination

Chain FORWARD (policy ACCEPT)

target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)

target     prot opt source               destination

second node

root@stage:~# iptables -L

Chain INPUT (policy ACCEPT)

target     prot opt source               destination

ACCEPT     all  --  anywhere             anywhere             state RELATED,ESTABLISHED

ACCEPT     all  --  anywhere             anywhere

ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:webmin

ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:ssh

ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:http

ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:mysql

ACCEPT     udp  --  anywhere             anywhere             udp dpt:mysql

ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:domain

ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:4567

ACCEPT     udp  --  anywhere             anywhere             udp dpt:4567

ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:4568

ACCEPT     udp  --  anywhere             anywhere             udp dpt:4568

ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:4444

ACCEPT     udp  --  anywhere             anywhere             udp dpt:4444

ACCEPT     icmp --  anywhere             anywhere

ACCEPT     udp  --  anywhere             anywhere             udp dpts:bootps:bootpc

LOG        all  --  anywhere             anywhere             limit: avg 5/min burst 5 LOG level debug prefix "iptables denied: "

DROP       all  --  anywhere             anywhere

Chain FORWARD (policy ACCEPT)

target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)

target     prot opt source               destination

if i telnet both servers each other are reachable on the ports your wrote...

[mat...@websolution.it]

Hey guys, still blocked here...

root@stage:~# systemctl status mariadb.service
● mariadb.service - MariaDB database server
   Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/mariadb.service.d
           └─migrated-from-my.cnf-settings.conf
   Active: failed (Result: timeout) since gio 2017-02-09 09:35:01 CET; 29s ago
  Process: 11502 ExecStartPost=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
  Process: 11499 ExecStartPost=/etc/mysql/debian-start (code=exited, status=0/SUCCESS)
  Process: 7877 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= ||   VAR=`/usr/bin/galera_recovery`; [ $? -eq 0 ]   && systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1 (code=ex
  Process: 7872 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
  Process: 7866 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
 Main PID: 8124
    Tasks: 14
   Memory: 61.6M
      CPU: 4.107s
   CGroup: /system.slice/mariadb.service
           ├─8124 /usr/sbin/mysqld --wsrep_start_position=0e58ea31-e798-11e6-a5e1-9ab12df4b919:0
           ├─8133 sh -c wsrep_sst_rsync --role 'joiner' --address '89.31.78.211' --datadir '/var/lib/mysql/'   --parent '8124' --binlog '/var/log/mysql/mariadb-bin'
           ├─8134 /bin/bash -ue /usr//bin/wsrep_sst_rsync --role joiner --address 89.31.78.211 --datadir /var/lib/mysql/ --parent 8124 --binlog /var/log/mysql/mariadb-bin
           ├─8166 rsync --daemon --no-detach --port 4444 --config /var/lib/mysql//rsync_sst.conf
           ├─8185 rsync --daemon --no-detach --port 4444 --config /var/lib/mysql//rsync_sst.conf
           ├─8186 rsync --daemon --no-detach --port 4444 --config /var/lib/mysql//rsync_sst.conf
           └─9139 sleep 1


feb 09 09:32:01 stage mysqld[8124]: 2017-02-09  9:32:01 140636448504064 [Note] /usr/sbin/mysqld (mysqld 10.1.21-MariaDB-1~xenial) starting as process 8124 ...
feb 09 09:32:02 stage rsyncd[8166]: rsyncd version 3.1.1 starting, listening on port 4444
feb 09 09:32:03 stage rsyncd[8185]: connect from 2-229-49-82.ip195.fastwebnet.it (2.229.49.82)
feb 09 09:32:03 stage rsyncd[8185]: rsync to rsync_sst/ from 2-229-49-82.ip195.fastwebnet.it (2.229.49.82)
feb 09 09:32:04 stage rsyncd[8185]: receiving file list
feb 09 09:33:31 stage systemd[1]: mariadb.service: Start operation timed out. Terminating.
feb 09 09:35:01 stage systemd[1]: mariadb.service: State 'stop-final-sigterm' timed out. Skipping SIGKILL. Entering failed mode.
feb 09 09:35:01 stage systemd[1]: Failed to start MariaDB database server.
feb 09 09:35:01 stage systemd[1]: mariadb.service: Unit entered failed state.
feb 09 09:35:01 stage systemd[1]: mariadb.service: Failed with result 'timeout'.

root@stage:~# journalctl -xe
feb 09 09:33:31 stage systemd[1]: mariadb.service: Start operation timed out. Terminating.

feb 09 09:35:01 stage systemd[1]: mariadb.service: State 'stop-final-sigterm' timed out. Skipping SIGKILL. Entering failed mode.

feb 09 09:35:01 stage systemd[1]: Failed to start MariaDB database server.

-- Subject: L'unità mariadb.service è fallita

-- Defined-By: systemd

-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel

--

-- L'unità mariadb.service è fallita.

--

-- Il risultato è failed.

feb 09 09:35:01 stage systemd[1]: mariadb.service: Unit entered failed state.

feb 09 09:35:01 stage systemd[1]: mariadb.service: Failed with result 'timeout'.

but if i

 watch "ps aux|grep mysql"

mysql     8124  0.0  1.8 400176 76824 ?        Ssl  09:32   0:00 /usr/sbin/mysqld --wsrep_start_position=0e58ea31-e798-11e6-a5e1-9ab12df4b919:0

mysql     8133  0.0  0.0   4508   748 ?        S    09:32   0:00 sh -c wsrep_sst_rsync --role 'joiner' --address 'blablabla' --datadir '/var/lib/mysql/'   --parent '8124' --binlog '/var/log/mysql/mariadb-b

mysql     8134  0.0  0.0  12732  3112 ?        S    09:32   0:00 /bin/bash -ue /usr//bin/wsrep_sst_rsync --role joiner --address blablabla --datadir /var/lib/mysql/ --parent 8124 --binlog /var/log/mysql/ma

mysql     8166  0.0  0.0  14060  1056 ?        S    09:32   0:00 rsync --daemon --no-detach --port 4444 --config /var/lib/mysql//rsync_sst.conf

mysql     8185  0.0  0.0  27412  2852 ?        S    09:32   0:00 rsync --daemon --no-detach --port 4444 --config /var/lib/mysql//rsync_sst.conf

mysql     8186  0.1  0.0  27672  1240 ?        S    09:32   0:01 rsync --daemon --no-detach --port 4444 --config /var/lib/mysql//rsync_sst.conf

it seems processes are still up and running...

[Aaron Morand]

Silly question, but are you specifying wsrep_node_name with the same value in both the my.cnf files?



To unsubscribe from this group and stop receiving emails from it, send an email to codership-team+unsubscribe@googlegroups.com.

[Matteo Lazzarin]

Of course not, 

1) "stage_server_local"
2) "stage_server_remote_1"

Matteo Lazzarin
FULLSTAK WEB DEVELOPER

	WebSolution S.n.c. Via Roma, 145 - 31020 Villorba (TV) T. +39 0422 910 205 F. +39 0422 611 038 mat...@websolution.it websolution.it Seguici su: LINKEDIN e TWITTER

You received this message because you are subscribed to a topic in the Google Groups "codership" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/codership-team/dLDXE1U6llg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to codership-tea...@googlegroups.com.

[Aaron Morand]

What order did you bring them down and how did you bring them back up?

To unsubscribe from this group and all its topics, send an email to codership-team+unsubscribe@googlegroups.com.

[Matteo Lazzarin]

I tried both of them with the same result....

Now, whit the "Avoid SST" method, everything is in order... but i think this SST timeout problem needs a bit of attention for WAN users...

To unsubscribe from this group and all its topics, send an email to codership-tea...@googlegroups.com.

[Andrew Garner]

This is running under systemd, which has a DefaultTimeoutStartSec of 90 seconds.  This matches roughly to the ~90 seconds timeout I see in the log timestamps:

gen 31 10:33:36 ubuntu mysqld[18699]: 2017-01-31 10:33:36 139695862314752 [Note] WSREP: (54cd2ab9, 'tcp://0.0.0.0:4567') turning message relay requesting off

gen 31 10:35:02 ubuntu systemd[1]: mariadb.service: Start operation timed out. Terminating.

You likely want to just bump up the service timeout.  The easiest way is to drop a file under /etc/systemd/system/mariadb.service.d/.  May something along the lines of:

$ sudo mkdir /etc/systemd/system/mariadb.service.d/ && echo $'[Service]\nTimeoutSec=2h\n' | sudo tee /etc/systemd/system/mariadb.service.d/timeouts.conf

That affects both Startup and Stop timeouts - you can set those independently if you want, and the actual values should be adjusted for your needs.

You have to reload the service config afterwards:

$ sudo systemctl daemon-reload

And then you can verify the actual timeout:

$ sudo systemctl show mariadb | grep Timeout

MariaDB should probably use a higher default out of the box, but I'm pretty sure it uses the default.  I've been adding a similar config to my environments to avoid this issue for some time.

~Andrew

To unsubscribe from this group and all its topics, send an email to codership-team+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Matteo Lazzarin]

So, could you explain too why connections remain opened and there's a loop on sst?

Matteo Lazzarin
FULLSTAK WEB DEVELOPER

	WebSolution S.n.c. Via Roma, 145 - 31020 Villorba (TV) T. +39 0422 910 205 F. +39 0422 611 038 mat...@websolution.it websolution.it Seguici su: LINKEDIN e TWITTER

To unsubscribe from this group and all its topics, send an email to codership-tea...@googlegroups.com.

[Matteo Lazzarin]

please see the loop here... https://github.com/codership/galera/issues/450

Matteo Lazzarin
FULLSTAK WEB DEVELOPER

	WebSolution S.n.c. Via Roma, 145 - 31020 Villorba (TV) T. +39 0422 910 205 F. +39 0422 611 038 mat...@websolution.it websolution.it Seguici su: LINKEDIN e TWITTER

[Andrew Garner]

The service timeout just sends a SIGTERM but mysqld doesn't actually act on that until the SST portion is complete.   The background processes keep running until they finish, even though the service start failed.

Offhand, this looks like spotty connectivity between hosts and SST is taking a really long time.  server2 seems to be losing connectivity to server1 periodically while the SST process is running.  I can see a similar sequence of events in the error log of my environment by introducing packet loss or some other communications slowdown and starting SST. I suppose other performance problems might lead to a similar result (bandwidth saturation, io saturation, etc.)

I imagine this would eventually complete if given enough time.  Although after the service times out, mysqld will just shut itself down after the SST portion - and you might run into https://jira.mariadb.org/browse/MDEV-9772 as part of that.

How much data are you transferring? You mentioned loading 30MB of databases, which seems small.  Maybe check what "sudo du -sh /var/lib/mysql/" shows.  Any idea how long it takes to copy data between sites without SST?

...

[Message clipped]  

[mat...@websolution.it]

I think this is exactly the problem i'm searching for....

After inspecting a while....

root@ubuntu:~# systemctl show mysql.service -p TimeoutSec
root@ubuntu:~# systemctl show mysql.service -p TimeoutStartSec
root@ubuntu:~# systemctl show mysql.service -p TimeoutStopSec
root@ubuntu:~# systemctl show mysql.service -p TimeoutUSec
root@ubuntu:~# systemctl show mysql.service -p TimeoutStartUSec
TimeoutStartUSec=1min 30s
root@ubuntu:~# systemctl show mysql.service -p TimeoutStopUSec
TimeoutStopUSec=1min 30s

I tried your solution above, but it isn't working....

This is what i've done:

echo $'[Service]\nTimeoutStartUSec=2h\nTimeoutStopUSec=2h' | sudo tee /etc/systemd/system/mariadb.service.d/timeouts.conf
systemctl daemon-reload

but then....

root@ubuntu:~# systemctl show mysql.service -p TimeoutStartUSec
TimeoutStartUSec=1min 30s
root@ubuntu:~# systemctl show mysql.service -p TimeoutStopUSec
TimeoutStopUSec=1min 30s

What i'm missing?

You received this message because you are subscribed to a topic in the Google Groups "codership" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/codership-team/dLDXE1U6llg/unsubscribe.

To unsubscribe from this group and all its topics, send an email to codership-tea...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to a topic in the Google Groups "codership" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/codership-team/dLDXE1U6llg/unsubscribe.

To unsubscribe from this group and all its topics, send an email to codership-tea...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Andrew Garner]

I don't think TimeoutStartUsec is a valid option - if you look in a log file like /var/log/syslog you'll probably see a line like the following after "systemctl daemon-reload":

systemd[1]: [/etc/systemd/system/mariadb.service.d/timeouts.conf:2] Unknown lvalue 'TimeoutStartUSec' in section 'Service'

systemd[1]: [/etc/systemd/system/mariadb.service.d/timeouts.conf:3] Unknown lvalue 'TimeoutStopUSec' in section 'Service'

This is certainly confusing, because "*USec" is the property name displayed in systemctl show output.  So I suggest using one of the documented timeout options listed here:

https://www.freedesktop.org/software/systemd/man/systemd.service.html#TimeoutStartSec=

In my command, I used TimeoutSec, which sets both the start and stop timeouts.   Using that option works.  Alternatively if you want to set separate start / stop timeouts, then use TimeoutStartSec and TimeoutStopSec.   Note, when you set those options and use "systemctl show mariadb", you'll see the associated value in the Timeout*Usec properties.

FWIW, I see there's an issue here talking about this systemd behavior here: https://github.com/systemd/systemd/issues/2047

~Andrew

To unsubscribe from this group and stop receiving emails from it, send an email to codership-team+unsubscribe@googlegroups.com.

[Azu Azu]

Hello, 
My apology if this is not the normal way but I just have to say thank you to @Andrew Garner.

I had the same problems and was almost 3 weeks into the issue. Stumble your post last night and *wham work as expected. Thank you!!!

Regards
Azu