"turning message relay requesting on" in log file

[Haris Zukanovic]

Hi,

Are these Notes in log indication of connectivity problems?
I am running 3 nodes cluster in a WAN environment. Connectivity should be good, but of course it is far away from Gigabit local network.
Ping response times between servers vary from 9-11ms.
Maybe I could optimize my configuration? Any advice?

This is a part of log from node1:
120711 9:26:32 [Note] WSREP: (e2b5bf6b-cac9-11e1-0800-c3d555400bc8, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://node3:4567 
120711 9:26:33 [Note] WSREP: (e2b5bf6b-cac9-11e1-0800-c3d555400bc8, 'tcp://0.0.0.0:4567') turning message relay requesting off
120711 10:11:09 [Warning] Invalid (old?) table or database name '#sql2-5750-5ced'
120711 10:11:10 [Warning] Invalid (old?) table or database name '#sql2-5750-5cee'
120711 11:36:32 [Note] WSREP: (e2b5bf6b-cac9-11e1-0800-c3d555400bc8, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://node3:4567 
120711 11:36:33 [Note] WSREP: (e2b5bf6b-cac9-11e1-0800-c3d555400bc8, 'tcp://0.0.0.0:4567') reconnecting to 9a8b2f60-cac9-11e1-0800-5704883a0b2d (tcp://node3:4567), attempt 0
120711 11:36:33 [Note] WSREP: (e2b5bf6b-cac9-11e1-0800-c3d555400bc8, 'tcp://0.0.0.0:4567') turning message relay requesting off
120711 11:36:32 [Note] WSREP: (e2b5bf6b-cac9-11e1-0800-c3d555400bc8, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://node3:4567 
120711 11:36:33 [Note] WSREP: (e2b5bf6b-cac9-11e1-0800-c3d555400bc8, 'tcp://0.0.0.0:4567') reconnecting to 9a8b2f60-cac9-11e1-0800-5704883a0b2d (tcp://node3:4567), attempt 0
120711 11:36:33 [Note] WSREP: (e2b5bf6b-cac9-11e1-0800-c3d555400bc8, 'tcp://0.0.0.0:4567') turning message relay requesting off

my.cnf

# All files in this package is subject to the GPL v2 license
# More information is in the COPYING file in the top directory of this package.
# Copyright (C) 2011 severalnines.com

[MYSQLD]
user=mysql
basedir=/usr/
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
pid-file=mysqld.pid
port=3306
log-error=error.log
#log-output=FILE
#relay-log=relay-bin
### INNODB 
innodb-buffer-pool-size=2867M
innodb-flush-log-at-trx_commit=2
innodb-file-per-table=1
innodb-doublewrite=0
innodb_log_file_size=256M
innodb-log-files-in-group=3
#innodb-buffer-pool-instances=4
innodb-thread-concurrency=0
#innodb-file-format=barracuda
innodb-flush-method = O_DIRECT
innodb_locks_unsafe_for_binlog=1
innodb_autoinc_lock_mode=2
engine-condition-pushdown=1

#REPLICATION SPECIFIC - GENERAL
#server-id must be unique across all mysql servers participating in replication.
#server-id=SERVERID
#auto_increment_increment=2
#auto_increment_offset=SERVERID
#REPLICATION SPECIFIC - MASTER
binlog_format=ROW
#log-bin=binlog
#log-slave-updates=1
#OTHER THINGS, BUFFERS ETC
key_buffer_size = 256M
max-allowed-packet = 512M
sort-buffer-size = 512K
read-buffer-size = 256K
read-rnd-buffer-size = 512K
#thread-cache-size=1024
myisam-sort-buffer_size = 8M
skip-name-resolve
memlock=0
sysdate-is-now=1
max-connections=512
thread-cache-size=512
query-cache-type = 0
query-cache-size = 0
table-open_cache=1024
lower-case-table-names=0
##
## WSREP options
##

# Full path to wsrep provider library or 'none'
wsrep_provider=/usr/lib64/galera/libgalera_smm.so

wsrep_node_address=node1
# Provider specific configuration options
#wsrep_provider_options=

# Logical cluster name. Should be the same for all nodes.
wsrep_cluster_name="my_wsrep_cluster"

# Group communication system handle
#wsrep_cluster_address="dummy://"

# Human-readable node name (non-unique). Hostname by default.
#wsrep_node_name=

# Address for incoming client connections. Autodetect by default.
#wsrep_node_incoming_address=

# How many threads will process writesets from other nodes
# (more than one untested)
wsrep_slave_threads=1

# DBUG options for wsrep provider
#wsrep_dbug_option

# Generate fake primary keys for non-PK tables (required for multi-master
# and parallel applying operation)
wsrep_certify_nonPK=1

# Location of the directory with data files. Needed for non-mysqldump
# state snapshot transfers. Defaults to mysql_real_data_home.
#wsrep_data_home_dir=

# Maximum number of rows in write set
wsrep_max_ws_rows=131072

# Maximum size of write set
wsrep_max_ws_size=1073741824

# to enable debug level logging, set this to 1
wsrep_debug=0

# convert locking sessions into transactions
wsrep_convert_LOCK_to_trx=0

# how many times to retry deadlocked autocommits
wsrep_retry_autocommit=1

# change auto_increment_increment and auto_increment_offset automatically
wsrep_auto_increment_control=1

# replicate myisam
wsrep_replicate_myisam=1
# retry autoinc insert, which failed for duplicate key error
wsrep_drupal_282555_workaround=0

# enable "strictly synchronous" semantics for read operations
wsrep_causal_reads=0

# Command to call when node status or cluster membership changes.
# Will be passed all or some of the following options:
# --status  - new status of this node
# --uuid    - UUID of the cluster
# --primary - whether the component is primary or not ("yes"/"no")
# --members - comma-separated list of members
# --index   - index of this node in the list
#wsrep_notify_cmd=

##
## WSREP State Transfer options
##

# State Snapshot Transfer method
# ClusterControl currently DOES NOT support wsrep_sst_method=mysqldump
wsrep_sst_method=rsync

# Address on THIS node to receive SST at. DON'T SET IT TO DONOR ADDRESS!!!
# (SST method dependent. Defaults to the first IP of the first interface)
#wsrep_sst_receive_address=

# SST authentication string. This will be used to send SST to joining nodes.
# Depends on SST method. For mysqldump method it is root:<root password>
wsrep_sst_auth=root:*

# Desired SST donor name.
#wsrep_sst_donor=

# Protocol version to use
# wsrep_protocol_version=
[MYSQL]
socket=/var/lib/mysql/mysql.sock
[client]
socket=/var/lib/mysql/mysql.sock
[mysqldump]
max-allowed-packet = 512M
[MYSQLD_SAFE]
wsrep_urls=gcomm://node3:4567,gcomm://node2:4567,gcomm://node1:4567
pid-file=mysqld.pid
log-error=error.log
basedir=/usr/
datadir=/var/lib/mysql

-- 
--
Haris Zukanovic

[Alex Yurchenko]

On 2012-07-11 16:02, Haris Zukanovic wrote:
> Hi,
>
> Are these Notes in log indication of connectivity problems?

Yes. Notably it seems like it is particularly node3 that has these
issues.

> I am running 3 nodes cluster in a WAN environment. Connectivity
> should be good, but of course it is far away from Gigabit local
> network.
> Ping response times between servers vary from 9-11ms.

Normal operational latency is not a concern. But there can be several
second hiccups once in a while.

> Maybe I could optimize my configuration? Any advice?

Have you seen this:
http://www.codership.com/wiki/doku.php?id=configuration_tips&#wan_replication
?

--
Alexey Yurchenko,
Codership Oy, www.codership.com
Skype: alexey.yurchenko, Phone: +358-400-516-011