WSREP_SST: [ERROR] rsync daemon port '4444' has been taken (20170908 12:46:33.N)

[Roland Giesler GTS]

mysql  Ver 15.1 Distrib 10.2.7-MariaDB, for FreeBSD11.0 (amd64) using readline 5.1

The error occurs on the 2nd node (as has been reported by various others in posts here and here (a bugfix that was closed) and here (how to troubleshoot SST errors)

I found the source script that generates this error here: /usr/local/bin/wsrep_sst_rsync and the code that is relevant is below:

    case $OS in
    FreeBSD)
        local port_info="$(sockstat -46lp ${rsync_port} 2>/dev/null | \
            grep ":${rsync_port}")"
        local is_rsync="$(echo $port_info | \
            grep -w '[[:space:]]\+rsync[[:space:]]\+'"$rsync_pid" 2>/dev/null)"
        ;;
    *)
        if ! which lsof > /dev/null; then
          wsrep_log_error "lsof tool not found in PATH! Make sure you have it installed."
          exit 2 # ENOENT
        fi

        local port_info="$(lsof -i :$rsync_port -Pn 2>/dev/null | \
            grep "(LISTEN)")"
        local is_rsync="$(echo $port_info | \
            grep -w '^rsync[[:space:]]\+'"$rsync_pid" 2>/dev/null)"
        ;;
    esac

    local is_listening_all="$(echo $port_info | \
        grep "*:$rsync_port" 2>/dev/null)"
    local is_listening_addr="$(echo $port_info | \
        grep "$rsync_addr:$rsync_port" 2>/dev/null)"

    if [ ! -z "$is_listening_all" -o ! -z "$is_listening_addr" ]; then
        if [ -z "$is_rsync" ]; then
            wsrep_log_error "rsync daemon port '$rsync_port' has been taken"
            exit 16 # EBUSY
        fi
    fi

Now, if I do:

# sockstat -46lp 4444
USER     COMMAND    PID   FD PROTO  LOCAL ADDRESS         FOREIGN ADDRESS

the port is clearly not in use.  Netstat also doesn't show port 4444 in use.

# netstat -an
Active Internet connections (including servers)
Proto Recv-Q Send-Q Local Address          Foreign Address        (state)
tcp4       0     36 172.16.15.21.22        192.168.120.224.44212  ESTABLISHED
tcp4       0      0 *.22                   *.*                    LISTEN
udp4       0      0 127.0.0.1.123          *.*                    
udp6       0      0 fe80::1%lo0.123        *.*                    
udp6       0      0 ::1.123                *.*                    
udp4       0      0 172.16.15.21.123       *.*                    
udp4       0      0 *.123                  *.*                    
udp6       0      0 *.123                  *.*                    
udp4       0      0 *.514                  *.*                    
udp6       0      0 *.514                  *.*                    
Active UNIX domain sockets
Address          Type   Recv-Q Send-Q            Inode             Conn             Refs          Nextref Addr
fffff8001a6b0870 stream      0      0                0 fffff8001a6b0960                0                0
fffff8001a6b0960 stream      0      0                0 fffff8001a6b0870                0                0
fffff8001a6b0b40 stream      0      0 fffff8001a6f8588                0                0                0 /var/run/devd.pipe
fffff8001a6b05a0 dgram       0      0                0 fffff8001a6b5780                0 fffff8001a6b5690
fffff8001a6b5690 dgram       0      0                0 fffff8001a6b5780                0                0
fffff8001a6b5780 dgram       0      0 fffff8001a664ce8                0 fffff8001a6b05a0                0 /var/run/logpriv
fffff8001a6b5870 dgram       0      0 fffff8003acaa000                0                0                0 /var/run/log
fffff8001a6b0a50 seqpac      0      0 fffff8001a6f83b0                0                0                0 /var/run/devd.seqpacket.pipe

My scripting experience and knowledge is not enough to spot the error the code.

Can comeone help maybe?

thanks

Roland

[Roland Giesler]

On Fri, Sep 8, 2017 at 3:56 PM, Markus Ueberall <uebe...@projektzentrisch.de> wrote:

>
> I found the source script that generates this error here:
> /usr/local/bin/wsrep_sst_rsync and the code that is relevant is below:

> [...]

Just to be sure, could you modify the script so that the contents of all
variables in question are logged (adding "set>/tmp/wsrep_sst_rsync.log"
right in front of the call to "wsrep_log_error" should be sufficient)
and provide us with the filtered/sanitised output?

(not sure if I should remove anything from this output...)

 

(This, e.g., will show whether $OS really is "FreeBSD" as expected.)

# cat /tmp/wsrep_sst_rsync.log
ADDR=172.16.15.21:4444
BINLOG_DIRNAME=.
BINLOG_FILENAME=mysql-bin
BINLOG_N_FILES=1
BINLOG_TAR_FILE=/var/db/mysql//wsrep_sst_binlog.tar
CLIENT_DIR=/usr/local/bin/../client
EXTRA_DIR=/usr/local/bin/../extra
FILTER=$'-f \'- /lost+found\' -f \'- /.fseventsd\' -f \'- /.Trashes\'
        -f \'+ /wsrep_sst_binlog.tar\' -f \'+ /ib_lru_dump\' -f \'+ /ibdata*\' -f \'+ /*/\' -f \'- /*\''
HOME=/
IFS=$' \t
'
MAGIC_FILE=/var/db/mysql//rsync_sst_complete
MODULE=rsync_sst
MYSQLDUMP=/usr/local/bin/mysqldump
MYSQLD_PID=4478
MYSQL_CLIENT=/usr/local/bin/mysql
MYSQL_HOME=/usr/local
MY_PRINT_DEFAULTS='/usr/local/bin/my_print_defaults  --defaults-extra-file=/var/db/mysql/my.cnf'
OPTIND=1
OS=FreeBSD
PATH=/usr/local/sbin:/usr/local/bin:/usr/local//bin:/sbin:/bin:/usr/sbin:/usr/bin
PPID=4478
PS1='$ '
PS2='> '
PS4='+ '
PWD=/var/db/mysql
RC_PID=4217
RSYNC_ADDR=172.16.15.21
RSYNC_CONF=/var/db/mysql//rsync_sst.conf
RSYNC_PID=/var/db/mysql//rsync_sst.pid
RSYNC_PORT=4444
RSYNC_REAL_PID=4517
SCRIPTS_DIR=/usr/local/bin
SHLVL=1
SILENT=''
SST_PROGRESS_FILE=/var/db/mysql//sst_in_progress
WSREP_LOG_DIR=/var/db/mysql
WSREP_SST_OPT_ADDR=172.16.15.21
WSREP_SST_OPT_AUTH=root:
WSREP_SST_OPT_BINLOG=mysql-bin
WSREP_SST_OPT_BYPASS=0
WSREP_SST_OPT_CONF=' --defaults-extra-file=/var/db/mysql/my.cnf'
WSREP_SST_OPT_CONF_SUFFIX=''
WSREP_SST_OPT_DATA=/var/db/mysql/
WSREP_SST_OPT_DEFAULT=''
WSREP_SST_OPT_EXTRA_DEFAULT='--defaults-extra-file=/var/db/mysql/my.cnf'
WSREP_SST_OPT_PARENT=4478
WSREP_SST_OPT_PSWD=''
WSREP_SST_OPT_ROLE=joiner
WSREP_SST_OPT_USER=root
_=/bin/sh
is_listening_addr='mysql rsync 4517 5 tcp4 172.16.15.21:4444 *:*'
is_listening_all=''
is_rsync=''
pid_file=/var/db/mysql//rsync_sst.pid
port_info='mysql    rsync      4517  5  tcp4   172.16.15.21:4444     *:*'
rsync_addr=172.16.15.21
rsync_pid=4517
rsync_port=4444

So it seems rsync is running at the time it reports that it's not able to use port 4444 because something is already running on that port??

thanks again

Roland

[Roland Giesler GTS]

Just to add:

# pkg info galera
galera-25.3.21
Name           : galera
Version        : 25.3.21
Installed on   : Mon Aug 21 19:48:37 2017 SAST
Origin         : databases/galera
Architecture   : FreeBSD:11:amd64
Prefix         : /usr/local
Categories     : databases
Licenses       : GPLv2
Maintainer     : de...@galeracluster.com
WWW            : http://galeracluster.com
Comment        : Synchronous multi-master replication engine
Options        :
    BOOSTPOOL      : off
    BPOSTATIC      : off
    DEBUG          : off
Shared Libs required:
    libboost_system.so.1.64.0
    libboost_program_options.so.1.64.0
Annotations    :
    repo_type      : binary
    repository     : FreeBSD
Flat size      : 3.69MiB
Description    :
Galera wsrep provider library for Galera Cluster for MySQL, an easy-to-use
high-availability solution with high system up-time, no data loss,
and scalability for future growth.

This seems to be the latest released version

[Markus Ueberall]

(I'm using the browser based Google Groups UI for this because neither my initial post nor your below answer landed in my mailbox.)

After looking at the variable contents and playing with a test script consisting of only of the relevant fragments, I conclude that the problem is somehow related to the way $is_rsync is initialised. When I execute the command in question on the command line, the output is as expected–however, when executing the same command in a subshell the 'grep part' will fail.

As a quick fix, please replace the following lines in the wsrep_sst_rsync script reading

local is_rsync="$(echo $port_info | \ 

    grep -w '[[:space:]]\+rsync[[:space:]]\+'"$rsync_pid" 2>/dev/null)"

with

local is_rsync="$(echo "$port_info" | grep -w rsync.*"$rsync_pid" 2>/dev/null)"

and rerun your test. (This is not exactly the same, but a good enough regular expression which should not lead to false positives.)

As a matter of fact, the original code also doesn't work for me, so https://jira.mariadb.org/browse/MDEV-9903 should probably be reopened and the codership team should be notified separately (from MariaDB point of view, SST related problems should be considered "upstream") ASAP, but I don't have the time to properly debug this atm (see above). Therefore, I am just reusing my own local workaround which removes this entire test as I know that the port in question will always be claimed by rsync on all nodes. Without a proper fix, however, every package update that overwrites the modified script will break the synchronisation.

Kind regards, Markus

Am Freitag, 8. September 2017 18:03:56 UTC+2 schrieb Roland Giesler GTS:

On Fri, Sep 8, 2017 at 3:56 PM, Markus Ueberall <uebe...@projektzentrisch.de> wrote:

>
> I found the source script that generates this error here:
> /usr/local/bin/wsrep_sst_rsync and the code that is relevant is below:
> [...]

Just to be sure, could you modify the script so that the contents of all
variables in question are logged (adding "set>/tmp/wsrep_sst_rsync.log"
right in front of the call to "wsrep_log_error" should be sufficient)
and provide us with the filtered/sanitised output?

(not sure if I should remove anything from this output...)

 

(This, e.g., will show whether $OS really is "FreeBSD" as expected.)

[...]

# cat /tmp/wsrep_sst_rsync.log

is_listening_addr='mysql rsync 4517 5 tcp4 172.16.15.21:4444 *:*'
is_listening_all=''
is_rsync=''
pid_file=/var/db/mysql//rsync_sst.pid
port_info='mysql    rsync      4517  5  tcp4   172.16.15.21:4444     *:*'
rsync_addr=172.16.15.21
rsync_pid=4517
rsync_port=4444

[...]

[Roland Giesler]

On Fri, Sep 8, 2017 at 8:02 PM, 'Markus Ueberall' via codership <codersh...@googlegroups.com> wrote:

(I'm using the browser based Google Groups UI for this because neither my initial post nor your below answer landed in my mailbox.)

After looking at the variable contents and playing with a test script consisting of only of the relevant fragments, I conclude that the problem is somehow related to the way $is_rsync is initialised. When I execute the command in question on the command line, the output is as expected–however, when executing the same command in a subshell the 'grep part' will fail.

As a quick fix, please replace the following lines in the wsrep_sst_rsync script reading

local is_rsync="$(echo $port_info | \ 

    grep -w '[[:space:]]\+rsync[[:space:]]\+'"$rsync_pid" 2>/dev/null)"

with

local is_rsync="$(echo "$port_info" | grep -w rsync.*"$rsync_pid" 2>/dev/null)"

and rerun your test. (This is not exactly the same, but a good enough regular expression which should not lead to false positives.)

I made that change and it prevented the false positive.

However, how I get a new error:

2017-09-08 22:24:39 34820008448 [Note] WSREP: Prepared SST request: rsync|172.16.15.21:4444/rsync_sst
2017-09-08 22:24:39 34820008448 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2017-09-08 22:24:39 34820008448 [Note] WSREP: REPL Protocols: 7 (3, 2)
2017-09-08 22:24:39 34820008448 [Note] WSREP: Assign initial position for certification: 0, protocol version: 3
2017-09-08 22:24:39 34424858112 [Note] WSREP: Service thread queue flushed.
2017-09-08 22:24:39 34820008448 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (2749a781-9248-11e7-83a2-0262448701d2): 1 (Operation not permitted)
         at galera/src/replicator_str.cpp:prepare_for_IST():482. IST will be unavailable.
2017-09-08 22:24:39 34820005888 [Note] WSREP: Member 0.0 (hout3) requested state transfer from '*any*'. Selected 1.0 (hout2)(SYNCED) as donor.
2017-09-08 22:24:39 34820005888 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 0)
2017-09-08 22:24:39 34820008448 [Note] WSREP: Requesting state transfer: success, donor: 1
2017-09-08 22:24:39 34820008448 [Note] WSREP: GCache history reset: 00000000-0000-0000-0000-000000000000:0 -> 2749a781-9248-11e7-83a2-0262448701d2:0
2017-09-08 22:24:42 34424859392 [Note] WSREP: (bd2fcb2e, 'tcp://0.0.0.0:4567') turning message relay requesting off
2017-09-08 22:25:14 34820005888 [Warning] WSREP: 1.0 (hout2): State transfer to 0.0 (hout3) failed: -255 (Unknown error: 255)
2017-09-08 22:25:14 34820005888 [ERROR] WSREP: gcs/src/gcs_group.cpp:gcs_group_handle_join_msg():736: Will never receive state. Need to abort.
2017-09-08 22:25:14 34820005888 [Note] WSREP: gcomm: terminating thread
2017-09-08 22:25:14 34820005888 [Note] WSREP: gcomm: joining thread
2017-09-08 22:25:14 34820005888 [Note] WSREP: gcomm: closing backend

[Markus Ueberall]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On 08.09.2017 13:15, Roland Giesler GTS wrote:
> mysql Ver 15.1 Distrib 10.2.7-MariaDB, for FreeBSD11.0 (amd64) using
> readline 5.1
>

> The error occurs on the 2nd node [...]

>
> I found the source script that generates this error here:
> /usr/local/bin/wsrep_sst_rsync and the code that is relevant is below:

> [...]

Just to be sure, could you modify the script so that the contents of all
variables in question are logged (adding "set>/tmp/wsrep_sst_rsync.log"
right in front of the call to "wsrep_log_error" should be sufficient)
and provide us with the filtered/sanitised output?

(This, e.g., will show whether $OS really is "FreeBSD" as expected.)

> Can comeone help maybe?
> thanks
> Roland

KR, Markus
-----BEGIN PGP SIGNATURE-----

iF4EARYKAAYFAlmyoaEACgkQ4ZPOaz1evUw8BgEA3ZDO/bTQVBEG+TkEVXlk7W3G
VNkQAoAbPnAonOKux7cBANAnFJVnurNei/seMsq+0VUaLemWveOEGZHTw4gsZ50N
=+SaZ
-----END PGP SIGNATURE-----

[Markus Ueberall]

On 08.09.2017 22:35, Roland Giesler wrote:

[N]ow I get a new error:

[...]

2017-09-08 22:24:39 34820008448 [Note] WSREP: Prepared SST request: rsync|172.16.15.21:4444/rsync_sst
2017-09-08 22:24:39 34820008448 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2017-09-08 22:24:39 34820008448 [Note] WSREP: REPL Protocols: 7 (3, 2)
2017-09-08 22:24:39 34820008448 [Note] WSREP: Assign initial position for certification: 0, protocol version: 3
2017-09-08 22:24:39 34424858112 [Note] WSREP: Service thread queue flushed.
2017-09-08 22:24:39 34820008448 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (2749a781-9248-11e7-83a2-0262448701d2): 1 (Operation not permitted)
         at galera/src/replicator_str.cpp:prepare_for_IST():482. IST will be unavailable.
2017-09-08 22:24:39 34820005888 [Note] WSREP: Member 0.0 (hout3) requested state transfer from '*any*'. Selected 1.0 (hout2)(SYNCED) as donor.
2017-09-08 22:24:39 34820005888 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 0)
2017-09-08 22:24:39 34820008448 [Note] WSREP: Requesting state transfer: success, donor: 1
2017-09-08 22:24:39 34820008448 [Note] WSREP: GCache history reset: 00000000-0000-0000-0000-000000000000:0 -> 2749a781-9248-11e7-83a2-0262448701d2:0
2017-09-08 22:24:42 34424859392 [Note] WSREP: (bd2fcb2e, 'tcp://0.0.0.0:4567') turning message relay requesting off
2017-09-08 22:25:14 34820005888 [Warning] WSREP: 1.0 (hout2): State transfer to 0.0 (hout3) failed: -255 (Unknown error: 255)
2017-09-08 22:25:14 34820005888 [ERROR] WSREP: gcs/src/gcs_group.cpp:gcs_group_handle_join_msg():736: Will never receive state. Need to abort.
2017-09-08 22:25:14 34820005888 [Note] WSREP: gcomm: terminating thread
2017-09-08 22:25:14 34820005888 [Note] WSREP: gcomm: joining thread
2017-09-08 22:25:14 34820005888 [Note] WSREP: gcomm: closing backend

The "standard" questions in this case:
    (a) What do you see in the logs of the other nodes (e.g., hout2, the selected donor)?
    (b) Have you checked that all required ports can be opened on all nodes (aside from port 4444 (SST), the nodes need 4567 (Cluster), 4568 (IST) as specified in the documentation)?

KR, Markus

[Roland Giesler]

There's no firewall between any of these machines, they're on their own network segment.

I'll check the hout2 logs tonight.

regards,

Roland
(sent from my phone)

--
You received this message because you are subscribed to the Google Groups "codership" group.
To unsubscribe from this group and stop receiving emails from it, send an email to codership-team+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Roland Giesler]

On Sun, Sep 10, 2017 at 3:25 PM, Roland Giesler <rol...@greentree.systems> wrote:

There's no firewall between any of these machines, they're on their own network segment.

I'll check the hout2 logs tonight.

Here's the hout2.err log:

WSREP_SST: [INFO] Preparing binlog files for transfer: (20170910 21:04:51.N)
a mysql-bin.000013
2017-09-10 21:04:53 34424859392 [Note] WSREP: (f698906e, 'tcp://0.0.0.0:4567') turning message relay requesting off
rsync: change_dir#1 "/test" (in rsync_sst) failed: Permission denied (13)
rsync error: errors selecting input/output files, dirs (code 3) at main.c(620) [Receiver=3.1.2]
rsync: change_dir#1 "/mysql" (in rsync_sst) failed: Permission denied (13)
rsync error: errors selecting input/output files, dirs (code 3) at main.c(620) [Receiver=3.1.2]
WSREP_SST: [ERROR] find/rsync returned code 1: (20170910 21:05:03.N)
2017-09-10 21:05:03 34825450496 [ERROR] WSREP: Failed to read from: wsrep_sst_rsync --role 'donor' --address '172.16.15.21:4444/rsync_sst' --socket '/tmp/mysql.sock' --datadir '/var/db/mysql/'  --defaults-extra-file '/var/db/mysql/my.cnf'   --binlog 'mysql-bin' --gtid '2749a781-9248-11e7-83a2-0262448701d2:0' --gtid-domain-id '0'
2017-09-10 21:05:03 34825450496 [ERROR] WSREP: Process completed with error: wsrep_sst_rsync --role 'donor' --address '172.16.15.21:4444/rsync_sst' --socket '/tmp/mysql.sock' --datadir '/var/db/mysql/'  --defaults-extra-file '/var/db/mysql/my.cnf'   --binlog 'mysql-bin' --gtid '2749a781-9248-11e7-83a2-0262448701d2:0' --gtid-domain-id '0': 255 (Unknown error: 255)
2017-09-10 21:05:03 34825450496 [Note] WSREP: resuming provider at 21
2017-09-10 21:05:03 34825450496 [Note] WSREP: Provider resumed.
2017-09-10 21:05:03 34825450496 [ERROR] WSREP: Command did not run: wsrep_sst_rsync --role 'donor' --address '172.16.15.21:4444/rsync_sst' --socket '/tmp/mysql.sock' --datadir '/var/db/mysql/'  --defaults-extra-file '/var/db/mysql/my.cnf'   --binlog 'mysql-bin' --gtid '2749a781-9248-11e7-83a2-0262448701d2:0' --gtid-domain-id '0'
2017-09-10 21:05:03 34820005888 [Warning] WSREP: 1.0 (hout2): State transfer to 0.0 (hout3) failed: -255 (Unknown error: 255)
2017-09-10 21:05:03 34820005888 [Note] WSREP: Shifting DONOR/DESYNCED -> JOINED (TO: 0)
2017-09-10 21:05:03 34820005888 [Note] WSREP: Member 1.0 (hout2) synced with group.
2017-09-10 21:05:03 34820005888 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 0)
2017-09-10 21:05:03 34820008448 [Note] WSREP: Synchronized with group, ready for connections
2017-09-10 21:05:03 34820008448 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2017-09-10 21:05:04 34424859392 [Note] WSREP: forgetting ed2551f2 (tcp://172.16.15.21:4567)
2017-09-10 21:05:04 34424859392 [Note] WSREP: Node f698906e state prim
2017-09-10 21:05:04 34424859392 [Note] WSREP: view(view_id(PRIM,f698906e,11) memb {
        f698906e,0
} joined {
} left {
} partitioned {
        ed2551f2,0
})
2017-09-10 21:05:04 34424859392 [Note] WSREP: save pc into disk
2017-09-10 21:05:04 34424859392 [Note] WSREP: forgetting ed2551f2 (tcp://172.16.15.21:4567)
2017-09-10 21:05:04 34820005888 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 1
2017-09-10 21:05:04 34820005888 [Note] WSREP: STATE_EXCHANGE: sent state UUID: f4050336-965a-11e7-8750-9364d431c517
2017-09-10 21:05:04 34820005888 [Note] WSREP: STATE EXCHANGE: sent state msg: f4050336-965a-11e7-8750-9364d431c517
2017-09-10 21:05:04 34820005888 [Note] WSREP: STATE EXCHANGE: got state msg: f4050336-965a-11e7-8750-9364d431c517 from 0 (hout2)
2017-09-10 21:05:04 34820005888 [Note] WSREP: Quorum results:
        version    = 4,
        component  = PRIMARY,
        conf_id    = 10,
        members    = 1/1 (joined/total),
        act_id     = 0,
        last_appl. = 0,
        protocols  = 0/7/3 (gcs/repl/appl),
        group UUID = 2749a781-9248-11e7-83a2-0262448701d2
2017-09-10 21:05:04 34820005888 [Note] WSREP: Flow-control interval: [16, 16]
2017-09-10 21:05:04 34820005888 [Note] WSREP: Trying to continue unpaused monitor
2017-09-10 21:05:04 34820008448 [Note] WSREP: New cluster view: global state: 2749a781-9248-11e7-83a2-0262448701d2:0, view# 11: Primary, number of nodes: 1, my index: 0, protocol version 3
2017-09-10 21:05:04 34820008448 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2017-09-10 21:05:04 34820008448 [Note] WSREP: REPL Protocols: 7 (3, 2)
2017-09-10 21:05:04 34820008448 [Note] WSREP: Assign initial position for certification: 0, protocol version: 3
2017-09-10 21:05:04 34424858112 [Note] WSREP: Service thread queue flushed.
2017-09-10 21:05:08 34424859392 [Note] WSREP: (f698906e, 'tcp://0.0.0.0:4567') connection established to ed2551f2 tcp://172.16.15.21:4567
2017-09-10 21:05:08 34424859392 [Warning] WSREP: discarding established (time wait) ed2551f2 (tcp://172.16.15.21:4567)

I cannot, however, find any reference to /test the config files.  Where does this come from?

thanks again

Roland

 

regards,

Roland
(sent from my phone)

[Roland Giesler]

On Sun, Sep 10, 2017 at 9:32 PM, Roland Giesler <rol...@greentree.systems> wrote:

<snip>

I cannot, however, find any reference to /test the config files.  Where does this come from?

Ah!

MariaDB [(none)]> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |
| test               |
+--------------------+

So these are databases rsync is attempting to transfer.  Of course, creating /test and /mysql would require root permissions, so where to I specify that these directories should be creates somewhere where rsync has permission?

Then I found this:https://www.percona.com/forums/questions-discussions/percona-xtradb-cluster/41511-error-join-cluster?p=43398#post43398

So deleted the files as per that post and restarted and now that rsync error is gone.

It only goes from bad to worse though it seems, since now mysql core dumps on hout3!

# cat /var/log/messages

Sep 10 21:44:03 hout3 kernel: pid 12107 (mysqld), uid 88: exited on signal 11 (core dumped)