Error: 113 (No route to host)

[DRivard]

Hi,

I am having a little issue connecting one of my node to the cluster. My cluster is made of 3 nodes:

galera:

  Installed: 23.2.2

mysql-server-wsrep:

  Installed: 5.5.28-23.7

It is running on ubunut 12.04

When I am trying to connect this server to the two other already in sync I am getting in the log this error message:

root@galera-2 ~:tail -f  /var/log/mysql/error.log
130201 11:45:57 [Note] WSREP: Shifting OPEN -> CLOSED (TO: 1024077)
130201 11:45:57 [Note] WSREP: RECV thread exiting 0: Success
130201 11:45:57 [Note] WSREP: recv_thread() joined.
130201 11:45:57 [Note] WSREP: Closing slave action queue.
130201 11:45:57 [Note] WSREP: /usr/sbin/mysqld: Terminated.
Aborted
130201 11:45:57 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
Parent mysqld process (PID:5262) terminated unexpectedly.
WSREP_SST: [INFO] Joiner cleanup. (20130201 11:45:57.442)
 done.
130201 11:59:05 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
130201 11:59:05 mysqld_safe WSREP: Running position recovery with --log_error=/tmp/tmp.ZEus0Rpk1y
130201 11:59:22 mysqld_safe WSREP: Recovered position 005131d0-ecd5-11e1-0800-766c848a56fb:7966782623
130201 11:59:22 [Note] WSREP: wsrep_start_position var submitted: '005131d0-ecd5-11e1-0800-766c848a56fb:7966782623'
130201 11:59:22 [Note] WSREP: Read nil XID from storage engines, skipping position init
130201 11:59:22 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so'
130201 11:59:23 [Note] WSREP: wsrep_load(): Galera 23.2.2(r137) by Codership Oy <info@codership.com> loaded succesfully.
130201 11:59:23 [Note] WSREP: Found saved state: 005131d0-ecd5-11e1-0800-766c848a56fb:7966782623
130201 11:59:23 [Note] WSREP: Reusing existing '/var/lib/mysql//galera.cache'.
130201 11:59:23 [Note] WSREP: Passing config to GCS: base_host = 192.168.25.12; base_port = 4567; cert.log_conflicts = no; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.size = 25G; gcs.fc_debug = 0; gcs.fc_factor = 1; gcs.fc_limit = 16; gcs.fc_master_slave = NO; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = NO; replicator.causal_read_timeout = PT30S; replicator.commit_order = 3
130201 11:59:23 [Note] WSREP: Assign initial position for certification: 7966782623, protocol version: -1
130201 11:59:23 [Note] WSREP: wsrep_sst_grab()
130201 11:59:23 [Note] WSREP: Start replication
130201 11:59:23 [Note] WSREP: Setting initial position to 005131d0-ecd5-11e1-0800-766c848a56fb:7966782623
130201 11:59:23 [Note] WSREP: protonet asio version 0
130201 11:59:23 [Note] WSREP: backend: asio
130201 11:59:23 [Note] WSREP: GMCast version 0
130201 11:59:23 [Note] WSREP: (ba3ded94-6c90-11e2-0800-4854cb5a0717, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
130201 11:59:23 [Note] WSREP: (ba3ded94-6c90-11e2-0800-4854cb5a0717, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
130201 11:59:23 [Note] WSREP: EVS version 0
130201 11:59:23 [Note] WSREP: PC version 0
130201 11:59:23 [Note] WSREP: gcomm: connecting to group 'datavalet_cluster', peer '192.168.25.13:4567'
130201 11:59:23 [Note] WSREP: (ba3ded94-6c90-11e2-0800-4854cb5a0717, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://192.168.25.11:4567
130201 11:59:23 [Note] WSREP: (ba3ded94-6c90-11e2-0800-4854cb5a0717, 'tcp://0.0.0.0:4567') turning message relay requesting off
130201 11:59:23 [Note] WSREP: declaring 71eb1f63-6c77-11e2-0800-675e02ddc4ef stable
130201 11:59:23 [Note] WSREP: declaring aa225cbd-6c77-11e2-0800-5d6736e20cb4 stable
130201 11:59:23 [Note] WSREP: view(view_id(PRIM,71eb1f63-6c77-11e2-0800-675e02ddc4ef,13) memb {
        71eb1f63-6c77-11e2-0800-675e02ddc4ef,
        aa225cbd-6c77-11e2-0800-5d6736e20cb4,
        ba3ded94-6c90-11e2-0800-4854cb5a0717,
} joined {
} left {
} partitioned {
})
130201 11:59:24 [Note] WSREP: gcomm: connected
130201 11:59:24 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
130201 11:59:24 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
130201 11:59:24 [Note] WSREP: Opened channel 'datavalet_cluster'
130201 11:59:24 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 2, memb_num = 3
130201 11:59:24 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
130201 11:59:24 [Note] WSREP: Waiting for SST to complete.
130201 11:59:24 [Note] WSREP: STATE EXCHANGE: sent state msg: ba3fe37d-6c90-11e2-0800-075f6b8a4876
130201 11:59:24 [Note] WSREP: STATE EXCHANGE: got state msg: ba3fe37d-6c90-11e2-0800-075f6b8a4876 from 0 (galera-3)
130201 11:59:24 [Note] WSREP: STATE EXCHANGE: got state msg: ba3fe37d-6c90-11e2-0800-075f6b8a4876 from 1 (galera-1)
130201 11:59:24 [Note] WSREP: STATE EXCHANGE: got state msg: ba3fe37d-6c90-11e2-0800-075f6b8a4876 from 2 (galera-2)
130201 11:59:24 [Note] WSREP: Quorum results:
        version    = 2,
        component  = PRIMARY,
        conf_id    = 12,
        members    = 2/3 (joined/total),
        act_id     = 2481874,
        last_appl. = -1,
        protocols  = 0/4/2 (gcs/repl/appl),
        group UUID = 71eb7cdf-6c77-11e2-0800-3d6da40f7cdb
130201 11:59:24 [Note] WSREP: Flow-control interval: [28, 28]
130201 11:59:24 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 2481874)
130201 11:59:24 [Note] WSREP: State transfer required:
        Group state: 71eb7cdf-6c77-11e2-0800-3d6da40f7cdb:2481874
        Local state: 005131d0-ecd5-11e1-0800-766c848a56fb:7966782623
130201 11:59:24 [Note] WSREP: New cluster view: global state: 71eb7cdf-6c77-11e2-0800-3d6da40f7cdb:2481874, view# 13: Primary, number of nodes: 3, my index: 2, protocol version 2
130201 11:59:24 [Warning] WSREP: Gap in state sequence. Need state transfer.
130201 11:59:26 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'joiner' --address '192.168.25.12' --auth '' --datadir '/var/lib/mysql/' --defaults-file '/etc/mysql/my.cnf' --parent '6215''
130201 11:59:26 [Note] WSREP: Prepared SST request: rsync|192.168.25.12:4444/rsync_sst
130201 11:59:26 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
130201 11:59:26 [Warning] WSREP: moving position backwards: 7966782623 -> 2481874
130201 11:59:26 [Note] WSREP: Assign initial position for certification: 2481874, protocol version: 2
130201 11:59:26 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (005131d0-ecd5-11e1-0800-766c848a56fb) does not match group state UUID (71eb7cdf-6c77-11e2-0800-3d6da40f7cdb): 1 (Operation not permitted)
         at galera/src/replicator_str.cpp:prepare_for_IST():440. IST will be unavailable.
130201 11:59:26 [ERROR] WSREP: Requesting state transfer failed: -113(No route to host)
130201 11:59:26 [ERROR] WSREP: State transfer request failed unrecoverably: 113 (No route to host). Most likely it is due to inability to communicate with the cluster primary component. Restart required.
130201 11:59:26 [Note] WSREP: Closing send monitor...
130201 11:59:26 [Note] WSREP: Closed send monitor.
130201 11:59:26 [Note] WSREP: gcomm: terminating thread
130201 11:59:26 [Note] WSREP: gcomm: joining thread
130201 11:59:26 [Note] WSREP: gcomm: closing backend
130201 11:59:26 [Note] WSREP: view(view_id(NON_PRIM,71eb1f63-6c77-11e2-0800-675e02ddc4ef,13) memb {
        ba3ded94-6c90-11e2-0800-4854cb5a0717,
} joined {
} left {
} partitioned {
        71eb1f63-6c77-11e2-0800-675e02ddc4ef,
        aa225cbd-6c77-11e2-0800-5d6736e20cb4,
})
130201 11:59:26 [Note] WSREP: view((empty))
130201 11:59:26 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
130201 11:59:26 [Note] WSREP: gcomm: closed
130201 11:59:26 [Note] WSREP: Flow-control interval: [16, 16]
130201 11:59:26 [Note] WSREP: Received NON-PRIMARY.
130201 11:59:26 [Note] WSREP: Shifting PRIMARY -> OPEN (TO: 2481875)
130201 11:59:26 [Note] WSREP: Received self-leave message.
130201 11:59:26 [Note] WSREP: Flow-control interval: [0, 0]
130201 11:59:26 [Note] WSREP: Received SELF-LEAVE. Closing connection.
130201 11:59:26 [Note] WSREP: Shifting OPEN -> CLOSED (TO: 2481875)
130201 11:59:26 [Note] WSREP: RECV thread exiting 0: Success
130201 11:59:26 [Note] WSREP: recv_thread() joined.
130201 11:59:26 [Note] WSREP: Closing slave action queue.
130201 11:59:26 [Note] WSREP: /usr/sbin/mysqld: Terminated.
Aborted
130201 11:59:26 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
Parent mysqld process (PID:6215) terminated unexpectedly.
WSREP_SST: [INFO] Joiner cleanup. (20130201 11:59:26.112)
 done.

I tried:

1 ) telnet 192.168.25.13 4567

that works and replies with 

Escape character is '^]'.

$ÇÉ0qëclwg^ÝÄïq Ålv!Ò;PuTTY

2 ) I can ping and traceroute the 2 other servers

So I don't understand why it tells me that there is no route to host.

3 ) iptables -L is empty I have no firewall rules and they are all on the same switch.

4 ) I can ping each server by their hostname, they are in the /etc/hosts file

Can you please help?

Regards

Dominick

[Alex Yurchenko]

Hi Dominick,

What's in wsrep_sst_donor on the joining node?
What's in wsrep_node_name on the other two?

> Oy <in...@codership.com> loaded succesfully.

> aa225cbd-6c77-11e2-0800-5d6736e20cb4stable

> 130201 11:59:23 [Note] WSREP:

> view(view_id(PRIM,71eb1f63-6c77-11e2-0800-

> 675e02ddc4ef,13) memb {
> 71eb1f63-6c77-11e2-0800-675e02ddc4ef,
> aa225cbd-6c77-11e2-0800-5d6736e20cb4,
> ba3ded94-6c90-11e2-0800-4854cb5a0717,
> } joined {
> } left {
> } partitioned {
> })
> 130201 11:59:24 [Note] WSREP: gcomm: connected
> 130201 11:59:24 [Note] WSREP: Changing maximum packet size to

> 64500,resulting msg size

> 130201 11:59:26 [ERROR] WSREP: *State transfer request failed
> unrecoverably:
> 113 (No route to host**)*. Most likely it is due to inability to

--
Alexey Yurchenko,
Codership Oy, www.codership.com
Skype: alexey.yurchenko, Phone: +358-400-516-011