Cant use rsync. How can rsync connect to other nodes without user and password?

[Sebastian Karpp]

I setted up a  galera cluster with 3 nodes. Fresh servers, fresh install (ubuntu server 16.04), fresh galera/mariaDB 10.1.

It runs perfectly from the beginning! WOW!

After a few days, I decided to set up a test-cluster. To play around, learn and so on.

But, if node 1 starts the cluster, I try to bring node 2 to the cluster. Will not work with rsync. There is something like "time out" in the syslog after wsrep starting rsync listening on port 4444. (Before it says "need state transafer...."). If I switch to SST method "mysqldump" ist works fine! But no way for me to get it running with rsync again.

Both servers are in the same network, I can use rsync manualy, scp, ssh, every connection works fine. No firewall installed. No imported data, fresh DB (justs system/mysql DBs with a few data).

Two questions to the community:

1. any hints for me?
2. how will galera use rsync without login to the other server?

Thanks and best,

Sebastian

[Brian :]

Hi Sebastian,

My limited experience is that rsync is problematic for SSTs if its
anything but default settings on all the servers. xtrabackup for SSTs
is much more reliable.

Brians

> --
> You received this message because you are subscribed to the Google Groups
> "codership" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to codership-tea...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Sebastian Karpp]

thanks Brian, but it is default on every node. I need to get it running again with rsync.

[Christopher Bowen]

You say no firewall installed, are you sure about that? Unless you disabled it then you should have iptables running, and blocking it. Typically rsync uses your SSH port, same with scp, so testing that will not necessarily determine the issue, unless you used ""ssh -p 4444" in the command.

I have 3 clusters currently setup and running, all 3 use the default rsync without password and without errors.

[Christopher Bowen]

Also ensure that you have 

wsrep_on=ON

in your conf file under the galera block, it must be set explicitly in 10.1, and that you have ran galera_new_cluster.

On Sunday, April 30, 2017 at 1:48:34 PM UTC-4, Sebastian Karpp wrote:

[suhendra su]

Hi Christopher,

Now i'm using mariadb cluster with 3 nodes but i can't start mariadb for second node after galera_new_cluster first node, what happen for that ?

I check the mariadb status and get info like below picture.

Thanks

Suhendra

[Quentin DESBOIS]

Hi Sebastian,

did you find any solution about this problem ? I'm in the same situation.

[Quentin DESBOIS]

Hi Sebastian.

Did you find any solution ? 

It's been 3 days I tried to set up a galera cluster with MariaDB on 3 freshly installed node. I use Ubuntu 16.04 LTS, MariaDB 10.2 (Stable).

My problem is exactly as your own.

On Sunday, April 30, 2017 at 7:48:34 PM UTC+2, Sebastian Karpp wrote:

[Christopher Bowen]

Did you guys get this working? From the screen shot it appears that maybe the firewall is not setup to allow traffic between the nodes.can you list the iptables rules related to this?

--
You received this message because you are subscribed to a topic in the Google Groups "codership" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/codership-team/_d-EgDBUFBo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to codership-team+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Chris Bowen
Systems Programmer

North Carolina State University
Hunt Library
Raleigh, NC 27695-8206
919.515.6182

While this E-mail is only intended for the addresses listed in the message headers, all electronic mail messages in connection with State business which are sent to or received by this account (ceb...@ncsu.edu) are subject to the NC Public Records Law and may be disclosed to third parties.

[Sebastian Karpp]

so I tried x times ago the installation. I followed exactly this both installation instrucions:

https://www.cyberciti.biz/faq/howto-install-configure-mariadb-galera-master-cluster-ubuntulinux/
https://www.digitalocean.com/community/tutorials/how-to-configure-a-galera-cluster-with-mariadb-10-1-on-ubuntu-16-04-servers

Everything is running, but sync via rsync not possible.

syslog shows the following:

Jun 12 08:32:45 ns3079510 mysqld[1185]: 2017-06-12  8:32:45 112371408574720 [Note] /usr/sbin/mysqld (mysqld 10.1.24-MariaDB-1~xenial) starting as process 1375 ...
Jun 12 08:32:49 ns3079510 mysqld[1185]: WSREP: Recovered position 00000000-0000-0000-0000-000000000000:-1
Jun 12 08:32:49 ns3079510 mysqld[1411]: 2017-06-12  8:32:49 110692474747136 [Note] /usr/sbin/mysqld (mysqld 10.1.24-MariaDB-1~xenial) starting as process 1411 ...
Jun 12 08:32:49 ns3079510 mysqld[1411]: 2017-06-12  8:32:49 110692474747136 [Note] WSREP: Read nil XID from storage engines, skipping position init
Jun 12 08:32:49 ns3079510 mysqld[1411]: 2017-06-12  8:32:49 110692474747136 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so'
Jun 12 08:32:49 ns3079510 mysqld[1411]: 2017-06-12  8:32:49 110692474747136 [Note] WSREP: wsrep_load(): Galera 25.3.20(r3703) by Codership Oy <in...@codership.com> loaded successfully.
Jun 12 08:32:49 ns3079510 mysqld[1411]: 2017-06-12  8:32:49 110692474747136 [Note] WSREP: CRC-32C: using hardware acceleration.
Jun 12 08:32:49 ns3079510 mysqld[1411]: 2017-06-12  8:32:49 110692474747136 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1, safe_to_bootsrap: 1
Jun 12 08:32:49 ns3079510 mysqld[1411]: 2017-06-12  8:32:49 110692474747136 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 172.16.0.51; base_port = 4567; cert.log_conflicts = no; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.recover = no; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ig
Jun 12 08:32:49 ns3079510 mysqld[1411]: 2017-06-12  8:32:49 110692474747136 [Note] WSREP: GCache history reset: old(00000000-0000-0000-0000-000000000000:0) -> new(00000000-0000-0000-0000-000000000000:-1)
Jun 12 08:32:49 ns3079510 mysqld[1411]: 2017-06-12  8:32:49 110692474747136 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1
Jun 12 08:32:49 ns3079510 mysqld[1411]: 2017-06-12  8:32:49 110692474747136 [Note] WSREP: wsrep_sst_grab()
Jun 12 08:32:49 ns3079510 mysqld[1411]: 2017-06-12  8:32:49 110692474747136 [Note] WSREP: Start replication
Jun 12 08:32:49 ns3079510 mysqld[1411]: 2017-06-12  8:32:49 110692474747136 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1
Jun 12 08:32:49 ns3079510 mysqld[1411]: 2017-06-12  8:32:49 110692474747136 [Note] WSREP: protonet asio version 0
Jun 12 08:32:49 ns3079510 mysqld[1411]: 2017-06-12  8:32:49 110692474747136 [Note] WSREP: Using CRC-32C for message checksums.
Jun 12 08:32:49 ns3079510 mysqld[1411]: 2017-06-12  8:32:49 110692474747136 [Note] WSREP: backend: asio
Jun 12 08:32:49 ns3079510 mysqld[1411]: 2017-06-12  8:32:49 110692474747136 [Note] WSREP: gcomm thread scheduling priority set to other:0
Jun 12 08:32:49 ns3079510 mysqld[1411]: 2017-06-12  8:32:49 110692474747136 [Warning] WSREP: access file(/var/lib/mysql//gvwstate.dat) failed(No such file or directory)
Jun 12 08:32:49 ns3079510 mysqld[1411]: 2017-06-12  8:32:49 110692474747136 [Note] WSREP: restore pc from disk failed
Jun 12 08:32:49 ns3079510 mysqld[1411]: 2017-06-12  8:32:49 110692474747136 [Note] WSREP: GMCast version 0
Jun 12 08:32:49 ns3079510 mysqld[1411]: 2017-06-12  8:32:49 110692474747136 [Note] WSREP: (f47d475f, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
Jun 12 08:32:49 ns3079510 mysqld[1411]: 2017-06-12  8:32:49 110692474747136 [Note] WSREP: (f47d475f, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
Jun 12 08:32:49 ns3079510 mysqld[1411]: 2017-06-12  8:32:49 110692474747136 [Note] WSREP: EVS version 0
Jun 12 08:32:49 ns3079510 mysqld[1411]: 2017-06-12  8:32:49 110692474747136 [Note] WSREP: gcomm: connecting to group 'test_cluster', peer '172.16.0.50:,172.16.0.51:'
Jun 12 08:32:49 ns3079510 mysqld[1411]: 2017-06-12  8:32:49 110692474747136 [Note] WSREP: (f47d475f, 'tcp://0.0.0.0:4567') connection established to f47d475f tcp://172.16.0.51:4567
Jun 12 08:32:49 ns3079510 mysqld[1411]: 2017-06-12  8:32:49 110692474747136 [Warning] WSREP: (f47d475f, 'tcp://0.0.0.0:4567') address 'tcp://172.16.0.51:4567' points to own listening address, blacklisting
Jun 12 08:32:49 ns3079510 mysqld[1411]: 2017-06-12  8:32:49 110692474747136 [Note] WSREP: (f47d475f, 'tcp://0.0.0.0:4567') connection established to 4681d2df tcp://172.16.0.50:4567
Jun 12 08:32:49 ns3079510 mysqld[1411]: 2017-06-12  8:32:49 110692474747136 [Note] WSREP: (f47d475f, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:
Jun 12 08:32:50 ns3079510 mysqld[1411]: 2017-06-12  8:32:50 110692474747136 [Note] WSREP: declaring 4681d2df at tcp://172.16.0.50:4567 stable
Jun 12 08:32:50 ns3079510 mysqld[1411]: 2017-06-12  8:32:50 110692474747136 [Note] WSREP: Node 4681d2df state prim
Jun 12 08:32:50 ns3079510 mysqld[1411]: 2017-06-12  8:32:50 110692474747136 [Note] WSREP: view(view_id(PRIM,4681d2df,4) memb {
Jun 12 08:32:50 ns3079510 mysqld[1411]: #0114681d2df,0
Jun 12 08:32:50 ns3079510 mysqld[1411]: #011f47d475f,0
Jun 12 08:32:50 ns3079510 mysqld[1411]: } joined {
Jun 12 08:32:50 ns3079510 mysqld[1411]: } left {
Jun 12 08:32:50 ns3079510 mysqld[1411]: } partitioned {
Jun 12 08:32:50 ns3079510 mysqld[1411]: })
Jun 12 08:32:50 ns3079510 mysqld[1411]: 2017-06-12  8:32:50 110692474747136 [Note] WSREP: save pc into disk
Jun 12 08:32:50 ns3079510 mysqld[1411]: 2017-06-12  8:32:50 110692474747136 [Note] WSREP: gcomm: connected
Jun 12 08:32:50 ns3079510 mysqld[1411]: 2017-06-12  8:32:50 110692474747136 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
Jun 12 08:32:50 ns3079510 mysqld[1411]: 2017-06-12  8:32:50 110692474747136 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
Jun 12 08:32:50 ns3079510 mysqld[1411]: 2017-06-12  8:32:50 110692474747136 [Note] WSREP: Opened channel 'test_cluster'
Jun 12 08:32:50 ns3079510 mysqld[1411]: 2017-06-12  8:32:50 110692474747136 [Note] WSREP: Waiting for SST to complete.
Jun 12 08:32:50 ns3079510 mysqld[1411]: 2017-06-12  8:32:50 110692118591232 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 2
Jun 12 08:32:50 ns3079510 mysqld[1411]: 2017-06-12  8:32:50 110692118591232 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
Jun 12 08:32:50 ns3079510 mysqld[1411]: 2017-06-12  8:32:50 110692118591232 [Note] WSREP: STATE EXCHANGE: sent state msg: f4cdb049-4f38-11e7-8302-02b7fe411841
Jun 12 08:32:50 ns3079510 mysqld[1411]: 2017-06-12  8:32:50 110692118591232 [Note] WSREP: STATE EXCHANGE: got state msg: f4cdb049-4f38-11e7-8302-02b7fe411841 from 0 (mdb01)
Jun 12 08:32:50 ns3079510 mysqld[1411]: 2017-06-12  8:32:50 110692118591232 [Note] WSREP: STATE EXCHANGE: got state msg: f4cdb049-4f38-11e7-8302-02b7fe411841 from 1 (mdb02)
Jun 12 08:32:50 ns3079510 mysqld[1411]: 2017-06-12  8:32:50 110692118591232 [Note] WSREP: Quorum results:
Jun 12 08:32:50 ns3079510 mysqld[1411]: #011version    = 4,
Jun 12 08:32:50 ns3079510 mysqld[1411]: #011component  = PRIMARY,
Jun 12 08:32:50 ns3079510 mysqld[1411]: #011conf_id    = 3,
Jun 12 08:32:50 ns3079510 mysqld[1411]: #011members    = 1/2 (joined/total),
Jun 12 08:32:50 ns3079510 mysqld[1411]: #011act_id     = 0,
Jun 12 08:32:50 ns3079510 mysqld[1411]: #011last_appl. = -1,
Jun 12 08:32:50 ns3079510 mysqld[1411]: #011protocols  = 0/7/3 (gcs/repl/appl),
Jun 12 08:32:50 ns3079510 mysqld[1411]: #011group UUID = c2482967-4f36-11e7-bdbb-1f53966b2ed7
Jun 12 08:32:50 ns3079510 mysqld[1411]: 2017-06-12  8:32:50 110692118591232 [Note] WSREP: Flow-control interval: [23, 23]
Jun 12 08:32:50 ns3079510 mysqld[1411]: 2017-06-12  8:32:50 110692118591232 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 0)
Jun 12 08:32:50 ns3079510 mysqld[1411]: 2017-06-12  8:32:50 110692422470400 [Note] WSREP: State transfer required:
Jun 12 08:32:50 ns3079510 mysqld[1411]: #011Group state: c2482967-4f36-11e7-bdbb-1f53966b2ed7:0
Jun 12 08:32:50 ns3079510 mysqld[1411]: #011Local state: 00000000-0000-0000-0000-000000000000:-1
Jun 12 08:32:50 ns3079510 mysqld[1411]: 2017-06-12  8:32:50 110692422470400 [Note] WSREP: New cluster view: global state: c2482967-4f36-11e7-bdbb-1f53966b2ed7:0, view# 4: Primary, number of nodes: 2, my index: 1, protocol version 3
Jun 12 08:32:50 ns3079510 mysqld[1411]: 2017-06-12  8:32:50 110692422470400 [Warning] WSREP: Gap in state sequence. Need state transfer.
Jun 12 08:32:50 ns3079510 mysqld[1411]: 2017-06-12  8:32:50 110692090525440 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'joiner' --address '172.16.0.51' --datadir '/var/lib/mysql/'   --parent '1411' --binlog '/var/log/mysql/mariadb-bin' '
Jun 12 08:32:51 ns3079510 rsyncd[1452]: rsyncd version 3.1.1 starting, listening on port 4444
Jun 12 08:32:52 ns3079510 mysqld[1411]: 2017-06-12  8:32:52 110692127508224 [Note] WSREP: (f47d475f, 'tcp://0.0.0.0:4567') connection to peer f47d475f with addr tcp://172.16.0.51:4567 timed out, no messages seen in PT3S
Jun 12 08:32:53 ns3079510 mysqld[1411]: 2017-06-12  8:32:53 110692127508224 [Note] WSREP: (f47d475f, 'tcp://0.0.0.0:4567') turning message relay requesting off

I tested again with installing ufw and a second try to uninstall iptables. But the same results.

Node 1 shows:
# mysql -u root -p -e "SHOW STATUS LIKE 'wsrep_cluster_size'"
Enter password: xxxx
+--------------------+-------+
| Variable_name      | Value |
+--------------------+-------+
| wsrep_cluster_size | 2     |
+--------------------+-------+

Node 2:
# mysql -u root -p -e "SHOW STATUS LIKE 'wsrep_cluster_size'"
Enter password: xxx
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111 "Connection refused")

Would be great to get a hint to get it running!

[Lammert Bies]

Despite uninstalling iptables It still feels like a firewalling issue to me. Uninstalling iptables only removes the iptables client from the OS. The table filtering capability stays in place because it is implemented in the kernel directly and it could be used now by other firewall services like firewalld.

You may want to test with telnet ip-address 4567 between the servers if you can reach the listening service on that port. If that fails, either the service to listen to port 4567 is not started properly, or packets are firewalled in the process.

[Christopher Bowen]

Or download nmap and run it against the other servers and see what ports it sees as open.

On Mon, Jun 12, 2017 at 8:04 AM, Lammert Bies <umb...@gmail.com> wrote:

Despite uninstalling iptables It still feels like a firewalling issue to me. Uninstalling iptables only removes the iptables client from the OS. The table filtering capability stays in place because it is implemented in the kernel directly and it could be used now by other firewall services like firewalld.

You may want to test with telnet ip-address 4567 between the servers if you can reach the listening service on that port. If that fails, either the service to listen to port 4567 is not started properly, or packets are firewalled in the process.

--

You received this message because you are subscribed to a topic in the Google Groups "codership" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/codership-team/_d-EgDBUFBo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to codership-team+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Sebastian Karpp]

thanks!


telnet:
root@ns3079510:~# telnet 172.16.0.50 4567
Trying 172.16.0.50...
Connected to 172.16.0.50.
Escape character is '^]'.
$▒▒r▒▒▒!▒P▒▒▒▒!MFo▒▒BP籢▒`tb▒▒Connection closed by foreign host.

nmap:
Scanning mdb01 (172.16.0.50) [1000 ports]
Discovered open port 3306/tcp on 172.16.0.50
Discovered open port 4567/tcp on 172.16.0.50
Completed SYN Stealth Scan at 10:34, 9.13s elapsed (1000 total ports)
Nmap scan report for mdb01 (172.16.0.50)
Host is up (0.00016s latency).
Not shown: 997 filtered ports
PORT     STATE  SERVICE
3306/tcp open   mysql
4444/tcp closed krb524
4567/tcp open   tram
MAC Address: A4:BF:01:1E:40:0B (Unknown)

.