SST error "nc: Address already in use"

[in...@rentrange.com]

Hello,
I did not set up the installation, and after a crash I am trying to get it working again.
After trying several ways to restart, I keep getting hung up on this issue.  I have looked around for countless hours to figure out what to do next to stop that "nc: Address already in use" error.

Error log output----
........
130619 20:30:59 [Note] WSREP: New cluster view: global state: c0545b8b-60f9-11e2-0800s-d92e4acc6436:187473848, view# 145: Primary, number of nodes: 3, my index: 0, protocol version 2
130619 20:30:59 [Warning] WSREP: Gap in state sequence. Need state transfer.
130619 20:31:01 [Note] WSREP: Running: 'wsrep_sst_xtrabackup --role 'joiner' --address '123.456.789.12' --auth 'sstroot:passgoeshere' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --parent '12261''
nc: Address already in use
tar: This does not look like a tar archive
tar: Exiting with failure status due to previous errors
130619 20:31:01 [Note] WSREP: Prepared SST request: xtrabackup|123.45.678.901:4444/xtrabackup_sst
130619 20:31:01 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
130619 20:31:01 [Note] WSREP: Assign initial position for certification: 187473848, protocol version: 2
130619 20:31:01 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (c0545b8b-60f9-11e2-0800-d92e4acc6436): 1 (Operation not permitted)
         at galera/src/replicator_str.cpp:prepare_for_IST():440. IST will be unavailable.
130619 20:31:01 [Note] WSREP: Node 0 (s1.int) requested state transfer from '*any*'. Selected 1 (s2.int)(SYNCED) as donor.
130619 20:31:01 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 187473860)
130619 20:31:01 [Note] WSREP: Requesting state transfer: success, donor: 1
WSREP_SST: [ERROR] Error while getting st data from donor node:  1, 2 (20130619 20:31:01.819)
130619 20:31:02 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup --role 'joiner' --address '123.456.789.12' --auth 'sstroot:passgoeshere' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --parent '12261': 32 (Broken pipe)
130619 20:31:02 [ERROR] WSREP: Failed to read uuid:seqno from joiner script.
130619 20:31:02 [ERROR] WSREP: SST failed: 32 (Broken pipe)
130619 20:31:02 [ERROR] Aborting

130619 20:31:04 [Note] WSREP: Closing send monitor...
130619 20:31:04 [Note] WSREP: Closed send monitor.
130619 20:31:04 [Note] WSREP: gcomm: terminating thread
130619 20:31:04 [Note] WSREP: gcomm: joining thread
130619 20:31:04 [Note] WSREP: gcomm: closing backend
.......


I found a command to check some connections:
"netstat -anpt | grep 4567"  (This does not return any results for the server that is throwing errors.)

Using this command on the other working servers I get:
tcp        0      0 0.0.0.0:4567                        0.0.0.0:*                          LISTEN             21613/mysqld
tcp        0      0 123.45.678.901:53485         123.45.678.901:4567          ESTABLISHED 21613/mysqld

How do I make my server open these tcp ports?
Please be specific with the command... I am really pulling my hair out as the server has been down for 3 days now...

Also, are there other ports that need to be specifically opened for SST/replication to work?

Thanks for any help!!

[Jay Janssen]

On Jun 19, 2013, at 9:39 PM, in...@rentrange.com wrote:

Also, are there other ports that need to be specifically opened for SST/replication to work?

SST uses port 4444 by default.  This is not opened by the mysqld process directly, but by a nc command forked by the SST script on the joiner.  

I'd check to see what process is listening on 4444 when you get this error:

lsof | grep TCP | grep 4444 

(or similar)

Jay Janssen, MySQL Consulting Lead, Percona

http://about.me/jay.janssen

[in...@rentrange.com]

Thanks for the reply.

When I run (on non-working server)

lsof | grep TCP | grep 4444

I get nothing all.

When I run (on non-working server)
# netstat -anpt | grep 4444
tcp        0      0 0.0.0.0:4444                0.0.0.0:*                   LISTEN      8714/nc

On one of the working boxes its different
# netstat -anpt | grep 4444
tcp        0      0 333.333.333.138:54444         333.333.333.154:3306          TIME_WAIT   -

Does that tell you anything?

I have seen a few references to port 4567 (like in the error log) - not sure if that needs to be configured as well.

Is the system supposed to spawn these port references automatically?  Or do I need to manually put them in IP tables?
If I need to put them in IP tables (or other), can you please write out the command?

Thank you

[Ovais Tariq]

Hi,

On Thu, Jun 20, 2013 at 11:21 PM, <in...@rentrange.com> wrote:

Thanks for the reply.

When I run (on non-working server)

lsof | grep TCP | grep 4444

I get nothing all.

When I run (on non-working server)
# netstat -anpt | grep 4444
tcp        0      0 0.0.0.0:4444                0.0.0.0:*                   LISTEN      8714/nc

That shows that nc is listening on port 4444, 8714 is the PID of the nc process that is listening on 4444.

 

On one of the working boxes its different
# netstat -anpt | grep 4444
tcp        0      0 333.333.333.138:54444         333.333.333.154:3306          TIME_WAIT   -

This is a different port shown in the output "54444".

 

Does that tell you anything?

There is probably a run-away nc process lurking around listening on port 4444. You need to kill that process first and then restart mysqld again:

kill -9 8714

 

I have seen a few references to port 4567 (like in the error log) - not sure if that needs to be configured as well.

Port 4567 is the default port that Galera uses for inter-node communication.

 

Is the system supposed to spawn these port references automatically?  Or do I need to manually put them in IP tables?
If I need to put them in IP tables (or other), can you please write out the command?

Do you have any iptables rules already setup that block access to ports? If yes then you need to write rules to enable access to the following ports:

4567 -  used by Galera for inter-node communication

4444 - used for SST

4568 - used for IST

However if you do not have any iptables rules setup to deny access then you do not need to open up these ports specifically.

Best,

 

Thank you

--
You received this message because you are subscribed to the Google Groups "Percona Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to percona-discuss...@googlegroups.com.
To post to this group, send email to percona-d...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--

Ovais Tariq, Principal Support Engineer, Percona

http://www.percona.com | http://www.mysqlperformanceblog.com
Phone               : +1 (888) 401-3401 Ext. 552
24/7 Emergency : +1 888 401 3401 ext 911
Skype               : ovaistariq

Training : http://www.percona.com/training/
Support  : http://www.percona.com/mysql-support/

Percona Live London MySQL Conference 2013
http://www.percona.com/live/london-2013/