adding new node giving joining node error

[Taylor Fort]

"Failed to add node [new node ip address]:8091 to cluster. Prepare join failed. Joining node to itself is not allowed."

I can confirm that both boxes have two different ip addresses and the new node is a recent kickstart.  I did notice that i'm getting "Decided to change address to [old ip address]" in the logs of the cluster that I'm joining the new node to.  Googling the error message isn't turning up much. 

[Aliaksey Kandratsenka]

On Tue, May 7, 2013 at 3:13 PM, Taylor Fort <taylo...@gmail.com> wrote:

"Failed to add node [new node ip address]:8091 to cluster. Prepare join failed. Joining node to itself is not allowed."

I can confirm that both boxes have two different ip addresses and the new node is a recent kickstart.  I did notice that i'm getting "Decided to change address to [old ip address]" in the logs of the cluster that I'm joining the new node to.  Googling the error message isn't turning up much. 

Double check that on that node that joins other node [new node ip address] actually routes to new node. I bet there's something in your environment or routing setup or ip addresses asignment. 

[Taylor Fort]

Not the case.  It took a long time to get around this issue, but I solved it.  For anyone interested, here's how I went about it... there were probably easier ways, but my solution worked.  

Context... 

Originally had a single node instance of couchbase that was serving traffic.  Kickstarted another machine in order to expand into a 2 node cluster (1 replica for buckets).  Went to add the new node and it continually gave me errors saying that "node cannot join itself."  Logs on the first node showed "node changed ip to [correct ip]".  

Solution... 

Tried several approaches, but finally decided that the original node was corrupt.  Started up the new node as its own instance and created XCDR replication from the old node to the new node and let it transfer over the data.  Once it was complete, I shut down the old node, wiped the data dir and started couchbase again into setup node.  Now I tried to join the older node to the newly created cluster.  Huzzah it worked and joined, but under the servers list it posted BOTH machines as the same IP address.  Lame.  Worse problem... if I tried to fail over / remove the newly added box, it attempted to delete both of the servers showing up as the same IP address in the Servers list in the GUI.  So I shutdown the older, corrupt box, deleted the index/data directory contents, and completely reinstalled the rpm package.  Afterwards, I joined back into the cluster and it was finally showing its proper IP address.  Started up a rebalance and everything worked itself out.  

Question... 

I know of the ip file that is generated in couchbase, but are there any editable or removable files or cache that I need to worry about that will store the initial IP address of a machine ounce couchbase is installed and started for the first time?  

either way... my problem is solved, but out of curiosity, I'd like to figure out why the couchbase server would obtain/retain the wrong ip address for a server.  

thanks!

[Aliaksey Kandratsenka]

On Fri, May 10, 2013 at 1:05 AM, Taylor Fort <taylo...@gmail.com> wrote:

Not the case.  It took a long time to get around this issue, but I solved it.  For anyone interested, here's how I went about it... there were probably easier ways, but my solution worked.  

Context... 

Originally had a single node instance of couchbase that was serving traffic.  Kickstarted another machine in order to expand into a 2 node cluster (1 replica for buckets).  Went to add the new node and it continually gave me errors saying that "node cannot join itself."  Logs on the first node showed "node changed ip to [correct ip]".  

Solution... 

Tried several approaches, but finally decided that the original node was corrupt.  Started up the new node as its own instance and created XCDR replication from the old node to the new node and let it transfer over the data.  Once it was complete, I shut down the old node, wiped the data dir and started couchbase again into setup node.  Now I tried to join the older node to the newly created cluster.  Huzzah it worked and joined, but under the servers list it posted BOTH machines as the same IP address.  Lame.  Worse problem... if I tried to fail over / remove the newly added box, it attempted to delete both of the servers showing up as the same IP address in the Servers list in the GUI.  So I shutdown the older, corrupt box, deleted the index/data directory contents, and completely reinstalled the rpm package.  Afterwards, I joined back into the cluster and it was finally showing its proper IP address.  Started up a rebalance and everything worked itself out.  

Well you're free to continue not believing me but I tell you this happens because two nodes believe they both have that same ip. It will, one way or the other, inevitably lead to disaster.

 

Question... 

I know of the ip file that is generated in couchbase, but are there any editable or removable files or cache that I need to worry about that will store the initial IP address of a machine ounce couchbase is installed and started for the first time?  

That's indeed ip file. ip is decided based on socket address that's used to reach the other node when we perform cluster join protocol over http.

[Andy Lane]

You wouldn't by chance be running into this issue because the instructions in this document were followed incorrectly for one of the nodes, would you?

http://www.couchbase.com/docs/couchbase-manual-1.8/couchbase-bestpractice-cloud-ip.html

Andrew Lane

--
You received this message because you are subscribed to the Google Groups "Couchbase" group.
To unsubscribe from this group and stop receiving emails from it, send an email to couchbase+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.