Rebalance failure after adding nodes

[Harsh Dev]

Hi ,

I get the following exception after i do a rebalance after succesfully adding a server int he cluster.

Rebalance exited with reason {unexpected_exit,
{'EXIT',<0.32686.0>,
{bulk_set_vbucket_state_failed,
[{'ns...@10.27.243.14',
{'EXIT',
{{{{case_clause,
{error,
{{{badmatch,
{error,
{{badmatch,{error,etimedout}},
[{mc_replication,connect,1,
[{file,
"src/mc_replication.erl"},
{line,30}]},
{mc_replication,connect,1,
[{file,
"src/mc_replication.erl"},
{line,49}]},
{dcp_proxy,connect,4,
[{file,"src/dcp_proxy.erl"},
{line,179}]},
{dcp_proxy,maybe_connect,1,
[{file,"src/dcp_proxy.erl"},
{line,166}]},
{dcp_producer_conn,init,2,
[{file,
"src/dcp_producer_conn.erl"},
{line,30}]},
{dcp_proxy,init,1,
[{file,"src/dcp_proxy.erl"},
{line,48}]},
{gen_server,init_it,6,
[{file,"gen_server.erl"},
{line,304}]},
{proc_lib,init_p_do_apply,3,
[{file,"proc_lib.erl"},
{line,239}]}]}}},
[{dcp_replicator,init,1,
[{file,"src/dcp_replicator.erl"},
{line,48}]},
{gen_server,init_it,6,
[{file,"gen_server.erl"},
{line,304}]},
{proc_lib,init_p_do_apply,3,
[{file,"proc_lib.erl"},
{line,239}]}]},
{child,undefined,
'ns...@10.28.221.83',
{dcp_replicator,start_link,
['ns...@10.28.221.83',"harsh_test"]},
temporary,60000,worker,
[dcp_replicator]}}}},
[{dcp_sup,start_replicator,2,
[{file,"src/dcp_sup.erl"},{line,78}]},
{dcp_sup,
'-set_desired_replications/2-lc$^2/1-2-',
2,
[{file,"src/dcp_sup.erl"},{line,55}]},
{dcp_sup,set_desired_replications,2,
[{file,"src/dcp_sup.erl"},{line,55}]},
{replication_manager,handle_call,3,
[{file,"src/replication_manager.erl"},
{line,130}]},
{gen_server,handle_msg,5,
[{file,"gen_server.erl"},{line,585}]},
{proc_lib,init_p_do_apply,3,
[{file,"proc_lib.erl"},{line,239}]}]},
{gen_server,call,
['replication_manager-harsh_test',
{change_vbucket_replication,511,
'ns...@10.28.221.83'},
infinity]}},
{gen_server,call,
[{'janitor_agent-harsh_test',
'ns...@10.27.243.14'},
{if_rebalance,<0.32669.0>,
{update_vbucket_state,511,replica,
passive,'ns...@10.28.221.83'}},
infinity]}}}}]}}}

Thanks in Advance for helping me out.

[Aliaksey Kandratsenka]

On Fri, Nov 14, 2014 at 3:18 PM, Harsh Dev <hars...@gmail.com> wrote:

Hi ,

I get the following exception after i do a rebalance after succesfully adding a server int he cluster.

Error below can be translated to human language as follows:

"During rebalance, while trying to establush dcp connection to node 10.27.22.83 from node 10.27.243.14 I've got ETIMEDOUT error from connect syscall".

This error is returned if several SYN packets got dropped. So it's something with network. What is less clear if it's something caused by couchbase server or not.

We'll need more details about your environment.

--
You received this message because you are subscribed to the Google Groups "Couchbase" group.
To unsubscribe from this group and stop receiving emails from it, send an email to couchbase+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Harsh Dev]

Thanks..What kind of information should i provide..ill also look into the network settings..

[Aliaksey Kandratsenka]

On Tue, Nov 18, 2014 at 8:51 AM, Harsh Dev <hars...@gmail.com> wrote:

Thanks..What kind of information should i provide..ill also look into the network settings..

Everything you can provide.

Plus run whatever network diagnostics you can to try to find cause of lost packets.