New node not joining to the ring (scylla 2.1.6)

Denis Zhdanov

<denis.zhdanov@gmail.com>

unread,

Jul 16, 2018, 6:00:08 PM7/16/18

to scylladb-users@googlegroups.com, asias@scylladb.com

Trying to add new node to 8 nodes Scylla 2.1.6 cluster. For some reason node stucks in "waiting for schema" phase forever. Restarting node and/or cleaning up /var/lib/scylla/data not helping.

New node has name mt-scylla003 / IP 10.56.128.21.

Full logs of new node - https://gist.github.com/deniszh/3d3cb2926ccab51385fd13f2c0afaea2

Cluster status - https://gist.github.com/deniszh/68f54e6c2a5998c5208541bc67271afd

Thanks a lot!

PS: new IP is visible to other nodes in gossip btw, e.g. when I restarting new node on other nodes I see

INFO 2018-07-16 23:27:15,381 [shard 0] rpc - client 10.56.128.21: fail to connect: Connection refused

INFO 2018-07-16 23:27:18,398 [shard 0] rpc - client 10.56.128.21: fail to connect: Connection refused

INFO 2018-07-16 23:27:33,505 [shard 0] gossip - InetAddress 10.56.128.21 is now DOWN, status =

INFO 2018-07-16 23:27:40,555 [shard 0] gossip - FatClient 10.56.128.21 has been silent for 30000ms, removing from gossip

INFO 2018-07-16 23:28:56,035 [shard 0] gossip - InetAddress 10.56.128.21 is now UP, status =

INFO 2018-07-16 23:28:56,129 [shard 0] gossip - Favor newly added node 10.56.128.21

Tzach Livyatan

<tzach@scylladb.com>

unread,

Jul 17, 2018, 1:20:54 AM7/17/18

to ScyllaDB users, Glauber Costa, Asias He

Maybe Glauber comment https://github.com/scylladb/scylla/issues/2921#issuecomment-339559855 will help

--
You received this message because you are subscribed to the Google Groups "ScyllaDB users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scylladb-users+unsubscribe@googlegroups.com.
To post to this group, send email to scylladb-users@googlegroups.com.
Visit this group at https://groups.google.com/group/scylladb-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-users/CAMacAtJYXRcQB3tH3a3uWji774Tcci3623aYjUetU36qyE_9iQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Asias He

<asias@scylladb.com>

unread,

Jul 17, 2018, 1:53:19 AM7/17/18

to Denis Zhdanov, Tomasz Grabiec, ScyllaDB users

On Tue, Jul 17, 2018 at 5:59 AM, Denis Zhdanov <denis....@gmail.com> wrote:

Trying to add new node to 8 nodes Scylla 2.1.6 cluster. For some reason node stucks in "waiting for schema" phase forever. Restarting node and/or cleaning up /var/lib/scylla/data not helping.
New node has name mt-scylla003 / IP 10.56.128.21.

Can you try:

$ rm -rf /var/lib/scylla/commitlog/* /var/lib/scylla/data/*

on the new node, and add the node again.

If it still hangs in 'wait for schema', do the cleanup and try again.

@Tomek, Do we have schema related fixes not backported in 2.1.6.

Full logs of new node - https://gist.github.com/deniszh/3d3cb2926ccab51385fd13f2c0afaea2
Cluster status - https://gist.github.com/deniszh/68f54e6c2a5998c5208541bc67271afd

Thanks a lot!

PS: new IP is visible to other nodes in gossip btw, e.g. when I restarting new node on other nodes I see
INFO 2018-07-16 23:27:15,381 [shard 0] rpc - client 10.56.128.21: fail to connect: Connection refused
INFO 2018-07-16 23:27:18,398 [shard 0] rpc - client 10.56.128.21: fail to connect: Connection refused
INFO 2018-07-16 23:27:33,505 [shard 0] gossip - InetAddress 10.56.128.21 is now DOWN, status =
INFO 2018-07-16 23:27:40,555 [shard 0] gossip - FatClient 10.56.128.21 has been silent for 30000ms, removing from gossip
INFO 2018-07-16 23:28:56,035 [shard 0] gossip - InetAddress 10.56.128.21 is now UP, status =
INFO 2018-07-16 23:28:56,129 [shard 0] gossip - Favor newly added node 10.56.128.21

This is fine. The new node is not added to the cluster yet, so others will remove it.

--

Asias

Denis Zhdanov

<denis.zhdanov@gmail.com>

unread,

Jul 17, 2018, 2:03:51 AM7/17/18

to scylladb-users@googlegroups.com, Asias He, Glauber Costa

On Tue, 17 Jul 2018 at 07:20, Tzach Livyatan <tz...@scylladb.com> wrote:

Looks similar to https://github.com/scylladb/scylla/issues/2921
Maybe Glauber comment https://github.com/scylladb/scylla/issues/2921#issuecomment-339559855 will help

Yes, looks similar, thanks. Looks like restart if seed nodes should help, let me try that.

On Tue, Jul 17, 2018 at 12:59 AM, Denis Zhdanov <denis....@gmail.com> wrote:

Trying to add new node to 8 nodes Scylla 2.1.6 cluster. For some reason node stucks in "waiting for schema" phase forever. Restarting node and/or cleaning up /var/lib/scylla/data not helping.
New node has name mt-scylla003 / IP 10.56.128.21.

Full logs of new node - https://gist.github.com/deniszh/3d3cb2926ccab51385fd13f2c0afaea2
Cluster status - https://gist.github.com/deniszh/68f54e6c2a5998c5208541bc67271afd

Thanks a lot!

PS: new IP is visible to other nodes in gossip btw, e.g. when I restarting new node on other nodes I see
INFO 2018-07-16 23:27:15,381 [shard 0] rpc - client 10.56.128.21: fail to connect: Connection refused
INFO 2018-07-16 23:27:18,398 [shard 0] rpc - client 10.56.128.21: fail to connect: Connection refused
INFO 2018-07-16 23:27:33,505 [shard 0] gossip - InetAddress 10.56.128.21 is now DOWN, status =
INFO 2018-07-16 23:27:40,555 [shard 0] gossip - FatClient 10.56.128.21 has been silent for 30000ms, removing from gossip
INFO 2018-07-16 23:28:56,035 [shard 0] gossip - InetAddress 10.56.128.21 is now UP, status =
INFO 2018-07-16 23:28:56,129 [shard 0] gossip - Favor newly added node 10.56.128.21

--
You received this message because you are subscribed to the Google Groups "ScyllaDB users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to scylladb-user...@googlegroups.com.
To post to this group, send email to scyllad...@googlegroups.com.

Visit this group at https://groups.google.com/group/scylladb-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-users/CAMacAtJYXRcQB3tH3a3uWji774Tcci3623aYjUetU36qyE_9iQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "ScyllaDB users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to scylladb-user...@googlegroups.com.
To post to this group, send email to scyllad...@googlegroups.com.

Visit this group at https://groups.google.com/group/scylladb-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-users/CAEB7%3D3ZNJkOQjd00eTWrS6d0ifqo5cjTFUPA%3DeY881Q2QTZaQw%40mail.gmail.com.

Denis Zhdanov

<denis.zhdanov@gmail.com>

unread,

Jul 17, 2018, 2:07:14 AM7/17/18

to Asias He, ScyllaDB users, Tomasz Grabiec

On Tue, 17 Jul 2018 at 07:53, Asias He <as...@scylladb.com> wrote:

On Tue, Jul 17, 2018 at 5:59 AM, Denis Zhdanov <denis....@gmail.com> wrote:
Trying to add new node to 8 nodes Scylla 2.1.6 cluster. For some reason node stucks in "waiting for schema" phase forever. Restarting node and/or cleaning up /var/lib/scylla/data not helping.
New node has name mt-scylla003 / IP 10.56.128.21.

Can you try:

$ rm -rf /var/lib/scylla/commitlog/* /var/lib/scylla/data/*

on the new node, and add the node again.

Did that couple of times - without any effect.

If it still hangs in 'wait for schema', do the cleanup and try again.

You mean try clean data directory one more time? Did that, no effect.

I will try to restart seed nodes, otherwise will try to upgrade cluster to 2.2

Asias He

<asias@scylladb.com>

unread,

Jul 17, 2018, 2:18:49 AM7/17/18

to Denis Zhdanov, ScyllaDB users, Tomasz Grabiec

On Tue, Jul 17, 2018 at 2:07 PM, Denis Zhdanov <denis....@gmail.com> wrote:

On Tue, 17 Jul 2018 at 07:53, Asias He <as...@scylladb.com> wrote:
On Tue, Jul 17, 2018 at 5:59 AM, Denis Zhdanov <denis....@gmail.com> wrote:
Trying to add new node to 8 nodes Scylla 2.1.6 cluster. For some reason node stucks in "waiting for schema" phase forever. Restarting node and/or cleaning up /var/lib/scylla/data not helping.
New node has name mt-scylla003 / IP 10.56.128.21.

Can you try:

$ rm -rf /var/lib/scylla/commitlog/* /var/lib/scylla/data/*

on the new node, and add the node again.
Did that couple of times - without any effect.

If it still hangs in 'wait for schema', do the cleanup and try again.
You mean try clean data directory one more time? Did that, no effect.

I meant clean up all the data on the new node before restart the new node.

I will try to restart seed nodes, otherwise will try to upgrade cluster to 2.2

OK. Let us know if it works.

@Tomek, Do we have schema related fixes not backported in 2.1.6.

Full logs of new node - https://gist.github.com/deniszh/3d3cb2926ccab51385fd13f2c0afaea2
Cluster status - https://gist.github.com/deniszh/68f54e6c2a5998c5208541bc67271afd

Thanks a lot!

PS: new IP is visible to other nodes in gossip btw, e.g. when I restarting new node on other nodes I see
INFO 2018-07-16 23:27:15,381 [shard 0] rpc - client 10.56.128.21: fail to connect: Connection refused
INFO 2018-07-16 23:27:18,398 [shard 0] rpc - client 10.56.128.21: fail to connect: Connection refused
INFO 2018-07-16 23:27:33,505 [shard 0] gossip - InetAddress 10.56.128.21 is now DOWN, status =
INFO 2018-07-16 23:27:40,555 [shard 0] gossip - FatClient 10.56.128.21 has been silent for 30000ms, removing from gossip
INFO 2018-07-16 23:28:56,035 [shard 0] gossip - InetAddress 10.56.128.21 is now UP, status =
INFO 2018-07-16 23:28:56,129 [shard 0] gossip - Favor newly added node 10.56.128.21

This is fine. The new node is not added to the cluster yet, so others will remove it.

--
Asias

--

Asias

Asias He

<asias@scylladb.com>

unread,

Jul 17, 2018, 7:59:56 PM7/17/18

to Denis Zhdanov, ScyllaDB users, Tomasz Grabiec

So Denis told me after restating the seed node. The schema sync issue is gone.

On Tue, Jul 17, 2018 at 2:07 PM, Denis Zhdanov <denis....@gmail.com> wrote:

--

Asias

Reply all

Reply to author

Forward