Down Leaving (DL) node

5 views
Skip to first unread message

Ariel Cohen

unread,
Nov 29, 2021, 5:06:44 PM11/29/21
to scyllad...@googlegroups.com
One of the node in my scylladb cluster is in DL status for over 3 days now
3 days before I had to stop the scylla process and call removenode from another node
avg load of each node is ~8TB with 11 nodes left in the cluster.
How should I proceed with DL node at this point and why it has not left the cluster so far.
 
Thanks
 

Asias He

unread,
Nov 29, 2021, 9:22:54 PM11/29/21
to ScyllaDB users
Hello Ariel,

nodetool removenode will trigger data streaming becasue after the node is removed existing nodes will own new token ranges. Did the 'nodetool removenode' command fail? If not this means the nodes are still streaming data. There are a lot of things that can make the streaming take a longer time, e.g., the amount of data per node, the compaction in the background. You can wait for the removenode to complete. If the compaction failed, you can still bring up the node that you tried to remove, e.g., fix the down node and restart the node. BTW, it is recommended to fix the down node. If it is not possible to fix, it is best to run the replace operation instead of the removenode operation.

--
You received this message because you are subscribed to the Google Groups "ScyllaDB users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scylladb-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-users/trinity-a6e32dce-fc0d-482d-832d-4d814aff8523-1638223602092%403c-app-mailcom-bs01.


--
Asias

Ariel Cohen

unread,
Nov 30, 2021, 10:28:16 AM11/30/21
to scyllad...@googlegroups.com
'nodetool removenode' have stopped, as I didnt run it in background
I have started 'nodetool removenode' again in background about 12hrs ago from another node
removed node is still showing DL status
'nodetool netstats' is showing ~20-25 data streams on each of the 11 nodes

Restore_replica_count-mydata-index-0 ff0072d7-5191-11ec-8755-000000000000

Thanks

-- 

Hello Ariel,
 
nodetool removenode will trigger data streaming becasue after the node is removed existing nodes will own new token ranges. Did the 'nodetool removenode' command fail? If not this means the nodes are still streaming data. There are a lot of things that can make the streaming take a longer time, e.g., the amount of data per node, the compaction in the background. You can wait for the removenode to complete. If the compaction failed, you can still bring up the node that you tried to remove, e.g., fix the down node and restart the node. BTW, it is recommended to fix the down node. If it is not possible to fix, it is best to run the replace operation instead of the removenode operation. 
 
 

--
You received this message because you are subscribed to the Google Groups "ScyllaDB users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scylladb-user...@googlegroups.com[https://groups.google.com/].
To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-users/trinity-a6e32dce-fc0d-482d-832d-4d814aff8523-1638223602092%403c-app-mailcom-bs01[https://groups.google.com/d/msgid/scylladb-users/trinity-a6e32dce-fc0d-482d-832d-4d814aff8523-1638223602092%403c-app-mailcom-bs01?utm_medium=email&utm_source=footer].

--

Asias
Reply all
Reply to author
Forward
0 new messages