Scylla node UP but unable to gossip with other nodes

[Rajsekhar Mallick]

Hello team,

One of the nodes in my 5 node cluster went down due to some hardware issues. 

When it was restarted, below logs were seen:

Scylla:[shard 33] storage_service - shutting down communications due to I/O errors until operator intervention.

The node then gets started. But when we run a nodetool status on the node, it doesn't list IP of other 4 nodes in the cluster.

Having checked the logs in other 4 nodes, they report

Scylla[shard 0] gossip - InetAddress <impacted IP> is now DOWN, status = shutdown

We have removed the impacted node from the cluster for the moment. Please comment on what may be the issue

Thanks

[Asias He]

On Sat, Sep 21, 2019, 08:47 Rajsekhar Mallick <raj.ma...@gmail.com> wrote:

Hello team,

One of the nodes in my 5 node cluster went down due to some hardware issues. 

When it was restarted, below logs were seen:

Scylla:[shard 33] storage_service - shutting down communications due to I/O errors until operator intervention.

When the  node see io error which is probably due to hardware issues, it will shutdown the cql and gossip to protect this bad node won't have impact on the result of the cluster. The process does not exit itself though. That is the reason other nodes  see this node as down.

The node then gets started. But when we run a nodetool status on the node, it doesn't list IP of other 4 nodes in the cluster.

Having checked the logs in other 4 nodes, they report

Scylla[shard 0] gossip - InetAddress <impacted IP> is now DOWN, status = shutdown

We have removed the impacted node from the cluster for the moment. Please comment on what may be the issue

Thanks

--
You received this message because you are subscribed to the Google Groups "ScyllaDB users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scylladb-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-users/CACYMiLGTEHX3e6JWStzNQoBC7hYPb2CKspZRndoA9UfbbyNESw%40mail.gmail.com.

[Avi Kivity]

On 21/09/2019 04.11, Asias He wrote:

On Sat, Sep 21, 2019, 08:47 Rajsekhar Mallick <raj.ma...@gmail.com> wrote:

Hello team,

One of the nodes in my 5 node cluster went down due to some hardware issues. 

When it was restarted, below logs were seen:

Scylla:[shard 33] storage_service - shutting down communications due to I/O errors until operator intervention.

When the  node see io error which is probably due to hardware issues, it will shutdown the cql and gossip to protect this bad node won't have impact on the result of the cluster. The process does not exit itself though. That is the reason other nodes  see this node as down.

The I/O error may be logged in dmesg.

The node then gets started. But when we run a nodetool status on the node, it doesn't list IP of other 4 nodes in the cluster.

Having checked the logs in other 4 nodes, they report

Scylla[shard 0] gossip - InetAddress <impacted IP> is now DOWN, status = shutdown

We have removed the impacted node from the cluster for the moment. Please comment on what may be the issue

Thanks

--
You received this message because you are subscribed to the Google Groups "ScyllaDB users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scylladb-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-users/CACYMiLGTEHX3e6JWStzNQoBC7hYPb2CKspZRndoA9UfbbyNESw%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "ScyllaDB users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scylladb-user...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-users/CAO1GqFbbzDctA%3DC2uNL51R_Li6BwFa8VECgomfcESNZx8vZC%3Dg%40mail.gmail.com.