Clarification on Differentiating Complete Node Failure vs. Network Partition in Raft

[Rajib Ghosal]

Dear Raft Community,

I have been studying the Raft consensus algorithm in depth and have encountered a subtle point regarding the behavior of a node when it stops receiving heartbeats. I would greatly appreciate your insights on the following scenario.

Background:
Consider a cluster of five nodes: A, B, C, D, and E. In Raft, a node relies on receiving regular heartbeat messages from the leader (or other nodes) to determine if the system is healthy.

Scenario 1 – Complete Cluster Failure:

• Suppose nodes A, B, C, and D have all failed (i.e., they are completely dead), leaving E as the only surviving node.
• In this situation, E, after not receiving any heartbeats, would eventually time out and attempt to start an election.

Scenario 2 – Network Partition (Isolated Node E):

• Now, imagine that nodes A, B, C, and D are alive and functioning normally, but due to a network partition, node E becomes isolated and cannot receive any heartbeats from them.
• From E’s perspective, it also does not receive any heartbeat messages and therefore might suspect that the leader (or the rest of the cluster) is down.

My Question:

Since node E relies solely on the heartbeat mechanism to detect activity, in both scenarios E “sees” no heartbeat. How does the Raft algorithm differentiate between the situation where all other nodes are truly dead (Scenario 1) versus the situation where a network partition isolates E (Scenario 2)? 

Thank you very much for your time and assistance in clarifying this matter. I look forward to your insights and any references or explanations you can provide.

[Philip O'Toole]

Inline.

On Fri, Feb 7, 2025 at 11:31 AM 'Rajib Ghosal' via raft-dev <raft...@googlegroups.com> wrote:

My Question:

Since node E relies solely on the heartbeat mechanism to detect activity, in both scenarios E “sees” no heartbeat. How does the Raft algorithm differentiate between the situation where all other nodes are truly dead (Scenario 1) versus the situation where a network partition isolates E (Scenario 2)? 

IIUC it doesn't, nor does it need to. Both situations are identical as far as node E is concerned.

The Raft protocol does explain how the node will handle the situation if and when the partition is healed and it is able to rejoin the cluster. But the only difference between nodes truly dying, or simply being uncontactable for an infinite period due to a network partition, is practical (in the former case you need to bring up new nodes, in the latter see if you can fix the network), not theoretical.

 

[Philip O'Toole]

On Fri, Feb 7, 2025 at 12:50 PM Philip O'Toole <oto...@google.com> wrote:

Inline.

On Fri, Feb 7, 2025 at 11:31 AM 'Rajib Ghosal' via raft-dev <raft...@googlegroups.com> wrote:

My Question:

Since node E relies solely on the heartbeat mechanism to detect activity, in both scenarios E “sees” no heartbeat. How does the Raft algorithm differentiate between the situation where all other nodes are truly dead (Scenario 1) versus the situation where a network partition isolates E (Scenario 2)? 

IIUC it doesn't, nor does it need to. Both situations are identical as far as node E is concerned.

Well, to be more precise, the situations are indistinguishable to Node E -- but it doesn't matter because the Raft protocol does not require them to be distinguishable. 

 

Philip

[A. Jesse Jiryu Davis]

In both cases, E is unable to win an election since it can't receive votes from a majority.

--
You received this message because you are subscribed to the Google Groups "raft-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to raft-dev+u...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/raft-dev/CAEajhJPSYq2p4%3DS9BtGKMMPoLavJ3F8WcxkxG1OOv%2BSCYXqfkA%40mail.gmail.com.

[Ejem Agbaeze]

I think from Raft fault tolerance assumption, N = 2f + 1 nodes is required to tolerate upto f crash failure, where N is the cluster size.

In reality, redundant nodes can step in to ensure reliability and consistency is maintained without the complete collapse of the system 

Theoretically, the assumption restrict the majority crashes from happening. If 4 nodes crash, only 1 node cannot form the majority quorum and without a majority, no new leader can be elected and the system cannot commit new log entries.

--

You received this message because you are subscribed to the Google Groups "raft-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to raft-dev+u...@googlegroups.com.

To view this discussion visit https://groups.google.com/d/msgid/raft-dev/eba0883d-bfc4-4d15-8fda-ef45f252a124n%40googlegroups.com.