AppendLogEntries creating logs with different term but same index and command

[Drew Eckhardt]

I assume this is OK because the log index and commands match?

On recovery:
1.  Absent optimizations , the leader iterates backwards over its log making an AppendEntries RPC for each log index until the follower matches prevLogIndex and prevLogTerm; truncates its log; and appends the entry.
2.  The leader iterates forwards from the next entry to its last making AppendEntries calls for groups of one or more log entries with the term field equal to their current term.

As a side effect, any log entries appended during recovery have a different term than in other nodes' logs.

[Chris Johnson]

Replicated logs should use the original term number.  It is not safe to change them to the new term number.

Example:

A is Leader, appends following:

A 1 1 1

B 1 1

C 1 1

B becomes leader, B and C commit up to entry 4:

A 1 1 1
B 1 1 2 2

C 1 1 2 2

C becomes leader, replicates log entry 3 to A with modified term 3:

A 1 1 3
B 1 1 2 2

C 1 1 2 2

At this point if C goes down only A can become leader as nodes only vote for servers where the most recent entry has a term number equal or greater than their own most recent entry, and if the terms match only then compare the lengths of the logs.  

When A becomes leader it will create a new operation for entry 4 that does not match entry 4 previously committed by B and C.

At best you have a sanity check to prevent B from overwriting committed logs and your cluster is stuck, at worst you overwrite your logs and your state machine diverges.