We know that raft is based on the state machines assumption that if the state machine is "deterministic" and all the state machines will reach the same final state when they are fed with same input sequence.
While I found that the state machine assumption may be violated, I'm trying to find a way to modify our business logic to suit the state machine model.
We have such a problem in our storage system, each disk we have a storage engine, and multiple raft groups run on the same storage engine, and each storage engine may have different raft group members.For example, three nodes A,B,C form one raft group, the engine which serves for A, B, C may contains different other groups.
Currently, our implementation logic is that, one user request will be written into raft logs and replicated at first(regardless some parameters checking), and after the log is committed, we try to apply the log.However, since the state of the storage engines are different, the apply result of each raft member may become different, that violate the state machine assumption(e.g., the user sends a wrong parameter request which cause duplicate allocation for some resource, some nodes have the duplicated resource so they can detect the fault while others can't).
I proposed one solution is that, the leader should do all the checks and makes sure that the request will be applied successful before it writes the request into WAL.However I still meet the problem that even the leader thinks it's okay, the followers may conclude different results since their state are not same with leader.
Another solution is that, for some undeterministic logic I can just bypass the raft mechanism so we can make sure only the deterministic logic goes to raft.
What is the standard way to solve this problem?