Possible split brain problem caused by async storage write?

[neog24]

Hi @nvanbenschoten @tbg , we known that etcd-raft use apply after commit style for membership change. When async storage write is enabled, it seems that split brain might happen in extreme case, consider the following situation in a three node cluster: {1,2, 3}:

1. User proposed a membership change request to change peer list from {1, 2, 3} to {1, 2, 3, 4, 5}
2. Peer 1 is a follower, then all the peers except 1 succeeded in changing config from {1, 2, 3, 4, 5}
   • peers in {2, 3, 4, 5} finished in entering joint consensus and leaving joint consensus, and their config became {1, 2, 3, 4, 5}
   • peer 1 has accepted and committed all the latest log but somehow the apply thread stalled and its config remained {1, 2, 3}
3. Then Network partitions happened, and there are two isolate network {1, 2} and {3, 4, 5}
4. peer 1 timeout and became candidate, peer 2 casted vote, peer 1 became leader
5. {3, 4, 5} might choose another leader at a same term, result in split brain