partial sync and expirations

[kostasrim]

On a 3 node setup and on takeover flow what happens if elements expire after waiting for the newly promoted master to catch up and before waiting for the rest of the nodes complete. In other words, expiration should be turned of once and until all nodes catch up.

From this comment #6353 (comment)

[romange]

it's a dangerous and flaky rule. why do we need it?

[kostasrim]

Example: node a(master), node b(replica 1) node c(replica 2)

node b initiates takeover on node a.

node a receives DFLY TAKEOVER. It waits node b to catch up and it disables expirations while it waits. Then expirations are enabled again and node a waits node c to also catch up and then shutsdown gracefully -- for non cluster. The problem here, is that if there are expirations after waiting node b but before waiting node c the LSN of node c will increase. This can:

• disable partial sync because the replica will try to sync from the future
• break data parity. If master receives the same number of commands as the number of expired entries (VERY unlikely -- partial sync will proceed from that LSN number but master(data) != replica(data))

What's the reason you think it's dangerous ? If the new master never received the newly expired elements (and since we will be shutting down) what's the reason for the second replica to receive those and break partial sync ?