[server] Add a heartbeat lag based replica auto resubscribe feature

[sixpluszero]

[server] Add a heartbeat lag based replica auto resubscribe feature

This is an optional feature that aims to improve consumption resiliency. When we observe current version resource heartbeat delay grows beyond certain threshold, it could potentially be that underlying consumer is not functioning well. While we are also working on figuring out the root cause of that, this feature aims to mitigate it in the runtime by asking the SIT to try to perform a resubscribe action that brings the replica to another consumer in the same consumer pool.

Solution

Code changes

• Added new code behind a config. If so list the config names and their default values in the PR description.
• Introduced new log lines.
  • Confirmed if logs need to be rate limited to avoid excessive logging.

Concurrency-Specific Checks

Both reviewer and PR author to verify

• Code has no race conditions or thread safety issues.
• Proper synchronization mechanisms (e.g., synchronized, RWLock) are used where needed.
• No blocking calls inside critical sections that could lead to deadlocks or performance degradation.
• Verified thread-safe collections are used (e.g., ConcurrentHashMap, CopyOnWriteArrayList).
• Validated proper exception handling in multi-threaded code to avoid silent thread termination.

How was this PR tested?

Work in progress

• New unit tests added.
• New integration tests added.
• Modified or extended existing tests.
• Verified backward compatibility (if applicable).

Does this PR introduce any user-facing or breaking changes?

• No. You can skip the rest of this section.
• Yes. Clearly explain the behavior change and its impact.

[haoxu07]

Thanks for preparing this! Left a few comments.

[haoxu07]

Seems we only concern lag here, should we consider both lag and very few records polled from consumer?

[haoxu07]

BTW, it is possible that underlying resubscribe operationh is very slow on waiting for the drainer to be cleared for specific partition, can we add some safeguard here to avoid enqueuing too many partititons into SIT resubscribe queue? We could check the SIT PSC # to prevent put too many partitions.

[sixpluszero]

Synced offline - we have one config to control each replica's resubscribe interval - it will make sure resubscribe behavior does not happen too frequent