[fix] Failed read entries after multiple decommissioning

[poorbarcode]

Motivation

Background

• ReadOnlyLedgerHandle opened without doRecovery, in other words, it has not been closed yet, its metadata in memory will be updated once modified.
• ReadOnlyLedgerHandle opened with doRecovery, in other words, it has been closed or it will be closed; its metadata in memory will never be updated

Issue

• Background: The ledger's metadata can also be modified by the auto-recovery component.
• There is a scenario in which a ledger handle always gets the error org.apache.bookkeeper.client.BKException$BKBookieHandleNotAvailableException: Bookie handle is not available` after multiple decommissions. The reproduction steps are as follows
  • Client service opens a read-only ledger handle, which has been closed.
  • All BKs that relate to the ledger have been decommissioned.
  • Auto recovery component moved the data into other BK instances who is alive.
  • The ledger handle in the client memory keeps connecting to the BKs who are in the original ensemble set, and the connection will always fail.
• You can reproduce the issue with the new test testOpenedLedgerHandleStillWorkAfterDecommissioning

Changes

Let Bookie LedgerHandle always accept the metadata updates, no matter "doRecovery" or not.

Provide a new API to open a read-only ledger handle with keepUpdateMetadata, the ledger's metadata in memory will be updated automatically after the auto-recovery component updates it.

[codelipenghui]

I'm curious about is there any cases that the ReadOnlyLedgerHandle cannot accept the updates from metadata server. As I understand, it should always accept the metadata updates no matter "doRecovery" or not. If the Ledger metadata updates to the Ledger which need to be recovered is not expected, we should fix the updates parts.

[poorbarcode]

@codelipenghui

I'm curious about is there any cases that the ReadOnlyLedgerHandle cannot accept the updates from metadata server. As I understand, it should always accept the metadata updates no matter "doRecovery" or not.

Changed the implementation.

[poorbarcode]

rerun failure checks

[zymap]

When doing recovery, it is also a metadata update operation. So it doesn't allow watching the metadata update. Maybe there is a race condition risk if we always get the metadata updated from two places.

[poorbarcode]

Added a new parameter to update metadata automatically, we can set the new parameter to true if needed, and it does not affect other use cases