Save backup manifests for snapshot backup (#232)

[michaelmdeng]

What problem does this PR solve?

Closes #6179

What is changed and how does it work?

Saves cluster manifests to external storage as part of the snapshot backup process, similar to EBS volume snapshot backups.

While EBS backups perform this step after the init job and prior to each data-plane's individual backup job (in fact each data plane will upload manifest to the same location, so only last-write-wins manifest will be backed up). For logical snapshot backup, there is no init job and there is only a single backup job that orchestrates the entire backup -- we choose to add the manifest backup step prior to the overall backup job. Similarly, this will only backup the cluster manifests from a single AZ, the control-plane AZ that performs the backup job, but this is sufficient for our use-cases.

Code changes

• Has Go code change
• Has CI related scripts change

Tests

• Unit test
• E2E test
• Manual test
• No code

Side effects

• Breaking backward compatibility
• Other side effects:

Adds additional external storage requirements for snapshot backup, additional step prior to the BR process that could fail.

Related changes

• Need to cherry-pick to the release branch
• Need to update the documentation

Release Notes

Please refer to Release Notes Language Style Guide before writing the release note.

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign wizardxiao for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 40.00000% with 3 lines in your changes missing coverage. Please review.

Project coverage is 56.60%. Comparing base (a16b2e8) to head (b3b47b5).
Report is 35 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #6180      +/-   ##
==========================================
- Coverage   57.19%   56.60%   -0.60%     
==========================================
  Files         259      259              
  Lines       33233    45446   +12213     
==========================================
+ Hits        19008    25725    +6717     
- Misses      12291    17696    +5405     
- Partials     1934     2025      +91     

Flag	Coverage Δ
unittest	56.60% <40.00%> (-0.60%)	⬇️

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[BornChanger]

please add a switch (by default false) to control if manifest backup is needed for snapshot backup since this change add more items to backup bucket.