fix(data): Prevent semaphore deadlock within PersistentMutationOutbox

[tylerjroach]

• PR title and description conform to Pull Request guidelines.

Issue #, if available:

Description of changes:
There's a scenario here that can happen with constant start/stops where neither doOnTerminate and doOnError get called to release the semaphore. If a completable is unsubscribed, doOnTerminate does not get called. doFinally will always get called, however, it runs on a different thread. This should be safe, and seemed to work ok if we replaced doOnTerminate entirely, but given the threading model, its best to release in doOnTerminate. The doFinally is a new safety check that will only release if the semaphore was not already released in doOnTerminate.

How did you test these changes?
(Please add a line here how the changes were tested)

Documentation update required?

• No
• Yes (Please include a PR link for the documentation update)

General Checklist

• Added Unit Tests
• Added Integration Tests
• Security oriented best practices and standards are followed (e.g. using input sanitization, principle of least privilege, etc)
• Ensure commit message has the appropriate scope (e.g fix(storage): message, feat(auth): message, chore(all): message)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[mattcreaser]

Concept looks good. Instead of repeating the logic could we make an extension function like this?

fun Completable.acquireSemaphore(semaphore: Semaphore): Completable {
    val semaphoreReleased = AtomicBoolean(false)
    return this.doOnSubscribe { semaphore.acquire() }
        .doOnTerminate {
            if (semaphoreReleased.compareAndSet(false, true)) {
                semaphore.release()
            }
        }
        .doFinally {
            if (semaphoreReleased.compareAndSet(false, true)) {
                semaphore.release()
            }
        }
}

[tylerjroach]

Good suggestion. Done

[codecov]

Codecov Report

❌ Patch coverage is 92.59259% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 54.23%. Comparing base (e7425e8) to head (94b7e05).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #3115   +/-   ##
=======================================
  Coverage   54.22%   54.23%           
=======================================
  Files        1038     1038           
  Lines       31989    32002   +13     
  Branches     4702     4704    +2     
=======================================
+ Hits        17346    17356   +10     
- Misses      12795    12799    +4     
+ Partials     1848     1847    -1     

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.