LaunchDarkly.ClientSDK - .Net iOS + MAUI iOS Crash - Bad Gateway (AWS Outage)

[justinwojo]

Is this a support request?
Yes

Describe the bug
During the AWS outage on 10/20/2025 that caused Bad Gateway responses for the clientstream.launchdarkly.com endpoints, the .NET SDK for iOS and MAUI caused all of our apps to crash during initialization. We have proper safeguards around initialization in case of any exceptions thrown or if it fails to initialize, to handle it gracefully. Unfortunately, this outage caused a EXC_BAD_ACCESS crash that didn't let us handle any exceptions or errors, and caused an immediate app crash for all of our iOS users.

This did not affect any of our Android users, only iOS, on both .Net for iOS and on .Net MAUI iOS apps.

To reproduce
Simulate bad gateway responses from the stream API while on an iOS app consuming the SDK, and see that the app crashes when initializing.

Expected behavior
The SDK should fail gracefully, return an error, and let us know it couldn't initialize.

SDK version
ClientSDK 5.5.2

Language version, developer tools
.NET 9.0.

OS/platform
iOS, all OS's exhibited the crash

[tanderson-ld]

Hello @justinwojo, thank you for reporting this. We will be investigating today.

[tanderson-ld]

Hi @justinwojo, I set up a test server that returned 502 for all responses and the SDK appears to be behaving as expected. Could you provide more information?

Questions:

1. Do you have any information about what phase of operation the apps were crashing in? Ex: already connected apps crashed, but newly launched apps did not.
2. Do you have any connection status event listeners that may be reacting to errors and executing logic?
3. Were there any devices that did not exhibit the crash, if so, could you describe those and how their operation differed from those that crashed?
4. During the outage, were you able to reproduce the crash yourself?
5. How many users were affected and what is this as a percentage of your customer base (if you aren't willing to share absolute numbers for business reasons, we understand).

Perhaps this is only when already connected. That will take more time to set up a mock server, so starting with the above questions will be helpful to steer this.

Thank you,
Todd

[justinwojo]

Hi @justinwojo, I set up a test server that returned 502 for all responses and the SDK appears to be behaving as expected. Could you provide more information?

Questions:

1. Do you have any information about what phase of operation the apps were crashing in? Ex: already connected apps crashed, but newly launched apps did not.
2. Do you have any connection status event listeners that may be reacting to errors and executing logic?
3. Were there any devices that did not exhibit the crash, if so, could you describe those and how their operation differed from those that crashed?
4. During the outage, were you able to reproduce the crash yourself?
5. How many users were affected and what is this as a percentage of your customer base (if you aren't willing to share absolute numbers for business reasons, we understand).

Perhaps this is only when already connected. That will take more time to set up a mock server, so starting with the above questions will be helpful to steer this.

Thank you, Todd

Hey @tanderson-ld. Absolutely, I'll provide as much detail as I can here:

1. Our apps were crashing on startup. I'd have to do more digging on if apps that were already running had issues, but absolutely during first-time initialization of the SDK.
2. I can share the code of our exact init process at the end here. Generally, we encapsulate our init/error in a base class, and have a singleton wrapper class that does the actual flags lookup, which will check if the SDK is initialized before doing any calls for flags.
3. From our data, pretty much all iOS apps were crashing on startup during this outage. Like I mentioned above, Android didn't have any issues here. Across both MAUI and .Net for iOS apps.
4. Yes, I could reproduce easily with both our store version and locally in a debug session. It was happening 100% of the time. If I removed the LaunchDarkly init code, it stopped crashing (was testing this live during the outage, so was able to confirm it was due to the LaunchDarkly Init).
5. We saw a massive spike in crashes, affecting more than 50% of our install base, and it led to multiple major incidents for our apps/company.

One thing I'll note, in some of our apps, we use the Async init (InitAsync). One of our MAUI apps does do a sync Init, though, and was also crashing.

I'll share our .Net for iOS init logic file with you (with some things removed). During our startup process, we do background init like this: _ = Task.Run(() => _featureFlagService.InitializeServiceAsync()); which just calls that InitializeLaunchDarklyServiceIfNeededAsync method.

A few helpful notes:

1. For our MAUI app that did a synchronous init with a timeout, we weren't getting crash reports into Sentry. Once it hit the timeout (be it 5 seconds, 3, whatever), it would then crash the app. For our .Net for iOS app using the background async init, we crashed after our 5 second timeout, but it seemed to be enough time for crash reports to get into Sentry.
2. A dev on one of our MAUI apps said locally he could change when the app crashed by adjusting the timeout. So, it seems to be directly related to the timeout function. It may be helpful if your test is failing quickly gracefully; whatever happened during the AWS outage caused the timeout to be reached, never an early exit.

NetForIOSClass.cs

^ Note this is a base class. We have a simple wrapper class that passes in various keys for feature flags. All logic is fully contained in this base class. It uses a semaphore to prevent multiple calls to init, in case of a slow init and someone calling for a feature flag.

Please let me know if there's anything else I can provide. If needed, I can share how we init sync on our MAUI app too, but hopefully the provided info helps.

[tanderson-ld]

Thank you for all the info. Will dig through it. Do you have any stack traces from Sentry?

We did see AWS's ALBs causing hanging sockets and partially transferred data. I'll test out slowing down the socket to trigger that timeout while the socket is open.

[justinwojo]

Thank you for all the info. Will dig through it. Do you have any stack traces from Sentry?

We did see AWS's ALBs causing hanging sockets and partially transferred data. I'll test out slowing down the socket to trigger that timeout while the socket is open.

Sure, I had Claude sanitize and evaluate the Sentry details for you. I can share the direct link to the reports in Sentry if you can provide me with an email to send it to, or some other way to share the link with you (can't share it publicly here).

Crash Report: EXC_BAD_ACCESS in NSUrlSessionHandler

Summary

Exception Type: EXC_BAD_ACCESS
Error Description: cancel > workQueue >
Culprit: xamarin_dyn_objc_msgSend
Last Seen: October 20, 2025
Status: Unresolved
Platform: iOS (Cocoa)

Error Details

Exception Message:

EXC_BAD_ACCESS: cancel > workQueue >
KERN_INVALID_ADDRESS at 0x60efa3462ef7b8.

Event Timestamp: October 20, 2025 at 20:19:24 UTC

Stack Trace

    at _pthread_start (<unknown>)
    at start_wrapper (threads.c:1276)
    at start_wrapper_internal (threads.c:1219)
    at mono_runtime_invoke_checked (object.c:2792)
    at do_runtime_invoke (object.c:2576)
    at mono_jit_runtime_invoke (mini-runtime.c:3649)
    at corlib_System_Threading_ThreadPoolWorkQueue_Dispatch (<unknown>)
    at Microsoft_iOS_System_Runtime_CompilerServices_AsyncTaskMethodBuilder_1_AsyncStateMachineBox_1_System_Net_Http_HttpResponseMessage_System_Net_Http_NSUrlSessionHandler__SendAsyncd__58_ExecutionContextCallback_object (<unknown>)
    at corlib_System_Threading_Tasks_Task_RunContinuations_object (<unknown>)
    at System_Net_Http_System_Runtime_CompilerServices_AsyncTaskMethodBuilder_1_AsyncStateMachineBox_1_System_Net_Http_HttpResponseMessage_System_Net_Http_HttpClient___SendAsyncg__Core_83_0d_ExecutionContextCallback_object (<unknown>)
    at corlib_System_Threading_Tasks_Task_RunContinuations_object (<unknown>)
    at LaunchDarkly_EventSource_System_Runtime_CompilerServices_AsyncTaskMethodBuilder_1_AsyncStateMachineBox_1_System_Threading_Tasks_VoidTaskResult_LaunchDarkly_EventSource_EventSourceService__ConnectToEventSourceApid__14_ExecutionContextCallback_object (<unknown>)
    at Microsoft_iOS_System_Net_Http_NSUrlSessionHandler_NSUrlSessionHandlerDelegate__c__DisplayClass5_0__DidReceiveResponseImplb__0 (NSUrlSessionHandler.cs:890)
    at Microsoft_iOS_Foundation_NSUrlSessionTask_Cancel (NSUrlSessionTask.g.cs:154)
    at wrapper_managed_to_native_ObjCRuntime_Messaging_void_objc_msgSend_intptr_intptr (<unknown>)
    at xamarin_dyn_objc_msgSend (<unknown>)
    at -[NSURLSessionTask cancel] (<unknown>)
    at objc_msgSend (<unknown>)

Key Stack Frames

The crash appears to originate from:

1. LaunchDarkly EventSource SDK - LaunchDarkly_EventSource_EventSourceService__ConnectToEventSourceApid__14 attempting to connect to EventSource API
2. NSUrlSessionHandler - NSUrlSessionHandler.cs:890 in DidReceiveResponseImpl callback
3. NSUrlSessionTask.Cancel - Attempting to cancel a session task
4. Crash Point - xamarin_dyn_objc_msgSend encountering KERN_INVALID_ADDRESS

Device & Environment Information

Device Details

• Device Model: iPhone 15 Pro (iPhone15,3)
• Architecture: arm64
• Memory: 5.9 GB total, 227 MB free at crash time
• Simulator: No (Physical device)

Operating System

• OS: iOS 18.1.1
• Build: 22B91
• Kernel: Darwin Kernel Version 24.1.0 (Mon Oct 7 21:11:37 PDT 2024)
• Rooted: No

Application Details

• Build Type: App Store
• Environment: Production
• Locale: en_US

Event Metadata

• Handled: No (Unhandled exception)
• Level: Fatal
• Mechanism: Mach exception
• App Start Time: October 20, 2025 at 20:11:04 UTC

Analysis

Crash Pattern

This crash exhibits the following characteristics:

1. Threading Context: Crash occurs on a background thread managed by the ThreadPool (ThreadPoolWorkQueue_Dispatch)

2. Async Operation Chain:

   • HTTP request via NSUrlSessionHandler
   • LaunchDarkly EventSource attempting to connect to streaming API
   • Task continuation callbacks executing

3. Memory Access Violation: KERN_INVALID_ADDRESS at 0x60efa3462ef7b8 suggests:

   • Accessing deallocated memory (use-after-free)
   • Invalid pointer dereferencing
   • Race condition between task cancellation and callback execution

4. Cancellation Context: The crash occurs during NSUrlSessionTask.Cancel(), specifically when:

   • A response callback (DidReceiveResponseImpl) is executing
   • Attempting to cancel the underlying NSURLSessionTask
   • The Objective-C message send (objc_msgSend) fails with bad memory access

Likely Root Cause

The stack trace suggests a race condition where:

1. LaunchDarkly EventSource initiates an HTTP connection to its streaming API
2. A response is received, triggering the DidReceiveResponseImpl callback
3. Concurrently, the task is being cancelled (possibly due to app backgrounding, timeout, or shutdown)
4. The callback attempts to access or message an already-deallocated object
5. This results in EXC_BAD_ACCESS when the Objective-C runtime tries to send a message to invalid memory

[tanderson-ld]

You can send sensitive info to tanderson@launchdarkly.com.

[tanderson-ld]

Hi again @justinwojo. I spent some more time today trying to reproduce the issue using the code you sent in NetForIOSClass.cs, but no luck.

I do notice that in that code it is possible for InitAsync to be invoked multiple times which is not recommended. I wonder if this is related. After the first call to Init returns or InitAsync unblocks, regardless of if it was able to initialize with data from the cloud or from cache, the client is running. If the client was unable to initialize with flag data, it automatically retries in the background.

You only need to call InitAsync once during the lifetime of the application as that creates the singleton instance internally. Subsequent calls may lead this logic inside the SDK to execute.

        static LdClient CreateInstance(Configuration configuration, Context initialContext, TimeSpan maxWaitTime)
        {
            lock (_createInstanceLock)
            {
                if (_instance != null)
                {
                    throw new Exception("LdClient instance already exists.");
                }

                var c = new LdClient(configuration, initialContext, maxWaitTime);
                _instance = c;
                return c;
            }
        }

_launchDarklyClient.Initialized may be false when there is no connectivity at startup, but then later after a successful retry _launchDarklyClient.Initialized may be true.

The timespan provided to the InitAsync call can be thought of as "what is the maximum amount of time you want to wait to get fresh flags before proceeding with defaults/cached flags".

Again, I am speculating this may be related since I'm having trouble reproducing it and I'm not currently aware of other support tickets. It does seem based on your stack trace that there is some way to get the EventSource into a race condition. Perhaps there are multiple LDClients starting and stopping in quick succession?

Is it possible for you to try to create a minimal reproducible project and share that?

[justinwojo]

@tanderson-ld It would never call InitAsync multiple times in parallel. The only time it would call it again is if LdClient shows as uninitialized after trying to initialize before. I can adjust the code here, but if the client didn't initialize, we'd certainly want to try to initialize it again when we need to fetch another feature flag. If we should treat it as "if it can't initialize once, never try it again for the entire app's lifecycle", I certainly can, but I'm not sure that's the smoking gun here.

For what it's worth, here's the MAUI code on another app that we had mass crashes on, completely different setup, no async usage, directly checks the Instance to see if it exists before it tries to init:

if (LdClient.Instance != null)
{
    LdClient.Instance.Identify(context, sdkTimeout);
}
else
{
    var config = Configuration
        .Builder(SdkKeyHere, ConfigurationBuilder.AutoEnvAttributes.Enabled)
        .Build();

    LdClient.Init(config, context, sdkTimeout);
}

Same thing here, we could reproduce the crashes locally during the outage with this code as well. It would call LdClient.Init, then crash as soon as it hits the timeout.

You mentioned not getting any reports from other customers. I wanted to call out that with this synchronous init, none of our crash reports were getting sent up to Sentry, even though it caused a complete app outage for us on iOS (repeated crash at launch).