Bug: IPC timeouts when deactivating multiple RDK services

[npoltorapavlo]

Problem/Opportunity

in log LinkType::Unsubscribe timeouts, IPC timeouts and "Unregistering an interface ... from a non-existing channel!!!"

2026-01-01T12:17:56.357Z WPEFramework[12192]:  [12235] INFO [DisplaySettings.cpp:5523] stopCecTimeAndUnsubscribeEvent: Unsubscribing event shortAudiodesciptorEvent
2026-01-01T12:17:57.359Z WPEFramework[12192]:  [12235] INFO [DisplaySettings.cpp:5523] stopCecTimeAndUnsubscribeEvent: Unsubscribing event setSystemAudioModeEvent
2026-01-01T12:17:58.359Z WPEFramework[12192]:  [12235] INFO [DisplaySettings.cpp:5523] stopCecTimeAndUnsubscribeEvent: Unsubscribing event reportAudioDeviceConnectedStatus
2026-01-01T12:17:59.360Z WPEFramework[12192]:  [12235] INFO [DisplaySettings.cpp:5523] stopCecTimeAndUnsubscribeEvent: Unsubscribing event reportCecEnabledEvent
2026-01-01T12:18:00.360Z WPEFramework[12192]:  [12235] INFO [DisplaySettings.cpp:5523] stopCecTimeAndUnsubscribeEvent: Unsubscribing event reportAudioDevicePowerStatus
2026-01-01T12:18:01.360Z WPEFramework[12192]:  [12235] INFO [DisplaySettings.cpp:5523] stopCecTimeAndUnsubscribeEvent: Unsubscribing event reportAudioStatusEvent
2026-01-01T12:18:13.435Z WPEFramework[12192]:  [SysLog]:[Error]: IPC method Invoke failed due to timeout (Interface ID 0x22, Method ID 0x4). Execution of code may or may not have happened. Side effects are to be expected after this message
2026-01-01T12:18:33.438Z WPEFramework[12192]:  [SysLog]:[Error]: IPC method Invoke failed due to timeout (Interface ID 0x22, Method ID 0x2). Execution of code may or may not have happened. Side effects are to be expected after this message
2026-01-01T12:18:53.463Z WPEFramework[12192]:  [SysLog]:[Error]: IPC method Invoke failed due to timeout (Interface ID 0x30, Method ID 0x22). Execution of code may or may not have happened. Side effects are to be expected after this message
2026-01-01T12:19:13.463Z WPEFramework[12192]:  [SysLog]:[Error]: IPC method Invoke failed due to timeout (Interface ID 0x480, Method ID 0x1). Execution of code may or may not have happened. Side effects are to be expected after this message
2026-01-01T12:19:33.464Z WPEFramework[12192]:  [SysLog]:[Error]: IPC method Invoke failed due to timeout (Interface ID 0x30, Method ID 0x1). Execution of code may or may not have happened. Side effects are to be expected after this message
2026-01-01T12:29:57.705Z WPEFramework[12192]:  ====> Unregistering an interface [0x30, 12192] from a non-existing channel!!!
2026-01-01T12:29:57.711Z /usr/bin/WPEFramework[12192]:  Signal received 11. in process [12192]
2026-01-01T12:29:57.711Z WPEFramework[12192]:  [SysLog]:[Error]: IPC method Invoke failed due to timeout (Interface ID 0x1043, Method ID 0x1). Execution of code may or may not have happened. Side effects are to be expected after this message

in coredump (ThreadPool queue 26, pool size 8):

• 6 threads wait _notificationLock in ServiceMap::Deinitialized owned by 12236
• Thread 9 (12236) waits ProxyStub::UnknownProxyType<WPEFramework::PluginHost::IPlugin::INotification>::Invoke IPC response in ServiceMap::Deinitialized
• Thread 2 (12192) waits ProxyStub::UnknownProxyType<WPEFramework::Exchange::ITextTrackTtmlStyle>::Release IPC response
• Thread 1 (12237) crashed in ServiceMap::RecursiveNotification

in ServiceMap _notifiers:

(gdb) print *(_notifiers._M_impl._M_start)
$42 = (WPEFramework::Core::Sink<WPEFramework::Plugin::Controller::Sink> *) 0x1b7fcc0
(gdb) print *(_notifiers._M_impl._M_start + 1)
$43 = (WPEFramework::Core::Sink<WPEFramework::Plugin::DeviceBridge::PluginsNotification> *) 0xac001a14
(gdb) print *(_notifiers._M_impl._M_start + 2)
$44 = (WPEFramework::Core::Sink<WPEFramework::Plugin::DeviceAppManager::PluginsNotification> *) 0xaa51017c
(gdb) print *(_notifiers._M_impl._M_start + 3)
$45 = (WPEFramework::ProxyStubs::PluginNotificationProxy *) 0xaa30dd48
(gdb) print *(_notifiers._M_impl._M_start + 4)
$46 = (WPEFramework::ProxyStubs::PluginNotificationProxy *) 0xa7628168
(gdb) print *(_notifiers._M_impl._M_start + 5)
$47 = (WPEFramework::ProxyStubs::PluginNotificationProxy *) 0xa7628168

in another coredump (ThreadPool queue 22, pool size 8):

• 6 threads wait _notificationLock in ServiceMap::Deinitialized/Deactivated/Unregister/... owned by 12144
• Thread 7 (12144) waits UnknownProxy::RemoteInterface/ProxyStub::UnknownProxy::Invoke IPC response in ServiceMap::Deinitialized
• Thread 6 (12143) waits ProxyStub::UnknownProxyType<WPEFramework::Exchange::IContentDecryption>::Invoke IPC response
• Thread 1 (12110) crashed in Service::ReleaseInterfaces

Looks like whenever deactivating multiple services, which may not be unusual situation, this will likely result in every deactivation being stuck in ServiceMap notifications as IPC responses require another thread, but if there isn't one available, it's a deadlock resolved by IPC timeout (20s)

Steps to reproduce

root@sky-llama-panel:~# cat /opt/stop.sh 
for item in "org.rdk.Bluetooth" "org.rdk.SecManager" "JSPP" "org.rdk.UsbMassStorage" \
  "org.rdk.Wifi" "org.rdk.MediaSettings" "PlayerInfo" "org.rdk.FrontPanel" \
  "org.rdk.AVInput" "DisplayInfo" "org.rdk.Privacy" "org.rdk.FirmwareUpdate" \
  "DeviceInfo" "Monitor" "org.rdk.SystemMode" "OCDM" \
  "org.rdk.SharedStorage" "org.rdk.OCIContainer" "org.rdk.RemoteControl" "org.rdk.MaintenanceManager" \
  "org.rdk.HdmiCecSource" "org.rdk.Analytics" "org.rdk.SystemAudioPlayer" "org.rdk.Xcast" \
  "org.rdk.LinchPinClient" "org.rdk.Cryptography" "org.rdk.IPControl" "org.rdk.TextTrack" \
  "org.rdk.CloudStore" "com.comcast.DeviceProvisioning" "org.rdk.UsbDevice" "org.rdk.VoiceControl" \
  "org.rdk.Telemetry" "org.rdk.AuthService" "org.rdk.System" "org.rdk.TextToSpeech" \
  "org.rdk.HdcpProfile" "org.rdk.UserSettings" "org.rdk.DisplaySettings" "org.rdk.Network"; do
  curl -X PUT http://127.0.0.1:9998/Service/Controller/Deactivate/$item &
done

Expected Behavior

Actual Behavior

Timeouts

Notes (Optional)

WPEFRAMEWORK-VERSION=R4.4.1