frontend: Adjust application shutdown logic

[Warchamp7]

Description

This PR does a bit of clean up and separation of logic in OBSBasic's shutdown process and makes a number of improvements to how OBS handles being closed by things other than a normal shutdown.

• Removes the app shutdown in response to Qt's commitDataRequest in favor of only saving data
• Removes the crash handler sentinel file earlier in the shutdown process.
  • It is now done after the main window is closed, which is after OBS has done most if it's own cleanup, but before the application is fully shutdown. Off thread discussion led to this decision as OBS' data is saved at this point, and plugins misbehaving at this point is not really a helpful reason to trigger the unclean shutdown dialog.
• On Windows, sets shutdown priority for OBS to be higher
  • This lets OBS respond to WM_QUERYENDSESSION before Windows kills off the browser subprocesses, as normally they'd be terminated at the same step.
  • This fixes a crash in the browser source when forcing a log off / shutdown with active outputs
• On Windows, adds explicit handling for WM_QUERYENDSESSION and WM_ENDSESSION which are sent during the log out process to inform apps the session is ending, and allow them to stall the process.

I have tried to keep my changes to the shutdown flow as minimal as possible, as it is particularly delicate and very sensitive to the order of operations. Testing on all operating systems will be valuable.

Motivation and Context

We've received a lot of great feedback on the Unclean Shutdown dialog after we removed the ability to suppress it. We believe that this dialog serves an important purpose but only when it is being shown accurately. This PR aims to address a majority of the cases where it was showing up unexpected or unintended.

How Has This Been Tested?

Painstakingly signing out of Windows multiple times with OBS running in various states.

Enabled the warning for confirmation on exit with outputs active. Ran OBS with no outputs active, with virtual camera active, and with virtual camera active then clicking Exit to display the confirmation prompt prior to log off. Then tested signing out in all scenarios, and both forcing the sign out to continue, or cancelling it.

When sign out is cancelled with outputs active, OBS stays running and displays the confirmation dialog if it was not showing already. When sign out continues, OBS starts normally the following time with no unclean shutdown warning.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Tweak (non-breaking change to improve existing functionality)

• Code cleanup (non-breaking change which makes code smaller or more readable)

Checklist:

• My code has been run through clang-format.
• I have read the contributing document.
• My code is not on the master branch.
• The code has been tested.
• All commit messages are properly formatted and commits squashed where appropriate.
• I have included updates to all appropriate documentation.

[Chiitoo]

Testing this on Gentoo Linux, it seems to be working fine, but I don't think I ever got to see the "unsafe shutdown detected" or so, even without it.

I have a way to prod a segmentation fault even (I promies I'll eventually actually report that one), and OBS Studio still starts just fine.

I haven't looked at the code, but I don't think it should be anything related to 'systemd' or something like that which I don't use, so it is a bit interesting to me.

It might be interesting if people who /do/ get the pop-up/logs for un-clean shutdowns explain how exactly they do get them.

In addition to the segfault, I've been sending OBS Studio the various signals ranging from 1 to 15, such as kill (9) and terminate (15) and nothing seems to prod up the message.

On that note, I think most if not all desktop environments will/should send the term (15) signal to all applications when shutting down, though I saw some discussion of 'systemd' also sending 'kill' (9) immediately after, but they may have made that better some time ago (I have no systems using 'systemd' to test right now).

In any case, thank you!

[Dankirk]

There is a potential application hang when experimental captions are enabled and trying to stop it. Fix candidate in #12729

[PatTheMav]

At least on macOS these changes seem to dangerously interfere with the orderly shutdown process of the app:

• When OBS is closed from the outside (by the OS), then
  1. commitData is called
  2. saveAll is called
  3. closeWindow is called via closeEvent
  4. saveAll is called again`
  5. OBSBasic's destructor runs (triggered by a QEvent::DeferredDelete event)
  6. The destructor crashes
• When OBS is closing itself, then
  1. closeWindow is called
  2. saveAll is called
  3. commitData is called
  4. saveAll is called again`
  5. closeWindow is called again (triggered by QApplication::aboutToQuit), but bails out early
  6. OBSBasic's destructor runs

Here's what STDERR gave me:

info: [OBSBasic] Destructor
libc++abi: terminating due to uncaught exception of type std::__1::system_error: mutex lock failed: Invalid argument
warning: QMetaObject::invokeMethod: No such method QObject::DoCefMessageLoop(int)

Confusingly enough the mutex in question is the log_mutex used by too_many_repeated_entries after the info message has been printed.

[m1neral-rocks]

@Warchamp7

Have you tested the above linked PR that is aimed at addressing the erroneous crash issues?

Sorry for pinging you twice. The other one got clogged by Joel's ego. My question was:

I started getting cmake errors a few days ago, it was compiling fine before. Is there an issue with dependencies relating to Qt::GuiPrivate or did I mess something up?

I want to try this pr but I feel like I'm missing something? I wasn't getting this dependency issue before.

[Warchamp7]

@Warchamp7

Have you tested the above linked PR that is aimed at addressing the erroneous crash issues?

Sorry for pinging you twice. The other one got clogged by Joel's ego. My question was:

I started getting cmake errors a few days ago, it was compiling fine before. Is there an issue with dependencies relating to Qt::GuiPrivate or did I mess something up?

I want to try this pr but I feel like I'm missing something? I wasn't getting this dependency issue before.

You can download artifacts compiled on CI of this PR:
https://github.com/obsproject/obs-studio/actions/runs/18130631287?pr=12668#artifacts

[PatTheMav]

Most recent pushed fixed the crash bug on macOS. Indeed QGuiApplicationPrivate::quit never seems to be executed at all (neither when terminated via macOS nor when quitting the app directly), probably the app is torn down before the invocation can take place.

[RytoEX]

Nits.

[RytoEX]

Squash the review commits and then this should be good to go. I would have accepted the extra CMake changes as a separate follow-up PR, but this is fine as a separate commit.

[RytoEX]

Seems fine. Much as I'd generally prefer not to include such a large change in a hotfix, I'd prefer that this change gets shipped to users sooner rather than later.