Add support for opt-in panic hooking

[Jake-Shadle]

It'd be nice to allow users to opt-in to panic handling that could also be reported with a minidump the same as crashes.

[ReactorScram]

Yeah right now I'm seeing that if I panic at any point after setting up crash handling, it leaks the server process? Maybe I've set something up wrong?

[ReactorScram]

nvm just found the docs for stale connections https://docs.rs/minidumper/latest/minidumper/struct.Server.html#method.run

Edit: on_client_disconnected seems to work, I told it to exit if there's 0 clients left connected. https://docs.rs/minidumper/latest/minidumper/trait.ServerHandler.html#method.on_client_disconnected

[P1n3appl3]

I'm currently trying to use minidumper in zed for both crash and panic reporting, and I'm interested in implementing support for panic-hook based minidump generation. I've got a few questions that I've run into while attempting to manually implement the feature on our end:

• Would you want the panic hook to generate the minidump in-process, or simply shell out to a server like is recommended for crashes?
  • On one hand, panics generally don't mean that the process is coming down around you like some signals do, so I'm assuming it'd be reasonably reliable to do the minidump generation in-process and avoid the complexity of the client/server architecture.
  • On the other hand, minidumper currently seems to have no support for using MinidumpWriter::dump_local_context, and that functionality is only implemented for windows/mac, not linux.
• The docs for Client::request_dump indicate that you should suspend all threads before requesting a dump on macos. Is that still the case during a panic or was it only relevant while the application was crashing? I've started cobbling something together with mach2::task_threads but it'd be great to see an example of someone doing it correctly.
• Since requesting a minidump requires a CrashContext, what would the best way of getting one of those from a panic hook be?
  • My current approach has been to raise a signal manually with crash_handler::make_crash_event inside the hook, but the struct fields are pub so it doesn't seem too hard to manually fabricate one right in the panic. Is either of these approaches desirable over the other?
• Is there any way to attach the panic message to the minidump? Our current approach is to write the message (as well as a couple other relevant pieces of information like a unique ID for the current machine) to disk out of band, and then when we upload/process the crash later we associate the minidump file with the json file containing extra data. It'd be great if there were some way to embed them directly in the minidump to have more self-contained crash info.
• Some sort of synchronization is needed if your application wants to use the same Client in both the crash handler AND panic hook. Is there a better strategy than take()ing a global static Mutex<Option<Client>> that's initialized lazily on application startup?
  • Does any special consideration need to be given to the fact that you can panic multiple times?
  • Right now I'm using a simple AtomicBool to avoid requesting minidumps more than once, which is acceptable for my application because we abort in our custom panic hook anyways. In the general case I imagine a more robust solution is warranted.
  • In the same vein many types of panics also generate signals which trigger the crash handler. For example running unsafe { *(0 as *const u64) } with a crash handler, a panic hook, and some println's gives:

running program
ran panic hook
thread caused non-unwinding panic. aborting.
ran crash handler

• Am I right in thinking that the panic strategy (unwind vs. abort) shouldn't change anything about generating a minidump in the panic hook, since the hook runs before any other panic machinery?
  • Once concern I have is that tools like symbolic seem to rely on the .eh_frame section of the binary to do backtrace symbolication, and panic=abort removes this since it's not needed at runtime. I'm not entirely clear on whether symbol files have all the relevant information even if generated from a build using panic=abort, but in any case this is more a problem with handling minidumps than generating them.

Sorry that's more than "a few" questions now that I look back over it 😅. Anyways I'll continue to hack on this and land at least one approach in zed's code base, but I'd love to encode best practices with an easy to use api so people don't have to do as much work to get minidumps from panics out of the box. Thanks for all the documentation on this project btw, it's been quite pleasant to work with so far.

[P1n3appl3]

We've landed support for this feature in zed-industries/zed#35263.

Here's the code we call in our panic handler. It waits up to 500ms for the crash handler process to start up and establish the socket connection, and it goes through the crash handler to request the dump.

@Jake-Shadle if you're interested in moving some of this functionality into the minidumper library then I'm happy to chat about that here. Given how many options there are for how you implement panic hooking + dump requests, maybe it'd make more sense to just add an example to the crate rather than library code that bakes in assumptions about what the user wants.