Migrating FrostWire to libtorrent 2.0 (jlibtorrent) getting random crashes when downloading, creating torrents, is try_signal the culprit?

[gubatron]

The notes on upgrading to libtorrent 2.0 mention

In libtorrent 2.0, the default disk I/O uses memory mapped files, which means a lot more of what used to belong in the disk caching subsystem is now handled by the kernel. This greatly simplifies the disk code and also has the potential of making a lot more efficient use of modern disks as well as physical memory.

In Java land with our jlibtorrent (C++ to Java wrapper) we get random silent crashes but eventually managed to get verbosity up and found this:

Warning: SIGSEGV handler modified!

Warning: SIGBUS handler modified!

Signal Handlers:

   SIGSEGV: boost::asio::execution_context::service* boost::asio::detail::service_registry::create<boost::asio::detail::resolver_service<boost::asio::ip::udp>, boost::asio::io_context>+176368 in jlibtorrent-18410287319277403849.dylib, mask=00000000000000000000000000000000, flags=SA_SIGINFO, unblocked

  *** Handler was modified!

  *** Expected: javaSignalHandler in libjvm.dylib, mask=11100110100111111111111111111111, flags=SA_ONSTACK|SA_RESTART|SA_SIGINFO

    SIGBUS: boost::asio::execution_context::service* boost::asio::detail::service_registry::create<boost::asio::detail::resolver_service<boost::asio::ip::udp>, boost::asio::io_context>+176368 in jlibtorrent-18410287319277403849.dylib, mask=00000000000000000000000000000000, flags=SA_SIGINFO, unblocked

  *** Handler was modified!

  *** Expected: javaSignalHandler in libjvm.dylib, mask=11100110100111111111111111111111, flags=SA_RESTART|SA_SIGINFO

    SIGFPE: javaSignalHandler in libjvm.dylib, mask=11100110000111110111111111111111, flags=SA_RESTART|SA_SIGINFO, unblocked

   SIGPIPE: javaSignalHandler in libjvm.dylib, mask=11100110000111110111111111111111, flags=SA_RESTART|SA_SIGINFO, unblocked

   SIGXFSZ: javaSignalHandler in libjvm.dylib, mask=11100110000111110111111111111111, flags=SA_RESTART|SA_SIGINFO, unblocked

    SIGILL: javaSignalHandler in libjvm.dylib, mask=11100110000111110111111111111111, flags=SA_RESTART|SA_SIGINFO, unblocked

   SIGUSR2: SR_handler in libjvm.dylib, mask=00000000000000000000000000000000, flags=SA_RESTART|SA_SIGINFO, unblocked

    SIGHUP: UserHandler in libjvm.dylib, mask=11100110000111110111111111111111, flags=SA_RESTART|SA_SIGINFO, unblocked

    SIGINT: UserHandler in libjvm.dylib, mask=11100110000111110111111111111111, flags=SA_RESTART|SA_SIGINFO, unblocked

   SIGTERM: UserHandler in libjvm.dylib, mask=11100110000111110111111111111111, flags=SA_RESTART|SA_SIGINFO, unblocked

   SIGQUIT: UserHandler in libjvm.dylib, mask=11100110000111110111111111111111, flags=SA_RESTART|SA_SIGINFO, blocked

   SIGTRAP: SIG_DFL, mask=00000000000000000000000000000000, flags=none, unblocked

Consider using jsig library.

I looked for possible changes in the way these signals are handled and found dep/try_signal and I suppose libtorrent's new default Disk IO is different than that of 1.2.

Are there other signals remapped, like when handling UDP packets? (perhaps hinted by this specific crash log)

Am I on the right track?

[gubatron]

I've added the recommended libjsig, the signal handling switches warnings no longer appear and now I can get proper crash logs caused by native calls to libtorrent on our UI thread, which are hopefully issues i need to fix in my application layer due to our wrongful use of the new 2.0 API.

But just out of curiosity and to rule out nmap issues, I'll see if there's a other disk handler available closer to 1.2s and see if that's truly the culprit of the crash.

[arvidn]

yes, libtorrent needs to handle certain signals during disk I/O operations. When using memory mapped files, disk writes are made by copying bytes into memory regions. There is no error code return path for memcopy, so the only way the kernel can signal errors (such as EPERM or ENOSPC) is via signals. When libtorrent is copying data into memory mapped files, it installs a temporary signal handle for the thread, and uninstalls it immediately afterwards.

I believe the JVM also installs signal handlers for managing its garbage collection, so it sounds plausible that there's some clash. Perhaps libtorrent is failing to properly restore the previous signal handler?

[gubatron]

Thank you so much for the clarification Arvid, hopefully soon we'll announce FrostWire 7 powered by libtorrent 2.0.

[gubatron]

So, other than disk IO done with mmap and having to hack the signal handlers to catch errors, is there any other mechanism in libtorrent 2.0 that's rewriting the signal handlers?

It's running perfectly in windows, which I imagine uses the posix disk IO.

I've tried creating session parameters using posix disk IO to see if the crashes go away, I can now create torrents, seed, and I have no crashes I can replicate that way, but just leaving the client running for a few minutes seeding can yield random crashes.

When I turn off libjsig (to mitigate the issues of signal handling conflicts, say when doing garbage collection like you mentioned) I can see the crashes are still due to signal handling issues, here's the latest one I saw:

Signal Handlers:
   SIGSEGV: boost::asio::execution_context::service* boost::asio::detail::service_registry::create<boost::asio::detail::resolver_service<boost::asio::ip::udp>, boost::asio::io_context>+173368 in jlibtorrent-4754436706698437257.dylib, mask=00000000000000000000000000000000, flags=SA_SIGINFO, unblocked
  *** Handler was modified!
  *** Expected: javaSignalHandler in libjvm.dylib, mask=11100110100111111111111111111111, flags=SA_ONSTACK|SA_RESTART|SA_SIGINFO
    SIGBUS: boost::asio::execution_context::service* boost::asio::detail::service_registry::create<boost::asio::detail::resolver_service<boost::asio::ip::udp>, boost::asio::io_context>+173368 in jlibtorrent-4754436706698437257.dylib, mask=00000000000000000000000000000000, flags=SA_SIGINFO, unblocked
  *** Handler was modified!
  *** Expected: javaSignalHandler in libjvm.dylib, mask=11100110100111111111111111111111, flags=SA_RESTART|SA_SIGINFO
    SIGFPE: javaSignalHandler in libjvm.dylib, mask=11100110000111110111111111111111, flags=SA_RESTART|SA_SIGINFO, unblocked
   SIGPIPE: javaSignalHandler in libjvm.dylib, mask=11100110000111110111111111111111, flags=SA_RESTART|SA_SIGINFO, unblocked
   SIGXFSZ: javaSignalHandler in libjvm.dylib, mask=11100110000111110111111111111111, flags=SA_RESTART|SA_SIGINFO, unblocked
    SIGILL: javaSignalHandler in libjvm.dylib, mask=11100110000111110111111111111111, flags=SA_RESTART|SA_SIGINFO, unblocked
   SIGUSR2: SR_handler in libjvm.dylib, mask=00000000000000000000000000000000, flags=SA_RESTART|SA_SIGINFO, unblocked
    SIGHUP: UserHandler in libjvm.dylib, mask=11100110000111110111111111111111, flags=SA_RESTART|SA_SIGINFO, unblocked
    SIGINT: UserHandler in libjvm.dylib, mask=11100110000111110111111111111111, flags=SA_RESTART|SA_SIGINFO, unblocked
   SIGTERM: UserHandler in libjvm.dylib, mask=11100110000111110111111111111111, flags=SA_RESTART|SA_SIGINFO, unblocked
   SIGQUIT: UserHandler in libjvm.dylib, mask=11100110000111110111111111111111, flags=SA_RESTART|SA_SIGINFO, blocked
   SIGTRAP: SIG_DFL, mask=00000000000000000000000000000000, flags=none, unblocked

I wonder if there's something you are doing with UDP that's similar:
boost::asio::execution_context::service* boost::asio::detail::service_registry::create<boost::asio::detail::resolver_service<boost::asio::ip::udp>, boost::asio::io_context>

boost::asio::ip::udp?

[arvidn]

libtorrent 2.0 uses memory mapped files on windows as well. However, the error handling there is done via structured exceptions, which may not interfere with the JVM memory manager.

[gubatron]

nvm, I believe it's something with set_piece_hashes that might be using the new mmap disk io, still getting crashes creating torrents without adding them to a session.

[gubatron]

Looks like that might be it, leaving these comments for others that may have this issue in the future on macos

	TORRENT_EXPORT void set_piece_hashes(create_torrent& t, std::string const& p
		, std::function<void(piece_index_t)> const& f, error_code& ec);
	TORRENT_EXPORT void set_piece_hashes(create_torrent& t, std::string const& p
		, settings_interface const& settings
		, std::function<void(piece_index_t)> const& f, error_code& ec);
	TORRENT_EXPORT void set_piece_hashes(create_torrent& t, std::string const& p
		, settings_interface const& settings, disk_io_constructor_type disk_io
		, std::function<void(piece_index_t)> const& f, error_code& ec);

My code was wrapping this version of set_piece_hashes

	TORRENT_EXPORT void set_piece_hashes(create_torrent& t, std::string const& p
		, std::function<void(piece_index_t)> const& f, error_code& ec);

which in turn calls the default disk_io_constructor

void set_piece_hashes(create_torrent& t, std::string const& p
		, std::function<void(piece_index_t)> const& f, error_code& ec)
	{
		aux::session_settings sett;
		set_piece_hashes(t, p, sett, f, ec);
	}

	void set_piece_hashes(create_torrent& t, std::string const& p
		, settings_interface const& sett
		, std::function<void(piece_index_t)> const& f, error_code& ec)
	{
	        // TADA!
		set_piece_hashes(t, p, sett, default_disk_io_constructor, f, ec);
	}

Will confirm later if a set_piece_hashes_posix_disk_io will solve the issue on macos 15

[gubatron]

That was it.

I've added a new set_piece_hashes_posix_disk_io for jlibtorrent which finally made the crash go away.