HTTP/2 Announce Support via libcurl #8025

TeholTheOnly · 2025-09-05T09:44:56Z

Introduces native HTTP/2 support for tracker announces through libcurl integration, enabling connection multiplexing and improved performance while maintaining full backward compatibility.

Key Changes

Added curl_tracker_client implementing tracker_client_interface for HTTP/2 announces
Implemented dedicated curl_thread_manager for async libcurl operations
Connection pooling and multiplexing for reduced overhead
Opt-in feature via settings_pack::use_libcurl_for_trackers
Comprehensive test suite including HTTP/2 test server

Performance

Reduction in connection overhead through multiplexing
SSL caching and reuse
30-40ms average latency improvement

Build

b2 use-libcurl=on

Breaking Changes

None - feature is disabled by default

…rectories

TeholTheOnly · 2025-09-05T22:21:17Z

Dedicated Thread vs ASIO integration

After prototyping both approaches, I chose a single-threaded libcurl design over direct ASIO integration due to fundamental socket handling incompatibilities that would have required extensive platform-specific code.

The Core Problem: Socket Descriptor Impedance Mismatch

Platform-Specific Socket Types

// Windows: SOCKET (UINT_PTR) vs ASIO's wrapped handles
curl_socket_t sock = ...; // Returns Windows SOCKET
// ASIO expects different wrapper, uses IOCP internally

// Unix: int fd vs ASIO's epoll/kqueue abstractions  
int fd = ...; // Raw file descriptor from curl
// ASIO manages its own epoll/kqueue state

Event Model Incompatibility

libcurl's curl_multi_socket_action requires precise per-socket event control that doesn't map cleanly to ASIO:

// libcurl needs: "Watch this socket for READ only, now WRITE only, now BOTH"
case CURL_POLL_IN:    // Register for read events only
case CURL_POLL_OUT:   // Register for write events only  
case CURL_POLL_REMOVE: // Unregister all events

// ASIO model: async operations own the socket
socket.async_read_some(...);  // Can't easily switch to write-only monitoring

Platform Event Systems Don't Mix

Each platform would need custom bridging code:

Windows: IOCP (ASIO) vs WSAEventSelect (curl) - incompatible event models
Linux: epoll with ASIO's reactor pattern vs curl's edge-triggered needs
macOS: kqueue filter management conflicts between libraries
BSD: Similar kqueue issues plus different socket option semantics

Socket Ownership Conflicts

// Race condition example:
// ASIO thinks it owns socket → schedules async_read
// libcurl closes socket → curl_easy_cleanup()  
// ASIO continues → crash on invalid socket

Pragmatic Solution

I chose a dedicated thread with curl_multi_wait/perform because:

Zero platform-specific code - Works identically everywhere
Clear ownership - One thread owns all curl resources
Proven Solution - Mirroring transmissions implementation

Trade-offs

One polling thread - Negligible CPU impact (<0.1% in profiling)

Conclusion

For tracker announces, the complexity of ASIO integration wasn't justified. The single-threaded design prioritizes correctness, portability, and maintainability over theoretical performance gains that might not materialize in practice.

arvidn · 2025-09-07T11:37:24Z

The two first commits don't belong to this PR. Are they a prerequisite for this to work?

arvidn

This is a big enough change to be making against the master branch, rather than RC_2_0. Please rebase it.

Also, it seems to be a lot more complex that it would need to be. I'm not done reviewing all of it yet though.

arvidn · 2025-09-07T11:39:07Z

include/libtorrent/aux_/mock_tracker_client.hpp

+namespace libtorrent::aux {
+
+// Mock implementation for testing
+class TORRENT_EXTRA_EXPORT mock_tracker_client {


This doesn't look like it ought to be part of the production code. I would have thought this would live under tests/. Is there a good reason to put it here?

arvidn · 2025-09-07T11:42:58Z

include/libtorrent/aux_/mock_tracker_client.hpp

+        announce(req, handler);
+    }
+
+    [[nodiscard]] bool can_reuse() const {


it really looks suspicious to have this feature just for tests. If the real curl_tracker_connection always is reusable, shouldn't the mock tracker connection mimic that behavior?
The production code where this returns true isn't necessarily well tested.

arvidn · 2025-09-07T11:49:55Z

include/libtorrent/aux_/curl_thread_manager.hpp

+class tracker_host_counter {
+private:
+    std::unordered_map<std::string, int> m_tracker_ref_counts;
+    mutable std::mutex m_mutex;


it doesn't look right that the counters have their own mutex. Shouldn't the counters be protected by the same mutex as the thing they are counting? i.e. the announce queue I assume.

arvidn · 2025-09-07T11:50:58Z

src/curl_thread_manager.cpp

+}
+
+void curl_thread_manager::wakeup_curl_thread() {
+    // Simple delegation to perform_wakeup for backward compatibility


backwards compatibility with what?

arvidn · 2025-09-07T11:51:33Z

src/curl_thread_manager.cpp

+    // FIX: Allow wakeup during shutdown to prevent deadlock
+    // The curl thread needs to wake up from curl_multi_poll() to check
+    // the shutdown flag and exit cleanly


this sounds important

arvidn · 2025-09-07T12:13:32Z

src/curl_thread_manager.cpp

+    // Wait for thread to be fully initialized with a timeout
+    // This prevents deadlock during session initialization when io_context isn't running yet
+    {
+        std::unique_lock<std::mutex> lock(manager->m_init_mutex);
+        // Wait up to 100ms for initialization
+        // If io_context isn't running yet, initialization will complete later
+        manager->m_init_cv.wait_for(lock, std::chrono::milliseconds(100), [&manager]{
+            return manager->m_init_status != InitStatus::Pending;
+        });
+
+        // Check if initialization failed (only if status changed from Pending)
+        if (manager->m_init_status == InitStatus::Failed) {
+            // Join the failed thread before throwing
+            if (manager->m_curl_thread.joinable()) {
+                manager->m_curl_thread.join();
+            }
+            throw std::runtime_error("Failed to initialize curl multi handle");
+        }
+        // If still pending, that's OK - initialization will complete asynchronously
+    }


this doesn't look right. std::thread will throw an exception if it fails. If it doesn't fail, the thread will be created. It seems quite unnecessary to handle the case where std::thread or the operating system isn't functioning properly.

arvidn · 2025-09-07T12:15:51Z

src/curl_thread_manager.cpp

+    }
+}
+
+std::vector<curl_request> curl_thread_manager::swap_pending_requests() {


this name does not describe what this function does.

I think what you want is, essentially:

std::unique_lock<std::mutex> lock(m_queue_mutex); return std::exchange(m_request_queue, std::vector<curl_request>());

But it's not clear why.

arvidn · 2025-09-07T12:17:30Z

src/curl_thread_manager.cpp

+        {
+            std::scoped_lock<std::mutex> lock(m_init_mutex);
+            m_init_status = InitStatus::Success;
+        }


why not just perform the initialization in the original thread instead?
Now you have to add synchronization just so the original thread can wait for the newly created one. I think it would be simpler if you just create the thread after the initialization is done.

and remove m_init_mutex and m_init_cv

arvidn · 2025-09-07T12:19:45Z

test/test_tracker_interface.cpp

@@ -0,0 +1,198 @@
+#include "test.hpp"
+#include "libtorrent/aux_/mock_tracker_client.hpp"


you should probably move the mock implementation into this test

arvidn · 2025-09-07T12:20:27Z

test/test_tracker_interface.cpp

+{
+    io_context ios;
+    settings_pack settings;
+    mock_tracker_client client(ios, settings);


what's the point of these tests? They don't seem to actually test any production code, just the mock object.

arvidn

I think it would be easier to review and land this in smaller pieces

arvidn · 2025-09-07T13:19:46Z

src/curl_tracker_connection.cpp

+		m_client->close();
+	}
+	cancel();
+	m_man.remove_request(static_cast<aux::http_tracker_connection const*>(this));


will this make the handler be called? If not, there won't be a way to synchronize the threads, is there?

arvidn · 2025-09-07T13:20:23Z

src/session_impl.cpp

 		, m_upload_rate(peer_connection::upload_channel)
 		, m_host_resolver(m_io_context)
+#ifdef TORRENT_USE_LIBCURL
+		, m_curl_thread_manager(curl_thread_manager::create(m_io_context, m_settings))


how come this isn't a member of tracker_manager instead?

arvidn · 2025-09-07T13:21:50Z

src/curl_tracker_connection.cpp

+	, m_curl_thread_manager(std::move(curl_mgr))
+{
+	aux::session_settings const& session_sett = m_man.settings();
+	settings_pack sett;


why are you making a copy of the settings here?

arvidn · 2025-09-07T13:24:18Z

include/libtorrent/settings_pack.hpp

+			// testing with self-signed certificates or when using a private
+			// certificate authority. The file must contain one or more CA
+			// certificates in PEM format. Only applies when using libcurl.
+			tracker_ca_certificate,


Does this only work for curl trackers? If so, that should be documented. If not, this seems like a great independent feature, that can be made as a separate PR.

arvidn · 2025-09-07T13:25:31Z

include/libtorrent/aux_/ssl_session_cache_interface.hpp

+namespace libtorrent::aux {
+
+// Non-template interface for SSL session caching
+// This allows C callbacks to interact with the templated connection pool


isn't curl managing the connection pool?

arvidn · 2025-09-07T14:03:24Z

src/curl_tracker_client.cpp

+	io_context&,
+	std::string const& url,
+	settings_pack const& settings,
+	std::shared_ptr<curl_thread_manager> curl_mgr)


passing in ownership of this object here creates a circular reference. I think this should be a raw reference

arvidn · 2025-09-07T14:04:33Z

src/curl_tracker_client.cpp

+	return scrape_url + "?" + build_tracker_query(req, true);
+}
+
+std::string curl_tracker_client::build_tracker_query(tracker_request const& req, bool scrape) const


this looks copy-pasted. it would be better to factor it out into a separate function

arvidn · 2025-09-07T14:05:08Z

src/curl_tracker_client.cpp

+}
+
+
+std::string curl_tracker_client::scrape_url_from_announce(std::string const& announce) const


this also looks like it should be a free function

arvidn · 2025-09-07T14:06:19Z

src/session_impl.cpp

+		// This ensures the worker thread is stopped before any member destruction begins,
+		// preventing dangling reference to m_settings
+		// C++17 style with if-initializer for cleaner scope management
+		if (auto manager = std::move(m_curl_thread_manager); manager)


Suggested change

if (auto manager = std::move(m_curl_thread_manager); manager)

if (auto manager = std::move(m_curl_thread_manager))

This has the same behavior, right?

arvidn · 2025-09-07T14:07:18Z

src/session_impl.cpp

+#ifndef TORRENT_DISABLE_LOGGING
+			session_log(" shutting down curl thread manager");
+#endif
+			manager->shutdown();


session_impl separates initiating shutdown from waiting for shutdown. I believe manager->shutdown() does both, right?

TeholTheOnly · 2025-09-08T20:07:44Z

I think it would be easier to review and land this in smaller pieces

Would you prefer me to go through and address the comments that you've made on this PR, or close this one out and start the process of making smaller PRs to land the main functionality of this and then slowly expand out? Just want to be mindful of your time, and I appreciate you taking a look!

loskutov · 2025-09-10T11:29:35Z

include/libtorrent/aux_/curl_handle_wrappers.hpp

+// Deleter for unique_ptr with custom cleanup
+struct curl_easy_deleter {
+    void operator()(CURL* handle) const noexcept {
+        if (handle) curl_easy_cleanup(handle);


unique_ptr doesn't invoke the deleter for nullptr anyways

seabashed · 2025-11-15T09:07:08Z

I'm really excited about this pull request, but noticed progress seems to have stalled. Is there any work happening behind the scenes that isn't visible in the repo?

FYI: rTorrent (and its libtorrent) recently adopted libcurl's modern connection reuse practices, which significantly reduces bandwidth usage. It would be fantastic to see similar improvements implemented here.

I wish I had the technical skills to contribute directly, but unfortunately that's beyond my current abilities. Still very much hoping to see this move forward!

xavier2k6 and others added 5 commits September 4, 2025 07:42

GHA CI: bumped checkout action to v5

49233cf

add test for torrent with two invalid directory names, merging two di…

2d107c8

…rectories

feat: http2 announces via libcurl

f10fc21

feat: c++ 17

964b18f

feat: CURL easy handle

d02bcbf

TeholTheOnly changed the title ~~HTTP/2 Tracker Support via libcurl~~ HTTP/2 Announce Support via libcurl Sep 5, 2025

fix: build auto-detect by default

2666ad3

TeholTheOnly force-pushed the 4334-libcurl-announces-master branch from 82432e3 to 2666ad3 Compare September 5, 2025 23:06

fix: resolve warnings

b962d2a

TeholTheOnly force-pushed the 4334-libcurl-announces-master branch from cc42017 to b962d2a Compare September 6, 2025 00:42

TeholTheOnly added 2 commits September 6, 2025 01:31

fix: test portability

cfd94bf

fix: test warnings

ff1b1a6

TeholTheOnly force-pushed the 4334-libcurl-announces-master branch from 2648005 to ff1b1a6 Compare September 6, 2025 03:07

arvidn reviewed Sep 7, 2025

View reviewed changes

loskutov reviewed Sep 10, 2025

View reviewed changes

This was referenced Oct 3, 2025

Announce optimization Tribler/tribler#8767

Closed

Announce optimization qbittorrent/qBittorrent#23339

Open

Announce optimization rakshasa/rtorrent#1595

Closed

		@@ -0,0 +1,198 @@
		#include "test.hpp"
		#include "libtorrent/aux_/mock_tracker_client.hpp"

		}


		std::string curl_tracker_client::scrape_url_from_announce(std::string const& announce) const

	if (auto manager = std::move(m_curl_thread_manager); manager)
	if (auto manager = std::move(m_curl_thread_manager))

Uh oh!

HTTP/2 Announce Support via libcurl #8025

Are you sure you want to change the base?

HTTP/2 Announce Support via libcurl #8025

Uh oh!

Conversation

TeholTheOnly commented Sep 5, 2025

Key Changes

Performance

Build

Breaking Changes

Uh oh!

TeholTheOnly commented Sep 5, 2025

Dedicated Thread vs ASIO integration

The Core Problem: Socket Descriptor Impedance Mismatch

Platform-Specific Socket Types

Event Model Incompatibility

Platform Event Systems Don't Mix

Socket Ownership Conflicts

Pragmatic Solution

Trade-offs

Conclusion

Uh oh!

arvidn commented Sep 7, 2025

Uh oh!

arvidn left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

arvidn left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

TeholTheOnly commented Sep 8, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

seabashed commented Nov 15, 2025

Uh oh!

Reviewers

Assignees

Labels