Skip to content

Conversation

Zenulous
Copy link

@Zenulous Zenulous commented Apr 3, 2025

Summary

This pull request addresses a rare edge case issue causing a thread deadlock with access to _optional_thread_lock in ConnectionPool. The solution involves changing ThreadLock to ThreadRLock to allow reentrant locking and resolve the deadlock.

We have encountered a rare deadlock issue in our production environment while using HTTP Core in a multithreaded setup. The issue manifests in rare cases as threads indefinitely waiting to acquire a lock, causing the entire worker to hang. This deadlock occurs when the same thread attempts to acquire the lock multiple times without releasing it.

This is important to meet the library's goal of being thread safe.

For more details, please refer to the discussion here.
This issue is also felt by other users of the lib although explained less clearly like in #997.

Checklist

  • I understand that this PR may be closed in case there was no previous discussion. (This doesn't apply to typos!)
  • I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change.
  • I've updated the documentation accordingly.

@encode encode deleted a comment from begilbert238 Apr 28, 2025
@Zenulous
Copy link
Author

Zenulous commented Jul 4, 2025

@tomchristie Is the team aware of this PR? I'd love to get an opinion on this PR because we still see this issue pop up. (#990)

@xiexianbin
Copy link

I tested it, and this pull request doesn't seem to work for #1029 ?

@Zenulous
Copy link
Author

Zenulous commented Sep 1, 2025

I tested it, and this pull request doesn't seem to work for #1029 ?

Could be, the issue we experience is not really related to large files so your problem likely has a different root cause

@lovelydinosaur
Copy link
Contributor

Unfortunately after quite some time trying we ourselves do not know the exact condition causing this.

I wouldn't make this change without a clear understanding of why a re-entrant lock would be required here.

Incidentally: The httpx 1..0 prelease has a simpler stack here.
I'd be more inclined to put my time into pushing that forward. https://www.encode.io/httpnext/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants