Skip to content

Commit be12983

Browse files
howardjohnseanmonstar
authored andcommitted
Fix race condition in connection termination
See hyperium/hyper#3652. What I have found is the final reference to a stream being dropped after the `maybe_close_connection_if_no_streams` but before the `inner.poll()` completes can lead to the connection dangling forever without any forward progress. No streams/references are alive, but the connection is not complete and never wakes up again. This seems like a classic TOCTOU race condition. In this fix, I check again at the end of poll and if this state is detected, wake up the task again. Wth the test in hyperium/hyper#3655, on my machine, it fails about 5% of the time: ``` 1876 runs so far, 100 failures (94.94% pass rate). 95.197349ms avg, 1.097347435s max, 5.398457ms min ``` With that PR, this test is 100% reliable ``` 64010 runs so far, 0 failures (100.00% pass rate). 44.484057ms avg, 121.454709ms max, 1.872657ms min ``` Note: we also have reproduced this using `h2` directly outside of `hyper`, which is what gives me confidence this issue lies in `h2` and not `hyper`.
1 parent 0d66e3c commit be12983

File tree

2 files changed

+13
-1
lines changed

2 files changed

+13
-1
lines changed

src/client.rs

+6-1
Original file line numberDiff line numberDiff line change
@@ -1428,7 +1428,12 @@ where
14281428

14291429
fn poll(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Self::Output> {
14301430
self.inner.maybe_close_connection_if_no_streams();
1431-
self.inner.poll(cx).map_err(Into::into)
1431+
let result = self.inner.poll(cx).map_err(Into::into);
1432+
if result.is_pending() && !self.inner.has_streams_or_other_references() {
1433+
tracing::trace!("last stream closed during poll, wake again");
1434+
cx.waker().wake_by_ref();
1435+
}
1436+
result
14321437
}
14331438
}
14341439

src/proto/connection.rs

+7
Original file line numberDiff line numberDiff line change
@@ -242,6 +242,13 @@ where
242242
}
243243
}
244244

245+
/// Checks if there are any streams or references left
246+
pub fn has_streams_or_other_references(&self) -> bool {
247+
// If we poll() and realize that there are no streams or references
248+
// then we can close the connection by transitioning to GOAWAY
249+
self.inner.streams.has_streams_or_other_references()
250+
}
251+
245252
pub(crate) fn take_user_pings(&mut self) -> Option<UserPings> {
246253
self.inner.ping_pong.take_user_pings()
247254
}

0 commit comments

Comments
 (0)