Send window update on STREAM_DATA_BLOCKED rx?

[mxinden]

If I understand the code below correctly, Quiche currently ignores incoming StreamDataBlocked frames, i.e. ignores if a sender signals that it is blocked on the provided stream data flow control window:

quiche/quiche/src/lib.rs

Line 8003 in b3af644

frame::Frame::StreamDataBlocked { .. } => (),

If that is correct, have you considered sending a stream flow control update instead? More specifically call FlowControl::max_data_next and send a window update:

quiche/quiche/src/flowcontrol.rs

Lines 95 to 98 in b3af644

	/// Returns the new max_data limit.
	pub fn max_data_next(&self) -> u64 {
	self.consumed + self.window
	}

Maybe additionally call FlowControl::autotune_window in case the BDP is larger than the stream window.

---

As a sender Firefox's QUIC stack will send a STREAM_DATA_BLOCKED when attempting to send more than the granted stream window allows. I.e. it will send notify the receiver ahead of time that it will be blocked soon.

https://github.com/mozilla/neqo/blob/98b69aabc1852b65242d59c5775e3b167e8f2a8d/neqo-transport/src/send_stream.rs#L1218-L1242

As a receiver Firefox's QUIC stack will send a flow control update when it receives a STREAM_DATA_BLOCKED:

https://github.com/mozilla/neqo/blob/98b69aabc1852b65242d59c5775e3b167e8f2a8d/neqo-transport/src/recv_stream.rs#L758-L765

Before sending it will auto-tune the window:

https://github.com/mozilla/neqo/blob/98b69aabc1852b65242d59c5775e3b167e8f2a8d/neqo-transport/src/fc.rs#L382

---

This showed up while benchmarking Firefox's QUIC stack using h3.speed.cloudflare.com.

Throughout a transfer, not just during the initial BDP auto tuning ramp up, Firefox is repeatedly blocked on the stream flow control limit. Example:

$ RUST_LOG=neqo_bin=info cargo run --bin neqo-client -- -m=POST --stats --upload-size=100000000 https://h3.speed.cloudflare.com/__up?measId=XXX

6.193 INFO stats for Client ...
  rx: 14582 drop 3 dup 0 saved 3
  tx: 73061 lost 172 lateack 0 ptoack 0 unackdrop 0
  pmtud: 6 sent 3 acked 3 lost 0 change 1500 iface_mtu 1470 pmtu
  resumed: false
  frames rx:
    crypto 6 done 1 token 0 close 0
    ack 14567 (max 73051) ping 569 padding 0
    stream 5 reset 0 stop 0
    max: stream 0 data 14 stream_data 24
    blocked: stream 0 data 0 stream_data 0
    datagram 0
    ncid 0 rcid 0 pchallenge 0 presponse 0
    ack_frequency 0
  frames tx:
    crypto 4 done 0 token 0 close 1
    ack 590 (max 14842) ping 7 padding 6
    stream 72950 reset 0 stop 0
    max: stream 1 data 0 stream_data 0
    blocked: stream 0 data 0 stream_data 21
    datagram 0
    ncid 0 rcid 0 pchallenge 0 presponse 0
    ack_frequency 0
  ecn:
    tx:
      Initial Count({NotEct: 4, Ect1: 0, Ect0: 0, Ce: 0})
      Handshake Count({NotEct: 2, Ect1: 0, Ect0: 0, Ce: 0})
      Short Count({NotEct: 73036, Ect1: 0, Ect0: 19, Ce: 0})
    acked:
    rx:
      Initial Count({NotEct: 4, Ect1: 0, Ect0: 0, Ce: 0})
      Handshake Count({NotEct: 3, Ect1: 0, Ect0: 0, Ce: 0})
      Short Count({NotEct: 14572, Ect1: 0, Ect0: 0, Ce: 0})
    path validation outcomes: ValidationCount({Capable: 0, NotCapable(BlackHole): 0, NotCapable(Bleaching): 1, NotCapable(ReceivedUnsentECT1): 0})
    mark transitions:
  dscp: Cs0: 14585 

Note the blocked: stream 0 data 0 stream_data 21, which means that Firefox has been blocked 21 times on stream flow control. Also note the low number of stream window updates sent by Cloudflare max: stream 0 data 14 stream_data 24, i.e. 24.

---

Caught by @larseggert 🙏
Related: #1384
Side note: Thank you for h3.speed.cloudflare.com! Very helpful.

[LPardue]

Taking any automated action in response to a X_DATA_BLOCKED frame seems like a bad idea IMO. Thats an easy way to surprise users of the library and trigger DoS via unexpected resource commit.

I agree there's room for improvement for the upload performance of the speed test service but the answer is probably just for the application to enable auto tuning explicitly itself, rather than build more smarts into quiche.

[mxinden]

Can you expand on these 3 points @LPardue. Sorry in case I am missing something obvious.

Thats an easy way to surprise users of the library

Say quiche grants a sender a window of 1 MB. Say the sender used up 250 KB and has 1 more MB to send (wants to send total of 1.25 MB). Say that the sender signals this fact to quiche via STREAM_DATA_BLOCKED. Say that the quiche user (i.e. consumer) consumed the already sent 250 KB. Why is it surprising to the user if quiche grants the sender the full window again just a bit early than usual, i.e. sends a MAX_STREAM_DATA with Maximum Stream Data at 1.25 KB, thus granting the original 1 MB window again?

and trigger DoS via unexpected resource commit.

Quiche already granted the sender the same size window before. Why not do so again, just a bit earlier. All under the premise that the data consumer has consumed the 250 KB in the meantime.

I doubt there is an amplification attack here. Even if, you could add a condition to only reply to a STREAM_DATA_BLOCKED when say 10% of the window have been used. Again, I am not suggesting to break backpressure, i.e. always make sure data has been consumed by the user.

If I understand correctly, currently Quiche will only send an update once half of the window has been consumed:

https://github.com/cloudflare/quiche/blob/4f633848683fd52f6c3f4d8ee873bd82adc0da3b/quiche/src/flowcontrol.rs#L92C1-L92C45

If the current window is not >= the current BDP, the window update might not arrive in time at the sender.

for the application to enable auto tuning explicitly itself, rather than build more smarts into quiche.

I am confused. Is FlowControl::autotune_window disabled on h3.speed.cloudflare.com? Or are you suggesting h3.speed.cloudflare.com should more agressively do auto tuning?

[LPardue]

I am confused. Is FlowControl::autotune_window disabled on h3.speed.cloudflare.com? Or are you suggesting h3.speed.cloudflare.com should more agressively do auto tuning?

Apologies, I misremembered how the flow control autotuning worked (I swear it was a enable/disable config option at some point, but I might be mistaken). Today, the config of autotuning depends on max_connection_window and max_stream_window defined in https://github.com/cloudflare/quiche/blob/master/quiche/src/lib.rs#L842-L843. There's opportunity for services to tweak these numbers themselves. There's also opportunity to tweak the flow control algorithm via further quiche library changes.

Say quiche grants a sender a window of 1 MB. Say the sender used up 250 KB and has 1 more MB to send (wants to send total of 1.25 MB). Say that the sender signals this fact to quiche via STREAM_DATA_BLOCKED. Say that the quiche user (i.e. consumer) consumed the already sent 250 KB. Why is it surprising to the user if quiche grants the sender the full window again just a bit early than usual, i.e. sends a MAX_STREAM_DATA with Maximum Stream Data at 1.25 KB, thus granting the original 1 MB window again?

Because that's not how it works today and any change would be surprising. For instance, https://datatracker.ietf.org/doc/html/rfc9000#section-4.2-3 discusses the tradeoffs between X_MAX_DATA frames and connecton overheads.

I'm still failing to fully understand the issue here though. If I grok the client output correctly, you sent 72950 STREAM frames and 21 STREAM_DATA_BLOCKED frames. That's about 0.03% of the frames. I might simply be that tweaking of the autotuning params noted about would reduce this. However, I'm curious what you think is a good target percentage.

The RFC states:

A sender SHOULD send a STREAM_DATA_BLOCKED frame (type=0x15) when it wishes to send data but is unable to do so due to stream-level flow control.

At the point you send STREAM_DATA_BLOCKED, are you actually blocked or not?

quiche decides to grant flow control based on what is sent, not what a client claims. It sounds like in this case you have a static file to upload so that aspect is not an application limitiation. It would be interesting to know more specific details about the relationship between your link capability, congestion window and flow control window in your test. I.e. is the flow control triggering you congestion controller to be app limited

[larseggert]

IIRC, any STREAM_DATA_BLOCKED means we are waiting for an RTT before we can send more data, which at high throughputs is quite a bit. Are you saying you are expecting the client to send STREAM_DATA_BLOCKED before it's actually blocked?

[mxinden]

Addressing these two comments from the two of you:

A sender SHOULD send a STREAM_DATA_BLOCKED frame (type=0x15) when it wishes to send data but is unable to do so due to stream-level flow control.

At the point you send STREAM_DATA_BLOCKED, are you actually blocked or not?

Are you saying you are expecting the client to send STREAM_DATA_BLOCKED before it's actually blocked?

Firefox will proactively, i.e. ahead of time, be sending STREAM_DATA_BLOCKED frames, when having more data to send than the current stream data window allows. Unless CC limited, it will then spend the remaining window in less than half an RTT, thus blocked until a window update arrives. In the meantime it will be blocked.

If I grok the client output correctly, you sent 72950 STREAM frames and 21 STREAM_DATA_BLOCKED frames. That's about 0.03% of the frames.

I think this ratio (# STREAM / # BLOCKED) is misleading. E.g. say a client is blocked for 1 RTT, that would result in 1 STREAM_DATA_BLOCKED frame, but potentially many unsent STREAM frames, i.e. there is no 1-to-1 replacement of the two.

It would be interesting to know more specific details about the relationship between your link capability, congestion window and flow control window in your test. I.e. is the flow control triggering you congestion controller to be app limited

I will provide more data, ideally a qlog. Thank you for the input thus far!

[LPardue]

I think this ratio (# STREAM / # BLOCKED) is misleading. E.g. say a client is blocked for 1 RTT, that would result in 1 STREAM_DATA_BLOCKED frame, but potentially many unsent STREAM frames, i.e. there is no 1-to-1 replacement of the two.

Its a bit janky I agree but that's all the data we have now. Measuring in terms of the time spent blocked, or the opportunity lost by being flow control blocked, via more-specific logs would help us rationalize.

I might just be the case that tuning the flow control threshold that we decide to send a window update (for instance, as a config param) addresses the underlying issue, without needing the complication of processing optional client signals. I.e. if the application is reading data anyway, then it could be more generous and timely with updates if it so chooses.