Patch for http.get only return after timeout

[jorgeluismireles]

@spytheman
@JalonSolov

I open a discussion here about issue #20283 here because

• I have a working patch for my particular case http.get only return after timeout #20283 (comment)
• I don't know what impact would have this patch in general and how to test or add tests for it.
• Wan't a preliminary revision about the ideas about the patch.

diff --git a/vlib/net/http/request.v b/vlib/net/http/request.v
index 914bd26..22e6a5f 100644
--- a/vlib/net/http/request.v
+++ b/vlib/net/http/request.v
@@ -302,16 +302,16 @@ fn (req &Request) receive_all_data_from_cb_in_builder(mut content strings.Builde
        for {
                readcounter++
                len := receive_chunk_cb(con, bp, bufsize) or { break }
+               new_len = old_len + u64(len)
                $if debug_http ? {
-                       eprintln('ssl_do, read ${readcounter:4d} | len: ${len}')
+                       eprintln('ssl_do, read ${readcounter:4d} | len: ${len} | new_len: ${new_len} | expected_size: ${expected_size}')
                        eprintln('-'.repeat(20))
                        eprintln(unsafe { tos(bp, len) })
                        eprintln('-'.repeat(20))
                }
-               if len <= 0 {
+               if len <= 0 || (expected_size != 0 && expected_size <= new_len) {
                        break
                }
-               new_len = old_len + u64(len)
                // Note: `schunk` and `bchunk` are used as convenient stack located views to the currently filled part of `buff`:
                schunk := unsafe { bp.vstring_literal_with_len(len) }
                mut bchunk := unsafe { bp.vbytes(len) }
@@ -331,16 +331,16 @@ fn (req &Request) receive_all_data_from_cb_in_builder(mut content strings.Builde
                        }
                }
                body_so_far := u64(new_len) - body_pos
-               if req.on_progress_body != unsafe { nil } {
-                       if expected_size == 0 {
-                               lidx := schunk.index('Content-Length: ') or { -1 }
-                               if lidx > 0 {
-                                       esize := schunk[lidx..].all_before('\r\n').all_after(': ').u64()
-                                       if esize > 0 {
-                                               expected_size = esize
-                                       }
+               if expected_size == 0 {
+                       lidx := schunk.index('Content-Length: ') or { -1 }
+                       if lidx > 0 {
+                               esize := schunk[lidx..].all_before('\r\n').all_after(': ').u64()
+                               if esize > 0 {
+                                       expected_size = esize
                                }
                        }
+               }
+               if req.on_progress_body != unsafe { nil } {
                        req.on_progress_body(req, bchunk, body_so_far, expected_size, status_code)!
                }
                if !(req.stop_copying_limit > 0 && new_len > req.stop_copying_limit) {

The patch consist in

• Move calculation of expected_size outside branch if req.on_progress_body != unsafe { nil } { otherwise remains in zero.
• Break after expected_size != 0 && expected_size <= new_len)
• Add to debug new_len and expected_size.

[jorgeluismireles]

Even though the large waiting time can be mitigated if we can reduce max_retries (only for not responding services) and timeouts, I think only using prepare and do:

...
    mut req := http.prepare(url: page) or {
        h.get_errs << ip
        h.println(ip, '-', '-', sw.elapsed(), 'prepare error:${err}')
        return
    }
    req.max_retries = 1
    req.read_timeout = 5 * time.second
    req.write_timeout = 5 * time.second
    resp := req.do() or {
        h.get_errs << ip
        h.println(ip, '-', '-', sw.elapsed(), 'request error:${err}')
        return
    }
    if resp.status_code == 200 {
...