Memory allocation in curl()

[jeroen]

See: r-lib/httr2#704 (comment). TLDR: reading from non blocking connections takes a lot of memory. The bench::mark shows that this is memory allocated by R in readBin.

The problem is that for non blocking connections, it is possible to call readBin() a million times without any data coming in. This results to high memory usage and busy waiting.

stream_data <- function() {
  url <- "https://jeroen.github.io/data/nycflights13.json"
  con <- curl::curl(url)
  open(con, "rb", blocking = FALSE)
  on.exit(close(con))
  
  while(isIncomplete(con)) {
    readBin(con, raw(), 10 * 1024)
  }
}

bm <- bench::mark(stream_data(), iterations = 1, filter_gc = FALSE)

[jeroen]

There is no real bug, but perhaps it can be improved.

For regular blocking connections, curl::curl() seems to be faster than base::url() and allocate the same memory, which matches the total size of the downloaded file.

stream_data <- function() {
  example <- "https://jeroen.github.io/data/nycflights13.json"
  con <- curl::curl(example, open = 'rb')
  # con <- base::url(example, open = 'rb')
  on.exit(close(con))
  total <- 0
  while(length( buf <- readBin(con, raw(), 10 * 1024))) {
    total <- total + length(buf)
    cat(sprintf("\rDownloaded %d bytes", total), file = stderr())
  }
}

bm <- bench::mark(stream_data(), iterations = 1, filter_gc = FALSE)

I think base::url() does not support isIncomplete() so we can't compare the non-blocking case?

But I suspect that the issue for non-blocking connections is simply that each call to readBin() reserves n (e.g. 10 * 1024) bytes, even though most of the calls to readBin will return much less, often zero bytes. So we end up making millions of calls to readBin(), pre-allocating 10k every time, even when no data is coming in.

One solution would be to use curl_multi_wait even in the case of non-blocking connections, to limit the number of times per second that readBin can be invoked when no new data is coming in.