Heap consumption until crash

[gluca]

Platform

ESP32

IDE / Tooling

Arduino (IDE/CLI)

What happened?

Free memory heavy reduction after each client request until crash. The effect is huge using firefox as browser while is relatively small using chrome.
Each time you request any URL (in the example i made 2 hooks, one for "/" and one for all others (not found) pages) the free memory in heap reduces drastically, using firefox as client browser (roughly 5 to 6kbytes for each request). Using chrome the heap size reduction is less evident but still here (500 to 600 bytes with recovery), as you can see from Serial output i provided and that can be reproduced by my example code.

Stack Trace

No Stack trace until it crashes for out of memory. Output of main loop (showing memory degradation) follows:

Chrome (Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36)

Memory - Free:174424, Min:171912, Largest:110580
Memory - Free:174424, Min:171912, Largest:110580
Serving request for /-Memory - Free:172332, Min:167872, Largest:110580
Memory - Free:172332, Min:167872, Largest:110580
Memory - Free:172332, Min:167872, Largest:110580
Memory - Free:173500, Min:167872, Largest:110580
Memory - Free:173500, Min:167872, Largest:110580
Serving request for /-Memory - Free:172688, Min:167312, Largest:110580
Memory - Free:172688, Min:167312, Largest:110580
Memory - Free:172688, Min:167312, Largest:110580
Memory - Free:173272, Min:167312, Largest:110580
Memory - Free:173272, Min:167312, Largest:110580
Serving request for /-Memory - Free:171648, Min:166260, Largest:110580
Memory - Free:171648, Min:166260, Largest:110580
Memory - Free:171648, Min:166260, Largest:110580
Memory - Free:172816, Min:166260, Largest:110580
Memory - Free:172816, Min:166260, Largest:110580
Serving request for /-Memory - Free:172004, Min:166260, Largest:110580
Memory - Free:172004, Min:166260, Largest:110580
Memory - Free:172004, Min:166260, Largest:110580
Memory - Free:172588, Min:166260, Largest:110580
Memory - Free:172588, Min:166260, Largest:110580
Serving request for /-Memory - Free:171776, Min:166260, Largest:110580
Memory - Free:171776, Min:166260, Largest:110580
Memory - Free:171776, Min:166260, Largest:110580
Memory - Free:172360, Min:166260, Largest:110580
Memory - Free:172360, Min:166260, Largest:110580
Serving request for /-Memory - Free:171548, Min:166152, Largest:110580
Memory - Free:171548, Min:166152, Largest:110580
Memory - Free:171548, Min:166152, Largest:110580
Memory - Free:172132, Min:166152, Largest:110580
Memory - Free:172132, Min:166152, Largest:110580

Firefox: (Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:141.0) Gecko/20100101 Firefox/141.0)

Memory - Free:172132, Min:166152, Largest:110580
Memory - Free:172132, Min:166152, Largest:110580
Serving request for /-Memory - Free:166292, Min:163768, Largest:110580
Memory - Free:166292, Min:163768, Largest:110580
Memory - Free:166292, Min:163768, Largest:110580
Serving request for /-Memory - Free:160492, Min:158056, Largest:110580
Memory - Free:160492, Min:157964, Largest:110580
Memory - Free:160492, Min:157964, Largest:110580
Memory - Free:160720, Min:157964, Largest:110580
Serving request for /-Memory - Free:154880, Min:152356, Largest:110580
Memory - Free:154880, Min:152356, Largest:110580
Memory - Free:154880, Min:152356, Largest:110580
Serving request for /-Memory - Free:149040, Min:146516, Largest:110580
Memory - Free:149040, Min:146516, Largest:110580
Memory - Free:149040, Min:146516, Largest:110580

Minimal Reproductible Example (MRE)

#define WEB_SERVER_PORT 8080

#include <WiFi.h>
#include <AsyncTCP.h>
#include <ESPAsyncWebServer.h>

static AsyncWebServer _server(WEB_SERVER_PORT);

const char *ssid          = "mySSD";
const char *password = "myPWD";

void wsConnectAP(const char * ssid, const char * pwd) {
  Serial.println("Connecting to wifi");
  WiFi.begin(ssid,pwd);
  int status = WiFi.status();
  while(status!= WL_CONNECTED) {
    delay(1000);
    status = WiFi.status();
  }
  Serial.println("Connected");
}

void wsStartServer() {
  wsSetHandlers();
  _server.begin();
}

void wsSetHandlers() {

  _server.on("/",HTTP_GET,     
    [](AsyncWebServerRequest * request){
        Serial.print("Serving request for /-");
        request->send(200,"application/json","{\"message\":\"Hello World\"}");
    });

    _server.onNotFound(
      [] (AsyncWebServerRequest *request) {
        Serial.print("Serving request for notFound-");
        request->send(404, "application/json", "{\"message\":\"Not Found\"}");
      });
}

void setup() {
  Serial.begin(9600);
  while(!Serial);
  wsConnectAP(ssid, password);
  wsStartServer();
}

void loop() {
    multi_heap_info_t info;
    heap_caps_get_info(&info, MALLOC_CAP_INTERNAL | MALLOC_CAP_8BIT); // internal RAM, memory capable to store data or to create new task
    Serial.printf("Memory - Free:%u, Min:%u, Largest:%u\n",   
    info.total_free_bytes,
    info.minimum_free_bytes,
    info.largest_free_block);
    delay(1000);
}

I confirm that:

• I have read the documentation.
• I have searched for similar discussions.
• I have searched for similar issues.
• I have looked at the examples.
• I have upgraded to the lasted version of ESPAsyncWebServer (and AsyncTCP for ESP32).

[mathieucarbou]

@gluca : please format your pos accordingly next time to speed up the reading. We are all participating in our free time so anything helping us is appreciated. I took the liberty to reformat it.

[gluca]

I don't know what happened but only a small part of the code was properly formatted, Tank you for the changes. Each row is one second apart from the previous one. I will put some timing on each output but i can anticipate that some memory recovery happens when using chrome but (as per my tests) no recovery when using firefox and, most important, each query consume more than 5k of heap (compared to 400 bytes of chrome queries).

[mathieucarbou]

@gluca : how did you do your tests ? ESP-IDF / LWIP layer use to allocate for the network buffers which are only freed after some time.

We do some perf tests in the project with even concurrent requests and there are indeed some heap decrease during the test but the amount of heap recovers after it finishes (sometimes after some time).

please add timestamps and wait a little more.

I will move that to a support ticket.

[gluca]

Ok. I did some other tests.

First i used a js to let firefox polls the esp32 and heap free space was not consuming. So i added a browser query (randomly) and it did happen again (see dump in attached file). Logging is every 5 seconds, js query is every 10 seconds, Manual query is random.

Then I checked what firefox was asking differently when calling the url from the browser instead of JS. I found (hopefully) the culprit:

When calling from the browser, firefox also asks the server for the favicon but on https! It gets a Network_Error (esp32 is not configured to answer https) but something is happening on ESP side and heap free space decreases (and memory is never recovered, at least after 1500s from the last call).

Now I am preparing new test using js to ask for a resource on https using other browsers as well to have a confirmation of that behavior, but I suspect something happen in the handshake part for SSL that makes ESP32 to allocate some memory that get wasted.

output.txt

[gluca]

Ok. Here we are.

I've attached the arduino sketch that recreates the problem.
If you wanna test it, add a secrets.h file to the project for wifi credential (const char * ssid="yourssid"; and const char * password="yourpassword";).

After upload the sketch to the ESP and waited it connects to wifi, point any browser to http://<ESP_IP_ADDRESS>:8080/client.html
Monitoring Serial output will produce something similar to my output (attached).

Tests I've made with both Firefox and Chrome reveal the issue (now i am sure is an issue) firefox produces a way bigger memory leak, something around 6K per query, while chrome is around 1.3k per query.

Not sure if is a problem of ESPAsyncWebServer or of AsyncTCP library. I have last (in the arduino repos) version of both.

output.txt

testAsyncWebServer.ino.zip

[mathieucarbou]

Can you tell me exactly what are the dependencies you are using, and if you are using Platformio ? Or, if you are using Arduino IDE, how are you making sure that the libraries compiled are the most recent ones (asking since Arduino IDE has a really poor dependency management system). Thanks!

When calling from the browser, firefox also asks the server for the favicon but on https! I

How is that possible ? When you open port 8080, only 8080 is opened, so the ESP (more precisely AsyncTCP) does not answer on any other port. The browser would need to call https://ip:8080 to ask for a resource with ssl protocol on rhe same non-ssl port, which I doubt it does.

[gluca]

use the sketch i provided, is all there. When asking for a resource in http, on the background firefox try to get favicon on the same port using **HTTPS** protocol. The example i provided implements this behavior. It alternates http calls to https calls on the same port. As you can also see from the output i provided, on http calls memory is handled properly while trying https on same port leads to memory leaks, 6k using firefox and 2k using chrome. Platform iuse is arduino ide 2.3.6 and the libraries used are the last one available there: AsyncTCP 3.4.6 ESPAsyncWebServer 3.7.10 I guess problem arise when initiating ssl handshake (from client side) and something happen server side that produce that leak. Don't know if in AsyncTVP or ESPAsyncWebServer. You can also replicate the same behavior setting a js calling the server and asking any resource (even invalid one) with https instead of http (it also works using browser normally).

[mathieucarbou]

Bug can be fixed by updating AsyncTCP.cpp abort() function like this:

int8_t AsyncClient::abort() {
  if (_pcb) {
    _tcp_abort(&_pcb, this);
    // _pcb is now NULL
  }
  if (_discard_cb) {
    // _pcb was closed here
    _discard_cb(_discard_cb_arg, this);
  }
  return ERR_ABRT;
}

[mathieucarbou]

Opened a fix PR: ESP32Async/AsyncTCP#81
Waiting for review from @willmmiles...

[gluca]

Why the heck FF is firing an SSL handshake when not told to do so lol !

Don't really know it, FF does many weird things behind the blinds. Let's see it positively. It helped in (probably) find a regression

[gluca]

@gluca : could you please try with AsyncTCP 3.4.5 (previous version). I cannot reproduce the issue with the previous version.

I confirm that probably is a regression. Using AsyncTCP 3.4.5 it works as expected. No memory leaks

[mathieucarbou]

Why the heck FF is firing an SSL handshake when not told to do so lol !

Don't really know it, FF does many weird things behind the blinds. Let's see it positively. It helped in (probably) find a regression

Yes!
And Thanks you for reporting it, and insisting! All our tests are mainly partly automated and with curl and do not focus on testing weird scenarios like ssl handshake over http lol. So this behavior was uncaught.

[gluca]

I am a grumpy old (ex) developer... No way I desist :-D

[mathieucarbou]

Will be fixed in main once PR #255 will be merged

[gluca]

Teested with AsyncTcp 3.4.7 and it works as expected