Mesh nodes lose connection to AP after 12 hr and enter unbreakable loop (AEGHB-1093)

[BR-Coding-cmd]

Checklist

• Checked the issue tracker for similar issues to ensure this is not a duplicate.
• Provided a clear description of your suggestion.
• Included any relevant context or examples.

Issue or Suggestion Description

I've had this issue across both dev kits and custom boards now intermittently for ~8 months so I'm left to thinking its a firmware based issue and not environmental or hardware.

on an off I see an issue where I get a warning message of

"wifi: Haven't to connect to a suitable AP now"

I follow the function init calls of the mesh_lite example of local_control for the WiFi and the mesh so only able to configure the settings via the menuconfig

What I'm seeing is a repeat of the warning shown above and that going from previous parent and new parent nodes for what I can only assume is it searching for a parent node that has an IP or is able to see the access point

like the below

W (1343075) wifi:Haven't to connect to a suitable AP now!
I (1343076) [vendor_ie]: now parent node   : bssid:e8:70:05:40:f3:70, rssi:0, level:-1 
I (1343082) [vendor_ie]: update parent node: bssid:94:a9:90:12:ec:65, rssi:-25, level:2 
W (1343091) wifi:Haven't to connect to a suitable AP now!
I (1347049) mqtt: Other event id:7
W (1350238) wifi:Password length matches WPA2 standards, authmode threshold changes from OPEN to WPA2
I (1350240) [vendor_ie]: esp_mesh_lite_wifi_scan_start return ESP_OK
I (1353074) [vendor_ie]: Mesh-Lite Scan done 
W (1353075) wifi:Haven't to connect to a suitable AP now!
I (1353076) [vendor_ie]: now parent node   : bssid:e8:70:05:40:f3:70, rssi:0, level:-1 
I (1353083) [vendor_ie]: update parent node: bssid:94:a9:90:12:ec:65, rssi:-23, level:2

...
W (1353103) wifi:Password length matches WPA2 standards, authmode threshold changes from OPEN to WPA2
W (1353112) wifi:Haven't to connect to a suitable AP now!

This issue happens every so often and across both the example and my written code. I'm confused as to why this loop will occure and what actually causes it. After multiple erases of the flash and writing the same code to the esp32 I often see the error clear or it clear after a few days only for it to come back after what seems to be a random amount of time; often after a weeks.

anyone got any suggestions as to this bizarre issue?
Thanks in advance for anything

BR

[tswen]

When calling the esp_wifi_sta_get_ap_info interface to retrieve AP information, if the device is not currently connected to an AP, this log message will appear. This log does not indicate an actual issue.

[BR-Coding-cmd]

Okay, thanks.

I'll close issue if by removing the unnecessary function call of 'esp_wifi_sta_get_ap_info' solves this issue or the real issues presents itself and I'll update the name of the issue accordingly

thanks,
BR

[BR-Coding-cmd]

After investigating the code, I will still get the above issue as there are no uses of that function outside of the tasks that are sleeping until there is a valid connection to a MQTT host (they are waiting for the setting of a event bit from the event handler call back for MQTT).

I've found that the function of "esp_mesh_lite_get_ap_record" uses the "esp_wifi_sta_get_ap_info " function call. would this not be called inbetween each of the parent node prints?

I (1343076) [vendor_ie]: now parent node : bssid:e8:70:05:40:f3:70, rssi:0, level:-1
I (1343082) [vendor_ie]: update parent node: bssid:94:a9:90:12:ec:65, rssi:-25, level:2

if so that would be where the "wifi:Haven't to connect to a suitable AP now!" would appear from. yet not explain why the driver is stuck in this loop.

Any further types of information required to help further, please let me know

thanks,
BR

[tswen]

Could you enable logging by calling esp_mesh_lite_core_log_enable(true) right after esp_mesh_lite_init(), then provide the complete log file from power-on until the issue occurs?

[BR-Coding-cmd]

I'll update this thread with logs once captured.

[BR-Coding-cmd]

Below is what I expect to see if the device disconnects from the MQTT broker due to the mesh.

Successful rejoin.txt

Within logs below I've been a little confused with the esp_mesh_lite_core_log_enable(true); printout of the below line

�[0;32mI (19557378) [vendor_ie]: Test Log 1030|94:a9:90:27:c1:15 �[0m

point of failiure.txt

Following the point of failure log, I don't see the mesh attempting to rejoin to the broker, naturally in the background, until the below log where it then later fails

rejoin into fail.txt

I'm very confused here as I can see that it can resolve it's issues but it takes too long for it to happen and it also doesn't create a stable connection.

any advice?

P.S. I doubt it's anything to do with hardware to proximity of devices as it's been documented in other open source tests that many esp kits have all been within 1 sqr meter and reportedly are stable. I've mine spread relatively evenly ( for testing purposes) across 4 sqr meters

[BR-Coding-cmd]

As an add-on to the above post, the final message that is then sent to the serial regarding any connection events can be seen captures in the below log.

last connection mention regarding mesh and other connections.txt

please note that the pattern seen in the last part of that log is repeated and never breaks

[tswen]

Periodically print free heap size to check if there's sufficient available memory.

[BR-Coding-cmd]

I've checked the heap and have noted that the heap does drop and have checked for leaks - but not found any in my running tasks. Currently I'm removing active tasks to check that way as my heap trace didn't turn up anything that wasn't recovered

When I did a system info print, the heap became stable at 7-11k remaining free. I've yet to see the heap reset due to aout of memory restart

Current test and changes since beginning of log.
LWIP and TCPIP has been changed

• stack size has been increased to 12k from 5120
• buffer size for send and receive has been increased to 24k from 9216
• task priority has been increased to 3 from 18
• message box for receive has been doubled 6->12

I've kept my MQTT messages being sent out at an average interval of 7 seconds.

• task 1 sends every 4 seconds
• task 2 sends every 10 seconds

mesh config remains the same - following ESP_MESH_LITE_DEFAULT_INIT with

 mesh_lite_config.join_mesh_ignore_router_status = true;
 mesh_lite_config.join_mesh_without_configured_wifi = true;

As I expect some devices to be out of range of a router so will require to connect to the nearest node rather than the router.

wifi.ap.beacon_interval is set at 800

I have a mesh of 7 (6 are currently in one mesh and 1 is orphaned) currently stable and a mesh of 21 than has collapsed in to 8 nodes with 3 nodes being on layer 1 and the rest being children of either of the 3.

The 7 node mesh currently has a stable heap after 2 nights running ~48hrs

19347912) main: System information, channel: 6, layer: 2, self mac: 94:a9:90:12:ec:d0, parent bssid: fc:01:2c:d5:b7:05, parent rssi: -29, free heap: 130668
I (19347924) main: All node number: 6

With a current running system from an overnight test from a starting of ~160500 free I'm now at 126144 free.

[BR-Coding-cmd]

It's worth noting, that within a current on going test, I'm viewing that the heap is dropped each time i'm publishing and there is a small memory leak there. I'm currently attempting to find it but it could be down to just a resource issue amungst mqtt/tcp publishing.

over a course of an hour, the custom F/W dropped 1564 bytes with no publishing but connected to the broker.
over the same time frame the mesh_lite example dropped 1812 bytes with only printing out the heap stack

[BR-Coding-cmd]

Periodically print free heap size to check if there's sufficient available memory.

I've checked against the memory and in the timeframe I'm having an issue with, the memory stays above 100k in heap

[tswen]

It's worth noting, that within a current on going test, I'm viewing that the heap is dropped each time i'm publishing and there is a small memory leak there. I'm currently attempting to find it but it could be down to just a resource issue amungst mqtt/tcp publishing.

over a course of an hour, the custom F/W dropped 1564 bytes with no publishing but connected to the broker. over the same time frame the mesh_lite example dropped 1812 bytes with only printing out the heap stack

https://github.com/tswen/esp-mesh-lite/tree/bugfix/Memory_leaks_caused_by_communication
This branch fixes an issue involving occasional memory leaks caused by communication interfaces.

For details, please refer to: #174 (comment)