esp_wifi eventually uses up all available memory and stops working

[Mossop]

Bug description

I have a project I'm playing with on an ESP32C6 board to publish sensor readings to mqtt. I have everything working but after about 12 hours of operation the wifi stops working and I see repeated Unable to allocate 2360 bytes warnings from esp_wifi in the log. esp_alloc has 96K of heap space and to my knowledge nothing else beyond esp_wifi is using it. I've tried increasing that as much as I can but nothing helps.

My repo is here: https://github.com/Mossop/garage-sensor.
Much of the esp-specific code is in here: https://github.com/Mossop/garage-sensor/blob/main/src/board/esp32c6.rs. Everything outside that uses the embassy_net APIs to interact with the network.

I have made some attempts to narrow the problem down but given the ~12 hour time to see if something works or not I haven't had much success. The basic embassy_net examples in the repo do not seem to do this. Is there some additional logging I can try to narrow down what might be going on?

Expected behavior

esp_wifi should not run out of memory.

Environment

• Target device:

Chip type:         esp32c6 (revision v0.1)
Crystal frequency: 40 MHz
Flash size:        4MB
Features:          WiFi 6, BT 5
MAC address:       9c:9e:6e:77:b8:2c

• Crate name and version: esp-wifi main

[okhsunrog]

You can add logging of heap usage stats, which was added in PR #2619. This way, you will be able to notice memory leaks earlier

[Mossop]

I added heap logging every five minutes. I wouldn't say there is any obvious jump anywhere, it starts at 50% (49836/98304) then slowly rises. Once it reaches 70% (69180/98304) esp_wifi starts complaining about failing to allocate occasionally. It gets really bad once we get to 80% (78668/98304). Usage continues to grow though, up to 96% last I looked,

[Mossop]

Memory usage eventually hit 97% and the device stopped with a "Load access fault"

[MabezDev]

I notice you have ESP_WIFI_CSI_ENABLE = "true" in https://github.com/Mossop/garage-sensor/blob/261b97db29764822d1920a5375ea9397584a186a/.cargo/config.toml#L8 but you don't seem to be using it in your app - could you try removing it (or setting it to false) and seeing if it makes a difference?

[Mossop]

Sure, I think I was just randomly trying things to fix it. After about 30 minutes of running heap usage has risen by 2% which is about in line with previously.

[Mossop]

The memory usage doesn't seem related to network traffic. I tried changing my code such that it sends and receives data much more frequently. It does however appear related to losing the connection to the wifi AP.

I ran a test where I watched the heap usage every 30 seconds and over the course of a few minutes the total heap usage was completely stable, however when I then forced the device to reconnect to the wifi (I can tell the AP to disconnect it) the heap started growing by about 30 bytes every 30 seconds. Reconnecting it a few more times didn't change that.

[bjoernQ]

Thanks for the hint regarding the reconnect.I think I found the reason for the leak and hope to have a fix soon

[Mossop]

Awesome. Let me know if you would like me to test anything.

[bjoernQ]

#2892 should fix the issue

[Mossop]

#2892 should fix the issue

Awesome thanks. I can confirm that it solves the problem. My device has now been running for 20 hours and heap usage is exactly the same as when it started. Thanks for the quick fix!