String management

[JosePineiro]

In C++ I can think of several ways to perform operations with strings of unknown length (we don't know their size at compile time).
In this case, two possibilities occur to me:

1. std::string

std::string gzPathStr = path.c_str();
gzPathStr += T__gz;

• Modern, portable, safe, and clean C++ code.
• Uses heap (can fragment memory), but less prone than Arduino String.
• Slightly slower than stack-only methods.
• On small strings, SBO (Small Buffer Optimization) may keep it on the stack, avoiding heap.
• Fully supported on ESP32 and ESP8266 with STL.
• Not available on tiny AVR-based platforms (e.g., ATtiny) without full STL support.

2. dinamic buffer

size_t pathLen = path.length();
char gzPath[pathLen + GZ_SUFFIX_LEN];
memcpy(gzPath, path.c_str(), pathLen);
memcpy(gzPath + pathLen, T__gz, GZ_SUFFIX_LEN);
Very fast and efficient, stack-only allocation.

• Risk of stack overflow if path is large (~512 bytes or more in esp32).
• Uses VLA (Variable Length Array), which is non-standard in C++ and only works with gnu++ extensions.
• Works fine on ESP32/ESP8266 using GCC.
• Not reliable on "tiny" platforms (limited stack + no VLA support).

1. Arduino String

      size_t pathLen = path.length();
      String gzPath;
      gzPath.reserve(pathLen + GZ_SUFFIX_LEN);  
      gzPath.concat(path);
      gzPath.concat(asyncsrv::T__gz);

• Simple, readable, Arduino-friendly syntax.
• Always uses the heap, causing memory fragmentation.
• Not recommended for embedded systems where stability matters.
• Only works inside Arduino environments (not portable C++).
• Adds more flash/program size. Risk of memory leaks.

In my opinion, there's no good choice, as each has its advantages and disadvantages, and none are 100% portable. Choosing one is a matter of taste. Therefore, I think it's up to you, as repository managers, to choose one option. I will use it for future optimizations.

NOTE: I prefer str::string, but since I only use an ESP32 and an Arduino-based platform, I'm not sure if it supports other configurations.

[mathieucarbou]

Note that String usage does not necessary cause heap fragmentation: for small allocations, the first free blocks are reused. Heap fragmentation is caused when allocating a lot of large segments of memory.

Esp-idf have some callbacks to track heap allocations and deallocations that you can use with Arduino using pioarduino hybrid compilation.

if you activate and print those you will see that esp-idf and Arduino itself are doing a LOT of heap allocations and dealloc, far more than your apps probably does and also for small quantities. WiFi is especially using heap allocations a lot and frequently.

heap is not an issue when used correctly and free the stack size.

Increasing stack size is a hell for apps using a lot of tasks because it creates an amount of memory that is not necessarily used by the app and count against the stack size when creating tasks.

[willmmiles]

Note that String usage does not necessary cause heap fragmentation: for small allocations, the first free blocks are reused. Heap fragmentation is caused when allocating a lot of large segments of memory.

Espressif's allocator (both ESP8266 and ESP32) will also coalesce adjacent free blocks, so fragmentation is only a problem for long-lived allocations. They've clearly spent some time optimizing it for heavy usage of small- or mid-(Ethernet packet size) sized transient allocations, ie. exactly IDF's usage pattern.

I concur 100% with your conclusions re heap usage.

I'd also note that the the Arduino String implementation on Espressif devices does small-string optimization, just like std::string.

[JosePineiro]

If I understand correctly, String on Arduino has been greatly optimized, and its performance isn't as bad as I thought.
But the question of compatibility still remains.
Which of the above options do you recommend?

[mathieucarbou]

ESPAsync uses String ,so please keep using String on heap, and reserve the right size upfront.