Serve large files from LittleFS efficiently with template processor

[szerwi]

Hi,

I am looking for the solution for efficiently serving large files from LittleFS partition. I also need the template processor feature.
Currently I serve web page files like this:

webServer.serveStatic("/style", LittleFS, "/www/style/")
    .setCacheControl("max-age=0");

webServer.serveStatic("/", LittleFS, "/www/")
    .setDefaultFile("index.html")
    .setTemplateProcessor(webVariables)
    .setCacheControl("max-age=0");

However, my page is not so small - 12 files that are 67kB in total.
During client connection I observe increased heap usage (minimum ever free heap size goes down from ~70000B to ~2000B).
What's more, web page files are often not served in full. It seems like sometimes the file transfer process is interrupted and the browser displays the page that is not complete or some JS scripts are not working.

What I have already tried:

1. Compressing files using gzip. It allowed me to shrink the size of the web page down from 67kB to 12kB. However, template processor was not working when serving compressed files. Is there any way to overcome that?
2. I have also read the wiki that contains the information about serving large files - https://github.com/ESP32Async/ESPAsyncWebServer/wiki#respond-with-file-content-using-a-callback-and-extra-headers and https://github.com/ESP32Async/ESPAsyncWebServer/wiki#serving-static-files-by-custom-handling. However, in this approach I also do not see a way to have the template processor.

Is there any way to serve large static files efficiently & keep the template processor functionality?

[mathieucarbou]

templating requires to load the template in memory.
also, templating is not a feature recommended to use because there are more advanced templating capabilities in other libs and becaue of the heap requirement.

MCU is low cpu / memory: all the templating should be done client-side, and mcu only called to get static pages plus json api for the data.

this mostly require a rework on how you process your UI to delegate some work to the browser instead of doing it for each client on the MCU.

[szerwi]

Are there any frameworks/libs for ESP32 that allow to create web dashboards that will have less memory requirements than templating? Can you recommend something? Or the best way will be to create own HTML/CSS static web page and send dynamic data via JSON as there are no efficient, ready solutions for it?

[EmileSpecialProducts]

I have an example that will serve lage files from LittleFS. ( not using template )
https://github.com/EmileSpecialProducts/ESP-LittleFS-Async-Web-Server

I use javascript and an API to get data on the webpage normaly.
In this case the content/files in the LittleFS.