Performance improvements

[boardend]

Rather large pull request consisting of various performance improvements:

• Toolchain Update
  • Emscripten 3.1.59
  • vcpkg 2024.02.14
  • pnpm 9.1.0
• Internalize Qt build
  • Update to qt 6.6.1
  • Removal of Qt CMake patches
• Add SIMD support
• Switch to "mimalloc"
  • see https://emscripten.org/docs/optimizing/Optimizing-Code.html#allocation
• Switch to QgsMapRendererParallelJob
• Performance Hints
  • Adding some GDAL/OGR hints to improove GPKG speed
  • Prevent WAL in Sqlite
  • QgsMapSetting hints
• Allow up to 16 threads and use "threadPoolSize" also in QgsApplication setup
• Progressive Rendering
  • see QgisApi_renderJob
  • see OpenLayers source QgisJobDataSrouce.ts
• Performance measurement tool (see sites/performance)

Ideas and progress tracked in docs/performance.md

[wonder-sk]

👏 Nice set of dependency & code & performance updates!

Please feel free to merge...

Few notes:

• it would be useful to have a small howto for the performance testing tool - how to launch it, what does it do
• would be great to mark in performance.md what was done / not done
• I have realized my Firefox is constantly using 1 full CPU when I have the qgis-js demo open, even if the computer is completely idle otherwise (does that happen to you as well?)
• was there a reason why not stick to emscripten 3.1.37 that is preferred for qt 6.6?
• 1928c39 - I got a bit lost how this is meant to work - I assume the idea is to have the progressive rendering like in QGIS map canvas, right? In my testing, it behaves a bit strangely, but I am not able to explain well why it's that 🙂
• minor typo: Srouce -> Source
• dff9618 - the API was not built to allow starting of jobs in a worker thread - in theory strange things could happen 😄

[boardend]

Hey @wonder-sk, huge thanks for your review 🙏

Feedback to your points:

• it would be useful to have a small howto for the performance testing tool - how to launch it, what does it do

• Sure, will do that

• would be great to mark in performance.md what was done / not done

• Just did with b593aaf

• I have realized my Firefox is constantly using 1 full CPU when I have the qgis-js demo open, even if the computer is completely idle otherwise (does that happen to you as well?)

• Not that I've noticed. But I will do some testing

• was there a reason why not stick to emscripten 3.1.37 that is preferred for qt 6.6?

• Just tried to find my source for "by anecdotal evidence, binaries compiled with the latest version are faster (latest LLVM)" but couldn't 😅
  • But I would like to stick to the latest Emscripten version in order to have the latest features and optimizations available (like for example the removal of the separate .worker.js file)
  • Note that Qt states: "You can build Qt from source if you require more flexibility when selecting the Emscripten version. In this case the versions above are minimum versions. Later versions are expected to work but may introduce behavior changes which require making changes to Qt."
    • So I assume this is rather a safe thing to do..?

• 1928c39 - I got a bit lost how this is meant to work - I assume the idea is to have the progressive rendering like in QGIS map canvas, right? In my testing, it behaves a bit strangely, but I am not able to explain well why it's that 🙂

• Could you create a video? 😄
• Note that this is already deployed to the GitHub Pages, see the "Preview" Tab 👀
  • And you can find an implementation here sites/dev/src/ol.ts

• minor typo: Srouce -> Source

• Good catch, fixed with 08158bc

• dff9618 - the API was not built to allow starting of jobs in a worker thread - in theory strange things could happen 😄

• Added a small comment with 412ba3f
  • I would like to stick to that, because without there is a noticeable freeze of the main thread

[boardend]

@wonder-sk I observe the same behavior in FF 😟 This needs to be fixed before merging this PR

[boardend]

@wonder-sk Did some more testing; This is likely something the FF does internally when loading such a large .wasm module:

• The high CPU load goes towards zero after some time (~4min on my machine). The website is still working/responsive after that and stays idle when doing more render calls
• The same thing happens with the current 0.0.3 version, see https://boardend.github.io/qgis-js-baseline/
• The same thing happens if I just return 0 in the main function
• This behavior was observed in FF on Ubuntu Wayland and X11, also on FF on Windows 11
• FF profiler only shows that the PollWrapper is busy polling
• Hard to judge, but it seems that other larger Qt projects show similar behavior, although not as extreme as in qgis-js:
  • https://ripes.me/ (32MB)
  • https://fwcd.dev/m1xxx/ (304MB)

Unsure what to do with this 🤔

(cc @andreasneumann)

[boardend]

Will merge this one in and add the Firefox high CPU load as a separate issue: #45