Memory leak in WayVNC on Raspberry Pi 5 8GB with Raspberry OS (v0.9.1)

[elchico92]

Useful information:

Please, try to gather as much of useful information as possible and follow
these instructions:

• Version:
  wayvnc: v0.9.1-e4ec935 (HEAD)
  neatvnc: v0.10-dev-6080662 (master)
  aml: v0.3.0-6-g56f51ad (master)

• System:
  Raspberry Pi 5, 8GB RAM OS: Raspberry Pi OS (based on Debian GNU/Linux 12 bookworm) PRETTY_NAME="Debian GNU/Linux 12 (bookworm)" Linux raspberrypi 6.6.62+rpt-rpi-2712 Fix output selection #1 SMP PREEMPT Debian 1:6.6.62-1+rpt1 (2024-11-25) aarch64 GNU/Linux

• Wayland compositor: Weston

• VNC client: RealVNC Viewer 4.9.2

• Graphics drivers: Default Raspberry Pi drivers (Mesa-based)

• Provide context, including but not limited to
  WayVNC's memory usage increases steadily over time. Additionally, every time a new window is opened or interacted with on the Raspberry Pi desktop, WayVNC's memory usage increases further and the allocated memory is not released. After a few minutes of operation (~10min), WayVNC consumes over 5 GB of memory and needs to be restarted. This depends highly on how many new windows I open. Closing windows does not reduce memory usage. Even opening and closing mousepad again and again leads to increasing memory usage even though it is only opened once at a time.

• Expected behavior:
  Memory usage should remain stable or within reasonable limits, even when windows are opened or closed on the desktop.

• Try to reproduce the problem while capturing a trace log:

  • wayvnc -Ltrace | tee wayvnc-bug.log
    Nothing changed during capturing:
    pi@raspberrypi:~ $ wayvnc -Ltrace | tee wayvnc-bug.log
    TRACE: ../src/main.c: 286: Registering new wlr_output_manager
    TRACE: ../src/main.c: 279: Registering new wlr_output_power_manager
    TRACE: ../src/main.c: 269: Registering new xdg_output_manager
    TRACE: ../src/main.c: 247: Registering new output 42
    TRACE: ../src/output-management.c: 186: New head, now at 1
    TRACE: ../src/output-management.c: 42: Got head name: NOOP-1
    TRACE: ../src/output-management.c: 51: Got head description: Headless output 1
    TRACE: ../src/output-management.c: 65: Got head mode
    TRACE: ../src/output-management.c: 71: Got head enabled: yes
    TRACE: ../src/output-management.c: 80: Got head current mode
    TRACE: ../src/output-management.c: 86: Got head position: 0,0
    TRACE: ../src/output-management.c: 92: Got head transform: 0
    TRACE: ../src/output-management.c: 100: Got head scale: 1.00
    TRACE: ../src/output-management.c: 164: Got new serial: 1
    TRACE: ../src/output.c: 227: Output 42 name: NOOP-1, headless: yes
    TRACE: ../src/output.c: 237: Output 42 description: Headless output 1
    Info: Capturing output NOOP-1
    Info: >> Headless output 1 1920x1080+0x0 Power:UNKNOWN
    DEBUG: ../src/ctl-server.c: 809: Initializing wayvncctl socket: /run/user/1000/wayvncctl
    DEBUG: ../src/server.c: 2160: Trying address: 127.0.0.1
    DEBUG: ../src/server.c: 2175: Successfully bound to address
    Info: Listening for connections on 127.0.0.1:5900

-output while monitoring consumption:
So 26. Jan 13:47:22 CET 2025: Checking WayVNC main PID: 6016
So 26. Jan 13:47:22 CET 2025: WayVNC memory usage: 2007 MB
So 26. Jan 13:47:22 CET 2025: Memory usage exceeded (2007 MB > 2000 MB), restarting WayVNC...
So 26. Jan 13:48:09 CET 2025: Checking WayVNC main PID: 6160
So 26. Jan 13:48:09 CET 2025: WayVNC memory usage: 117 MB
So 26. Jan 13:49:09 CET 2025: Checking WayVNC main PID: 6160
So 26. Jan 13:49:09 CET 2025: WayVNC memory usage: 920 MB
So 26. Jan 13:50:09 CET 2025: Checking WayVNC main PID: 6160
So 26. Jan 13:50:09 CET 2025: WayVNC memory usage: 1078 MB
So 26. Jan 13:51:09 CET 2025: Checking WayVNC main PID: 6160
So 26. Jan 13:51:09 CET 2025: WayVNC memory usage: 1078 MB
So 26. Jan 13:52:09 CET 2025: Checking WayVNC main PID: 6160
So 26. Jan 13:52:09 CET 2025: WayVNC memory usage: 1180 MB
So 26. Jan 13:53:09 CET 2025: Checking WayVNC main PID: 6160
So 26. Jan 13:53:09 CET 2025: WayVNC memory usage: 1591 MB
So 26. Jan 13:54:09 CET 2025: Checking WayVNC main PID: 6160
So 26. Jan 13:54:09 CET 2025: WayVNC memory usage: 1839 MB
So 26. Jan 13:55:09 CET 2025: Checking WayVNC main PID: 6160
So 26. Jan 13:55:09 CET 2025: WayVNC memory usage: 1841 MB
So 26. Jan 13:56:09 CET 2025: Checking WayVNC main PID: 6160
So 26. Jan 13:56:09 CET 2025: WayVNC memory usage: 2144 MB
So 26. Jan 13:56:09 CET 2025: Memory usage exceeded (2144 MB > 2000 MB), restarting WayVNC...
So 26. Jan 13:57:09 CET 2025: Checking WayVNC main PID: 6367
So 26. Jan 13:57:09 CET 2025: WayVNC memory usage: 208 MB
So 26. Jan 13:58:10 CET 2025: Checking WayVNC main PID: 6367
So 26. Jan 13:58:10 CET 2025: WayVNC memory usage: 669 MB
So 26. Jan 13:59:10 CET 2025: Checking WayVNC main PID: 6367
So 26. Jan 13:59:10 CET 2025: WayVNC memory usage: 830 MB
So 26. Jan 14:00:10 CET 2025: Checking WayVNC main PID: 6367
So 26. Jan 14:00:10 CET 2025: WayVNC memory usage: 830 MB
So 26. Jan 14:01:10 CET 2025: Checking WayVNC main PID: 6367
So 26. Jan 14:01:10 CET 2025: WayVNC memory usage: 830 MB
So 26. Jan 14:02:10 CET 2025: Checking WayVNC main PID: 6367
So 26. Jan 14:02:10 CET 2025: WayVNC memory usage: 830 MB
So 26. Jan 14:05:10 CET 2025: Checking WayVNC main PID: 6367
So 26. Jan 14:05:10 CET 2025: WayVNC memory usage: 1800 MB
So 26. Jan 14:07:10 CET 2025: Checking WayVNC main PID: 6367
So 26. Jan 14:07:10 CET 2025: WayVNC memory usage: 1974 MB
So 26. Jan 14:08:10 CET 2025: Checking WayVNC main PID: 6367
So 26. Jan 14:08:10 CET 2025: WayVNC memory usage: 2510 MB
So 26. Jan 14:08:10 CET 2025: Memory usage exceeded (2510 MB > 2000 MB), restarting WayVNC...

• Describe how to reproduce the problem

1. Start WayVNC with the following command:
   bash wayvnc
2. Connect a VNC client (e.g., RealVNC Viewer).
3. Interact with the desktop environment on the Raspberry Pi by opening and closing application windows.
4. Monitor memory usage over time using htop or ps.

• Try to think about your problem beyond these instructions and include
  Wayvnc runs on port 5901. Earlier I used X11 on 5900 and that worked.

[any1]

Please compile again from git master. Don't check out v9.1 specifically.