malloc(): invalid size (unsorted)

[EricClaeys]

Useful information:

• Version:
  wayvnc: 0.9.1
  neatvnc: 0.9.1
  aml: 0.3.0

The wayvnc package was updated today on my Raspberry Pi 5 running Bookworm and I can no longer connect.
Trying to connect from my PC using TigerVNC view returns a dialog box with:

An unexpeted error occured when communicating with the server:
readTLS: The TLS connection was non-propery terminated. (-110)

Trying to connect with RealVNC Viewer displays the connection window but it goes away in about a second.

With both TigerVNC and RealVNC Viewer /var/log/syslog has:

2025-07-14T02:51:18.119257-05:00 allsky-test4 systemd[1]: Started wayvnc.service - VNC Server.
2025-07-14T02:51:18.149270-05:00 allsky-test4 systemd[1]: Started wayvnc-control.service - VNC Control Service.
2025-07-14T02:51:31.982170-05:00 allsky-test4 wayvnc[8289]: malloc(): invalid size (unsorted)
2025-07-14T02:51:31.987164-05:00 allsky-test4 wayvnc[8270]: Aborted
2025-07-14T02:51:31.987533-05:00 allsky-test4 systemd[1]: wayvnc.service: Main process exited, code=exited, status=134/n/a
2025-07-14T02:51:31.987695-05:00 allsky-test4 systemd[1]: wayvnc.service: Failed with result 'exit-code'.
2025-07-14T02:51:31.988473-05:00 allsky-test4 systemd[1]: Stopping wayvnc-control.service - VNC Control Service...
2025-07-14T02:51:31.989344-05:00 allsky-test4 systemd[1]: wayvnc-control.service: Deactivated successfully.
2025-07-14T02:51:31.989769-05:00 allsky-test4 systemd[1]: Stopped wayvnc-control.service - VNC Control Service.
2025-07-14T02:51:32.321792-05:00 allsky-test4 systemd[1]: wayvnc.service: Scheduled restart job, restart counter is at 1.
2025-07-14T02:51:32.323958-05:00 allsky-test4 systemd[1]: Stopped wayvnc.service - VNC Server.

I ran wayvnc -Ltrace | tee /tmp/wayvnc-bug.log and got this on the screen:

Info: Capturing output NOOP-1
Info: >> Headless output 2 1920x1080+0x0 Power:UNKNOWN
DEBUG: ../src/ctl-server.c: 809: Initializing wayvncctl socket: /run/user/1000/wayvncctl
DEBUG: ../src/server.c: 2114: Trying address: 127.0.0.1
DEBUG: ../src/server.c: 2129: Successfully bound to address
Info: Listening for connections on 127.0.0.1:5900
DEBUG: ../src/ext-image-copy-capture.c: 331: Buffer dimensions: 1920x1080
DEBUG: ../src/buffer.c: 607: Reconfiguring buffer pool
DEBUG: ../src/ext-image-copy-capture.c: 475: Init done

[EricClaeys]

Is it possible to revert to the wayvnc version I had yesterday?

[miceno]

I also have this issue, on Raspi4, running Bookworm with all the latest updates applied.

As a workaround, I manage to make it run turning the picture quality on the client (RealVNC(R) Viewer 7.13.1 (r57) ARMv8-A (Dec 6 2024 18:04:04)) to "Low". Any attempt to set it to Medium or High made wayvnc server to crash.

[EricClaeys]

Setting the picture quality to "Low" also worked for me, but the bug in the server still needs to be fixed.

[any1]

@spl237, I notice that the neatvnc, version is 0.9.1, but the newest patch release is 0.9.4. I think that upgrading has a good change of fixing this. To be clear, neatvnc follows semantic versioning, so there are no new features that need testing in 0.9.4, only bug fixes.

[EricClaeys]

@any1,
the upgrade that appears to have broken my VNC connections was to the wayvnc server. How does neatvnc server relate?

Eric