Unable to connect using RealVNC client: bad desktop size 32561x35048

[xorrvin]

Freshly compiled wayvnc and its deps from master, using Alpine Linux Docker (musl libc v.1.2.5) container.

wayvnc --version
wayvnc: v0.9-dev-15660cd (master)
neatvnc: v0.9-dev-0e93aa9 (master)
aml: v0.3.0-2-gef33f2d (master)

First of all, sometimes it crashes randomly with a segfault. I've fired up gdb and it looks like this:

# gdb wayvnc 
GNU gdb (GDB) 14.2
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-alpine-linux-musl".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from wayvnc...
(gdb) r -Ltrace 0.0.0.0
Starting program: /usr/bin/wayvnc -Ltrace 0.0.0.0
TRACE: ../src/main.c: 251: Registering new xdg_output_manager
TRACE: ../src/main.c: 269: Registering new wlr_output_manager
TRACE: ../src/main.c: 261: Registering new wlr_output_power_manager
TRACE: ../src/output-management.c: 162: Got new serial: 0
Info: Capturing output
[New LWP 2242]
[New LWP 2243]
[New LWP 2244]
[New LWP 2245]
DEBUG: ../subprojects/neatvnc/src/server.c: 1919: Trying address: 0.0.0.0
DEBUG: ../subprojects/neatvnc/src/server.c: 1939: Successfully bound to address
Info: Listening for connections on 0.0.0.0:5900

Thread 1 "wayvnc" received signal SIGSEGV, Segmentation fault.
memset () at src/string/x86_64/memset.s:55
warning: 55	src/string/x86_64/memset.s: No such file or directory
(gdb) bt
#0  memset () at src/string/x86_64/memset.s:55
#1  0x000055555555f349 in create_placeholder_buffer (width=32767, height=59624) at ../src/main.c:844
#2  0x000055555555f3ca in blank_screen (self=0x7fffffff6500) at ../src/main.c:859
#3  0x000055555555faa9 in init_nvnc (self=0x7fffffff6500, addr=0x7fffffff6828 "0.0.0.0", port=5900, socket_type=SOCKET_TYPE_TCP)
    at ../src/main.c:973
#4  0x0000555555562a3f in main (argc=3, argv=0x7fffffffeb88) at ../src/main.c:1932

One can see that create_placeholder_buffer creates a buffer with absolutely enormous screen dimensions.

When it does not segfault and starts away, an attempt to connect produces expected client error and following trace log:

# wayvnc -Ltrace 0.0.0.0
TRACE: ../src/main.c: 251: Registering new xdg_output_manager
TRACE: ../src/main.c: 269: Registering new wlr_output_manager
TRACE: ../src/main.c: 261: Registering new wlr_output_power_manager
TRACE: ../src/output-management.c: 162: Got new serial: 0
Info: Capturing output
DEBUG: ../subprojects/neatvnc/src/server.c: 1919: Trying address: 0.0.0.0
DEBUG: ../subprojects/neatvnc/src/server.c: 1939: Successfully bound to address
Info: Listening for connections on 0.0.0.0:5900
*** BUG ***
In pixman_region_init_rect: Invalid rectangle passed
Set a breakpoint on '_pixman_log_error' to debug

DEBUG: ../src/ctl-server.c: 809: Initializing wayvncctl socket: /tmp/wayvncctl
Info: New client connection from 192.168.33.23: 0x7f31236d9070 (ref 1)
DEBUG: ../subprojects/neatvnc/src/server.c: 800: Client chose security type: 1
Info: Starting screen capture
DEBUG: ../src/main.c: 1348: Client connected, new client count: 1
DEBUG: ../src/ctl-server.c: 941: Enqueueing client-connected event: {"id":"1","address":"192.168.33.23","username":null,"seat":"seat0","connection_count":1}
DEBUG: ../src/ctl-server.c: 968: Enqueued client-connected event for 0 clients
wl_display@1: error 1: invalid arguments for [email protected]_virtual_pointer_with_output
ERROR: ../src/main.c: 508: Failed to dispatch pending
*** BUG ***
In pixman_region_init_rect: Invalid rectangle passed
Set a breakpoint on '_pixman_log_error' to debug

DEBUG: ../src/ctl-server.c: 941: Enqueueing detached event: {}
DEBUG: ../src/ctl-server.c: 968: Enqueued detached event for 0 clients
Info: Client 0x7f31236d9070 (1) hung up
Info: Closing client connection 0x7f31236d9070: ref 0
DEBUG: ../src/main.c: 1296: Client disconnected, new client count: 0
DEBUG: ../src/ctl-server.c: 380: Don't know how to convert sa_family 0 to string
DEBUG: ../src/ctl-server.c: 941: Enqueueing client-disconnected event: {"id":"1","address":null,"username":null,"seat":null,"connection_count":0}
DEBUG: ../src/ctl-server.c: 968: Enqueued client-disconnected event for 0 clients

[any1]

Is this also a problem with the latest release?

[xorrvin]

@any1 I think yes, but I can retest with specific versions pinned if needed

[any1]

I would also be good to know which compositor you're using.

[husshpup]

I had this same issue connecting with viewers on both Windows and Linux (viewers tried on Windows: Realvnc and tigervnc; viewers tried on the Linux box: Realvnc and Remmina). Compositors on the server: Hyprland and Sway. Disabling and disconnecting the external monitor on the server had no effect. Initially, I launched the server from a ssh session separate from the desktop environment. When I killed this server and launched it within the desktop environment, I connected with no issue. However, I was also able to successfully connect when rerunning the command from the ssh session so not entirely clear what went wrong.