Mixed IPv4/IPv6 code - address family mismatch

[matt335672]

xrdp version

All IPv6

Detailed xrdp version, build options

xrdp 0.10.80
  A Remote Desktop Protocol Server.
  Copyright (C) 2004-2025 Jay Sorg, Neutrino Labs, and all contributors.
  See https://github.com/neutrinolabs/xrdp for more information.

  Configure options:
      --enable-devel-all
      --enable-fuse
      --enable-pixman
      --enable-ipv6
      --enable-painter
      --enable-jpeg
      --with-imlib2
      --enable-vsock
      --with-freetype2
      --enable-ibus
      --enable-x264
      --enable-openh264
      --enable-utmp
      --with-libpcsclite
      --enable-neutrinordp
      CC=gcc
      CFLAGS=-g -fvar-tracking -Wl,-z,now -Wno-error=deprecated-declarations

  Compiled with OpenSSL 3.0.2 15 Mar 2022

Operating system & version

All

Installation method

Doesn't matter

Which backend do you use?

All

What desktop environment do you use?

any

Environment xrdp running on

All

What's your client?

No response

Area(s) with issue?

Network

Steps to reproduce

No steps as such - see below.

✔️ Expected Behavior

See below

❌ Actual Behavior

See below

Anything else?

While testing #3390, I found some interesting messages in xrdp-sesman.log:-

Linux

[2025-01-10T14:39:05.122+0000] [ERROR] [pid:401338 tid:125582510497920] [g_tcp_bind(os_calls.c:976)] g_tcp_bind(15, 6010) failed bind IPv6 (errno=98) and IPv4 (errno=22).

98 is EADDRINUSE and 22 is EINVAL

FreeBSD

[2025-01-10T14:46:48.309+0000] [ERROR] [g_tcp_bind(os_calls.c:978)] g_tcp_bind(15, 6010) failed bind IP
v6 (errno=48) and IPv4 (errno=47).

48 is EADDRINUSE, and 47 is EAFNOSUPPORT.

In both cases, the IPv6 error was expected, and the IPv4 error wasn't.

The problem is caused by this code:-

int
g_tcp_bind(int sck, const char *port)
{
    struct sockaddr_in6 sa;
    struct sockaddr_in s;
    int errno6;

    // First IPv6
    g_memset(&sa, 0, sizeof(sa));
    sa.sin6_family = AF_INET6;
    sa.sin6_addr = in6addr_any;                 // IPv6 ::
    sa.sin6_port = htons((tui16)atoi(port));
    if (bind(sck, (struct sockaddr *)&sa, sizeof(sa)) == 0)
    {
        return 0;
    }
    errno6 = errno;

    // else IPv4
    g_memset(&s, 0, sizeof(s));
    s.sin_family = AF_INET;
    s.sin_addr.s_addr = htonl(INADDR_ANY);     // IPv4 0.0.0.0
    s.sin_port = htons((tui16)atoi(port));
    if (bind(sck, (struct sockaddr *)&s, sizeof(s)) == 0)
    {
        return 0;
    }

    LOG(LOG_LEVEL_ERROR, "g_tcp_bind(%d, %s) failed "
        "bind IPv6 (errno=%d) and IPv4 (errno=%d).",
        sck, port, errno6, errno);
    return -1;
}

The code tries to bind the same socket to IPv6, or IPv4.

However, the socket has been created with one of these these calls in g_tcp_socket():-

(int)socket(AF_INET6, SOCK_STREAM, 0);

or

rv = (int)socket(AF_INET, SOCK_STREAM, 0);

In other words, the same socket can't be used for binding on both address families.

There's a similar problem with connect(), in that in g_tcp_connect() we try to connect a socket to both address families.

This looks pretty fundamental to me, and probably needs quite a big change. The good news is we don't connect a lot any more in v0.10.x over TCP sockets - it's mostly a listen() problem. We know there are definitely a few niggles with the IPv6 implementation, and this looks like one.

I'd appreciate any thoughts on this at all. @metalefty, @jsorg71, @Nexarian - please let me know what you think.