Bad quality visuals when using x264

[sshaikh]

xrdp version

0.10.2

Detailed xrdp version, build options

❯ xrdp --version
xrdp 0.10.2
  A Remote Desktop Protocol Server.
  Copyright (C) 2004-2024 Jay Sorg, Neutrino Labs, and all contributors.
  See https://github.com/neutrinolabs/xrdp for more information.

  Configure options:
      --prefix=/usr
      --sysconfdir=/etc
      --libexecdir=/usr/lib
      --localstatedir=/var
      --sbindir=/usr/bin
      --with-systemdsystemunitdir=/usr/lib/systemd/system
      --enable-jpeg
      --enable-tjpeg
      --enable-fuse
      --enable-fdkaac
      --enable-opus
      --enable-rfxcodec
      --enable-mp3lame
      --enable-pixman
      --enable-painter
      --enable-vsock
      --enable-ipv6
      --enable-pam-config=arch
      --enable-rdpsndaudin
      --with-imlib2
      --enable-x264
      CFLAGS=-march=x86-64 -mtune=generic -O2 -pipe -fno-plt -fexceptions         -Wp,-D_FORTIFY_SOURCE=3 -Wformat -Werror=format-security         -fstack-clash-protection -fcf-protection         -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -g -ffile-prefix-map=/build/xrdp/src=/usr/src/debug/xrdp -flto=auto
      LDFLAGS=-Wl,-O1 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now          -Wl,-z,pack-relative-relocs -flto=auto
      CXXFLAGS=-march=x86-64 -mtune=generic -O2 -pipe -fno-plt -fexceptions         -Wp,-D_FORTIFY_SOURCE=3 -Wformat -Werror=format-security         -fstack-clash-protection -fcf-protection         -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -Wp,-D_GLIBCXX_ASSERTIONS -g -ffile-prefix-map=/build/xrdp/src=/usr/src/debug/xrdp -flto=auto

  Compiled with OpenSSL 3.4.0 22 Oct 2024

Operating system & version

Arch Linux

Installation method

other

Which backend do you use?

FreeRDP

What desktop environment do you use?

i3wm

Environment xrdp running on

Headless VM without GPU

What's your client?

Remmina

Area(s) with issue?

Graphic glitches, Performance

Steps to reproduce

Build and install package via https://aur.archlinux.org/packages/xrdp. Effect is seen next reboot. Recompiling without the enable-x264 flag restores previous performance.

✔️ Expected Behavior

Lag free.
Sharp graphics.

❌ Actual Behavior

Some subjective lag experienced.
"Fuzzy" rendering of graphics, particularly fonts. Most seen with red writing on black, but also other contrasting colours. It reminds me of bad aliasing or the effects of interpolation.

Anything else?

No response

[metalefty]

Why don’t you share screenshots when discussing graphic glitches? We should be looking at the same thing to have a proper discussion.

[sshaikh]

As requested. The effect is subtle and might be obscured by the capture tools. If these don't work then I'll try to use an external device to capture it (eg my phone).

First the screenshots taken "remotely". We should expect these to be identical (and they seem like they are).

Remote, no x264:

Remote, with x264:

This next image was taken from the client. The top is without x264, the bottom with (apologies for the image hack):

I'm wondering if there is a test image I can use instead?

[pnowack]

I'm wondering if there is a test image I can use instead?

Just use a thinner font (or not making the text fat). The description of your issue with x264 combined with "Fuzzy" rendering of graphics, particularly fonts. Most seen with red writing on black was actually super precise. What you experience here is the result of subsampling, as the H.264 encoded stream uses a YUV420 surface for the source frame.
Red text on black background, red text on baby blue background, or baby blue text on red background are one of the best reproducers for this issue (the text however needs to be thin, as the chroma subsampling works on 2x2 grids).

The human eye can actually not differentiate colours very well. However, it can differentiate brightness pretty well. That characteristic is used by most codecs these days (by subsampling the chroma values), because in video content, you won't notice it. There is a case, where this is noticed and that is usual text (unless it is fat or the font size is relatively big).
The colours in your screenshot appear here "just" more pale. With a thinner font (or by making the text smaller), the result on the client side will become blurry, while if you take the screenshot in the server side, it would be perfectly readable.

[sshaikh]

That helped:

Top two are with x264 disabled. Bottom two is with it enabled (it's the exact same image). I included green because unlike with the red (which becomes blurry) the green text has a fringe.

My actual question was to see whether anyone else was seeing this or if it's just me. I did play around with some of the config in gpu.toml but it didn't help. Is there a way to (or even point in) disabling this subsampling?

If this is x264 by design then that's fair enough, I'll just have to make do without it.

[tsz8899]

My tests are similar:

xrdp or Windows RFX:
Display Effect: Video playback and text display are perfect at both 1920 and 4K resolutions.
Bandwidth Usage: 2K-4K video plays smoothly at 60-100 Mb/s, text displays smoothly at 30-50 Mb/s.
Resource Usage: Server 1core CPU at 75%, but the client requires strong software decoding capability; many ARM processors cannot display 4K text smoothly.

xrdp + x264.v155-167 + AVC420:
Display Effect: Video playback is good, and text display on a 4K resolution LCD monitor is excellent, but at 1920 resolution, the text and icon edges feel blurred, particularly noticeable at intersections of different colors, especially in specific colors like red.
Bandwidth Usage: 2K-4K video plays smoothly at 15-30 Mb/s, text displays smoothly at 3-10 Mb/s.
Resource Usage: Server CPU at 85%, but most clients support 264 hardware decoding, allowing most CPUs to display video and text smoothly.

Windows RDP + AVC444 + GPU:
Display Effect: Video playback and text quality below 4K are good; video quality is weaker than x264, but text display is better than x264. Frame drops can be felt at 4K resolution.
Bandwidth Usage: 2K-4K video plays smoothly at 15-25 Mb/s, text displays smoothly at 3-10 Mb/s.
Resource Usage: 10% CPU on the server, 25% GPU usage.

xrdp + OpenH264.v2.5.0 + AVC420:
Display Effect: Video playback is relatively good, and text display on a 4K LCD monitor is excellent; however, with gfx.toml default values, the display quality is slightly weaker than x264.
Bandwidth Usage: 2K-4K video plays smoothly at 5-10 Mb/s, text displays smoothly at 3-10 Mb/s.
Resource Usage: Server CPU at 87%.

Each solution has its applicable scenarios. x264 is the most versatile; it would be great if text display is clear. Would x264 + AVC444 be helpful for text display?

Text Display Effects:
xrdp GFX: RFX:client screen 1920x1080 scale 100%,xrdp screen 1920x1080 scale 100%

xrdp + x264 + AVC420:client screen 1920x1080 scale 100%,xrdp screen 1920x1080 scale 100%

Windows RDP + AVC444:client screen 1920x1080 scale 100%,windows screen 1920x1080 scale 100%

xrdp + OpenH264 + AVC420:client screen 1920x1080 scale 100%,xrdp screen 1920x1080 scale 100%

xrdp+x264+avc420:client screen 3840x2160 scale 200%,xrdp screen 3840x2160 scale 100%. Why has it become clearer???

[metalefty]

Thank you all for the detailed test and report. Fundamentally, H.264 is a lossy video compression, so there's no way to avoid graphical deterioration. However, we can control the quality by configuring encoding parameters. The default configuration is performance (low CPU usage) oriented rather than quality. I'm not sure @sshaikh satisfy but we can MITIGATE the quality issue.

My actual question was to see whether anyone else was seeing this or if it's just me. I did play around with some of the config in gpu.toml but it didn't help. Is there a way to (or even point in) disabling this subsampling?

I guess everyone sees. that. And what exactly did you try?

[metalefty]

Also we need to support YUV444 mode for the fundamental quality enhancement as @pnowack mentioned.

[sshaikh]

I guess everyone sees. that. And what exactly did you try?

Only to set the profile to medium. I couldn't detect any difference so didn't explore further.

[metalefty]

Then, you're probably already seeing the better quality, such as red or black. YUV444 is needed for further quality I'm afraid.

[ThelloD]

Are there any updates on that issue already, or can the quality improved by changing some settings?

So far I am very happy with the performance and quality of xrdp when the new x264 codec is used, my remote VM finally feels as responsive as a VM running locally. The only issue is red text, pariticularly bold red text.

[metalefty]

As I mentioned just above, we need YUV444.