winit-core:winit: add tablet input support

[kchibisov]

The API is integrated into the WindowEvent::Pointer* API and is present in form of TabletTool variant on corresponding data entries.

For now implemented for Web, Windows, and with limitations for Wayland.

Replaces #3810 and #4287.

[kchibisov]

@wareya Hi, would you mind to look into this wrt Windows stuff? I've did some basic testing(qemu + usb passthrough), and it kind of works.

Though, there's an issue with button not being registered, if you 1. press a button, 2. touch surface, 3. lift pen 4. release the button. This sequence results only in 3 button events and not sure that anything can be done here (we don't get more messages from windows).

[wareya]

Thanks for the ping, I'll try to take a look soon!

[kchibisov]

As for the API.

Do we really need Option for tilt, etc if we can have just default to 0 and so on? I guess the point is the logic to derive azimuth and such? And also, do we really need it separate if we can just compute it from tilt (browsers also do that from what I've seen, since platforms report tilt, but not azimuth, and even if's the case we can do transition from one to another)?

I'd also go forward with adding new types to ensure the boundaries of the values instead of writing those boundaries in docs.

[wareya]

Do we really need Option for tilt, etc if we can have just default to 0 and so on? I guess the point is the logic to derive azimuth and such?

Morally, yes, because applications may need to synthesize tilt if it's not actually supported. But in practice a lot of hardware or drivers will report having tilt but then always report a dummy value, so it might actually be misleading. Either you go with Option and document that some systems falsely report Some even when there's no real data, or you go with bare tilt and document that the application needs to watch for changes in the value to see if it's supported. (The same is true of azimuth/altitude, by the way!)

[kchibisov]

Morally, yes, because applications may need to synthesize tilt if it's not actually supported. But in practice a lot of hardware or drivers will report having tilt but then always report a dummy value, so it might actually be misleading.

The thing is that on win32/wayland/etc it seems like there's only tilt and you synth the other one. Though, I haven't looked into what macOS delivers actually. The thing is that my tablet doesn't support tilt but windows still reported that I have tilt of 0,0 even though it doesn't support it. Like the default tilt is 0,0 IIRC.

[wareya]

The thing is that my tablet doesn't support tilt but windows still reported that I have tilt of 0,0 even though it doesn't support it. Like the default tilt is 0,0 IIRC.

Yep, this is the main problem. I guess winit could track tilt over time and look for changes for a given pointer, and only start reporting it as Some if it ever changes...? But that's oddly high-level behavior for a windowing library, maybe.

[kchibisov]

Yep, this is the main problem. I guess winit could track tilt over time and look for changes for a given pointer, and only start reporting it as Some if it ever changes...? But that's oddly high-level behavior for a windowing library, maybe.

The thing is that windows reports TILT capability, so not obeying would be strange I guess. In general, tilt 0,0 just means that you have it orthogonal to surface, which is fine?

[wareya]

Initial notes after poking around for a few minutes on windows (in a slightly modified version of #4219):

1. The event layer seems to correctly get high-resolution positions from the windows API instead of the low-resolution positions that are easier to get but useless for art and CAD applications. Thumbs up.
2. I'm surprised that the ButtonSource version of the tablet input event doesn't carry very much data; I would expect to at least have tilt, and hopefully pressure too. Maybe it's in a different field of WindowEvent::PointerButton? I didn't fully explore the API changes, just scanned the diff looking for what I was missing.
3. I'm currently having issues with phantom press/release events when pressing the pen buttons during "drawing" but this is likely to be a skill issue on my part rather than an issue with the API. It might also be an issue with my setup (I'm currently using opentabletdriver instead of the standard commercial drivers for my device).
4. Unrelated to this PR: running examples on windows is currently impossible because the examples directory in the winit directory is a symlink and git on windows doesn't understand how to convert between symlinks and windows hardlinks (at least the version of git I'm using). It ends up checking out as a text file containing ../examples. I had to delete it and manually ln -s from a msys2 shell to get examples working. (EDIT: according to a friend, git on windows expects an admin-controlled windows setting (that controls symlink permissions) to be a non-default value, so it SHOULD work, but won't work on setups that don't use the git-for-windows installer, like msys2's git builds. Still a problem and worth considering changing IMO.)

[kchibisov]

2

it is not there, yes. I also think that probably should. But also, at the same time, Move is emitted so many times, that it's kind of always there if you stash it, I guess.

I'm not against adding though.

3

I'd suggest to check events that are coming, but generally, it seems like a button state.

Seems like due to how you setup git.

[wareya]

it is not there, yes. I also think that probably should. But also, at the same time, Move is emitted so many times, that it's kind of always there if you stash it, I guess.

I'm trying to think through this: if my application ignores inputs until it sees a button press, it'll be lacking tilt info until the first move event after the button press. If my application is trying to do something like "rotate 3d object to specific orientation when you press on it with the pen", it shouldn't have to wait for a movement event after the button press to be able to do that, I think. As another example, a painting program that works by splatting oriented copies of a brush image onto the canvas, which for some reason also ignores all the data from the pen until it's pressed onto the the screen, may have the first splat use the wrong orientation.

[Doublonmousse]

I have the same remarks as I did back here #2396 (comment) for the window backend. That boils down to

• On precision, using GET_POINTER_PEN_INFO means that pressure is quantized down to 1024 levels of pressure (as https://learn.microsoft.com/en-us/windows/win32/api/winuser/ns-winuser-pointer_pen_info indicates). So you lose on precision if the pen supports more than that (1/4th the resolution if you have 4096 levels)
• On pen button support, the Win32 API is limited, in particular no information is given for button presses when hovering
  See master...Doublonmousse:winit:winit-pen#diff-bb05c5e1236bdaa7f268bcbbbb8cc9f378a48eb3ed480884f0c6471a9cc933edR2020-R2031 for the winRT api using https://microsoft.github.io/windows-docs-rs/doc/windows/UI/Input/struct.PointerPoint.html#method.GetCurrentPoint that returns all properties we would want to get from the pen (isInContact, and all properties from https://microsoft.github.io/windows-docs-rs/doc/windows/UI/Input/struct.PointerPointProperties.html)

3 - I'll try to setup things to retest on my side (on surface official drivers). Having a somewhat state-less API (send button presses/releases)
works as long as this info is fully reliable!. As opposed to passing the button state for each event for example.

4 - Yes this is git config, do a git clone -c core.symlinks=true to keep the symlinks alive

Another issue: is the event a button AND position or one XOR the other? I don't think it's great if that's an exclusive as you'd totally lose the first pen position on the screen (chances are the BARREL signal is sent at the same exact time because of the lack of this information on hover)

Though, there's an issue with button not being registered, if you 1. press a button, 2. touch surface, 3. lift pen 4. release the button. This sequence results only in 3 button events and not sure that anything can be done here (we don't get more messages from windows).

When are these 3 events sent? Do you get a pen down/up + a release on lifting the pen? My guess is that it's related to Win32 API limitations (hence my suggestion to use the WinRT one)

[wareya]

Re: precision: the way tablet pens communicate auxiliary data like pressure and tilt to the digitizer is so noisy that 1024 levels is easily far more than enough. 8k pressure levels is basically marketing snake oil.

Re: button vs position: yes, absolutely, for tablets in particular it's very important for the user to get all the position data and auxiliary data in the same event as whatever event they get button press changes in. I'm not sure why this is the way that it is.

Re: WinRT: Do you mean RealTimeStylus, or the "Inking" API, or something else? RealTimeStylus is meant for handwriting-related stuff and is buggy for anything else, while the "Inking" API adds a whole lot of abstractions that are problematic for graphics program use (it has its own canvas context and stroke management system etc), and IIRC it's annoying to use from not-C#. Firefox uses WM_POINTER events (like this PR does) because of these problems.

[Doublonmousse]

WinRT : I mean using this method from the windows rust crate https://microsoft.github.io/windows-docs-rs/doc/windows/UI/Input/struct.PointerPoint.html#method.GetCurrentPoint

[wareya]

I thiiink that one's probably connected to the same system as WM_POINTER, but I don't know if it's a wrapper around WM_POINTER or an alternative API to the same data. If it is part of the same system as WM_POINTER like I think then it should be safe? But if the data is mostly the same and it's just a different API, it could probably be a later refactoring thing.

[kchibisov]

On pen button support, the Win32 API is limited, in particular no information is given for button presses when hovering

This matches what I've observed, but it's partially true, since it only breaks if you "mix" button presses release/press sequences. See example I've provided, since I've tried to test that.

Another issue: is the event a button AND position or one XOR the other? I don't think it's great if that's an exclusive as you'd totally lose the first pen position on the screen (chances are the BARREL signal is sent at the same exact time because of the lack of this information on hover)

as you can see position is a part of each button event, unless you mean Move. What is not part yet is the auxiliary data, since it's only attached to Move.

I think you can not mix win-rt and win32, and in any way, the windows backend is implemented straight away to not regress impl, while having better impl would be good, I don't think we're in position to block it because of that.

[Doublonmousse]

I think you can not mix win-rt and win32

Seems it can be done and I've succeeded in making it work. Get the pen event and pointer id from win32 (so as to not change the event loop logic) and query more pen info from the winrt api through getCurrentPoint. Maybe there's a little less window version support though (WinRT probably is > Windows 8). Though in any case it's not hard to change that later and not blocking.

For the button press/release mixes, I can't reproduce getting 3 on my side, but I can get the barrel press/contact release situation (and nothing sent on hover).

[kchibisov]

For the button press/release mixes, I can't reproduce getting 3 on my side, but I can get the barrel press/contact release situation (and nothing sent on hover).

Yeah, I count Contact as press, since I'm talking about Up/Down events, pressing barrel, contact, lift, release barrel, should result in 4 button events, while it ends up with 3 for me, where I don't get event for releasing barrel at the end.

[kchibisov]

Added the tool data to the Button as well, since it's just around and no point in not sending it.

[kchibisov]

I wonder if we should just derive release/press from the movement, at least for barrel I mean, since the information is generally available due to them being state and not really an event. Though, I'm not familiar with how other windows toolkits deal with it, maybe there's generally a better way to deal with that.