Pen/Stylus support #3810

daxpedda · 2024-07-22T10:13:03Z

Waiting for #3833.

This implements Pen/Stylus support inspired by the Web Pointer API.

Instead of adding new events I decided to add them to CursorMoved and MouseInput (now renamed to CursorInput.
This does preclude any larger changes in the spirit of Richer Touch events / unified pointer event model #336, but I'm happy to include some changes in this PR.
Despite the Web API not giving erasers their own device type, but simply a button, I decided to add a new device type instead. See Stylus eraser: should it be a new pointerType instead of a button state? w3c/pointerevents#134 for why this was considered a mistake to begin with.
X11 was emitting pen input in DeviceEvent::MouseWheel, DeviceEvent::MouseMotion, WindowEvent::CursorMoved, WindowEvent::MouseInput, which I disabled, because users would think all these inputs are mouse. This is also in line with all the other backends, X11 and iOS were the only ones emitting any pen events.
iOS was emitting Force::Calibrated::altitude_angle when a pen is used. I removed this and disabled pen events as well, as this information is now part of the ToolState, which can be used with Force::normalized() to get the old behavior back.

Open questions:

Rename "Tool" to "Stylus": Pen/Stylus support #3810 (comment).
Rename "Cursor" to "Pointer": Pen/Stylus support #3810 (comment).
Change DeviceEvent::MouseMotion to DeviceEvent::CursorMotion?
Implement X11 before merging? (I'm happy to do it)

Follow-up:

More compatibility issues with browser: Browser support tracking #2875.
- Linux Firefox has no pen support (Bugzilla).
Richer Touch events / unified pointer event model #336
Gesture API? This might be able to include things like feat(#3759): Implements Apple Pencil double tap functionality #3768 and all the Apple specific gestures.
Implement more backends.
X11 improvements/platform consistency (maybe after Browser support tracking #2875 and a touch overhaul)
- Don't move mouse with touch (?): Pen/Stylus support #3810 (comment).
- Don't move mouse on focus (?): Pen/Stylus support #3810 (comment).
- Better device detection: Pen/Stylus support #3810 (comment).
- Use correct device ID (?): Pen/Stylus support #3810 (comment).

Public API changes

pub enum WindowEvent {
    CursorMoved {
        device_id: DeviceId,
        position: PhysicalPosition<f64>,
        // new field
        r#type: CursorType,
    },
    // renamed from `MouseInput`
    CursorInput {
        device_id: DeviceId,
        state: ElementState,
        // changed type from `MouseButton`
        button: CursorButton
    },
    ...
}

pub enum Force {
    Calibrated {
        force: f64,
        max_possible_force: f64,
        // removed field
        // altitude_angle: Option<f64>,
    },
    Normalized(f64),
}

impl Force {
    // takes new `angle` parameter
    // now always emits normalized force, only emits perpendicular force if `angle` is passed
    pub fn normalized(&self, angle: Option<ToolAngle>) -> f64 { ... }
}

// new types from here on

pub enum CursorType {
    Mouse,
    Pen(ToolState),
    Eraser(ToolState),
}

pub struct ToolState {
    pub force: Force,
    pub tangential_force: Option<f32>,
    pub twist: Option<u16>,
    pub tilt: Option<ToolTilt>,
    pub angle: Option<ToolAngle>,
}

impl ToolState {
    pub fn tilt(self) -> Option<ToolTilt> { ... }
    pub fn angle(self) -> Option<ToolAngle> { ... }
}

pub struct ToolTilt { pub x: i8, pub y: i8 }

impl ToolTilt {
    pub fn angle(self) -> ToolAngle { ... }
}

pub struct ToolAngle { pub altitude: f64, pub azimuth: f64 }

impl Default for ToolAngle { ... }

impl ToolAngle {
    pub fn tilt(self) -> ToolTilt { ... }
}

pub enum CursorButton {
    Mouse(MouseButton),
    Pen(ToolButton),
    Eraser(ToolButton),
}

impl From<MouseButton> for CursorButton { ... }

pub enum ToolButton {
    Contact,
    Barrel,
    Other(u16),
}

impl From<ToolButton> for MouseButton { ... }

Fixes #99.

daxpedda · 2024-07-22T10:16:47Z

src/event.rs

+
+#[derive(Clone, Copy, Debug, PartialEq)]
+#[cfg_attr(feature = "serde", derive(Deserialize, Serialize))]
+pub struct ToolState {


I would be glad for an alternative to the term "Tool" or "State".
Originally got the idea from Linux, because we need a term that applies to all sorts of different tools.
But honestly I think "Pen" would have been fine as well?

Maybe "Stylus" would fit all the possible tools:

Pen

Eraser

Brush

Pencil

Airbrush

src/event.rs

This aligns with the future Std type name and allows us to introduce `LazyCell`.

daxpedda · 2024-07-22T11:12:11Z

src/platform_impl/linux/x11/event_processor.rs

                    event: WindowEvent::CursorMoved {
                        device_id: mkdid(util::VIRTUAL_CORE_POINTER),
                        position: location.cast(),
+                        r#type: CursorType::Mouse,
                    },
                };
                callback(&self.target, event);


Mouse cursor position changes when touch events are received.
Only the first concurrently active touch ID moves the mouse cursor.

This is a bit confusing to me, especially because I believe it doesn't map to other backends. Why do we tell the user that the mouse moves if the cursor moves with touch input?
Mouse input is cursor movement, but not all cursor movement is mouse input.

I'm really not sure what the purpose of this is.

daxpedda · 2024-07-22T12:26:10Z

src/event.rs

    CursorLeft { device_id: DeviceId },

+    /// A cursor button press has been received.
+    CursorInput { device_id: DeviceId, state: ElementState, button: CursorButton },


At this point we should probably rename "Cursor" to "Pointer".
"Cursor" commonly refers to the visual representation of an input device on the screen.
But "Pointer" is a broader category that can include any device and is not specific to its visual representation on the screen.

I couldn't really find anything concrete on this online.
WDYT?

daxpedda · 2024-07-22T12:33:58Z

src/platform_impl/linux/x11/event_processor.rs

            let event = Event::WindowEvent {
                window_id,
-                event: WindowEvent::CursorMoved { device_id, position },
+                event: WindowEvent::CursorMoved { device_id, position, r#type: CursorType::Mouse },


I don't know how to ask X11 what kind of device this is.
Right now it reports my pen as a mouse.
Same in CursorEntered and CursorLeft.

Same issue in DeviceEvents.
Interestingly MouseWheel contains the pen angles, but MouseMotion should probably be excluded for pens as well, unless we want to rename it to CursorMotion (or PointerMotion).

Okay I figured this one out as well.
After digging around in Chromium for a while I found how they determined if this was a stylus or not.

After some digging, I also found that Glazier was doing something quite similar, in addition to detecting erasers (in a hacky way, but it works). There is a lot to learn from them when actually implementing the backends for Pens/Stylus's.

Additionally, after some more digging into X11, I found that there is an alternative to figuring out the type: XListInputDevices. That's right, using the old XInput version and no way to query individual devices. This exposes what they call "Device Types", which unfortunately don't include pens, but they let you recognize the drawing tablet, which works as well.

I found that Chromium was doing something similar for touchpads, but not for pens (and a comment in a much older revision explaining this as well). So maybe this will be useful for us in the future.

daxpedda · 2024-07-22T16:24:00Z

src/platform_impl/linux/x11/event_processor.rs

        let event = Event::WindowEvent {
            window_id,
-            event: WindowEvent::CursorMoved { device_id: mkdid(pointer_id as _), position },
+            event: WindowEvent::CursorMoved {
+                device_id: mkdid(pointer_id as _),
+                position,
+                r#type: CursorType::Mouse,
+            },
        };


This is also a bit confusing to me.
AFAIU focusing the window is not a mouse event and the mouse didn't move.

daxpedda · 2024-07-22T17:01:26Z

src/platform_impl/linux/x11/event_processor.rs

+        let Some(DeviceType::Mouse) = self
+            .devices
+            .borrow()
+            .get(&DeviceId(event.sourceid as xinput::DeviceId))


This came as a surprise to me, but I had to spend quite some time to figure out why this wasn't working for me.
deviceId here uses the cursor, but sourceId actually uses the device that moved the cursor. Shouldn't we use that to send DeviceId to the user?

src/event.rs

- This adds pen/stylus support. - Move `Force::Calibrated::altitude_angle` to `ToolAngle::altitude`. - `Force::normalized()` now takes a `Option<ToolAngle>` to calculate the perpendicular force. - Just introducing types, no implementation yet!

Add a new `CursorButton` and `ToolButton` type.

Fixed: device events are emitted regardless of cursor type.

daxpedda · 2024-07-27T23:02:17Z

After some discussion and thinking, I created #3833, which would basically result in a pre-cursor to this PR.
The implementation of #3833 will be partly based on the code here.

I will resume this PR when we are done with #3833.

DorianRudolph · 2024-12-30T23:52:11Z

Great work on this PR and the pointer event overhaul! May I ask what the status of this PR is? This is something I have wanted for quite some time.

I made an implementation based on this PR (with some modifications) for wayland https://github.com/WhiteboardCX/winit/tree/tablet

I use f64 for tilt/twist/angle because wayland reports these as floats and it feels more natural.
I only use a single Tool kind for PointerKind, instead of separate eraser and pen. This aligns more with the wayland interface. The buttons on the pen are handled as PointerButton events and mapping these to pen/eraser/... then can be handled by the application.
I also added a ToolType, currently only pen and eraser, though wayland supports many more. However, I don't have hardware with different tool types. Presumably pens with an eraser on the back should work.
Wayland provides some extra features like distance, hardware ids for pens, etc., which presumably are also only available on higher end wacom tablets (I only have Huion and "One by Wacom"). I did not implement these.
Strangely, the wayland protocol does not seem to have a way to associate the tablet tools with the drawing tablet if there are multiple (not sure if it matters). The device_id is just assigned in the order in which the pens appear (my tablet don't report any of the ids in the protocol). So the same tool could get a different id when it reappears.

If my changes are OK, I can also make a PR. I should have hardware to test all platforms, but implementing all of them will take some time.

madsmtm · 2024-12-31T09:32:51Z

May I ask what the status of this PR is

It's nominated for our next meeting, which should happen sometime in start January

DorianRudolph · 2025-01-05T17:38:39Z

Thanks. In the meantime, I added support for windows to my fork and created a small demo based on vello.

wareya · 2025-05-03T12:07:12Z

Does this PR need to be redone on top of #3876? I have a lot of free time right now and like the above poster I did a test implementation of exposing tablet events on Windows. I could do the same for linux, android, and web too if this PR's creator is busy. https://github.com/wareya/winit/tree/tablet

DorianRudolph · 2025-05-03T12:35:00Z

Hey, I think my version of this was already on top of that commit https://github.com/WhiteboardCX/winit/tree/tablet
I also had Linux/Wayland and windows support.

I would be happy to work with you on this. In principle, I also have Mac and iPad hardware, so I could try adding support for these as well.

At this point my fork probably needs some work again to run on top of the latest master branch, and it would be helpful if someone else went over my design decisions.

madsmtm · 2025-05-05T21:39:29Z

Thanks for the interest. I unfortunately don't think we're equipped to review such work at the moment (I plan to post something about that soon), would it be okay if I maybe got back to you in a few months?

wareya · 2025-05-05T22:37:34Z

Are you planning on doing work that would require us to rewrite whatever we come up with, like swapping out platform integrations? If not, I'm willing to do whatever you guys need to make reviewing easier (e.g. limiting its scope, or sticking to a basic design that won't need breakage to be extended later, limiting the platforms supported at first to ones that are unlikely to need changing, etc).

DasLixou · 2025-06-20T12:43:08Z

Don't want to promise anything but it seems that after the backends are now split in seperate crates, the awaited "refactor" is done?
I might try to adopt this PR, make it work with the split and also try to integrate Apple Pencil support. (Still, nothing promised, but I'd like to know if now is the time and if it would be appreciated)

daxpedda added the S - enhancement Wouldn't this be the coolest? label Jul 22, 2024

daxpedda commented Jul 22, 2024

View reviewed changes

src/event.rs Outdated Show resolved Hide resolved

Internal: rename Lazy to LazyLock

cb0874a

This aligns with the future Std type name and allows us to introduce `LazyCell`.

daxpedda force-pushed the pen branch 2 times, most recently from aac4f31 to c3b84f8 Compare July 22, 2024 10:31

daxpedda commented Jul 22, 2024

View reviewed changes

daxpedda force-pushed the pen branch from c3b84f8 to f0b9bdd Compare July 22, 2024 12:25

daxpedda commented Jul 22, 2024

View reviewed changes

daxpedda force-pushed the pen branch from f0b9bdd to 74bc843 Compare July 22, 2024 16:47

daxpedda commented Jul 22, 2024

View reviewed changes

src/event.rs Show resolved Hide resolved

daxpedda added 3 commits July 23, 2024 00:16

Rename WindowEvent::MouseButton to CursorButton

584a073

Add a new `CursorButton` and `ToolButton` type.

Web: Pen/Stylus implementation

67e2fec

Fixed: device events are emitted regardless of cursor type.

daxpedda force-pushed the pen branch from 74bc843 to 67e2fec Compare July 22, 2024 22:19

daxpedda marked this pull request as ready for review July 22, 2024 22:52

daxpedda requested review from jackpot51, kchibisov, madsmtm and notgull as code owners July 22, 2024 22:52

This was referenced Jul 23, 2024

Web: don't block pen input #3813

Merged

Pointer Event Overhaul #3833

Closed

daxpedda marked this pull request as draft July 27, 2024 23:02

daxpedda added the DS - web Affects the Web backend (WebAssembly/WASM) label Jul 27, 2024

This was referenced Aug 16, 2024

Remove non-platform-consistent mouse events from X11 #3874

Open

Implement the pointer event overhaul #3876

Merged

madsmtm added the C - nominated Nominated for discussion in the next meeting label Dec 2, 2024

madsmtm modified the milestones: Version 0.30.11, Version 0.31.0 Jun 5, 2025

DasLixou mentioned this pull request Jun 21, 2025

Pen/Stylus support #4287

Closed

16 tasks

madsmtm mentioned this pull request Sep 4, 2025

winit-core:winit: add tablet input support #4318

Merged

madsmtm self-assigned this Sep 4, 2025

Pen/Stylus support #3810

Are you sure you want to change the base?

Pen/Stylus support #3810

Uh oh!

Conversation

daxpedda commented Jul 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

daxpedda Jul 22, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

daxpedda Jul 22, 2024

Choose a reason for hiding this comment

Uh oh!

daxpedda Jul 22, 2024

Choose a reason for hiding this comment

Uh oh!

daxpedda Jul 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

daxpedda Jul 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

daxpedda Jul 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

daxpedda Jul 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

daxpedda commented Jul 27, 2024

Uh oh!

DorianRudolph commented Dec 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

madsmtm commented Dec 31, 2024

Uh oh!

DorianRudolph commented Jan 5, 2025

Uh oh!

wareya commented May 3, 2025

Uh oh!

DorianRudolph commented May 3, 2025

Uh oh!

madsmtm commented May 5, 2025

Uh oh!

wareya commented May 5, 2025

Uh oh!

DasLixou commented Jun 20, 2025

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

5 participants

daxpedda commented Jul 22, 2024 •

edited

Loading

daxpedda Jul 22, 2024 •

edited

Loading

daxpedda Jul 22, 2024 •

edited

Loading

daxpedda Jul 22, 2024 •

edited

Loading

daxpedda Jul 22, 2024 •

edited

Loading

DorianRudolph commented Dec 30, 2024 •

edited

Loading