[feat] Expose WindowEvent::KeyboardInput in Rust

[stijnfrishert]

Describe the problem

I want to be able to react to key events coming from the application event loop.

I am building a shortcut system for my Desktop app, on macOS. I would like to be able to trigger hotkeys, even if the application doesn't have any windows open. This rules out any js key listener solutions, because they are tied to windows. Plus, this feels like something that should be done over in Rust-land, anyway.

The current recommended solution is to use rdev. This works up to a point (see #11670), but requires my app to ask for full access to all OS-level keyboard inputs (Accessibility), which is a permission I would rather not depend on.

In reality, this is something that should be solvable by propagating events from the event loop. I scoured through Tao a bit and found key event types, so this seems to be within the realm of possibility?

Describe the solution you'd like

I want to be able to register a key event handler on my app. I envision this would look something like this, but that's of course up for debate:

tauri::Builder::default()
    .on_key_event(|app, event|{ }))
    .run()

I can imagine plug-ins might want to do this too, so maybe we should have a system to register (and unregister) multiple handlers.

Alternatives considered

• Binding to key event listeners in js, but this can only be done on windows. Besides, imho this is something to be solved on the Rust side, using the application event loop.

• Using rdev, but this requires my app to ask for full keyboard monitoring on an OS-level, which is not a permission I want to ask for, really.

  In addition, people have to twiddle with device_event_filter, which is neither intuitive nor easy to find.

Additional context

I've searched through the Discord and issue list, and it seems more people have been requesting this.

[amrbashir]

similar to the other issue, your problem may just be solved by our global shortcut plugin, https://tauri.app/plugin/global-shortcut/

[FabianLars]

i disagree, app-local shortcuts and/or an equivalent to js key events in rust are a regularly requested features (i was sure there was one on github already before this but i can't find it...)

[amrbashir]

app-local shortcuts and/or an equivalent to js key events in rust

not sure what you mean by this, please explain more

[FabianLars]

basically key events that only work when the app is in focus.

for example, let's say you want to listen to ctrl-s to save data. in your app most of the data is hosted on the rust side though so listening to the ctrl-s event on the rust side would be the easiest solution.

All current "solutions" have drawbacks:

• global-shortcut: has to be registered and unregistered on focus change. also the "stealing" nature of it may not be always desirable.
• rdev: also listens to events globally meaning you have to keep track of the focus state. The bigger issue is the macos accessability permission requirement though. And it doesn't work on wayland.
• listening to "keydown" (or similar) in js and forwarding that with tauri events to the backend: works alright, but still weird. Also potentially taxes the ipc more than needed

i'm not exactly sure how it behaves on macos but at least on windows tao's WindowEvent::KeyboardInput behaves exactly as desired

---

i realize i said mostly the same as OP but idk what else to say

[FabianLars]

for example, let's say you want to listen to ctrl-s to save data. in your app most of the data is hosted on the rust side though so listening to the ctrl-s event on the rust side would be the easiest solution.

I also often get asked how to have global-shortcuts not be global because a shortcut like api is easier (for shortcuts) than listening to key events but that would just be an extra imo

[amrbashir]

You keep calling it app-local, app-level where but I assume you mean window-level keyboard input but in Rust which is just forwarding taos' WindowEvent::KeyboardInput event, right? which is fine, we can expose it

[FabianLars]

yes

[stijnfrishert]

You keep calling it app-local, app-level where but I assume you mean window-level keyboard input but in Rust which is just forwarding taos' WindowEvent::KeyboardInput event, right? which is fine, we can expose it

You keep calling it app-local, app-level where but I assume you mean window-level keyboard input but in Rust which is just forwarding taos' WindowEvent::KeyboardInput event, right? which is fine, we can expose it

Yes, and no. In any case, I'd already be very happy if we could receive Tao's keyboard events in Rust Tauri by forwarding them.

There is a difference between app- and window-level key events, but only on macOS. Mac apps can launch without having any window open, and can stay active when your last window closes. This is also why the menu resides at the top of your screen, and isn't tied to any specific window.

A hotkey like Cmd + N to create a new "project" should be triggerable even without having any window open. To Windows and Linux users that seems exotic, but a lot of macOS apps work this way, including all Apple apps that come with the OS itself (Finder, for example). Technically this works, because key events are received by NSResponders, and NSWindow is one, but so is theNSApplication itself.

But to reiterate, at this stage I'm already happy if I can just receive window events forwarded from Tao.

All current "solutions" have drawbacks:

• global-shortcut: has to be registered and unregistered on focus change. also the "stealing" nature of it may not be always desirable.

• rdev: also listens to events globally meaning you have to keep track of the focus state. The bigger issue is the macos accessability permission requirement though. And it doesn't work on wayland.

• listening to "keydown" (or similar) in js and forwarding that with tauri events to the backend: works alright, but still weird. Also potentially taxes the ipc more than needed

Yeah, so receiving window events directly alleviates the accessibility permission, because apps have access to their own keyboard events by default.

For something like the rdev solution to work, I would need app-level focus/gain listeners to figure out when to register and unregister the key listeners. If not, they do a hostile take-over of hotkeys across your OS, and Cmd + N also triggers when, say, VSCode is in focus.

And, for macOS users, JS keydown events also don't fix this entirely, because you can't listen to non-window keydown events. :/

In any case, thank you for taking this request seriously. I know more people have been requesting this, and although it's not a waterproof argument, this is something most other app frameworks also support out-of-the-box.

I'm happy to work with you and send in a PR, especially because the amount of macOS devs is probably low, but I would need some guidance/mentorship on where to look in the code.

[amrbashir]

There is a difference between app- and window-level key events, but only on macOS. Mac apps can launch without having any window open, and can stay active when your last window closes. This is also why the menu resides at the top of your screen, and isn't tied to any specific window.

macOS is weird tbh.

[amrbashir]

I'm happy to work with you and send in a PR, especially because the amount of macOS devs is probably low, but I would need some guidance/mentorship on where to look in the code.

Thanks for the offer, we appreciate this.

• For exposing the tao's WindowEvent::KeyboardInput, you can search for WindowEvent::Moved in tauri repo and you can see how this event is exposed and replicate it.
• For the app-level keyboard input, you'd have to first add the event in tao repo, probably in https://github.com/tauri-apps/tao/blob/dev/src/platform_impl/macos/app_delegate.rs

[crackleeeessyp]

Agree, app-level hot key is necessary

[stijnfrishert]

Been a while, but I've started work on this.

@amrbashir I've got two questions:

• How much of the underlying event in Tao do we want to expose to Tauri? Just the KeyEvent, or also the DevideId from which it came and whether the event was synthetic?
• Do we also want to emit a "tauri://keyboard-input" event to the window?

[amrbashir]

@stijnfrishert Let's start by just exposing KeyEvent but not all of it though, just physical_key, logical_key and state. I would also use types from the keyboard-types crate.

As for the JS event, that sounds good too.

[stijnfrishert]

@amrbashir I'm not sure what you're suggesting here. The keyboard types used in tao aren't based on keyboard-types, from what I can see.

I can pass along physical_key, logical_key and state just fine, but I don't need keyboard-types for that.

Or are you suggesting to re-export and use keyboard-types as the public tauri API? In that case we need to write conversion functions from the one crate to the other. Why isn't tao internally based on keyboard-types in the first place, then?

[amrbashir]

Yes I meant the latter, and tao doesn't use it because it was a fork of winit, but nonetheless a wrapper in tauri ensures there is no breaking changes from tao would creep in.

[stijnfrishert]

@amrbashir I've pushed some work and created a draft PR. There's a question about non-exhaustive enums in there.

Can I ask you to take a look and review? I don't know what the workflow is for Tauri here.