Asking for more KeyAction events when using android keyboard.

[mvvvv]

Description

This issue also concerns the PR rust-mobile/android-activity#178 from @jancespivo

With Openxr/StereoKit-rust, as I can't use neither Window::set_ime_allowed nor app.show_soft_input(), I tested the Android keyboard launched via JNI on the Meta Quest2. It works exactly like any Android device.
https://github.com/mvvvv/StereoKit-rust/blob/master/src/tools/os_api.rs#L191

Winit blocks many characters including accented characters which it replaces completely with a backspace and erases the character typed previously. I made a short video: https://youtu.be/os8cucfF3D8

The problem is that I can only rely on event::ElementState::Pressed and event::ElementState::Released which are the only events retained (

winit/src/platform_impl/android/mod.rs

Line 461 in 5ea81ef

let state = match key.action() {

) when in the log I can see the events I need:

12-30 15:16:17.601 23830 23830 W ViewRootImpl[NativeActivity]: Dropping event (no window focus):KeyEvent { action=ACTION_DOWN, keyCode=KEYCODE_DEL, scanCode=0, metaState=0, flags=0x0, repeatCount=0, eventTime=0, downTime=0, deviceId=-1, source=0x0, displayId=0 }
12-30 15:16:17.601 23830 23830 W ViewRootImpl[NativeActivity]: Cancelling event (no window focus):KeyEvent { action=ACTION_UP, keyCode=KEYCODE_DEL, scanCode=0, metaState=0, flags=0x20, repeatCount=0, eventTime=0, downTime=0, deviceId=-1, source=0x0, displayId=0 }
12-30 15:16:17.601 23830 23830 W ViewRootImpl[NativeActivity]: Cancelling event (no window focus):KeyEvent { action=ACTION_UP, keyCode=KEYCODE_DEL, scanCode=0, metaState=0, flags=0x20, repeatCount=0, eventTime=0, downTime=0, deviceId=-1, source=0x0, displayId=0 }
12-30 15:16:17.602 23830 23830 W ViewRootImpl[NativeActivity]: Dropping event (no window focus):KeyEvent { action=ACTION_MULTIPLE, keyCode=KEYCODE_UNKNOWN, scanCode=0, characters="ò", metaState=0, flags=0x0, repeatCount=0, eventTime=19026369, downTime=19026369, deviceId=-1, source=0x101, displayId=-1 }
12-30 15:16:17.605 23830 23882 D SK_rust_Demos: !!!KeyboardInput: KeyEvent { physical_key: Code(Backspace), logical_key: Named(Backspace), text: None, location: Standard, state: Released, repeat: false, platform_specific: KeyEventExtra }

Having access to ACTION_MULTIPLE event seems to be the only way to get enhanced characters(here ò), smileys and utf8 characters.

Device and Android version

Meta Quest 2 Horizon OS v72 (android 12.1)

Winit version

0.30.7

[mvvvv]

The idea is to keep only action_down and action_multiple. For example to make an ô you will have action_down ("o") + action_down (del) + action_multiple ("ô")

[jancespivo]

Hi, I investigated it a bit more and IMO the only way how to get a character when ACTION_MULTIPLE and KEYCODE_UNKNOWN is set is to call KeyEvent.getCharacters (which is deprecated btw https://developer.android.com/reference/android/view/KeyEvent.html#getCharacters()) Unfortunately NDK does not expose AKeyEvent_getCharacters so neither android-activity nor winit. It is not isolated to Rust world see https://issuetracker.google.com/issues/36950127?pli=1
I think the only way how to workaround the problem is to call KeyEvent.getCharacters with JNI (probably with fallback for the newer android versions - I don't have such a device and I don't know how it behaves there). Problem with JNI approach is the place where the logic should sit. We could hack it (in bad meaning) here https://github.com/rust-mobile/android-activity/blob/main/android-activity/src/lib.rs#L756 or pass jvm inside and do it on lower levels, which seems hacky to me as well.

[mvvvv]

Can't we enhance event::ElementState with these events ? here:

winit/src/platform_impl/android/mod.rs

Line 461 in 5ea81ef

let state = match key.action() {

[jancespivo]

@mvvvv Where the dropping event log line with all information we need is fired? I can't see these lines anywhere (ndk, / android-activity / winit). I can see some information in Android logs, but all information about key pressed / character picked is not get over the Android NDK layer, so IMO we can't do it anything useful here in winit.

[jancespivo]

This is very specific to the older versions of android. ACTION_MULTIPLE is deprecated see https://developer.android.com/reference/android/view/KeyEvent#ACTION_MULTIPLE. So I wouldn't pollute winit events with something that is platform and version specific. I would rather like to fix it somewhere close to the source of the event (probably android-activity).

[mvvvv]

The warning messages are from Android ViewRootImpl because there is a problem of focus or event handling as I do not have a view.

As I said I receive event::ElementState::Pressed and event::ElementState::Released. I believe (but I can't test) it's from this code:

winit/src/platform_impl/android/mod.rs

Line 461 in 5ea81ef

let state = match key.action() {

[jancespivo]

Unfortunately I can't see any of these events in logs on my Android device.

After picking letter ž on SoftKeyBoard these logs are fired (only RustStdoutStderr are from Rust code).

18047 18047 V InputMethodManager: dispatchKeyEventFromInputMethod: KeyEvent=KeyEvent { action=ACTION_MULTIPLE, keyCode=KEYCODE_UNKNOWN, scanCode=0, characters="ž", metaState=0, flags=0x0, repeatCount=0, eventTime=2540073973, downTime=2540073973, deviceId=-1, source=0x101, displayId=-1 }
18047 18079 I RustStdoutStderr: 10:06:40.482Z TRACE Mainloop iteration
18047 18079 I RustStdoutStderr: 10:06:40.482Z TRACE Handling main event InputAvailable
18047 18079 I RustStdoutStderr: 10:06:40.482Z TRACE Unknown MainEvent InputAvailable (ignored)
18047 18079 I RustStdoutStderr: ndk::event::InputEvent::KeyEvent -- KeyEvent { ptr: 0x761f33b360 }
18047 18079 I RustStdoutStderr: winit InputEvent::KeyEvent -- KeyEvent { ndk_event: KeyEvent { ptr: 0x761f33b360 }, _lifetime: PhantomData<&ndk::event::KeyEvent> }
18047 18079 I RustStdoutStderr: Window Event KeyboardInput { device_id: DeviceId(DeviceId(-1)), event: KeyEvent { physical_key: Unidentified(Android(0x0000)), logical_key: Unidentified(Android(0x0000)), text: None, location: Standard, state: Released, repeat: false, platform_specific: KeyEventExtra }, is_synthetic: false }
18047 18079 I RustStdoutStderr: 10:06:40.482Z DEBUG dispatch_text_event: Running ON_TEXT_EVENT pass with KeyboardKey
18047 18079 I RustStdoutStderr: 10:06:40.482Z TRACE dispatch_text_event:Textbox{id=#5}: Widget 'TextArea' #5 visited
18047 18079 I RustStdoutStderr: 10:06:40.482Z TRACE dispatch_text_event:Prose{id=#6}: Widget 'Textbox' #6 visited
18047 18079 I RustStdoutStderr: 10:06:40.482Z TRACE dispatch_text_event:Flex{id=#11}: Widget 'Flex' #11 visited
18047 18079 I RustStdoutStderr: 10:06:40.482Z TRACE dispatch_text_event:RootWidget{id=#12}: Widget 'RootWidget' #12 visited
18047 18079 I RustStdoutStderr: 10:06:40.482Z DEBUG dispatch_text_event: ON_TEXT_EVENT finished focused_widget=5 handled=false

I enhanced the logs with 3 println! (android-actvity, winit and application layer)

18047 18079 I RustStdoutStderr: ndk::event::InputEvent::KeyEvent -- KeyEvent { ptr: 0x761f33b360 }
18047 18079 I RustStdoutStderr: winit InputEvent::KeyEvent -- KeyEvent { ndk_event: KeyEvent { ptr: 0x761f33b360 }, _lifetime: PhantomData<&ndk::event::KeyEvent> }
18047 18079 I RustStdoutStderr: Window Event KeyboardInput { device_id: DeviceId(DeviceId(-1)), event: KeyEvent { physical_key: Unidentified(Android(0x0000)), logical_key: Unidentified(Android(0x0000)), text: None, location: Standard, state: Released, repeat: false, platform_specific: KeyEventExtra }, is_synthetic: false }

The information about pressed key is lost and not visible for winit at all. It might be different from your case, maybe different version of Android (mine is Android 12). I tracked the problem down to the android-activity crate, and found the pressed key is not available there as well (the solution that can work I proposed here #4067 (comment)). From my point of view the only way to work is to make it available there and it should theoretically start to work in winit out of the box (will see though :)

[jancespivo]

Unfortunately I haven't been successful. Apparently there is no possible workaround unless you have the newest Android 15, where a new function AInputEvent_toJava is introduced https://developer.android.com/ndk/reference/group/input#group___input_1gaed0e10fd09a1560029a870455257a247 IMO without that function it is not possible to convert NDK event to the Java one and call KeyEvent.getCharacters function with JNI. I tried to cast ndk event directly to the jobject, but it didn't work - it is not java accessible object :( The finding is in align with https://issuetracker.google.com/issues/36950127 I researched before.