Evolution of the Android backend

[rib]

Hi,

Sorry in advance this is a fairly long read :) This this aims to open a discussion as much as it does try to highlight some practical issues I found recently while working with the Android backend. The winit Discussions don't seem super active and since I also had some practical issues to bring up I figured I'd post here, but happy to try and split this up if it makes sense...

I've recently been experimenting with an alternative "glue" layer for building native Rust applications on Android that's based on the GameActivity class (in turn based on the widely used AppCompatActivity class) that's part of Google's Android Game Development Kit:

Ref: https://developer.android.com/games/agdk
and https://developer.android.com/games/agdk/integrate-game-activity

Here's the glue layer I've implemented so far: https://github.com/rib/agdk-rust/tree/main/game-activity

The README there also has a bit more context, including a bit about my motivation for looking at this but essentially NativeActivity isn't practically usable in many circumstances (at least not without subclassing in Java to augment with additional functionality, but even then it's impossible to use AppCompatActivity which has become a very standard foundational class for Android development that provides numerous back-ported compatibility APIs on Android).

I originally aimed to base the game-activity glue API on ndk-glue but as I made progress I found I needed to diverge in a number of ways from the ndk-glue API, such as for robustly handling synchronization between the java main thread and native thread, and I also chose to define a standard extern "C" android_main() entry point ABI for applications that doesn't require any special macros.

So although I originally imagined I might be able to enable winit support simply by being a drop in alternative to ndk-glue I ended up working on more substantial winit backend changes to be able to build on this work further.

Just for reference at this point I've published these three minimal Android app examples based on this work, to test creating a winit + wgpu app and a winit-based egui app. I have an experimental Bevy app I've been poking at locally but for now Bevy doesn't really work on Android due to it's lack of handling of lifecycle events.

Please find my example test apps here:
https://github.com/rib/agdk-rust/tree/main/examples

Please find my initial Android backend for Winit here: https://github.com/rib/winit/tree/agdk-game-activity

Now it's a bit tricky to determine the next steps for this work, considering that ideally I'd like to be able to enable this functionality in winit upstream, not just a private branch.

For starters I figured it would be good to share the current status of this work, and then I was thinking I'd list some of the most actionable topics that have come to light in the process of working on this that might help determine some next steps...

Technical backend issues that were noticed:

• The existing Android backend in winit could be made more consistent with e.g. the X11 backend considering being similarly based on polling file descriptors.

Something I found while working on the Android backend was that it was initially unclear (at least to me) if there were potentially multiple points where the loop could block on IO while it didn't follow (what I would consider) a more traditional polling model where there would be one clear point of polling (via a Looper which is Android's wrapper over epoll). For some time I was convinced the existing backend was going to block on its main event reads, in addition to doing a looper poll, but I think to some extent it's just the layout that I found a little misleading to follow. In comparison I found the organisation of the X11 backend quite clear, with a separate function for running a single iteration of the loop and a single point where the loop would block to poll for file descriptor events. When I enabled my own glue layer I ended up following the same structure as the X11 backend.

• It's subjective but the Android event loop feels like it's kind of upside down - where it dispatches events and then polls in different ways according to the control_flow. Again I found the X11 backend structure clearer in this regard - it would poll for events and then run an iteration of event dispatching (which was encapsulated in a function where it was clear to see the logical order of dispatching matched the documented order that's expected).

• A small detail, but the Android backend uses a call_event_handler macro which doesn't need to be a macro. I ended up swapping in a sticky_exit_callback function that was consistent with the X11/Wayland backends.

• Redraw requests are converted into an event in such a way that if they don't get handled by triggering a redraw then the request is lost. I think redraw requests are conceptually expected to be persistent requests that should only be cleared when they are fulfilled. E.g. if a redraw is requested while the loop isn't 'running'  (i.e. suspended) then (currently) the internal event is take()en after waking from the poll but redraws aren't emitted while an app is suspended. I'd intuitively expect that the request should be queued until the app resumes.

• Queuing redraw requests as events introduces some unnecessary latency, and potentially multiple iterations of the event loop before the request might be honoured. At the point of queuing the redraw request there might already be other events pending that will wake up the looper, such as input events and Android's Looper implementation doesn't prioritize delivering a Poll::Wake over any other file descriptor events. Since the redraw request will only be acknowledged once the looper specifically wakes with a Poll::Wake it's possible to handle multiple other pending events before catching up with the redraw request.

Something I did a little differently here, which could potentially be re-used in other backends too was to create a RedrawRequester that encapsulated a shared atomic boolean flag and a 'waker' (i.e. looper.wake()). This gives more ready access to the flag for any subsequent iteration of the event loop (woken for any reason) and doesn't lead to any buffering of data (except for potentially redundant wakes which Android already tries to expunge automatically). The lack of buffering is notable compared to e.g. the X11 backend that uses an mpsc channel for buffering redraw requests which also doesn't seem like an ideal fit for the problem.

Java <-> Native Synchronization in ndk-glue

This is a key area where I was very uncertain about the current ndk-glue design and how robustly is handles synchronization between the java main thread and the rust native thread for operations such as destroying the applications native window and for saving state.

As far as I've seen, ndk-glue employs a purely cooperative synchronization scheme that documents what downstream users of the API must do to ensure synchronization.

There is this comment for the WindowDestroyed event enum:

/// If the window is in use by ie. a graphics API, make sure the lock from
/// [`native_window()`] is held on to until after freeing those resources.
///
/// After receiving this [`Event`] `ndk_glue` will block until that read-lock
/// is released before returning to Android and allowing it to free up the window.
WindowDestroyed,

which essentially documents an implementation detail that says: if you keep the the read lock guard after querying the current native_window() you can effectively block ndk-glue from being able to clear the native window, which will ultimately force synchronization with the Java thread in case it gets notified of a window termination (because it won't be able to write the change).

In practice though winit's Android backend doesn't appear to take a read lock during particularly critical event callbacks, such as for Redraw events so winit doesn't seem to honor the documented synchronization scheme to help block the native window from being torn out in the middle of rendering.

What's also notably different to the original android_native_app_glue provided for use with NativeActivity by Google is that there's no guarantee that the native window remains accessible at least until the WindowDestroyed event has been received and the application has had an opportunity to react. By the time the application sees a WindowDestroyed event the native window could already be long gone.

Something I did differently in the game-activity glue implementation was provide a poll_events() API that takes an FnMut closure that is in some ways comparable to how winit takes a closure that's called for each event. The important thing this enables is that the implementation can place arbitrary pre- and -post logic around the handling of any Android event, which provides a robust place to handle any necessary synchronization with the Java main thread as required for different events, including for window termination and state saving. Since this design also fully encapsulates synchronization, there's no need to downstream users to handle anything and synchronization details can also be changed without affecting applications.

This is something that would also be good to discuss as an ndk-glue issue to see if it makes sense to change its current design but I figure it also makes sense to highlight here too.

High-level Android portability questions

In the process of testing the winit backend and e.g. looking to get egui and Bevy running I realized that if you follow most existing examples for how to use Winit and look at existing integrations you don't tend to end up with an application that is portable to Android (ignoring things like main() function quirks.)

In particular Android is currently unique in requiring applications to be aware that the .native_window() for a winit window will be invalidated between Suspended and Resumed event pairs, and also requiring applications to recreate any render surfaces each time their application is Resumed.

Existing winit integrations tend to assume they can create a window and initialize all graphics state + rendering surfaces up front when they create their event loop, which will ultimately just lead to a panic on Android once the app tries to access a native_window before it has been Resumed.

Here's an example of an upstream PR for egui that attempts to update their winit + wgpu integration abstractions so that it can support Android: emilk/egui#1634

I think their may be some opportunities within Winit to help steer downstream users into building portable integrations and applications. E.g. one idea I've wondered about is whether we could make all platforms consistently deliver a Resumed event (even desktop window systems) and then encourage (by updating examples) that this should be the standard place for all applications to lazily initialize all their graphics state and create their rendering surfaces. There would still be the separate requirement on Android to have to re-create new surfaces for each future Resumed event but it would already be quite a big improvement in terms of consistent application structure that could help encourage portability by default.

Next steps

... okey, I guess I'll stop here for now, since this is already a pretty big dump of information. :)

I'd be interested to gauge interest in any of this, and would be happy to split out separate issues for some of the things mentioned above.

One big question that could be interesting to discuss is whether there might be an interest in updating Winit's Android backend to work with this game-activity glue layer, or something similar considering:

• Some of the concerns around synchronization that the current ndk-glue design appears to have?
• The ability to support AppCompatActivity based Android applications which includes back-ported Android APIs that make it much more practical to develop Android applications that are compatible with a wider range of Android versions.
• Being in a better position to leverage more of the AGDK native libraries, e.g. for improved ime text input handling, game controller support, "swappy" synchronization for rendering etc.

Alternatively maybe it'd be possible to define a standard "glue" API, and maybe have something similar to ndk-context for the glue layer that would make it possible for applications to choose their glue (though I suspect that adding additional abstractions here might also just impede improvements to Android support more than help). I think initially the main challenge with this direction would be with defining a standard input API considering that game-activity is not based on AInputQueue which NativeActivity apps tend to use.

Thanks for your time if you made it this far! :-D

[maroider]

I very much agree that Android is a bit of an ugly duckling compared to our other backends in terms of the handling of the lifetime of the native window, and rectifying that situations is definitely on my radar, although it's not at the top of my list of priorities as of now.

PS: I do appreciate the long issue text, as a fellow poster of such issues/comments :D

[rib]

PS: I do appreciate the long issue text, as a fellow poster of such issues/comments :D

hehe, thanks 👍

I very much agree that Android is a bit of an ugly duckling compared to our other backends in terms of the handling of the lifetime of the native window, and rectifying that situations is definitely on my radar, although it's not at the top of my list of priorities as of now.

I'd be interested to hear if you had any particular ideas in mind already re: handling the way in which Android's native windows come and go over the lifetime of the application?

There are a few discussions open atm relating to the handling of lifecycle events in Winit:
#2185
#2144

where I've noted that the Android backend currently tells a bit of a white lie with regards to the application's lifecycle state. The Android backend currently drives it Suspended / Resumed state based on when it's native surface is destroyed (via an on onSurfaceDestroyed callback from Java. The general assumption / observation is that this usually corresponds to the onPaused lifecycle event, but this isn't something I've ever seen as documented / promised by Android - and I can imagine it's something that could change between versions of Android. A quick search today for a documented connection actually led me to find this page: https://stackoverflow.com/questions/69422482/lifecycle-of-surfaceview which seems to demonstrate an example where that assumption is broken.

I don't think there's anything we can do on Android to simply bypass this protocol of having the surface be destroyed, so I think I currently see two general approaches for better supporting the behaviour in Winit:

1. Embrace the notion that a Winit Window is only a logical window which may conceptually get detached/reattached to some underlying native window/surface if necessary for some backends (like Android). In this case I'd consider adding explicit events for this so we don't have to overload the Suspended/Resumed events and it could be good to change the .native_window() getter to return an Option<> instead of panicking like it does currently on Android.

2. Explicitly start emitting WindowEvent::Destroyed events on Android whenever the native surface is destroyed and then require downstream users to be prepared to create a new window when.... the app next resumes?

When I think about how I've seen downstream integrations build on top of Winit, such as in egui or Bevy my impression is that the more practical option is (1), which is generally similar to the current situation but possibly slightly formalized.

[kchibisov]

I think their may be some opportunities within Winit to help steer downstream users into building portable integrations and applications. E.g. one idea I've wondered about is whether we could make all platforms consistently deliver a Resumed event (even desktop window systems) and then encourage (by updating examples) that this should be the standard place for all applications to lazily initialize all their graphics state and create their rendering surfaces. There would still be the separate requirement on Android to have to re-create new surfaces for each future Resumed event but it would already be quite a big improvement in terms of consistent application structure that could help encourage portability by default.

That could make sense for Wayland. So winit can support layer-shell. It basically gets destroyed from time to time, but the problem is that it differs a lot from normal windows...

Not sure if it'll be a good idea to add special shells in winit.

Embrace the notion that a Winit Window is only a logical window which may conceptually get detached/reattached to some underlying native window/surface if necessary for some backends (like Android). In this case I'd consider adding explicit events for this so we don't have to overload the Suspended/Resumed events and it could be good to change the .native_window() getter to return an Option<> instead of panicking like it does currently on Android

That won't work with applications requiring RawWindowHandle. Since it would mean that it could invalidate and everything should be recreated. Not sure something like that makes sense for other platforms. I mean, that's your intention, but it could be confusing. The clearer aproach is notyfind that the window got destroyed.

The problem is that there should be some notion when system expects you to recreate the window...

[rib]

That won't work with applications requiring RawWindowHandle. Since it would mean that it could invalidate and everything should be recreated. Not sure something like that makes sense for other platforms. I mean, that's your intention, but it could be confusing. The clearer aproach is notyfind that the window got destroyed.

Just to clarify; this is how things currently work on Android - the RawWindowHandle changes over the lifetime of the Winit window, and from an integration pov (e.g. looking at what's convenient in things like Bevy + egui) my impression is that it would complicate things to start mapping this into really destroying the top-level window.

I think terminology matters a lot here and unfortunately I think it's a little misleading on Android; especially the fact that 'surface views' have to be mapped to 'raw window' handles since surface views are a lower level object for rendering - distinct from the the Android 'Window' class (https://developer.android.com/reference/android/view/Window) that doesn't get taken away. (Note: the RawWindowHandle is a native handle that can be used to create a GPU graphics context - where it's platform specific whether this really maps to a logical 'window')

On Android 'raw window handle' can be seen as a bit of a misnomer for that platform. Even the underlying 'surface view' is a bit of a misnomer when it comes to understanding conceptually what it is. A better name could be 'swap chain' or 'buffer queue'. It's more closely related to the kind of surface you create and configure via wgpu for rendering - so more of an implementation detail that impacts rendering. Android has it's own high level "Window" class and it's notable that the application's Android Window isn't getting destroyed / recreated. It probably doesn't help that the original android_native_app_glue from Google refers to surface view destruction/creation as a 'window'.

A Winit Window is something that tends to be most meaningful to applications, and typically they don't care much about the details and just say "I want a window please - and give it this title if possible". From an app pov they'd rather not deal with windows being torn away arbitrarily due to system issues, and it can make sense to associate high-level state with a window that you probably expect to last for the lifetime of the application.

To briefly try and compare it more to Wayland (considering there is overlapping terminology that has different meanings); with Wayland you have wl_surface objects that correspond to 'windows' in winit and then you post wl_buffers to present to those windows/surfaces. On Android when your surface view is destroyed (what a raw handle relates to) that's kind of like an embargo on creating buffers. A surface view conceptually represents a kind of swap chain that's managed by the OS, which you need for rendering, which is notably different to how wayland allocates its buffers). Imagine you had an intermediate 'swap chain' object that you requested wl_buffers from, and that sometimes you'd get notified that the swap chain was destroyed. You would still have your wl_surfaces in tact (your windows for the application are logically still fine and you wouldn't want to tell the application their window has been destroyed) but now you need to wait until you're given a new swap chain before you can allocate any more wl_buffers to post to your surface/window - that's essentially what's happening on Android.

I think the situation with Wayland layer shells is a little different here. In that case an application acting as a shell layer will know what it's designed for. The protocol allows for the shell to notify the application that the layer has effectively become redundant because there's no longer an output for it to be anchored too or the user chose to close the layer widget and at a high level the application would be prepared to respond to that.

I think in the case of layer shells it might be reasonable to map the closed notifications to Destroyed events, but I'd guess it would also be reasonable to expect that this kind of application is Wayland specific (or at least have Wayland specific implementation details that can directly respond to the closed event in its own way.) One of the notable reasons a shell layer would get a closed event is that the user wants decides to close the widget - and in that sense mapping to Destroyed would probably be ok.

So in summary, I'd say I tend to see the transient nature of raw (surface view) handles as a lower-level implementation detail on Android related to rendering, and it doesn't really represent a high level event that applications should be closely concerned with - except as far as knowing they are blocked from rendering. The terminology is what's confusing things here. It's just that other OSs don't have this same architecture where they can take away your swap chain and take away your ability to allocate new back buffers in such an explicit way. Mapping this to window Destroyed events on Android would be a bit like notifying desktop applications that their window was destroyed whenever the user alt tabbed to another app or minimized the app - which would be quite frustrating for applications to deal with.

Hope that all makes sense?

[Hoodad]

Pitching in here. I like to stress that support for getting the Unicode characters from the virtual keyboard is (to my knowledge at least) missing in NativeActivity and resolved in the GameActivity which adds proper support for TextInput.
You can do a dirty conversion (and I have done that) to convert key codes together with meta state and get support for characters A-Z and 0-9 plus some limited number of special characters. But that's where it currently ends. Adding full support does not seem possible.

So if anything this is just more wood on the fire 🔥

[rib]

Cool, thanks, yeah I'm not exactly sure what the ideal next steps here should be atm.

Since I was thinking there might still be an interest in maintaining support for NativeActivity in winit (which might be appealing for anyone that really despises building stuff with Gradle perhaps ;-p) I actually also recently implemented an alternative native-activity glue layer in the same repo: https://github.com/rib/agdk-rust/tree/main/native-activity and with a little iteration I made it API compatible with game-activity to the extent that my Winit branch is now able to conditionally build against either crate.

(If you see the examples here there are now some with an na- prefix that test running with NativeActivity, including with winit, and they can also be built with cargo apk.)

In this case the native-activity crate would essentially be a replacement for ndk-glue but I think some of the design differences (such as with how it handles synchronization with Java) could make it worthwhile to consider. One other feature native-activity currently supports is the ability to save/restore application state which is notably something that depends on correct synchronization between Rust and Java.

I'm half wondering if there would be interest in a PR that would first make some of the smaller tidy up changes I made to the Android backend and switch over to native-activity. Since that's API compatible with game-activity it might then be easier to figure out incremental ways of enabling GameActivity as an optional, alternative Activity.

[msiglreith]

To provide a larger coverage of general features and mobile specific parts, having a custom Activity or moving away from NativeActivity in general seems necessary. SDL has also a non-trivial amount of Java code in their codebase.

Currently I see support for Java/gradle in cargo-apk or xbuild (which seems to be the future) as one main blocking tasks. Porting stuff to Android seems to be a common point of friction, so reducing the overhead for it would be great :D

[MarijnS95]

I'd like to not require additional Java code for pure native applications, or at the very least have that option available with winit. When the user requires Java interop and/or additional features they're probably better off building their app in Java/Kotlin/Flutter and using Mozilla's rust-gradle plugin to take care of the rest.

(Implying lifecycle management and other bits stay in Java-land too - the ndk already has support for rendering to individual surfaces etc).

[rib]

I'd like to not require additional Java code for pure native applications, or at the very least have that option available with winit. When the user requires Java interop and/or additional features they're probably better off building their app in Java/Kotlin/Flutter and using Mozilla's rust-gradle plugin to take care of the rest.

(Implying lifecycle management and other bits stay in Java-land too - the ndk already has support for rendering to individual surfaces etc).

I might be misunderstanding this but I'm not sure I agree with the assertions here...

It's really easy to stray beyond what's possible purely based on NativeActivity because there are a number of important Activity features that require you to subclass and override methods in the Activity and there's no way around that. (I'd probably go so far as to say it's inevitable for any Android application that's more than a bare-bones rendering test)

An example that affected me recently was with using the Bluetooth API on Android and wanting to use the Companion API to select a device. For this you need to launch another activity with an Intent and get a result back (with the user's choice) via the onActivityResult method.

I think there are lots of different use cases for an Android app to launch another Activity via an Intent and want to get a result back (for various system helpers), since that's a fairly primitive Android mechanism.

Another example is the onNewIntent callback which @Hoodad above has recently found they needed: rust-mobile/ndk#275 (comment)

If you look at the docs for the Activity class you can find lots of onXyz methods that aren't all integrated with NativeActivity and there's a good chance for any app to eventually need to integrate with one of them eventually.

I don't think needing these features really implies much about the complexity of the application or whether it should be built via Java/Kotlin/Flutter instead of being a native Rust application.

If your app wants to render with opengl/vulkan/wgpu such as a Bevy game, or using a UI like Egui (or any Rust native UI that renders via wgpu/gl), and especially if you want the app to be portable between mobile and desktop etc then I can see that it'd be desirable to want to structure that as a standard native application (as opposed to writing a Rust plugin for a Java/Kotlin application).

Overall I think that, in practice, any application for Android that isn't a just a quick test/demo/toy will inevitably need to integrate with Java for something - It's just the nature of the beast with Android applications all being JVM based. Like with iOS you wouldn't get far without needing to write some objective C code to access the rest of the system.

When it comes to Winit I think it could be nice to try and preserve support for NativeActivity if possible - due to the convenience of being able to build and run minimal apps without compiling any extra Java code but I also think it'd be good to also help support applications that are based on different Activity subclasses (not necessarily sucblasses of NativeActivity).

[MarijnS95]

When it comes to Winit I think it could be nice to try and preserve support for NativeActivity if possible - due to the convenience of being able to build and run minimal apps without compiling any extra Java code but I also think it'd be good to also help support applications that are based on different Activity subclasses (not necessarily sucblasses of NativeActivity).

That sums it up nicely. Winit doesn't need to lock into "pure" native apps without Java, but it should definitely not be a requirement.