[pigeon]: Support Swift async/await and Kotlin suspend methods #8341

feduke-nukem · 2024-12-23T10:37:56Z

adds support of generating Swift async/await and Kotlin suspend methods in HostApi.
adds Async annotation and AsyncType.callback/AsyncType.await to specify the style of generating async methods for Swift and Kotlin

Pre-launch Checklist

If you need help, consider asking for advice on the #hackers-new channel on Discord.

hellohuanlin · 2024-12-23T19:13:07Z

packages/pigeon/example/app/ios/Runner/AppDelegate.swift

@@ -25,6 +25,16 @@ private class PigeonApiImplementation: ExampleHostApi {
    }
    completion(.success(true))
  }
+
+  func sendMessageModernAsync(message: MessageData) async throws -> Bool {


you can just overload the function name func sendMessage(...)

I don't think that I understood.

It's generated Host API code. We could not overload functions in Dart even if we do in Swift

hellohuanlin · 2024-12-23T19:14:27Z

packages/pigeon/example/app/ios/Runner/AppDelegate.swift

@@ -25,6 +25,16 @@ private class PigeonApiImplementation: ExampleHostApi {
    }
    completion(.success(true))
  }
+
+  func sendMessageModernAsync(message: MessageData) async throws -> Bool {
+    try? await Task.sleep(nanoseconds: 2_000_000_000)


is this used for unit test? we shouldn't wait for so long as it slows down the tests

This is used in example app for demonstration purposes

I am not sure if it is used in any tests. I could remove this code if it is necessary

Removed Task.sleep

hellohuanlin · 2024-12-23T19:15:42Z

packages/pigeon/example/app/ios/Runner/Messages.g.swift

+        Task {
+          do {
+            let result = try await api.sendMessageModernAsync(message: messageArg)
+            DispatchQueue.main.async {


you can do Task { @MainActor } here.

you can do Task { @MainActor } here.

Do you mean:

sendMessageModernAsyncChannel.setMessageHandler { message, reply in let args = message as! [Any?] let messageArg = args[0] as! MessageData Task { do { @MainActor let result = try await api.sendMessageModernAsync(message: messageArg) reply(wrapResult(result)) } catch { reply(wrapError(error)) } } }

Will it lead to executing api.sendMessageModernAsync on main thread that could block it?

No, i meant you can replace DispatchQueue.main.async {} with Task { @MainActor}

No, i meant you can replace DispatchQueue.main.async {} with Task { @MainActor}

What's the point/profit?

The purpose of swift concurrency is to replace GCD. So since you are using swift concurrency here, there's no reason to use GCD at all.

GCD

Okay I get it.

I am not very familiar with swift concurrency actually. Is that what you meant?

sendMessageModernAsyncChannel.setMessageHandler { message, reply in let args = message as! [Any?] let messageArg = args[0] as! MessageData Task { do { let result = try await api.sendMessageModernAsync(message: messageArg) await Task { @MainActor in reply(wrapResult(result)) } } catch { await Task { @MainActor in reply(wrapError(error)) } } } }

Maybe we could use MainActor.run instead? I think it is more readable.

sendMessageModernAsyncChannel.setMessageHandler { message, reply in let args = message as! [Any?] let messageArg = args[0] as! MessageData Task { do { let result = try await api.sendMessageModernAsync(message: messageArg) await MainActor.run(){ reply(wrapResult(result)) } } catch { await MainActor.run(){ reply(wrapError(error)) } } } }

In your first chunk of code, don't put await in front of Task.

For this particular case, you can use MainActor.run { ... }, since reply doesn't require an async context.

@hellohuanlin done

packages/pigeon/example/app/ios/Runner/Messages.g.swift

feduke-nukem · 2024-12-24T08:43:04Z

@hellohuanlin @LouiseHsu @tarrinneal CI check fails:

Changed packages: pigeon
[0:00] Running for packages/pigeon...
  No version change.
  Found NEXT; validating next version in the CHANGELOG.
No version change found, but the change to this package could not be verified to be exempt
from version changes according to repository policy.
If this is a false positive, please comment in the PR to explain why the PR
is exempt, and add (or ask your reviewer to add) the "override: no versioning needed" label.

Do I need to bump version in CHANGELOG.md? Currently I append related to pr changes below NEXT label.

hellohuanlin · 2024-12-26T18:55:55Z

packages/pigeon/example/app/ios/Runner/Messages.g.swift

+        let messageArg = args[0] as! MessageData
+        Task {
+          let result = await api.sendMessageModernAsync(message: messageArg)
+          await MainActor.run {


Why doesn't the old version switch thread? Is setMessageHandler's callback guaranteed to be called on main thread?

The old version uses the callback approach that doesn't introduce asynchronous things like task so there was no need to switch to main to reply via channel

Is setMessageHandler's callback guaranteed to be called on main thread?

I guess so as it is platform channel way of communication

Have found implementation where I see dispatch_async(dispatch_get_main_queue()

After marking the func definition as @MainActor discussed here, here we can just do:

Task { @MainActor in let result = await api.send... reply(wrapResult(result)) }

After marking the func definition as @MainActor discussed here, here we can just do:

Task { @MainActor in let result = await api.send... reply(wrapResult(result)) }

No problem it could be easily done.

But could you explain to me how it would work?

If we mark the whole function with @mainactor doesn't that mean that some expensive async/await function call could potentially be called at main?

If we mark the whole function with @mainactor doesn't that mean that some expensive async/await function call could potentially be called at main?

The API contract is that this function will be called on main. The old API is also called on main and this does not change that. Marking it as @MainActor will allow compiler to enforce the implementor of this API to only use API that's safe to be used on main thread. That's one of the benefit of swift concurrency.

An expensive aync/await call inside this function will be called on the thread specified by that call. For example,

@MainActor func sendMessageModernAsync() { let foo = await myActor.getExpensiveFoo() }

Here getExpensiveFoo will be called under the async context of myActor, rather than that of the main actor.

hellohuanlin · 2024-12-26T19:09:15Z

packages/pigeon/example/app/ios/Runner/AppDelegate.swift

@@ -25,6 +25,10 @@ private class PigeonApiImplementation: ExampleHostApi {
    }
    completion(.success(true))
  }
+
+  func sendMessageModernAsync(message: MessageData) async -> Bool {


I just looked at the code again - it looks like the contract of pigeon is that the API (e.g. sendMessage) should be called on main thread. If that's the contract we want to preserve, we should mark the async API as main actor.

@MainActor func sendMessageModernAsync(message: MessageData) async -> Bool {}

The only thing that should be called on the main thread is dispatching messages via channel

https://docs.flutter.dev/platform-integration/platform-channels#jumping-to-the-main-thread-in-ios

If we mark the whole function with @MainActor doesn't that mean that some expensive async/await function call could potentially be called at main?

iiuc both way of communication needs to be on main thread.

The link you add here is calling from platform to dart side. This is the opposite side communication, which needs to be on main as well.

Actually it is the same from the point of dispatching reply via channel, be it call from platform to dart or replying to call from dart

it looks like this isn't addressed yet?

I would expect that would be an easy fix.

Is this meant to be a separate pull request?

A prequel PR might be cleaner, but it's since it's closely related to this PR and should be small, including it in this PR would probably also be reasonable unless @tarrinneal would prefer it separate.

wow, I don't know how that one has been sitting for so long. I'm fine with whatever is easier for @feduke-nukem. I think having a second pr would be cleaner, easier to review. I am fine with either way though.

I prefer separate PRs.

But what would the scenario be in such a case?

I create a new PR.

This PR waits for the new one.

As soon as the new PR is approved or merged, I merge the changes into this PR.

Or are there any other steps?

# Conflicts: # packages/pigeon/CHANGELOG.md

tarrinneal

I'll get to an in depth review as soon as I can. Just wanted to answer your question about failing checks.

Thank you for putting the effort into this pr.

packages/pigeon/CHANGELOG.md

…compile with /bigobj fix

packages/pigeon/platform_tests/test_plugin/windows/CMakeLists.txt

tarrinneal

Just realized I had unpublished review comments. Still not a complete review though.

packages/pigeon/CHANGELOG.md

packages/pigeon/README.md

tarrinneal · 2025-01-17T17:41:45Z

packages/pigeon/example/README.md

+  func sendMessageModernAsyncThrows(message: MessageData) async throws -> Bool {
+    throw PigeonError(code: "code", message: "message", details: "details")
+  }


Maybe throw a (if message.whatever == whatever) return whatever else throw just so people can see some use here.

That's a great idea!

What do you think about also simplifying the core_tests methods by reducing them to one?

/// Returns the passed object, to test async serialization and deserialization using `await`-style /// and Swift can throw an exception. @Async(type: AsyncType.await(isSwiftThrows: true)) @ObjCSelector('echoModernAsyncAllTypesAndNotThrow:') @SwiftFunction('echoModernAsyncAllTypesAndNotThrow(_:)') AllTypes echoModernAsyncAllTypesAndNotThrow(AllTypes everything); /// Returns the passed object, to test async serialization and deserialization using `await`-style /// and throws an exception. @Async(type: AsyncType.await(isSwiftThrows: true)) @ObjCSelector('echoModernAsyncAllTypesAndThrow:') @SwiftFunction('echoModernAsyncAllTypesAndThrow(_:)') AllTypes echoModernAsyncAllTypesAndThrow(AllTypes everything);

to a single AllTypes echoModernAsyncAllTypesAndThrow(AllTypes everything);

This way, we could handle both cases in the tests with the condition you mentioned earlier. Would that work for you?

no :( We need to handle both generated code paths, even though they are mostly redundant.

tarrinneal · 2025-01-17T17:42:19Z

packages/pigeon/example/README.md

+Future<bool> sendMessageModernAsyncAndThrow(String messageText) {
+  final MessageData message = MessageData(
+    code: Code.two,
+    data: <String, String>{'header': 'this is a header'},
+    description: 'uri text',
+  );
+
+  return _api.sendMessageModernAsyncThrows(message);
+}


Let's remove the redundant methods from languages that don't support the exampled feature.

Could you clarify how that should be done? The tool/generate.dart generates code for all supported languages and platforms.

I just mean they don't need to be included in the README

I just mean they don't need to be included in the README

This README is generated with script based on the code from example app.

I am not sure what should be done then.

Looks like that it should be isolated HostApi and excluded from being generated for unsupported platforms like EventChannelTests and EventChannelApi was done.

It will significantly reduce the amount of useless code (only Kotlin and Swift is supported)

I left another comment with instructions, I don't think it's worth adding a new pigeon file for this. We can leave the code in the other language example files, it just needs to be removed from the README

Have removed for Linux and Windows

tarrinneal · 2025-01-17T17:45:38Z

packages/pigeon/example/README.md

+static void handle_send_message_modern_async_throws(
+    PigeonExamplePackageMessageData* message,
+    PigeonExamplePackageExampleHostApiResponseHandle* response_handle,
+    gpointer user_data) {
+  g_autoptr(FlValue) details = fl_value_new_string("details");
+  pigeon_example_package_example_host_api_respond_error_send_message_modern_async_throws(
+      response_handle, "code", "message", details);
+}


Just noting them all to help with tracking (it's useful for me at least)

tarrinneal · 2025-01-17T17:45:49Z

packages/pigeon/example/README.md

+    .send_message_modern_async_throws =
+        handle_send_message_modern_async_throws};


tarrinneal · 2025-01-21T23:50:20Z

packages/pigeon/example/app/ios/Runner/AppDelegate.swift

@@ -25,6 +25,10 @@ private class PigeonApiImplementation: ExampleHostApi {
    }
    completion(.success(true))
  }
+
+  func sendMessageModernAsync(message: MessageData) async -> Bool {


wow, I don't know how that one has been sitting for so long. I'm fine with whatever is easier for @feduke-nukem. I think having a second pr would be cleaner, easier to review. I am fine with either way though.

packages/pigeon/lib/ast.dart

tarrinneal · 2025-01-21T23:59:44Z

packages/pigeon/lib/kotlin_generator.dart

+            if (asynchronousType.isAwait) {
+              indent.writeln('withContext(Dispatchers.Main) {');
+              indent.inc();
+            }
            indent.writeln('reply.reply(wrapped)');
+            if (asynchronousType.isAwait) {
+              indent.dec();
+              indent.writeln('}');
+            }
          }
        });
+        if (asynchronousType.isAwait) {
+          indent.dec();
+          indent.writeln('}');
+        }


I need to add a method for handling optional indents, and maybe open/close scope across [pigeon] scopes.

Just a note for myself, you don't need to change anything.

Changed CHANGELOG.md description Changed README.md

…nukem/packages into issue-123867-async-await

tarrinneal · 2025-01-23T17:50:20Z

packages/pigeon/example/app/windows/runner/flutter_window.cpp

+      const MessageData& message,
+      std::function<void(ErrorOr<bool> reply)> result) {
+    result(FlutterError("code", "message", "details"));
+  }
 };
 // #enddocregion cpp-class


This comment tells the script where to end the snippet. You can start and stop a snippet multiple times to add a ... and skip lines you don't want included as well. So here I would end the snippet after SendMessageModernAsync then start it again after SendMessageModernAsyncThrows and leave the end comment where it is.

Let me know if that's too confusing of an explanation.

This comment tells the script where to end the snippet. You can start and stop a snippet multiple times to add a ... and skip lines you don't want included as well. So here I would end the snippet after SendMessageModernAsync then start it again after SendMessageModernAsyncThrows and leave the end comment where it is.

Did I get it correct?

// #docregion host-definitions @HostApi() abstract class ExampleHostApi { String getHostLanguage(); // These annotations create more idiomatic naming of methods in Objc and Swift. @ObjCSelector('addNumber:toNumber:') @SwiftFunction('add(_:to:)') int add(int a, int b); @async bool sendMessage(MessageData message); // #enddocregion host-definitions bool unwantedMethodInReadme(); bool anotherUnwantedMethodInReadme(); // #docregion host-definitions } // #enddocregion host-definitions

Will produce README:

@HostApi() abstract class ExampleHostApi { String getHostLanguage(); // These annotations create more idiomatic naming of methods in Objc and Swift. @ObjCSelector('addNumber:toNumber:') @SwiftFunction('add(_:to:)') int add(int a, int b); @async bool sendMessage(MessageData message); // ··· }

# Conflicts: # packages/pigeon/CHANGELOG.md # packages/pigeon/lib/generator_tools.dart # packages/pigeon/pubspec.yaml

…d implementation

…nukem/packages into issue-123867-async-await

tarrinneal · 2025-01-28T23:34:30Z

You don't need to merge main unless there is a conflict that needs resolving. It actually makes it harder to review (and sends people an email every time).

feduke-nukem · 2025-01-28T23:39:10Z

You don't need to merge main unless there is a conflict that needs resolving. It actually makes it harder to review (and sends people an email every time).

Whoops, my bad.

Didn't mean to do it like that. It won't happen again.

tarrinneal · 2025-01-28T23:40:12Z

It's not a big deal, just thought I'd let you know

feduke-nukem requested review from hellohuanlin, LouiseHsu and tarrinneal as code owners December 23, 2024 10:37

github-actions bot added p: pigeon platform-android platform-ios platform-linux platform-macos platform-windows labels Dec 23, 2024

feduke-nukem changed the title ~~[pigeon]: added support of modern asynchronous api for Swift and Kotlin~~ [pigeon]: Support of modern asynchronous api for Swift and Kotlin Dec 23, 2024

hellohuanlin reviewed Dec 23, 2024

View reviewed changes

packages/pigeon/example/app/ios/Runner/Messages.g.swift Show resolved Hide resolved

feduke-nukem requested a review from hellohuanlin December 23, 2024 21:22

feduke-nukem force-pushed the issue-123867-async-await branch from fdb7254 to a8df692 Compare December 23, 2024 21:58

added support of modern async api Swift/Kotlin

583d6d7

feduke-nukem force-pushed the issue-123867-async-await branch from a8df692 to 583d6d7 Compare December 23, 2024 22:00

feduke-nukem and others added 3 commits December 24, 2024 10:53

issue(123867): added ability to specify swift throws in modern async

934c5c7

Merge branch 'main' into issue-123867-async-await

8ec409d

issue(123867): fix ci checks

52de826

hellohuanlin reviewed Dec 26, 2024

View reviewed changes

feduke-nukem added 2 commits December 27, 2024 15:28

issue(123867): added @mainactor for Swift modernAsync

cd9c381

Merge branch 'main' into issue-123867-async-await

84ef399

# Conflicts: # packages/pigeon/CHANGELOG.md

feduke-nukem requested a review from hellohuanlin December 27, 2024 12:35

tarrinneal requested changes Dec 27, 2024

View reviewed changes

packages/pigeon/CHANGELOG.md Outdated Show resolved Hide resolved

feduke-nukem added 2 commits December 27, 2024 15:40

issue(123867): run tool/generate.dart

0ac2004

issue(123867): fix @mainactor

70e0aa3

feduke-nukem and others added 3 commits January 19, 2025 18:35

issue(123867): number of sections exceeded object file format limit: …

785cff5

…compile with /bigobj fix

Merge branch 'main' into issue-123867-async-await

7966ead

Merge branch 'main' into issue-123867-async-await

9ceaccd

stuartmorgan reviewed Jan 21, 2025

View reviewed changes

packages/pigeon/platform_tests/test_plugin/windows/CMakeLists.txt Outdated Show resolved Hide resolved

feduke-nukem and others added 2 commits January 21, 2025 19:23

issue(123867): removed condition for add_compile_options("/bigobj")

a3718dd

Merge branch 'main' into issue-123867-async-await

693aaf6

tarrinneal requested changes Jan 22, 2025

View reviewed changes

feduke-nukem and others added 4 commits January 23, 2025 11:23

Merge branch 'main' into issue-123867-async-await

d47d0a0

issue(123867): fix some review comments

8f196a4

Changed CHANGELOG.md description Changed README.md

Merge branch 'issue-123867-async-await' of https://github.com/feduke-…

eb9cdb4

…nukem/packages into issue-123867-async-await

issue(123867): fix typo in docs

3b74da1

tarrinneal reviewed Jan 23, 2025

View reviewed changes

feduke-nukem and others added 10 commits January 23, 2025 23:07

Merge branch 'main' into issue-123867-async-await

f9b7922

fix(issue-123867): removed unsupported platforms' examples

3dbd764

Merge branch 'main' into issue-123867-async-await

0652344

Merge branch 'main' into issue-123867-async-await

f803cfc

# Conflicts: # packages/pigeon/CHANGELOG.md # packages/pigeon/lib/generator_tools.dart # packages/pigeon/pubspec.yaml

Merge branch 'main' into issue-123867-async-await

8b95706

Merge branch 'main' into issue-123867-async-await

9b18fab

issue(123867): changed example app sendMessageModernAsyncThrows metho…

8d0211d

…d implementation

Merge branch 'issue-123867-async-await' of https://github.com/feduke-…

fc5c987

…nukem/packages into issue-123867-async-await

issue(123867): fix handle_send_message_modern_async_throws windows

4a05c46

Merge branch 'main' into issue-123867-async-await

3823553

stuartmorgan mentioned this pull request Jan 24, 2025

[pigeon] Adds support of TaskQueue for Swift #8501

Open

11 tasks

feduke-nukem requested a review from tarrinneal January 24, 2025 23:23

feduke-nukem added 3 commits January 27, 2025 11:14

Merge branch 'main' into issue-123867-async-await

5bb8568

Merge branch 'main' into issue-123867-async-await

2644975

Merge branch 'main' into issue-123867-async-await

7214802

		.send_message_modern_async_throws =
		handle_send_message_modern_async_throws};

[pigeon]: Support Swift async/await and Kotlin suspend methods #8341

Are you sure you want to change the base?

[pigeon]: Support Swift async/await and Kotlin suspend methods #8341

Conversation

feduke-nukem commented Dec 23, 2024 • edited Loading

Pre-launch Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

feduke-nukem Dec 23, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

feduke-nukem Dec 23, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

feduke-nukem commented Dec 24, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

feduke-nukem Dec 26, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hellohuanlin Dec 26, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

feduke-nukem Dec 26, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tarrinneal left a comment

Choose a reason for hiding this comment

tarrinneal left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tarrinneal Jan 22, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tarrinneal commented Jan 28, 2025

feduke-nukem commented Jan 28, 2025

tarrinneal commented Jan 28, 2025

feduke-nukem commented Dec 23, 2024 •

edited

Loading

feduke-nukem Dec 23, 2024 •

edited

Loading

feduke-nukem Dec 23, 2024 •

edited

Loading

feduke-nukem Dec 26, 2024 •

edited

Loading

hellohuanlin Dec 26, 2024 •

edited

Loading

feduke-nukem Dec 26, 2024 •

edited

Loading

tarrinneal Jan 22, 2025 •

edited

Loading