Emit events for all known types #127

dmdashenkov · 2023-03-17T09:36:54Z

In this PR we enable ProtoData plugin authors to listen to Protobuf compiler events that describe types from dependencies.

Previously, events like TypeEntered were only emitted for Proto definitions from within the module that ProtoData is processing. This is good for many cases, since this removes the need to check whether or not any code has to be generated based on a given event. However, this limited the users to only the types defined in the module. There was no way to know about a Proto message type from outside the module.

Now, we introduce a new DepdendencyDiscovered event and a ProtobufDependencyFile view. The two new types allow ProtoData plugin authors to generate code with awareness of the module's dependencies.

codecov · 2023-03-17T09:59:32Z

Codecov Report

Merging #127 (ff3ab1e) into master (23108f9) will increase coverage by 2.38%.
The diff coverage is 94.17%.

Additional details and impacted files

@@             Coverage Diff              @@
##             master     #127      +/-   ##
============================================
+ Coverage     79.35%   81.73%   +2.38%     
- Complexity      236      240       +4     
============================================
  Files            73       79       +6     
  Lines          1889     1993     +104     
  Branches        140      146       +6     
============================================
+ Hits           1499     1629     +130     
+ Misses          335      313      -22     
+ Partials         55       51       -4

dmdashenkov · 2023-03-17T11:26:50Z

@armiol, @alexander-yevsyukov, PTAL.

This is a small yet consequential change set. Let's further discuss it a video call if necessary.

alexander-yevsyukov

Please see my comments. In general, I think that the generation_requested flag approach should be discussed.

alexander-yevsyukov · 2023-03-17T12:07:56Z

.github/workflows/build-on-ubuntu.yml

          report_paths: '**/build/test-results/test/TEST-*.xml'
          require_tests: true # will fail workflow if test reports not found

-      # See: https://github.com/marketplace/actions/junit-report-action


Can we revert these changes please? Separation of test reports was needed some time ago when investigating build failures at CI.

@dmdashenkov I think this comment still stands.

alexander-yevsyukov · 2023-03-17T12:12:58Z

api/src/main/kotlin/io/spine/protodata/event/EnumCompilerEvents.kt

@@ -68,13 +68,15 @@ internal class EnumCompilerEvents(
            EnumEntered.newBuilder()


Please consider using Kotlin DSL for Protobuf. It should be already available here. I've just tried it in another branch and it worked for me.

api/src/main/kotlin/io/spine/protodata/event/CompilerEvent.kt

armiol

@dmdashenkov I am now requesting changes, so that you could re-request from me, once this PR is ready. Alexander is off for a while, so it will be just me.

…s produce the same output

dmdashenkov · 2023-03-24T15:59:18Z

@armiol, PTAL.

armiol

@dmdashenkov please see my comments.

armiol · 2023-03-24T16:21:55Z

.github/workflows/build-on-ubuntu.yml

          report_paths: '**/build/test-results/test/TEST-*.xml'
          require_tests: true # will fail workflow if test reports not found

-      # See: https://github.com/marketplace/actions/junit-report-action


@dmdashenkov I think this comment still stands.

armiol · 2023-03-24T16:23:21Z

api/src/main/proto/spine/protodata/events.proto

@@ -264,3 +265,15 @@ message RpcExited {

    RpcName rpc = 3;
 }
+
+// Emitted when a Protobuf dependency file is discovered.


Is this about discovering an import in our code? I am not sure what do you mean by "discovered".

armiol · 2023-03-24T16:24:06Z

api/src/main/proto/spine/protodata/events.proto

+// Normally, no code should be generated for dependencies. However, they can be used for additional
+// info when generating code for the module's own types.
+//
+message DependencyDiscovered {


I would be more precise with this name, so that Java- and Proto-level dependencies could be distinguished.

I've updated the doc somewhat. I'd like to avoid adding the Proto prefix because:

we only work with Protobuf, no Java or JS or other dependency is ever considered in ProtoData (not talking about the Gradle compiler here, only ProtoData proper);

this is one of a series of events such as TypeEntered, OptionDiscovered, etc., which do not specify the language either.

armiol · 2023-03-24T16:27:02Z

compiler/src/main/kotlin/io/spine/protodata/backend/AstComponents.kt

+}
+
+/**
+ * Assigns the field type and cardinality (`map`/`list`/`oneof_name`/`single`) to the receiver


Is this, in fact, copying the field type and the cardinality? If so, maybe we could get rid of this "assign" term which feels a bit over-used in our domain?

armiol · 2023-03-24T16:28:32Z

compiler/src/main/kotlin/io/spine/protodata/backend/AstComponents.kt

+ * Assigns the field type and cardinality (`map`/`list`/`oneof_name`/`single`) to the receiver
+ * builder.
+ *
+ * @return the receiver for method chaining.


I think, in Kotlin we still don't put periods in @return statement descriptions.

armiol · 2023-03-24T16:32:46Z

compiler/src/main/kotlin/io/spine/protodata/backend/DependencyEvents.kt

+/**
+ * Produces a sequence by walking through all the nested message definitions staring with `type`.
+ *
+ * @param type the message definition which may contain nested message definition to walk though


armiol · 2023-03-24T16:33:35Z

compiler/src/main/kotlin/io/spine/protodata/backend/DependencyEvents.kt

+ *
+ * The event reflects all the definitions from the file.
+ */
+internal fun toDependencyEvent(fileDescriptor: Descriptors.FileDescriptor) =


How about discoverDependencies(fileDescriptor: Descriptors.FileDescriptor)?

How about discoverDependencies(fd: FileDescriptor)? :)

armiol · 2023-03-24T16:34:19Z

compiler/src/main/kotlin/io/spine/protodata/backend/Documentation.kt

 import com.google.protobuf.Descriptors.MethodDescriptor
 import com.google.protobuf.Descriptors.OneofDescriptor
 import com.google.protobuf.Descriptors.ServiceDescriptor
+import io.spine.protodata.Doc
 import io.spine.string.trimWhitespace
 import io.spine.util.interlaced

 /**
 * Documentation contained in a Protobuf file.


Given its description, I don't really understand why such a generic thing is now in .backend.

I'm not a fan of the word .backend either.

This is the top-level package name for the :compiler module. :compiler serves as main implementation module for ProtoData.

Previously, Documentation, CompilerEvents, and other types where placed in the :api module in ...event package. However, it's not clear that those components belong in the :api module. Moving them to :compiler is logical since this is clearly an implementation detail (hence the internal modifier). The change of the package is a mere consequence.

I could repackage said components into ...backend.event, but I don't see much value in that right now. Would love to hear what you think.

armiol · 2023-03-24T16:36:02Z

compiler/src/main/kotlin/io/spine/protodata/backend/Documentation.kt

@@ -149,10 +154,10 @@ private constructor(private val value: List<Int>) {
         */
        fun fromMessage(descriptor: Descriptor): LocationPath {
            val numbers = mutableListOf<Int>()
-            numbers.add(FileDescriptorProto.MESSAGE_TYPE_FIELD_NUMBER)
+            numbers.add(DescriptorProtos.FileDescriptorProto.MESSAGE_TYPE_FIELD_NUMBER)


There is a lot of such changes in this PR. Don't you think it's less convenient to address the types and their members by specifying a "full" (so to speak) name, rather than stripping some common parts by having them "statically" imported?

armiol · 2023-03-24T16:37:11Z

compiler/src/main/kotlin/io/spine/protodata/backend/MessageCompilerEvents.kt

 import io.spine.protodata.OneofGroup
 import io.spine.protodata.TypeName
+import io.spine.protodata.event.FieldEntered


Given that the events apparently are still in io.spine.protodata.event package, I don't understand why MessageCompilerEvents isn't.

I have the same question regarding all the similar cases. Maybe, we should discuss the idea of re-packaging things into .backend as a whole. Because right now I don't understand the goal.

@dmdashenkov, we do need event package so that Javadocs, KDocs, and source code JARs are easier to understand. Please do have events coming under the event package.

alexander-yevsyukov · 2023-03-28T15:49:47Z

api/src/main/kotlin/io/spine/protodata/ProtoDeclaration.kt

+     * A type URL contains the type URL prefix and the qualified name of the type separated by
+     * the slash (`/`) symbol. See the docs of `google.protobuf.Any.type_url` for more info.
+     */
+    public fun typeUrl(): String? {


If the return value can really be null, please document when such a case can occur.

ProtoDeclarationName has a similar method that returns non-nullable String. How these two types are different in that regards?

Oops, that was a mistake. Fixed it.

alexander-yevsyukov · 2023-03-28T15:56:29Z

compiler/src/main/kotlin/io/spine/protodata/backend/AstComponents.kt

+    doc = documentation.forField(this@buildField)
+    [email protected] = [email protected]
+    [email protected] = declaringType
+    copyTypeAndCardinality(this@buildField)


Please extract this@buildField into a variable (e.g. fieldDescr, or thisField). This way you'd have fewer “dogs” around.

alexander-yevsyukov

Please see my additional comments.

alexander-yevsyukov · 2023-03-28T15:58:46Z

compiler/src/main/kotlin/io/spine/protodata/backend/AstComponents.kt

+    declaredIn = declaringType
+    [email protected] = [email protected]
+    orderOfDeclaration = index
+    doc = documentation.forEnumConstant(this@buildConstant)


Similarly to the comment above, extracting this@buildConstant into a val in this block could improve the readability.

compiler/src/main/kotlin/io/spine/protodata/backend/ServiceCompilerEvents.kt

dmdashenkov · 2023-03-30T12:09:18Z

@alexander-yevsyukov, @armiol, PTAL again.

alexander-yevsyukov

Please see my comments. Almost there.

alexander-yevsyukov · 2023-03-30T13:08:21Z

api/src/main/kotlin/io/spine/protodata/ProtoDeclaration.kt

+     * A type URL contains the type URL prefix and the qualified name of the type separated by
+     * the slash (`/`) symbol. See the docs of `google.protobuf.Any.type_url` for more info.
+     */
+    public fun typeUrl(): String {


Why don't we have it as another property rather than a function? Just curious, how you see it.

One reason I can think of is to have shorter names in Java code. But we aim to Kotlin anyway, and we always can have @JvmName defined, should we want to.

alexander-yevsyukov · 2023-03-30T13:43:19Z

compiler/src/test/kotlin/io/spine/protodata/backend/event/CompilerEventsSpec.kt

@@ -183,8 +201,8 @@ class CompilerEventsSpec {

        event.option.name shouldBe  "idempotency_level"
        event.option.value.unpackGuessingType() shouldBe enumValue {
-            name = IdempotencyLevel.NO_SIDE_EFFECTS.name
-            number = IdempotencyLevel.NO_SIDE_EFFECTS_VALUE
+            name = DescriptorProtos.MethodOptions.IdempotencyLevel.NO_SIDE_EFFECTS.name


Can we import MethodOptions, please?

dmdashenkov · 2023-03-30T15:21:58Z

@alexander-yevsyukov, PTAL again.

alexander-yevsyukov

LGTM

Since we've discussed this PR in a call already.

dmdashenkov added 3 commits March 16, 2023 13:31

Emit events about all types

fdc909b

Add a test for non-request files

a4f32df

Bump version

1534077

dmdashenkov self-assigned this Mar 17, 2023

Add doc

77a711d

dmdashenkov marked this pull request as ready for review March 17, 2023 11:25

dmdashenkov requested review from alexander-yevsyukov and armiol March 17, 2023 11:25

alexander-yevsyukov requested changes Mar 17, 2023

View reviewed changes

dmdashenkov added 3 commits March 17, 2023 17:08

Add definitions for a dependency-based view

c551eb9

Implement a view collecting dependency file data

643ad58

Move some implementation details to :compiler

b5963fd

armiol requested changes Mar 21, 2023

View reviewed changes

dmdashenkov added 10 commits March 22, 2023 16:26

Emit events for dependencies

f8ca578

More testing

9811ba2

Add file path in services and remove the shouldGenerate flag

a412d59

Split new model-building code into a separate file

ea7025a

Assemble separate functions into a context-class

47884bb

Add missing type details

b363bf2

Add a test for ensuring event-based and procedure-based model builder…

7a0617e

…s produce the same output

Clean up Proto definition

85c211e

Fix imports

b105840

More doc

e078f86

dmdashenkov requested a review from armiol March 24, 2023 15:59

armiol previously requested changes Mar 24, 2023

View reviewed changes

dmdashenkov added 2 commits March 27, 2023 11:39

Revert changes from config

281d9b1

More details on the new compiler event

2527f70

dmdashenkov added 2 commits March 28, 2023 18:14

Simplify proto definition construction

51ed614

Improve readability

07dea0f

alexander-yevsyukov reviewed Mar 28, 2023

View reviewed changes

alexander-yevsyukov requested changes Mar 28, 2023

View reviewed changes

alexander-yevsyukov mentioned this pull request Mar 28, 2023

same_as validation option SpineEventEngine/validation#83

Open

dmdashenkov added 9 commits March 30, 2023 10:55

Fix type nullability

c477f1b

Refactor functions to avoid this@smth-s

c8b2a60

Extract duplicates

0fa23c7

Repackage event factories

d6133ec

Fix typo

af59295

Add doc

cd18964

Fix typo

113e558

Clear up imports and use Kotest where possible

518c8b2

Remove redundant Companion specifier

8e207c1

dmdashenkov requested review from alexander-yevsyukov and armiol March 30, 2023 12:09

alexander-yevsyukov requested changes Mar 30, 2023

View reviewed changes

dmdashenkov added 2 commits March 30, 2023 17:45

Add static imports

5a11ddd

Use val instead of fun

ff3ab1e

dmdashenkov requested a review from alexander-yevsyukov March 30, 2023 15:21

alexander-yevsyukov approved these changes Mar 30, 2023

View reviewed changes

dmdashenkov enabled auto-merge March 31, 2023 08:26

dmdashenkov added this pull request to the merge queue Mar 31, 2023

Merged via the queue into master with commit 621a33a Mar 31, 2023

armiol deleted the emit-events-for-all-known-types branch June 13, 2023 16:50

		@@ -68,13 +68,15 @@ internal class EnumCompilerEvents(
		EnumEntered.newBuilder()

Emit events for all known types #127

Emit events for all known types #127

Conversation

dmdashenkov commented Mar 17, 2023 • edited Loading

codecov bot commented Mar 17, 2023 • edited Loading

Codecov Report

dmdashenkov commented Mar 17, 2023

alexander-yevsyukov left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

armiol left a comment

Choose a reason for hiding this comment

dmdashenkov commented Mar 24, 2023

armiol left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dmdashenkov Mar 27, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alexander-yevsyukov Mar 28, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alexander-yevsyukov left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dmdashenkov commented Mar 30, 2023

alexander-yevsyukov left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dmdashenkov commented Mar 30, 2023

alexander-yevsyukov left a comment

Choose a reason for hiding this comment

dmdashenkov commented Mar 17, 2023 •

edited

Loading

codecov bot commented Mar 17, 2023 •

edited

Loading

dmdashenkov Mar 27, 2023 •

edited

Loading

alexander-yevsyukov Mar 28, 2023 •

edited

Loading