[New Tool] Spring Cloud Stream plugin for AsyncAPI

[fmvilas]

We currently have a Spring Cloud Stream code generator, which takes an AsyncAPI file and generates a project using Spring Cloud Stream. This is fine but it's not all the good it could be. If you're familiar with our Glee framework, we're advocating for using AsyncAPI at runtime too, instead of using it as a mere asset when starting your project.

We got both ways of working covered, however, there are a few limitations with our current tools:

1. Glee is based in JS/TS. It's great but not everyone is comfortable with these languages. We added support for Java at Glee but that resulted in a much complex codebase and ditched the idea. See related pull requests.
2. Many people don't start a fresh new project but they already have an existing one with lots of code they obviously don't want to refactor in a another language and framework 😅
3. Incorporating the code generation workflow in an existing project is nearly impossible because requires changing the way all developers in the team work.

Proposal

So my proposal is that we create a plugin for Spring Cloud Stream that will be able to intercept messages before they arrive to the handler and before they are published to the broker. In other words, intercept messages when they're received and sent and validate them according to the existing AsyncAPI file in the project. It's not as powerful as Glee but would make things much better and people will instantly see the value.

Benefits

1. Teams only have to add this new plugin. Developer workflow doesn't change. They can still work on the code first or on the design first. Up to them.
2. Messages get validated "for free". They only have to define them in the AsyncAPI file. This means their code is protected against malformed messages and they will never send malformed messages themselves.
3. Automatically generating docs in Markdown, HTML, or other formats can also be offered as part of this plugin easily. Even starting a new HTTP server/endpoint where the application will serve the docs.

Challenges

1. I'm not 100% sure this is completely possible in Spring Cloud Stream. I see they have plugins but have no idea what can be done at plugin level. In any case, if that's not possible, that shouldn't stop us from chatting with the Spring Cloud Stream maintainers and see if that's something they can enable. Need help from Java engineers here.
2. A common complain I've gotten is that developers have to define the data shape in two places and keep them in sync: one in their code (e.g., a Java class) and one in the AsyncAPI file (using JSON Schema or Avro most probably). So anything that can be done to keep the two in sync would be really great.
3. We have to figure out what to do when the message is not valid. We should probably offer a few choices: log the error, send the error to an external service by calling an HTTP endpoint, integrating with New Relic and Datadog, etc.

---

Would be great to hear from you folks. Especially, on how to solve the challenges :)

[abelgomez]

Hi @fmvilas , I know our eclipse-based AsyncAPI Toolkit does not explicitly focus on Spring Cloud Stream (but standard Java for MQTT), and it is not implemented using the official AsyncAPI template engine either, but I think that our proposal directly targets challenge #2 (correct me if I'm wrong).

As you may remember, we're following a generative approach, and what we do is generate the Java classes that support data serialization and parsing directly from the AsyncAPI specification. Thus, developers only need to focus on "client code" (i.e., the logic of their apps), and delegate all the boilerplate code (especially if JSON messages are exchanged) to the code generator.

Coincidentally, today we have released a new version of the tool, and we have done extensive changes. Now the generated code is much more generic and complete, and what's more, it is maven compatible (so hopefully, generated code can be more easily integrated with 3rd party code now). The tool also has extended it's modeling support, but maybe it is not so interesting in this case.

I'm telling this, just in case you want to give it a look. Maybe the new release can serve as an inspiration for solving the challenges you face 😊

[fmvilas]

I'm absolutely open to suggestions here, Abel. In any case, we need to target common frameworks because that's what people are using and we don't want them to change their code but just "plug" an AsyncAPI plugin and move on. About challenge #2, we can leverage AsyncAPI Toolkit to generate AsyncAPI from existing code (initial step) and then leverage Modelina to keep generating code from AsyncAPI changes.

Just an idea though. We'll definitely have to explore more and see what fits and what doesn't. You may want to take the lead on this tool regarding your extensive experience building AsyncAPI Toolkit :)