Is a rewrite really desirable?

[danielmarbach]

Please don't get me wrong, this question bugs my mind and I wanted to hear your opinion. Now and then on issues or PRs it is talked about the new client and that the work will start soonish. I was wondering if going for a new client is really necessary or desirable. Yes we know that this client has "legacy" and a long history but isn't that really so bad to justify to rewrite? Just by looking at it from an async perspective like I've shown in a PR it would be possible to evolve the existing client step by step into an async future. This would also allow to gradually evolve the clients of this client towards the async future.

I have not enough context nor history to be a good sounding board here. What I can say based on my experience with Software Development is that whenever we got the requirement to rewrite an existing system from scratch with the one and only requirement of "it shall do the same but different" the efforts were huge and we were usually significantly lagging behind with feature parity with the "old software piece".

Consider this more as me thinking out loud and interested in hearing your opinions @michaelklishin @kjnilsson

[michaelklishin]

This client was designed ~ 10 years ago for .NET 1.1 (which it still supported as recently as early 2013). By 2013 we moved it to require 3.5, then 4.0, and now all the way to 4.6 and .NET Core but some API and implementation details are not too different from the state of the art in 2007.

@danielmarbach I agree with your impression but there are two things to consider:

• I don't know if I'd like the new client to do exactly the same thing
• RabbitMQ clients have been fairly stable for years now, with enough test suites available and common problems/solutions discovered that developing a new client is mostly a matter of coming up with an API that makes sense for language X and integrating the above knowledge
• I am personally not looking forward to supporting all .NET "flavours" going forward (and no, I don't see "regular" .NET disappearing any time soon). It is frustrating and time consuming.

Last client we developed from scratch was the Objective-C/Swift one. We had someone with much less experience with RabbitMQ and its clients than @kjnilsson and I have; we have to follow development of an even less mature language/tooling (Swift); and we had to debug native code crashes (Objective-C, C), something that I rarely see with .NET runtimes. Even then it took us maybe 5 months from start to finish. The client has a few known issues that every client had at some point
(how per-operation timeouts are integrated, for example) but beyond that, it requires next to no maintenance.

5 months is not a small amount of time. Let's take a look how long it took us to support Windows RT? I think at least a month. .NET Core last year? ~ 3 weeks at first , then several more over the course of 2016 to keep up, and I know already that another couple of weeks are coming because newest .NET Core tooling seems to break some things in ways I cannot explain.

How long did it take for Particular Software folks to integrate the new async dyspatcher? IIRC a few of weeks as well.

What am I getting at is that we have probably spent 1-2 person/months if not more maintaining this client in the last ~ 18 months, with only one major improvement, and we are already working on 5.1.

If we are to start making more significant changes to this client, we'd be at 10.0 or 15.0 in a year and create a hell of an upgrade path for everyone, including NServiceBus. Or we can spend more time on a new API that won't require 5 major version bumps in 1 year to get where we want it to be, and free ourselves from supporting classic .NET, 10 years of quirks accumulated in this codebase and Semantic Versioning (which is a liability for a maintainer as well as an asset, let's not pretend it's all 🦄🌈).

This client will then be limited to bug fixes and other changes we cannot realistically avoid (security patches, support for any changes coming in RabbitMQ 4.0 w.r.t. client semantics, etc).

Back in 2010 Ruby had a popular client, amqp gem, which had too many fundamental problems with its underlying I/O and concurrency library, which made it too easy to misuse. It also had Bunny, another client with a weird API that hid channels entirely, tried
to make messaging look "a bit like Rack" (an HTTP server interface) and had no concurrency anywhere to be found.
So I spent maybe 6 months rewriting Bunny from scratch, integrating everything we've learned from amqp gem and in the end well over 90% of users could migrate with next to no changes. Some improvements were even incorporated back to amqp gem (as a shared library, for example). Then someone ported Bunny to JRuby, creating March Hare, and some users can change a few lines of code to move from one to the other.
There was no way to evolve amqp gem into the Bunny and March Hare we have today, and while I started from scratch,
Bunny 0.7.x (the original one) and 0.8.0+ had no important API differences.

So, having contributed to, developed and "rewritten" multiple clients I can tell you that the domain is largely static, there's plenty of other clients to learn from (including 10 years of this one), many things (e.g. test cases) that can be more or less stolen from other places, and plenty of reasons for us to hate working on this client.

Now you see why I am personally all for developing a new, .NET Core only client.

[michaelklishin]

That said, I'm happy that major users of this client speak up and suggest other options :)

[adamralph]

@michaelklishin when you say "a new, .NET Core only client", I assume you mean "a new, .NET Standard only client"?

[michaelklishin]

@adamralph well, yes, but I think the end game for MSFT is to have as few .NET flavours around as possible, and maybe even just one. We definitely would like to support .NET Framework, Mono and such in the new client, just hope it can be a lot taxing than it is today.

[YulerB]

If your planning AMQP 1.0, id rewrite.

If you want to go pure async, and are willing to take on dependencies on some TPL packages, I'd rewrite.

Otherwise, no real reason to upgrade; no compelling argument to upgrade mentioned above.

But decision is ultimately yours.

[michaelklishin]

We are not planning to target a different protocol but we do want to have the freedom to pursue any API change we want, including targeting async/await and the TPL and renaming of major API elements. We also would like to drop most of our10 years worth of baggage along the way.

…

On Wed, Oct 18, 2017 at 10:11 PM, Brian Yule ***@***.***> wrote: If your planning AMQP 1.0, id rewrite. If you want to go pure async, and are willing to take on dependencies on some TPL packages, I'd rewrite. Otherwise, no real reason to upgrade; no compelling argument to upgrade mentioned above. But decision is ultimately yours. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#356 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAEQhvBkZV76-m6ADL8WZVHzFokTo_tks5stk1VgaJpZM4PfORP> .

-- MK Staff Software Engineer, Pivotal/RabbitMQ

[Leon99]

Any update on the state of the async support? The current client doesn't fit the idea of having "async all the way", which is how all modern systems are built.

[michaelklishin]

@Leon99 there is no update and you'd be surprised to know what % of users opposed the idea of a rewrite. Publishing in the protocol has always been asynchronous (there is simply no response for the basic.publish frame) and an an asynchronous consumer operation dispatch has been available for a while. Whether that's "async all the way", I don't know, but it means both hot paths are in decent shape.

I am locking this as we don't need more iterations of the same question. When we have the cycles and decide to do something (e.g. develop a new client), this issue will be updated.

[lukebakken]

For people watching this issue, please look at #694 and #687, two recent community contributions that enhance this library.