Invalid request number, must be greater than 0 when sending message to outbound channel

[timterle]

We use Artemis and have a Quarkus application that sends lots of messages to Artemis. Sometimes we see the stacktrace below. When this happens the application silently stops sending messages on the channel. There are no more errors, but no messages are sent. We have to restart the application when this happens.

java.lang.IllegalArgumentException: Invalid request number, must be greater than 0
	at io.smallrye.mutiny.helpers.Subscriptions.getInvalidRequestException(Subscriptions.java:24)
	at io.smallrye.mutiny.operators.multi.builders.BaseMultiEmitter.request(BaseMultiEmitter.java:133)
	at io.smallrye.reactive.messaging.amqp.AmqpCreditBasedSender.setCreditsAndRequest(AmqpCreditBasedSender.java:163)
	at io.smallrye.reactive.messaging.amqp.AmqpCreditBasedSender.lambda$onNoMoreCredit$7(AmqpCreditBasedSender.java:218)
	at io.vertx.mutiny.core.Context.lambda$runOnContext$1(Context.java:136)
	at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:270)
	at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:252)
	at io.vertx.core.impl.ContextInternal.lambda$runOnContext$0(ContextInternal.java:50)
	at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:566)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:1583)

I commented on this issue in quarkus:
quarkusio/quarkus#48624

The code in AmqpCreditBasedSender.java assumes credit is always positive, but according to Proton's documentation it looks like it can be negative:
https://qpid.apache.org/releases/qpid-proton-j-0.34.1/api/org/apache/qpid/proton/engine/Link.html#getCredit()

I do have a patch of smallrye-reactive-messaging-amqp that we have been testing and so far we haven't been able to reproduce this issue.

[ozangunalp]

@timterle Thanks for the detailed description.

Is it possible to reproduce the error in a test? Maybe configuring a queue with a small enough max-size and sending a lot of messages can trigger this?

We would be very interested in the patch if you'd like to contribute.

[timterle]

I made a pull request:
#3197

This issue has been pretty easy to reproduce in OpenShift, but when running Artemis in docker on the same machine it's not been reproducible.

[timterle]

@ozangunalp Did you check the pull request? Do you think it looks reasonable?

[ozangunalp]

@timterle sorry I thought I responded. The change looks good to me. Do you think we can create a test case for it?

[timterle]

@ozangunalp Do you mean a unit test? I guess it should be possible to trigger the bug somehow. I'm not familiar with the code and the tests, but I'll try and take a look.

[timterle]

Fixed here:
#3213

[timterle]

@ozangunalp When roughly will there be a release with this fix?

[ozangunalp]

This will be part of the 4.31.0 release. It'll be released very soon, today or Monday.