Substitution of RabbitMQ for Redis Queues

[clostao]

Problem

For implementing a robust system sometimes comes with the tradeoff of implementing redundant safety mechanisms. For example the payment manager in #477 implements two simultaneous approaches for tracking a transaction:

• Listens to events emitted by the target contract: This approach is really clean but I had experience in the past missing events
• The user can submit a transaction hash to make the payment manager: This is much more effective but it's reactive

I think that this kind of designs with redundancy are generally good and enable much more robust systems though it comes with the challenge of having to avoid the processing of duplicated non-idempotent events/tasks.

Solution

Substitute the current rabbitmq for a redis instance with the objective of having deduplication mechanism already built-in in our message queue service.

Impact

Currently, we have few cases in which two services could trigger the same event just because we were aware of this limitation. But this update would enable more robust solutions for subsystems like on-chain publisher, archiving manager or the payment manager.

[jfrank-summit]

How difficult would this implementation be? What are the risks?

[clostao]

The implementation in terms of coding I think that would be relatively easy. There's this library bulmq that implements a simple interface for managing redis queues with which we could just keep the current event router interface but with a new driver.

The complexity and therefore the risk I think is more at the infra level. I yet have to research the workload that redis queues can handle. Obviously the migration isn't trivial though as I like to say not rocket science.

Anyways, I created the issue because I had this in mind but I don't think is a priority right now but could be sometime