rabbitmq-event-exchange missing events

[Jianping Roth]

In our application, we add and delete queues once in a while; each queue is for one customer; this happens as new customers come and old customers go, 

We have about 1K queues. In the past, we would get all the queues via RabbitMQ api periodically, but we found getting the queues take a heavy toll on the RabbitMQ cluster's performance and so we switched to using rabbitmq-event-exchange to detect new queues and add consumers to new queues automatically. 

However, we found once in a while we would not receive new queue events and thus we would not have consumers for the requests from new customers.

We use RabbitMQ 3.7.4 and Erlang 20.3.

My questions are: 

is rabbitmq-event-exchange reliable?

if not, how can be compensated for it using RabbitMQ's other features.

Here are some context:

- We have a cluster of 3 nodes

- Each cluster has about 15 virtual hosts

- Each host has about 1K queues

[Luke Bakken]

Hello,

The event exchange should be reliable. If there's a way we could reproduce this issue, we could fix it quickly.

Is there any pattern to the missing new queue events? Are you certain this isn't an issue with the application consuming these events?

Thanks -

Luke

[Jianping Roth]

Hi Luke, 

Thank you for your prompt response. 

I am fairly certain that our application, with 3 nodes consuming from the same RabbitMQ cluster, didn't run into any exception in this case. 

This happened rarely and so we haven't found a pattern of when and how it happened. The RabbitMQ cluster and the application load were fairly normal.

I will provide you with more detailed information when they become available. I am trying to get the RabbitMQ cluster logs.

Regards,

Jianping Roth

[Jianping Roth]

I looked at the RabbitMQ server logs and I didn't see any exceptions, nor did I see anything of interest. For example, when new queues were added, I didn't see any log message indicating so. I didn't see any activities for event.exchange. I wonder how I can turn on logging for the event.exchange? 

thank you!

[Jianping Roth]

Hi Luke,

We found there is bug that has been fixed. We will try to upgrade our RabbitMQ server to get the fix.

Event Exchange Plugin
Bug Fixes
Events could fail to be emitted in multi-node clusters.

GitHub issue: rabbitmq/rabbitmq-event-exchange#41

[Luke Bakken]

That bug does not affect RabbitMQ 3.7.4 thus it is not affecting your environment.

https://groups.google.com/forum/#!topic/rabbitmq-users/ufAHJzDivwY/discussion

https://github.com/rabbitmq/rabbitmq-event-exchange/issues/41

The event exchange doesn't support logging at this time. Since this issue hasn't been reported elsewhere the most likely explanation is an error in a consumer, or something specific to your environment.

Thanks,

Luke

On Tuesday, October 15, 2019 at 5:02:50 PM UTC-7, Jianping Roth wrote:

Hi Luke,

[Jianping Roth]

Thank you for the information. We will keep an eye out for this issue and see if it will happen again and keep you posted.  

--
Jianping Roth
Software Developer
xMatters: Digital Service Availability Platform
jianpi...@xmatters.com | 250-380-0304 ext 427
----------------
Free Forever | Meet Flow Designer | See it in Action 

---

From: rabbitm...@googlegroups.com <rabbitm...@googlegroups.com> on behalf of Luke Bakken <lba...@pivotal.io>
Sent: Wednesday, October 16, 2019 7:43 AM
To: rabbitmq-users <rabbitm...@googlegroups.com>
Subject: [rabbitmq-users] Re: rabbitmq-event-exchange missing events

 

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/4d3bf879-67a5-4c2f-a74b-7bee6e332e70%40googlegroups.com.