Timed out waiting for a reply to message ID

456 views
Skip to first unread message

Hamza Mahmoud Abdallah Alqtaishat

unread,
Jan 26, 2021, 5:00:05 PM1/26/21
to rabbitmq-users
hi i have a question about oslo_messaging and rabbitmq when ever the rabbitmq restarted we get `Timed out waiting for a reply to message ID` and we need to restart all services in openstack (sometimes u need to restart rabbitmq more than once for it to work) any idea what could be the cause on this

Hamza Mahmoud Abdallah Alqtaishat

unread,
Jan 26, 2021, 5:08:38 PM1/26/21
to rabbitmq-users
Maybe i should explain more 

I have an openstack system which use RabbitMQ as RPC,  and use oslo_messaging to communicate with it
for some reason sometimes when RabbitMQ is restarted the error Timed out waiting for a reply to message ID
is raised in some openstack services and they stop working

is that a known behavior for the RRabbitMQ
any info about this will be appreciated 
thanks

Wesley Peng

unread,
Jan 26, 2021, 7:46:48 PM1/26/21
to rabbitm...@googlegroups.com
For this issue, you'd better ask it on openstack mailing list.
If you were sure this is a rabbitmq issue, please provide the full info such as versions, error log, operations, running environment and architecture etc.

Thanks.
--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.

Hamza Mahmoud Abdallah Alqtaishat

unread,
Feb 1, 2021, 1:58:19 PM2/1/21
to rabbitmq-users
I asked in the openstack side and they think 
I guess it could be due to multiple cause, on of them could be a rabbitmq split brain due to a network issue, I also heard about race conditions on rabbit, but I'm not a rabbitmq expert either

rabbitmq version: 3.8.5 it also happen on older versions
error log: does not show anything out of the ordinary ( i will attach it later )
operations: send a message and wait for reply
environment: redhat 7.9
architecture: the package shows noarch but its 64 machine

thanks

Hamza Mahmoud Abdallah Alqtaishat

unread,
Feb 1, 2021, 2:04:03 PM2/1/21
to rabbitmq-users
and there a bug in openstack with the same issue

Hamza Mahmoud Abdallah Alqtaishat

unread,
Feb 1, 2021, 2:47:44 PM2/1/21
to rabbitmq-users
i was able to find this error in the log when the issue happened
[error] <0.30671.1> Channel error on connection <0.14073.1> (***:59260 -> ***:5671, vhost: '***', user: '***'), channel 1: operation queue.declare caused a channel exception not_found: queue 'reply_567e2f8d94ef476a9a73c32de95f7df1' in vhost '***' process is stopped by supervisor

which happened when the rabbitmq was restarted

Wesley Peng

unread,
Feb 1, 2021, 7:27:15 PM2/1/21
to RabbitMQ Users
Both network and filesystem issues can cause the problem.
For example, you were running Ceph which needs good network connection, any network issue (either physical connection or virtual connection such as SDN) will cause RMQ to have problems.

Thanks.
Reply all
Reply to author
Forward
0 new messages