debug rabbit mq for issues

[Sagar]

Good Morning ..

We are using rabbit mq 3.2.4   .. we see issues where the client/consumers get throttled and does  not consume at times ... the client side thread dumps doesnt show anything alarming ...

Not sure how to debug from rabbitmq end  to check the state / resources etc  ... also any tools we can use to monitor the rabbit mq .. we use rabbitmq console  but doesnt have much info on what is happening internally wrt threads /speed /resources etc  

Appreciate all the help  

[Michael Klishin]

Upgrade. There is I/O and consumer utilisation stats available in the more recent
management plugin versions, besides other improvements :

https://www.rabbitmq.com/blog/2014/04/14/finding-bottlenecks-with-rabbitmq-3-3/
http://www.rabbitmq.com/changelog.html
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[Vijaya Sagar]

Thanks Michael ...

Below is a scenario we are seeing in production ...

We see a backlog on queues and the consumer( clients )  boxes has lot of available free resources to process more messages  but there is some disconnect on why consumers are not able to consume more ...

Is there anything in rabbit mq which is holding and not delivering fast ... We have plenty of memory and still see paging happening .. not sure why ..

also increasing the consumers  is not helping with the delivery rate ....  just trying to understand how we should go about to debug ..

--

Thanks,
Sagar
4047548920

[Michael Klishin]

 On 21 August 2015 at 18:49:22, Vijaya Sagar (sagar....@gmail.com) wrote:
> We see a backlog on queues and the consumer( clients ) boxes has
> lot of available free resources to process more messages but
> there is some disconnect on why consumers are not able to consume
> more ...
> Is there anything in rabbit mq which is holding and not delivering
> fast ... We have plenty of memory and still see paging happening
> .. not sure why ..
>
> also increasing the consumers is not helping with the delivery
> rate .... just trying to understand how we should go about to debug

There were multiple improvements in prioritising delivery to consumers
over other things, as well as fixes to throughput regressions and statistics
that would help investigating exactly this kind of problems:

25888, 25882, and a bunch more in 3.3.0:
http://www.rabbitmq.com/release-notes/README-3.3.0.txt

26389 in 3.4.0:
http://www.rabbitmq.com/release-notes/README-3.4.0.txt

and so on.

3.2.4 is 16 releases behind, soon to be 17:

[Vijaya Sagar]

Thanks Michael ..

Does the delivery rate of the messages depends on the server ?? or is it the bottleneck at consumer end (client)..

We see quite a messages in the queue and and see paging happening for the queue as well ...  but for some reason the client (consumers ) is not able to consume the messages at the rate we are publishing   ... 

Is the paging causing the issue at server end ... From client end we still see lot of capacity and are trying to figure out where the bottleneck is ..

-Sagar

--

Thanks,
Sagar
4047548920

[Michael Klishin]

The issues mentioned are in the server. Clients and applications can be slow as well: what client and version do you use?

MK

[Vijaya Sagar]

rabbit mq 3.2.4 is the server  ...

we use simple java client  with spring  rabbit 1.2.0 , spring amqp 1.2.0  , amqp-client-2.8.1 

--

Thanks,
Sagar
4047548920

[Michael Klishin]

On 24 August 2015 at 21:23:32, Vijaya Sagar (sagar....@gmail.com) wrote:
> we use simple java client with spring rabbit 1.2.0 , spring amqp
> 1.2.0 , amqp-client-2.8.1

There were no significant concurrency improvements in consumer method dispatch
since Java client 2.7.0. 

[Vijaya Sagar]

Good Morning ..

Wile we are working on the upgrade of rabbit MQ  .. Can u suggest us tools on how to debug rabbit mq 3.2.4  on why the delivery rate is slow ..

We noticed this issues again this weekend and looks like once paging  starts things go worst   ..

We have the messages persistent and do ack ... we have 2 nodes  in the cluster .. do u recommend  any changes ...

 From below  it looks like one node memory is being utilized more ....let us know if u need any.. Appreciate all the help ..

 

Also from the below image ..notice only one Node  and we can see the paging stuff for this queue ... other queues are doing good where we did not notice any paging  ..

--

Thanks,
Sagar
4047548920