Rabbit mq flow state

[springusr]

I have read lot of blogs, comments on forums on this topic. My question is as application developer , what I am supposed to do to recover from it. Right now I have connection in flow state, application is hung. How can I detect this and take action. Also while I am having this not all applications impacted with flow state. And my load is also not high at all.

[springusr]

I am using spring amqp abstraction so org.springframework.amqp.rabbit.connection.CachingConnectionFactory and org.springframework.amqp.rabbit.core.RabbitAdmin and RabbitTemplate.

[Arnaud Cogoluègnes]

Are you still experiencing the same issue as in [1] or is it something else?

For [1], we filled in an issue and the fix should ship in 4.1.0 of the Java Client (in the next upcoming weeks probably).

If this is something else, we need more information (e.g. version of the server, server logs, kind of loads in the application).

Have you read [2] and [3]? They should help you diagnose flow control issues.

[1] https://groups.google.com/forum/#!searchin/rabbitmq-users/Rabbit$20mq$20channel$20hangs%7Csort:relevance/rabbitmq-users/Xc1TH-518qY/FDbbluQXBgAJ

[2] https://www.rabbitmq.com/flow-control.html

[3] http://www.rabbitmq.com/blog/2014/04/14/finding-bottlenecks-with-rabbitmq-3-3/

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[springusr]

For [1] , yes that channel timeout enhancement will help. Right locslnow we made few changes and now we are running into flow state - where server is healthy, we are not producing too fast messages etc, right now no clue on reasons behind flow state. Ideally when I produced flow state on my local it was flowing messages though but in test env once flow state it just hangs. Threads which are communicating to rabbit hangs which actually leads [1]. I will keep posted on what we learn.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

[Arnaud Cogoluègnes]

I uploaded a snapshot of the Java Client [1] with the fix for [2]. Can you give it a try? Use the appropriate method on the ConnectionFactory [3] to set the timeout.

[1] https://oss.sonatype.org/content/repositories/snapshots/com/rabbitmq/amqp-client/4.1.0-SNAPSHOT/

[2] https://github.com/rabbitmq/rabbitmq-java-client/issues/219

[3] https://github.com/rabbitmq/rabbitmq-java-client/blob/master/src/main/java/com/rabbitmq/client/ConnectionFactory.java#L1100

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

[springusr]

Thanks I will give a try..just so you know we will need way to set this in spring abstraction as well like in caching conn factory, right ? Though for this I will try to use library direct or I will just add this new version and can live with 10 min default timeout.

[Michael Klishin]

I believe spring-amqp allows you to access RabbitMQ's ConnectionFactory.

That's not to say spring-amqp won't have those respective settings but it's going to happen

in Spring AMQP 2.x (which can require RabbitMQ Java client 4.x).

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Arnaud Cogoluègnes]

There are ways to provide the RabbitMQ native ConnectionFactory to Spring AMQP CachingConnectionFactory, take a look at the Spring AMQP documentation [1] (link is for XML, but Java configuration equivalent should be straightforward).

You should provide a shorter timeout (e.g. 10 seconds), at least for easier testing. FYI, automatic recovery is enabled by default as of RabbitMQ Java Client 4.0. Take this into account when testing/observing your application. 

[1] http://docs.spring.io/spring-amqp/docs/1.6.5.RELEASE/reference/htmlsingle/#connection-factory

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

[springusr]

General question - is it okay to see channels in running mode even if there is no connection (Application stopped). I think those hanging channels locking application.

[Michael Klishin]

It depends where you see them. Management UI records cannot possibly lock anything.

I doubt that channel processes in the server with a closed connection can affect your application

in general: all channel objects in the client should be garbage collected with their closed connection.

If that's not the case then most likely something is holding a reference to the closed connection object.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

[springusr]

Ya it was on mgmt console, first I saw applicattion hung as mentioned above on getting queue properties call (From stack trace) , so I stopped application, so that closed connection as it should but even then those channels in running mode.

[Michael Klishin]

You can reset your stats DB as documented in http://www.rabbitmq.com/management.html.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.