publisher threads BLOCKED waiting for a lock at com.rabbitmq.client.impl.SocketFrameHandler.writeFra

[Sagar]

We have the below spring configuration  and use amqp client 3.5.5  and spring 1.5.0  ...

<rabbit:connection-factory id="connectionFactory" connection-factory="cf" />

<rabbit:admin connection-factory="connectionFactory"/>

<rabbit:template id="amqpTemplate" connection-factory="connectionFactory"/>

Irrespective of the numbers of threads used to publish we see all the threads except 1 in the below ..Not sure why ...Please advice...Thanks and appreciate all you help ..

"pool-4-thread-100" prio=6 tid=0x0000000008163800 nid=0xa53c waiting for monitor entry [0x000000000fd2e000]

   java.lang.Thread.State: BLOCKED (on object monitor)

at com.rabbitmq.client.impl.SocketFrameHandler.writeFrame(SocketFrameHandler.java:145)

- waiting to lock <0x0000000795e3ab30> (a java.io.DataOutputStream)

at com.rabbitmq.client.impl.AMQConnection.writeFrame(AMQConnection.java:508)

at com.rabbitmq.client.impl.AMQCommand.transmit(AMQCommand.java:102)

- locked <0x00000007e3c0bb50> (a com.rabbitmq.client.impl.CommandAssembler)

at com.rabbitmq.client.impl.AMQChannel.quiescingTransmit(AMQChannel.java:334)

- locked <0x00000007bbe8d650> (a java.lang.Object)

at com.rabbitmq.client.impl.AMQChannel.quiescingTransmit(AMQChannel.java:316)

- locked <0x00000007bbe8d650> (a java.lang.Object)

at com.rabbitmq.client.impl.AMQChannel.quiescingRpc(AMQChannel.java:251)

- locked <0x00000007bbe8d650> (a java.lang.Object)

at com.rabbitmq.client.impl.ChannelN.close(ChannelN.java:574)

- locked <0x00000007bbe8d650> (a java.lang.Object)

at com.rabbitmq.client.impl.ChannelN.close(ChannelN.java:509)

at com.rabbitmq.client.impl.ChannelN.close(ChannelN.java:503)

at org.springframework.amqp.rabbit.connection.CachingConnectionFactory$CachedChannelInvocationHandler.physicalClose(CachingConnectionFactory.java:807)

at org.springframework.amqp.rabbit.connection.CachingConnectionFactory$CachedChannelInvocationHandler.invoke(CachingConnectionFactory.java:677)

at com.sun.proxy.$Proxy17.close(Unknown Source)

at org.springframework.amqp.rabbit.connection.RabbitUtils.closeChannel(RabbitUtils.java:80)

at org.springframework.amqp.rabbit.connection.ConnectionFactoryUtils.releaseResources(ConnectionFactoryUtils.java:157)

at org.springframework.amqp.rabbit.core.RabbitTemplate.doExecute(RabbitTemplate.java:1307)

at org.springframework.amqp.rabbit.core.RabbitTemplate.execute(RabbitTemplate.java:1271)

at org.springframework.amqp.rabbit.core.RabbitTemplate.send(RabbitTemplate.java:619)

at org.springframework.amqp.rabbit.core.RabbitTemplate.send(RabbitTemplate.java:613)

at com.att.dlife.dlcore.mbus.client.publisher.AmqpMessageSenderImpl.send(AmqpMessageSenderImpl.java:98)

at com.att.dlife.dlcore.mbus.client.demo.MessageSendTask.run(MessageSendTask.java:35)

at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)

at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:

- <0x00000007962f6bd8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

[Michael Klishin]

 On 3 Oct 2015 at 21:59:54, Sagar (sagar....@gmail.com) wrote:
> "pool-4-thread-100" prio=6 tid=0x0000000008163800 nid=0xa53c
> waiting for monitor entry [0x000000000fd2e000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at com.rabbitmq.client.impl.SocketFrameHandler.writeFrame(SocketFrameHandler.java:145)
> - waiting to lock <0x0000000795e3ab30> (a java.io.DataOutputStream)

Please see http://www.rabbitmq.com/alarms.html. Your connection is blocked either due to
internal flow control (temporary) or a resource-driven one (e.g. RAM watermark).

Please *always* inspect RabbitMQ logs when investigating things like this. The regular log
will contain entries for alarms, for example, so inspecting the log often alleviates the need
to post to the list because it’s more or less obvious what’s going on.
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[Jaskey Lam]

Hi Michael, 

I have the same problem but I do have enough memory and disk space from the as the management plugins shows, what will be the problem??

Would you please also answer in the SO post :https://stackoverflow.com/questions/48728703/threads-are-all-blocked-when-publish-with-multi-threads-and-chanels-rabbitmq

在 2015年10月4日星期日 UTC+8上午3:04:23，Michael Klishin写道：

[Luke Bakken]

Hi Jaskey,

The RabbitMQ core team prefers to answer questions and discuss issues on this mailing list.

In any case, the answer you received is correct and you are observing the expected behavior in your application (https://stackoverflow.com/a/48729039). In order to preserve correct order in the data being sent to and received from RabbitMQ there has to be synchronization somewhere.

In your case, why are you opening so many channels per connection? If you read that as a best practice somewhere on rabbitmq.com, I would like to make sure that the documentation gives good advice.

Thanks -

Luke

[Arnaud Cogoluègnes]

Frame writing must be synchronized at the socket level, or frames from different channels would be interleaved and the broker wouldn't be able to parse those frames correctly. This is no different from any network clients.

This is a lot of open channels for one connection. On SO, you mention 4000 messages / seconds, is it for the whole connection or per channel? (A single producer can publish 1000s of messages / second, on a single channel). What is the average size of a message?

Once you observe that channels hang on frame writing, what do you experience next? Do they unblock little by little? Or does the connection fail and automatic connection recovery kicks in?

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jaskey Lam]

Hi 

在 2018年2月12日星期一 UTC+8下午10:32:02，Arnaud Cogoluègnes写道：

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

[Jaskey Lam]

Hi Luke,

Thank you very much for your help.

For the question of

1."why are you opening so many channels per connection":

that is because rabbitmq's Chanel is not thread safe, and I have at most 200 threads that are publishing messages with high tps, so for now I have to create channel every time I pubilsh and close, which means when broker is flow control, I have have 200 open chanels.

2.you mention 4000 messages / seconds, is it for the whole connection or per channel

That is for my whole broker cluster, actually, I have multiple connections for one publisher. That is the existing code design, which create a single connection to a single queue so If I have 20 queues to publish, I have 20 connections. PS:  I have tens of publish clients, and 9 brokers.

And every time I pubish, I create a channel from one of this connections. Indeed, I guess this is a bad design, is it a single connection for all publishers(in one single process) is good enough and better than multiple connections? 

3. What is the average size of a message?

It should be small since the body is a small JSON with two keys like: {messageId:"32-CHAR", clientId:"32-CHAR"}.

4 One you observe that channels hang on frame writing, what do you experience next? Do they unblock little by little?

The answer is frustrating, what I can see is that my publisher client is always blocking which means no threads are able to responding the requests, and the thread which is blocked are always blocked after minutes and at last I have to restart the client. And sometimes, indeed the thread will become safe. So we guess the broker have some bottle neck that client are no longer able to flush and always not able to release the monitor lock(exit synchronize)

在 2018年2月12日星期一 UTC+8上午12:55:40，Luke Bakken写道：

[Michael Klishin]

Are you sure that having a concurrency rate of 200 or similar is actually productive? If your machine doesn't have at least 64 cores

and a lot of network bandwidth, I kind of doubt it.

Opening a channel for every publish operations is *incredibly* inefficient and must be avoided. It's a synchronous protocol

operation, which means the client has to wait for a response and there is a network roundtrip.

--

You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Michael Klishin]

What you are looking for is:

 * A concurrency level that makes sense for the number of CPU cores both in the client and on the server

 * Channel pooling (a solved problem in several projects, e.g. Spring AMQP)

 * Lots of server and infra metrics (network, CPU context switching, channel state) and how they change as you change the concurrency level and other variables

 * Client-level introspection with VisualVM or similar

To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Jaskey Lam]

Hi Michael 

My machine is productive with more than 64 cores.

I have two applications using two different ways, 

1.one is using thread local to hold channels, since I have 200 threads in a thread pool so I have 200 channels that can be reused,  the requests using this implementation has tps more than 5000

2.Another simple implementation is using creating channel and destory it every time, these tps is around 2000. Also, this way I use the select confrim way to ensure that the messages will not be lost as much as possible, which is why this tps is slower than application 1

But both of these two applications send messages to the the broker cluster. I am finding a best practice to refactor the code for using channels.

My questions are :

1.Do you mean the #1 is a better one. but I don't think thread local is a graceful way.

2.Does the way using #2 gives burden to broker which influence application1, since I can find flow control for the connection of application1.

3 Also, I was shocked that I can't find a good example from the doc or blog for publish in multithread in java, would you please give a link for this to show a best practice.

Thank you 

Best Regards

Jaskey

在 2018年2月22日星期四 UTC+8下午10:44:31，Michael Klishin写道：

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Michael Klishin]

With more than 64 cores you can get contention in areas you don’t expect,

both in the client and the server. A single connection can be such bottleneck,

in particular on the server, since it has a realistically limited throughput that 200 channels using 64 cores can surpass easily.

Connections, channels and queues are three main units of concurrency in RabbitMQ.

rabbitmq-top and management UI metrics such as context switching can reveal the node processes that take most the runtime scheduler time.

If the server has more than 64 cores (I sure hope it does with that kind of clients), VM scheduler binding strategy becomes very important. It has been discussed on this list multiple times in the past. The optimal value is workload-specific and the default cannot be optimal by definition, in particular with more than 64 cores.

There is a concurrency considerations section in the Java client doc guide and channel pooling in Spring AMQP. There are no “better practices”.

[Jaskey Lam]

Hi Michael,

We don't use Spring AMQP, but pure java api. 

In a word, what's your advice for the massive publishing for multi threading. we have 5000-10000/s in total for the cluster. All messages must be persistent.

 Here are my questions.

1. How do we deal with connections. Should I create a single connection to 10 queues(we have XX_0 to XX10), or we create 10 connections for this. Now we use the latter one, but we have now around 1000 connections in the cluster , is this a burden to rabbitmq broker?

2. How should we deal with channels, as I said , we create a channel in the thread local, and since we have around 1000 connections, now, we have over 100000 channels, is this really bad design? As you  pointed out, should we use channel pooling and how?(we don't use spirng amqp)

3. We don't use any special exchange, which means we only send to the default exchange and route to the queue directly, it is better to have multiple exchanges?

 

4. We are always suffering flow control, what important metrics  should we notice?

Thank you very much!

Best Regards

Jaskey

在 2018年2月23日星期五 UTC+8下午12:14:13，Michael Klishin写道：

[Michael Klishin]

On Mon, Mar 5, 2018 at 11:29 AM, Jaskey Lam <linjun...@gmail.com> wrote:

Hi Michael,

We don't use Spring AMQP, but pure java api. 

In a word, what's your advice for the massive publishing for multi threading. we have 5000-10000/s in total for the cluster. All messages must be persistent.

 Here are my questions.

1. How do we deal with connections. Should I create a single connection to 10 queues(we have XX_0 to XX10), or we create 10 connections for this. Now we use the latter one, but we have now around 1000 connections in the cluster , is this a burden to rabbitmq broker?

1K connections is not much, see https://www.rabbitmq.com/networking.html and http://www.rabbitmq.com/production-checklist.html

as some OS defaults would be inadequate even for that many.

 

2. How should we deal with channels, as I said , we create a channel in the thread local, and since we have around 1000 connections, now, we have over 100000 channels, is this really bad design? As you  pointed out, should we use channel pooling and how?(we don't use spirng amqp)

100K channels will consume resources. Are you sure you need that many? Would a small pool per connection not work? Combined with a small

connection pool (say, 64 or so).

Unless you have hundreds of thousands of cores, I don't see any reason to have 1K connections and 100K channels. Most JVMs don't handle

100K threads very well without extensive tuning, for example. Spring AMQP has figured it out years ago, I'm not suggesting that you

use it but at least you can take a look at what they do.

 

3. We don't use any special exchange, which means we only send to the default exchange and route to the queue directly, it is better to have multiple exchanges?

It doesn't matter. Channels do the routing and it is rarely the bottleneck, everything that queues have to do is (from writing to disk to mirroring to interacting with potentially slow consumers).

 

 

4. We are always suffering flow control, what important metrics  should we notice?

 Transient flow control under load is expected. Publishers have to do very little work compared to RabbitMQ and (usually) consumers.

Don't mirror what doesn't have to be mirrored, use at least as many queues as there are cores and see

context switching metrics. Search list archives for threads about Erlang VM scheduler-to-CPU core binding.

Tose and http://www.rabbitmq.com/monitoring.html and http://rabbitmq.com/ha.html are good starting points.

PerfTest can be used to simulate using a lot of topologies, although except for master it can use a lot more threads

than really necessary, making some workloads harder to simulate:
http://www.rabbitmq.com/java-tools.html.

 

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.