Blocking threads while publishing to RabbitMQ channel

[getaju69]

We are facing some issues where the we use a thread pool executor and a thread is blocking other threads resulting in the core pool size and max queue size to fill up

The thread which other threads are waiting on is:

AMQP Publisher [cameramanager]-135" #4626 prio=4 os_prio=0 tid=0x00007f0ca4824000 nid=0x7c70 runnable [0x00007f0a4b30b000]

   java.lang.Thread.State: RUNNABLE

at java.net.SocketOutputStream.socketWrite0(Native Method)

at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)

at java.net.SocketOutputStream.write(SocketOutputStream.java:155)

at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)

at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)

- locked <0x00000002e5f800c0> (a java.io.BufferedOutputStream)

at java.io.DataOutputStream.flush(DataOutputStream.java:123)

at com.rabbitmq.client.impl.SocketFrameHandler.flush(SocketFrameHandler.java:177)

at com.rabbitmq.client.impl.AMQConnection.flush(AMQConnection.java:557)

at com.rabbitmq.client.impl.AMQCommand.transmit(AMQCommand.java:127)

at com.rabbitmq.client.impl.AMQChannel.quiescingTransmit(AMQChannel.java:377)

- locked <0x00000002e5f7f9b8> (a java.lang.Object)

at com.rabbitmq.client.impl.AMQChannel.transmit(AMQChannel.java:353)

- locked <0x00000002e5f7f9b8> (a java.lang.Object)

at com.rabbitmq.client.impl.ChannelN.basicPublish(ChannelN.java:690)

at com.rabbitmq.client.impl.ChannelN.basicPublish(ChannelN.java:672)

at com.rabbitmq.client.impl.ChannelN.basicPublish(ChannelN.java:662)

at com.rabbitmq.client.impl.recovery.AutorecoveringChannel.basicPublish(AutorecoveringChannel.java:192)

at com.pcmseu.rabbitMQ.AMQPAbstractExchangePublisher.startPublisherProcess(AMQPAbstractExchangePublisher.java:239)

at com.pcmseu.rabbitMQ.AMQPAbstractExchangePublisher.lambda$send$0(AMQPAbstractExchangePublisher.java:211)

at com.pcmseu.rabbitMQ.AMQPAbstractExchangePublisher$$Lambda$377/2046803569.run(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748)

From the documentation, I see:

As a rule of thumb, sharing Channel instances between threads is something to be avoided. Applications should prefer using a Channel per thread instead of sharing the same Channel across multiple threads.

But we are creating a connection and channel only once, and the same channel is used for multiple threads for publishing messages using channel.basicPublish()

So could this be the problem, that we are sharing the channel across threads ? How can we use multiple channels - do we create a channel for each message we need to publish - or is there any option of pooled channels or likewise? 

If this indeed is not the problem, then what could probably cause this ? We did check the Rabbit MQ logs and couldn't find anything there.

Saw a similar issue: https://github.com/rabbitmq/rabbitmq-java-client/issues/236 , could it be same\similar issue ?

The amqp-client version we are using is 4.1.1

Also read in the documentatino that it could be due to Memory and Disk Alarms, but we checked that and it is fine.

RabbitMQ versio:3.7.15

Erlang version: 22.0.5

AMQP version:4.1.1

[Wesley Peng]

getaju69 wrote:
> As a rule of thumb, sharing Channel instances between threads is
> something to be avoided. Applications should prefer using a Channel per
> thread instead of sharing the same Channel across multiple threads.
>
> But we are creating a connection and channel only once, and the same
> channel is used for multiple threads for publishing messages using
> channel.basicPublish()
>
> So could this be the problem, that we are sharing the channel across
> threads ? How can we use multiple channels - do we create a channel for
> each message we need to publish - or is there any option of pooled
> channels or likewise?

Hi

The documentation has described that clearly:

Applications should prefer using a Channel per thread instead of
sharing the same Channel across multiple threads.

regards.

[getaju69]

HI Wesley,

Indeed, the documentation says so. But until now we haven't faced any issues with this, and we see this only on some servers.

We have around 1000 messages per second being published to the exchange - do you think it would be ok to create a new channel for each of the thread?

If so, can you point to any piece of code which does the same?

[Wesley Peng]

getaju69 wrote:
>
> We have around 1000 messages per second being published to the exchange
> - do you think it would be ok to create a new channel for each of the
> thread?
>

hello

each thread for a new channel is fine.
we have 9k/s messages for consuming.

regards.

[getaju69]

This is for publishing, and not for consuming. This application only publishes. Is it ok also to create a new channel for each of the messages published ? 

[Wesley Peng]

getaju69 wrote:
> This is for publishing, and not for consuming. This application only
> publishes. Is it ok also to create a new channel for each of the
> messages published ?
>

hello

why don't you create a channel in a thread, then publish as many as
possible messages in that channel?

regards.

[getaju69]

At the moment, we create a thread for each of the message - which uses the channel already created at first.

I think what we need to do is create a new channel when this happens , so 1 message = 1 thread = 1 channel.

We need to send the publish the messages in parallel, so I think it is not a good idea to have the messages published asynchronously using 1 channel for multiple messages.

What do you think?

[Wesley Peng]

Hello

getaju69 wrote:
> We need to send the publish the messages in parallel, so I think it is
> not a good idea to have the messages published asynchronously using 1
> channel for multiple messages.
>
> What do you think?

If I was you, I will prepare a thread pool, in each thread, open a
channel standing there. After that every thread try to publish messages
using the already openned channel. This is done in parallel.

I in fact never did thread stuff. I do programming with ruby, which is
not thread safe for most cases. If you are using Java, that is not the
case I consider for.

Thanks.

[getaju69]

Also - it seems that channel creation is not expensive- so we create a new channel every time and close it as soon as we publish - correct ? 

[Arnaud Cogoluègnes]

It is hard to say whether this blocking is due to using a channel in
multiple threads or to the network connectivity.

Some recommendations:
* do not use a channel instance from several threads at the same time
* do not create a channel for every message you publish. Channels
are not super costly to create compared to a connection, but their
creation implies a network roundtrip, which would slow down the
publishing. Re-use channels.
* do not create a thread for every message you publish, use an
abstraction like ExecutorService.
* please upgrade, both the server (it's a little behind) and the
client (the version you're using is 2.5 years old). For the Java
client, upgrade to 4.11.3 (backward compatible) or, better, to 5.7.3
if you can use Java 8 (there are some minor breaking changes between
4.x and 5.x [1])
* if your processing is already happening in long-lived threads,
having a dedicated publishing channel for each thread is a good idea
* there is no channel pool in the Java client. Spring AMQP has a channel pool.

[1] https://github.com/rabbitmq/rabbitmq-java-client/releases/tag/v5.0.0#user-content-breaking-changes

> --
> You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
> To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/1983e172-6deb-4135-a962-8347cd89c903%40googlegroups.com.

[Arnaud Cogoluègnes]

>
> Also - it seems that channel creation is not expensive- so we create a new channel every time and close it as soon as we publish - correct ?
>

Please don't do that, this will kill the throughput.

[Wesley Peng]

Please see Arnaud’s suggestions, especially this one:

if your processing is already happening in long-lived threads,
having a dedicated publishing channel for each thread is a good idea

This is exactly what I want to say. 

Regards 

--

You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/9abe231a-25e4-492c-9ad5-766ece721016%40googlegroups.com.

[getaju69]

Hi Arnaud,

We are already using a thread executor meaning we do not create a new thread always , but use the one from pool.

So we need to make sure we do not use the same channel across multiple threads. So use one channel per thread. But at the moment, we are using a thread pool with core pool size of 200 and max pool size of 2000. So as many channels would be created per application (there are other applications too connecting to the same RMQ server) . Is that ok ? Do we also need to close the channels explicitly ?

I also read in the documentation:

Given both of these factors, limiting the number of channels used per connection is highly recommended. As a rule of thumb, most applications can use a single digit number of channels per connection. 

Does this mean that it is recommended not to use a large thread pool - to limit the number of channels?

So the short term steps we could take are:

• Upgrade the client to 4.11.3
• Use one channel per thread and not share channel across threads.

And in longer term:

• Upgrade the client to 5.7.3
• Upgrade the server

Do you agree?

> To unsubscribe from this group and stop receiving emails from it, send an email to rabbitm...@googlegroups.com.

[getaju69]

@Wesley

Indeed, I get your point. We would be doing the changes to have once channel per thread.

Still we think the number of channels would be very high (as per the documentation as pasted above, it is suggested to keep in single digits). What do you guys think about it ? Could we have as many channels as threads which would be 200+ ? Or should we use a smaller threadpool? 

[Arnaud Cogoluègnes]

>
> So we need to make sure we do not use the same channel across multiple threads. So use one channel per thread. But at the moment, we are using a thread pool with core pool size of 200 and max pool size of 2000. So as many channels would be created per application (there are other applications too connecting to the same RMQ server) . Is that ok ? Do we also need to close the channels explicitly ?
>

200 is ok, 2000 is a lot, but OK for short spikes. You can close
channels if you know they won't be used before a fair amount of time
(dozens of minutes or more). Do not open and close channels several
times a second, try re-using them for these cases.

Note with these numbers you should get plenty of concurrency and parallelism.

> I also read in the documentation:
>
>> Given both of these factors, limiting the number of channels used per connection is highly recommended. As a rule of thumb, most applications can use a single digit number of channels per connection.
>
>
> Does this mean that it is recommended not to use a large thread pool - to limit the number of channels?

Well, I don't know about the specifics of your use case. These
recommendations are a bit conservative to avoid abuse, but I'm pretty
sure there are plenty of properly-working applications using 200+
channels in the wild.

>
> So the short term steps we could take are:
>
> Upgrade the client to 4.11.3
> Use one channel per thread and not share channel across threads.
>
> And in longer term:
>
> Upgrade the client to 5.7.3
> Upgrade the server
>
> Do you agree?

This looks reasonable.

[Wesley Peng]

A channel is just like a socket handle, the server will estimate its RW ability by realtime. So keeping large number of channels is not good for performance.

Regards 

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/333634cc-69b6-4b6f-bf20-242d6c65a0c5%40googlegroups.com.

[getaju69]

@Arnaud,

In https://github.com/rabbitmq/rabbitmq-java-client/issues/236

You have written:

The issue looks like #194 nevertheless and the workaround needs some specific settings:

connectionFactory.setRequestedHeartbeat(5);
ExecutorService shutdownExecutor = Executors.newSingleThreadExecutor();
connectionFactory.setShutdownExecutor(shutdownExecutor);

Could this solve my issue for now (since implementing a channel per thread would need more refactoring and testing)

If so, can you explain how it could work? I also see a similar issue in : https://groups.google.com/forum/#!msg/rabbitmq-users/3039DKxDxZg/SrK9rMm-BgAJ

Currently the request heartbeat is set as 60 seconds (which I think is also the default). What would happen if this value is set to 5 seconds ? How would the timeout work in this case? 

What exactly does the shut down executor do in here?

If we know the answers for these - we could apply this changes for now as a hotfix and then work on moving towards using a dedicated channel per thread for publishing. 

[Arnaud Cogoluègnes]

>
>> The issue looks like #194 nevertheless and the workaround needs some specific settings:
>>
>> connectionFactory.setRequestedHeartbeat(5);
>> ExecutorService shutdownExecutor = Executors.newSingleThreadExecutor();
>> connectionFactory.setShutdownExecutor(shutdownExecutor);
>
>
> Could this solve my issue for now (since implementing a channel per thread would need more refactoring and testing)
>

It could, but the original GitHub issue blames also a network
equipment that could contribute heavily to the problem. You may be in
the same situation.

> If so, can you explain how it could work? I also see a similar issue in : https://groups.google.com/forum/#!msg/rabbitmq-users/3039DKxDxZg/SrK9rMm-BgAJ
>
> Currently the request heartbeat is set as 60 seconds (which I think is also the default). What would happen if this value is set to 5 seconds ? How would the timeout work in this case?
>
> What exactly does the shut down executor do in here?

IIRC making the heartbeat lower and setting the shutdown executor will
make the connection close faster. The publishing thread may still be
stuck for a while, but new publishing should happen on a new workable
connection, because the dead one is now closed and non-functional.

Note these are assumptions: this is a problem that pops up once in a
while, I've never managed to reproduce it. The cause seems to be
buried in the JRE and in the TCP layer, and there's no clean way to
fix it in the Java client. Steps to reproduce it reliably would be
great.

>
> If we know the answers for these - we could apply this changes for now as a hotfix and then work on moving towards using a dedicated channel per thread for publishing.

You can apply those changes, but you can still run into problems if
channels are shared across threads.

>
>
> On Tuesday, 19 November 2019 13:55:46 UTC+1, Arnaud Cogoluègnes wrote:
>>
>> >
>> > So we need to make sure we do not use the same channel across multiple threads. So use one channel per thread. But at the moment, we are using a thread pool with core pool size of 200 and max pool size of 2000. So as many channels would be created per application (there are other applications too connecting to the same RMQ server) . Is that ok ? Do we also need to close the channels explicitly ?
>> >
>>
>> 200 is ok, 2000 is a lot, but OK for short spikes. You can close
>> channels if you know they won't be used before a fair amount of time
>> (dozens of minutes or more). Do not open and close channels several
>> times a second, try re-using them for these cases.
>>
>> Note with these numbers you should get plenty of concurrency and parallelism.
>>
>> > I also read in the documentation:
>> >
>> >> Given both of these factors, limiting the number of channels used per connection is highly recommended. As a rule of thumb, most applications can use a single digit number of channels per connection.
>> >
>> >
>> > Does this mean that it is recommended not to use a large thread pool - to limit the number of channels?
>>
>> Well, I don't know about the specifics of your use case. These
>> recommendations are a bit conservative to avoid abuse, but I'm pretty
>> sure there are plenty of properly-working applications using 200+
>> channels in the wild.
>>
>> >
>> > So the short term steps we could take are:
>> >
>> > Upgrade the client to 4.11.3
>> > Use one channel per thread and not share channel across threads.
>> >
>> > And in longer term:
>> >
>> > Upgrade the client to 5.7.3
>> > Upgrade the server
>> >
>> > Do you agree?
>>
>> This looks reasonable.
>

> --
> You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
> To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/08c9d7f4-c653-4d7f-8942-aa0e3a5cd8ff%40googlegroups.com.

[getaju69]

Hello @Arnaud and @Wesley,

It is only now we have decided to implement the one channel per thread architecture. We have decided to use apache commons pool library for the pooling of objects (channels here). I have a few questions though, if you could help answer:

1. So I would be borrowing an object (channel) from the pool of channels for each thread. Earlier I used to have a single channel (and connection) and declare the queues\exchanges on the channel (once at the start when the channel is null) . We need to do this as we want the application to create the exchange\queues when it starts up (if not existing already). But now I would be having a pool of channels - do I need to declare the exchanges and queues for all? Or is there a way I could check if it is already declared.

2. Should I close the channels explicitly ? I'm guessing when the connection is closed - the channel will also be closed?

3. Is there a maximum channels per connection that we can\should (or not) create ?

Regards

Ajmal

> To unsubscribe from this group and stop receiving emails from it, send an email to rabbitm...@googlegroups.com.

[Arnaud Cogoluègnes]

> 1. So I would be borrowing an object (channel) from the pool of channels for each thread. Earlier I used to have a single channel (and connection) and declare the queues\exchanges on the channel (once at the start when the channel is null) . We need to do this as we want the application to create the exchange\queues when it starts up (if not existing already). But now I would be having a pool of channels - do I need to declare the exchanges and queues for all? Or is there a way I could check if it is already declared.

Exchanges and queues are global to the virtual host, so they are "visible" to all the channels. One exception is exclusive queues, which are exclusive to the connection that created them, so by extension to the channels of the connection. If you don't use exclusive queues, you don't have worry about visibility of resources among your channels.

> 2. Should I close the channels explicitly ? I'm guessing when the connection is closed - the channel will also be closed?

Yes, the channels of a closed connection are not usable anymore.

> 3. Is there a maximum channels per connection that we can\should (or not) create ?

There is a limit, the default being 2047 channels per connection [1] (see [2] for the rational behind this value). Yet this is configurable and even negotiable between the client and the server [3]. Nevertheless the defaults should be fine for most applications.

The whole channels page [4] could be a useful read to tackle this task.

[1] https://github.com/rabbitmq/rabbitmq-server/blob/b8edec02392fba0788db0ccfea5af5017fc64485/Makefile#L28
[2] https://github.com/rabbitmq/rabbitmq-server/issues/1593
[3] https://www.rabbitmq.com/channels.html#channel-max
[4] https://www.rabbitmq.com/channels.html

[getaju69]

Hi @Arnaud,

Thanks, that clears many things indeed.

Our architecture is such that we declare the exchanges, queues and bindings for the one channel we had. I am planning to have 5 channels per connection now - as a pool. Plan is to declare the exchanges, queues and bindings for each channel.

Do you see any issues in declaring the exchanges multiple times, on different channels (on the same connection) ?

Regards,

Ajmal

[Arnaud Cogoluègnes]

Re-declaring should be fine (creation is idempotent), as long as the declarations are exactly the same (same name obviously, but also durable flag, parameters, etc).

________________________________________
From: rabbitm...@googlegroups.com <rabbitm...@googlegroups.com> on behalf of getaju69 <geta...@gmail.com>
Sent: Wednesday, July 8, 2020 11:43 AM
To: rabbitmq-users
Subject: Re: [rabbitmq-users] Blocking threads while publishing to RabbitMQ channel

[1] https://github.com/rabbitmq/rabbitmq-server/blob/b8edec02392fba0788db0ccfea5af5017fc64485/Makefile#L28<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Frabbitmq%2Frabbitmq-server%2Fblob%2Fb8edec02392fba0788db0ccfea5af5017fc64485%2FMakefile%23L28&data=02%7C01%7Cacogoluegnes%40vmware.com%7C190659dea6cb4f18630b08d823235a0a%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637297982035174827&sdata=W4r67f9umGS3mgZ99V1a0fPlg5xM8zkGsa19TIVj3AY%3D&reserved=0>
[2] https://github.com/rabbitmq/rabbitmq-server/issues/1593<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Frabbitmq%2Frabbitmq-server%2Fissues%2F1593&data=02%7C01%7Cacogoluegnes%40vmware.com%7C190659dea6cb4f18630b08d823235a0a%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637297982035184826&sdata=eS2%2BXNqqth07LVWRu8W4JutTHyt5GkA0YYArdETXItY%3D&reserved=0>
[3] https://www.rabbitmq.com/channels.html#channel-max<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.rabbitmq.com%2Fchannels.html%23channel-max&data=02%7C01%7Cacogoluegnes%40vmware.com%7C190659dea6cb4f18630b08d823235a0a%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637297982035184826&sdata=IJZcPano231%2FzCRGueB6Apip4WFlhViwVDHbciRCFdw%3D&reserved=0>
[4] https://www.rabbitmq.com/channels.html<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.rabbitmq.com%2Fchannels.html&data=02%7C01%7Cacogoluegnes%40vmware.com%7C190659dea6cb4f18630b08d823235a0a%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637297982035184826&sdata=kITyaemccuty7Gh1UF2Og4UiRKHYNR2rxnfoYqtfLEY%3D&reserved=0>

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com<mailto:rabbitmq-user...@googlegroups.com>.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/b43b6749-619d-4673-9e85-b70c4bab894co%40googlegroups.com<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Frabbitmq-users%2Fb43b6749-619d-4673-9e85-b70c4bab894co%2540googlegroups.com%3Futm_medium%3Demail%26utm_source%3Dfooter&data=02%7C01%7Cacogoluegnes%40vmware.com%7C190659dea6cb4f18630b08d823235a0a%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637297982035194824&sdata=nUegdgzGZLgZ5U5IboMZXsHN0p7Hc0vkis72kS9%2B9P4%3D&reserved=0>.