Issue || RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS

[Kushagra Bindal]

Hi Luke,

We are currently using 3.7.17 version. And we were observing continuous CPU spikes in that as compare to 3.6.10. Based on our below discussion you have suggested us to move to the latest version of 3.8.x and so we are now planning to upgrade our system to 3.8.2, but this is something subject of approval from Sr.leadership

https://groups.google.com/forum/#!topic/rabbitmq-users/N32iJTa6law  

Now, in parallel we observed one more thing in 3.7.17 version. In high CPU consumed machine (approx CPU spikes was 60%), we observed that there is a variable name as sbwt in erlang (default value of this in 22.0.7 erlang is MEDIUM). We edit our below file and override sbwt value to NONE.

vi ./usr/lib/rabbitmq/lib/rabbitmq_server-3.7.17/sbin/rabbitmq-env

RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="+sbwt none"

Once we perform this and restart our server, we found that CPU spikes suddenly come to 1%. We found this variable on url http://erlang.org/doc/man/erl.html, it states that

+sbwt none|very_short|short|medium|long|very_long

Sets scheduler busy wait threshold. Defaults to medium. The threshold determines how long schedulers are to busy wait when running out of work before going to sleep.

 & based on the details provided in  https://www.rabbitmq.com/runtime.html  we override this value.

Apart from this what we have also observed that in current version we also seems to be configuring memory for rabbitmq. I may be wrong but in older version i don't see following arguments  getting passed at startup, where as for new version it's there and these seems to be default values as well.

-MBas ageffcbf -MHas ageffcbf -MBlmbcs 512 -MHlmbcs 512 -MMmcs 30

Please confirm following items.

1. If there is any side impact of changing sbwt value from medium to none.

2. What are usage of allocating default configuration memory.

3. As I mentioned above as well that we are migrating to rabbit-3.8.2 & erlang-22.1.8 version, then these settings will be by default  handled or there will be any specific action required to lower down the CPU usage like we mentioned with the version 3.7.17 ?

--

Regards,
Kushagra Bindal

[Luke Bakken]

Hi Kushagra,

I was going to suggest setting +swbt none in your environment after confirming that was the issue. That's why I asked for the output of rabbitmq-diagnostics runtime_thread_stats from your cluster in this message:

https://groups.google.com/d/msg/rabbitmq-users/N32iJTa6law/TZAX9ecRAQAJ

But, I didn't get a response, so I didn't suggest +swbt none. The RabbitMQ team prefers to not guess as to what might fix issues.

Please do not modify the rabbitmq-env file. Instead, create the file /etc/rabbitmq/rabbitmq-env.conf with this content:

SERVER_ADDITIONAL_ERL_ARGS='+sbwt none'

1. If there is any side impact of changing sbwt value from medium to none.

One impact this setting could have is in message latency, but it is very unlikely. You should run benchmarks to confirm, if latency is important in your environment.

 

2. What are usage of allocating default configuration memory.

You should not change those arguments.

 

3. As I mentioned above as well that we are migrating to rabbit-3.8.2 & erlang-22.1.8 version, then these settings will be by default  handled or there will be any specific action required to lower down the CPU usage like we mentioned with the version 3.7.17

You should continue to set  +swbt none in /etc/rabbitmq/rabbitmq-env.conf if you see the high CPU issue without that setting. No other changes should be necessary.

Thanks,

Luke

[Kushagra Bindal]

Hi Luke

Thanks for the reply. Actually by looking at regular high cpu spikes there was some discussion initiated to change it to another version. 🙂 

So, do you still think that after making neccessary changes into rabbitmq-env.conf file, there is any immediate need to upgrade to 3.8.2 version or our current 3.7.17 version will work properly? And do we need to add this property in case of 3.8.2 version as well?

And yes I will perform proper performance test on this after making neccessary changes in both the cases.

Please confirm.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/906d22c7-5d76-4386-9e48-0e0da0212fe0%40googlegroups.com.

[Luke Bakken]

So, do you still think that after making neccessary changes into rabbitmq-env.conf file, there is any immediate need to upgrade to 3.8.2 version or our current 3.7.17 version will work properly?

That is your call. If you are planning an upgrade anyway, I suggest at least upgrading to the latest 3.7 version.

 

And do we need to add this property in case of 3.8.2 version as well?

See the answer I already provided.

[Kushagra Bindal]

Hi Luke,

We are doing some POC around 3.8.2 version.

Just wanted to understand that from which version this +swbt flag was introduced in Erlang? I understand that this flag is present in 3.7.x & 3.8.x.

Does +swbt flag exist in 3.6.10 version as well?

And if that is the case what was the default value of +swbt in 3.6.10.

--

You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/51347c51-63bb-4dcf-9adc-b983856923db%40googlegroups.com.

--

Regards,
Kushagra Bindal
+91-9013792807

[Kushagra Bindal]

Hi

What does that mean? I I didn't get it. Can you please provide response to my queries 

On Wed, Dec 11, 2019, 7:08 PM Amy Parker <amypar...@gmail.com> wrote:

DJTogether/DJTogether/ConnectButton.swift

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/1938f18a-f285-4a81-aa83-a385a0005190%40googlegroups.com.

[Luke Bakken]

Hello,

The correct argument is +sbwt, not +swbt

+sbwt is an Erlang VM argument. By default, this argument is not used by RabbitMQ so it doesn't make sense to associate the argument with a RabbitMQ version.

I just tested and Erlang 20.3 supports +sbwt, so it has been around at least that long.

Thanks,

Luke

[Kushagra Bindal]

Thanks Luke. That was a typo.

--

You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/f09e19f4-9975-4900-b4f9-973be50eebb5%40googlegroups.com.

[Pranith Vanaparthi]

Hi Luke,

We are facing a similar issue of High CPU spike on RabbitMQ servers. After reading RabbitMQ documentation for performance improvement we have configured
schedulers count to 16 (reduced the value to 16 from available cores) using SERVER_ADDITIONAL_ERL_ARGS="+S 16:16" parameter. But we did not see much difference in the CPU spike.

As suggested in this group we have run rabbitmq-diagnostics runtime_thread_stats statistics and here is the report of the same

        Thread      aux check_io emulator       gc    other     port    sleep

 scheduler( 1)    0.91%    0.33%   10.93%    0.81%   35.06%    1.71%   50.25%
 scheduler( 2)    0.85%    0.32%   10.22%    0.71%   33.34%    1.77%   52.79%
 scheduler( 3)    0.85%    0.32%   10.10%    0.73%   32.56%    1.70%   53.75%
 scheduler( 4)    0.69%    0.27%    9.84%    0.67%   32.49%    1.60%   54.45%
 scheduler( 5)    0.82%    0.30%   10.25%    0.73%   32.84%    1.67%   53.39%
 scheduler( 6)    0.85%    0.35%   12.20%    0.85%   39.26%    2.11%   44.39%
 scheduler( 7)    0.65%    0.27%    9.57%    0.72%   30.47%    1.56%   56.76%
 scheduler( 8)    0.01%    0.00%    0.00%    0.00%    0.36%    0.00%   99.63%
 scheduler( 9)    0.01%    0.00%    0.00%    0.00%    0.37%    0.00%   99.62%
 scheduler(10)    0.00%    0.00%    0.00%    0.00%    0.27%    0.00%   99.73%
 scheduler(11)    0.00%    0.00%    0.00%    0.00%    0.08%    0.00%   99.91%
 scheduler(12)    0.00%    0.00%    0.00%    0.00%    0.07%    0.00%   99.93%
 scheduler(13)    0.00%    0.00%    0.00%    0.00%    0.07%    0.00%   99.93%
 scheduler(14)    0.00%    0.00%    0.00%    0.00%    0.07%    0.00%   99.93%
 scheduler(15)    0.00%    0.00%    0.00%    0.00%    0.07%    0.00%   99.93%
 scheduler(16)    0.00%    0.00%    0.00%    0.00%    0.07%    0.00%   99.93%
 
We are using rabbitMq 3.8.2 version and we have not modified default sbwt parameter. Our application runs with a load of 2000 messages/second of produce and 12000 messages/second consumption rate.

Please suggest on high CPU usage of beam.smp.

Thanks in advance

Regards

Pranith

[Luke Bakken]

Hello,

Please take the time to search this group for answers. This recent discussion covers this CPU usage:

https://groups.google.com/d/topic/rabbitmq-users/dTP6sCSlNpM/discussion

Finally, don't respond to old topics - start new ones.

Thanks,

Luke