Unexpectedly high CPU usage with streams

[Yoakim Persson]

Hi,

I’ve been having some issues regarding higher then expected CPU usage when using ~1000 streams. This behaviour has been observed on both RabbitMQ 3.9.13 and the newer 3.10.1.

 Points:

• We are running on CloudAMQP’s single node “Roaring Rabbit” tier which relates to a Standard_D2as_v4 Azure machine.
• On creation of the ~1000 streams the CPU usage included:  and holds at approx. ~35%.
  • Worth noting that there is no traffic on the majority of these.
• Speculative busy waiting is disabled with: RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="+sbwt none +sbwtdcpu none +sbwtdio none"
  • This was checked as our instance was initially hosted on a lower tier which is run on a burstable machine (Standard_B1ms)
• Metric gathering has been disabled with:
  • {rabbitmq_management, [{rates_mode,basic}, {disable_stats, true}]}
  • {rabbitmq_management_agent,[{disable_metrics_collector,true}]}·         
• As a comparison we created the same number of quorum queues however we did not see an increase to the same level as what we saw with the streams. 
  

[Running on "Big Bunny"/Standard_B1ms. Deletion and re-creation of the stream queues.  The jump from ~50% CPU usage to 100% usage can be attributed to exhausting burstable credits]  

[Running on upgraded “Roaring Rabbit”/Standard_D2as_v4. Taken when creating the additional 1000 quorum queues to the existing steam queues.]

 

 

Is anyone could shed some light on this it would be much appreciated.

[Michal Kuratczyk]

Hi,

A few questions:

1. You only mention stream declarations (not publishing/consumption) but your screenshots cover hours. On my system, declaring 1000 streams takes a few seconds. What's going on here? Is it just the declarations taking so long?

2. Why are you concerned with CPU usage during stream declaration? If the declaration takes hours, that's obvious but if it's more like on my system, why would that worry you? Streams are meant to be long-lived, so the initial declaration should not be much of a problem. We obviously wouldn't mind it being faster/less CPU-intensive, and probably can optimize it to some extent, but I'm trying to understand how important that is.

3. Can you see if you can reproduce this with https://github.com/rabbitmq/rabbitmq-stream-java-client and if yes - share the flags used? A simple `stream-perf-test -x 1000 -y 1000 -sc 1000` declares 1000 streams and starts a publisher and consumer for each stream. On my system, it takes about 15s for this initial setup, until publishers start publishing and consumers start consuming.

Best,

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/b997fcd5-1864-46c5-a76b-a195fa51a319n%40googlegroups.com.

--

Michał

RabbitMQ team

[Yoakim Persson]

Hi Michał,

I appreciate your response. I apologise for the misunderstanding however I have no issue with the actual time required to declare/create the streams. This is consistent regardless of what type of queues I am creating.

My issue is actually the unexpectedly high CPU usage once these have been created, idling, compared to others such as quorum queues. These queues actually have no traffic on then.

According to the docs (https://www.rabbitmq.com/streams.html#resource-use) steams are indicated to have a lower CPU (and memory) footprint the quorum queues.

However what my second screenshot is attempting to show is that with an existing 1000 streams the creation of an additional 1000 quorum queues appeared to have little to no impact on the CPU usage compared to what is required to have the 1000 streams running.

Hopefully this clarifies my issue and again appreciate your response.

Regards,
Yoakim

[Michal Kuratczyk]

Hi,

Thanks a lot for reporting this, we can see the problem now. The details depend on the number of streams and CPU cores available but stream metrics collection,

which happens every 5 seconds even when the streams are idle, is indeed quite expensive. You can expect lower CPU usage in the near future, perhaps in the next patch release.

Most likely the changes will happen as part of this PR: https://github.com/rabbitmq/rabbitmq-server/pull/4928.

Best,

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/f19bb975-0d8a-40e5-84fb-d849eed6d2cen%40googlegroups.com.

--

Michał

RabbitMQ team

[Yoakim Persson]

Hi,

Thanks, that is great to hear.

Just curiously, I do know some testing was performed by disabling certain metrics, configuring RabbitMQ with:

{rabbitmq_management, [{rates_mode,basic}, {disable_stats, true} ]}, {rabbitmq_management_agent, [{disable_metrics_collector,true} ]}

Would the stream metrics that you mentioned still have been enabled with a cluster configured with the above?

Again, great to hear that this is likely to be solved shortly.

Kind regards,

Yoakim

[Michal Kuratczyk]

Hi,

These settings don't affect stream metrics collection. For historical reasons, there are (too) many ways of collecting and exposing metrics in RabbitMQ,

so it can be confusing. We definitely want to simplify this in the future.

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/3afd02b0-b259-4119-97d6-980027025dcbn%40googlegroups.com.

--

Michał

RabbitMQ team

[Michal Kuratczyk]

Hi,

I'm happy to report that this has already been improved. On my machine, without the fix, I saw 1-1.5 cores used for metric collection every

few seconds (with 1000 idle streams). Right now, all cores stay under 5% for me. The improvement will most likely be included in 3.10.6.

Best,

--

Michał

RabbitMQ team

[Yoakim Persson]

Hi,

Thanks for the clarification regarding the metrics. 

Thanks, that is fantastic to hear, looking forward to updating to the newer version.

Appreciate all your help,

Kind regards,

Yoakim