High Scheduler CPU usage on one RabbitMQ 3.8.6 cluster node

Ruian

unread,

Jan 10, 2021, 2:25:07 AM1/10/21

to rabbitmq-users

Hi,

I have one rabbitmq node that having cpu usage on scheduler be much higher than other nodes in a 3.8.6 cluster.

The problem node doesn't hold any master queue and has only one client connection with 40iB/s traffic, so the high cpu usage on the scheduler seems weird to me.

The rabbitmqctl list_queues --local output:

root@rabbitmq-2-rabbitmq-1:/# rabbitmqctl list_queues --local
Timeout: 60.0 seconds ...
Listing queues for vhost / ...
root@rabbitmq-2-rabbitmq-1:/#

The rabbitmq-diagnostics runtime_thread_stats:

rroot@rabbitmq-2-rabbitmq-1:/# rabbitmq-diagnostics runtime_thread_stats --sample-interval 15

Will collect runtime thread stats on rab...@rabbitmq-2-rabbitmq-1.rabbitmq-2-rabbitmq-discovery.default.svc.cluster.local for 15 seconds...

Average thread real-time : 15023779 us

Accumulated system run-time : 67253958 us

Average scheduler run-time : 14971634 us

Thread alloc aux bifbusy_wait check_io emulator ets gc gc_full nif other port send sleep timers

Stats per thread:

async( 0) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% 0.00%

...

async(63) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% 0.00%

aux( 1) 0.26% 0.49% 0.00% 0.00% 0.14% 0.00% 0.00% 0.00% 0.00% 0.00% 0.01% 0.00% 0.00% 99.10% 0.00%

dirty_cpu_( 1) 1.08% 0.00% 0.00% 8.22% 0.00% 1.34% 0.00% 11.11% 28.85% 0.00% 0.01% 0.00% 0.00% 49.40% 0.00%

...

dirty_io_s(10) 0.04% 0.00% 0.00% 12.94% 0.00% 1.07% 0.00% 0.00% 0.00% 9.19% 0.04% 0.00% 0.00% 76.71% 0.00%

poll( 0) 0.00% 0.00% 0.00% 0.00% 0.29% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 99.71% 0.00%

scheduler( 1) 5.94% 1.00% 15.12% 1.19% 1.60% 41.03% 2.83% 3.02% 4.12% 0.73% 15.29% 3.10% 3.78% 0.37% 0.86%

scheduler( 2) 6.47% 2.72% 16.40% 0.98% 1.30% 39.41% 2.29% 3.55% 3.31% 0.20% 8.80% 10.33% 2.54% 0.33% 1.37%

Stats per type:

async 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% 0.00%

aux 0.26% 0.49% 0.00% 0.00% 0.14% 0.00% 0.00% 0.00% 0.00% 0.00% 0.01% 0.00% 0.00% 99.10% 0.00%

dirty_cpu_sche 1.15% 0.00% 0.00% 7.41% 0.00% 1.07% 0.00% 10.72% 26.75% 0.00% 0.01% 0.00% 0.00% 52.90% 0.00%

dirty_io_sched 0.04% 0.00% 0.00% 8.54% 0.00% 0.62% 0.00% 0.00% 0.00% 6.06% 0.04% 0.00% 0.00% 84.71% 0.00%

poll 0.00% 0.00% 0.00% 0.00% 0.29% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 99.71% 0.00%

scheduler 6.20% 1.86% 15.76% 1.08% 1.45% 40.22% 2.56% 3.29% 3.72% 0.47% 12.05% 6.71% 3.16% 0.35% 1.12%

The problem scheduler keeps 40.22% usage on emulator.

On the contrast, scheduler on other nodes keep around 50% usage on sleep.

Does someone could help me to do further investigation why the node keep busy?

Thank you very much.

Ruian

unread,

Jan 10, 2021, 2:32:02 AM1/10/21

to rabbitmq-users

Here is the process stats of the problem node:

Ruian 在 2021年1月10日星期日下午3:25:07 [UTC+8] 的信中寫道：

Ruian

unread,

Jan 10, 2021, 1:34:01 PM1/10/21

to rabbitmq-users

some of the rabbitmq-diagnostics observer screenshot

Ruian 在 2021年1月10日星期日下午3:32:02 [UTC+8] 的信中寫道：

Reply all

Reply to author

Forward