RabbitMQ Extremely Slow Publishing

[kashir hasan]

We are using Masstransit 8.0.2 with RabbitMQ(3.8.1 Erlang 22.1.5) and .Net6 . The message is being published from a TCPClient application. The message publishing time is increasing gradually and taking upto 30 minutes in publishing a single message. All the messages are being published in a background TCP receiver service. The data rate is fast as 60-70 messages are received at TCP client and being published to Rabbit using Masstransit asynchornously.

• DataMessage consumer is deployed on 3 different servers with total 5 consumer having consumer utilization of 100% (RabbitMQ Portal).
• All consumers have prefetch count of 100(we have tried with 1,5,16,50,100 and prefetch count having same results)
• The message published and acknowledging rate on this queue is on avg 10/s. (RabbitMQ server configuration 4 core cpu, 16 gb ram,Disk I/O 6400)

We have tried multiple configurations by playing with prefetch count and increasing and decreasing the consumers. We have also viewed the server cpu/memory/network utilization which all are under 50% on average Bus is being starting in application Startup 

public static void AddServiceBus(this IServiceCollection services, IConfiguration configuration, int prefetchCount = 0, params Type[] consumers)
        {
            services.AddMassTransit(x =>
            {
                if (consumers != null && consumers.Any())
                {
                    x.AddConsumers(consumers);
                }
                x.UsingRabbitMq((context, configurator) =>
                {
                    var rabbitMqSettings = configuration.GetSection(nameof(RabbitMqConfiguration)).Get<RabbitMqConfiguration>();
                    configurator.Host(rabbitMqSettings.Host, d =>
                    {
                        d.Username(rabbitMqSettings.Username);
                        d.Password(rabbitMqSettings.Password);
                    });
                    configurator.ConfigureEndpoints(context);
                    configurator.UseRetry(b =>
                    {
                        b.Immediate(3);
                    });

                    if (prefetchCount > 0)
                        configurator.PrefetchCount = prefetchCount;
                });
                x.Configure<MassTransitHostOptions>(options =>
                {
                    options.WaitUntilStarted = true;
                    options.StartTimeout = TimeSpan.FromSeconds(30);
                    options.StopTimeout = TimeSpan.FromMinutes(1);

                });
            });
        }

private async Task<ErrorCode> PublishDataAsync(BaseData Data, string _messageGuid)
        {
            try
            {
                using var scope = _serviceProviderFactory.CreateScope();
                var publishEndpoint = scope.ServiceProvider.GetRequiredService<IPublishEndpoint>();

                var DataMessage = new DataMessage(Data);
                await _publishEndpoint.Publish(DataMessage);
                _logger.LogInformation("{messgeguid} Data Published to MassTransit", _messageGuid);
                return ErrorCodes.SUCCESS;
            }
            catch (Exception e)
            {
                _logger.LogError(e, $"Message could not be published. {JsonConvert.SerializeObject(Data)}");
            }

        }
We are trying to increasing our message publishing rate. Our publishing message size is on avg 2kb.

I've also posted the question on StackOverFlow

[Michal Kuratczyk]

Hi,

First, upgrade to a supported RabbitMQ version: https://www.rabbitmq.com/versions.html.

If the issue persists, please provide https://perftest.rabbitmq.com/ flags we can use to simulate your workload or a github repo with an executable app that we can use to test this.

Best,

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/aba8de94-a9ff-4b62-8cd5-fc2d0094f40cn%40googlegroups.com.

--

Michał

RabbitMQ team

[Luke Bakken]

As Michal said, you should be using supported RabbitMQ and Erlang versions, though that is probably not the root cause of your issue.

Right now you are asking us to guess what is the issue. I suggest doing some or all of the following:

• Provide a project that I can clone, compile and run to see the same behavior you describe. Providing two incomplete code snippets is not sufficient.
• Ask your question in a MassTransit-specific support forum - https://github.com/MassTransit/MassTransit/discussions. Again, you should provide code that someone can clone, compile and run to reproduce the issue.

The RabbitMQ team maintains the .NET client library used by MassTransit. We know that fast performance is possible using it, and using MassTransit, so you have an issue in how you are using the library.

Thanks,

Luke