Very slow performance with RabbitMQ

[Gregor Cieslak]

I am searching an ESB for our enterprise and i got in love with MassTransit! :-)
However the performance with RabbitMQ is very low on my dev machine (i7-2600, 8 GB).
The hard disk is very active when pub/sub messages,there seem to be a lot of read/write operations?!
CPU activity is almost 0%, but erl.exe is reading/writing hard all the time.
RabbitMQ was installed with the default settings 2 days ago... So its basicly a fresh install.

I reduced load test to 1000 messages, otherwise it would never finish..

HeavyLoad - MassTransit Load Generator

RabbitMQ Load Test:
Started at 03.10.2013 13:15:55
  Publishing 1000 messages
    1000 messages in 93,16 seconds
      10,73 messages/second
  Receiving 1000 messages
    2000 messages in 93,18 seconds
      21,46 messages/second
Finished at 03.10.2013 13:17:28
Total elapsed time: 00:01:33.1799040


End of line.



--------------------------
Local Loopback Test:

HeavyLoad - MassTransit Load Generator

Starting Local Loopback Consumer Load Test
Loopback Load Consumer Test:
Started at 03.10.2013 13:20:44
  Publishing 10000 messages
    10000 messages in 1,71 seconds
      5.838,58 messages/second
  Receiving 10000 messages
    20000 messages in 3,93 seconds
      5.086,11 messages/second
Finished at 03.10.2013 13:20:48
Total elapsed time: 00:00:03.9325971


End of line.


Am i missing something?

[Travis Smith]

We do a ton of volume with effectively zero configuration changes to RMQ. At our lowest, we're pumping ~100 msg/s through RMQ with a peek well over 1000 msg/s. IO isn't our bottleneck. 

-Travis

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.
To post to this group, send email to masstrans...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/a2a388fa-e118-46e9-afee-70061e2366d0%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Gregor Cieslak]

Wish i could have the same results....

I did more testing and found out:

Running the Heavy Load Test with MSMQ on the same machine: ~1600 msg / sec
Running the Heavy Load Test on another machine with RabbitMQ: 9 Message / sec :-(

I installed EasyNetQ. Publishing/Consuming is very fast with RabbitMQ with that client, i don't know what MassTransit is doing different?


Probe/WriteIntrospectionToConsole:

host.machine_name: USER-PC
mt.concurrent_receive_threads: 1
mt.control_bus: rabbitmq://localhost/heavy_load
mt.default_serializer: JsonMessageSerializer
mt.max_consumer_threads: 32
mt.network: user-pc
mt.receive_from: rabbitmq://localhost/heavy_load
mt.receive_timeout: 00:00:03
mt.service_count: 3
mt.transport: [rabbitmq] RabbitMqTransportFactory
mt.transport: [loopback] LoopbackTransportFactory
mt.version: 2.8.0.0
net.version: 4.0.30319.296
os.bits: x64
os.version: Microsoft Windows NT 6.1.7601 Service Pack 1
process.bits: x32
process.fulltrust: True
process.id: 5560
zz.mt.inbound_pipeline: Pipeline
        Routed (MassTransit.IConsumeContext, MassTransit)
                Translated to MassTransit.Diagnostics.Tracing.GetMessageTraceLis
t, MassTransit
                Routed (MassTransit.Diagnostics.Tracing.GetMessageTraceList, Mas
sTransit)
                        Consumed by Instance (MassTransit.Diagnostics.Tracing.Ge
tMessageTraceList, MassTransit)
                Translated to HeavyLoad.GeneralMessage, HeavyLoad
                Routed (HeavyLoad.GeneralMessage, HeavyLoad)
                        Consumed by Instance (HeavyLoad.GeneralMessage, HeavyLoa
d)
                Translated to HeavyLoad.Load.SimpleResponse, HeavyLoad
                Routed (HeavyLoad.Load.SimpleResponse, HeavyLoad)
                        Consumed by Instance (HeavyLoad.Load.SimpleResponse, Hea
vyLoad)

zz.mt.outbound_pipeline: Pipeline
        Interceptor
        Interceptor
        Routed (MassTransit.ISendContext, MassTransit)

-Travis

To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.

[Travis Smith]

The HeavyLoad sample might not actually be that great of a measure. What version of MT are you using? 

-Travis

To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/918b3a6c-9dc4-47f0-a7fa-8e0254f140d0%40googlegroups.com.

[Gregor Cieslak]

It turned out we were using the latest source from github, but compiled in debug mode! This affected RabbitMQ (but not MSMQ), bummer!
HeavyLoad publishes now ~100msg/sec and consuming 135 msg/sec..that looks much better then before.

By the way: Does it make sense to use MassTransit for request/response scenarios as well?
Besides pub/sub events, we need to query other back ends over the atlantic ocean and vice versa.
For example:
Users looking into their orders on a Webserver in US
MT US sends a request to another instace of MT EU(rope)
MT EU checks its database in the back end, replies with a response back to MT US.
Webserver delivers.
Or should traditional Webservices be implemented for this scenarios?
Perhaps MassTransit could switch in that case to a HTTP Transport, instead of RabbitMQ?

Ah..and you guys rock!
Greg

-Travis

-Travis

To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsubscribe...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/a2a388fa-e118-46e9-afee-70061e2366d0%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.
To post to this group, send email to masstrans...@googlegroups.com.

[Chris Patterson]

Not sure about heavy load, I haven't run it in ages. 

The new stress tool I wrote (I haven't published yet to GitHub, but will soon enough) typically shows significantly better message throughput than you're seeing. On the order of 2,000 message/second with multiple concurrent consumers doing request/response messages to a single host server.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

[Chris Patterson]

Here is the output from the new stress console testing 300,000 messages via RabbitMQ (default install, running on a 2008 R2 VM on my Retina MBP).

09:11:17,301 INFO  Configuration Result:

[Success] Name MassTransitStressConsole

[Success] DisplayName MassTransit RabbitMQ Stress Console

[Success] Description Generates a stressful load on RabbitMQ using MassTransit

[Success] ServiceName MassTransitStressConsole

09:11:17,336 INFO  Topshelf v3.1.107.0, .NET Framework v4.0.30319.18052

09:11:17,497 INFO  Creating service bus at rabbitmq://localhost/stress_service?prefetch=10

09:11:21,115 INFO  Creating stress client at rabbitmq://localhost/stress_service_client_0

09:11:21,738 INFO  Creating stress client at rabbitmq://localhost/stress_service_client_1

09:11:21,824 INFO  Creating stress client at rabbitmq://localhost/stress_service_client_2

09:11:21,824 INFO  Creating stress client at rabbitmq://localhost/stress_service_client_3

09:11:21,824 INFO  Creating stress client at rabbitmq://localhost/stress_service_client_4

09:11:21,824 INFO  Creating stress client at rabbitmq://localhost/stress_service_client_5

09:11:22,413 INFO  Creating stress client at rabbitmq://localhost/stress_service_client_6

09:11:22,414 INFO  The MassTransitStressConsole service is now running, press Control+C to exit.

09:11:22,413 INFO  Creating stress client at rabbitmq://localhost/stress_service_client_7

09:11:22,415 INFO  Creating stress client at rabbitmq://localhost/stress_service_client_8

09:11:22,416 INFO  Creating stress client at rabbitmq://localhost/stress_service_client_9

09:16:03,533 INFO  Service Stop requested, exiting.

09:16:03,536 INFO  Stopping the MassTransitStressConsole service

09:16:03,544 INFO  Stress completed

09:16:03,545 INFO  Request Count: 300000

09:16:03,547 INFO  Response Count: 300000

09:16:03,548 INFO  Average Response Time: 5ms

09:16:03,557 INFO  Max Response Time: 918ms

09:16:03,562 INFO  Min Response Time: 0ms

09:16:03,760 INFO  Med Response Time: 4ms

09:16:03,931 INFO  95% Response Time: 10ms

09:16:03,932 INFO  Elapsed Test Time: 00:04:42.4450039

09:16:03,934 INFO  Total Client Time: 2766621ms

09:16:03,935 INFO  Per Client Time: 276662ms

09:16:03,936 INFO  Message Throughput: 2168m/s

09:16:06,535 INFO  The MassTransitStressConsole service has stopped.

[Chris Patterson]

And yeah, I decided to publish it to GitHub this morning:

https://github.com/phatboyg/MassTransit-Stress

[Dru Sellers]

should have called it trafficjam ;)

To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/CAHX4WYe4Z0MkntTrc2qiquqv-SxWS8uetUE3%3DvA6XBGB%3DGDLWQ%40mail.gmail.com.

[Travis Smith]

should have called it trafficjam ;)

+1

And if you bridging across WAN connections, then I would use web services for the bridge. You can use MT to publish to a bridge service that does the work. But you'll want some sort of fault tolerance and control around the bridge that isn't as easy to get with messaging alone. MT does support a Request-Response pattern for messaging though. 

[Chris Patterson]

Yeah, my naming consultant (Dru) was non-responsive when I clicked File, New Project...

To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/CAHwtU2mvbLFFWgtZh6khGo-RRn6%3DKgnNTmUNbr5%2B6yet6Z7H8Q%40mail.gmail.com.

[Dru Sellers]

<3

To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/CAHX4WYc30PS2rnjwt9Xn849vf5CtHhos9nSQrEEBNStb10hEkQ%40mail.gmail.com.

[Gregor Cieslak]

Downloaded TrafficJam from github, restored the packages and the hard disk is writing like crazy again:

19:05:39,236 INFO  Configuration Result:

[Success] Name MassTransitStressConsole
[Success] DisplayName MassTransit RabbitMQ Stress Console
[Success] Description Generates a stressful load on RabbitMQ using MassTransit
[Success] ServiceName MassTransitStressConsole

19:05:39,259 INFO  Topshelf v3.1.107.0, .NET Framework v4.0.30319.296
19:05:39,292 INFO  Creating service bus at rabbitmq://localhost/stress_service?p
refetch=10
19:05:42,074 INFO  Creating stress client at rabbitmq://localhost/stress_service
_client_0
19:05:42,492 INFO  Creating stress client at rabbitmq://localhost/stress_service
_client_1
19:05:42,573 INFO  Creating stress client at rabbitmq://localhost/stress_service
_client_2
19:05:42,573 INFO  Creating stress client at rabbitmq://localhost/stress_service
_client_4
19:05:42,573 INFO  Creating stress client at rabbitmq://localhost/stress_service
_client_6
19:05:42,574 INFO  Creating stress client at rabbitmq://localhost/stress_service
_client_5
19:05:42,574 INFO  Creating stress client at rabbitmq://localhost/stress_service
_client_8
19:05:42,574 INFO  Creating stress client at rabbitmq://localhost/stress_service
_client_7
19:05:42,577 INFO  Creating stress client at rabbitmq://localhost/stress_service
_client_3
19:05:42,729 INFO  Creating stress client at rabbitmq://localhost/stress_service
_client_9
19:05:42,574 INFO  The MassTransitStressConsole service is now running, press Co
ntrol+C to exit.
19:06:13,447 INFO  Service Stop requested, exiting.
19:06:13,453 INFO  Stopping the MassTransitStressConsole service
19:06:13,463 INFO  Stress completed
19:06:13,463 INFO  Request Count: 1000
19:06:13,463 INFO  Response Count: 1000
19:06:13,464 INFO  Average Response Time: 187ms
19:06:13,469 INFO  Max Response Time: 1133ms
19:06:13,470 INFO  Min Response Time: 1ms
19:06:13,474 INFO  Med Response Time: 183ms
19:06:13,476 INFO  95% Response Time: 321ms
19:06:13,476 INFO  Elapsed Test Time: 00:00:31.3998995
19:06:13,477 INFO  Total Client Time: 259564ms
19:06:13,477 INFO  Per Client Time: 25956ms
19:06:13,478 INFO  Message Throughput: 77m/s
19:06:16,397 INFO  The MassTransitStressConsole service has stopped.

[Chris Patterson]

Debug vs Release? 

Are you running it on an external USB floppy drive? 

Half a dozen virus scanners running at the same time?

Durable messaging requires disk IO, so I can't be sure what to say there.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.
To post to this group, send email to masstrans...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/955c117c-456b-43ba-b90e-5ea5dd4e978f%40googlegroups.com.

[Chris Patterson]

Also, open the RabbitMQ management console and look at the use of things like file handles, etc. to see why RabbitMQ is having slowness. Because I see all green boxes on mine while running the stressful load.

On Thu, Oct 3, 2013 at 10:07 AM, Gregor Cieslak <gregor....@gmail.com> wrote:

[Gregor Cieslak]

RabbitMQ management console shows all green, no usb drive, no anti virus or any other exostic configuration. compiled in release mode (will not make the same mistake again)
i did again a clean install .. erlang.exe, rabbitmq.exe, nuget masstransit.
there is something really strange going on, since the hard disk is working so hard when publishing messages. procmon shows erl.exe writing into queues and journal.pif

will try to find out what it is..

To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

[Chris Patterson]

You might check things like available file descriptors, and things of that nature. For fun, I installed RabbitMQ 3.1.5 on my Raspberry Pi and managed to get these results:

13:01:08,443 INFO  Stress completed

13:01:08,444 INFO  Request Count: 1000

13:01:08,445 INFO  Response Count: 1000

13:01:08,446 INFO  Average Response Time: 275ms

13:01:08,452 INFO  Max Response Time: 5836ms

13:01:08,454 INFO  Min Response Time: 19ms

13:01:08,459 INFO  Med Response Time: 216ms

13:01:08,462 INFO  95% Response Time: 360ms

13:01:08,463 INFO  Elapsed Test Time: 00:00:45.8632307

13:01:08,465 INFO  Total Client Time: 369645ms

13:01:08,470 INFO  Per Client Time: 36964ms

13:01:08,471 INFO  Message Throughput: 54m/s

13:01:11,457 INFO  The MassTransitStressConsole service has stopped.

To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/c52f7883-6c3e-4774-a902-b17772de851a%40googlegroups.com.

[Gregor Cieslak]

http://i.imgur.com/V9xU8zr.jpg

To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsubscribe...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/955c117c-456b-43ba-b90e-5ea5dd4e978f%40googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.
To post to this group, send email to masstrans...@googlegroups.com.