Has anybody did Performance test PUB/SUB on ZeroMQ its slow for me

2006 views
Skip to first unread message

Meetu Maltiar

unread,
Mar 28, 2012, 1:13:41 AM3/28/12
to Akka User List
Hi There,

I ran a really simple ZeroMQ one PUB and one SUB on my machine which
is i5 processor macbook and am getting low numbers. It is just 3300
msgs/sec.

Here is the code link
https://github.com/knoldus/Akka2Bench/blob/master/src/main/scala/com/knoldus/akka/bench/zmq/ZMQApplication.scala

and sometimes I am also missing some messages. Please let me know if
these are expected numbers or I am missing something here.


Regards,
Meetu Maltiar
www.knoldus.com
www.knoldus.wordpress.com

Patrik Nordwall

unread,
Mar 28, 2012, 1:44:19 AM3/28/12
to akka...@googlegroups.com
On Wed, Mar 28, 2012 at 7:13 AM, Meetu Maltiar <me...@knoldus.com> wrote:
Hi There,

I ran a really simple ZeroMQ one PUB and one SUB on my machine which
is i5 processor macbook and am getting low numbers. It is just 3300
msgs/sec.

Yes, that is low. 
You should always do a warm-up before starting measurement.
I don't think that will do that big difference that you expect, but it's a prerequisite to almost any benchmark.
 
and sometimes I am also missing some messages. Please let me know if
these are expected numbers or I am missing something here.


I don't know what the real problem is. Make sure you have correct prerequisites (warm-up period, correct jvm parameters etc) and then try to isolate and identify the bottleneck.

--
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To post to this group, send email to akka...@googlegroups.com.
To unsubscribe from this group, send email to akka-user+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/akka-user?hl=en.




--
Patrik Nordwall
Typesafe - The software stack for applications that scale
Twitter: patriknw

Meetu Maltiar

unread,
Mar 28, 2012, 3:26:58 AM3/28/12
to Akka User List
Thanks Patrick,

Yes true having a warmup period will not give a significant boost. I
will test this code on other boxes as well just to be sure of the
numbers. I will run Scala examples for ZeroMQ. Akka zeroMQ module
gotta be faster than Scala examples with ZeroMQ. Will keep you
posted :)

Regards,
Meetu Maltiar
www.Knoldus.com
Twitter: meetumaltiar

On Mar 28, 10:44 am, Patrik Nordwall <patrik.nordw...@gmail.com>
wrote:
> On Wed, Mar 28, 2012 at 7:13 AM, Meetu Maltiar <me...@knoldus.com> wrote:
> > Hi There,
>
> > I ran a really simple ZeroMQ one PUB and one SUB on my machine which
> > is i5 processor macbook and am getting low numbers. It is just 3300
> > msgs/sec.
>
> Yes, that is low.
>
>
>
> > Here is the code link
>
> >https://github.com/knoldus/Akka2Bench/blob/master/src/main/scala/com/...
>
> > You should always do a warm-up before starting measurement.
>
> I don't think that will do that big difference that you expect, but it's
> a prerequisite to almost any benchmark.
>
> > and sometimes I am also missing some messages. Please let me know if
> > these are expected numbers or I am missing something here.
>
> Look herehttp://www.zeromq.org/area:results
>
> I don't know what the real problem is. Make sure you have
> correct prerequisites (warm-up period, correct jvm parameters etc) and then
> try to isolate and identify the bottleneck.
>
>
>
>
>
>
>
>
>
>
>
> > Regards,
> > Meetu Maltiar
> >www.knoldus.com
> >www.knoldus.wordpress.com
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Akka User List" group.
> > To post to this group, send email to akka...@googlegroups.com.
> > To unsubscribe from this group, send email to
> > akka-user+...@googlegroups.com.
> > For more options, visit this group at
> >http://groups.google.com/group/akka-user?hl=en.
>
> --
> Patrik Nordwall
> Typesafe <http://typesafe.com/> - The software stack for applications that
> scale
> Twitter: patriknw

Meetu Maltiar

unread,
Mar 28, 2012, 7:38:47 AM3/28/12
to Akka User List
I have written PUB/SUB example in Scala it has one publisher and one
Subscriber. The results are not good at all. On my box it is around
18000 msgs/sec and when I compare it with 3300 msgs/sec I am
disappointed. Here is the Scala code for Publisher and Subscriber

https://github.com/knoldus/Akka2Bench/blob/master/src/main/scala/com/knoldus/akka/bench/zmq/ZMQPub.scala
https://github.com/knoldus/Akka2Bench/blob/master/src/main/scala/com/knoldus/akka/bench/zmq/ZMQSub.scala

There is something wrong somewhere I guess.

Regards,
Meetu Maltiar
www.Knoldus.com
Twitter: meetumaltiar

Jonas Boner

unread,
Mar 28, 2012, 8:01:56 AM3/28/12
to akka...@googlegroups.com
Is it CPU, memory or IO bound? Can you profile to see where the time is spent? 
Jonas Bonér
CTO
Typesafe - The software stack for applications that scale
Phone: +46 733 777 123
Twitter: @jboner

Vikas Hazrati

unread,
Mar 28, 2012, 7:55:30 AM3/28/12
to akka...@googlegroups.com
This has been a known issue for quite some time now. Though there is scala binding available but probably it still needs to be perf tested, see the note here


Also, I had a similar discussion 3 months back when the numbers were not that good either 


At that time the Scala binding version was zeromq-scala-binding" % "0.0.3", now i see you are using 0.0.5 but still the same issues.

Vikas / Knoldus Software

Meetu Maltiar

unread,
Mar 29, 2012, 5:30:35 AM3/29/12
to Akka User List
It is neither CPU bound nor it is IO bound. Here is the snippet on the
subscriber side.
def receive = {
case msg =>
counter = counter + 1
if (counter == 1) startTime = System.currentTimeMillis
if (counter == ZMQApplication.numberOfMessages) {
val elapsedTime = System.currentTimeMillis - startTime
val throughput = (ZMQApplication.numberOfMessages.toDouble *
1000.0) / elapsedTime
println("Elapsed Time millis: " + elapsedTime)
println("Throughput msgs/sec: " + throughput)
}
}
It just increments a counter and measures throughput.
The publisher simply sends message one by one on the socket. Here is
the code snippet:
(1 to numberOfMessages) foreach { x: Int => pubSocket !
ZMQMessage(Seq(Frame("foo.bar"), Frame("hello"))) }

I have the similar code for the benchmarking app written just in scala
without Akka ZeroMQ module. The difference is too large roughly 6x
times the Akka based version. Please do have a look at the code:
https://github.com/knoldus/Akka2Bench/blob/master/src/main/scala/com/knoldus/akka/bench/zmq/ZMQApplication.scala
https://github.com/knoldus/Akka2Bench/blob/master/src/main/scala/com/knoldus/akka/bench/zmq/ZMQPub.scala
https://github.com/knoldus/Akka2Bench/blob/master/src/main/scala/com/knoldus/akka/bench/zmq/ZMQSub.scala

Regards,
Meetu Maltiar
Twitter: @meetumaltiar
www.knoldus.com


On Mar 28, 4:55 pm, Vikas Hazrati <vi...@knoldus.com> wrote:
> This has been a known issue for quite some time now. Though there is scala
> binding available but probably it still needs to be perf tested, see the
> note here
>
> https://github.com/kro/zeromq-scala-binding/issues/6
>
> Also, I had a similar discussion 3 months back when the numbers were not
> that good either
>
> https://groups.google.com/forum/#!topic/scala-user/EKPxFTuWq7o/discus...
>
> At that time the Scala binding version was zeromq-scala-binding" % "0.0.3",
> now i see you are using 0.0.5 but still the same issues.
>
> Vikas / Knoldus Software
>
>
>
>
>
>
>
> On Wed, Mar 28, 2012 at 5:08 PM, Meetu Maltiar <me...@knoldus.com> wrote:
> > I have written PUB/SUB example in Scala it has one publisher and one
> > Subscriber. The results are not good at all. On my box it is around
> > 18000 msgs/sec and when I compare it with 3300 msgs/sec I am
> > disappointed. Here is the Scala code for Publisher and Subscriber
>
> >https://github.com/knoldus/Akka2Bench/blob/master/src/main/scala/com/...
>
> >https://github.com/knoldus/Akka2Bench/blob/master/src/main/scala/com/...

Jonas Boner

unread,
Mar 29, 2012, 6:43:56 AM3/29/12
to akka...@googlegroups.com
Thanks for your report. Could you please file a ticket with all this info and we'll look into it? 

Meetu Maltiar

unread,
Mar 29, 2012, 7:48:29 AM3/29/12
to Akka User List
Thanks,

I created the issue here
http://www.assembla.com/spaces/akka/tickets/1963-akka-zeromq-module-performance-is-way-slower-than-standalone-scala-based-examples

Meanwhile I am looking for a workaround. I am thinking of using Akka
based version of publisher and subscriber. I can override prestart in
them to create publisher socket in Publisher and Subscriber socket in
Subscriber and call recv in a loop.

Meetu Maltiar
Twitter: @meetumaltiar
> >https://github.com/knoldus/Akka2Bench/blob/master/src/main/scala/com/...
>
> >https://github.com/knoldus/Akka2Bench/blob/master/src/main/scala/com/...
>
> >https://github.com/knoldus/Akka2Bench/blob/master/src/main/scala/com/...

Roland Kuhn

unread,
Mar 29, 2012, 8:35:27 AM3/29/12
to akka...@googlegroups.com
Hi Meetu,

On Mar 29, 2012, at 13:48 , Meetu Maltiar wrote:

Thanks, I’m looking into it. Initial findings:

- ZMQApplication ca 4000msg/sec, but that goes up to 6600msg/sec when using 100,000 messages (not that it matters that much)
- ZMQWithoutAkkaExtensionApp ca. 22000msg/sec
- ZMQPub/Sub ca. 27000msg/sec

all on MacBook Pro 15" 2.2GHz, and I did “brew install zeromq” which gave me 2.1.11. I’ve seen that the first one uses twice the #cores for the first half of the time and then drops, which is a marked difference to the second.

> Meanwhile I am looking for a workaround. I am thinking of using Akka
> based version of publisher and subscriber. I can override prestart in
> them to create publisher socket in Publisher and Subscriber socket in
> Subscriber and call recv in a loop.
>

My guess is that trying to avoid that last half sentence is what’s ruining it (not sure at all, just a hunch).

Regards,

Roland

Roland Kuhn
Typesafe – The software stack for applications that scale.
twitter: @rolandkuhn


Meetu Maltiar

unread,
Mar 29, 2012, 10:47:21 AM3/29/12
to Akka User List
Hi Ronald,

Our dev machines are same and the results are similar as well. For
your reference; we had found earlier that scala binding for ZeroMQ was
also slowing our application. Here is the link for your reference
https://github.com/kro/zeromq-scala-binding/issues/6.

Regards,
Meetu Maltiar
Twitter: @meetumaltiar

On Mar 29, 5:35 pm, Roland Kuhn <goo...@rkuhn.info> wrote:
> Hi Meetu,
>
> On Mar 29, 2012, at 13:48 , Meetu Maltiar wrote:
>
> > I created the issue here
> >http://www.assembla.com/spaces/akka/tickets/1963-akka-zeromq-module-p...
> > For more options, visit this group athttp://groups.google.com/group/akka-user?hl=en.

Roland Kuhn

unread,
Mar 29, 2012, 11:09:45 AM3/29/12
to akka...@googlegroups.com
Hi Meetu,

On Mar 29, 2012, at 16:47 , Meetu Maltiar wrote:

> Our dev machines are same and the results are similar as well. For
> your reference; we had found earlier that scala binding for ZeroMQ was
> also slowing our application. Here is the link for your reference
> https://github.com/kro/zeromq-scala-binding/issues/6.
>

Yes, seeing that, I am considering replacing the zeromq-scala-binding dependency with jzmq, will see how big the differences are. And I think I also found at least one reason for slowness, and it is within the implementation of the polling loop (bouncing Futures across different thread pools). I’ll ping you as soon as I have something you can test.

Regards,

Roland

Meetu Maltiar

unread,
Mar 29, 2012, 12:56:16 PM3/29/12
to Akka User List
Thanks Roland :) I think we are nearing solution.

Jonas Boner

unread,
Mar 29, 2012, 1:32:47 PM3/29/12
to akka...@googlegroups.com
Awesome Roland. 

Roland Kuhn

unread,
Mar 30, 2012, 9:29:42 AM3/30/12
to akka...@googlegroups.com
Time for a status update: I’ve found and fixed some issues, most notably when/where/how the polling loop is executed in the Subscriber actor. This brings the test up to 11500msg/sec, but to make the comparison fair these are two-part messages, whereas the other benchmark participants use only single-part messages, which means that my current solution is even slightly faster than the ZMQWithoutAkkaExtensionApp (23000 vs. 22000; although if I switch ZMQApplication to do the single-part exchange, it also drops to 22000).

So, the good news is that the performance problem is fixed, and hopefully also the strange crashes reported (assertion failures in native code), because now everything wrt. one socket really only ever runs on the same thread.

The bad news is that I’m seeing sporadic message losses: if it occurs, it is one message out of the first 1000 which is lost, no others. And it only seems to happen when multi-part messages are exchanged. If anybody with more experience in ØMQ could tell me more about this, I’d be most grateful.

The branch with my current state of the work-in-progress is here: https://github.com/akka/akka/tree/wip-1963-zeromq-%E2%88%82%CF%80

Regards,

Roland

√iktor Ҡlang

unread,
Mar 30, 2012, 9:34:06 AM3/30/12
to akka...@googlegroups.com
Thanks Roland!
Viktor Klang

Akka Tech Lead
Typesafe - The software stack for applications that scale

Twitter: @viktorklang

Jonas Boner

unread,
Mar 30, 2012, 9:49:07 AM3/30/12
to akka...@googlegroups.com
Awesome job Roland. 

Meetu Maltiar

unread,
Mar 30, 2012, 2:13:35 PM3/30/12
to Akka User List

Ivan Porto Carrero

unread,
Mar 30, 2012, 4:32:15 PM3/30/12
to akka...@googlegroups.com
the message that gets lost is that the very first one?
Roland

Awesome Roland. 


>>>>>>>>>> For more options, visit this group at
>>>>>>>>>> http://groups.google.com/group/akka-user?hl=en.
>>
>>>>>>>>> --
>>>>>>>>> Patrik Nordwall
>>>>>>>>> Typesafe <http://typesafe.com/> - The software stack for
>>>>> applications
>>>>>>> that
>>>>>>>>> scale
>>>>>>>>> Twitter: patriknw
>>
>>>>>>> --
>>>>>>> You received this message because you are subscribed to the Google
>>>>> Groups
>>>>>>> "Akka User List" group.
>>>>>>> To post to this group, send email to akka...@googlegroups.com.
>>>>>>> To unsubscribe from this group, send email to

>>>>>>> For more options, visit this group at
>>>>>>> http://groups.google.com/group/akka-user?hl=en.
>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google Groups
>>>>> "Akka User List" group.
>>>>> To post to this group, send email to akka...@googlegroups.com.
>>>>> To unsubscribe from this group, send email to

>>>>> For more options, visit this group at
>>>>> http://groups.google.com/group/akka-user?hl=en.
>>
>>>> --
>>>> Jonas Bonér
>>>> CTO
>>>> Typesafe - The software stack for applications that scale
>>>> Phone: +46 733 777 123
>>>> Twitter: @jboner
>>
>>> --
>>> You received this message because you are subscribed to the Google Groups "Akka User List" group.
>>> To post to this group, send email to akka...@googlegroups.com.
>>> To unsubscribe from this group, send email to akka-user+unsubscribe@googlegroups.com.

>>> For more options, visit this group athttp://groups.google.com/group/akka-user?hl=en.
>>
>> Roland Kuhn
>> Typesafe – The software stack for applications that scale.
>> twitter: @rolandkuhn
>
> --
> You received this message because you are subscribed to the Google Groups "Akka User List" group.
> To post to this group, send email to akka...@googlegroups.com.
> To unsubscribe from this group, send email to akka-user+unsubscribe@googlegroups.com.

> For more options, visit this group at http://groups.google.com/group/akka-user?hl=en.
>

Roland Kuhn
Typesafe – The software stack for applications that scale.
twitter: @rolandkuhn


--
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To post to this group, send email to akka...@googlegroups.com.
To unsubscribe from this group, send email to akka-user+unsubscribe@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/akka-user?hl=en.




--
Jonas Bonér
CTO
Typesafe - The software stack for applications that scale
Phone: +46 733 777 123
Twitter: @jboner


--
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To post to this group, send email to akka...@googlegroups.com.
To unsubscribe from this group, send email to akka-user+unsubscribe@googlegroups.com.
Roland

Awesome Roland. 


>>>>>>>>>> For more options, visit this group at
>>>>>>>>>> http://groups.google.com/group/akka-user?hl=en.
>>
>>>>>>>>> --
>>>>>>>>> Patrik Nordwall
>>>>>>>>> Typesafe <http://typesafe.com/> - The software stack for
>>>>> applications
>>>>>>> that
>>>>>>>>> scale
>>>>>>>>> Twitter: patriknw
>>
>>>>>>> --
>>>>>>> You received this message because you are subscribed to the Google
>>>>> Groups
>>>>>>> "Akka User List" group.
>>>>>>> To post to this group, send email to akka...@googlegroups.com.
>>>>>>> To unsubscribe from this group, send email to

>>>>>>> For more options, visit this group at
>>>>>>> http://groups.google.com/group/akka-user?hl=en.
>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google Groups
>>>>> "Akka User List" group.
>>>>> To post to this group, send email to akka...@googlegroups.com.
>>>>> To unsubscribe from this group, send email to

>>>>> For more options, visit this group at
>>>>> http://groups.google.com/group/akka-user?hl=en.
>>
>>>> --
>>>> Jonas Bonér
>>>> CTO
>>>> Typesafe - The software stack for applications that scale
>>>> Phone: +46 733 777 123
>>>> Twitter: @jboner
>>
>>> --
>>> You received this message because you are subscribed to the Google Groups "Akka User List" group.
>>> To post to this group, send email to akka...@googlegroups.com.
>>> To unsubscribe from this group, send email to akka-user+unsubscribe@googlegroups.com.

>>> For more options, visit this group athttp://groups.google.com/group/akka-user?hl=en.
>>
>> Roland Kuhn
>> Typesafe – The software stack for applications that scale.
>> twitter: @rolandkuhn
>
> --
> You received this message because you are subscribed to the Google Groups "Akka User List" group.
> To post to this group, send email to akka...@googlegroups.com.
> To unsubscribe from this group, send email to akka-user+unsubscribe@googlegroups.com.

> For more options, visit this group at http://groups.google.com/group/akka-user?hl=en.
>

Roland Kuhn
Typesafe – The software stack for applications that scale.
twitter: @rolandkuhn


--
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To post to this group, send email to akka...@googlegroups.com.
To unsubscribe from this group, send email to akka-user+unsubscribe@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/akka-user?hl=en.




--
Jonas Bonér
CTO
Typesafe - The software stack for applications that scale
Phone: +46 733 777 123
Twitter: @jboner


--
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To post to this group, send email to akka...@googlegroups.com.
To unsubscribe from this group, send email to akka-user+unsubscribe@googlegroups.com.
Roland

Awesome Roland. 


>>>>>>>>>> For more options, visit this group at
>>>>>>>>>> http://groups.google.com/group/akka-user?hl=en.
>>
>>>>>>>>> --
>>>>>>>>> Patrik Nordwall
>>>>>>>>> Typesafe <http://typesafe.com/> - The software stack for
>>>>> applications
>>>>>>> that
>>>>>>>>> scale
>>>>>>>>> Twitter: patriknw
>>
>>>>>>> --
>>>>>>> You received this message because you are subscribed to the Google
>>>>> Groups
>>>>>>> "Akka User List" group.
>>>>>>> To post to this group, send email to akka...@googlegroups.com.
>>>>>>> To unsubscribe from this group, send email to

>>>>>>> For more options, visit this group at
>>>>>>> http://groups.google.com/group/akka-user?hl=en.
>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google Groups
>>>>> "Akka User List" group.
>>>>> To post to this group, send email to akka...@googlegroups.com.
>>>>> To unsubscribe from this group, send email to

>>>>> For more options, visit this group at
>>>>> http://groups.google.com/group/akka-user?hl=en.
>>
>>>> --
>>>> Jonas Bonér
>>>> CTO
>>>> Typesafe - The software stack for applications that scale
>>>> Phone: +46 733 777 123
>>>> Twitter: @jboner
>>
>>> --
>>> You received this message because you are subscribed to the Google Groups "Akka User List" group.
>>> To post to this group, send email to akka...@googlegroups.com.
>>> To unsubscribe from this group, send email to akka-user+unsubscribe@googlegroups.com.

>>> For more options, visit this group athttp://groups.google.com/group/akka-user?hl=en.
>>
>> Roland Kuhn
>> Typesafe – The software stack for applications that scale.
>> twitter: @rolandkuhn
>
> --
> You received this message because you are subscribed to the Google Groups "Akka User List" group.
> To post to this group, send email to akka...@googlegroups.com.
> To unsubscribe from this group, send email to akka-user+unsubscribe@googlegroups.com.

> For more options, visit this group at http://groups.google.com/group/akka-user?hl=en.
>

Roland Kuhn
Typesafe – The software stack for applications that scale.
twitter: @rolandkuhn


--
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To post to this group, send email to akka...@googlegroups.com.
To unsubscribe from this group, send email to akka-user+unsubscribe@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/akka-user?hl=en.




--
Jonas Bonér
CTO
Typesafe - The software stack for applications that scale
Phone: +46 733 777 123
Twitter: @jboner


--
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To post to this group, send email to akka...@googlegroups.com.
To unsubscribe from this group, send email to akka-user+unsubscribe@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/akka-user?hl=en.

Roland Kuhn

unread,
Mar 30, 2012, 4:44:05 PM3/30/12
to akka...@googlegroups.com
no, unfortunately not: in the case I logged it lost the 227th message, so I assume it has nothing to do with slow/too late subscriber.

Regards,

Roland

To view this discussion on the web visit https://groups.google.com/d/msg/akka-user/-/pQP7OXPySdgJ.

To post to this group, send email to akka...@googlegroups.com.
To unsubscribe from this group, send email to akka-user+...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/akka-user?hl=en.
Message has been deleted

beagledude

unread,
Jul 11, 2012, 11:24:12 PM7/11/12
to akka...@googlegroups.com
was there a release this was fixed in that's publicly available? I'm seeing poor zmq performance with dealer/router sockets as well. 
VisualVM shows almost all the time spent in zmq_poll

screen shot attached

Screen Shot 2012-07-11 at 8.23.42 PM.png

√iktor Ҡlang

unread,
Jul 12, 2012, 6:19:11 AM7/12/12
to akka...@googlegroups.com

Is there a version number difference between the version you are using and the latest version?

Have you benched against master and/or the release-2.0 branch?

Cheers,
V

To view this discussion on the web visit https://groups.google.com/d/msg/akka-user/-/NQb5OBMEOAkJ.
Reply all
Reply to author
Forward
0 new messages