Has anybody did Performance test PUB/SUB on ZeroMQ its slow for me

[Meetu Maltiar]

Hi There,

I ran a really simple ZeroMQ one PUB and one SUB on my machine which
is i5 processor macbook and am getting low numbers. It is just 3300
msgs/sec.

Here is the code link
https://github.com/knoldus/Akka2Bench/blob/master/src/main/scala/com/knoldus/akka/bench/zmq/ZMQApplication.scala

and sometimes I am also missing some messages. Please let me know if
these are expected numbers or I am missing something here.


Regards,
Meetu Maltiar
www.knoldus.com
www.knoldus.wordpress.com

[Patrik Nordwall]

On Wed, Mar 28, 2012 at 7:13 AM, Meetu Maltiar <me...@knoldus.com> wrote:

Hi There,

I ran a really simple ZeroMQ one PUB and one SUB on my machine which
is i5 processor macbook and am getting low numbers. It is just 3300
msgs/sec.

Yes, that is low. 

Here is the code link
https://github.com/knoldus/Akka2Bench/blob/master/src/main/scala/com/knoldus/akka/bench/zmq/ZMQApplication.scala

You should always do a warm-up before starting measurement.

I don't think that will do that big difference that you expect, but it's a prerequisite to almost any benchmark.

 

and sometimes I am also missing some messages. Please let me know if
these are expected numbers or I am missing something here.

Look here http://www.zeromq.org/area:results

I don't know what the real problem is. Make sure you have correct prerequisites (warm-up period, correct jvm parameters etc) and then try to isolate and identify the bottleneck.

Regards,
Meetu Maltiar
www.knoldus.com
www.knoldus.wordpress.com

--
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To post to this group, send email to akka...@googlegroups.com.
To unsubscribe from this group, send email to akka-user+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/akka-user?hl=en.

--

Patrik Nordwall

Typesafe - The software stack for applications that scale

Twitter: patriknw

[Meetu Maltiar]

Thanks Patrick,

Yes true having a warmup period will not give a significant boost. I
will test this code on other boxes as well just to be sure of the
numbers. I will run Scala examples for ZeroMQ. Akka zeroMQ module
gotta be faster than Scala examples with ZeroMQ. Will keep you
posted :)

Regards,
Meetu Maltiar
www.Knoldus.com
Twitter: meetumaltiar

On Mar 28, 10:44 am, Patrik Nordwall <patrik.nordw...@gmail.com>
wrote:

> On Wed, Mar 28, 2012 at 7:13 AM, Meetu Maltiar <me...@knoldus.com> wrote:
> > Hi There,
>
> > I ran a really simple ZeroMQ one PUB and one SUB on my machine which
> > is i5 processor macbook and am getting low numbers. It is just 3300
> > msgs/sec.
>
> Yes, that is low.
>
>
>
> > Here is the code link
>

> >https://github.com/knoldus/Akka2Bench/blob/master/src/main/scala/com/...

>
> > You should always do a warm-up before starting measurement.
>
> I don't think that will do that big difference that you expect, but it's
> a prerequisite to almost any benchmark.
>
> > and sometimes I am also missing some messages. Please let me know if
> > these are expected numbers or I am missing something here.
>

> Look herehttp://www.zeromq.org/area:results

>
> I don't know what the real problem is. Make sure you have
> correct prerequisites (warm-up period, correct jvm parameters etc) and then
> try to isolate and identify the bottleneck.
>
>
>
>
>
>
>
>
>
>
>
> > Regards,
> > Meetu Maltiar
> >www.knoldus.com
> >www.knoldus.wordpress.com
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Akka User List" group.
> > To post to this group, send email to akka...@googlegroups.com.
> > To unsubscribe from this group, send email to
> > akka-user+...@googlegroups.com.
> > For more options, visit this group at
> >http://groups.google.com/group/akka-user?hl=en.
>
> --
> Patrik Nordwall

> Typesafe <http://typesafe.com/> - The software stack for applications that
> scale
> Twitter: patriknw

[Meetu Maltiar]

I have written PUB/SUB example in Scala it has one publisher and one
Subscriber. The results are not good at all. On my box it is around
18000 msgs/sec and when I compare it with 3300 msgs/sec I am
disappointed. Here is the Scala code for Publisher and Subscriber

https://github.com/knoldus/Akka2Bench/blob/master/src/main/scala/com/knoldus/akka/bench/zmq/ZMQPub.scala
https://github.com/knoldus/Akka2Bench/blob/master/src/main/scala/com/knoldus/akka/bench/zmq/ZMQSub.scala

There is something wrong somewhere I guess.

Regards,
Meetu Maltiar
www.Knoldus.com
Twitter: meetumaltiar

[Jonas Boner]

Is it CPU, memory or IO bound? Can you profile to see where the time is spent? 

Jonas Bonér

CTO
Typesafe - The software stack for applications that scale
Phone: +46 733 777 123
Twitter: @jboner

[Vikas Hazrati]

This has been a known issue for quite some time now. Though there is scala binding available but probably it still needs to be perf tested, see the note here

https://github.com/kro/zeromq-scala-binding/issues/6

Also, I had a similar discussion 3 months back when the numbers were not that good either 

https://groups.google.com/forum/#!topic/scala-user/EKPxFTuWq7o/discussion

At that time the Scala binding version was zeromq-scala-binding" % "0.0.3", now i see you are using 0.0.5 but still the same issues.

Vikas / Knoldus Software

[Meetu Maltiar]

It is neither CPU bound nor it is IO bound. Here is the snippet on the
subscriber side.
def receive = {
case msg =>
counter = counter + 1
if (counter == 1) startTime = System.currentTimeMillis
if (counter == ZMQApplication.numberOfMessages) {
val elapsedTime = System.currentTimeMillis - startTime
val throughput = (ZMQApplication.numberOfMessages.toDouble *
1000.0) / elapsedTime
println("Elapsed Time millis: " + elapsedTime)
println("Throughput msgs/sec: " + throughput)
}
}
It just increments a counter and measures throughput.
The publisher simply sends message one by one on the socket. Here is
the code snippet:
(1 to numberOfMessages) foreach { x: Int => pubSocket !
ZMQMessage(Seq(Frame("foo.bar"), Frame("hello"))) }

I have the similar code for the benchmarking app written just in scala
without Akka ZeroMQ module. The difference is too large roughly 6x
times the Akka based version. Please do have a look at the code:
https://github.com/knoldus/Akka2Bench/blob/master/src/main/scala/com/knoldus/akka/bench/zmq/ZMQApplication.scala
https://github.com/knoldus/Akka2Bench/blob/master/src/main/scala/com/knoldus/akka/bench/zmq/ZMQPub.scala
https://github.com/knoldus/Akka2Bench/blob/master/src/main/scala/com/knoldus/akka/bench/zmq/ZMQSub.scala

Regards,
Meetu Maltiar
Twitter: @meetumaltiar
www.knoldus.com

On Mar 28, 4:55 pm, Vikas Hazrati <vi...@knoldus.com> wrote:
> This has been a known issue for quite some time now. Though there is scala
> binding available but probably it still needs to be perf tested, see the
> note here
>
> https://github.com/kro/zeromq-scala-binding/issues/6
>
> Also, I had a similar discussion 3 months back when the numbers were not
> that good either
>

> https://groups.google.com/forum/#!topic/scala-user/EKPxFTuWq7o/discus...

>
> At that time the Scala binding version was zeromq-scala-binding" % "0.0.3",
> now i see you are using 0.0.5 but still the same issues.
>
> Vikas / Knoldus Software
>
>
>
>
>
>
>
> On Wed, Mar 28, 2012 at 5:08 PM, Meetu Maltiar <me...@knoldus.com> wrote:
> > I have written PUB/SUB example in Scala it has one publisher and one
> > Subscriber. The results are not good at all. On my box it is around
> > 18000 msgs/sec and when I compare it with 3300 msgs/sec I am
> > disappointed. Here is the Scala code for Publisher and Subscriber
>

> >https://github.com/knoldus/Akka2Bench/blob/master/src/main/scala/com/...
>
> >https://github.com/knoldus/Akka2Bench/blob/master/src/main/scala/com/...

[Jonas Boner]

Thanks for your report. Could you please file a ticket with all this info and we'll look into it? 

http://doc.akka.io/docs/akka/snapshot/project/issue-tracking.html

[Meetu Maltiar]

Thanks,

I created the issue here
http://www.assembla.com/spaces/akka/tickets/1963-akka-zeromq-module-performance-is-way-slower-than-standalone-scala-based-examples

Meanwhile I am looking for a workaround. I am thinking of using Akka
based version of publisher and subscriber. I can override prestart in
them to create publisher socket in Publisher and Subscriber socket in
Subscriber and call recv in a loop.

Meetu Maltiar
Twitter: @meetumaltiar

> >https://github.com/knoldus/Akka2Bench/blob/master/src/main/scala/com/...
>
> >https://github.com/knoldus/Akka2Bench/blob/master/src/main/scala/com/...
>
> >https://github.com/knoldus/Akka2Bench/blob/master/src/main/scala/com/...

[Roland Kuhn]

Hi Meetu,

On Mar 29, 2012, at 13:48 , Meetu Maltiar wrote:

> I created the issue here
> http://www.assembla.com/spaces/akka/tickets/1963-akka-zeromq-module-performance-is-way-slower-than-standalone-scala-based-examples
>

Thanks, I’m looking into it. Initial findings:

- ZMQApplication ca 4000msg/sec, but that goes up to 6600msg/sec when using 100,000 messages (not that it matters that much)
- ZMQWithoutAkkaExtensionApp ca. 22000msg/sec
- ZMQPub/Sub ca. 27000msg/sec

all on MacBook Pro 15" 2.2GHz, and I did “brew install zeromq” which gave me 2.1.11. I’ve seen that the first one uses twice the #cores for the first half of the time and then drops, which is a marked difference to the second.

> Meanwhile I am looking for a workaround. I am thinking of using Akka
> based version of publisher and subscriber. I can override prestart in
> them to create publisher socket in Publisher and Subscriber socket in
> Subscriber and call recv in a loop.
>

My guess is that trying to avoid that last half sentence is what’s ruining it (not sure at all, just a hunch).

Regards,

Roland

Roland Kuhn
Typesafe – The software stack for applications that scale.
twitter: @rolandkuhn

[Meetu Maltiar]

Hi Ronald,

Our dev machines are same and the results are similar as well. For
your reference; we had found earlier that scala binding for ZeroMQ was
also slowing our application. Here is the link for your reference
https://github.com/kro/zeromq-scala-binding/issues/6.

Regards,
Meetu Maltiar
Twitter: @meetumaltiar

On Mar 29, 5:35 pm, Roland Kuhn <goo...@rkuhn.info> wrote:
> Hi Meetu,
>
> On Mar 29, 2012, at 13:48 , Meetu Maltiar wrote:
>
> > I created the issue here

> >http://www.assembla.com/spaces/akka/tickets/1963-akka-zeromq-module-p...

> > For more options, visit this group athttp://groups.google.com/group/akka-user?hl=en.

[Roland Kuhn]

Hi Meetu,

On Mar 29, 2012, at 16:47 , Meetu Maltiar wrote:

> Our dev machines are same and the results are similar as well. For
> your reference; we had found earlier that scala binding for ZeroMQ was
> also slowing our application. Here is the link for your reference
> https://github.com/kro/zeromq-scala-binding/issues/6.
>

Yes, seeing that, I am considering replacing the zeromq-scala-binding dependency with jzmq, will see how big the differences are. And I think I also found at least one reason for slowness, and it is within the implementation of the polling loop (bouncing Futures across different thread pools). I’ll ping you as soon as I have something you can test.

Regards,

Roland

[Meetu Maltiar]

Thanks Roland :) I think we are nearing solution.

[Jonas Boner]

Awesome Roland. 

[Roland Kuhn]

Time for a status update: I’ve found and fixed some issues, most notably when/where/how the polling loop is executed in the Subscriber actor. This brings the test up to 11500msg/sec, but to make the comparison fair these are two-part messages, whereas the other benchmark participants use only single-part messages, which means that my current solution is even slightly faster than the ZMQWithoutAkkaExtensionApp (23000 vs. 22000; although if I switch ZMQApplication to do the single-part exchange, it also drops to 22000).

So, the good news is that the performance problem is fixed, and hopefully also the strange crashes reported (assertion failures in native code), because now everything wrt. one socket really only ever runs on the same thread.

The bad news is that I’m seeing sporadic message losses: if it occurs, it is one message out of the first 1000 which is lost, no others. And it only seems to happen when multi-part messages are exchanged. If anybody with more experience in ØMQ could tell me more about this, I’d be most grateful.

The branch with my current state of the work-in-progress is here: https://github.com/akka/akka/tree/wip-1963-zeromq-%E2%88%82%CF%80

Regards,

Roland

[√iktor Ҡlang]

Thanks Roland!

Viktor Klang

Akka Tech Lead

Typesafe - The software stack for applications that scale

Twitter: @viktorklang

[Jonas Boner]

Awesome job Roland. 

[Meetu Maltiar]

Thanks Roland. Great job by Typesafe and that too so quickly. I also
noticed loss of messages sometimes but I do not know the reason why
this is happening.

Regards,
Meetu Maltiar

> >>https://github.com/knoldus/Akka2Bench/blob/master/src/main/scala/com/.<https://github.com/knoldus/Akka2Bench/blob/master/src/main/scala/com/>
> >> ..
>
> >>https://github.com/knoldus/Akka2Bench/blob/master/src/main/scala/com/.<https://github.com/knoldus/Akka2Bench/blob/master/src/main/scala/com/>
> >> ..
>
> >>https://github.com/knoldus/Akka2Bench/blob/master/src/main/scala/com/.<https://github.com/knoldus/Akka2Bench/blob/master/src/main/scala/com/>
> >> ..

> ...
>
> read more »

[Ivan Porto Carrero]

the message that gets lost is that the very first one?

Roland

Awesome Roland. 

>>>>>>>>>> akka-user+unsubscribe@googlegroups.com.

>>>>>>>>>> For more options, visit this group at
>>>>>>>>>> http://groups.google.com/group/akka-user?hl=en.
>>
>>>>>>>>> --
>>>>>>>>> Patrik Nordwall
>>>>>>>>> Typesafe <http://typesafe.com/> - The software stack for
>>>>> applications
>>>>>>> that
>>>>>>>>> scale
>>>>>>>>> Twitter: patriknw
>>
>>>>>>> --
>>>>>>> You received this message because you are subscribed to the Google
>>>>> Groups
>>>>>>> "Akka User List" group.
>>>>>>> To post to this group, send email to akka...@googlegroups.com.
>>>>>>> To unsubscribe from this group, send email to

>>>>>>> akka-user+unsubscribe@googlegroups.com.

>>>>>>> For more options, visit this group at
>>>>>>> http://groups.google.com/group/akka-user?hl=en.
>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google Groups
>>>>> "Akka User List" group.
>>>>> To post to this group, send email to akka...@googlegroups.com.
>>>>> To unsubscribe from this group, send email to

>>>>> akka-user+unsubscribe@googlegroups.com.

>>>>> For more options, visit this group at
>>>>> http://groups.google.com/group/akka-user?hl=en.
>>
>>>> --
>>>> Jonas Bonér
>>>> CTO
>>>> Typesafe - The software stack for applications that scale
>>>> Phone: +46 733 777 123
>>>> Twitter: @jboner
>>
>>> --
>>> You received this message because you are subscribed to the Google Groups "Akka User List" group.
>>> To post to this group, send email to akka...@googlegroups.com.

>>> To unsubscribe from this group, send email to akka-user+unsubscribe@googlegroups.com.

>>> For more options, visit this group athttp://groups.google.com/group/akka-user?hl=en.
>>
>> Roland Kuhn
>> Typesafe – The software stack for applications that scale.
>> twitter: @rolandkuhn
>
> --
> You received this message because you are subscribed to the Google Groups "Akka User List" group.
> To post to this group, send email to akka...@googlegroups.com.

> To unsubscribe from this group, send email to akka-user+unsubscribe@googlegroups.com.

> For more options, visit this group at http://groups.google.com/group/akka-user?hl=en.
>

Roland Kuhn
Typesafe – The software stack for applications that scale.
twitter: @rolandkuhn


--
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To post to this group, send email to akka...@googlegroups.com.

To unsubscribe from this group, send email to akka-user+unsubscribe@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/akka-user?hl=en.

--

Jonas Bonér

CTO
Typesafe - The software stack for applications that scale
Phone: +46 733 777 123
Twitter: @jboner

--
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To post to this group, send email to akka...@googlegroups.com.

To unsubscribe from this group, send email to akka-user+unsubscribe@googlegroups.com.

Roland

Awesome Roland. 

>>>>>>>>>> akka-user+unsubscribe@googlegroups.com.

>>>>>>>>>> For more options, visit this group at
>>>>>>>>>> http://groups.google.com/group/akka-user?hl=en.
>>
>>>>>>>>> --
>>>>>>>>> Patrik Nordwall
>>>>>>>>> Typesafe <http://typesafe.com/> - The software stack for
>>>>> applications
>>>>>>> that
>>>>>>>>> scale
>>>>>>>>> Twitter: patriknw
>>
>>>>>>> --
>>>>>>> You received this message because you are subscribed to the Google
>>>>> Groups
>>>>>>> "Akka User List" group.
>>>>>>> To post to this group, send email to akka...@googlegroups.com.
>>>>>>> To unsubscribe from this group, send email to

>>>>>>> akka-user+unsubscribe@googlegroups.com.

>>>>>>> For more options, visit this group at
>>>>>>> http://groups.google.com/group/akka-user?hl=en.
>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google Groups
>>>>> "Akka User List" group.
>>>>> To post to this group, send email to akka...@googlegroups.com.
>>>>> To unsubscribe from this group, send email to

>>>>> akka-user+unsubscribe@googlegroups.com.

>>>>> For more options, visit this group at
>>>>> http://groups.google.com/group/akka-user?hl=en.
>>
>>>> --
>>>> Jonas Bonér
>>>> CTO
>>>> Typesafe - The software stack for applications that scale
>>>> Phone: +46 733 777 123
>>>> Twitter: @jboner
>>
>>> --
>>> You received this message because you are subscribed to the Google Groups "Akka User List" group.
>>> To post to this group, send email to akka...@googlegroups.com.

>>> To unsubscribe from this group, send email to akka-user+unsubscribe@googlegroups.com.

>>> For more options, visit this group athttp://groups.google.com/group/akka-user?hl=en.
>>
>> Roland Kuhn
>> Typesafe – The software stack for applications that scale.
>> twitter: @rolandkuhn
>
> --
> You received this message because you are subscribed to the Google Groups "Akka User List" group.
> To post to this group, send email to akka...@googlegroups.com.

> To unsubscribe from this group, send email to akka-user+unsubscribe@googlegroups.com.

> For more options, visit this group at http://groups.google.com/group/akka-user?hl=en.
>

Roland Kuhn
Typesafe – The software stack for applications that scale.
twitter: @rolandkuhn


--
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To post to this group, send email to akka...@googlegroups.com.

To unsubscribe from this group, send email to akka-user+unsubscribe@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/akka-user?hl=en.

--

Jonas Bonér

CTO
Typesafe - The software stack for applications that scale
Phone: +46 733 777 123
Twitter: @jboner

--
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To post to this group, send email to akka...@googlegroups.com.

To unsubscribe from this group, send email to akka-user+unsubscribe@googlegroups.com.

Roland

Awesome Roland. 

>>>>>>>>>> akka-user+unsubscribe@googlegroups.com.

>>>>>>>>>> For more options, visit this group at
>>>>>>>>>> http://groups.google.com/group/akka-user?hl=en.
>>
>>>>>>>>> --
>>>>>>>>> Patrik Nordwall
>>>>>>>>> Typesafe <http://typesafe.com/> - The software stack for
>>>>> applications
>>>>>>> that
>>>>>>>>> scale
>>>>>>>>> Twitter: patriknw
>>
>>>>>>> --
>>>>>>> You received this message because you are subscribed to the Google
>>>>> Groups
>>>>>>> "Akka User List" group.
>>>>>>> To post to this group, send email to akka...@googlegroups.com.
>>>>>>> To unsubscribe from this group, send email to

>>>>>>> akka-user+unsubscribe@googlegroups.com.

>>>>>>> For more options, visit this group at
>>>>>>> http://groups.google.com/group/akka-user?hl=en.
>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google Groups
>>>>> "Akka User List" group.
>>>>> To post to this group, send email to akka...@googlegroups.com.
>>>>> To unsubscribe from this group, send email to

>>>>> akka-user+unsubscribe@googlegroups.com.

>>>>> For more options, visit this group at
>>>>> http://groups.google.com/group/akka-user?hl=en.
>>
>>>> --
>>>> Jonas Bonér
>>>> CTO
>>>> Typesafe - The software stack for applications that scale
>>>> Phone: +46 733 777 123
>>>> Twitter: @jboner
>>
>>> --
>>> You received this message because you are subscribed to the Google Groups "Akka User List" group.
>>> To post to this group, send email to akka...@googlegroups.com.

>>> To unsubscribe from this group, send email to akka-user+unsubscribe@googlegroups.com.

>>> For more options, visit this group athttp://groups.google.com/group/akka-user?hl=en.
>>
>> Roland Kuhn
>> Typesafe – The software stack for applications that scale.
>> twitter: @rolandkuhn
>
> --
> You received this message because you are subscribed to the Google Groups "Akka User List" group.
> To post to this group, send email to akka...@googlegroups.com.

> To unsubscribe from this group, send email to akka-user+unsubscribe@googlegroups.com.

> For more options, visit this group at http://groups.google.com/group/akka-user?hl=en.
>

Roland Kuhn
Typesafe – The software stack for applications that scale.
twitter: @rolandkuhn


--
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To post to this group, send email to akka...@googlegroups.com.

To unsubscribe from this group, send email to akka-user+unsubscribe@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/akka-user?hl=en.

--

Jonas Bonér

CTO
Typesafe - The software stack for applications that scale
Phone: +46 733 777 123
Twitter: @jboner

--
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To post to this group, send email to akka...@googlegroups.com.

To unsubscribe from this group, send email to akka-user+unsubscribe@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/akka-user?hl=en.

[Roland Kuhn]

no, unfortunately not: in the case I logged it lost the 227th message, so I assume it has nothing to do with slow/too late subscriber.

Regards,

Roland

To view this discussion on the web visit https://groups.google.com/d/msg/akka-user/-/pQP7OXPySdgJ.

To post to this group, send email to akka...@googlegroups.com.

To unsubscribe from this group, send email to akka-user+...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/akka-user?hl=en.

[beagledude]

was there a release this was fixed in that's publicly available? I'm seeing poor zmq performance with dealer/router sockets as well. 

VisualVM shows almost all the time spent in zmq_poll

screen shot attached

[√iktor Ҡlang]

Is there a version number difference between the version you are using and the latest version?

Have you benched against master and/or the release-2.0 branch?

Cheers,
V

To view this discussion on the web visit https://groups.google.com/d/msg/akka-user/-/NQb5OBMEOAkJ.