perfomance / profiling akka

[Gonzalo Vasquez]

I switched from a round robin strategy to a request-work strategy for remote load balancing. Upon switching, performance decreased almost to half speed. Trying to find out what could be wrong I started JVisualVM on the remote nodes to perhaps figure out the root cause, but upon that I just found more puzzling data, with no answer from me.

Please see attached image, which I'll describe now:

- "AltiuzReportsBatchWorker" is the system name, thus it's dispatcher threads are named AltiuzReportsBatchWorker-akka-actor.defatult-dispatcher-1,AltiuzReportsBatchWorker-akka-actor.defatult-dispatcher-2, AltiuzReportsBatchWorker-akka-actor.defatult-dispatcher-3 and AltiuzReportsBatchWorker-akka-actor.defatult-dispatcher-4 (4 threads as I have 4 cores on that machine).

- jvisualvm shows almost 50/50 % use of CPU time of that thread divided between my "job class" (actor) and the Mailbox's run method, which in turn invokes the onreceive method of the actor, so it seems to me that somehow work is getting done twice, but as you can see one part is "ahead" of the other with 33 invocations v/s 27....any hints on how to dig deeper on this matter?

Regards,

Gonzalo Vásquez Sáez

Gerente Investigación y Desarrollo (R&D)
Altiuz Soluciones Tecnológicas de Negocios Ltda.
Av. Nueva Tajamar 555 Of. 802, Las Condes
(56-2) 335 2461
gvasquez@altiuz.cl

http://www.altiuz.cl

 

[√iktor Ҡlang]

Do you request one item of work at a time? If so, ask for batches.

Cheers,

√

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://akka.io/faq/
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To post to this group, send email to akka...@googlegroups.com.
To unsubscribe from this group, send email to akka-user+...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user?hl=en.
 
 

--
Viktor Klang

Akka Tech Lead

Typesafe - The software stack for applications that scale

Twitter: @viktorklang

[Gonzalo Vasquez]

No, I've already batched the process, using "packets" of requests to minimize network traffic.-

Gonzalo Vásquez Sáez

Gerente Investigación y Desarrollo (R&D)
Altiuz Soluciones Tecnológicas de Negocios Ltda.
Av. Nueva Tajamar 555 Of. 802, Las Condes
(56-2) 335 2461
gvasquez@altiuz.cl

http://www.altiuz.cl

 

El 16-08-2012, a las 18:10, √iktor Ҡlang <viktor...@gmail.com> escribió:

Do you request one item of work at a time? If so, ask for batches.

Cheers,

√

On Thu, Aug 16, 2012 at 11:57 PM, Gonzalo Vasquez <gvas...@altiuz.cl> wrote:

I switched from a round robin strategy to a request-work strategy for remote load balancing. Upon switching, performance decreased almost to half speed. Trying to find out what could be wrong I started JVisualVM on the remote nodes to perhaps figure out the root cause, but upon that I just found more puzzling data, with no answer from me.

Please see attached image, which I'll describe now:

- "AltiuzReportsBatchWorker" is the system name, thus it's dispatcher threads are named AltiuzReportsBatchWorker-akka-actor.defatult-dispatcher-1,AltiuzReportsBatchWorker-akka-actor.defatult-dispatcher-2, AltiuzReportsBatchWorker-akka-actor.defatult-dispatcher-3 and AltiuzReportsBatchWorker-akka-actor.defatult-dispatcher-4 (4 threads as I have 4 cores on that machine).

- jvisualvm shows almost 50/50 % use of CPU time of that thread divided between my "job class" (actor) and the Mailbox's run method, which in turn invokes the onreceive method of the actor, so it seems to me that somehow work is getting done twice, but as you can see one part is "ahead" of the other with 33 invocations v/s 27....any hints on how to dig deeper on this matter?

Regards,

Gonzalo Vásquez Sáez

Gerente Investigación y Desarrollo (R&D)
Altiuz Soluciones Tecnológicas de Negocios Ltda.
Av. Nueva Tajamar 555 Of. 802, Las Condes
(56-2) 335 2461
gvasquez@altiuz.cl

http://www.altiuz.cl

 

<Firma_medio ambiente_mail.jpg>

<akka-profiling.png>

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://akka.io/faq/
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To post to this group, send email to akka...@googlegroups.com.
To unsubscribe from this group, send email to akka-user+...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user?hl=en.
 
 

--
Viktor Klang

Akka Tech Lead

Typesafe - The software stack for applications that scale

Twitter: @viktorklang

[Viktor Klang]

Hi,

Will be nigh impossible to come up with any answers without the code.

Cheers,

√

On Thu, Aug 16, 2012 at 11:57 PM, Gonzalo Vasquez <gvas...@altiuz.cl> wrote:

--

[Gonzalo Vasquez]

Which pieces of code do I send? Ask for and I'll send them!

Gonzalo Vásquez Sáez

Gerente Investigación y Desarrollo (R&D)
Altiuz Soluciones Tecnológicas de Negocios Ltda.
Av. Nueva Tajamar 555 Of. 802, Las Condes
(56-2) 335 2461
gvasquez@altiuz.cl

http://www.altiuz.cl

 

El 16-08-2012, a las 18:25, Viktor Klang <viktor...@gmail.com> escribió:

Hi,

Will be nigh impossible to come up with any answers without the code.

Cheers,

√

On Thu, Aug 16, 2012 at 11:57 PM, Gonzalo Vasquez <gvas...@altiuz.cl> wrote:

I switched from a round robin strategy to a request-work strategy for remote load balancing. Upon switching, performance decreased almost to half speed. Trying to find out what could be wrong I started JVisualVM on the remote nodes to perhaps figure out the root cause, but upon that I just found more puzzling data, with no answer from me.

Please see attached image, which I'll describe now:

- "AltiuzReportsBatchWorker" is the system name, thus it's dispatcher threads are named AltiuzReportsBatchWorker-akka-actor.defatult-dispatcher-1,AltiuzReportsBatchWorker-akka-actor.defatult-dispatcher-2, AltiuzReportsBatchWorker-akka-actor.defatult-dispatcher-3 and AltiuzReportsBatchWorker-akka-actor.defatult-dispatcher-4 (4 threads as I have 4 cores on that machine).

- jvisualvm shows almost 50/50 % use of CPU time of that thread divided between my "job class" (actor) and the Mailbox's run method, which in turn invokes the onreceive method of the actor, so it seems to me that somehow work is getting done twice, but as you can see one part is "ahead" of the other with 33 invocations v/s 27....any hints on how to dig deeper on this matter?

Regards,

Gonzalo Vásquez Sáez

Gerente Investigación y Desarrollo (R&D)
Altiuz Soluciones Tecnológicas de Negocios Ltda.
Av. Nueva Tajamar 555 Of. 802, Las Condes
(56-2) 335 2461
gvasquez@altiuz.cl

http://www.altiuz.cl

 

<Firma_medio ambiente_mail.jpg>

<akka-profiling.png>

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://akka.io/faq/
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To post to this group, send email to akka...@googlegroups.com.
To unsubscribe from this group, send email to akka-user+...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user?hl=en.
 
 

[Viktor Klang]

What pieces do you have?

[Gonzalo Vasquez]

Victor,

I have three different parts: master, worker and messages. I've included the 3 projects in a single zip, along with their pom.xml maven descriptors

Both worker and master parts have Main-classes as defined in their pom files, as this a sample approach and the final arquitecture will consider webservice based job triggering instead of running a master main-class. The worker is supposed to be run as a daemon on the remote servers.

My project's logic is in separate java-projects, so it's not included in here, only the akka-related parts.

[√iktor Ҡlang]

Hi Gonzalo,

I will unfortunately not have time to have a look at this, I'm leaving shortly and will be off fo about 2 weeks.

Would be great if someone else could offer to help you out.

Thanks,

√

--
>>>>>>>>>>      Read the docs: http://akka.io/docs/
>>>>>>>>>>      Check the FAQ: http://akka.io/faq/
>>>>>>>>>>      Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To post to this group, send email to akka...@googlegroups.com.
To unsubscribe from this group, send email to akka-user+...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user?hl=en.

El 16-08-2012, a las 18:37, Viktor Klang <viktor...@gmail.com> escribió:

What pieces do you have?

On Fri, Aug 17, 2012 at 12:33 AM, Gonzalo Vasquez <gvas...@altiuz.cl> wrote:

Which pieces of code do I send? Ask for and I'll send them!

Gonzalo Vásquez Sáez

Gerente Investigación y Desarrollo (R&D)
Altiuz Soluciones Tecnológicas de Negocios Ltda.
Av. Nueva Tajamar 555 Of. 802, Las Condes
(56-2) 335 2461
gvasquez@altiuz.cl

http://www.altiuz.cl

 

<Firma_medio ambiente_mail.jpg>

El 16-08-2012, a las 18:25, Viktor Klang <viktor...@gmail.com> escribió:

Hi,

Will be nigh impossible to come up with any answers without the code.

Cheers,

√

On Thu, Aug 16, 2012 at 11:57 PM, Gonzalo Vasquez <gvas...@altiuz.cl> wrote:

I switched from a round robin strategy to a request-work strategy for remote load balancing. Upon switching, performance decreased almost to half speed. Trying to find out what could be wrong I started JVisualVM on the remote nodes to perhaps figure out the root cause, but upon that I just found more puzzling data, with no answer from me.

Please see attached image, which I'll describe now:

- "AltiuzReportsBatchWorker" is the system name, thus it's dispatcher threads are named AltiuzReportsBatchWorker-akka-actor.defatult-dispatcher-1,AltiuzReportsBatchWorker-akka-actor.defatult-dispatcher-2, AltiuzReportsBatchWorker-akka-actor.defatult-dispatcher-3 and AltiuzReportsBatchWorker-akka-actor.defatult-dispatcher-4 (4 threads as I have 4 cores on that machine).

- jvisualvm shows almost 50/50 % use of CPU time of that thread divided between my "job class" (actor) and the Mailbox's run method, which in turn invokes the onreceive method of the actor, so it seems to me that somehow work is getting done twice, but as you can see one part is "ahead" of the other with 33 invocations v/s 27....any hints on how to dig deeper on this matter?

Regards,

Gonzalo Vásquez Sáez

Gerente Investigación y Desarrollo (R&D)
Altiuz Soluciones Tecnológicas de Negocios Ltda.
Av. Nueva Tajamar 555 Of. 802, Las Condes
(56-2) 335 2461
gvasquez@altiuz.cl

http://www.altiuz.cl

 

<Firma_medio ambiente_mail.jpg>

<akka-profiling.png>

--

[Jonas Bonér]

Create a ticket, attach all information and it will be prioritized and
planned along with everything else.

On Fri, Aug 17, 2012 at 12:14 PM, √iktor Ҡlang <viktor...@gmail.com> wrote:
> Hi Gonzalo,
>
> I will unfortunately not have time to have a look at this, I'm leaving
> shortly and will be off fo about 2 weeks.
>
> Would be great if someone else could offer to help you out.
>
> Thanks,
> √
>
> On Fri, Aug 17, 2012 at 12:49 AM, Gonzalo Vasquez <gvas...@altiuz.cl>
> wrote:
>>
>> Victor,
>>
>> I have three different parts: master, worker and messages. I've included
>> the 3 projects in a single zip, along with their pom.xml maven descriptors
>>
>> Both worker and master parts have Main-classes as defined in their pom
>> files, as this a sample approach and the final arquitecture will consider
>> webservice based job triggering instead of running a master main-class. The
>> worker is supposed to be run as a daemon on the remote servers.
>>
>> My project's logic is in separate java-projects, so it's not included in
>> here, only the akka-related parts.
>>
>>
>> Gonzalo Vásquez Sáez
>> Gerente Investigación y Desarrollo (R&D)
>> Altiuz Soluciones Tecnológicas de Negocios Ltda.
>> Av. Nueva Tajamar 555 Of. 802, Las Condes
>> (56-2) 335 2461

>> gvas...@altiuz.cl

>> http://www.altiuz.cl
>>
>>
>>
>> --
>> >>>>>>>>>> Read the docs: http://akka.io/docs/
>> >>>>>>>>>> Check the FAQ: http://akka.io/faq/
>> >>>>>>>>>> Search the archives:
>> >>>>>>>>>> https://groups.google.com/group/akka-user
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Akka User List" group.
>> To post to this group, send email to akka...@googlegroups.com.
>> To unsubscribe from this group, send email to
>> akka-user+...@googlegroups.com.
>> Visit this group at http://groups.google.com/group/akka-user?hl=en.
>>
>>
>>
>>
>> El 16-08-2012, a las 18:37, Viktor Klang <viktor...@gmail.com>
>> escribió:
>>
>> What pieces do you have?
>>
>> On Fri, Aug 17, 2012 at 12:33 AM, Gonzalo Vasquez <gvas...@altiuz.cl>
>> wrote:
>>>
>>> Which pieces of code do I send? Ask for and I'll send them!
>>>
>>> Gonzalo Vásquez Sáez
>>> Gerente Investigación y Desarrollo (R&D)
>>> Altiuz Soluciones Tecnológicas de Negocios Ltda.
>>> Av. Nueva Tajamar 555 Of. 802, Las Condes
>>> (56-2) 335 2461

>>> gvas...@altiuz.cl

>>>> gvas...@altiuz.cl

Jonas Bonér
Phone: +46 733 777 123
Home: http://jonasboner.com
Twitter: @jboner

[√iktor Ҡlang]

On Fri, Aug 17, 2012 at 12:15 PM, Jonas Bonér <jo...@jonasboner.com> wrote:

Create a ticket, attach all information and it will be prioritized and
planned along with everything else.

It's not a bug or a feature and not docs, it's helping a user out why it's not performing as expected.

[√iktor Ҡlang]

On Fri, Aug 17, 2012 at 12:18 PM, √iktor Ҡlang <viktor...@gmail.com> wrote:

On Fri, Aug 17, 2012 at 12:15 PM, Jonas Bonér <jo...@jonasboner.com> wrote:

Create a ticket, attach all information and it will be prioritized and
planned along with everything else.

It's not a bug or a feature and not docs, it's helping a user out why it's not performing as expected.

i.e. exactly what the mailing list is for.

[Jonas Bonér]

On Fri, Aug 17, 2012 at 12:18 PM, √iktor Ҡlang <viktor...@gmail.com> wrote:
>
>
> On Fri, Aug 17, 2012 at 12:15 PM, Jonas Bonér <jo...@jonasboner.com> wrote:
>>
>> Create a ticket, attach all information and it will be prioritized and
>> planned along with everything else.
>
>
> It's not a bug or a feature and not docs, it's helping a user out why it's
> not performing as expected.
>

I assumed that a regression in Akka could not be ruled out. But if you say so.

[Viktor Klang]

On Fri, Aug 17, 2012 at 12:22 PM, Jonas Bonér <jo...@jonasboner.com> wrote:
> On Fri, Aug 17, 2012 at 12:18 PM, √iktor Ҡlang <viktor...@gmail.com> wrote:
>>
>>
>> On Fri, Aug 17, 2012 at 12:15 PM, Jonas Bonér <jo...@jonasboner.com> wrote:
>>>
>>> Create a ticket, attach all information and it will be prioritized and
>>> planned along with everything else.
>>
>>
>> It's not a bug or a feature and not docs, it's helping a user out why it's
>> not performing as expected.
>>
>
> I assumed that a regression in Akka could not be ruled out. But if you say so.

I'd seconded that sentiment if the case was that the approach degraded
when updating versions, but here he has switched approaches and the
second doesn't perform as well as the first.

[Jonas Bonér]

On Fri, Aug 17, 2012 at 12:25 PM, Viktor Klang <viktor...@gmail.com> wrote:
> On Fri, Aug 17, 2012 at 12:22 PM, Jonas Bonér <jo...@jonasboner.com> wrote:
>> On Fri, Aug 17, 2012 at 12:18 PM, √iktor Ҡlang <viktor...@gmail.com> wrote:
>>>
>>>
>>> On Fri, Aug 17, 2012 at 12:15 PM, Jonas Bonér <jo...@jonasboner.com> wrote:
>>>>
>>>> Create a ticket, attach all information and it will be prioritized and
>>>> planned along with everything else.
>>>
>>>
>>> It's not a bug or a feature and not docs, it's helping a user out why it's
>>> not performing as expected.
>>>
>>
>> I assumed that a regression in Akka could not be ruled out. But if you say so.
>
> I'd seconded that sentiment if the case was that the approach degraded
> when updating versions, but here he has switched approaches and the
> second doesn't perform as well as the first.
>

True.

[Gonzalo Vasquez]

Ok, thanks Victor, could anyone else help me please?

Thanks again!

Gonzalo Vásquez Sáez
Gerente Investigación y Desarrollo (R&D)
Altiuz Soluciones Tecnológicas de Negocios Ltda.
Av. Nueva Tajamar 555 Of. 802, Las Condes
(56-2) 335 2461

gvasquez@altiuz.cl
http://www.altiuz.cl
 

[Patrik Nordwall]

Hi Gonzalo,

I don't understand the stack traces in your jvisulvm image. At the top it looks like processRequest is processed directly from the default-dispatcher without going through onReceive of the actor. I guess that is what puzzles you also. Could it be that jvisualvm is fooling us here. It is sampling, isn't it?

Can you verify with another tool or with plain println? It would also be interesting to know how long your work time is compared to the network roundtrip.

If the bottleneck is the remote send (network) and not the work it is not strange if you get half speed, since you send double amount of remote messages.

Cheers, Patrik

Patrik Nordwall

Typesafe -  The software stack for applications that scale

Twitter: @patriknw

[Gonzalo Vasquez]

Patrik,

Neither do I, I've added some extra logging and the processRequest method is only getting invoked once per job, so the stacktrace seems to be buggy. It's sampling indeed.

Network time could be an issue, but how do I measure it? It seems that I haven't doubled the amount of data messages, but perhaps the "sync" messages.

Gonzalo Vásquez Sáez

Gerente Investigación y Desarrollo (R&D)
Altiuz Soluciones Tecnológicas de Negocios Ltda.
Av. Nueva Tajamar 555 Of. 802, Las Condes
(56-2) 335 2461
gvasquez@altiuz.cl

http://www.altiuz.cl

 

El 17-08-2012, a las 9:32, Patrik Nordwall <patrik....@gmail.com> escribió:

Hi Gonzalo,

I don't understand the stack traces in your jvisulvm image. At the top it looks like processRequest is processed directly from the default-dispatcher without going through onReceive of the actor. I guess that is what puzzles you also. Could it be that jvisualvm is fooling us here. It is sampling, isn't it?

Can you verify with another tool or with plain println? It would also be interesting to know how long your work time is compared to the network roundtrip.

If the bottleneck is the remote send (network) and not the work it is not strange if you get half speed, since you send double amount of remote messages.

Cheers, Patrik

On Fri, Aug 17, 2012 at 3:07 PM, Gonzalo Vasquez <gvas...@altiuz.cl> wrote:

Ok, thanks Victor, could anyone else help me please?

Thanks again!

Gonzalo Vásquez Sáez
Gerente Investigación y Desarrollo (R&D)
Altiuz Soluciones Tecnológicas de Negocios Ltda.
Av. Nueva Tajamar 555 Of. 802, Las Condes
(56-2) 335 2461
gvasquez@altiuz.cl
http://www.altiuz.cl
 

<Firma_medio ambiente_mail.jpg>

[Patrik Nordwall]

I'd probably start with some basic System.nanoTime measurements to get a feeling of how long the worker time is compared to the network round trip.

If you pass a nanoTime in the messages you can measure the round trip. You must measure the elapsed duration in the same jvm as you originally created the start time nanoTime. Pass it back.

I'd also subscribe to RemoteLifeCycleEvent to see if there are many connects/disconnects.

See end of http://doc.akka.io/docs/akka/current/java/remoting.html

/Patrik

17 aug 2012 kl. 21:51 skrev Gonzalo Vasquez <gvas...@altiuz.cl>:

Patrik,

Neither do I, I've added some extra logging and the processRequest method is only getting invoked once per job, so the stacktrace seems to be buggy. It's sampling indeed.

Network time could be an issue, but how do I measure it? It seems that I haven't doubled the amount of data messages, but perhaps the "sync" messages.

Gonzalo Vásquez Sáez

Gerente Investigación y Desarrollo (R&D)
Altiuz Soluciones Tecnológicas de Negocios Ltda.
Av. Nueva Tajamar 555 Of. 802, Las Condes
(56-2) 335 2461
gvasquez@altiuz.cl

http://www.altiuz.cl

 

<Firma_medio ambiente_mail.jpg>

[Gonzalo Vásquez]

Havent used nanotime, I'll check it.

Regarding monitoring, I'll add that too monday.

Thanks for the tips.

Enviado desde mi iPhone

[Gonzalo Vasquez]

I've just finished measuring network time (not roundtrip, only the time used in tell method invocations), and it's about 300 smaller than the actual processing time. I've also tried different packet sizes, with no relevant change. 

Next step is to actually measure round trip, and try the RemoteLifeCycleEvent subscription.

Any other ideas?

Gonzalo Vásquez Sáez

Gerente Investigación y Desarrollo (R&D)
Altiuz Soluciones Tecnológicas de Negocios Ltda.
Av. Nueva Tajamar 555 Of. 802, Las Condes
(56-2) 335 2461
gvasquez@altiuz.cl

http://www.altiuz.cl

 

El 18-08-2012, a las 3:47, Patrik Nordwall <patrik....@gmail.com> escribió:

[Patrik Nordwall]

On Mon, Aug 20, 2012 at 8:56 PM, Gonzalo Vasquez <gvas...@altiuz.cl> wrote:

I've just finished measuring network time (not roundtrip, only the time used in tell method invocations), and it's about 300 smaller than the actual processing time. I've also tried different packet sizes, with no relevant change. 

and what is the processing time, in absolute numbers

 

Next step is to actually measure round trip, and try the RemoteLifeCycleEvent subscription.

Easiest is probably to turn on logging of these in your conf file.

akka.remote.log-remote-lifecycle-events = on

[Gonzalo Vásquez]

Patrik, processing each work unit takes about 100ms, but as I'm sending packets there is no network traffic on each unit, but upen each packet completion. I've also tested different packet sizes, coming with 25 as the best value in the dev environment.

I'll configure suggested logging, and enable prefetching as currently a new packet is only requested when there is no more data to be processed, so I guess there might be some extra lag / overhead between packets.

Regards.-

Enviado desde mi iPhone

El 21-08-2012, a las 2:21, Patrik Nordwall <patrik....@gmail.com> escribió:

On Mon, Aug 20, 2012 at 8:56 PM, Gonzalo Vasquez <gvas...@altiuz.cl> wrote:

I've just finished measuring network time (not roundtrip, only the time used in tell method invocations), and it's about 300 smaller than the actual processing time. I've also tried different packet sizes, with no relevant change. 

and what is the processing time, in absolute numbers

 

Next step is to actually measure round trip, and try the RemoteLifeCycleEvent subscription.

Easiest is probably to turn on logging of these in your conf file.

akka.remote.log-remote-lifecycle-events = on

 

Any other ideas?

Gonzalo Vásquez Sáez

Gerente Investigación y Desarrollo (R&D)
Altiuz Soluciones Tecnológicas de Negocios Ltda.
Av. Nueva Tajamar 555 Of. 802, Las Condes
(56-2) 335 2461
gvasquez@altiuz.cl

http://www.altiuz.cl

 

<Firma_medio ambiente_mail.jpg>