Remote Actor Performance

[Fatih Dönmez]

Hi,

I'm using remote actor to generate some content but facing some issues with big chunks around 128KB on concurrent requests. BalancingPool used with number of workers to support concurrent users. 

If I run the test on local to local, I'm getting 256 req / sec but if I run the same test on local to remote, I get 10 req / sec. In addition to that I also find these logs;

DEBUG test-app akka.remote.EndpointWriter - Drained buffer with maxWriteCount: 50, fullBackoffCount: 546, smallBackoffCount: 2, noBackoffCount: 1 , adaptiveBackoff: 2000

Processing time is same on actor's receive and delay on mailbox relatively same. I expect some performance decrease because of the network but not that much.

Also If I run same test (local - remote) with small chunks around 10B again with same number of concurrent requests, throughput is around 256 per sec. 

I'll do the same test with http api instead of remote as alternative but still curious about the issue.

Can you help me on this? 

[Patrik Nordwall]

256 req/s doesn't sound like much.

What serialization do you use. Could that be the bottleneck?

What kind of flow control do you use in the test? How many messages do you send?

/Patrik

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscribe@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

--

Patrik Nordwall
Akka Tech Lead
Lightbend -  Reactive apps on the JVM
Twitter: @patriknw

[Fatih Dönmez]

Hey Patrik,

I used default java serialisation. I was assuming it could have some drawback but still I don't understand why it performing this much different with same serialisation and test, only change is instead of local, I deployed remotely.  

I send some initial requests to warm up jvm and threads then sent 256 request, concurrency level is 128. Between following batches I put some delay (500ms). 

Cheers,

Fatih

24 Mart 2017 Cuma 23:03:09 UTC+7 tarihinde Patrik Nordwall yazdı:

256 req/s doesn't sound like much.

What serialization do you use. Could that be the bottleneck?

What kind of flow control do you use in the test? How many messages do you send?

/Patrik

On Fri, Mar 24, 2017 at 5:51 AM, Fatih Dönmez <fathd...@gmail.com> wrote:

Hi,

I'm using remote actor to generate some content but facing some issues with big chunks around 128KB on concurrent requests. BalancingPool used with number of workers to support concurrent users. 

If I run the test on local to local, I'm getting 256 req / sec but if I run the same test on local to remote, I get 10 req / sec. In addition to that I also find these logs;

DEBUG test-app akka.remote.EndpointWriter - Drained buffer with maxWriteCount: 50, fullBackoffCount: 546, smallBackoffCount: 2, noBackoffCount: 1 , adaptiveBackoff: 2000

Processing time is same on actor's receive and delay on mailbox relatively same. I expect some performance decrease because of the network but not that much.

Also If I run same test (local - remote) with small chunks around 10B again with same number of concurrent requests, throughput is around 256 per sec. 

I'll do the same test with http api instead of remote as alternative but still curious about the issue.

Can you help me on this? 

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.

To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.

To post to this group, send email to akka...@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

[Patrik Nordwall]

Serialization is not used for the local case.
As a test you can compare with sending a dummy byte array.

[Fatih Dönmez]

Sorry I think I didn't explain well. When I was saying local to local it's not same jvm. I run two jvm apps and connect each other with remoting (on localhost)

I was thinking serialisation still is happening in this case or am I wrong?

 

Cheers,

Fatih

To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscribe@googlegroups.com.

To post to this group, send email to akka...@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

--

>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---

You received this message because you are subscribed to a topic in the Google Groups "Akka User List" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/akka-user/szVVa7Ue5XI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to akka-user+unsubscribe@googlegroups.com.

To post to this group, send email to akka...@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

--

Fatih.

[Patrik Nordwall]

Ah, yes then serialization is used. Then it might actually be the network link. That can also explain the log message you noticed. What do you use? Only 10 mbit/s ?

--

>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---

You received this message because you are subscribed to a topic in the Google Groups "Akka User List" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/akka-user/szVVa7Ue5XI/unsubscribe.

To unsubscribe from this group and all its topics, send an email to akka-user+...@googlegroups.com.

To post to this group, send email to akka...@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

--

Fatih.

[Jasim V S]

We finally figured out this issue.
 

We had to optimize the send-buffer-size and receive-buffer-size on both client and server to be able to increase the number of concurrent requests.  This settings is on remote.netty.tcp.  

Measuring the time delay between sending and receiving the message helped up to figure out and optimize it.

Regards,
Jasim

[Patrik Nordwall]

Thanks for sharing. Interesting that it would make such big difference.

If you ever need better remoting performance you should try the new implementation, called Artery. 

It should not be a big difference for these message rates, though.

http://doc.akka.io/docs/akka/2.4/scala/remoting-artery.html

/Patrik

To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscribe@googlegroups.com.

To post to this group, send email to akka...@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.