What is akka message network overhead?

[Sean Zhong]

Suppose I want to transfer a msg to another machine 

otherActor ! “hello" 

what is the wire message overhead?

1. the wire messge need to have information about ActorRef, will the ActorRef be attached for every message, or will it be cached on target machine, so the sender only need to attach an Id?

2. How string "hello" is serialized on the wire, with java serialization? or protobuf? Can this be configured?

3. What is other overhead for this message?

[Konrad Malawski]

Hello Sean,

actual overhead in terms of how many bytes – depends on your serialiser.

1) Yes, a message includes the sender. Not much optimisations in there currently, although we have nice ideas on what we can do within a cluster (short "handles" instead of full addresses),

2) Refer to: http://doc.akka.io/docs/akka/snapshot/scala/serialization.html Strings are a byte array, the envelope is protobuf (you can find it in `akka-remote`), other kinds of payload is whatever you configured – defaults to java serialisation.

3) Of the entire message? It depends on your serialiser.

Hope this helps, happy hakking :-)

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

--

Cheers,

Konrad 'ktoso' Malawski

hAkker @ Typesafe

[Sean Zhong]

Thanks, Konrad,


Are there other akka data/attributes attached to a remote sending message?  Or just?

serialized(msg) + actorRef

Which part of akka source code I can check for answer?

Besides, is there LOG flags I can enable so that I can check which serilization framework is in effect?

In my experiment, I cannot explain half network bandwitdh usage with akka remote messaging. My message throughput is 60MB/s, but the network bandwidth is 140MB/s. How can I trouble shooting this.

[Konrad Malawski]

Hi Sean,

On the wire:
You can look into https://github.com/akka/akka/tree/master/akka-remote/src/main/protobuf for what exactly we pass around on the wire.

Which serializer is used:
enable debug logging, we log this info in Serialization.scala log.debug("Using serializer[{}] for message [{}]", ser.getClass.getName, clazz.getName)

Other hints:
Are you sure you’re not CPU or something else -bound? And you should be able to stuff the network?
Have you played around with the number of threads etc?
What hardware are you benchmarking on?

​

[Sean Zhong]

Konrad, thanks.

After enabling the debug flag, 

I saw the system message like Terminate are using javaSerialiaer, is this expected?

[DEBUG] [08/06/2014 22:19:11.836] [0e6fb647-7893-4328-a335-5e26e2ab080c-akka.actor.default-dispatcher-4] [akka.serialization.Serialization(akka://0e6fb647-7893-4328-a335-5e26e2ab080c)] Using serializer[akka.serialization.JavaSerializer] for message [akka.dispatch.sysmsg.Terminate]

Besides, as my message is String, I cannot find related serialization log for type java.lang.String. How to know for sure protobuf is used for String?

Are you sure you’re not CPU or something else -bound? And you should be able to stuff the network?

What I means is that 140MB/s network are occupied, but source message throughput is only 60MB/s, so there are 80MB/s bandwidth I cannot explain.

[Michael Frank]

sean- how are you measuring your network thoroughput?  does 140MB/s include TCP, IP, ethernet header data?  are you communicating across local network or across the internet?  the greater the distance your packets have to travel (specifically the number of hops), the higher chance that they will get dropped and retransmitted, or fragmented.  a tool like Wireshark, tcpdump, or ScaPy could help you differentiate utilization at different protocol layers and also identify network instability.

-Michael

[Sean Zhong]

Hi Michael,

I used wireshark to capture the traffic. I found for each message sent(the message is sent with option noSender), there is an extra cost of ActorPath. 

For example, for the following msg example, message payload length length is 100 bytes(bold), but there is also a target actor path for 221 bytes(red), which is much bigger than message itself.

Can the actorPath overhead cost be reduced?

. 

akka.tcp://app0ex...@192.168.1.53:51582/remote/akka.tcp/2120193a-e10b-474e...@192.168.1.53:48948/remote/akka.tcp/clu...@192.168.1.54:43676/user/master/Worker1/app_0_executor_0/group_1_task_0#-768886794o
h1880512348131407402383073127833013356174562750285568666448502416582566533241241053122856142164774120:

[Sean Zhong]

My application.conf

akka {

  loglevel = "INFO"

  extensions = ["com.romix.akka.serialization.kryo.KryoSerializationExtension$"]

  actor {

    provider = "akka.remote.RemoteActorRefProvider"

    serializers {

      kryo = "com.romix.akka.serialization.kryo.KryoSerializer"

    }

    serialization-bindings {

       "java.lang.String" = kryo

    }

    default-dispatcher {

      throughput = 1024

 fork-join-executor {

        parallelism-max = 4

      }

    }

   default-mailbox {

      mailbox-type = "akka.dispatch.SingleConsumerOnlyUnboundedMailbox"

    }

  }

  remote {

    enabled-transports = ["akka.remote.netty.tcp"]

    netty.tcp {

      port = 0

    }

  }

}

On Thursday, August 7, 2014 1:45:12 PM UTC+8, Sean Zhong wrote:

Hi Michael,

I used wireshark to capture the traffic. I found for each message sent(the message is sent with option noSender), there is an extra cost of ActorPath. 

For example, for the following msg example, message payload length length is 100 bytes(bold), but there is also a target actor path for 221 bytes(red), which is much bigger than message itself.

Can the actorPath overhead cost be reduced?

. 

akka.tcp://app0executor0@192.168.1.53:51582/remote/akka.tcp/2120193a-e10b-474e-bccb-8ebc4b...@192.168.1.53:48948/remote/akka.tcp/cluster@192.168.1.54:43676/user/master/Worker1/app_0_executor_0/group_1_task_0#-768886794o
h1880512348131407402383073127833013356174562750285568666448502416582566533241241053122856142164774120:

[Sean Zhong]

Is it possible to reduce the average message overhead?

200 bytes extra cost per remote message doesn't looks good...

On Thursday, August 7, 2014 1:45:12 PM UTC+8, Sean Zhong wrote:

Hi Michael,

I used wireshark to capture the traffic. I found for each message sent(the message is sent with option noSender), there is an extra cost of ActorPath. 

For example, for the following msg example, message payload length length is 100 bytes(bold), but there is also a target actor path for 221 bytes(red), which is much bigger than message itself.

Can the actorPath overhead cost be reduced?

. 

akka.tcp://app0executor0@192.168.1.53:51582/remote/akka.tcp/2120193a-e10b-474e-bccb-8ebc4b...@192.168.1.53:48948/remote/akka.tcp/cluster@192.168.1.54:43676/user/master/Worker1/app_0_executor_0/group_1_task_0#-768886794o
h1880512348131407402383073127833013356174562750285568666448502416582566533241241053122856142164774120:

[Endre Varga]

Hi Sean,

Unfortunately there is no way to reduce this overhead without changing the wire layer format, which we cannot do now. As you correctly see, practically all the overhead comes from the path of the destination and sender actor. In the future we have plans to implement a scheme which allows the sender to abbreviate the most common paths used to a single number, but this needs a new protocol.

So the answer currently is that you cannot reduce this overhead without introducing some batching scheme yourself: instead of sending MyMessage you can send Array[MyMessage], so the cost of the recipient path is only suffered once for the batch, but not for the individual messages -- i.e. you can amortize the overhead.

-Endre

[√iktor Ҡlang]

Or add compression.

[Endre Varga]

On Thu, Aug 7, 2014 at 10:05 AM, √iktor Ҡlang <viktor...@gmail.com> wrote:

Or add compression.

This is the Akka wire level envelope, cannot be directly controlled by users (unless someone writes a new transport of course).

-Endre

[√iktor Ҡlang]

You can do wire-level compression.

Cheers,

√

[Sean Zhong]

Hi Viktor,

About wire-compression, do you mean this?

akka {
remote {
compression-scheme = "zlib" # Options: "zlib" (lzf to come), leave out for no compression
zlib-compression-level = 6 # Options: 0-9 (1 being fastest and 9 being the most compressed), default is 6
}
}

Will it do compression at message level? or at a batch level(share same source machine and target machine)?

Hi Endre,

This is the Akka wire level envelope, cannot be directly controlled by users (unless someone writes a new transport of course).

Which part of source code I can look at to write a new transport?  

On Thursday, August 7, 2014 4:22:16 PM UTC+8, √ wrote:

You can do wire-level compression.

On Thu, Aug 7, 2014 at 10:09 AM, Endre Varga <endre...@typesafe.com> wrote:

On Thu, Aug 7, 2014 at 10:05 AM, √iktor Ҡlang <viktor...@gmail.com> wrote:

Or add compression.

This is the Akka wire level envelope, cannot be directly controlled by users (unless someone writes a new transport of course).

-Endre

On Aug 7, 2014 9:52 AM, "Endre Varga" <endre...@typesafe.com> wrote:

Hi Sean,

Unfortunately there is no way to reduce this overhead without changing the wire layer format, which we cannot do now. As you correctly see, practically all the overhead comes from the path of the destination and sender actor. In the future we have plans to implement a scheme which allows the sender to abbreviate the most common paths used to a single number, but this needs a new protocol.

So the answer currently is that you cannot reduce this overhead without introducing some batching scheme yourself: instead of sending MyMessage you can send Array[MyMessage], so the cost of the recipient path is only suffered once for the batch, but not for the individual messages -- i.e. you can amortize the overhead.

-Endre

On Thu, Aug 7, 2014 at 8:11 AM, Sean Zhong <cloc...@gmail.com> wrote:

Is it possible to reduce the average message overhead?

200 bytes extra cost per remote message doesn't looks good...

On Thursday, August 7, 2014 1:45:12 PM UTC+8, Sean Zhong wrote:

Hi Michael,

I used wireshark to capture the traffic. I found for each message sent(the message is sent with option noSender), there is an extra cost of ActorPath. 

For example, for the following msg example, message payload length length is 100 bytes(bold), but there is also a target actor path for 221 bytes(red), which is much bigger than message itself.

Can the actorPath overhead cost be reduced?

. 

akka.tcp://app0executor0@192.168.1.53:51582/remote/akka.tcp/2120193a-e10b-474e-bccb-8ebc4b3a0...@192.168.1.53:48948/remote/akka.tcp/cluster@192.168.1.54:43676/user/master/Worker1/app_0_executor_0/group_1_task_0#-768886794o
h1880512348131407402383073127833013356174562750285568666448502416582566533241241053122856142164774120:

--

Cheers,

√

[√iktor Ҡlang]

Hi Sean,

On Thu, Aug 7, 2014 at 10:49 AM, Sean Zhong <cloc...@gmail.com> wrote:

Hi Viktor,

About wire-compression, do you mean this?

akka {
remote {
compression-scheme = "zlib" # Options: "zlib" (lzf to come), leave out for no compression
zlib-compression-level = 6 # Options: 0-9 (1 being fastest and 9 being the most compressed), default is 6
}
}

No, that's from a legacy version of Akka. I mean using a compressed link/interface.

--

Cheers,

√

[Sean Zhong]

compressed link/interface

Is this configuration inside Akka conf? I cannot find the document, do you have pointer to this?

Hi Sean,

For more options, visit https://groups.google.com/d/optout</

...

[√iktor Ҡlang]

That would be completely outside of Akka.

Cheers,

√

[Sean Zhong]

Unfortunately there is no way to reduce this overhead without changing the wire layer format, which we cannot do now. As you correctly see, practically all the overhead comes from the path of the destination and sender actor. In the future we have plans to implement a scheme which allows the sender to abbreviate the most common paths used to a single number, but this needs a new protocol.

Hi, Endre and Viktor, 

suppose I want to hack the wire format transport, which source file I should change? 

...

[Konrad Malawski]

Hi Sean,

to summarise our thoughts about the on-the-wire overhead:

* Viktor hinted at this technique: http://en.wikipedia.org/wiki/TCP_acceleration which requires special hardware AFAIK, but is able to transparently compress data sent over TCP connections. It's probably costly but can be applied without app modifications.

* We are aware of the large overhead and want to fix it soon – this will require changes in the remoting protocol, so it won't happen sooner than 2.4.x. The biggest part of the overhead is the ActorPath and Endre has suggested a neat "actor path shortening" trick which will save a lot of space on the wire for popular senders / receivers.

* You could implement your own Transport (like our NettyTransport) and hook additional Gzip stages to the netty pipeline in there... This is quite some work but depending on your use-case perhaps it's worth it? In order to gzip to give anything here it would have to span multiple messages.

Hope this helps!

If your case is not strong enough for implementing this yourself you'll have to wait for our remoting updates which will address this problem (we totally agree the overhead is too big and will bring it down).

[√iktor Ҡlang]

On Fri, Aug 8, 2014 at 3:17 PM, Konrad Malawski <kt...@typesafe.com> wrote:

Hi Sean,

to summarise our thoughts about the on-the-wire overhead:

* Viktor hinted at this technique: http://en.wikipedia.org/wiki/TCP_acceleration which requires special hardware AFAIK, but is able to transparently compress data sent over TCP connections. It's probably costly but can be applied without app modifications.

* We are aware of the large overhead and want to fix it soon – this will require changes in the remoting protocol, so it won't happen sooner than 2.4.x. The biggest part of the overhead is the ActorPath and Endre has suggested a neat "actor path shortening" trick which will save a lot of space on the wire for popular senders / receivers.

* You could implement your own Transport (like our NettyTransport) and hook additional Gzip stages to the netty pipeline in there... This is quite some work but depending on your use-case perhaps it's worth it? In order to gzip to give anything here it would have to span multiple messages.

Thanks for the summary, Konrad.

Sean: I'd say that this third option is your best and cheapest option until 2.4.

--

Cheers,

√

[Kam Kasravi]

What was Endre's idea - is it related to URL shortening as discussed here? http://stackoverflow.com/questions/742013/how-to-code-a-url-shortener - ie using a bijective function?

...

[Sean Zhong]

Thanks, I hacked the RemoteActorRefProvider, and now be able to reduce to per message overhead to 34 byes(26 bytes are fixed overhead of wire format, not sure what it is).

Hi Sean,

Cheers,

<a href="http://typesafe.com" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Ftypesafe.com\46sa\75D\46sntz\0751\46usg\75AFQjCNFC6SplTJxAP7AExZl1lClfJ-tq6w';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Ftypesafe.com\46sa\75D\46sntz\0751\46u

...

[Sean Zhong]

Besides the actorPath, mesage payload, there are also 21 bytes overhead in the wire format, which is also not trivival. You may want to consider this when changing the wire format in future.

Hi Sean,

Cheers,

Konrad 'ktoso' Malawski

<font style="margin:0px;padding:0px;border:0px;vertical-align:

...

[Prakhyat Mallikarjun]

Hi Sean,

I was just reading through your discussions. I was quiet curious on how did you measure message throughput in your app?

-prakhyat m m

[Sean Zhong]

Hi Prakhyat,

I created an test benchmark which have two layers, source actors shuffle data to target actors. I add metrics in the source actors to determine how many messages are passed to downstream target actors. That is for message throughput. I then place the source actors and target actors at my will to different machines, that I know whehther the link from one selected source actor to a selected target actor requires network transfer traffic cross machine, I then use this to compute how many network bandwidth is required. I use ganglia to gather the overall network traffic bandwidth.

Thanks

Sean

[Sean Zhong]

Hi Michael,

 does 140MB/s include TCP, IP, ethernet header data?

Yes, it includes. when I use wireshark to capture the traffic, I found netty will assemble multiple message to use single tcp packet. So, the amortized tcp/ip header cost is not that big.

are you communicating across local network or across the internet?

I use lan. there are several machines in a private cluster. 140MB/s represents real network traffic, which means local loopback traffic is not counted. 

the greater the distance your packets have to travel (specifically the number of hops), the higher chance that they will get dropped and retransmitted

The test is in LAN, it is not a problem. The travel latency from machine to machine should be less than 1 ms.

On Thursday, August 7, 2014 1:34:42 AM UTC+8, Michael Frank wrote:

[Sean Zhong]

In case, someone is also looking for the workaround, here is the hacked implementation 

https://github.com/clockfly/akka_ShortActorRefProvider

Hi Sean,

Cheers,

Konrad 'ktoso' Malawski

<font style="margin:0px;padding:0px;border:0px;vertical-align:

...

[Patrik Nordwall]

Thanks for sharing! I have create an issue for future investigations: https://github.com/akka/akka/issues/15784

/Patrik

Patrik Nordwall
Typesafe -  Reactive apps on the JVM
Twitter: @patriknw

[Prakhyat Mallikarjun]

Hi Sean,

How did you carry out the performance tests? Which tool you used? What are your suggestions on the tools?

Along with above, If you can forward me to related article to get me started on this will be event better.

-Prakhyat M M