Announcement: akka-kryo-serialization - yet another efficient serialization implementation for akka-remote

[Roman Levenstein]

Hi,

As a follow-up of my announcement of akka-protostuff-serialization and akka-quickser-serialization libraries, I'd like to announce today the availability of a kryo-based serialization library for akka: https://github.com/romix/akka-kryo-serialization

This library has a much better performance (e.g. 6-7 times faster!) and much smaller serialized representation size than Java-based serializers. It also seems to have a bit better performance than akka-protostuff-serialization and akka-quickser-serialization on the same test cases. The library does not require any additional build steps, as it is the case e.g. with protobuf's serializers when generating the marshalling and unmarshalling code from protobuf's proto files.

All configuration for using the announced serialization backend can be done in the application.conf of your Akka application. It means that it can be used almost as a drop-in replacement for the Java-based serialization.

The library is available under Apache 2.0 license.

To improve this library even further, Akka users community is very welcome to try it out using real applications and real use-cases.

Any comments and feedback are welcome!

Regards,
  Roman

P.S. Please update the docs with a link to this library.

[Jonas Boner]

Very cool. Thanks. We'll add links to it.

> --
> You received this message because you are subscribed to the Google Groups
> "Akka User List" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/akka-user/-/wuMRsUMNQn4J.
> To post to this group, send email to akka...@googlegroups.com.
> To unsubscribe from this group, send email to
> akka-user+...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/akka-user?hl=en.



--
Jonas Bonér
CTO
Typesafe - The software stack for applications that scale
Phone: +46 733 777 123
Twitter: @jboner

[Guillaume Simard]

This is very interesting indeed, I'll try it :)

Guillaume

> --
> You received this message because you are subscribed to the Google Groups
> "Akka User List" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/akka-user/-/wuMRsUMNQn4J.
> To post to this group, send email to akka...@googlegroups.com.
> To unsubscribe from this group, send email to
> akka-user+...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/akka-user?hl=en.



--

Guillaume Simard, B.Ing., M.Sc.A.
e.   gsi...@gmail.com
t.    514-501-5707

[√iktor Ҡlang]

Links added!

Viktor Klang

Akka Tech Lead

Typesafe - The software stack for applications that scale

Twitter: @viktorklang

[Evan Chan]

Nice.  With Kryo 2.0, you no longer have to add zero-arg constructors to every case class, right?

[Roman Levenstein]

On Friday, June 1, 2012 8:41:59 PM UTC+2, Evan Chan wrote:

Nice.  With Kryo 2.0, you no longer have to add zero-arg constructors to every case class, right?

That's correct. 

[Philip]

Hi Akka team,

I am using Roman's Kryo serialization extension and it is performing much better than the default Java serialization. If configured properly, Kryo can be used almost as a drop-in replacement for Java serialization. Could you maybe consider turning Kryo serialization into an official Akka module? It might even make sense to make it the default serialization option at some point.

Philip

[√iktor Ҡlang]

Hi Philip,

On Wed, Oct 31, 2012 at 11:39 PM, Philip <philip...@gmail.com> wrote:

Hi Akka team,

I am using Roman's Kryo serialization extension and it is performing much better than the default Java serialization. If configured properly, Kryo can be used almost as a drop-in replacement for Java serialization. Could you maybe consider turning Kryo serialization into an official Akka module?

What's the problem with having it as it is?

 

It might even make sense to make it the default serialization option at some point.

I'm not seeing that happening any time soon, mainly because that would mean that akka-actors.jar would have yet another dependency (we want to keep that number near 0).

I'm really glad you're enjoying Kryo serialization!

Cheers,

√

 

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://akka.io/faq/
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---

You received this message because you are subscribed to the Google Groups "Akka User List" group.

To post to this group, send email to akka...@googlegroups.com.
To unsubscribe from this group, send email to akka-user+...@googlegroups.com.

Visit this group at http://groups.google.com/group/akka-user?hl=en.
 
 

--

Viktor Klang

Akka Tech Lead

Typesafe - The software stack for applications that scale

Twitter: @viktorklang

[Philip]

Hi Viktor,

I am using Roman's Kryo serialization extension and it is performing much better than the default Java serialization. If configured properly, Kryo can be used almost as a drop-in replacement for Java serialization. Could you maybe consider turning Kryo serialization into an official Akka module?

What's the problem with having it as it is?

If it were an official Akka module, then there would be an implied assurance that the module is maintained, high-quality, and available from the Typesafe repository. This would probably lead to more Akka users adopting it and experiencing a massive increase in remote actor performance. The improvement is even available without the constraints and hassle of defining protocol buffers.

 

It might even make sense to make it the default serialization option at some point.

I'm not seeing that happening any time soon, mainly because that would mean that akka-actors.jar would have yet another dependency (we want to keep that number near 0).

I understand and appreciate this approach. At the same time efficient serialization is very important for the performance of remote actors and the dependency would only have to be part of the akka-remote module. If you instead come up with your own magical macro that implements the functions in Externalizable, that would be cool too. Anything that makes writing high-performance code easier would be appreciated.

 

I'm really glad you're enjoying Kryo serialization!

I'm enjoying Akka too, it is a fine piece of software. Thanks a lot for all the great work!

Philip

[√iktor Ҡlang]

On Thu, Nov 1, 2012 at 1:59 AM, Philip <philip...@gmail.com> wrote:

Hi Viktor,

I am using Roman's Kryo serialization extension and it is performing much better than the default Java serialization. If configured properly, Kryo can be used almost as a drop-in replacement for Java serialization. Could you maybe consider turning Kryo serialization into an official Akka module?

What's the problem with having it as it is?

If it were an official Akka module, then there would be an implied assurance that the module is maintained, high-quality, and available from the Typesafe repository. This would probably lead to more Akka users adopting it and experiencing a massive increase in remote actor performance. The improvement is even available without the constraints and hassle of defining protocol buffers. 

 

It might even make sense to make it the default serialization option at some point.

I'm not seeing that happening any time soon, mainly because that would mean that akka-actors.jar would have yet another dependency (we want to keep that number near 0).

I understand and appreciate this approach. At the same time efficient serialization is very important for the performance of remote actors and the dependency would only have to be part of the akka-remote module.

The mapping of serializers is in akka-actor, not in akka-remote, and the default is that Serializable goes to JavaSerializer: https://github.com/akka/akka/blob/master/akka-actor/src/main/resources/reference.conf#L338

 

If you instead come up with your own magical macro that implements the functions in Externalizable, that would be cool too. Anything that makes writing high-performance code easier would be appreciated.

Even better, how about _you_ come up with something like that and create a pull request, that would be awesome! :-)

 

 

I'm really glad you're enjoying Kryo serialization!

I'm enjoying Akka too, it is a fine piece of software. Thanks a lot for all the great work!

I'm very glad to hear that, there's been a lot of hard work going into it :-)

Cheers,

√

 

Philip

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://akka.io/faq/
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To post to this group, send email to akka...@googlegroups.com.
To unsubscribe from this group, send email to akka-user+...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user?hl=en.
 
 

[√iktor Ҡlang]

http://akka.io/community/

On Sat, Dec 15, 2012 at 11:49 AM, Anthony <jibb...@gmail.com> wrote:

Where are these links posted?

--Anthony

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://akka.io/faq/
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---

You received this message because you are subscribed to the Google Groups "Akka User List" group.
To post to this group, send email to akka...@googlegroups.com.
To unsubscribe from this group, send email to akka-user+...@googlegroups.com.

Visit this group at http://groups.google.com/group/akka-user?hl=en.
 
 

--
Viktor Klang

Director of Engineering

[Roman Levenstein]

Hi Kevin,

Yes I plan to add support for Akka 2.1, but most likely it will happen in January, because I'm leaving for vacations next week.

On Thursday, December 13, 2012 7:14:34 PM UTC+1, Kevin Osborn wrote:

Any plans on updating this module to work with Akka 2.1?  Akka uses Scala 2.10.0, but your library uses scala.reflect.ClassManifest, which was apparently removed in Scala 2.10.0. I really want to use this library, but don't know enough about Scala to patch it myself.

[Kevin Osborn]

Sounds great. I can't wait. Enjoy your vacation.

KEVIN OSBORN
LEAD SOFTWARE ENGINEER
CNET Content Solutions

OFFICE 949.399.8714     
CELL 949.310.4677      SKYPE osbornk     

5 Park Plaza, Suite 600, Irvine, CA 92614

[Amit Mula]

hi Roman,

I am using kryo with the configurations specified in application.config. The akka remoting application was working correctly ealier with Java serialization. But as I switched to Kyro, messages are going to dead letters.

My message class is below which I am serializing with kyro:

public class Message {

ActorRef messageSender;
String messageContents;
public Message(String messageContents, ActorRef messageSender) {
this.messageSender = messageSender;
this.messageContents = messageContents;
}
public ActorRef getMessageSender() {
return messageSender;
}
public void setMessageSender(ActorRef messageSender) {
this.messageSender = messageSender;
}
public String getMessageContents() {
return messageContents;
}
public void setMessageContents(String messageContents) {
this.messageContents = messageContents;
}
}

Please help in resolving the problem.

Amit

[Roman Levenstein]

Hi Amit,

On Thursday, December 5, 2013 5:42:59 PM UTC+1, Amit Mula wrote:

hi Roman,

I am using kryo with the configurations specified in application.config. The akka remoting application was working correctly ealier with Java serialization. But as I switched to Kyro, messages are going to dead letters.

To be able to help you, I'd need a bit more information. Which  version of akka-kryo-serialization do you use? Which versions of Scala and Akka?

Do you have a self-contained example that could help to reproduce a problem?

I'd suggest that you file an issue on github and we continue the discussion there:

https://github.com/romix/akka-kryo-serialization

-Roman

[Amit Mula]

I have the following set of dependencies in POM :

akka-remote_2.10-2.3-M1.jar

akka-actor_2.10-2.3-M1.jar
scala-library-2.10.2.jar
akka-kryo-serialization_2.10-0.3.0.jar

kryo-2.22.jar

I am attaching two java programs one of which runs like a akka remote actor(Server) and other one is another akka remote actor running as a client(sending message to the server). On running both and sending the above message class after kryo serilization between the two causes messages to go to dead letters. Please suggest better strategies for this loop. 

[Amit Mula]

the problem arises whenever the message class contains some members of type java.util.ArrayList or an element of type ActorRef(anything which is not a java primitive type). Is there a need to serialize the non primitive types separately apart from the message class ? 

[Roman Levenstein]

Thanks for providing your test-case. I haven't run it yet, but at the first glance I see the following problem:

You use "akka.remote.message.Message" directly. Normally, people define their own message types, which are specific to their applications. Often it is done e.g. by means of case classes. Could you try to do the same?

One you have your custom classes, please register them in corresponding sections of the config file, e.g. assign an id in "mappings", mention in "serialization-bindings", etc. And let me know if it helps.

In the meantime, I'll try to run your code to what happens.

On Tuesday, December 10, 2013 2:00:59 PM UTC+1, Amit Mula wrote:

the problem arises whenever the message class contains some members of type java.util.ArrayList or an element of type ActorRef(anything which is not a java primitive type). Is there a need to serialize the non primitive types separately apart from the message class ?

It depends. Kryo can serialize pretty much anything, but it may require registering a class first. Some of the JDK classes are pre-registered in Kryo. Some others need to be explicitly registered (e.g. in mappings sections, etc). You should see something in the traces, if you forget to register a class and Kryo does not know how to serialize it. 

 

-Roman

[Roman Levenstein]

On Tuesday, December 10, 2013 1:09:56 PM UTC+1, Amit Mula wrote:

I have the following set of dependencies in POM :

akka-remote_2.10-2.3-M1.jar

akka-actor_2.10-2.3-M1.jar
scala-library-2.10.2.jar
akka-kryo-serialization_2.10-0.3.0.jar

kryo-2.22.jar

I am attaching two java programs one of which runs like a akka remote actor(Server) and other one is another akka remote actor running as a client(sending message to the server). On running both and sending the above message class after kryo serilization between the two causes messages to go to dead letters. Please suggest better strategies for this loop. 

I'm not able to run your code. 

- You refer to RemoteSys in your code, but config file contains LocalSys. 

- You refer to akka.remote.message.Message class, but you have not provide it.

Could you provide a self-contained test-case project? So that I only need to do "sbt compile" and "sbt run" or something similar to reproduce your issue? 

 

Thanks,

   Roman

[Amit Mula]

Hey Roman,

thanks..now it worked for me. I was not registering the type java.util.ArrayList with an unique id after including in the serialization-binding section. After registering the id for the type, its now working the way I wanted. Thanks anyway for the help.

[Roman Levenstein]

On Tuesday, December 10, 2013 4:33:44 PM UTC+1, Amit Mula wrote:

Hey Roman,

thanks..now it worked for me. I was not registering the type java.util.ArrayList with an unique id after including in the serialization-binding section. After registering the id for the type, its now working the way I wanted. Thanks anyway for the help.

OK. Glad that it works for you now.

-Roman