experiences with serialization: json vs protobuf vs avro vs kryo...

[Joost de Vries]

I'm trying to find a suitable serialization for akka persistence. 

There are a lot of options but not all of them are equally used and are as mature. F.i. we picked avro with avro4s but hit some problems that weren't easy to fix.

Of course then it's tempting to fall back on json: it's familiar from rest, easy to analyse and there's stamina for schema evolution.

But I wonder if there's another option. So my question is: which non json serialisation are you using and what are your experiences with schema evolution and otherwise?

A special concern is that we'd rather not have to duplicate our case class structure in IDL files. 

[Richard Rodseth]

Because we're using Avro for Kafka, we also looked at avro and avro4s, but ended up using Protobuf (with Maven plugin) for Akka Persistence. Mapping between the IDL-generated classes and case classes is indeed unfortunate, but no other issues so far.

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscribe@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

[Justin du coeur]

I'm using Kryo, and that's working reasonably well so far, but it's not trivial to set up.  As I have time, I'm gradually writing that up, but it's only half-done.  Happy to answer questions if you want to explore that route.

My general impression is that protobuf is the well-established route, and the one that most people are using, but that does involve some duplication of information.  (Or generating the case classes from the protobuf IDL.)  I don't much care for that, which is why I went for Kryo; the end result is broadly similar in terms of the amount of effort, I believe, but with the serialization information encoded directly into the classes themselves.

On Mon, May 1, 2017 at 12:54 AM, Joost de Vries <joost....@gmail.com> wrote:

--

[Alan Burlison]

On 01/05/2017 17:15, Justin du coeur wrote:

> My general impression is that protobuf is the well-established route, and
> the one that most people are using

I wrote a wrapper around Akka-Spray to save JSON in flat files. It is
fairly primitive but has the (for my use case) big advantage that I can
examine and manipulate the persisted Actor data using other tools, which
is extremely useful during development.

--
Alan Burlison
--

[Joost de Vries]

Thanks for your experiences.

It sounds that protobuf is the most often used serialisation. And perhaps the need to write IDL is something where we'll have to bite the bullet. It's a bit of double book keeping but at least it should be easily testable....

I think the standard serialisation for Lagom is json. I haven't seen information on the why of that decision anywhere. Would be interesting to know.

[Guido Medina]

You might want to try Json Smile which is a Json binary, it is very effective and can favor of re-using Jackson ObjectMapper

https://github.com/FasterXML/jackson-dataformats-binary/tree/master/smile

I also use Kryo for mainly Java classes where schema evolution and compatibility is not important as all message classes are shared and cluster restarted fully.

HTH,

Guido.