Is kryo good for long term projects

[Ankur]

Hi,

I am planning on using kryo for a long term project which would
probably evolve over time and will be changing the class hierarchy.

Also, being a large scale app, it will probably run simultaneously in
multiple JVMs and consume each others messages serialized by kryo, we
want to be sure that the serialization will always be the same and
usable across all the instances without any gotchas.

Does the order of registering classes work? Can the above scenario be
accomplished with kryo? Finally, is there a document that details how
classes are serialized?

[Bruce Ritchie]

Our application uses Kryo for serialization and the main potential shortfalls that I can think of are

1. No versioning by default

2. Registered class ordering must be identical on all nodes

#1 can be handled by a custom serializer(s), #2 is something that is easily handled by just appending, never modifying the existing list of registered classes.

For us we just require a full cluster restart on upgrades which negates all of these issues and other similar ones.

Regards,

Bruce Ritchie

--
You received this message because you are subscribed to the "kryo-users" group.
http://groups.google.com/group/kryo-users

[Nate]

Hi,

> Does the order of registering classes work?

I don't understand this question.

> Can the above scenario be accomplished with kryo?

Your description of how you will use Kryo is vague. Long term and large scale don't pose any issues. Any "gotchas" you might run into depend on how exactly you want to use the library.

> Finally, is there a document that details how classes are serialized?

Docs:
http://code.google.com/p/kryo/
See the javadocs and serializers in SVN for more detail.

As Bruce mentioned, if you are doing long term persistence, you may want to use a serializer other than FieldSerializer. If you are just sending messages over the network, this isn't a problem. Also there is a difference between keeping all instances of your app up to date, and having some of them use old versions of classes.

-Nate

On Tue, May 3, 2011 at 11:26 AM, Ankur <ankur...@gmail.com> wrote:

[Ankur]

> > Does the order of registering classes work?

This was meant to be "Does the order of registering classes matter?"
and I got my answer as yes it does.

By long term and large scale projects i mean projects which would
evolve, probably get more classes
maybe some of the old ones get re-factored into better abstractions, I
am sure this is pretty common
in enterprise systems where multiple teams collaborate on huge
projects and eventually refactor each other's
code to make it better or add features.

Though i am not doing long term persistence, I am using the serialized
versions(these have a limit of 1KB) of the objects to be put as
payloads of a message in a queue which would be acted upon by workers.
Now as it is the case that refactoring invalidates
the de-serializability of older messages(as it is the case with the
stock serializers) each time a newer version of the
system is deployed the team would have to either do a migration of the
serialized objects to the new format and re-insert
them to the queue, or stop newer tasks to be added, or have the system
manage older and newer versions.

Also, as per the earlier response I can get around by writing a custom
serializer to manage versioning but then the fact that I need to make
sure that the developer "ONLY" appends to the list is somewhat of a
magic bean when it comes to the long term usage.
The case in point is that in case 2 months from now if a different
team decides to add some new classes and adds stuff to the register
methods list in random order (not necessarily appends) this would
cause the system to become unusable and would become insanely
difficult to debug.

I think for my particular use case a schema based approach such as
avro or protobuf/protostuff is better especially because the results
are a but more reproducible and atleast avro addresses these problems
directly.

I do not mind writing some extra code to make kryo work but why do it!
Kryo is a good library but i don't really think its suitable in such a
scenario.

Unless someone can give me a good explanation about some features that
address these concerns or if they have had success in implementing a
solution that dealt with the above mentioned challenges I would love
to hear their suggestions.

[Nate]

If you aren't doing long term persistence and you don't mind invalidating existing serialized bytes, then the order in which you register classes doesn't matter. If at all possible, your life will be simplest if you allow the class definitions to be the schema and don't require compatibility. Kryo excels at this, other libraries cannot do this at all AFAIK. Any other solution requires you to maintain some sort of schema and increases the complexity. Kryo can be used for this, but other libraries may be more mature at this specific task. Note some serialization libraries that use schemas do not support forward/backward compatibility, which means you must maintain a schema for every version of serialized bytes you have. No free lunch! :)

-Nate