Kryo schema evolution support

[Shushant Arora]

Hi

I have a java class serialized using kryo-shaded 4.0.1

class definition :

class A{
private int c1;
  private Map<Integer, String> c2;
}

After serialization when I deserialized its fine,But if If after serilization using existing definition and then add new field as

class A{
private int c1;
  private Map<Integer, String> c2;
private Map<Integer, String> c3;
}

And then tries to deserialize existing serialized content(serialized using old definition) it throws

com.esotericsoftware.kryo.KryoException: Buffer underflow.
Serialization trace:
....
at com.esotericsoftware.kryo.io.Input.require(Input.java:199)
    at com.esotericsoftware.kryo.io.Input.readVarInt(Input.java:373)
    at com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:127)
    at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:693)
    at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:118)
    at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:543)
    at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:709)

My code is thread safe as I am using Kryopool. Does kryo not support schema evolution.

[Joachim Durchholz]

Am 21.02.2018 um 04:31 schrieb Shushant Arora:
> Does kryo not support
> schema evolution.

Kryo is not built with that use case in mind. Which excludes Kryo from
some scenarios, but the authors consider this out of scope, so there you
have it.

That said, schema evolution is a pretty hairy concept.

First of all, you somehow need to have the list of fields for old and
new class. If the classes have the same qualified name, the only way
that the JVM can deal with this is to put them into different
classloaders, which comes with its own set of hairballs.
The alternative is to give the classes different names. Say, A is the
old class, A2 is the new version. Now Kryo can deserialize the old class
just fine, all you need to do is to write adapter code that converts A
to A2, and suddenly you don't need any schema evolution support anymore.

Yet another alternative would be to put the class information in the
data stream. This is somewhat ugly, because (a) you need a whole lot of
information (superclasses, plus class information for all serialized
fields), all the while being pretty careful that you don't end up
serializing the class information for the entire JDK, and (b) you want
to send that information just once per communication partner, but Kryo
does not know about communication partners, is knows about individual
streams.

The second bit hairiness is that you'd need mechanisms to deal with all
kinds of class rewriting. What to do if old class has
List<TelephoneNumber>, new class has TelephoneNumber phone1, phone2,
phone3 (this happens if two systems try to communicate and the new
system needs to talk to the old system). What do to if there is no
information loss but the data structures are different. What to do if
some data structure has was demoted to cache because it turned out it's
redundant - the deserializer has to rebuild the cache.
Nothing of that is rocket science, but it requires an object graph
rewriting engine. It's probably something that's easier to do at the
Java level than as part of Kryo; Kryo would have to play catch-up with
ever-more-complicated scenarios.

An object graph rewriting engine would be an interesting project, but
probably out of scope for Kryo :-)

[Shushant Arora]

what if my schema has new fields added only and no delete support required, is there any easy alternative, 

I tried VersionFieldSerializer for my class while creating kryo instance but its still calling FieldSerializer for deserializing my class ,Is  VersionFieldSerializer  helpful here can you share some sample code ?

--
You received this message because you are subscribed to the "kryo-users" group.
http://groups.google.com/group/kryo-users
--- You received this message because you are subscribed to a topic in the Google Groups "kryo-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/kryo-users/lKVoiZneP-M/unsubscribe.
To unsubscribe from this group and all its topics, send an email to kryo-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Joachim Durchholz]

Am 21.02.2018 um 18:31 schrieb Shushant Arora:
> what if my schema has new fields added only and no delete support
> required,

Trust me: This is not what you are going to have in the long term.

> is there any easy alternative,
> I tried VersionFieldSerializer for my class while creating kryo instance
> but its still calling FieldSerializer for deserializing my class ,Is
> VersionFieldSerializer  helpful here can you share some sample code ?

Something like that might work, but I haven't use that in anger.
Maybe somebody else can help with that.