From default java Serialization to CompatibleFieldSerializer with backward compatibility

[Alex Bélisle-Turcot]

Hi,

I joined a project where the business objects are being serialized and propagated to other systems. The default java serialization mechanism is used, and thus the slightest change requires the objects to be rebuilt from scratch, since the serialized file is incompatible. The rebuild process is costly and may take up to 6 hours, keeping in mind this is a system that must be up 24/7. It has been the reality for the past few years, but I intend to change that for the better, using Kryo ;)

Now, I'm currently in the process of adding/removing fields (which will invalidate the serialized file) so I want to switch to Kryo altogether. However, it may not be so easy to do everything at once... Just bare with me:

(1) After doing some tests, I believe CompatibleFieldSerializer is what I want (versus TaggedFieldSerializer).

(2) I added new fields in the BOs and marked them as transient (so that the default java implementation still works)

(4) once the BO are loaded from the old serialized file, I will save them back using CompatibleFieldSerializer => Unfortunately the new fields are transient and will not be saved.

What would be nice is to be able to specify a custom annotation which would override transient. Thus, the default serialization would work, and then CompatibleFieldSerializer would save the additional fields.

Ex:  

@CustomAnnotationToOverrideTransient

private transient int newValue;

Unless such functionality exists (or another way?), the 2 options I see are:

Implement a CustomCompatibleFieldSerializer just to add the CustomAnnotation feature. And then, in a future realease I would change to CompatibleFieldSerializer seemlessly.

OR

Use TaggedFieldSerializer in the next release and only mark the wanted field regardless of whether they are transient or not. Then, switch to CompatibleFieldSerializer in the following release (keeping TaggedFieldSerializer for an initial load then CompatibleFieldSerializer to re-write the whole thing).

 

Do you see any other blocker? Or other/better options?

[Nate]

FieldSerializer has a private boolean field "serializeTransient". It is false by default but it could be exposed through a setter and may do what you need.

I prefer TaggedFieldSerializer because the overhead is minimal. Do you really need old code to be able to load newer serialized objects?

-Nate

--
You received this message because you are subscribed to the "kryo-users" group.
http://groups.google.com/group/kryo-users
---
You received this message because you are subscribed to the Google Groups "kryo-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kryo-users+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Alex Bélisle-Turcot]

Thanks for the reply!

No I don't need "old code to load newer serialized objects", I need "new code to load old serialized objects and at the same time produce a new serialized format using Kryo".

My understanding is that serializeTransient would either write none or all transient fields. In my case I only want the flag to apply for a couple new fields (other fields are transient and must be ignored).

I'm leaning towards writing a custom CompatibleFieldSerializer, which would bypass the transient check when a field is annotated by @NotTransientForKryo. Is that something you would consider to include in your framework ?

Thanks!

[Nate]

On Tue, Oct 14, 2014 at 12:56 AM, Alex Bélisle-Turcot <alex.beli...@gmail.com> wrote:

No I don't need "old code to load newer serialized objects", I need "new code to load old serialized objects and at the same time produce a new serialized format using Kryo".

I understand you need to load your objects using Java built-in serialization, then write it with Kryo. That problem is separate from how you need to evolve your classes after switching to Kryo.

In the new Kryo serialization, do you need forward compatibility? Ie, do you ever need to load serialized bytes from newer class files? Ie, you have a class and you deploy that code somewhere. Later you add a field to the class and serialize some data to bytes. You send those bytes to the code that has the old version of the class and expect the bytes to be deserialized.

If the answer is no, then you don't need CompatibleFieldSerializer. Use TaggedFieldSerializer and you can add new fields as much as you like. Instead of removing a field, rename the field to "ignored" and put @Deprecated on it.

 

My understanding is that serializeTransient would either write none or all transient fields. In my case I only want the flag to apply for a couple new fields (other fields are transient and must be ignored).

I'm leaning towards writing a custom CompatibleFieldSerializer, which would bypass the transient check when a field is annotated by @NotTransientForKryo. Is that something you would consider to include in your framework ?

It seems pretty specific to your application, and even then (if I understand correctly) you only need the feature to do a one time move of the data from Java built-in serialization. I'd say hack it up, migrate your data, and then you don't need the hack anymore.

[Joachim Durchholz]

Am 14.10.2014 um 00:39 schrieb Alex Bélisle-Turcot:
>The default java serialization mechanism is
> used, and thus the slightest change requires the objects to be rebuilt from
> scratch, since the serialized file is incompatible. The rebuild process is
> costly and may take up to 6 hours, keeping in mind this is a system that
> must be up 24/7.

We feel your pain.

Does that mean that you have persisted serialized data?
In that case, you might run into trouble when upgrading Kryo.
At least that was what I concluded last time the issue was discussed; I
might be wrong, or things might have improved.

Regards,
Jo