Replacing classes during custom serialization

[Jeff Schnitzer]

When doing custom serialization, is it possible to swap out one class
for another?

This would fix a _lot_ of problems using Objectify's Ref<?> and Key<?>
client-side, including, I think, being able to instantiate Refs and
Keys intelligently.

But I have an immediate problem I'm trying to fix - which is that
Ref<?> is a class hierarchy on the server side, but I'd really like to
simplify this to a single concrete Ref class client-side. So even
though the sever might return StdRef<?> or NullRef<?>, these should be
converted to a simplified, concrete Ref<?> on the client side.

Is this possible? It would be even better if I could somehow just
define one custom serializer that handles all Ref subclasses instead
of having to make separate custom serializers for each.

Thanks in advance,
Jeff

[Jeff Schnitzer]

As a second question, is there any document anywhere which describes
how custom serializers actually work? The tiny section in
DevGuideServerCommunication is not helpful. Some basic questions:

* Are custom serializers server-side only? Or do they get executed on
the client-side too?

* What's the relationship between the static methods and the instance
methods? I presume we have both for some sort of historical reason,
but can we get rid of the static methods? (last time I tried it didn't
work)

Jeff

[Thomas Broyer]

On Sunday, March 31, 2013 11:14:28 PM UTC+2, Jeff Schnitzer wrote:

As a second question, is there any document anywhere which describes
how custom serializers actually work?

I don't think so; at least not outside Google.

There's https://docs.google.com/document/d/1eG0YocsYYbNAtivkLtcaiEE5IOF5u4LUol8-LL0TIKU/edit about GWT-RPC serialization, but it's really light when it comes to custom serializers and simply defer to the DevGuide.

 

The tiny section in
DevGuideServerCommunication is not helpful. Some basic questions:

 * Are custom serializers server-side only? Or do they get executed on
the client-side too?

Custom serializers are used on both sides (isn't this clear from the doc?). If you need different implementations for the client and server, you can either have a Ref_ServerCustomFieldSerializer<Ref> or use super-sources (EnumMap and LinkedHashMap uses this, actually in addition to a ServerCustomFieldSerializer: the CustomFieldSerializer from "sources" is used in DevMode, the one from super-sources is annotated with @GwtScriptOnly and thus only used when compiled to JS, and the ServerCustomFieldSerializer is used on the server-side).

 * What's the relationship between the static methods and the instance
methods? I presume we have both for some sort of historical reason,
but can we get rid of the static methods? (last time I tried it didn't
work)

Actually, the static methods are used on client-side, while the instance methods are used on server-side (when they generally just defer to the static methods). Historically, custom serializers were only convention-based: must be named with the _CustomFieldSerializer suffix and have the serialize and deserialize static methods and optionally an instantiate static method. The CustomFieldSerializer interface was added in GWT 2.3, and the ServerCustomFieldSerializer added in GWT 2.5.0. The CustomFieldSerializer interface was added as an optimization, to avoid too much reflection at runtime on the server-side.

[Thomas Broyer]

I don't think it's possible, because the type signature is written independently from the custom serializers, so serializing a StdRef on the server-side will put the StdRef type signature in the stream, to be decoded on the client-side.

You can however just make StdRef and NullRef "super-source" classes that simply inherit from Ref without adding anything, and have their respective custom serializers call the Ref_CustomFieldSerializer methods (at least for the client-side implementation)

[Jeff Schnitzer]

Ah hah! The magic secret ingredient that will save me is the ability
to have different custom serializers on client and server, probably
via super-source. I hadn't thought of that.

Re-creating my Ref type hierarchy in super-source on the client is
annoying, but as you point out it's not an intractable problem since
all but the root can just be stub subclasses.

Thanks,
Jeff

[Jeff Schnitzer]

I have it "mostly" working, although the inability to replace a class (ala readResolve() or writeReplace()) pretty much means it's impossible for an object roundtrip through the RPC layer to work properly. A quick explanation:

 * I have a Ref impl of type LiveRef which holds a reference to the live datastore so it can resolve the entity pointed at.

 * With java serialization, a LiveRef writeReplace()s to a DeadRef which holds a reference to the actual entity.

My GWT client-only version of LiveRef is now implemented just like DeadRef, so clients can obtain a Ref from the server and work with it. But passing a LiveRef back to the server reconstitutes a real LiveRef which loses the entity part and "reattaches" to the datastore. So it's impossible to do a real roundtrip.

It sounds like I will have to live with this behavior for now.

One thing I want to say is that GWT custom serialization seems unbearably complicated and poorly documented. Here is why it is confusing:

 * what runs on client vs what runs on server

 * _CustomFieldSerializer vs _ServerCustomFieldSerializer??

 * static methods vs instance methods

 * instantiate vs deserialize (what's the difference?)

From the outside, this feels like something in desperate need of refactoring. Never mind backwards compatibility; as documented, this feature is basically unusable by anyone who isn't on the GWT team.

Jeff

[V.B.]

This is slightly OT but... Thomas, do you think it would be possible to use a custom serialization approach server-side to translate protocol buffers on the fly to/from parallel GWT classes?
   Our situation is this: We have lots of protocol buffer definitions that we pass around between AppEngine, Android, iOS, and WP8. Since GWT doesn't have PB support yet, we maintain parallel classes specifically for GWT and already perform a lot of switch on type and translation on the server. It's ugly but we have it working.
   I feel as though custom serialization might be able to offer a slightly cleaner approach for switching between GWT and PB types. Your thoughts?

[Thomas Broyer]

Revisiting this in response to feedback on https://code.google.com/p/google-web-toolkit/issues/detail?id=3303

I got something working in a few minutes without the need for super-source. I can't tell whether this is supported though (or whether it might break without notice ina future version of GWT).

The thing is that the actual return type from instantiate(), and parameter type for deserialize(), are only checked for assignability, and the deserialization code (that calls the CustomFieldSerializers) do not check the deserialized objects' types; so in LiveRef_CustomFieldSerializer you can actually have a "public static Ref instantiate(SerializationStreamReader reader) throws SerializationException" and "public void deserialize(SerializationStreamReader reader, Ref instance)" methods, and return a DeadRef instead of a LiveRef.

 

[Jeff Schnitzer]

Oh, that's fantastic - if it works (I will try it out in a couple days when I get some free time) then you should really close the issue as "Fixed" or "not a bug". That is a great solution. It should be documented though :)

Thanks Thomas.

Jeff

[Thomas Broyer]

On Tuesday, June 11, 2013 8:34:58 AM UTC+2, Jeff Schnitzer wrote:

Oh, that's fantastic - if it works (I will try it out in a couple days when I get some free time) then you should really close the issue as "Fixed" or "not a bug". That is a great solution. It should be documented though :)

Someone familiar with RPC would have to first acknowledge it as a supported solution, otherwise it could break in a future release of GWT without notice. Pending that, it's no more than a hack I'm afraid.