What is scala-gwt hoping to support w.r.t. RPC serializable types?

[Nate Bauernfeind]

One of the best things about scala-gwt is the ability to re-use the same models on the client as on the server. I setup a simple RPC service and found that case classes don't properly serialize (even with a default constructor and extending Serializable). This would be a huge win for me if it can be done.

Is there anything on the roadmap for this? Any implementation details you can provide if I choose to tackle this, if it's even possible? 

For a quicker work around I'd be willing to consider serializing/deserializing to/from another type (like json string) but I'd like to be able to write a limited framework once instead of lots of one-of marshalling and translation code (already have had plenty of that with a java client and scala server). We heavily use jackson (and the jackson-module-scala open-source library), but since it heavily uses reflection I doubt that will be very feasible.

[Lex Spoon]

I'm curious, too, if anyone gets the standard GWT serialization to
work with Scala+GWT.

Note there are other serialization options that would also be valuable
to try. Here's one of them:

http://code.google.com/p/protobuf-gwt/

This might be an area where some development effort would really help.
Find a way to get case classes to serialize, so long as their argument
types are relatively simple (primitive types and other case classes).

Lex Spoon

[Nate Bauernfeind]

Standard RPC serialization does work with Service, ServiceAsync and ServiceImpl all written in scala. I can get it to send a case class with a definition like this:

// Note GWT serialization requires a no-arg constructor.

case class ProjectId(name: String) extends Serializable {

  def this() = this(null)

}

But the deserialized object in the client keeps the null value.

I'll look into the protobuf-gwt later tonight, but it looks like it might be a perfect example of how to add support.

[Lex Spoon]

On Mon, Apr 9, 2012 at 12:09 PM, Nate Bauernfeind
<nate.bau...@gmail.com> wrote:
> Standard RPC serialization does work with Service, ServiceAsync and
> ServiceImpl all written in scala. I can get it to send a case class with a
> definition like this:
>
> // Note GWT serialization requires a no-arg constructor.
> case class ProjectId(name: String) extends Serializable {
>   def this() = this(null)
> }
>
> But the deserialized object in the client keeps the null value.

Ah, right, I see the problem.

Have you verified that everything works for non-case classes in Scala?

For case classes, can you try putting a "var" in front of all the
field names, in addition to adding the extra constructor? It's ugly,
but it would be a starting point for coming up with something cleaner.

As far as what the cleaner thing is, if the above works, the following
ancient issue would be helpful:

http://code.google.com/p/google-web-toolkit/issues/detail?id=1054

GWT has long been talking about supporting serialization of final
fields, and there's even a patch, but it somehow still hasn't landed.

-Lex

[Nate Bauernfeind]

Serialization works normally with scala objects that are serializable and define an empty constructor.

Using var's on the case class values work. 

Empty Array[]'s do not seem to work (if you wait long enough there is some complaining about not being able to verify a parameter.. unfortunately takes a long time for the error to pop up and I didn't copy one of them -- and I've got to head home now).

Array[] objects of length 1 or more seem to work fine. Hmph.

So yes, looks like final field serialization would be useful here. Thanks for the var hint... not _exactly_ as we'd all like, but it's a great start.

[Nate Bauernfeind]

BTW it looks like GWT has a CustomFieldSerializers and it uses them heavily to add support for the java collections library. All typed instances still need to be serializable objects (simply calls readObject/writeObject).

So it'll probably be easier to add generic set/map/list support than it will be to add generic case class support.

I wish I could have access to a manifest and the ability to lookup a constructor at this layer... because if so then implementing this for case classes would actually be possible. If only this part of GWT was more like Jackson (using TypeReferences and for a module to take a type and return a boolean as to whether or not it is able to ser/des a particular object).

[Lex Spoon]

Excellent news about the var plus the default constructor!

I'm a little surprised about the empty arrays. It might be worth
pursuing that and trying to smooth out that ugly corner case.

Custom field serializers are an interesting idea! As you say, you
probably won't be able to write a single one that works for any case
class. However, there could be a scalac plugin that does the trick.
Every time the plugin saw a case class, it could generate a custom
serializer on the side for it.

It would be better if the core tools just worked right, though.
Plugins are a pain on users. I have dug a little into the relevant
parts of the GWT code--bringing back all sorts of memories!--and I
believe I have identified the line of code that emits the Java "new"
expressions that reconstruct objects coming off the wire:

https://github.com/scalagwt/scalagwt-gwt/blob/scalagwt/user/src/com/google/gwt/user/rebind/rpc/FieldSerializerCreator.java#L356

If you scroll up a little, you will see a number of special cases,
e.g. for custom serializers. Perhaps there is a special case we could
come up with that would work for Scala?

This doesn't seem like something good to include in the initial patch
to GWT, but it is potentially a good followon patch. The first patch
would make Scala+GWT work at all, and this second one would make it
work more conveniently.

-Lex

[Nate Bauernfeind]

Something I did not need to know about until now is that GWT generates a Whitelist (at compile-time) of types that are allowed to be serialized/deserialized at run-time. The restriction seems to have come about when java.io.Serializable was chosen to be "supported" (over IsSerializable). Details here: https://developers.google.com/web-toolkit/doc/latest/FAQ_Server#Does_the_GWT_RPC_system_support_the_use_of_java.io.Serializable

Unfortunately, this means we can't use CustomFieldSerializer for things like scala.collection.{List, Map, or even Set}. This is because each of these actually use different implementations under the covers. List uses :: and Nil, and Set/Map tend to actually use mutable/immutable versions, and there are tons of specialized versions of everything. My first sample custom serializer was for Tuple2 and it works for Strings, other objects, arrays (haven't had a chance to look into the empty array issue), and even Tuple2's. It did not work for Tuple2[String, Int] however since it uses a specialized impl (Tuple2$mCLI$sp) which was not whitelisted.

Maybe we'll have to prepare to introduce something sooner than later. My probing around suggests a new-age replacement for CustomFieldSerializer (at the same places those currently hook in), modeled after a very light set of jackson-like functionality, would fit very well.

From a high level:

- do not require static methods (come on, let's write these things in scala!)

- features a resolver layer that attempts to provide a serializer/deserializer based on the JClassType

- serializer/deserializer will use newBuilder() to reconstruct collections

- case classes will still need to be particularly special but a good starting place is assignability to Product.

- ignore whitelist for anything that a resolver supports

Note that I am implying that sometimes specific implementations of things server side do not making it over client side... so suppose you have an rpc that returns Iterable[T], but server side it's really a subclass of Set[T]... then it must be acceptable that the client be able to use the Iterable.newBuilder (which currently resolves to a ListBuffer). 

Opinions, comments, suggestions? Would it be a terrible idea to attempt to tackle this so early?

[Lex Spoon]

The new age replacement sounds great, but would be a significant amount of work.

I would not give up yet on custom serializers. The problems you
describe for collections already apply to Java. Java's List, Map, and
Set also have a gazillion subclasses, and serializing them doesn't
work all that well in practice. If I recall correctly, the best
practice we could come up with was to advise people to make final
subclasses of the major collection types and to use those final
subclasses in their DTO classes.

To do something similar for Scala, you could make a custom subclass of
Scala's Seq, Map, and Set. Internally, they would wrap a normal List,
Map, and Set. For serialization, you would supply a custom serializer
that does something reasonable and does *not* recurse into the member
types.

Note that a class with a custom serializer is always whitelisted as
serializable, even if it doesn't implement Serializable or
IsSerializable.

Also, note that the point of the whitelist is to avoid needing runtime
reflection, which leads to problems with optimization.

All this said, it might be good to stop and write up a wiki page or
some other form of doc summarizing what you have found with stock GWT
RPC. It sounds like a ton already works, despite the disappointments
about case classes and collections.

Lex Spoon

[Nate Bauernfeind]

On Wed, Apr 11, 2012 at 9:11 AM, Lex Spoon <l...@lexspoon.org> wrote:

I would not give up yet on custom serializers. The problems you
describe for collections already apply to Java. Java's List, Map, and Set also have a gazillion subclasses, and serializing them doesn't work all that well in practice. If I recall correctly, the best
practice we could come up with was to advise people to make final subclasses of the major collection types and to use those final subclasses in their DTO classes.

Interesting. What GWT did for java collections is write a generic Collection_CustomFieldSerializer and then implementations like  ArrayList_CustomFieldSerializer would call into the Collection's version of ser/deser for consistency sake.

However, it does look like some subsets of collections know to use their super class' CustomFieldSerializer. For example java.util.Stack has a generated field serializer that delegates to Vector_CustomFieldSerializer. I'll try to figure out tonight how that happens. 

 

To do something similar for Scala, you could make a custom subclass of
Scala's Seq, Map, and Set. Internally, they would wrap a normal List,
Map, and Set. For serialization, you would supply a custom serializer
that does something reasonable and does *not* recurse into the member
types.

Fantastic idea, that brings this back to a manageable subset.

 

All this said, it might be good to stop and write up a wiki page or
some other form of doc summarizing what you have found with stock GWT RPC. It sounds like a ton already works, despite the disappointments about case classes and collections.

Do you guys have a wiki setup somewhere already?

[Grzegorz Kossakowski]

On 11 April 2012 17:33, Nate Bauernfeind <nate.bau...@gmail.com> wrote:

Do you guys have a wiki setup somewhere already?

There are two options:

1. enable wiki for scalagwt-gwt project https://github.com/scalagwt/scalagwt-gwt
2. Add content to our website: https://github.com/scalagwt/scalagwt.github.com

Not sure which option is better, though.

--
Grzegorz Kossakowski

[Grzegorz Kossakowski]

I went with wiki, it's easier to edit:

https://github.com/scalagwt/scalagwt-gwt/wiki

--
Grzegorz Kossakowski