Scala trait deserialization

[j...@joescii.com]

As I've been working on getting Lift to take advantage of clustering capabilities of servlet containers like Jetty, I've been having to clean up serialization and deserialization issues in the Lift code base. This is necessary because the container will rely on Java serialization to migrate the session data (I know for sure that Jetty works this way, but I've not investigated more. It's hard for me to imagine others working much differently). 

I'm currently making an effort to cleanly support either java serialization or Kryo serialization via Twitter chill. They each have advantages and disadvantages for serializing session state into a servlet cluster. Java serialization is a bit brittle, but at least you're going to be quite aware of what is and isn't getting serialized. Kryo (and chill in particular) handle Scala details much better, but because it's more aggressive with serialization you could potentially be putting more objects on the wire than you intended. Since they both have their merits, I intend to support either as long as I don't hit any blocking issues.

It turns out that some of the neato tricks that the Scala compiler pulls to work on the JVM doesn't play well with Java serialization. In particular I've discovered that traits do not deserialize cleanly. That is, even though it has readObject appropriately defined (a method that is supposed to be called during the deserialization process as detailed here), the readObject method is never invoked. Hence if you have a trait such as Loggable, any transient fields are left null. (as a side note about Kryo/chill, it will reinitialize the fields to the original constructed values. It's quite ideal in that regard)

The best solution I've been able to think of is to expose a protected init function in any traits which have transient fields. Anything serializable which extends the interface will need to implement its own readObject can call up to the init function. 

Allow me to show it in code: 

Loggable will look like this: 

trait Loggable {
  @transient private[this] var _logger: Logger = null

  /**
    * Initializes the logger field. This is exposed for subclasses to invoke during deserialization to work
    * around java serialization limitations with Scala traits.
    */
  protected def loggableInit(): Unit = _logger = Logger(this.getClass)
  loggableInit()

  protected def logger = _logger
}

So now rather than having a protected logger value, it has a private logger field with the loggableInit() function which can be called to initialize the _logger field. You can see it is immediately invoked after definition to have the field initialized in the object constructor. 

Then a subclass like I have in lift-ng can be serializable with the Loggable trait like the following:

class JsObjFactory() extends Factory with Serializable with Loggable {
  @transient private var promiseMapper = DefaultApiSuccessMapper

  @throws(classOf[IOException])
  @throws(classOf[ClassNotFoundException])
  private[this] def readObject(in: ObjectInputStream): Unit = {
    promiseMapper = DefaultApiSuccessMapper
    loggableInit()
  }

  // Other irrelevant code below ... 
}

Here you can see I have my own transient field which I handle in readObject, and I'm able to tell the logger to initialize as well.

Another approach I have considered is making logger always reinitialize itself anytime it finds the logger is null, but I feel that would need to be thread-safe and produce a lot of unnecessary overhead for the 99.999% of the time it's not a previously deserialized object. 

Does anyone have some insight or opinions on the approach I'm currently considering? 

Thanks!

Joe

 

[Matt Farmer]

This is tangental, but I'm strongly of the opinion that we shouldn't change framework code to make Java serialization work for arbitrary classes, at least in part because using Java serialization is a practice we should be discouraging. The other part being that we're inheriting a lot of boilerplate that would be annoying to impossible to test well. Further, Java serialization is notoriously finicky about any kind of changes to the class and it makes doing rolling releases of new code virtually impossible. 

My experience with Kryo has been mildly better. Given that this is a volunteer project for most of us, and our time is limited, I'm inclined to say that Kryo serialization should be our blessed way of doing this. I have some concerns about how Kryo keeps track of what it serializes - as best I understand it there some internal integer offset that keeps things straight - but it still seems far more capable for this kind of work than Java serialization.

That said, I'm somewhat skeptical of fork lifting classes over to a new copy of an application as-is. Could we, instead, transmit enough information using JSON to rebuild the desired state instead of transmitting that state itself? Such a design would be much more resilient to rolling releases and, I think, handling the failure conditions would be more explicit for the actual application developer.

--
--
Lift, the simply functional web framework: http://liftweb.net
Code: http://github.com/lift
Discussion: http://groups.google.com/group/liftweb
Stuck? Help us help you: https://www.assembla.com/wiki/show/liftweb/Posting_example_code

---
You received this message because you are subscribed to the Google Groups "Lift" group.
To unsubscribe from this group and stop receiving emails from it, send an email to liftweb+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[j...@joescii.com]

Thank you for your affirming feedback Matt. :)

So yeah, as I continued working with Kryo I came to the same conclusion you did. Things got worse I continued to pull on the java serialization thread. I had a lot of experience with it prior to learning Scala and felt pretty comfortable with it. I didn't have a good appreciation for how poorly Scala objects behave because of how much java serialization is coupled to language semantics. Last week I started experimenting with pushing the limits on Kryo and got pretty good results from it.

Before I go there... I have to say while I understand your skepticism about "fork lifting classes over to a new copy of the application as-is", the alternatives are not very practical I found. Firstly, an ascii-based serialization like JSON will never work because much of the session state is stored in the form of lambdas. My first attempt at this (regardless of serialization library) was to try and find the important pieces inside LiftSession. Unfortunately it turned into a never ending hunt for the important pieces. It's doable, but requires a good bit of tedious and hard to test code. Go back and take a look at the branch/PR. You had concerns :) 

What's interesting is that with some tweaks I've been able to get the entire LiftSession to serialize with much less code thanks to Kryo. It has some drawbacks in its current implementation (just trying to prove it right now), but I think I can get it down to mostly sprinkling in @transient and private default constructors. 

Given that, back drop. A possible path is to make these refactorings in the Lift code base which should have no impact to non-cluster users. The session management and serialization can be implemented as a Lift module. This would also allow us to publish the code using the Kryo library without adding it as a dependency for Lift itself. The fact that it can be done as an optional model is the most appealing part of this idea. In my mind, I just have to see how minimally I can refactor LiftSession etc to behave well.

Any thoughts on what I've discovered recently?

Joe

[Antonio Salazar Cardozo]

This would also allow us to publish the code using the Kryo library without adding it as a dependency for Lift itself. The fact that it can be done as an optional model is the most appealing part of this idea.

This sounds excellent to me. Started getting worried when a new library was bandied about :)

I agree that there is room for considerable pain with binary serialization, but am convinced that the

goal it's facilitating is useful, and that if we succeed at that goal the pain will be worth it.

Thanks,

Antonio

[Matt Farmer]

Hah, I feel like I'm starting to lose my mind. That previous PR had totally slipped my memory.

I'm good with the lift module plan. I think that this will present a lot of value. The only things I can think of that would give me hesitation are:

• How code behaves when a new version of the framework is deployed to an app
• How code behaves when a totally incompatible change is made to an app and then deployed
• How code behaves when the lamba we serialize closes over some external value

I think the answers to these concerns are probably unchangeable - but we'll deliver a much better experience by making them a part of a walkthrough of how to use the code and setting the expectations of the developers accordingly.

TL;DR - I'm on board with kryo serialization. Build dat module. :)

[Janos Lele]

Hi Guys/Joe,

Hope I’m writing to the right place and you can help me.

Just started using Lift-NG (which is awesome BTW, thanks for pulling it together!),

managed to configure it, and now I’d like to use an external Angular module, namely Smart Table (http://lorenzofox3.github.io/smart-table-website/) using its webjar (using maven BTW), so configured the following in my pom:

…

<dependency>

<groupId>net.liftweb</groupId>

<artifactId>lift-webkit_2.11</artifactId>

<version>3.1.0</version>

<scope>provided</scope>

</dependency>

<dependency>

    <groupId>net.liftmodules</groupId>

    <artifactId>ng_3.1_2.11</artifactId>

    <version>0.10.1</version>

</dependency>

<!-- https://mvnrepository.com/artifact/net.liftmodules/ng-js_2.6_2.11 -->

<dependency>

    <groupId>net.liftmodules</groupId>

    <artifactId>ng-js_2.6_2.11</artifactId>

    <version>0.2_1.3.1</version>

</dependency>

<!-- https://mvnrepository.com/artifact/org.webjars.npm/angular -->

<dependency>

    <groupId>org.webjars.npm</groupId>

    <artifactId>angular</artifactId>

    <version>1.6.5</version>

</dependency>

<!-- https://mvnrepository.com/artifact/org.webjars.npm/angular-smart-table -->

<dependency>

    <groupId>org.webjars.npm</groupId>

    <artifactId>angular-smart-table</artifactId>

    <version>2.1.6</version>

</dependency>

/*

configured 'includeAngularJs = true,’

in ‘net.liftmodules.ng.Angular.init'

in Boot.scala

and I just add the following in my .html file:

<script data-lift="Angular"></script>

and Angular works perfectly, yet

*/

I cannot work out how to configure Smart Table, i.e.

Do I need Scala code? Or just need to add some <script> tag in my .html file?

If you could help me or direct me to some doc that would be awesome.

Or/and if you can recommend a better way / angular module for displaying tables, please let me know!

Keep up the good work,

Thanks for any help in advance,

Janos

[Janos Lele]

Hi Guys,

Just a bit of more info:

The angular module is configured using:

angular.module(‘Foo', ['smart-table’])…

So the only thin I would need is how to make the smart-table.js resource available so that it can be ‘included’ in the .html - I guess.

So the alternative of '<script data-lift="Angular"></script>’ for Smart Table.

But please feel fee to correct me if I’m wrong - I’m quite new to Angular and webjars as well.

The error message I’m getting in the browser BTW:

Error: [$injector:modulerr] Failed to instantiate module Foo due to:

[$injector:modulerr] Failed to instantiate module smart-table due to:

[$injector:nomod] Module 'smart-table' is not available! You either misspelled the module name or forgot to load it

Thanks a lot for any help:

Janos

[Peter Petersson]

Hi,
No Scala code needed for that, you just have to add the script resources and reference it.

Best regards Peter Petersson

[Janos Lele]

Hi Peter,

Thanks a lot for the answer!

I’ve tried adding:

<script src="/webjars/angular-smart-table/2.1.6/dist/smart-table.js"></script>

even

<script src="webjars/angular-smart-table/2.1.6/dist/smart-table.js"></script>

But no luck so far.

(Error message is the same:

[$injector:modulerr] Failed to instantiate module smart-table due to:

[$injector:nomod] Module 'smart-table' is not available! You either misspelled the module name or forgot to load it... )

Was wondering if I need to update my web.xml?

Currently it looks like:

<web-app>

<filter>

<filter-name>LiftFilter</filter-name>

<display-name>Lift Filter</display-name>

<description>The Filter that intercepts lift calls</description>

<filter-class>net.liftweb.http.LiftFilter</filter-class>

</filter>

<filter-mapping>

<filter-name>LiftFilter</filter-name>

<url-pattern>/*</url-pattern>

</filter-mapping>

</web-app>

And how does Lift know what resources should be made available?

No additional config in Boot.scala is needed e.g.?

What would be the framework specific settings like for the others:

http://www.webjars.org/documentation#play2

Any help/pointers much appreciated,

Thanks,

Janos

[j...@joescii.com]

Thanks for the feedback as always Antonio and Matt. 

Matt: To answer your excellent questions, the first two are bad things one shouldn't do with a clustered webapp for the reasons I'm sure you're able to imagine. The cluster should ONLY contain webapps with the same version of code. Deployment of new code should canary (or whatever) to the new cluster. All of this of course will the outlined very clearly with in the module's README.

As for your third question about lambdas closing, I'm still experimenting with the behavior of kryo... but it seems to *just work*. The good news is the client I'm working with right now has a pretty big Lift app that is going to allow me a LOT of valuable testing to prove this out.

I'm still quite anxious about finding a solid solution to this as it is an extremely complicated problem to solve. But I'm optimistic that if there is a reasonable solution, we're finally circling around it.

Joe

[Peter Petersson]

Hi

There are a lot of ways to add resources to Lift but one easy way to get started is to download the resources (npm install or what ever method you like) then copy them to a folder (like assets/js/... assets/css...) inside of your /webapp folder (like you would do with simple app specific .css files).

In your "default.html" then add a script tag like this

<script src="/assets/js/smart-table.js"></script>

Hope that help you get going, then you can explore other ways to add resources.

Best regards Peter Petersson

[Matt Farmer]

Awesome thanks for those answers! Excited to see some progress!

[Janos Lele]

Hi Peter,

Awesome, this is great help!

I’ll pass on webjars then.

Meanwhile I found Joe’s and Tim’s solutions and the discussion on https://groups.google.com/forum/#!topic/liftweb/6pZ5YggJtK8

Though the records are quite old but I guessed there’s no built-in support for webjars in Lift (yet).

Anyway, Lift still rocks!

Keep up the good work!

All the best and thanks again,

Janos