Serialization and deserialization of org.jpmml.evaluator.Evaluator

[Sashikanth Chandrasekaran]

Hi,

We need to serialize (and deserialize) the Evaluator we create using the LoadingModelEvaluatorBuilder. We tried Kyro serialization and while Kyro is optimized, we would like to further reduce the cost (both space and time) of serialization and deserialization.

It is not a spark application, so we can't use JPMML-Spark

Can we write a custom serializer that saves the parts of the Evaluator that is expensive to recreate from the PMML and perhaps recreates the other parts to rebuild the Evaluator? Anyone have experience with serializing Evaluator in a non-spark latency sensitive application?

Thanks,

-sashi.

[Villu Ruusmann]

Hi Sashi,

> We need to serialize (and deserialize) the Evaluator
> we create using the LoadingModelEvaluatorBuilder.
>

There's an old issue (closed as "won't fix") regarding Kryo serialization:
https://github.com/jpmml/jpmml-evaluator/issues/114

Did Kryo work with the latest JPMML-Evaluator version 1.5.X now? Were
you using the default serializer, or did you write your own(s)?

All important classes in the JPMML software stack should be Java
serializable (ie. implementing java.io.Serializable interface). What;s
wrong with that, how much faster do you need?

> Can we write a custom serializer that saves the parts
> of the Evaluator that is expensive to recreate from the PMML
> and perhaps recreates the other parts to rebuild the Evaluator?
>

The current technical design has the following layers:

1) abstract base class org.jpmml.evaluator.ModelManager. Holds the
org.dmg.pmml.PMML class model object, and a reference to an
org.dmg.pmml.Model contained therein:
https://github.com/jpmml/jpmml-evaluator/blob/1.5.1/pmml-evaluator/src/main/java/org/jpmml/evaluator/ModelManager.java#L70-L72

The extra state consists of various indices:
https://github.com/jpmml/jpmml-evaluator/blob/1.5.1/pmml-evaluator/src/main/java/org/jpmml/evaluator/ModelManager.java#L74-L90

2) abstract base class org.jpmml.evaluatorModelEvaluator. Holds
evaluation configuration (input and result field aliases, output field
filters, reporting configuration etc):
https://github.com/jpmml/jpmml-evaluator/blob/1.5.1/pmml-evaluator/src/main/java/org/jpmml/evaluator/ModelEvaluator.java#L69-L75

3) concrete classes org.jpmml.evaluator.<model_type>.<Model>Evaluator.
Hold model type-specific indices.

Speaking about SerDe, then IMHO you should design your approach around
shipping the org.dmg.pmml.PMML class model object. All helper state
(eg. index of fields by name, as a java.util.Map) are very
cheap/lightweight compared to that.

Read an o.d.p.PMML class model object from the PMML document, and see
which SerDe approach works the best. Please note that right after XML
unmarshalling this is a graph of unique objects, but when you apply
one or more Visitors to it, its properties may change quite a bit,
affecting SerDe performance.

For example, if you apply
org.jpmml.evaluator.visitors.AttributeInternerBattery and
ElementInternerBattery to it, then you'd be replacing duplicate
objects with a reference to the "master copy" object. A SerDe approach
that works best for a non-interned PMML class model object may fall to
the last position for a heavily interned one, and vice versa.

Please share your results and observations once you have them!


VR

[Sashikanth Chandrasekaran]

Hi Villu,

Thank you once again for the detailed explanation.

Kyro did not work with JPMML 1.5.1. I have pasted the stack trace below. We were able to workaround the error by extending the Kyro Serializer. The custom serializer uses Java serialization. 

Serializing o.d.p.PMML using Java serialization also works and was indeed the first thing we tried. We are using JPMML from within another framework, and we have not been able to identify the root cause of the slow down. We observed a slow down when we serialized and deserialized PMML frequently (may have been a GC issue).

-sashi.

Serialization trace:

dataFields (org.jpmml.evaluator.mining.MiningModelEvaluator)

...

at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)

at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)

at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679)

at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)

at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)

at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679)

at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)

at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)

at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)

...

Caused by: java.lang.UnsupportedOperationException: null

at java.util.Collections$UnmodifiableMap.put(Collections.java:1457)

at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:144)

at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:21)

at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679)

at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)

--

-sashi.

[Villu Ruusmann]

Hi Sashi,

>
> Kyro did not work with JPMML 1.5.1.
> I have pasted the stack trace below.
>

This UnsupportedOperationException is caused by the fact that Kryo is
trying to deserialize an immutable Guava collection (here,
ImmutableMap) using JDK collection deserializers.

I tried configuring Kryo with
https://github.com/magro/kryo-serializers, but it didn't solve this
issue - something is broken there.

However, I did succeed serializing/deserializing complete
org.dmg.pmml.PMML class model objects using the following code:
Kryo kryo = new Kryo();
kryo.setRegistrationRequired(false);
kryo.register(ComplexArray.class, new JavaSerializer());
kryo.register(FieldName.class, new JavaSerializer());
kryo.register(JAXBElement.class, new JavaSerializer());

Most org.dmg.pmml.* classes can be regarded as data transfer classes
(ie. hold state in the form of Java primitive values), and are
therefore rather suitable for transferring via Java or Kryo
serialization. Whenever there's a "difficult" class around, such as
the enum-like org.dmg.pmml.FieldName, simply map it to a
com.esotericsoftware.kryo.serializers.JavaSerializer.

> We were able to workaround the error by extending
> the Kyro Serializer. The custom serializer uses Java serialization.
>

Do you have some code to share?

When it comes to transferring complex (ie. non-data transfer classes)
classes such as org.jpmml.evaluator.ModelManager and
o.j.e.ModelEvaluator subclasses, then I'd probably map them all to
JavaSerializer as well.

kryo.register(MiningModelEvaluator.class, new JavaSerializer());
kryo.register(TreeModelEvaluator.class, new JavaSerializer());

The thinking goes that performance is gained/lost around transfering
the central org.dmg.pmml.PMML object, which should be given to Kryo.
All the surrounding helper state can be left to Java serialization.
Perhaps java.io.Serializable is a bit more robust in handling all
those immutable collections, caches and (anynoymous-) inner classes.

I did get the native Kryo serialization working (to a degree) for
o.j.e.ModelEvaluator subclasses. I didn't perform any exact
measurements, but my perception was that while the Kryo serialization
of o.d.p.PMML objects was "fast", then the Kryo serialization of
o.j.e.ModelEvaluator objects was "slow" - the JPMML-SkLearn
integration testing suite ran something like 3-5x longer.

What to make of this thread? Perhaps the JPMML-Evaluator should
provide an utility method for properly configuring a Kryo instance?


VR

[Sashikanth Chandrasekaran]

Hi Villlu,

Regarding the UnsupportedOperationException: We are getting this stack when we try to serialize ModelEvaluator in Apache Flink.

I just found that Flink uses its own type serializer and falls back to Kryo - so perhaps the "UnsupportedOperationException" happens when there Flink has "partially" deserialized the object and delegated to Kryo for specific fields - I am not sure, I could not reproduce the stack with a standalone program outside Flink.

To summarize: my observations are the same as yours:

1. o.d.p.PMML can be serialized with Kryo if we use kryo.register to use JavaSerializer for FieldName and other types.
2. MiningModelEvaluator can be serialized with JavaSerializer. The custom serializer we wrote was very similar to JavaSerializer. I could replace our custom code with JavaSerializer and it worked.
3. For our sample PMML document and sample Evaluator, it was about 10% faster to serialize and deserialize the MiningModelEvaluator class instead of serializing and deserializing o.d.p.PMML and recreating the Evaluator.

It could help if JPMML Evaluator has a utility method for working with Kryo. Or, some documentation would also suffice.

Thank you for the great support!

-sashi.

--

-sashi.

[Villu Ruusmann]

Hi Sashi,

Thanks for making me think about serializability in the JPMML software stack!

The picture is rather clear with the JPMML-Model base layer - all PMML
class model classes (ie. org.dmg.pmml.PMML itself, and everything that
can be put in it) should be regarded as value object/data transfer
classes, which can be read/written using Java primitives only (eg.
read/write int, double, String, Number).

The situation might become more complicated after the PMML class model
has been transformed, which typically happens by applying Visitor API
to it. For example, interning repeated class elements, pre-parsing
String values, introducing custom class model classes (eg. replacing
org.dmg.pmml.DataField instances with
org.jpmml.evaluator.RichDataField instances). I'm currently unsure if
interning is a significant transformation from Kryo perspective or
not. Perhaps a full serialize-deserialize roundtrip will destroy the
interning effect, because Kryo deserializer allocates a new
org.dmg.pmml.Predicate or o.d.p.ScoreDistribution for every
re-encounter? Needs investigation.

The picture is much more "fluid" with the JPMML-Evaluator layer. Code
inspection shows that some org.jpmml.evaluator.ModelManager and
o.j.e.ModelEvaluator fields are serializable, whereas others are
transient:
https://github.com/jpmml/jpmml-evaluator/blob/1.5.1/pmml-evaluator/src/main/java/org/jpmml/evaluator/ModelManager.java#L92-L105

I can't recall the original technical decision between this
non-transient/transient distinction, but if I had to explain it now,
then it must be about primary state (eg. ModelManager#pmml) vs derived
secondary state (eg. ModelManager#visibleFields). The derived
secondary state can vary depending on configuration (eg.
ModelEvaluator#configuration). Basically, "why persist the secondary
state, while it can be lazily computed and cached from the primary
state?"

However, when the SerDe performance is critical, then it would be
highly desirable to avoid the computation of the secondary state
completely. This would necessitate removing the "transient" modifiers
from many (if not all) o.j.e.ModelManager and o.j.e.ModelEvaluator
fields.

> For our sample PMML document and sample Evaluator,
> it was about 10% faster to serialize and deserialize the
> MiningModelEvaluator class instead of serializing and deserializing
> o.d.p.PMML and recreating the Evaluator.
>

Do you know, if you map MiningModelEvaluator to Kryo's JavaSerializer,
then does JavaSerializer handle the full object graph using
java.io.Serializable, or is it doing a mix of java.io.Serializable and
com.esotericsoftware.kryo.KryoSerializable? In other words, is the
enclosed big and heavy o.d.p.PMML instance transferred using Java or
Kryo serialization.

A mere 10% performance difference suggests that the o.d.p.PMML object
is still being transferred using Java serialization?

> It could help if JPMML Evaluator has a utility method
> for working with Kryo. Or, some documentation would also suffice.
>

I'm going to introduce a org.jpmml:pmml-model-kryo module into the
JPMML-Model library soon, which should provide easy API for getting
the transfer of PMML class model objects right (including details such
as interned vs non-interned object graph).

Still need to think about making the JPMML-Evaluator better
interoperable with Kryo:
1) Carefully revise the use of the "transient" keyword. Avoid
recomputing the state, which is likely to be identical/stable between
the master object instance and its copies.
2) Provide specialized Kryo serializers for all ModelManager
subclasses. I don;t particularly like the idea of implementing
c.e.k.KryoSerializable interface directly, because it would lead to a
direct coupling with the Kryo library (although the license terms and
conditions of the Kryo library seem supportive of my business model).
Perhaps there will be a separate org.jpmml:pmml-evaluator-kryo module
also.

The goal of the exercise should be to improve the SerDe performance 3-10x.


VR

[Villu Ruusmann]

Hi Sashi,

I've added org.jpmml:pmml-model-kryo SerDe module to the JPMML-Model
library. It provides org.jpmml.model.kryo.KryoUtil utility class,
which should take care of most common use cases:
https://github.com/jpmml/jpmml-model/blob/master/pmml-model-kryo/src/main/java/org/jpmml/model/kryo/KryoUtil.java

I'm using the following Kryo configuration in my uni tests:
https://github.com/jpmml/jpmml-model/blob/master/pmml-model-kryo/src/test/java/org/jpmml/model/kryo/KryoUtilTest.java#L21-L31

If the unmarshalled PMML XML document contains an InlineTable element
(eg. model verification dataset), then Kryo config needs to contain a
mapper for the org.w3c.dom.Element class.

Now, we should figure out if Kryo serialization has any real benefits
over the standard Java serialization in the JPMML software stack. A
relevant benchmark would be to choose a number of representative PMML
objects, and make clones of them using both
org.jpmml.model.SerializationUtil#clone(Object) and
org.jpmml.model.kryo.KryoUtil#clone(Kryo, Object) utility methods:
https://github.com/jpmml/jpmml-model/blob/master/pmml-model/src/main/java/org/jpmml/model/SerializationUtil.java#L94-L107
https://github.com/jpmml/jpmml-model/blob/master/pmml-model-kryo/src/main/java/org/jpmml/model/kryo/KryoUtil.java#L25-L40

Does Kryo beat plain old Java by a sufficient margin (>3X) at all? If
not (and we can't find a way to improve it), then there's perhaps no
point in trying to rework the JPMML-Evaluator library at all.


VR

[Villu Ruusmann]

Hi Sashi,

> I am not familiar with Kryo but a cursory look at the JavaSerializer
> suggests that when we map MiningModelEvaluator to Kryo's JavaSerializer,
> Kryo's JavaSerializer handles the full object graph using java.io.Serializable.
>

I've got the same impression about JavaSerializer.

However, maybe there's a KryoJavaHybridSerializer available somewhere,
which is able to toggle between serialization engines for different
object instances in the object graph.

I ran some crude experiments yesterday, and my readings suggest that
Kryo serialization (as implemented in
org.jpmml.model.kryo.KryoUtil#clone(Kryo, Object)) is roughly 2x
faster than Java serialization (as implemented in
org.jpmml.model.SerializationUtil#clone(Object)) when working with
org.dmg.pmml.PMML class model objects.

I have a feeling that it should be possible to improve from this level
by writing custom Kryo serializers for most common PMML class model
classes. For example, when serializing decision tree ensemble models,
then the most frequent element types are Node, SimplePredicate,
SimpleSetPredicate, ScoreDistribution. These classes have fairly rigid
internal structure, and it should be possible to write "optimizing"
Kryo serializers for them. For example, predicates typically involve a
field name reference in the form of an org.dmg.pmml.FieldName object.
An optimizing serializer could write it out as an integer (referencing
some constant table), instead of redirecting to JavaSerializer, which
would write it out as a String.

Should write some custom Kryo serializers manually, and if they prove
useful, could auto-generate them for all class model classes during
the project build.

I checked Kryo documentation some more, and Kryo seems to have API
switches for toggling the serialization of transient fields. If you
activate it, then org.jpmml.evaluator.ModelEvaluator instances should
be transferred in their current "fully-initialized" form, which should
eliminate any warm-up costs on the receiving end.


VR

[Villu Ruusmann]

Hi Sashi,

I've added full support for Kryo serialization into JPMML-Model and
JPMML-Evaluator projects now (released yesterday and today,
respectively, as 1.5.3 versions).

Here's how it works with the JPMML-Evaluator library:

1) Add org.jpmml:pmml-evaluator-kryo dependency to your project.
2) Import the org.jpmml.evaluator.kryo.KryoUtil utility class, and use
its #init(Kryo) and #register(Kryo) utility methods to initialize a
Kryo instance:

Kryo kryo = new Kryo();

KryoUtil.init(kryo);
KryoUtil.register(kryo);

See https://github.com/jpmml/jpmml-evaluator/blob/1.5.3/pmml-evaluator-kryo/src/main/java/org/jpmml/evaluator/kryo/KryoUtil.java

3) Use this Kryo instance to read/write ModelEvaluator instances as needed:

Evaluator evaluator = ...
kryo.writeClassAndObject(new Output(...), evaluator);

Evaluator clonedEvaluator = (Evaluator)kryo.readClassAndObject(new Input(...));

---

The JPMML-Evaluator version 1.5.3 mostly about improved Java/Kryo
serialization. I thought long and hard, but couldn't find any
justification for marking some ModelEvaluator instance fields with the
"transient" modifier - so they were removed:
https://github.com/jpmml/jpmml-evaluator/commit/e0c81d60448fc203a252d3f4bb5e13435e2470c0

It means that serialization now preserves a ModelEvaluator instance in
its last known state. For example, if you "warmed up" model evaluator
by invoking ModelEvaluator#verify(), then a full
serialization/deserialization cycle will give you a "warm" model
evaluator back (previously you'd gotten back a "cooled down" variant).
So, the time from deserialization to first prediction should be
considerably shorter now.


VR

[Sashikanth Chandrasekaran]

Hi Villu,

I tried org.jpmml:pmml-evaluator-kryo with 1.5.3 and see a visible improvement in a microbenchmark compared to the default serialization of the Evaluator object.

-sashi.

--

-sashi.

[Sashikanth Chandrasekaran]

Hi Villu,

Was there an incompatible change from 1.5.1 to 1.5.3? I am getting the following exception. Wanted to check if there are known incompatibilities before digging further.

Thanks!

-sashi.

org.jpmml.evaluator.TypeCheckException: Expected float value, got string value

at org.jpmml.evaluator.TypeUtil.toFloat(TypeUtil.java:702)

at org.jpmml.evaluator.TypeUtil.cast(TypeUtil.java:531)

at org.jpmml.evaluator.tree.SimpleTreeModelEvaluator.evaluateAny(SimpleTreeModelEvaluator.java:97)

at org.jpmml.evaluator.tree.SimpleTreeModelEvaluator.evaluateRegression(SimpleTreeModelEvaluator.java:80)

at org.jpmml.evaluator.ModelEvaluator.evaluateInternal(ModelEvaluator.java:474)

at org.jpmml.evaluator.mining.MiningModelEvaluator.evaluateSegmentation(MiningModelEvaluator.java:543)

at org.jpmml.evaluator.mining.MiningModelEvaluator.evaluateRegression(MiningModelEvaluator.java:249)

at org.jpmml.evaluator.ModelEvaluator.evaluateInternal(ModelEvaluator.java:474)

at org.jpmml.evaluator.mining.MiningModelEvaluator.evaluateInternal(MiningModelEvaluator.java:242)

at org.jpmml.evaluator.mining.MiningModelEvaluator.evaluateSegmentation(MiningModelEvaluator.java:543)

at org.jpmml.evaluator.mining.MiningModelEvaluator.evaluateClassification(MiningModelEvaluator.java:309)

at org.jpmml.evaluator.ModelEvaluator.evaluateInternal(ModelEvaluator.java:477)

at org.jpmml.evaluator.mining.MiningModelEvaluator.evaluateInternal(MiningModelEvaluator.java:242)

at org.jpmml.evaluator.ModelEvaluator.evaluate(ModelEvaluator.java:306)

--

-sashi.

[Villu Ruusmann]

Hi Sashi,

> Was there an incompatible change from 1.5.1 to 1.5.3?

There was an internal API change, which was supposed to be invisible
to end users. Namely, there are two tree model evaluators available
now - one new/lightweight (intended nested tree models such as the
members of a ensemble model) and one classical/heavy-weight (intended
for top-level tree models).

1) New: org.jpmml.evaluator.tree.SimpleTreeModelEvaluator
2) Old: org.jpmml.evaluator.tree.ComplexTreeModelEvaluator

> I am getting the following exception.

> org.jpmml.evaluator.TypeCheckException: Expected float value, got string value
> at org.jpmml.evaluator.TypeUtil.toFloat(TypeUtil.java:702)
> at org.jpmml.evaluator.TypeUtil.cast(TypeUtil.java:531)
> at org.jpmml.evaluator.tree.SimpleTreeModelEvaluator.evaluateAny(SimpleTreeModelEvaluator.java:97)
>

I ran into this exception two days ago myself while investigating
https://github.com/jpmml/sklearn2pmml/issues/245.

The fix is straightforward:
https://github.com/jpmml/jpmml-evaluator/commit/46453afd954d74b69da590fc54e4a17604a9bc33

The strange thing is that this exception didn't occur when I was
running all my integration tests (JPMML-SkLearn, JPMML-R,
JPMML-SparkML, and all their transitive dependencies). Should
investigate some more what's so special about this particular XGBoost
configuration.

Anyway, the above is a serious matter, which warrants a new
JPMML-Evaluator release ASAP. I hope to have it ready (will be called
1.5.4 then) by Monday morning.


VR

[Villu Ruusmann]

Hi Sashi,

> Was there an incompatible change from 1.5.1 to
> 1.5.3? I am getting the following exception.
>

> org.jpmml.evaluator.TypeCheckException: Expected float value, got string value
> at org.jpmml.evaluator.TypeUtil.toFloat(TypeUtil.java:702)
> at org.jpmml.evaluator.TypeUtil.cast(TypeUtil.java:531)
> at org.jpmml.evaluator.tree.SimpleTreeModelEvaluator.evaluateAny(SimpleTreeModelEvaluator.java:97)
>

The above exception happens when the new/lightweight tree model
evaluator encounters a non pre-parsed PMML class model object.

This issue can be fixed by changing JPMML-Evaluator library code (as
shown in the commit a couple of days ago), but it would be much better
fixed in your own Java application code, by changing how models are
loaded from PMML files.

In brief, when a tree model data structure is loaded from a PMML file,
then the JAXB frameworks keeps all Node@score attribute values as
java.lang.String. When applying the
org.jpmml.evaluator.visitors.NodeScoreParser visitor class
(https://github.com/jpmml/jpmml-evaluator/blob/master/pmml-evaluator/src/main/java/org/jpmml/evaluator/visitors/NodeScoreParser.java)
to such tree model data structures, then all node scores will be
pre-parsed to java.lang.Number objects (eg. java.lang.Float for
XGBoost and java.lang.Double for Scikit-Learn).

The NodeScoreParser visitor class is applied applied automatically
when you load models using
org.jpmml.evaluator.LoadingModelEvaluatorBuilder:

ModelEvaluator evaluator = new LoadingModelEvaluatorBuilder()
.load(new File("xgboost.pmml"))
.build();

Now, the question becomes, what are you doing differently? Are you
loading the org.dmg.pmml.PMML object "manually" using eg. the
org.jpmml.model.JAXBUtil#unmarshal(InputStream) utility method, and
then applying a different set of visitor classes to it (that doesn't
include NodeScoreParser)?

Something like this:

PMML pmml = PMMLUtil.load(new File("xgboost.pmml"))
VisitorBattery visitors = new
org.jpmml.evaluator.visitors.DefaultModelEvaluatorBattery();
visitors.applyTo(pmml);
ModelEvaluator evaluator = new ModelEvaluatorBuilder(pmml)
.build();

If you happen to use o.j.e.ModelEvaluatorBuilder right now, then you
should definitely consider switching to
o.j.e.LoadingModelEvaluatorBuilder. This should reduce the need for
Java boilerplate code considerably, and ensure that all the latest
features and functionality are triggered in correct order.

By pre-parsing tree model data structures using
o.j.e.visitors.NodeScoreParser, you should see the following benefits:
1) Reduced memory consumption (heavy java.lang.String objects replaced
with much lighter java.lang.Number objects)
2) Reduced SerDe file size and processing times - this is what you're
really after!
3) Improved evaluation times (no need to parse java.lang.String
objects to numbers anymore).


Villu