[kryo] Kryo serializer handling of Circular references in Object graph

[Vootla]

Nate and others,

I'm trying to serialize a large object graph (running into GBs on the
disk with current Java serialization) and encounter stack overflows.
I'm wondering if this could be due to any circularity (cycles) in the
object graph. Does kryo and its serializers handle this already or do
we need to do do something to ensure that objects are serialized only
once in the graph.

Thanks,
-Prasad

--
You received this message because you are subscribed to the "kryo-users" group.
http://groups.google.com/group/kryo-users

[Martin Grotzke]

Hi Prasad,

On Fri, Apr 16, 2010 at 4:28 PM, Vootla <vvo...@gmail.com> wrote:
> Nate and others,
>
> I'm trying to serialize a large object graph (running into GBs on the
> disk with current Java serialization) and encounter stack overflows.
> I'm wondering if this could be due to any circularity (cycles) in the
> object graph.

Cycles are handled by java serialization, at least I haven't seen any
that were not detected. Perhaps the stack trace might help here.

> Does kryo and its serializers handle this already or do
> we need to do do something to ensure that objects are serialized only
> once in the graph.

Kryo comes with the ReferenceFieldSerializer that you need to return
in newDefaultSerializer:

Kryo kryo = new Kryo() {
protected Serializer newDefaultSerializer( final Class type ) {
return new ReferenceFieldSerializer( this, type );
}
};

Additionally it might make sense to write custom serializers for some
objects if they'd pull in stuff that you actually don't need.

I just wrote some custom serializer, you might have a look at
http://github.com/magro/kryo-serializers/tree/master/src/main/java/de/javakaffee/kryoserializers/

Cheers,
Martin

>
> Thanks,
> -Prasad
>
> --
> You received this message because you are subscribed to the "kryo-users" group.
> http://groups.google.com/group/kryo-users

--
Martin Grotzke
http://www.javakaffee.de/blog/

[Nate]

Hi Prasad,

This is a limitation of the current Kryo serializer implementations. The serializers use stack-based recursion, so the depth of your object graph is limited by the stack size. The easy fix is to increase the stack size via the "-Xss" JVM option. If you are running a server where many threads will be concurrently performing serialization of deep object graphs, this may not be a viable solution. In this case, you would need to rewrite ReferenceFieldSerializer to avoid using stack memory.

FWIW, built-in Java serialization has the same problem:
http://bugs.sun.com/view_bug.do?bug_id=4152790
At least with Kryo you have the option of implementing a serializer to solve it. :)

-Nate