serializing java.util.Arrays$ArrayList?

[Reynold Xin]

I am running into a problem that I have to serialize a java.util.Arrays$ArrayList. Kryo serializes it successfully, but when it is deserializing the java.util.Arrays$ArrayList, CollectionSerializer.java:113 throws a null pointer exception. Is there a way I can get around it? (perhaps some special serializer?)

Thanks.

It appears that in the following lines failed when kryo was trying to create a new collection. 

if (type == ArrayList.class)

collection = new ArrayList(length);

else

collection = (Collection)newInstance(kryo, type);

com.sun.jdi.InvocationException occurred invoking method.

--

Reynold Xin

Algorithms, Machines, People Lab | Database Group

Electrical Engineering and Computer Science, UC Berkeley

http://www.cs.berkeley.edu/~rxin/

[Reynold Xin]

I found a workaround by using 

import de.javakaffee.kryoserializers.ArraysAsListSerializer

kryo.register(Arrays.asList("dummy").getClass, new ArraysAsListSerializer(kryo))

Is there a more principled way to deal with this?

[Nate]

Hmm. I don't like that StdInstantiatorStrategy is used by default. I've changed this so only reflection is used by default, so it is less confusing. In your case Arrays$ArrayList is being instantiated without its constructor being called, and the class was not designed to handle this.

I've improved some things in trunk just now. ArraysAsListSerializer as it is on the kryo-serializers project could now implemented using:

static public class ArraysAsListSerializer extends FieldSerializer {
    public ArraysAsListSerializer (Kryo kryo, Class type) {
        super(kryo, type);
        getField("a").setSerializer(kryo.getArraySerializer());
    }
}
//...
kryo.setInstantiatorStrategy(new StdInstantiatorStrategy());
kryo.addDefaultSerializer(Arrays.asList().getClass(), ArraysAsListSerializer.class);

Note 1) this will only work with StdInstantiatorStrategy and 2) Arrays$ArrayList is a private class, so a different JRE implementation may not have this class at all.

Since the concrete type returned by Arrays.asList is private anyway, there is no reason we can't just use ArrayList. Then the solution becomes:

public class ArraysAsListSerializer extends CollectionSerializer {
    public ArraysAsListSerializer (Kryo kryo) {
        super(kryo);
    }

    public Collection create (Kryo kryo, Input input, Class type) {
        return new ArrayList();
    }
}

This is checked in and added by default as a default serializer. See DefaultSerializersTest#testArraysAsList() if needed.

-Nate

--
You received this message because you are subscribed to the "kryo-users" group.
http://groups.google.com/group/kryo-users

[Nate]

On a related note, I just added default serializers from kryo-serializers for these methods, so they will work out of the box:

Collections.emptyList
Collections.emptyMap
Collections.emptySet
Collections.singletonList
Collections.singletonMap
Collections.singleton

Thanks Martin! I looked at your other serializers again, but I don't think we should include the ones that work by accessing private fields.

-Nate

[Martin Grotzke]

On 04/17/2012 06:49 AM, Nate wrote:
> Since the concrete type returned by Arrays.asList is private anyway,
> there is no reason we can't just use ArrayList. Then the solution becomes:
>
> public class ArraysAsListSerializer extends CollectionSerializer {
> public ArraysAsListSerializer (Kryo kryo) {
> super(kryo);
> }
>
> public Collection create (Kryo kryo, Input input, Class type) {
> return new ArrayList();
> }
> }

An argument against this (and the reason why I used Arrays.asList in
ArraysAsListSerializer.create) is that Arrays.asList creates a fixed
size list, so that Arrays.asList().add("foo") throws an
UnsupportedOperationException.

If you use a j.u.ArrayList for deserialization behaviour is different
after deserialization.

Cheers,
Martin

[Martin Grotzke]

Great that you're pulling more serializers from kryo-serializers into core!

Cheers,
Martin

> the java.util.Arrays$__ArrayList, __CollectionSerializer.java:113

> throws a null pointer exception. Is there a way I can get
> around it? (perhaps some special serializer?)
>
> Thanks.
>
>
> It appears that in the following lines failed when kryo was
> trying to create a new collection.
>
> if (type == ArrayList.class)
>
> collection = new ArrayList(length);
>
> else
>
> collection = (Collection)newInstance(kryo, type);
>
>

> com.sun.jdi.__InvocationException occurred invoking method.

>
>
>
> --
> Reynold Xin
> Algorithms, Machines, People Lab | Database Group
> Electrical Engineering and Computer Science, UC Berkeley

> http://www.cs.berkeley.edu/~__rxin/
> <http://www.cs.berkeley.edu/%7Erxin/>

>
> --
> You received this message because you are subscribed to the
> "kryo-users" group.
> http://groups.google.com/group/kryo-users
>
>
>
> --
> You received this message because you are subscribed to the "kryo-users"
> group.
> http://groups.google.com/group/kryo-users

--
Brakhane, Grotzke und Langbehn Informatiker und Physiker PartG
Breitenfelder Str. 13c, 20251 Hamburg
Amtsgericht Hamburg, PR 795
http://inoio.de

[Nate]

Gah, you're right. Worse, an object graph may reference the wrapped array twice. For references to be written correctly, the wrapped array must be serialized itself and not just exposed via the Arrays$ArrayList. We could do something like this...

static public class ArraysAsListSerializer extends Serializer<List> {
    public void write (Kryo kryo, Output output, List object) {
        kryo.writeObject(output, object.toArray(new Object[object.size()]));
    }

    public List create (Kryo kryo, Input input, Class type) {
        return Arrays.asList(kryo.readObject(input, Object[].class));
    }
}

This sucks because it allocates during write, but is ultimately unacceptable because multiple references to the wrapped array won't work correctly. I'll remove ArraysAsListSerializer from the core then. Lesson learned: Arrays.asList sucks.

You could use my concise version of ArraysAsListSerializer that extends FieldSerializer in kryo-serializers if you like.

-Nate

[Martin Grotzke]

On 04/17/2012 09:26 AM, Nate wrote:
> Gah, you're right. Worse, an object graph may reference the wrapped
> array twice. For references to be written correctly, the wrapped array
> must be serialized itself and not just exposed via the Arrays$ArrayList.
> We could do something like this...
>
> static public class ArraysAsListSerializer extends Serializer<List> {
> public void write (Kryo kryo, Output output, List object) {
> kryo.writeObject(output, object.toArray(new Object[object.size()]));
> }
>
> public List create (Kryo kryo, Input input, Class type) {
> return Arrays.asList(kryo.readObject(input, Object[].class));
> }
> }
>
> This sucks because it allocates during write, but is ultimately
> unacceptable because multiple references to the wrapped array won't work
> correctly. I'll remove ArraysAsListSerializer from the core then. Lesson
> learned: Arrays.asList sucks.

Yes. And other stuff like List.subList as well.

>
> You could use my concise version of ArraysAsListSerializer that extends
> FieldSerializer in kryo-serializers if you like.

I think I'll do that.

Cheers,
Martin

[Reynold Xin]

Glad that my email provoked this discussion. :)

--

Reynold Xin

Algorithms, Machines, People Lab | Database Group

Electrical Engineering and Computer Science, UC Berkeley

http://www.cs.berkeley.edu/~rxin/

[Nate]

I don't have a clear idea of what you are doing. Can you provide a class with a main method that shows the error when run?

-Nate

On Sat, Aug 18, 2012 at 1:40 PM, Christopher fabri <christop...@gmail.com> wrote:

Hi - I'm experiencing a similar issue but am not quite sure how to resolve it.  My object is serializing but the deserialization fails with

com.esotericsoftware.kryo.KryoException: java.lang.NullPointerException

Serialization trace:

_patternList (com.elise.bitoflife.botworld.aiml.Pattern)

_pattern (com.elise.bitoflife.botworld.aiml.Category)

_categoryList (com.elise.bitoflife.botworld.aiml.Aiml)

_aimlList (com.elise.bitoflife.botworld.brain.Brain)

at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.read(FieldSerializer.java:536)

at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:221)

at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:651)

......

Caused by: java.lang.NullPointerException

at java.util.Arrays$ArrayList.size(Unknown Source)

at java.util.AbstractList.add(Unknown Source)

at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:104)

at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18)

at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:651)

at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.read(FieldSerializer.java:515)

I'm using kryo 2.19 - following the directions from  https://code.google.com/p/kryo/; 

my code looks like:

_kryo = new Kryo(); // ReflectionFactorySupport();

_kryo.register(Category.class);

....

output = new Output(new FileOutputStream(resourceLocation));

_kryo.writeObject(output, MyClass_ );

...

input = new Input(new FileInputStream(fileName));

b = _kryo.readObject(input, MyClass.class);

I've tried (separately)

_kryo.setInstantiatorStrategy(new StdInstantiatorStrategy());

or

_kryo.setInstantiatorStrategy(new SerializingInstantiatorStrategy());

but both lead to other, more cryptic errors - and the serialization fails.

also, I've tried

_kryo.register(Arrays.asList("").getClass(), new ArraysAsListSerializer(_kryo));

but that also fails - this class comes from .10 or .9 kryo-serializers

any suggestions as to how to resolve this issue - which appears to be around serializing/deserialziing Arrays$ArraysList  (apparently, the size of the list on creation)?

Any help appreciated.

CF>

[Martin Grotzke]

kryo-serializers are not yet updated for kryo2, but there's a branch for an older version of kryo2 and forks of that branch that should be more up2date. I'm currently (and the next 4 weeks) on vacation but will work on kryo-serializers when I'm back.

Cheers,
Martin

[Christopher fabri]

Hi Martin - yes, it seemed so - yet there is a link from the V2 page
to it - so I thought I would try it.

what are the correct versions to use for V2 kryo, associated V2 serializers?

tx.


On Sun, Aug 19, 2012 at 1:29 AM, Martin Grotzke

[Christopher fabri]

Hi Nate - my code uses Arrays.toList() in various places - the
question is - does kryo work out of the box serializing/deserializing
such code? If not, what needs to be done to make it work properly?

I've attached a working example that produces a different error upon
deserialization but still associated with Arrays$ArrayList and perhaps
symptomatic of the general issue:

09:41:43.203 [main] ERROR MyKryoTestMain : Exception during
kryo restoration e:Class cannot be created (missing no-arg
constructor): java.util.Arrays$ArrayList
Serialization trace:
_stringList (com.elise.bitoflife.botworld.MyPattern)
_mc (com.elise.bitoflife.botworld.MyCategory)
_list (com.elise.bitoflife.botworld.MyA):
09:41:43.218 [main] WARN MyKryoTestMain : a :null:
com.esotericsoftware.kryo.KryoException: Class cannot be created
(missing no-arg constructor): java.util.Arrays$ArrayList
Serialization trace:
_stringList (com.elise.bitoflife.botworld.MyPattern)
_mc (com.elise.bitoflife.botworld.MyCategory)
_list (com.elise.bitoflife.botworld.MyA)
at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1047)
at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1059)
at com.esotericsoftware.kryo.serializers.CollectionSerializer.create(CollectionSerializer.java:84)
at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:88)

at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18)
at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:651)
at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.read(FieldSerializer.java:515)

at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:221)
at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:651)

at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.read(FieldSerializer.java:515)
at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:221)
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:732)
at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:111)

at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18)
at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:651)
at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.read(FieldSerializer.java:515)

at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:221)
at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:629)
at com.elise.bitoflife.botworld.MyKryoTestMain.read(MyKryoTestMain.java:82)
at com.elise.bitoflife.botworld.MyKryoTestMain.main(MyKryoTestMain.java:40)

[Nate]

On Sun, Aug 19, 2012 at 5:47 AM, Christopher fabri <christop...@gmail.com> wrote:

Hi Nate - my code uses Arrays.toList() in various places - the
question is - does kryo work out of the box serializing/deserializing

uch code?  If not, what needs to be done to make it work properly?

No, it doesn't out of the box.

I've attached a working example that produces a different error upon
deserialization but still associated with Arrays$ArrayList and perhaps
symptomatic of the general issue:

Your code was more than I have time to dig into. Try this...

static public class ArraysAsListSerializer extends Serializer<List> {
    public void write (Kryo kryo, Output output, List object) {
        kryo.writeObject(output, object.toArray(new Object[object.size()]));
    }

    public List read (Kryo kryo, Input input, Class<List> type) {
        return Arrays.asList(kryo.readObject(input, Object[].class));
    }
}

public static void main (String[] args) throws Exception {
    Kryo kryo = new Kryo();
    kryo.register(Arrays.asList().getClass(), new ArraysAsListSerializer());
    Output output = new Output(1024);
    kryo.writeClassAndObject(output, Arrays.asList(1, 2, 3, 4, 5));
    Object object = kryo.readClassAndObject(new Input(output.toBytes()));
    System.out.println(object);
}

Note this won't correctly handle references to the Arrays.asList list. If you don't need references to this list to be serialized, then this could be acceptable.

-Nate

[Martin Grotzke]

Please check pull requests and forks of kryo-serializers (they should have set the version of used kryo2 in pom.xml). I'm sorry I can't rule this out for you as I'm on vacation with my phone only (and right now need to get on the road again).

Cheers,
Martin

[Christopher fabri]

Hi Nate - I replaced the use of Arrays.ToList(...) throughout the code and added _kryo.setInstantiatorStrategy(new SerializingInstantiatorStrategy());

serialization/deserialization now works on some instances of the object type I'm working with but not on other instances -  I don't see what could be different that might the cause of the issue.  The instances that work however do so consistently, similarly with those that fail.

I've attached kryo tracer output for a particular run where the serialization 'worked' but the deserialization fails with 

Exception during kryo brain restoration e:Unable to find class:XXXXcom.elise.bitoflife.botworld.aiml.Category

The class name reported has 4 'characters' in front of the 'com' which should not be there - they don't render in the post here (though they render in eclipse console, box with inverted c inside of that means anything to you) - I've just represented them here as 'X' so we can see them.  it looks like either the write or the read is corrupt somehow.

I walked through the FieldSerializer.read method; it first creates an instance of the main object by calling its zero-arg constructor - that works fine; it then tries to populate each field of that object.  The first field it tries to populate is an ArrayList which it loads using the data from the file, however, each object in the array get rendered as a class name prefixed with these various X characters - in each case, the class should be com.elise.bitoflife.botworld.aiml.Category but it's being loaded as XXXXcom.elise.bitoflife.botworld.aiml.Category, which then cannot be loaded, of course leading to the stack trace error of class not found.

any idea what is going on there?

any help appreciated.

CF>

[X com.elise.bitoflife.botworld.aiml.Aiml]

[Nate]

On Mon, Aug 20, 2012 at 8:37 AM, Christopher fabri <christop...@gmail.com> wrote:

Hi Nate - I replaced the use of Arrays.ToList(...) throughout the code and added _kryo.setInstantiatorStrategy(new SerializingInstantiatorStrategy());

serialization/deserialization now works on some instances of the object type I'm working with but not on other instances -  I don't see what could be different that might the cause of the issue.  The instances that work however do so consistently, similarly with those that fail.

I've attached kryo tracer output for a particular run where the serialization 'worked' but the deserialization fails with 

Exception during kryo brain restoration e:Unable to find class:XXXXcom.elise.bitoflife.botworld.aiml.Category

The class name reported has 4 'characters' in front of the 'com' which should not be there - they don't render in the post here (though they render in eclipse console, box with inverted c inside of that means anything to you) - I've just represented them here as 'X' so we can see them.  it looks like either the write or the read is corrupt somehow.

Sounds like something is being written but not read. To debug, you need to compare what is being written with what is being read. It can be a bit of a pain. Use Log.TRACE() first and compare write with read. Trace can't output everything though, so if you don't see a discrepancy there you have to step through the problematic serialization and deserialization. It is worth simplifying what you are serializing as much as possible so there is less code to step through. If you are still stuck, post the simple example.

-Nate