IndexOutOfBoundsException when attempting to read snapshot containing nested Objects

[Darren Bathgate]

Hello,

I am attempting to serialize and deserialize an Object that contains a list of sub Objects with the same type.

For example, I have an Object called Pojo that contains a List of sub Pojo's.

Writing a snapshot seems to work fine, but reading in the snapshot results in an IndexOutOfBoundsException.

Here is my code:

//  Pojo object

    private class Pojo {

        private String name;

        private List<Pojo> subPojos;

        public Pojo(String name) {

            this.name = name;

            this.subPojos = new ArrayList<Pojo>();

        }

 

        public Pojo(String name, List<Pojo> subPojos) {

            this.name = name;

            this.subPojos = subPojos;

        }

        public String getName() {

            return name;

        }

        public List<Pojo> getSubPojos() {

            return subPojos;

        }

    }

//  Serializer 

    private class PojoSerializer extends NFTypeSerializer<Pojo>{

        public PojoSerializer(String name) {

            super(name);

        }

        @Override

        protected void doSerialize(Pojo value, NFSerializationRecord rec) {

            serializePrimitive(rec, "name", value.getName());

            serializeObject(rec, "subPojos", value.getSubPojos());

        }

        @Override

        protected Pojo doDeserialize(NFDeserializationRecord rec) {

            return new Pojo(deserializePrimitiveString(rec, "name"), deserializeObject(rec, "subPojos"));

        }

        @Override

        protected FastBlobSchema createSchema() {

            return schema(

                    field("name", FieldType.STRING),

                    field("subPojos", "ListOfPojos")

                    );

        }

        @Override

        public Collection<NFTypeSerializer<?>> requiredSubSerializers() {

            return serializers(new ListSerializer<Pojo>("ListOfPojos", new PojoSerializer("SubPojo")));

        }

    }

//  Writing the snapshot

    Pojo pojo = new Pojo("Pojo", Lists.newArrayList(new Pojo("SubPojo")));

    Pojo pojo2 = new Pojo("Pojo2", Lists.newArrayList(new Pojo("SubPojo", Lists.newArrayList(new Pojo("SubSubPojo")))));

    

    FastBlobStateEngine engine1 = new FastBlobStateEngine(new SerializerFactory() {

        

        @Override

        public NFTypeSerializer<?>[] createSerializers() {

            return new NFTypeSerializer<?>[] {new PojoSerializer("Pojo")};

        }

    });

    

    FastBlobWriter writer = new FastBlobWriter(engine1);

    engine1.prepareForNextCycle();

    

    engine1.add("Pojo", pojo);

    engine1.add("Pojo", pojo2);

    

    engine1.prepareForWrite();

    

    try (DataOutputStream outputStream = new DataOutputStream(new FileOutputStream(new File("./pojos")))) {

        writer.writeSnapshot(outputStream);

    }

//  Reading the snapshot

    final FastBlobStateEngine engine2 = new FastBlobStateEngine(new SerializerFactory() {

        

        @Override

        public NFTypeSerializer<?>[] createSerializers() {

            return new NFTypeSerializer<?>[] {new PojoSerializer("Pojo")};

        }

    });

    final FastBlobReader reader = new FastBlobReader(engine2);

    try (DataInputStream inputStream = new DataInputStream(new FileInputStream(new File("./pojos")))) {

        reader.readSnapshot(inputStream);

    }

The IndexOutOfBoundsException is thrown upon calling reader.readSnapshot(...)

java.lang.IndexOutOfBoundsException: Index: 0, Size: 0

at java.util.ArrayList.rangeCheck(ArrayList.java:653)

at java.util.ArrayList.get(ArrayList.java:429)

at com.netflix.zeno.fastblob.state.FastBlobTypeDeserializationState.get(FastBlobTypeDeserializationState.java:66)

at com.netflix.zeno.fastblob.FastBlobFrameworkDeserializer.deserializeObject(FastBlobFrameworkDeserializer.java:308)

at com.netflix.zeno.fastblob.FastBlobFrameworkDeserializer.deserializeObject(FastBlobFrameworkDeserializer.java:281)

at com.netflix.zeno.fastblob.FastBlobFrameworkDeserializer.deserializeObject(FastBlobFrameworkDeserializer.java:48)

at com.netflix.zeno.serializer.NFTypeSerializer.deserializeObject(NFTypeSerializer.java:107)

at zeno.test.NestedPojoTest$PojoSerializer.doDeserialize(NestedPojoTest.java:70)

at zeno.test.NestedPojoTest$PojoSerializer.doDeserialize(NestedPojoTest.java:1)

at com.netflix.zeno.serializer.NFTypeSerializer.deserialize(NFTypeSerializer.java:61)

at com.netflix.zeno.fastblob.state.FastBlobTypeDeserializationState.add(FastBlobTypeDeserializationState.java:71)

at com.netflix.zeno.fastblob.io.FastBlobReader.readTypeStateObjects(FastBlobReader.java:200)

at com.netflix.zeno.fastblob.io.FastBlobReader.readSnapshotTypes(FastBlobReader.java:93)

at com.netflix.zeno.fastblob.io.FastBlobReader.readSnapshot(FastBlobReader.java:73)

at zeno.test.NestedPojoTest.test(NestedPojoTest.java:126)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:483)

The only solution I have come up with is to pre-instantiate sub serializers with unique names:

return serializers(new ListSerializer<Pojo>("ListOfPojos1", new PojoSerializer("SubPojo1")));

return serializers(new ListSerializer<Pojo>("ListOfPojos2", new PojoSerializer("SubPojo2")));

return serializers(new ListSerializer<Pojo>("ListOfPojos3", new PojoSerializer("SubPojo3")));

...

The only issue with this is that I am limited to a fixed number of nested objects.

Is there another way around this that I am overlooking?

[Jonathan Stockdill]

Do you get the exception only for empty lists?  The below code throws the error.

Does the error still get thrown if the list is null?

An example of the exception is below..

—jon

    @Test

    public void testIndex(){

        List<String> list = new ArrayList<>();

        list.get(0);

    }

--
You received this message because you are subscribed to the Google Groups "Netflix Zeno Discussion Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to netflix-zeno...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Darren Bathgate]

I should probably also note that this is netflix-zeno version 2.6

[Darren Bathgate]

When I attempted this issue with netflix-zeno 2.22, the result is that subPojos returns null.

I added debug log statements to FastBlobTypeDeserializationState on version 2.2.

public class FastBlobTypeDeserializationState<T> implements Iterable<T> {

     }

     public T get(int ordinal) {

-        if(ordinal >= objects.size())

+        System.out.println("attempting to get object at ordinal " + ordinal);

+        if(ordinal >= objects.size()) {

+            System.out.println("failed to get object at ordinal " + ordinal);

             return null;

+        }

+        System.out.println("obtained object at ordinal " + ordinal);

         return objects.get(ordinal);

     }

     @SuppressWarnings("deprecation")

     public void add(int ordinal, FastBlobDeserializationRecord rec) {

+        System.out.println("attempting to add object to ordinal " + ordinal);

         T obj = serializer.deserialize(rec);

         ensureCapacity(ordinal + 1);

+        System.out.println(obj);

         objects.set(ordinal, obj);

+        System.out.println("Object added at ordinal " + ordinal);

         stateListener.addedObject(obj);

         stateListener.addedObject(obj, ordinal);

     }

Which produced the following log lines:

attempting to add object to ordinal 0

attempting to get object at ordinal 0

failed to get object at ordinal 0

Pojo [name=SubPojo, subPojos=null]

Object added at ordinal 0

attempting to add object to ordinal 1

attempting to get object at ordinal 0

failed to get object at ordinal 0

Pojo [name=SubSubPojo, subPojos=null]

Object added at ordinal 1

Seems like what is happening is FastBlobTypeDeserializationState.get(..) is being called before FastBlobTypeDeserializationState.add(..) is complete.

On Thursday, January 15, 2015 at 9:54:58 AM UTC-5, Darren Bathgate wrote:

[Drew Koszewnik]

Hello Darren,

I think I understand the problem.  It appears that you're creating a circular reference chain:

Pojo -> ListOfPojo -> SupPojo -> ListOfPojo -> SubPojo -> ....

Unfortunately, Zeno won't support this type of circular reference.

When we have run into this problem in our data model, we have referenced the sub-pojos by some identifier, rather than holding the sub-pojos themselves in a list.  Then, our serializers are simply lists of the appropriate identifiers, which we use to look-up the POJOs at runtime.  

This does require maintenance of an additional index to lookup those POJOs at runtime.

For example, instead of:

class Pojo {

   List<Pojo> subPojos;

}

You might instead try:

class Pojo {

   List<ID> subPojoIds;

}

Along with some kind of:

Map<ID, Pojo> pojoLookup;

I wish I could provide you with a simple switch which was overlooked, but hopefully this helps a little bit.

Thanks,

Drew.

[Darren Bathgate]

Hi Drew,

Thanks for the response, I believe this gives me the answer I was looking for.

I wasn't sure if there was another way in zeno to do a circular reference like this.

-Darren