Set<String> is decoded as ArrayList<String> ?

[Miroslav Genov]

Hello,

I'm encountering an strange issue when entity has property Set<String> and the same property in the database is stored as empty list. 

Here is a complete stack trace that is causing the issue

java.lang.IllegalStateException: Could not set value [] to field private java.util.Set com.test.Foo.items

at com.google.code.twig.translator.FieldTranslator.setFieldValue(FieldTranslator.java:201)

at com.google.code.twig.translator.FieldTranslator.decodeField(FieldTranslator.java:188)

at com.google.code.twig.standard.TranslatorObjectDatastore$ObjectFieldTranslator.decodeField(TranslatorObjectDatastore.java:981)

at com.google.code.twig.translator.FieldTranslator.decode(FieldTranslator.java:127)

at com.google.code.twig.standard.StandardDecodeCommand.entityToInstance(StandardDecodeCommand.java:118)

at com.google.code.twig.standard.StandardDecodeCommand$1.next(StandardDecodeCommand.java:180)

at com.google.common.collect.ForwardingIterator.next(ForwardingIterator.java:48)

at com.google.common.collect.Lists.newArrayList(Lists.java:145)

at com.google.code.twig.standard.StandardRootFindCommand$3.now(StandardRootFindCommand.java:278)

at com.google.code.twig.standard.StandardRootFindCommand$3.now(StandardRootFindCommand.java:274)

As it seems the problem is in the createCollection method of IterableTranslator:

protected Collection<Object> createCollection(Type type)

{

// support reusing existing implementations 

if (datastore.refresh != null)

{

@SuppressWarnings("unchecked")

Collection<Object> result = (Collection<Object>) datastore.refresh;

datastore.refresh = null;

return result;

}

else

{

return new ArrayList<Object>();

}

}

so, the type (Set<String> in my case) is not used to recognize where Set or List implementation need to be created.

I made a small test and fix to make it working. Here is a gist of my fix: https://gist.github.com/2971101

[Miroslav Genov]

Just found that this fix only works with Set<String>, not for Set<RelatedEntity> which belongs to another entity group. I'll dig into it to find out whats going wrong. 

[Miroslav Genov]

Just saw the comment:

 // collections always come back as ArrayLists

 Collection<Object> collection = PropertySets.firstValue(properties);

So, is this a normal behavior for working with collections?  i.e Only lists to be used ?

[John Patterson]

Could you post the full stack trace?  Need to see the underlying exception why the field value cannot be set.

All iterable types are supported as long as there is a Converter registered for them.  the last line of IterableTranslator.decode does:

Iterable<?> iterable = datastore.getTypeConverter().convert(list, type);

Also, if your model class declares the implantation e.g. Set<Foo> food = new MySet() - then that implementation is clear()'ed and reused, a new one is not created.

[John Patterson]

On 22/06/2012, at 4:06 PM, Miroslav Genov wrote:

> Just saw the comment:
>
> // collections always come back as ArrayLists
> Collection<Object> collection = PropertySets.firstValue(properties);
>
> So, is this a normal behavior for working with collections? i.e Only lists to be used ?

No this just means that the low-level datastore always returns ArrayList even if you put a HashSet in it.

[John Patterson]

On 22/06/2012, at 2:49 PM, Miroslav Genov wrote:

java.lang.IllegalStateException: Could not set value [] to field private java.util.Set com.test.Foo.items

Is this declared as a raw Set?

[Miroslav Genov]

Jun 22, 2012 10:56:53 AM com.google.appengine.api.datastore.dev.LocalDatastoreService init

INFO: Local Datastore initialized: 

Type: Master/Slave

Storage: In-memory

Here is the full stack trace:

java.lang.IllegalStateException: Could not set value [com.google.code.twig.SetPersistenceTest$Bar@e4600c0] to field private java.util.Set com.google.code.twig.SetPersistenceTest$Foo.bars

at com.google.code.twig.translator.FieldTranslator.setFieldValue(FieldTranslator.java:201)

at com.google.code.twig.translator.FieldTranslator.decodeField(FieldTranslator.java:188)

at com.google.code.twig.standard.TranslatorObjectDatastore$ObjectFieldTranslator.decodeField(TranslatorObjectDatastore.java:981)

at com.google.code.twig.translator.FieldTranslator.decode(FieldTranslator.java:127)

at com.google.code.twig.standard.StandardDecodeCommand.entityToInstance(StandardDecodeCommand.java:118)

at com.google.code.twig.standard.StandardDecodeCommand.keyToInstance(StandardDecodeCommand.java:205)

at com.google.code.twig.standard.StandardSingleTypedLoadCommand.keyToInstance(StandardSingleTypedLoadCommand.java:9)

at com.google.code.twig.standard.StandardSingleTypedLoadCommand.now(StandardSingleTypedLoadCommand.java:34)

at com.google.code.twig.SetPersistenceTest.getStoredSetOfRelatedEntities(SetPersistenceTest.java:69)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

at org.junit.internal.runners.TestMethod.invoke(TestMethod.java:59)

at org.junit.internal.runners.MethodRoadie.runTestMethod(MethodRoadie.java:98)

at org.junit.internal.runners.MethodRoadie$2.run(MethodRoadie.java:79)

at org.junit.internal.runners.MethodRoadie.runBeforesThenTestThenAfters(MethodRoadie.java:87)

at org.junit.internal.runners.MethodRoadie.runTest(MethodRoadie.java:77)

at org.junit.internal.runners.MethodRoadie.run(MethodRoadie.java:42)

at org.junit.internal.runners.JUnit4ClassRunner.invokeTestMethod(JUnit4ClassRunner.java:88)

at org.junit.internal.runners.JUnit4ClassRunner.runMethods(JUnit4ClassRunner.java:51)

at org.junit.internal.runners.JUnit4ClassRunner$1.run(JUnit4ClassRunner.java:44)

at org.junit.internal.runners.ClassRoadie.runUnprotected(ClassRoadie.java:27)

at org.junit.internal.runners.ClassRoadie.runProtected(ClassRoadie.java:37)

at org.junit.internal.runners.JUnit4ClassRunner.run(JUnit4ClassRunner.java:42)

at org.junit.runner.JUnitCore.run(JUnitCore.java:130)

at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:63)

Caused by: java.lang.RuntimeException: java.lang.IllegalArgumentException: Can not set java.util.Set field com.google.code.twig.SetPersistenceTest$Foo.bars to java.util.ArrayList

at com.google.code.twig.util.Reflection.set(Reflection.java:140)

at com.google.code.twig.translator.FieldTranslator.setFieldValue(FieldTranslator.java:197)

... 28 more

Caused by: java.lang.IllegalArgumentException: Can not set java.util.Set field com.google.code.twig.SetPersistenceTest$Foo.bars to java.util.ArrayList

at sun.reflect.UnsafeFieldAccessorImpl.throwSetIllegalArgumentException(UnsafeFieldAccessorImpl.java:146)

at sun.reflect.UnsafeFieldAccessorImpl.throwSetIllegalArgumentException(UnsafeFieldAccessorImpl.java:150)

at sun.reflect.UnsafeObjectFieldAccessorImpl.set(UnsafeObjectFieldAccessorImpl.java:63)

at java.lang.reflect.Field.set(Field.java:657)

at com.google.code.twig.util.Reflection.set(Reflection.java:136)

... 29 more

Test that is verifying the behavior: 

  static class Foo {

    @Id

    private String key;

    private Set<String> items;

    private Set<Bar> bars;

  }

  static class Bar {

    @Id

    private String id;

  }

 @Test

  public void getStoredSetOfRelatedEntities() {

    Bar bar = new Bar();

    bar.id = "123";

    ObjectDatastore datastore = new AnnotationObjectDatastore(true);

    datastore.store().instance(bar).now();

    Foo foo = new Foo();

    foo.key = "key2";

    foo.bars = new HashSet<Bar>();

    foo.bars.add(bar);

    datastore.store().instance(foo).now();

    datastore.disassociateAll();

    Foo existing = datastore.load().type(Foo.class).id(foo.key).now();

    assertThat(existing.bars.size(),is(equalTo(1)));

  }

[John Patterson]

Make sure you have the latest source.  Do you see this line?

http://code.google.com/p/twig-persist/source/browse/src/main/java/com/google/code/twig/standard/IterableTranslator.java?r=43b3f7821597b33c595957345569e4f1b09abb73#102

[Miroslav Genov]

Huh, you were right. 

10 files updated, 0 files merged, 0 files removed, 0 files unresolved

I'm not familiar with mercurial :( Sorry for the bothering. 

It seems that the issue still exists: 

mgenov:twig-persist() $ hg update -C v2.0                              

0 files updated, 0 files merged, 0 files removed, 0 files unresolved

mgenov:twig-persist() $ hg diff

mgenov:twig-persist() $ 

So no changed files. Now the same test that I wrote before is throwing the following exception: 

Jun 22, 2012 11:25:14 AM com.google.appengine.api.datastore.dev.LocalDatastoreService init

INFO: Local Datastore initialized: 

Type: Master/Slave

Storage: In-memory

java.lang.IllegalStateException: Could not set value [] to field private java.util.Set com.google.code.twig.SetPersistenceTest$Foo.bars

at com.google.code.twig.translator.FieldTranslator.setFieldValue(FieldTranslator.java:201)

at com.google.code.twig.translator.FieldTranslator.decodeField(FieldTranslator.java:188)

at com.google.code.twig.standard.TranslatorObjectDatastore$ObjectFieldTranslator.decodeField(TranslatorObjectDatastore.java:975)

Note that Iterable<?> iterable = datastore.getTypeConverter().convert(list, type); now exists on line #102 in my code. 

[Miroslav Genov]

btw, are all tests pass in the latest version  ? 

Here is what I get when I run all tests from the package: 

[John Patterson]

I have just run the tests and also some are failing for me. Must be a recent check in that broke something, sorry. I'll have a look later tonight.

On 22/06/2012, at 6:29 PM, Miroslav Genov wrote:

[Miroslav Genov]

John, did you have time to take a look ? 

[John Patterson]

Yes, I just pushed the fix. It was a bug that was fixed but got re-introduced as I merged changes from another project.

[Georgi Georgiev]

Hi John,

I have the same problem. I’m attaching a patch which contains several tests and fix of this problem.

I would appreciate if you look at it.