Why are java_lang_Object Entities are being created?

[IlyaE]

I posted this in the JAVA App Engine group but it may be related to
Twig but hopefully not.

I look at the Database Viewer and besides the models and _ah_SESSION
i
see there is a java_lang_Object entity that only has a ID/Name
attribute. What is this and why are they being created?


I define my models using DefaultConfiguration.registerTypeName(class,
name)

[John Patterson]

It looks like Twig is creating these (by the entity kind name) probably
because something in your data model references an Object.

One possibility I can think of: if you have an @Embedded raw Collection
or Collection<Object> that is declared polymorphic=false then Twig will
try to create Objects to put in it when it reads instances back again.

I would clean the local datastore and run tests to see when the Objects
appear in the datastore. You could also set a conditional break point
in BaseObjectDatastore.servicePut() to see when an Entity with that kind
name is put in the datastore.

[IlyaE]

Well this only happened in the deployed version. I do have an idea of
what it is since on of my pages bombed due to a casting error from
Object to Item.
Here is my model:
public class Registry implements Serializable {
List<Item> items;
}

An Item entity got created as an object but added to the the items
list.. Any idea why this sporadically happened? See 48002L.

items
value: [datastore_types.Key.from_path(u'Item', 33002L, _app=u's~g'),
datastore_types.Key.from_path(u'Item', 41001L, _app=u's~g'),
datastore_types.Key.from_path(u'Item', 44002L, _app=u's~g'),
datastore_types.Key.from_path(u'Item', 44004L, _app=u's~g'),
datastore_types.Key.from_path(u'Item', 47005L, _app=u's~g'),
datastore_types.Key.from_path(u'java_lang_Object', 48002L,
_app=u's~g'), datastore_types.Key.from_path(u'Item', 40011L,
_app=u's~g')]

[John Patterson]

On 20/09/2011 08:34, IlyaE wrote:
> An Item entity got created as an object but added to the the items
> list.. Any idea why this sporadically happened? See 48002L.

This could be a bug related to entering a null value into a Collection
of references. Add a check that you are not storing null when you add a
reference to an Item. null is not a valid referenced item, but none of
the tests cover trying to add null.

Make sure you have the latest source
http://code.google.com/p/twig-persist/wiki/Integration2_0 and with your
null check in place you should be able to remove the references to
Object with the datastore viewer and delete the Entities with kind
java_lang_Object.

[John Patterson]

On 20/09/2011 08:34, IlyaE wrote:

Well this only happened in the deployed version. I do have an idea of
what it is since on of my pages bombed due to a casting error from
Object to Item.
Here is my model:
public class Registry implements Serializable {
   List<Item> items;
}

An Item entity got created as an object but added to the the items
list.. Any idea why this sporadically happened? See 48002L.

items
value: [datastore_types.Key.from_path(u'Item', 33002L, _app=u's~g'),
datastore_types.Key.from_path(u'Item', 41001L, _app=u's~g'),
datastore_types.Key.from_path(u'Item', 44002L, _app=u's~g'),
datastore_types.Key.from_path(u'Item', 44004L, _app=u's~g'),
datastore_types.Key.from_path(u'Item', 47005L, _app=u's~g'),
datastore_types.Key.from_path(u'java_lang_Object', 48002L,
_app=u's~g'), datastore_types.Key.from_path(u'Item', 40011L,
_app=u's~g')]

For some explanation of why null could end up stored as Object - PropertyTranslators return null to indicate that they could not translate a value into Property's (i.e. try the next translator please) so to indicate that a value is to be actually stored as null in the datastore, they return a special value PropertyTranslator.NULL_VALUE which is actually an object.  Now all of the translator code should detect this and store null... but there may be a bug with Collections of relations (could already be fixed) where the NULL_VALUE is being stored.

[IlyaE]

i was able to build the jar with the link you provided. but now have
to change all my imports to com.vercer and some of my methods are
missing. Is this right behavior?

Like the returnUnique and the DefaultConfiguration.registerTypeName
methods... what can i replace these with?

On Sep 19, 10:00 pm, John Patterson <jdpatter...@gmail.com> wrote:
> On 20/09/2011 08:34, IlyaE wrote:
>
>
>
>
>
>
>
>
>
> > Well this only happened in the deployed version. I do have an idea of
> > what it is since on of my pages bombed due to a casting error from
> > Object to Item.
> > Here is my model:
> > public class Registry implements Serializable {
> >     List<Item>  items;
> > }
>
> > An Item entity got created as an object but added to the the items
> > list.. Any idea why this sporadically happened? See 48002L.
>
> > items
> > value: [datastore_types.Key.from_path(u'Item', 33002L, _app=u's~g'),
> > datastore_types.Key.from_path(u'Item', 41001L, _app=u's~g'),
> > datastore_types.Key.from_path(u'Item', 44002L, _app=u's~g'),
> > datastore_types.Key.from_path(u'Item', 44004L, _app=u's~g'),
> > datastore_types.Key.from_path(u'Item', 47005L, _app=u's~g'),
> > datastore_types.Key.from_path(u'java_lang_Object', 48002L,
> > _app=u's~g'), datastore_types.Key.from_path(u'Item', 40011L,
> > _app=u's~g')]
>

> For some explanation of why null /could /end up stored as Object -

[IlyaE]

AH! I just went through all my code replacing returnAll().now with
return .returnAllResultsNow()... and got
UnsupportedOperationException: Not yet implemented...

Guess I'm going back to beta4

[John Patterson]

On 21/09/2011 00:55, IlyaE wrote:
> Do i need to remove the jars i added and linked? Do I need to do
> anything prior to deploying?
>

No, if you include the source in your project you don't need any twig
jars. Thats what makes it such an easy way to deploy. Everything gets
put into your WEB-INF/classes folder by Eclipse

[John Patterson]

On 21/09/2011 03:07, IlyaE wrote:

i was able to build the jar with the link you provided. but now have
to change all my imports to com.vercer and some of my methods are
missing. Is this right behavior?

Like the returnUnique and the DefaultConfiguration.registerTypeName
methods... what can i replace these with?

You missed the first step after getting the source; you must "switch to the v2.0 branch using the command "hg update -C v2.0". If you use TortoiseHg you can choose "TortoiseHg > Update > Update to" and choose v2.0."

[IlyaE]

OK i ran the command, and now i get beta5 jar created.. though i
still
get an UnsupportedOperationException: Not yet implemented... error
and
I see the google twig package again so i'll try to revert everything
back.

[John Patterson]

On 21/09/2011 19:37, IlyaE wrote:
> OK i ran the command, and now i get beta5 jar created.. though i
> still
> get an UnsupportedOperationException: Not yet implemented... error
> and
> I see the google twig package again so i'll try to revert everything
> back.
>

What exactly are you doing and what exactly is the stack trace?

[IlyaE]

This was my original code
List<News> news = ds.find().type(News.class).addSort("creationDate",
SortDirection.DESCENDING).fetchMaximum(3).returnAll().now();

which after update i changed it to
List<News> news = ds.find().type(News.class).addSort("creationDate",
SortDirection.DESCENDING).maximumResults(3).returnAllResultsNow();

and then got the stacktrace that i can't produce anymore since i've
changed things back and forth so many times now...

Now when i start i get this error
java.lang.NoSuchMethodError:
com.google.common.collect.Maps.newConcurrentMap()Ljava/util/concurrent/
ConcurrentMap;

at the ObjectDatastore ds = new AnnotationObjectDatastore();

Am i supposed to be including the com.google.code.twig or the
com.vercer.engine packages?

[John Patterson]

On 21/09/2011 19:56, IlyaE wrote:
> This was my original code
> List<News> news = ds.find().type(News.class).addSort("creationDate",
> SortDirection.DESCENDING).fetchMaximum(3).returnAll().now();
>
> which after update i changed it to
> List<News> news = ds.find().type(News.class).addSort("creationDate",
> SortDirection.DESCENDING).maximumResults(3).returnAllResultsNow();
>
> and then got the stacktrace that i can't produce anymore since i've
> changed things back and forth so many times now...
>
> Now when i start i get this error
> java.lang.NoSuchMethodError:
> com.google.common.collect.Maps.newConcurrentMap()Ljava/util/concurrent/
> ConcurrentMap;
>
> at the ObjectDatastore ds = new AnnotationObjectDatastore();
>
> Am i supposed to be including the com.google.code.twig or the
> com.vercer.engine packages?

OK, you're original code is correct. The second form you showed above
is the v1.0 API which you mistakenly built. Forget that.

The error you are getting is due to a missing Guava jar. Follow the
directions in the link I sent earlier which explain where to get it and
where to put it.

[IlyaE]

I suspected the guava issue and i already have the guava-r09.jar in
the lib folder and in my build path.. any other ideas? I had a google
shopper api that had the old r08 jar but i wasn't using it so i
removed the entire api and cleaned the project.

Still the same issue.

[John Patterson]

On 21/09/2011 20:15, IlyaE wrote:
> I suspected the guava issue and i already have the guava-r09.jar in
> the lib folder and in my build path.. any other ideas? I had a google
> shopper api that had the old r08 jar but i wasn't using it so i
> removed the entire api and cleaned the project.
>
> Still the same issue.

Definitely look like you are missing the guava jar from WEB-INF/lib or
it could be corrupt. Try replacing it.

[IlyaE]

Well i removed it.. downloaded it again.. this time dropped both the
guava-r09.jar and guava-r09-gwt.jar files in my WEB-INF/lib folder.
Added it to my build path and cleaned the project.

I still get the same error. I'm using eclipse 3.7 and twig-persist-2.0-
beta5.jar that i compiled today.

Do i need to include the guava anywhere when i compile the jar? I
wouldn't think so but i'm completely out of ideas now since my project
is not showing any errors as i am also using methods from that
library.

[John Patterson]

On 21/09/2011 21:28, IlyaE wrote:
> Well i removed it.. downloaded it again.. this time dropped both the
> guava-r09.jar and guava-r09-gwt.jar files in my WEB-INF/lib folder.
> Added it to my build path and cleaned the project.
>
> I still get the same error. I'm using eclipse 3.7 and twig-persist-2.0-
> beta5.jar that i compiled today.
>
> Do i need to include the guava anywhere when i compile the jar? I
> wouldn't think so but i'm completely out of ideas now since my project
> is not showing any errors as i am also using methods from that
> library.
>

If you get this error:

java.lang.NoSuchMethodError:
com.google.common.collect.Maps.newConcurrentMap()Ljava/util/concurrent/
ConcurrentMap;

then the guava jar is not being found. The method is there:

http://code.google.com/p/guava-libraries/source/browse/src/com/google/common/collect/Maps.java?name=v9.0#183

twig built successfully against its own version of it.

The only possibility I can think of is that you have an older version of guava or google collections in your lib directory.

[IlyaE]

OK.. you figured it out.. i had a google-collect-1.0-rc1 jar file in
my lib. Once i removed that i got the app to start. Here is the
stacktrace you were looking for. I've reverted everything back to my
original code and now i can see if i get those Objects created.

When you wrote .. .

Add a check that you are not storing null when you add a
reference to an Item.

Here is my code..
List<Item> items = registry.getItems()==null?new ArrayList<Item>():
(ArrayList<Item>) registry.getItems();
Item item = new Item();
...item.setX...
items.add(item);
registry.setItems(items);
ds.update(registry);

Did you mean that i need to check all my setter values to make sure
nothing is null? or check if item itself is null before setting the
collection in registry?


Here is the stacktrace that i got before.
java.lang.UnsupportedOperationException: Not yet implemented
at
com.vercer.engine.persist.standard.StandardRootFindCommand.returnAllResultsNow(StandardRootFindCommand.java:
184)
at
com.vercer.engine.persist.standard.StandardRootFindCommand.returnAllResultsNow(StandardRootFindCommand.java:
20)
at org.apache.jsp.index_jsp._jspService(index_jsp.java:77)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at
org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:
377)
at
org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:
313)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260)
at com.google.appengine.tools.development.PrivilegedJspServlet.access
$101(PrivilegedJspServlet.java:23)
at com.google.appengine.tools.development.PrivilegedJspServlet
$2.run(PrivilegedJspServlet.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at
com.google.appengine.tools.development.PrivilegedJspServlet.service(PrivilegedJspServlet.java:
57)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:
511)
at org.mortbay.jetty.servlet.ServletHandler
$CachedChain.doFilter(ServletHandler.java:1166)
at
com.google.appengine.tools.development.HeaderVerificationFilter.doFilter(HeaderVerificationFilter.java:
35)
at org.mortbay.jetty.servlet.ServletHandler
$CachedChain.doFilter(ServletHandler.java:1157)
at
com.google.appengine.api.blobstore.dev.ServeBlobFilter.doFilter(ServeBlobFilter.java:
58)
at org.mortbay.jetty.servlet.ServletHandler
$CachedChain.doFilter(ServletHandler.java:1157)
at
com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter(TransactionCleanupFilter.java:
43)
at org.mortbay.jetty.servlet.ServletHandler
$CachedChain.doFilter(ServletHandler.java:1157)
at
com.google.appengine.tools.development.StaticFileFilter.doFilter(StaticFileFilter.java:
122)
at org.mortbay.jetty.servlet.ServletHandler
$CachedChain.doFilter(ServletHandler.java:1157)
at
com.google.appengine.tools.development.BackendServersFilter.doFilter(BackendServersFilter.java:
97)
at org.mortbay.jetty.servlet.ServletHandler
$CachedChain.doFilter(ServletHandler.java:1157)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:
388)
at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:
216)
at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:
182)
at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:
765)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:
418)
at
com.google.apphosting.utils.jetty.DevAppEngineWebAppContext.handle(DevAppEngineWebAppContext.java:
70)
at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:
152)
at com.google.appengine.tools.development.JettyContainerService
$ApiProxyHandler.handle(JettyContainerService.java:351)
at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:
152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:
542)
at org.mortbay.jetty.HttpConnection
$RequestHandler.headerComplete(HttpConnection.java:923)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:547)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at
org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:
409)
at org.mortbay.thread.QueuedThreadPool
$PoolThread.run(QueuedThreadPool.java:582)

On Sep 21, 10:47 am, John Patterson <jdpatter...@gmail.com> wrote:
> On 21/09/2011 21:28, IlyaE wrote:
>
> > Well i removed it.. downloaded it again.. this time dropped both the
> > guava-r09.jar and guava-r09-gwt.jar files in my WEB-INF/lib folder.
> > Added it to my build path and cleaned the project.
>
> > I still get the same error. I'm using eclipse 3.7 and twig-persist-2.0-
> > beta5.jar that i compiled today.
>
> > Do i need to include the guava anywhere when i compile the jar? I
> > wouldn't think so but i'm completely out of ideas now since my project
> > is not showing any errors as i am also using methods from that
> > library.
>
> If you get this error:
>
> java.lang.NoSuchMethodError:
> com.google.common.collect.Maps.newConcurrentMap()Ljava/util/concurrent/
> ConcurrentMap;
>
> then the guava jar is not being found.  The method is there:
>

> http://code.google.com/p/guava-libraries/source/browse/src/com/google...

[John Patterson]

On 21/09/2011 22:08, IlyaE wrote:
> OK.. you figured it out.. i had a google-collect-1.0-rc1 jar file in
> my lib. Once i removed that i got the app to start. Here is the
> stacktrace you were looking for. I've reverted everything back to my
> original code and now i can see if i get those Objects created.
>
> When you wrote .. .
> Add a check that you are not storing null when you add a
> reference to an Item.
>
> Here is my code..
> List<Item> items = registry.getItems()==null?new ArrayList<Item>():
> (ArrayList<Item>) registry.getItems();
> Item item = new Item();
> ...item.setX...
> items.add(item);
> registry.setItems(items);
> ds.update(registry);
>
> Did you mean that i need to check all my setter values to make sure
> nothing is null? or check if item itself is null before setting the
> collection in registry?

Yes I mean do not add any null values to the collection of references
items.

I have just been looking through the code in
StandardMultipleStoreCommand that is used to store the referenced
items. I can see a check that I added for instances that were already
stored to filter out any null values. This could happen if an
collection of instances are stored and one of them (ItemA) references
another one (ItemB) in the collection. Then Twig would store the ItemB
as it stored itemA because it follows references. Then when tries to
store ItemB again but finds that it is already stored.

In other words I think there was a bug that has been fixed that could
have allowed a NULL_VALUE to be stored.

I'm not sure that this is what you are coming across but it could be

>
> Here is the stacktrace that i got before.
> java.lang.UnsupportedOperationException: Not yet implemented
> at

That is from v1.0 which you built by mistake.

[John Patterson]

On 21/09/2011 22:08, IlyaE wrote:

> OK.. you figured it out.. i had a google-collect-1.0-rc1 jar file in
> my lib. Once i removed that i got the app to start. Here is the
> stacktrace you were looking for. I've reverted everything back to my
> original code and now i can see if i get those Objects created.
>
> When you wrote .. .
> Add a check that you are not storing null when you add a
> reference to an Item.
>
> Here is my code..
> List<Item> items = registry.getItems()==null?new ArrayList<Item>():
> (ArrayList<Item>) registry.getItems();
> Item item = new Item();
> ...item.setX...
> items.add(item);
> registry.setItems(items);
> ds.update(registry);
>
> Did you mean that i need to check all my setter values to make sure
> nothing is null? or check if item itself is null before setting the
> collection in registry?

Yes I mean do not add any null values to the collection of references items.

I have just been looking through the code in
StandardMultipleStoreCommand that is used to store the referenced items.
I can see a check that I added for instances that were already stored
to filter out any null values. This could happen if an collection of
instances are stored and one of them (ItemA) references another one
(ItemB) in the collection. Then Twig would store the ItemB as it stored
itemA because it follows references. Then when tries to store ItemB

again but finds that it is already stored and returns null.

In other words I think there was a bug that has been fixed that could
have allowed a NULL_VALUE to be stored.

I'm not sure that this is what you are coming across but it could be if
Item references other Items

>
> Here is the stacktrace that i got before.
> java.lang.UnsupportedOperationException: Not yet implemented
> at

That is from v1.0 which you built by mistake.

[IlyaE]

OK i'll take a look and keep playing with it.. Thanks for your help
John.