JDO startup apparently reads through entire table, causing quota problems - why and how to fix?

168 views
Skip to first unread message

Ken Traub

unread,
Nov 21, 2013, 2:39:11 PM11/21/13
to google-a...@googlegroups.com
I've got a GAE app which uses JDO to manage a single persistent class.  Memcache not used.  Every time a new instance of the app starts up, I see 3524 data store read operations added to my quota usage.  That number happens to be the number of rows in the persistent table, currently.  So it seems that JDO is reading through the entire table upon startup.  I've done thorough logging and testing to confirm that my own app code is not doing this -- my code only does queries thatl read at most 1-12 rows at a time, and the effect on quota usage for an already started instance matches my expectations.  It's only when GAE starts a new instance that I see the huge increment to data store read ops.
 
Because usage of my app varies up and down through a typical day, GAE starts new instances frequently (which come and go).  So this 3500+ hit to quota each time easily exceeds 50K total a day, so this is costing me $.
 
Questions:  why in the world does JDO need to read through the entire table upon startup, and is there any way to prevent it?
 
Below is the log output at startup from JDO.
 
Thanks very much in advance for any advice anybody provides.
 
  1. 2013-11-21 13:45:01.543
  2. org.datanucleus.plugin.NonManagedPluginRegistry resolveConstraints: Bundle "org.datanucleus.jpa" has an optional dependency to "org.datanucleus.enhancer" but it cannot be resolved
  3. I2013-11-21 13:45:01.543
    org.datanucleus.plugin.NonManagedPluginRegistry resolveConstraints: Bundle "org.datanucleus" has an optional dependency to "org.eclipse.equinox.registry" but it cannot be resolved
  4. I2013-11-21 13:45:01.544
    org.datanucleus.plugin.NonManagedPluginRegistry resolveConstraints: Bundle "org.datanucleus" has an optional dependency to "org.eclipse.core.runtime" but it cannot be resolved
  5. I2013-11-21 13:45:02.237
    org.datanucleus.PersistenceConfiguration setProperty: Property datanucleus.rdbms.sql.allowAllSQLStatements unknown - will be ignored
  6. I2013-11-21 13:45:02.244
    org.datanucleus.PersistenceConfiguration setProperty: Property datanucleus.rdbms.stringDefaultLength unknown - will be ignored
  7. I2013-11-21 13:45:02.263
    org.datanucleus.PersistenceConfiguration setProperty: Property datanucleus.appengine.autoCreateDatastoreTxns unknown - will be ignored
  8. I2013-11-21 13:45:02.268
    org.datanucleus.ObjectManagerFactoryImpl logConfiguration: ================= Persistence Configuration ===============
  9. I2013-11-21 13:45:02.270
    org.datanucleus.ObjectManagerFactoryImpl logConfiguration: DataNucleus Persistence Factory - Vendor: "DataNucleus"  Version: "1.1.5"
  10. I2013-11-21 13:45:02.271
    org.datanucleus.ObjectManagerFactoryImpl logConfiguration: DataNucleus Persistence Factory initialised for datastore URL="appengine" driver="" userName=""
  11. I2013-11-21 13:45:02.271
    org.datanucleus.ObjectManagerFactoryImpl logConfiguration: ===========================================================
  12. I2013-11-21 13:45:02.839
    org.datanucleus.PersistenceConfiguration setProperty: Property datanucleus.query.cached unknown - will be ignored
  13. I2013-11-21 13:45:03.840
    org.datanucleus.jdo.metadata.JDOMetaDataManager <init>: Registering listener for metadata initialisation
  14. I2013-11-21 13:45:04.692
    org.datanucleus.jdo.metadata.JDOAnnotationReader processClassAnnotations: Class "net.traub.tagxlate.server.dao.gae.GaeGcpRuleRecord" has been specified with JDO annotations so using those.
  15. I2013-11-21 13:45:04.851
    org.datanucleus.store.appengine.MetaDataValidator validate: Performing appengine-specific metadata validation for net.traub.tagxlate.server.dao.gae.GaeGcpRuleRecord
  16. I2013-11-21 13:45:04.851
    org.datanucleus.store.appengine.MetaDataValidator validate: Finished performing appengine-specific metadata validation for net.traub.tagxlate.server.dao.gae.GaeGcpRuleRecord
  17. I2013-11-21 13:45:04.936
    org.datanucleus.store.StoreDataManager registerStoreData: Managing Persistence of Class : net.traub.tagxlate.server.dao.gae.GaeGcpRuleRecord [Table : <class name="GaeGcpRuleRecord"
       identity-type="application"
       objectid-class="javax.jdo.identity.LongIdentity"
       persistence-modifier="persistence-capable"
>
[details omitted here to save space] </class>
, InheritanceStrategy : new-table]
  18. I2013-11-21 13:45:05.034
    org.datanucleus.store.StoreDataManager registerStoreData: Managing Persistence of Class : net.traub.tagxlate.server.dao.gae.GaeGcpRuleRecord [Table : <class name="GaeGcpRuleRecord"
       identity-type="application"
       objectid-class="javax.jdo.identity.LongIdentity"
       persistence-modifier="persistence-capable"
>
<inheritance strategy="new-table">
</inheritance>
[details omitted here to save space] </class>
, InheritanceStrategy : new-table]
  19. I2013-11-21 13:45:06.235
    org.datanucleus.jdo.metadata.JDOMetaDataManager$MetaDataRegisterClassListener registerClass: Listener found initialisation for persistable class net.traub.tagxlate.server.dao.gae.GaeGcpRuleRecord
  20. W2013-11-21 13:45:07.758
    com.google.appengine.api.datastore.QueryResultsSourceImpl logChunkSizeWarning: This query does not have a chunk size set in FetchOptions and has returned over 1000 results.  If result sets of this size are common for this query, consider setting a chunk size to improve performance.
  To disable this warning set the following system property in appengine-web.xml (the value of the property doesn't matter): 'appengine.datastore.disableChunkSizeWarning'

Vinny P

unread,
Nov 26, 2013, 12:53:16 AM11/26/13
to google-a...@googlegroups.com
On Thu, Nov 21, 2013 at 1:39 PM, Ken Traub <goo...@kentraub.com> wrote:
Questions:  why in the world does JDO need to read through the entire table upon startup, and is there any way to prevent it?



How did you configure your persistence.xml file? Certain settings can cause caching/preloading requests, which this could be.


On Thu, Nov 21, 2013 at 1:39 PM, Ken Traub <goo...@kentraub.com> wrote:
com.google.appengine.api.datastore.QueryResultsSourceImpl logChunkSizeWarning: This query does not have a chunk size set in FetchOptions and has returned over 1000 results. If result sets of this size are common for this query, consider setting a chunk size to improve performance. To disable this warning set the following system property in appengine-web.xml (the value of the property doesn't matter): 'appengine.datastore.disableChunkSizeWarning' 


This informational message from the datastore would seem to hint that something is running a query. Perhaps your application isn't running the query, but a library somewhere is. Try installing AppStats to see details about the query: https://developers.google.com/appengine/docs/java/tools/appstats
 
 
-----------------
-Vinny P
Technology & Media Advisor
Chicago, IL

App Engine Code Samples: http://www.learntogoogleit.com

Reply all
Reply to author
Forward
0 new messages