java.lang.OutOfMemoryError when doing a query for 26000 keys

[Chister Nordvik]

I am getting about 26K of objects via a list of keys and getting the out of memory exception. Is this "normal"? 26K objects doesn't seem that much to me? 

I have not enabled caching (but the stacktrace seems to go via some cache code?) and I am still using Objectify v2. 

for (AlertSubscription s: filteredSubscriptions.values()) 
{
  dKeys.add(new com.googlecode.objectify.Key<DeviceInfo>(DeviceInfo.class,s.user));
}
Map<com.googlecode.objectify.Key<DeviceInfo>, DeviceInfo> fetched = ofy.get(dKeys); //this line crashes sometimes with OOM

Error for /tasks/sendfile
java.lang.OutOfMemoryError: Java heap space
	at java.util.AbstractCollection.toArray(AbstractCollection.java:136)
	at java.util.ArrayList.<init>(ArrayList.java:151)
	at com.google.appengine.repackaged.com.google.common.collect.Lists.newArrayList(Lists.java:128)
	at com.google.appengine.api.datastore.PostLoadFuture.executeCallbacks(PostLoadFuture.java:24)
	at com.google.appengine.api.datastore.PostLoadFuture.executeCallbacks(PostLoadFuture.java:13)
	at com.google.appengine.api.datastore.PostOpFuture.wrap(PostOpFuture.java:27)
	at com.google.appengine.api.utils.FutureWrapper.wrapAndCache(FutureWrapper.java:53)
	at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:90)
	at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:86)
	at com.googlecode.objectify.cache.ListenableFuture.get(ListenableFuture.java:91)
	at com.googlecode.objectify.cache.MergeFuture.get(MergeFuture.java:81)
	at com.googlecode.objectify.cache.MergeFuture.get(MergeFuture.java:17)
	at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:86)
	at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:86)
	at com.googlecode.objectify.impl.ResultAdapter.get(ResultAdapter.java:29)
	at com.googlecode.objectify.impl.ObjectifyImpl.get(ObjectifyImpl.java:45)
	at com.google.android.chrometophone.server.SubscriptionController.sendAlertToSubscribers(SubscriptionController.java:328)

[Jeff Schnitzer]

An F1 instance has 128MB of RAM. Your 2-year-old cellphone has an order of magnitude more than that. You really have to be careful in this kind of environment; loading big chunks of data is pretty much impossible, and 26k objects could easily push past the limit.

Either consider how to break down the problem so you hold less state in RAM, or upgrade your instance size. It is disappointing that GAE instances are so spendy; Jon and I are in the process of figuring out how to run a couple batch processes on F2s using the new modules because it will allow us to go back to F1s for the frontend and save us money.

Jeff

--
You received this message because you are subscribed to the Google Groups "objectify-appengine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to objectify-appen...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Chister Nordvik]

Thanks!. I can't believe I missed the option to set your instance size when I've been using it for 2 years :-) I changed it to F2 and will see if that is enough for now.  

To unsubscribe from this group and stop receiving emails from it, send an email to objectify-appengine+unsub...@googlegroups.com.

[Emanuele Ziglioli]

Hi,

are you still using App Engine?

I'm experimenting with big queries and batch queries too. Did you ever find an 'optimal size'? quickest query response time/batch size, that's what I'm looking for

Thank you

Emanuele

[Christer Nordvik]

No, we moved over to Google SQL... 

-Christer

--
You received this message because you are subscribed to a topic in the Google Groups "objectify-appengine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/objectify-appengine/5vcUa_JiWeU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to objectify-appen...@googlegroups.com.