Instances + Memcache + Users

[jay]

Quick question,

In my app users "do work" on some data. Every action they do I have to
pull the data out of the datastore, unpickle it, do the transform,
pickle it again and put it back in the datastore.


Do today I'm doing some optimization where I'll look in the memcache
first for the unpicked data.

I was wondering if the user is sitting at their desk doing something
every 5-10 seconds or so, and I have 5 or 6 instances running, are
they likely to be hitting the same instance over and over, based on
geography (or something else), or will they be making requests of a
completely random one of my instances.

Just wondering.

Thanks in advance.

[jay]

Oh, I just read that memcache is constant across instances.

[Robert Kluin]

Hi Jay,
Correct, memcache is global to your app. The same user's requests
could go to any of your instances. So caching user datain local
memory is probably not worth it.

Robert

> --
> You received this message because you are subscribed to the Google Groups "Google App Engine" group.
> To post to this group, send email to google-a...@googlegroups.com.
> To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
>
>

[jay]

do we know how big the memcache is? It would be good to know roughly
how much data we can jam in there.

[Robert Kluin]

I don't think they publish any specific amount. You might want to do
some basic testing to see. Also, they've lowered the RAM on
front-ends to 128mb. I would not be surprised if they've also reduced
the available memcahce space, so I wouldn't rely on any old
information.

Robert

[Waleed Abdulla]

There is a function, get_stats(), that gives you some data about your memcache usage, including the total size of stored data. You might want to run a test by starting as much as you can in memcache while watching the total stored data and when that hits a ceiling, you'll know how much is available to you. My guess, though, is that the value probably changes because it wouldn't be efficient to set aside some memcache space for each app whether it's used or not, so the memcache space is probably shared among apps. 

http://code.google.com/appengine/docs/python/memcache/functions.html

[Simon Knott]

Robert, where did you get the 128MB limit from?  Is this limit only in effect when the new billing comes in?

I noticed in the recent IRC chat log that Ikai mentioned that they had noticed more applications hitting a 180MB soft limit, reduced from 200MB, with a 300MB hard limit.

[Brandon Wirtz]

Green line is memory in Megabytes Peaking at 2000 and change

Instances range up to 26 total, and like 8 active.

 

Sorry they don’t quiet line up I was being fast…

 

26 total instances with 2100 MB of Memory = 80 MB per Instance. Then very clearly something Flushes and it goes back up…

 

I don’t have a way to prove I am hitting a synthetic limit, but it would appear that way.

 

I’m using Instance Memory, before Mem-Cache, before Datastore.  So I should be pretty memory intensive (by design)

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/O9LZKiDgTJQJ.

[Robert Kluin]

Hey Simon,
I know it is documented on the backends page:
http://code.google.com/appengine/docs/python/backends/overview.html#Properties_of_Backends

I'm not sure it is documented anywhere else. They are aligning the
front-end instances with the most low-end backend instance:
http://code.google.com/appengine/docs/python/backends/overview.html#Billing_Quotas_and_Limits

Robert

> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.

> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/O9LZKiDgTJQJ.