Unexplained over quota error and some benchmarks and a question on ListProperty limits

[Aral]

I've started getting over quota errors while testing in my sandbox
application on appengine but I can't see why by looking at the
dashboard or the logs.

In the logs there is just one warning (see below).

I'm testing the responsiveness/limits of ListProperty entities and had
managed to put about 4700 keys in a list property before I got two 500
errors (not from my app but from Google) and then I started getting
the over quota errors.

Screen grab of the dashboard, showing usage at the time of the over
quota errors:

http://aralbalkan.com/images/SafariScreenSnapz022.png

Would appreciate it if someone could look into it.

Also, I know that we're told not to put a large amount of items in a
ListProperty but (a) you basically have to if you're building any sort
of messaging functionality and (b) in my tests, lookup of
ListProperties and saving large ListProperties is very responsive
(lookup on ListProperty with 4000+ items ~= 0.69 secs, saving record
with ListProperty with 4000+ items ~= 1.67 items). Is there some sort
of artificial limit around the 5000 key mark or is this a bug?

Thanks,
Aral

The log, showing the one error (and the entry before it from a few
days ago) follows:

78.143.202.194 - - [27/06/2008:14:06:19 -0700] "GET /add-friends/aral/
4201/ HTTP/1.1" 0 0 - -
E 06-27 02:06PM 19.166
<class 'django.template.TemplateSyntaxError'>: Caught an exception
while rendering:

Original Traceback (most recent call last):
File "/base/data/home/apps/aral/1.30/django/template/debug.py", line
71, in render_node
result = node.render(context)
File "/base/data/home/apps/aral/1.30/django/template/
defaulttags.py", line 149, in render
nodelist.append(node.render(context))
File "/base/data/home/apps/aral/1.30/django/template/debug.py", line
87, in render
output = force_unicode(self.filter_expression.resolve(context))
File "/base/data/home/apps/aral/1.30/django/template/__init__.py",
line 542, in resolve
new_obj = func(obj, *arg_vals)
File "/base/data/home/apps/aral/1.30/django/template/
defaultfilters.py", line 31, in _dec
args[0] = force_unicode(args[0])
File "/base/data/home/apps/aral/1.30/django/utils/encoding.py", line
58, in force_unicode
s = s.decode(encoding, errors)
File "/base/python_dist/lib/python2.5/encodings/utf_8.py", line 16,
in decode
return codecs.utf_8_decode(input, errors, True)
MemoryError

Traceback (most recent call last):
File "/base/data/home/apps/aral/1.30/main.py", line 45, in <module>
main()
File "/base/data/home/apps/aral/1.30/main.py", line 42, in main
util.run_wsgi_app(application)
File "/base/python_lib/versions/1/google/appengine/ext/webapp/
util.py", line 76, in run_wsgi_app
result = application(env, _start_response)
File "/base/data/home/apps/aral/1.30/django/core/handlers/wsgi.py",
line 205, in __call__
response = self.get_response(request)
File "/base/data/home/apps/aral/1.30/django/core/handlers/base.py",
line 123, in get_response
return debug.technical_500_response(request, *exc_info)
File "/base/data/home/apps/aral/1.30/django/views/debug.py", line
74, in technical_500_response
html = get_traceback_html(request, exc_type, exc_value, tb)
File "/base/data/home/apps/aral/1.30/django/views/debug.py", line
169, in get_traceback_html
return t.render(c)
File "/base/data/home/apps/aral/1.30/django/template/__init__.py",
line 176, in render
return self.nodelist.render(context)
File "/base/data/home/apps/aral/1.30/django/template/__init__.py",
line 751, in render
bits.append(self.render_node(node, context))
File "/base/data/home/apps/aral/1.30/django/template/debug.py", line
71, in render_node
result = node.render(context)
File "/base/data/home/apps/aral/1.30/django/template/
defaulttags.py", line 29, in render
output = self.nodelist.render(context)
File "/base/data/home/apps/aral/1.30/django/template/__init__.py",
line 751, in render
bits.append(self.render_node(node, context))
File "/base/data/home/apps/aral/1.30/django/template/debug.py", line
71, in render_node
result = node.render(context)
File "/base/data/home/apps/aral/1.30/django/template/
defaulttags.py", line 149, in render
nodelist.append(node.render(context))
File "/base/data/home/apps/aral/1.30/django/template/
defaulttags.py", line 243, in render
return self.nodelist_true.render(context)
File "/base/data/home/apps/aral/1.30/django/template/__init__.py",
line 751, in render
bits.append(self.render_node(node, context))
File "/base/data/home/apps/aral/1.30/django/template/debug.py", line
81, in render_node
raise wrapped

W 06-27 02:06PM 19.168
This request used a high amount of CPU, and was roughly 11.7 times
over the average request CPU limit. High CPU requests have a small
quota, and if you exceed this quota, your app will be temporarily
disabled.

06-16 06:01AM 25.898 /log/ 0 475ms 0 kb ...
E 06-16 06:01AM 26.371 <type 'exceptions.ImportError'>: No module
named a

[javaDinosaur]

I recall that a 100 item property list maximum has been mentioned.

Google is going to penalise and shutdown your App if pages routinely
take longer than 1 second to execute. 1 second is a very long running
web page in a well tuned web application on any platform.

[Aral]

> I recall that a 100 item property list maximum has been mentioned.

Well, as I mentioned in my post, I've tested successfully with a 4700
item PropertyList so empirical evidence would suggest that there isn't
a 100 item property list limit.

> Google is going to penalise and shutdown your App if pages routinely
> take longer than 1 second to execute. 1 second is a very long running
> web page in a well tuned web application on any platform.

Well, hosting really isn't about penalties. My other web hosts do not
penalize me, I pay more if I want more. If app engine is to be
successful, Google has to go beyond a mentality of penalties to
understanding that they are providing a commercial service and
competing with other web hosts. And the most important factor in
hosting is support. I'd say that that's where Google App Engine is
weakest at the moment.

Currently, I don't see any other way to create a messaging/micro-
blogging system (e.g., you write a message and it gets sent to your
3,000 friends) other than by using a ListProperty that's duplicated on
the Person entity and on the Message entities. You cannot use a
separate model and map reference properties as you would have to
create 3,000 mappings and we don't have long running processes. Ditto
for not being able to actually copy the message into the Person
entities of the 3,000 recipients (no long running processes).

ListProperties appear to work really well. Yes, the writes are not the
fastest but, as I mentioned in my post, none of those calls resulted
in an over CPU quota warning and, based on other testing, Google App
Engine appears to be fine with calls that last up to 8-9 seconds (see
the thread on faking a long running process using urlfetch, for
example.) 1.67 seconds to write an entity with a ListProperty that has
4,700 items in it is a metric I'm quite happy to live with (i.e.,
sending a message to 4,700 of your friends takes 1.67 seconds.) I can
live with that.

What I want to know is why I'm getting a Google App Engine 500 error
when I put an entity that has a ListProperty field with more than
4,700 or so items in it. (This is not an app 500 error but a Google
App Engine 500 error, where the error message points to Google
Support). This sounds like a bug in the system.

Based on my testing, Google App Engine should be able to handle
entities with ListProperties that have 10,000 or so items in them in
about half the allocated time for a call (I'm guesstimating around 3-4
seconds for writes and 1 sec for a read) which should be more than
enough to handle the boundary cases for most messaging/micro-blogging
apps.

Actually, I'm happy with even limiting things at 5,000 ala Facebook's
friend number limit but it does look like there's actually a bug (not
a limitation) on ListProperty fields with close to 5,000 items in
them. I'm going to open an issue for this on the tracker and see what
the response is.

[Aral]

I just logged a bug on the issue tracker concerning the 500 server
error issue with large ListProperty properties:

http://code.google.com/p/googleappengine/issues/detail?id=527

Aral

[javaDinosaur]

> Well, as I mentioned in my post, I've tested successfully with a 4700
> item PropertyList so empirical evidence would suggest that there isn't
> a 100 item property list limit.

Not a hard runtime limit but 100 was mentioned as a reasonable-use
design policy.

> Well, hosting really isn't about penalties. My other web hosts do not
> penalize me, I pay more if I want more.

The typical monthly hosting free for any infrastructure with the
redundancy that Google provides with AppEngine would be $100 to $5,000
per month.

The fact that Google can offer such a platform free of charge for up
to half a million hits per day is amazing. In order to achieve this
Google has to penalise unreasonable use. Putting 4000 child elements
into a persistent entity property counts as unreasonable use in my
book.

> I'm quite happy to live with (i.e., sending a message to 4,700 of your friends takes
> 1.67 seconds.) I can live with that.

You can, but can the Google hosting economics? Most developers manage
normal page processing in 100 to 300ms. Google allows the occasional
hit on long running pages but the 1.67 sec page sounds central to
your application’s functionality.

> And the most important factor in
> hosting is support. I'd say that that's where Google App Engine is
> weakest at the moment.

It is a preview/beta system at the moment so it is far too early to
assess support.

[Brett Morgan]

On Sat, Jun 28, 2008 at 9:53 PM, javaDinosaur <jonat...@hotmail.co.uk> wrote:

> Well, as I mentioned in my post, I've tested successfully with a 4700
> item PropertyList so empirical evidence would suggest that there isn't
> a 100 item property list limit.

Not a hard runtime limit but 100 was mentioned as a reasonable-use
design policy.

If I remember correctly, the 100 design policy was actually cribbed from the google io fireside chat, and the design point was that it is reasonable to be able to fetch 100 items for each request. It was basically refuting my earlier design gambit of trying to have a single entity housing all the data for each query. My design gambit sucked anyways. =)

In something I am playing with I need to get to the point of having a list of 30k ish items work. It hits high cpu at the moment, but that is going to be fixed at some point. Or if i get really ambitious there is always monkey patching. Ahem. Anyways.
 

> Well, hosting really isn't about penalties. My other web hosts do not
> penalize me, I pay more if I want more.

The typical monthly hosting free for any infrastructure with the
redundancy that Google provides with AppEngine would be $100 to $5,000
per month.

The fact that Google can offer such a platform free of charge for up
to half a million hits per day is amazing. In order to achieve this
Google has to penalise unreasonable use. Putting 4000 child elements
into a persistent entity property counts as unreasonable use in my
book.

It's the only way to achieve my design aims on top of GAE. So I will be doing it.
 

> I'm quite happy to live with (i.e., sending a message to 4,700 of your friends takes
> 1.67 seconds.) I can live with that.

You can, but can the Google hosting economics? Most developers manage
normal page processing in 100 to 300ms. Google allows the occasional
hit on long running  pages but the 1.67 sec page sounds central to
your application's functionality.

> And the most important factor in
> hosting is support. I'd say that that's where Google App Engine is
> weakest at the moment.

It is a preview/beta system at the moment so it is far too early to
assess support.

I think the support is brilliant. But I'm biased. ;-)

--

Brett Morgan http://brett.morgan.googlepages.com/

[Canis]

> Is there some sort of artificial limit around the 5000 key mark

Yes, there is:

http://groups.google.com/group/google-appengine/browse_thread/thread/d5f4dcb7d00ed4c6/88160410fcfdb372

[Aral]

Thanks you so much for the link, Canis.

I'm going to modify my issue ticket based on that to ask for a better
error message (a generic 500 error without any explanation does not a
good error make.) :)

Aral

On Jun 28, 3:53 pm, Canis <wooji.ju...@googlemail.com> wrote:
> > Is there some sort of artificial limit around the 5000 key mark
>
> Yes, there is:
>

> http://groups.google.com/group/google-appengine/browse_thread/thread/...

[Aral]

In my tests, my test entity just had the ListProperty in it. So for a
real entity (e.g., a message entity which a searchable index of
words), this limit is going to be lower.

I'm still all right with the limit (I don't mind limiting the number
of friends you have in the app at, say, 2-3K) but I still hope that
there will at least be a paid option for this cap and, worst case
scenario, at least have Google App Engine throw an error that we can
catch in the application.

Thanks,
Aral

[Canis]

There's a number of options you could look into, depending on the
needs of your application.

However, most of them are still going to run into resource limits of
one sort or another. Google will hopefully enhance GAE in future, but
in general, it's designed to cope with huge, huge numbers of separate
Entities, but not huge huge numbers of properties on any given
Entity.

Each Entity typically benefits from being quite simple and no list or
query can contain very many items before you start to hit resource
limits _somewhere_. It's just a fact of GAE's implementation at this
point.

Now, the way to get around this is, since Entities are the one thing
you can have lots of, you should generally try to convert values to
Entities. You're still limited on how many you can fetch at once,
though, so in the current GAE implementation, you're never going to be
able to "walk through" the full friends list simply by fetching it, if
it's over 1000 entities.

(However, if you distinguish them in some _ordered_ way, perhaps by
the time they were first added, you can page through them by modifying
your Query, rather than by using the fetch() limit/offset values. This
appears to be the recommended approach.)

Two things come to mind for your specific problem:

1. Blob and Text properties aren't indexed, so you can take a huge
list, turn it into a string (either by pickling or with the string
join/split methods) and store it in a Blob. This is simple, but you're
likely to burn CPU like crazy if the list is large. Still, it could be
of use.

2. You might be better off removing the list of friends altogether,
and instead having an Entity that represents the relationship:

class Friendship(db.Model):
person1 = db.ReferenceProperty(Person)
person2 = db.ReferenceProperty(Person)
timestamp = db.DateTimeProperty(auto_now_add=True)

A ReferenceProperty creates a back-link on the target, so you can use
Person.Friendship_set to find the friends of a particular user. I seem
to remember this is a Query so the usual fetch() limits apply -- but
if you order them by timestamp, you can use an inequality filter to
page through the list. You can also use equality filters to quickly
find out whether or not A is a friend of B.

-c