counter

[rjcarr]

I have a simple app engine app that offers a download. I catch the
download request and record the time of the download in the data store
before sending the file. I'd also like to keep track of the total
number of downloads. Obviously, the number of rows in the data store
represents the total number of downloads.

However, looking at the API, the count() method of Query return a
maximum of 1000. This won't work because my number will grow much
larger than that.

So as a fix I added a count field to my data model so that the count
is updated upon every request. It looks something like this:

class Counter(db.Model):
count = db.IntegerProperty()
date = db.DateTimeProperty(auto_now_add=True)

class DownloadHandler(webapp.RequestHandler):
def get(self):
query = Counter.all()
query.order("-count")

counter = query.fetch(1)
count = counter[0].count

counter = Counter()
counter.count = count + 1
counter.put()

This works, but I'm seeing quite a few timeout errors in the logs.

So, what am I missing? Is there a better way to keep track of a
counter? I'd prefer to make the counter persistent and not simply a
member of the class as that seems volatile and likely to reset (say,
on a redeploy).

[Sylvain]

Use this : http://paste.blixt.org/1581

From Google I/O

[Jeff S]

The reason that you are seeing timeouts, is that you have too many put
transactions overlapping one another (contention). When a transaction
fails, it is automatically retried. The sample that Sylvain pointed to
is an excellent solution to this common design problem. The sharded
counter avoids write contention by splitting the writes over multiple
entities. The faster the counter is being incremented, the more shards
you will need. This technique is an example of "scaling horizontally".
Here's another great design for a sharded counter, written by Bill
Katz:

http://oji.me/px

Happy coding,

Jeff

P.S. The Google I/O presentation that Sylvain's sharded counter was
from can be found here: http://sites.google.com/site/io/building-scalable-web-applications-with-google-app-engine
Counters are explained starting at around 19 minutes in.

[Robert J. Carr]

Hey Jeff and Sylvain-

Thanks for the info on the shard counter. I've been looking it over
and I don't completely understand the code, I understand the idea.

However, it is hard for me to believe it is a contention issue because
this is a very low traffic download. Something around 1 download
every two hours. The shard solution degrades to my counting solution
when the cache has expired.

Anyway ... thanks for the notes, just seems a strange solution when
simple aggregates would make it simple.

[Peter Recore]

On Oct 17, 1:32 pm, "Robert J. Carr" <rjc...@gmail.com> wrote:
<snip>

> Anyway ... thanks for the notes, just seems a strange solution when
> simple aggregates would make it simple.

Sometimes I think the datastore API *should* have generic sharded
counter functionality built in, as this would make programming
slightly easier. But I also *like* the lack of such a basic feature,
because it shocks many newcomers into finally realizing that the
datastore is not in fact a SQL database with a funny name, while also
providing a gentle introduction to powerful concepts like sharding.
In fact, maybe implementing a sharded counter should be a rite of
passage for everyone in the GAE community :)