Strange Backend Behavior

[Volker Schönefeld]

Hey,

we've accumulated a large dataset on our appengine (almost a terrabyte
now), and we're struggling to delete it (mostly old data that is no
longer needed).

After years of trying this and that, I gave backends a shot, but the
behavior is really strange.

Essentially, I've got a little deleter script set up like this:

backends:
- name: deleter
class: B2
start: backends/deleter.py
instances: 1

In the backends/deleter.py I then do:

import logging
logging.info("Starting deleter...")

import datetime
import common
now = datetime.datetime.now()
old_game_date = now+common.MAXIMUM_GAME_AGE
logging.info("Finding games before %s"%old_game_date)
from models import Game
from google.appengine.ext import db
q = db.Query(Game.Game, keys_only=True)
q.filter("date <", old_game_date)
games = q.fetch(500)
count = 0
while games and len(games)>0:
db.delete(games)
games = q.fetch(500)
count += len(games)
logging.info("Done, deleted %s games."%count)

This produces the following log message:

A serious problem was encountered with the process that handled this
request, causing it to exit. This is likely to cause a new process to
be used for the next request to your application. If you see this
message frequently, you may have a memory leak in your application.
(Error code 201)

However, if I stop the deletion after a few games, like

if count > 1000:
break

It actually works:

2011-09-03 05:13:51.386
Starting deleter...
I 2011-09-03 05:13:51.390
Finding games before 2011-08-04 12:13:51.390348
I 2011-09-03 05:14:01.034
Done, deleted 1500 games.

Also, the logservice.flush() doesn't seem to work at all, I was
getting the same error I'm getting above and _no_ output at all, even
if i only had a simple script like:

from google.appengine.api import logservice
import logging
logging.info("Starting deleter...")
logservice.flush()

So something seems to be up in the land of the backends. Please
advise.

Cheers,
Volker

PS: our appengine id is "towermadness", the backend is "deleter"

[Tim Hoffman]

With this code you I think you won't be reclaiming any memory.

You could try a gc.collect, in addition move the deletions in your code so that the deletes occurr in a function.

I would have the outer loop use a cursor, pass the query to a function that does the fetch of the keys

and does the delete then returns.

Have stuff full out of scope could help.

then call a gc.collect in the outer loop before getting the next cursor.

Rgds

T

[Volker Schönefeld]

Hey,

thanks, that actually seemed to have helped. I still don't get any log
messages, but I'm now using memcache to output some debug information
into a handler running on the server, and it's actually deleting data.

Thanks for the help!