Cron Job and DeadlineExceededError

[Jorge Velázquez]

I guess I probably misunderstood and thought that cron jobs weren't
subject to the same 10 second request deadline for HTTP requests? I'm
trying to run a cron job that sends out a mailer. Unfortunately, it's
taking longer than 10 seconds so the job will fail about halfway
through. I guess I could rewrite it to only do batches at a time.
The means extra tables in my database to keep track of who has been
mailed, etc. Not a big deal, I guess. It just seems that this
restriction makes cron jobs fairly limited in what they can do. A
database cleanup job, for example, is going to take longer and longer
as tables grow and will almost certainly always hit this limit, and
create a situation where the cron scripts will have to be rewritten
and additional tables written to manage intermediate state, etc.

Is the intent that the cron jobs will remain subject to the same
restrictions as other requests?

[Wooble]

The request deadline is actually 30 seconds now, but yeah, as far as I
can tell cron jobs are just intended to free you from needing a cron
job on your own machine to hit a URL with wget or whatever
periodically.

Long-running processes are probably what you're looking for; hopefully
we'll see them soon.

In the meantime, the remote API might be the best solution for big
database cleanup jobs.

[Jorge Velázquez]

Thanks, Wooble, that cleared things up. I was a little confused by
the following log entries:

04-12 07:58PM 50.107 /mailer 500 28785ms 10749ms-cpu 0kb
See details
0.1.0.1 - - [12/Apr/2009:19:59:18 -0700] "GET /mailer HTTP/1.1" 500 0
- -
04-12 07:59PM 18.778
<class 'google.appengine.runtime.DeadlineExceededError'>:

So the 28785ms is where the problem is (not the 10749ms cpu)...this is
consistent with the recent post on the AppEngine blog about the
removal of the high CPU errors.

I guess my next concern is the fact that what I consider to be a
fairly simple script is taking so long to run. It is basically doing
a query of a user table that has ~500 rows and about 4 columns. From
this, it sends out an HTML email (usually about 3-4k in size) to each
user. This would take 2-3 seconds tops when I used to run it in a
Perl script when I was hosting my site elsewhere.

I did offload some of the Django template work on to Python. Since
there is some minor custom info in each email (user name, user id), I
was using the Django templates to format each message as I iterated
over the user list. Although there were only 3 variables that were
being substituted in the template, it seems to me that this is the
only possible place that could be causing such a huge performance
bottleneck. I haven't taken the time to profile it yet so I'll have
to do that whenever time allows.

Thanks again.

[DarkCoiote]

Hi... I have a similar situation:
I want to iterate over a large set of database entrys with a scheduled
cron job.

by now, I can manually use a page to do that using this:
http://code.google.com/appengine/articles/update_schema.html
(basically: iterates over small parts, and state is mainteined using
urls
strings used via 'get' and meta refresh stuff)

I havent tested, but what does the cron job do with the page
response?
Would the technic described in the above link work with cron jobs?

I guess it will not work...but.....
Thanks

> > can tellcronjobs are just intended to free you from needing acron

> > job on your own machine to hit a URL with wget or whatever
> > periodically.
>
> > Long-running processes are probably what you're looking for; hopefully
> > we'll see them soon.
>
> > In the meantime, the remote API might be the best solution for big
> > database cleanup jobs.
>
> > On Apr 12, 11:41 pm, jorge <jorge.velazq...@gmail.com> wrote:
>

> > > I guess I probably misunderstood and thought thatcronjobs weren't

> > > subject to the same 10 second request deadline for HTTP requests?  I'm

> > > trying to run acronjob that sends out a mailer.  Unfortunately, it's

> > > taking longer than 10 seconds so the job will fail about halfway
> > > through.  I guess I could rewrite it to only do batches at a time.
> > > The means extra tables in my database to keep track of who has been
> > > mailed, etc.  Not a big deal, I guess.  It just seems that this

> > > restriction makescronjobs fairly limited in what they can do.  A

> > > database cleanup job, for example, is going to take longer and longer
> > > as tables grow and will almost certainly always hit this limit, and

> > > create a situation where thecronscripts will have to be rewritten

> > > and additional tables written to manage intermediate state, etc.
>

> > > Is the intent that thecronjobs will remain subject to the same
> > > restrictions as other requests?