What's the definition of overly CPU-intensive requests

[Alex Popescu]

Hi,

This question is a bit long, so I'm doing my best to express it as
short as possible but keep it clear.

Basically, due to storage rules I am looking into having some "batch"
tasks that would be executed (mostly) executed at regular intervals (I
know that croned tasks are not supported). However, these will be
storage intensive and so they may require some time to be executed. I
am wondering if this falls under the definition/limitation explained
as above?

I would say that this kind of ops are IO intensive and so they
shouldn't be included in this quota. However, the request execution
time may be much longer than for normal requests.

I would really appreciate an authoritative answer before jumping to a
solution considering that it will require a quite complex setup.

thanks in advance,

./alex
--
.w( the_mindstorm )p.
Alexandru Popescu

[palp]

I don't know what the actual quota is for CPU intensive requests, but
if you're only doing it occasionally (once every hour or so?) you
shouldn't have any issues.
However, bear in mind that any execution that exceeds a certain amount
of time (8 seconds, I think) will stop and throw an exception.

On May 1, 6:35 pm, Alex Popescu <the.mindstorm.mailingl...@gmail.com>
wrote:

[Alex Popescu]

On May 2, 10:25 am, palp <palp...@gmail.com> wrote:
> I don't know what the actual quota is for CPU intensive requests, but
> if you're only doing it occasionally (once every hour or so?) you
> shouldn't have any issues.
> However, bear in mind that any execution that exceeds a certain amount
> of time (8 seconds, I think) will stop and throw an exception.
>

Thanks. Can you point me to the right spot in the documentation where
I can read about these? I don't think I've seen the 8seconds execution
limit mentioned anywhere.

tia

[Alex Popescu]

On May 2, 10:36 am, Alex Popescu <the.mindstorm.mailingl...@gmail.com>
wrote:

> On May 2, 10:25 am, palp <palp...@gmail.com> wrote:
>
> > I don't know what the actual quota is for CPU intensive requests, but
> > if you're only doing it occasionally (once every hour or so?) you
> > shouldn't have any issues.
> > However, bear in mind that any execution that exceeds a certain amount
> > of time (8 seconds, I think) will stop and throw an exception.
>
> Thanks. Can you point me to the right spot in the documentation where
> I can read about these? I don't think I've seen the 8seconds execution
> limit mentioned anywhere.
>

I couldn't find anywhere the 8 seconds limitation, but I have found
references to this limitation. Here are the quotes:

[1] http://code.google.com/appengine/docs/whatisgoogleappengine.html

Application code only runs in response to a web request, and must
return response data within a few seconds. A request handler cannot
spawn a sub-process or execute code after the response has been sent.

[2] http://code.google.com/appengine/docs/python/sandbox.html

An App Engine application cannot:
spawn a sub-process or thread. A web request to an application must be
handled in a single process within a few seconds. Processes that take
a very long time to respond are terminated to avoid overloading the
web server.

[3] http://code.google.com/appengine/docs/urlfetch/overview.html

Since your application must respond to the user's request within
several seconds, a URL fetch action to a slow remote server may cause
your application to return a server error to the user. There is
currently no way to specify a time limit to the URL fetch action.

One way to control the user experience of a slow remote server is to
use browser JavaScript to call a separate handler on your application
that performs the action that requires URL fetching, then report the
error to the user if the handler fails.

[end quotes/]

Unfortunately, all these are leading me to think once again that there
are way too many limitations or constraints set on the App Engine that
are making the life of a dev quite a hell :(. I usually don't like to
complain, but I must confess that I am having very hard times figuring
out how to write an app (which is quite simple) that would fit into
the app engine. To me it looks like App Engine is way too young and
has way too many constraints.

cheers,

[javaDinosaur]

> To me it looks like App Engine is way too young
> and has way too many constraints.

On a typical web software stack (C# or Java), 8 seconds is a large
time span for processing an http request. A web page that takes more
than a second including a few single row DB operations is a candidate
for tuning.

Consuming more than 1 second of pure CPU time (excluding DB I-O) in an
http request takes some really intensive and inefficient coding. So
the only things that should push you over the 8 second limit would be
too many or blocking Datastore operations or very slow URLfetches to a
3rd party site.

If your remote URLfetch might take more than a couple of seconds to do
its thing, you could code around this situation by passing a task-id
token to the remote site and design the remote site to process the
request asynchronously. Your App Engine http can store the job-id in
the Datastore and return well under 8 seconds. When the remote server
finishes the task it can invoke a "job finished" http interface on the
App Engine site. In the mean time your Ajax style browser can poll for
job finished once every 10 seconds.

Confused? :-)

[Alex Popescu]

I really appreciate your comments, but you should have read also my
post: this is for 'batch' -like processing requests. Basically no end
user involved, but a means to have cron-like tasks. So, even if your
comments do apply for normal end-user requests, they completely don't
fit in this thread.

Moreover, if you read carefully the quotes, you'll notice that the
limitation is expressed in response time so it is not CPU-only time,
but rather request processing time/wall-clock time (which includes
both CPU and IO time).

cheers,

./alex
--
.w( the_mindstorm )p.
Alexandru Popescu

[javaDinosaur]

I take your point about 8 seconds being a stopwatch limit including
CPU and I-O, I was aware of this.

What volume of data are you considering in these batch events?

Have you thought about crunching this data down via pickle and storing
the data in Blob properties? If you could stash 100 data items per
Datastore write you might be able to do more batch work per 8 second
http hit.

[redsox2005]

So, are you hitting a "page" with some scheduled job to start your
"batch" jobs? If so, I suppose the response would be "ok, I finished,
and I processed 52010 records". And, if you break your task into many
small parts, you could perhaps hit your "job page" with a batch of
requests, and finish your job.

I've done this. I have a "job" that takes about 30 seconds. I
structured a model to keep track of my progress, and I only process a
few thousand entities at a time. It takes about 4 seconds per
request, and I return the "count" to the remote caller. My first
setup call returns the "total count" in my batch job, and each
processing call returns the "processed count". I delay a second
between each new call, and call again, until my returned count is
zero. I have a little script on my PC scheduled to run twice a day
to start the process.

This would take to long if I were processing millions of records, but
I'm not, so it works fine.

> > Confused? :-)- Hide quoted text -
>
> - Show quoted text -

[javaDinosaur]

> I have a "job" that takes about 30 seconds.  I
> structured a model to keep track of my progress, and I only process a
> few thousand entities at a time.  It takes about 4 seconds per
> request

These numbers give me a lot of encouragement about what could be
possible with App Engine.

Do you have a figure at hand for the number of Model.put() requests
you process in a 4 second window?

[Ken Rimey]

I wonder if talking about the hard time limit of "a few
seconds" (apparently 8) might be optimistic. Executing a request that
does nothing but time.sleep(2.0) elicits the following warning: "This
request used a high amount of CPU, and was roughly 6.7 times over the
average request CPU limit. High CPU requests have a small quota, and
if you exceed this quota, your app will be temporarily disabled."

[javaDinosaur]

> Executing a request that
> does nothing but time.sleep(2.0) elicits the following warning: "This
> request used a high amount of CPU, and was roughly 6.7 times over the
> average request CPU limit.

Ou err.

Two thoughts:

1: Someone raised a ticket for the App Engine team this week about the
CPU quota consumption maths seeming a bit skewed.

2: Could the sleep() function have a faulty implementation, that spins
a CPU at 100%? Two seconds of uninterrupted CPU consumption for a
single http request is unreasonable and deserves to be flagged.

[Alex Popescu]

I have to emphasize again that according to the documentation there is
NO reference to CPU time. If you carefully read the above quotes
(those are copyied'n'pasted from the published docs), you'll notice
that they are talking in fact about wall clock time (total execution
time) and Ken's stats are clearly proving me right :).

[javaDinosaur]

Alex I have not doubted the existence of a wall clock time cutoff per
http request.

The docs also refer to daily a CPU quota. The text of the warning
message up few posts above leads me to believe that Google is rightly
monitoring for pages that burn a CPU core at a high level for perhaps
a second or more. I think the warning message is saying, too many hits
on high CPU pages will trigger a block on a site before other daily
limits come into effect.

[Alex Popescu]

I guess the only chance to figure this out is to wait for a developer
to comment on it :-).

[palp]

The best way to deal with this is probably to find a way to split the
entire job into a set of smaller tasks, then execute those tasks in
parallel. This should end up with the task actually completing much
faster than it would have using a traditional large single job.

This brings up the reason why these limits are in place, to force you
to write a scalable application. Doing a single large batch may
usually be considered reasonable, but if you are dealing with millions
of operations in that batch, it becomes a huge problem. If you split
it up into finite chunks, it can then scale on a cluster
appropriately. Is this kind of work needed for small, simple apps?
Absolutely not. But GAE's main draw is the built-in scalability,
which a lot of people seem to overlook when complaining about the
constraints placed on them.

[Alex Popescu]

On May 2, 8:54 pm, palp <palp...@gmail.com> wrote:
> The best way to deal with this is probably to find a way to split the
> entire job into a set of smaller tasks, then execute those tasks in
> parallel.  This should end up with the task actually completing much
> faster than it would have using a traditional large single job.
>
> This brings up the reason why these limits are in place, to force you
> to write a scalable application.  Doing a single large batch may
> usually be considered reasonable, but if you are dealing with millions
> of operations in that batch, it becomes a huge problem.  If you split
> it up into finite chunks, it can then scale on a cluster
> appropriately.   Is this kind of work needed for small, simple apps?
> Absolutely not.  But GAE's main draw is the built-in scalability,
> which a lot of people seem to overlook when complaining about the
> constraints placed on them.
>

I do agree with all your points, but basically it is not GAE's scaling
out, but rather your own code. The App Engine is currently missing
almost all mechanism for helping you scaling out. It is rather
imposing you a set of rules that will make your app scale out, but you
really have to jump through hoops to get there.

[palp]

Agreed, that was the intent of what I was saying. One of the requests
currently on the tracker is to implement a way to use Google's own Map/
Reduce functionality within GAE, which I think would be a great
solution to this problem.

On May 2, 11:40 am, Alex Popescu <the.mindstorm.mailingl...@gmail.com>
wrote: