about max execution time

[changhefirst]

Is there any limit of execution time of scripts? If there is, how long
is this max execution time for a function?

Is there any way to set the max_execution_time or functions like
set_time_limit() in php?

If I want my script to execut in a long time to finish its job, maybe
20 mininuts, just looks like a service, is there any way in App
Engine ?

thanks.

ps, sorry for my poor English.

[Ross Ridge]

changhefirst wrote:
> Is there any limit of execution time of scripts? If there is, how long
> is this max execution time for a function?

A few seconds, eight or ten seconds at the most.

> Is there any way to set the max_execution_time or functions like
> set_time_limit() in php?

I don't think so.

> If I want my script to execut in a long time to finish its job, maybe
> 20 mininuts, just looks like a service, is there any way in App
> Engine ?

Nope. Google App Engine is only designed for applications that can
respond to requests in under 300 ms or so.

Ross Ridge

[javaDinosaur]

> If I want my script to execut in a long time to finish its job, maybe
> 20 mininuts, just looks like a service,  is there any way in App
> Engine ?

Not yet but scheduled and long running tasks are one of the most
popular requests. I think in one of the Google I-O videos an AppEngine
developer hinted Google is actively working on such a feature.

In the mean time you could checkout Amazon's EC2 virtual machine
offering, trouble is the entrylevel cost starts at $72 per month for
the smallest VM then you need to add storage and bandwidth usage costs.

[changhefirst]

Thank you for your reply, Ross Ridge.

Is there any official document from google about this yet, I did not
find one? Actually if we do not know how much CPU power and time we
could use, it is really difficult to do more than a simple web site.
But I want to create web services on App Engine...

On Jun 21, 5:35 pm, javaDinosaur <jonathan...@hotmail.co.uk> wrote:
> > If I want my script to execut in a long time to finish its job, maybe
> > 20 mininuts, just looks like a service, is there any way in App
> > Engine ?
>

> Not yet but scheduled and long running tasks are one of the most
> popular requests. I think in one of the Google I-O videos an AppEngine
> developer hinted Google is actively working on such a feature.
>
> In the mean time you could checkout Amazon's EC2 virtual machine
> offering, trouble is the entrylevel cost starts at $72 per month for
> the smallest VM then you need to add storage and bandwidth usage costs.

OH YEAH !

Google, plaese rush for this feature, it is really important !

Thank you for your suggetion, javaDinosaur.

Amazon's EC2 is a good option, but even the smallest VM is too much
powerful to new borned web service... If Amazon could set some smaller
VM servers or Google could provide some API for long time
function...it would be awosome!

[Ross Ridge]

changhefirst wrote:
> Is there any official document from google about this yet, I did not
> find one?

The a "few seconds" for a request comes from the "What is Google App
Engine?" section of the docs:

http://code.google.com/appengine/docs/whatisgoogleappengine.html

Some features impose limits unrelated to quotas to protect the
stability of the system. For example, when an application is
called to serve a web request, it must issue a response within
a few seconds. If the application takes too long, the process
is terminated and the server returns an error code to the
user. The request timeout is dynamic, and may be shortened if
a request handler reaches its timeout frequently to conserve
resources

The 8 or 10 seconds absolute upper limit comes from various messages
on this group, and the 300 ms upper limit for a typical request comes
from this warning that can appear in the logs:

This request used a high amount of CPU, and was roughly 1.3
times over the average request CPU limit. High CPU requests
have a small quota, and if you exceed this quota, your app
will be temporarily disabled

> Actually if we do not know how much CPU power and time we
> could use, it is really difficult to do more than a simple web site.
> But I want to create web services on App Engine...

I think it's possible to do some pretty complex WWW sites within the
context of GAE actually. Most WWW sites want to respond to requests
in well under a second, otherwise the site will appear to be slow and
sluggish. I think Google is only looking at providing support for
longer requests only for periodic administrative tasks, things like
cleaning out stale entities out of the datastore, for example. I
don't think they're intending to support computationally expensive web
services. Pure Python, without any extension libraries, isn't really
the language for that sort of task.

Ross Ridge

[changhefirst]

Thank you for your reply, Ross Ridge.

I think I should study EC2 for main web service function, and use GAE
like a new kind of virtual host.

At least the url fetch API of AGE support https, so I can combind
them.

[Brett Morgan]

On Sat, Jun 21, 2008 at 7:35 PM, javaDinosaur <jonat...@hotmail.co.uk> wrote:

> If I want my script to execut in a long time to finish its job, maybe
> 20 mininuts, just looks like a service,  is there any way in App
> Engine ?

Not yet but scheduled and long running tasks are one of the most
popular requests. I think in one of the Google I-O videos an AppEngine
developer hinted Google is actively working on such a feature.

More than hinted, there is an AppEngine blog post dated April 11 saying that background processing is a focus:

http://googleappengine.blogspot.com/2008/04/were-up-and-running.html

If I understand things correctly, one of the major things about background processing for the AppEngine team is how to make background processing work. There are concerns about distributability, fairness of allocation, and dealing with restarts in the case of server failure.

They effectively need to do something similar to what happened with re-shaping people's data storing habbits to allow for background processing to scale.

Being the guy that I am, I'd really like to play with map-reduce functionality directly in python. But that is a fairly massive mindshift for most developers.

So, the question for the group is, how to design a background processing api such that it is distributable, and works in with DataStore's transactions? What do we need to be able to do what we want to do. In fact, what is it that you guys want to achieve with background processing?

--

Brett Morgan http://brett.morgan.googlepages.com/

[Michael R. Bernstein]

On Jun 26, 10:18 pm, "Brett Morgan" <brett.mor...@gmail.com> wrote:
>
> So, the question for the group is, how to design a background processing api
> such that it is distributable, and works in with DataStore's transactions?
> What do we need to be able to do what we want to do. In fact, what is it
> that you guys want to achieve with background processing?

Well, generally speaking, what I'd like to do is chew through rather
large amounts of collected data (say, user ratings or user bookmarks),
and extract recommendations ('we think you'll like these items') of
various sorts. An extreme example that comes to mind (in terms of the
size of the corpus and the CPU-intensiveness of the iterated
algorithm) is pagerank, but even much smaller problems like movie
recommendations are quite large.

There are any number of data structures and algorithms that fit into
this general pattern, but from my perspective, what they have in
common is:

1) operating across multiple users' data
1a) a subset of these problems can be sharded into more manageable
groups of users, ie. recommendations based on my friends' (or even
friends-of-friends') data only.
1b) a different subset cannot, ie. finding a set of users whose
recommendations are similar to mine.
2) periodic (episodic?) offline processing. ie. no need to recalculate
every time a user adds data, nor should the recalculation prevent
adding new data. This potentially implies operating on a snapshot.
3) there are native-code libraries (such as NumPy or LIBSVM) that
would be *very* helpful, but not strictly necessary.

There is a completely different set of problems that involve only
infrequent (even one-time) CPU-intensive transformation of a single
user's data that you would want to fire-and-forget without being bound
to an HTTP request's limits. However, most (if not quite all) of these
more-or-less *require* native-code libraries that aren't usable in GAE
right now (even for non-CPU-intensive purposes that *could* be so
bound) anyway.

- Michael

[changhefirst]

On Jun 27, 1:18 pm, "Brett Morgan" <brett.mor...@gmail.com> wrote:

> More than hinted, there is an AppEngine blog post dated April 11 saying that
> background processing is a focus:
>
> http://googleappengine.blogspot.com/2008/04/were-up-and-running.html

Thank you for your information. I read this blog post.

In my opinion, google have to solve these two thing, or it is hard to
say App Engine can be used to develop bussiness applications :

"Support for offline processing. Right now Google App Engine is great
for web apps that do all of their processing in response to user
requests, but what about apps that need to perform scheduled tasks or
larger-scale data migration? We'd like to support those apps too.

Support for large files. Google App Engine currently imposes a limit
of 1MB on all requests, both inbound and outbound. We're interested in
providing efficient support for much larger uploads and downloads."

What I need is smiple :

1 A setting like max_execution_time or functions like set_time_limit()
in php. In this way, I handle how long a funtion could go, and pay for
it.

2 Cron jobs.

3 Let me set the limit size of data returned in every request.

Otherwise, google's Cloud Computing is just a toy of google itself,
but a joke for us, as there is no much computation you could do every
request...