is that possible to run some time-costing task with GAE?

[J]

hey all,

in my app, i got to do time-costing task, for example, reading huge
data from several "big table" and computing result with those data
based on special algorithms, and what's important again, it's VERY
time-costing.

is that possible for my to perform those tasks/function on GAE?

after reading some intro and docs from GAE, i think a frankly way is
to call time-costing tasks/function in a "RequestHandler", but this
way will make browser keep waiting ..., may i to run the time-costing
task just like on a desktop pc? easy to start and stop, and dont need
trigger from browser.

thanks.

[Blixt]

You've got several obstacles preventing you from doing that:
1) Google App Engine will kill your request after 8 seconds or so.
2) If the process is CPU intensive, and uses many more CPU cycles than
a normal defined by Google, you'll get huge penalties to your CPU
quota (I managed to get almost 20% of my CPU quota used up in one 1-
second-request once.)

The only solution right now is to allow your activity to do a little
work each request and make multiple requests.

[J]

Hey Blixt,
Many thanks!

[glenc]

Yikes! 20% of quota in 1 second! Holy crap.

I needed to do something similar, where I need to basically run an A-
Star search algorithm to solve an optimisation problem on GAE. Now
I'm generally expecting the datasets to be fairly small and therefore
the runtime of the search should be small, however, I'm now very
worried about scalability if a single second of runtime is 20% of your
quota.

Sounds like it might get very very expensive over time. And breaking
up the computation into chunks-per-request shouldn't make a difference
to the total CPU bill ...

[Andrew Redkin]

Should be possible to run some javascript on client side which will
make server to execute small parts of your task. Of course, there will
be a problem where to store all the intermediate data, that depends on
your algorithm, if its small enough it could be possible to store it
in a "big table".

[Blixt]

@glenc
I actually think that particular 20% quota expenditure was a hiccup on
GAE's part, as it hasn't happened again and I still have some pretty
CPU intensive requests running in my app. But your quota will be used
up quickly once you start seeing those "a request used x times more
CPU cycles than expected" (or some such.) That was what I saw when my
used quota went up like a rocket.

[Steven M. Parker]

You may also be able to persists some data on the client side using
google gears, no? Perhaps it's not the solution for this particular
algorithm, but javascript and google gears could offload a lot of cpu
processing time for some apps.

[Glen Coates]

Yeah I was thinking Gears might be a good way to offload the
processing too. I think I'll prototype the search server-side and
then move it to Gears once the CPU cost becomes too high.

--
Cheers,
Glen

- myspace: http://www.myspace.com/sleepinggimp
- mixtapes: http://glen.coates.googlepages.com/mixtapearchive