TaskQueue and genetic algo tasks

[J.Ganesan]

Each task runs a genetic program. In desktop, each task takes 5 mins
and consumes 250 MB ram. A user-level job consists of 10 such tasks.
In desktop ( a normal Windows PC ), It takes about 50 minutes to
complete one user-level job.

Can I use App Engine to run the tasks in parallel and achieve a
response of ~5 mins for the user-level job ?

Specifically, a single http request triggers 10 DeferredTasks. The
DeferredTask need not do any datastore operation. Can taskqueue
process the DeferredTasks in parallel and handle the 10 DeferredTasks
simultaneously?

As each DeferredTask runs, it has to notify the progress to the
client using Channels API. Is it possible ? Channel being not
serializable, how can I keep a channel as a member variable to a
DeferredTask?

Could you please share your experience in CPU and memory intensive
DeferredTasks ?

J.Ganesan

[Jason Collins]

Each deferred task has the potential to spin up a new (parallel) instance, though there is some dependence on the scheduler and your Min/Max Pending Latency settings (in Application Settings). If $$ is not an object, you could increase the Min Idle Instances to ensure the instances are around. There is some instance start-up overhead, though that mostly depends on your own code/framework.

You'd need to use larger front-end instances to get the memory footprint you need. F2's have 256MB, so you'd likely need the big F4 (512MB).

For the channel API, you only need to know the channel ID to send the message. It is just a string, so you can include it in your task context easily.

Sounds like your plan will work out fine.

j

[J.Ganesan]

Clear. Thanks. I'll go ahead with the implementation. 

J.Ganesan