App Engine really isn't designed for use as a backend for huge computing jobs

[le yu]

FYI.

http://stackoverflow.com/questions/8105564/can-google-app-engine-be-used-for-massively-parallel-computation

check out the one and also the only answer in this thread

[Dongxi Zheng]

Do we have a better choice?

> --
>
>

[le yu]

another thread on stackoverflow. 

http://stackoverflow.com/questions/4783580/is-google-app-engine-mapreduce-my-best-bet-for-a-massively-parallel-solution-in

summary of the thread above: 

- App Engine is focused on serving web requests, not pure number crunching

- long running jobs are not possible (10 mins limit)

- Amazon EC2 could be a backup

Let's need look into more on EC2, GAE and hadoop.

Cheers!!!

[Dongxi Zheng]

Hadoop would probably does the trick, but we won't have a physical cluster available.

--
 
 

[le yu]

Currently two issues trouble us:

one, the algorithm

another, the platform 

I suggest this stage we familiarize the algorithm as soon as possible. It is not just the Floyd-Warshall itself, there are one more basic dynamic programming algorithm (in the slide-like document I shared last time) which Floyd-Warshall is related to. Also, the divide-and-conquer version is closely related the matrix multiplication. So I estimate some effort will be required on understanding the divide-and-conquer algorithm of matrix multiplication. 

I still don't have chance to dive a lot on the computing platform. But currently, I think we need to have a general idea of the CANs and CAN'ts of the three major platform. 

I will post more if I get progress. 

Le

[le yu]

hmm.... that make choice simple, EC2 or GAE. I once had a glance on EC2, not bad at least.....

[Po-Chun Chang]

I now are working on understanding the algorithm, if possible, I think
I will be ready in Fri or Sat.

> --
>
>

[Po-Chun Chang]

Though primitive, it is there I feel.
https://developers.google.com/appengine/docs/python/dataprocessing/mapreducepipelineclass#Introduction

[Po-Chun Chang]

On Wed, Nov 7, 2012 at 2:07 PM, le yu <yoyos...@gmail.com> wrote:
> http://www.norstad.org/matrix-multiply/index.html
> matrix multiplication on mapreduce. Get closer....
> to dongdong: does condor provide the api from parallel programming ?

I guess no, unless we could take advantage of MW facility. Which, by
my understanding, is a master-worker scheduler.
I don't think there are explicit synchronization we could leverage there.

>
>
> 2012/11/7 Po-Chun Chang <pocha...@gmail.com>

[Po-Chun Chang]

There there, more on MapReduce of GAE

http://code.google.com/p/appengine-mapreduce/source/browse/#svn%2Ftrunk%2Fpython%2Fdemo
http://code.google.com/p/appengine-mapreduce/wiki/MapReduceDemoApp