Is it possible to use multiprocessing Pool in Python 2.7 on App Engine?

[Luca de Alfaro]

I have some processing that would benefit from parallelism -- the multiprocessing Pool class would be ideal.  

Is it available in App Engine?  The Mapping API does not work for my purpose, since it is in-memory processing that I am trying to do. 

Thanks! 

Luca

[Brian Quinlan]

Have you considered using concurrent.futures:
http://pypi.python.org/pypi/futures ?



Cheers,
Brian

> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/yzxvmQQyWAIJ.
> To post to this group, send email to google-a...@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengi...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

[Luca de Alfaro]

No, and that would be perfect... is that supported on AppEngine? 

And btw, how do I find out about which packages are supported?  I did not know that some futures packages from pypi could by supported.  I ask because there are a few more packages (scipy, for instance) that I really wish would be supported...

Luca

On Sunday, May 27, 2012 2:39:18 AM UTC-7, Brian Quinlan wrote:

Have you considered using concurrent.futures:
http://pypi.python.org/pypi/futures ?



Cheers,
Brian

On Sun, May 27, 2012 at 5:47 PM, Luca de Alfaro
<luca.de...@gmail.com> wrote:
> I have some processing that would benefit from parallelism -- the
> multiprocessing Pool class would be ideal.
> Is it available in App Engine?  The Mapping API does not work for my
> purpose, since it is in-memory processing that I am trying to do.
> Thanks!
>
> Luca
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/yzxvmQQyWAIJ.

> To post to this group, send email to google-appengine@googlegroups.com.

> To unsubscribe from this group, send email to

> google-appengine+unsubscribe@googlegroups.com.

> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

On Sunday, May 27, 2012 2:39:18 AM UTC-7, Brian Quinlan wrote:

Have you considered using concurrent.futures:
http://pypi.python.org/pypi/futures ?



Cheers,
Brian

On Sun, May 27, 2012 at 5:47 PM, Luca de Alfaro
<luca.de...@gmail.com> wrote:
> I have some processing that would benefit from parallelism -- the
> multiprocessing Pool class would be ideal.
> Is it available in App Engine?  The Mapping API does not work for my
> purpose, since it is in-memory processing that I am trying to do.
> Thanks!
>
> Luca
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/yzxvmQQyWAIJ.

> To post to this group, send email to google-appengine@googlegroups.com.

> To unsubscribe from this group, send email to

> google-appengine+unsubscribe@googlegroups.com.

[Brian Quinlan]

On Mon, May 28, 2012 at 2:08 AM, Luca de Alfaro
<luca.de...@gmail.com> wrote:
> No, and that would be perfect... is that supported on AppEngine?

The ThreadPool functionality will work fine but you might have to
remove the multiprocessing imports.

> And btw, how do I find out about which packages are supported?

There is no list of packages supported on App Engine - you have to
look into the ones that you want to use.

> I did not
> know that some futures packages from pypi could by supported.  I ask because
> there are a few more packages (scipy, for instance) that I really wish would
> be supported...

Many (possibly most) packages from pypi are supported. But packages
won't work if:
- they have C/C++/FORTRAN components (scipy has these)
- they require features that App Engine doesn't support such as a
writeable filesystem

Cheers,
Brian

>
> Luca
>
> On Sunday, May 27, 2012 2:39:18 AM UTC-7, Brian Quinlan wrote:
>>
>> Have you considered using concurrent.futures:
>> http://pypi.python.org/pypi/futures ?
>>
>>
>>
>> Cheers,
>> Brian
>>
>> On Sun, May 27, 2012 at 5:47 PM, Luca de Alfaro
>> <luca.de...@gmail.com> wrote:
>> > I have some processing that would benefit from parallelism -- the
>> > multiprocessing Pool class would be ideal.
>> > Is it available in App Engine?  The Mapping API does not work for my
>> > purpose, since it is in-memory processing that I am trying to do.
>> > Thanks!
>> >
>> > Luca
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "Google App Engine" group.
>> > To view this discussion on the web visit
>> > https://groups.google.com/d/msg/google-appengine/-/yzxvmQQyWAIJ.

>> > To post to this group, send email to google-a...@googlegroups.com.

>> > To unsubscribe from this group, send email to

>> > google-appengi...@googlegroups.com.

>> > For more options, visit this group at
>> > http://groups.google.com/group/google-appengine?hl=en.
>
>
> On Sunday, May 27, 2012 2:39:18 AM UTC-7, Brian Quinlan wrote:
>>
>> Have you considered using concurrent.futures:
>> http://pypi.python.org/pypi/futures ?
>>
>>
>>
>> Cheers,
>> Brian
>>
>> On Sun, May 27, 2012 at 5:47 PM, Luca de Alfaro
>> <luca.de...@gmail.com> wrote:
>> > I have some processing that would benefit from parallelism -- the
>> > multiprocessing Pool class would be ideal.
>> > Is it available in App Engine?  The Mapping API does not work for my
>> > purpose, since it is in-memory processing that I am trying to do.
>> > Thanks!
>> >
>> > Luca
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "Google App Engine" group.
>> > To view this discussion on the web visit
>> > https://groups.google.com/d/msg/google-appengine/-/yzxvmQQyWAIJ.

>> > To post to this group, send email to google-a...@googlegroups.com.

>> > To unsubscribe from this group, send email to

>> > google-appengi...@googlegroups.com.

>> > For more options, visit this group at
>> > http://groups.google.com/group/google-appengine?hl=en.
>

> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit

> https://groups.google.com/d/msg/google-appengine/-/HNyRzPrDYQUJ.
>
> To post to this group, send email to google-a...@googlegroups.com.

> To unsubscribe from this group, send email to

> google-appengi...@googlegroups.com.

[Dale Humby]

@Luca: Have you seen the list of supported Python libraries? https://developers.google.com/appengine/docs/python/runtime#Pure_Python