Multiprocessing in python

[Jungho Ahn]

Hello,

Is it allowed to use python multiprocessing in appengine?

I tried, but it looks like only one core is utilized.

Thanks,

[Adam (Cloud Platform Support)]

Multiple cores are not available to App Engine standard runtime instances. If you need access to multiple cores on the same instance, consider using the App Engine flexible runtime, which uses Compute Engine as the underlying VM and allows you to configure the number of cores available.

[Jungho Ahn]

On Aug 27, 2016 4:16 PM, "'Adam (Cloud Platform Support)' via Google App Engine" <google-a...@googlegroups.com> wrote:
>
> Multiple cores are not available to App Engine standard runtime instances. If you need access to multiple cores on the same instance, consider using the App Engine flexible runtime, which uses Compute Engine as the underlying VM and allows you to configure the number of cores available.
>

Yes, but i can't use multiprocess in Python flexible appengine. When I created a process, it was a fake process which is a thread like in appengine.

>
> On Wednesday, August 24, 2016 at 4:27:17 PM UTC-4, Jungho Ahn wrote:
>>
>> Hello,
>>
>> Is it allowed to use python multiprocessing in appengine?
>> I tried, but it looks like only one core is utilized.
>>
>> Thanks,
>

> --
> You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/D6z_WoLsBH8/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to google-appengi...@googlegroups.com.
> To post to this group, send email to google-a...@googlegroups.com.
> Visit this group at https://groups.google.com/group/google-appengine.
> To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/88dafadf-a328-423e-8ddc-7c4f585f88a9%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Adam (Cloud Platform Support)]

Which Python runtime are you using? (the 'runtime' line in your app.yaml)

On Saturday, August 27, 2016 at 7:36:31 PM UTC-4, Jungho Ahn wrote:

On Aug 27, 2016 4:16 PM, "'Adam (Cloud Platform Support)' via Google App Engine" <google-appengine@googlegroups.com> wrote:
>
> Multiple cores are not available to App Engine standard runtime instances. If you need access to multiple cores on the same instance, consider using the App Engine flexible runtime, which uses Compute Engine as the underlying VM and allows you to configure the number of cores available.
>

Yes, but i can't use multiprocess in Python flexible appengine. When I created a process, it was a fake process which is a thread like in appengine.

>
> On Wednesday, August 24, 2016 at 4:27:17 PM UTC-4, Jungho Ahn wrote:
>>
>> Hello,
>>
>> Is it allowed to use python multiprocessing in appengine?
>> I tried, but it looks like only one core is utilized.
>>
>> Thanks,
>
> --
> You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/D6z_WoLsBH8/unsubscribe.

> To unsubscribe from this group and all its topics, send an email to google-appengine+unsubscribe@googlegroups.com.
> To post to this group, send email to google-appengine@googlegroups.com.

[Jungho Ahn]

I'm using python:

runtime: python

runtime_config:

    python_version: 2

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/2fd63c21-f33f-48a2-932d-bd7db89cb56c%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Jungho Ahn

[Adam (Cloud Platform Support)]

Possibly a silly question, but are you specifying at least 2 cores in your 'resources' configuration? The normal 'python' runtime should not be sandboxed and allow you to utilize more than one core.

On Monday, August 29, 2016 at 3:52:33 PM UTC-4, Jungho Ahn wrote:

I'm using python:

runtime: python

runtime_config:

    python_version: 2

[Jungho Ahn]

Yes, right. More than 2 cores.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/cc1bc1bd-188d-4490-a4dc-5c5138a68f2e%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Jungho Ahn

[Adam (Cloud Platform Support)]

Could you post an example of the code you're using to test, and how you're checking the CPU utilization on the target instance?

On Friday, September 2, 2016 at 6:01:03 PM UTC-4, Jungho Ahn wrote:

Yes, right. More than 2 cores.

[Jungho Ahn]

You mean the python in App Engine supports multiprocessing?

multiprocessing.Process() was not a real process.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/46012019-4c23-4e0f-a854-12449e34c1e7%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Jungho Ahn

[Adam (Cloud Platform Support)]

There's no special 'App Engine Python' on the 'python' runtime; it's just standard Python running in a Docker container. There are no core restrictions imposed by App Engine or by Docker.

In your app, are you checking the number of cores available using 'multiprocessing.cpu_count()'? When you say 'multiprocessing.Process() was not a real process' can you clarify what you mean, and describe how are you determining this (eg. pool._processes)?

You can also test the number of cores available to your deployed app from the Cloud Console by going to 'Compute' -> 'VM instances' -> 'SSH' to SSH to your instance in Debug mode, then from the shell type this to connect to your app's Docker container:

$ sudo docker exec -it gaeapp /bin/bash

Once there you can run 'nproc' to see the number of compute units or 'top' to see the cpu usage of each process eg.

root@d0f61e0950fe:/home/vmagent/app# nproc
2

root@d0f61e0950fe:/home/vmagent/app# TERM=xterm top
...

In my case I have 2 cores available, and my app.yaml looks like the following:

runtime: python
vm: true
entrypoint: gunicorn -w 4 -b :$PORT main:app

resources:
  cpu: 2.0
  memory_gb: 2.6
  disk_size_gb: 10

On Tuesday, September 6, 2016 at 1:14:48 PM UTC-4, Jungho Ahn wrote:

You mean the python in App Engine supports multiprocessing?

multiprocessing.Process() was not a real process.

[Jungho Ahn]

Yes, there are no core restrictions, but multiprocessing.Process() is a fake thread.

I received an answer from Google like

'For security reasons, multiprocessing is not supported in App Engine Standard Environment.  More specifically, the multiprocess module is replaced with a stub that provides limited compatibility (hence the "fake" PIDs you are seeing), but does not actually spawn processes.'

AFAIK, python threads cannot utilize all CPU cores due to GIL. So, I'm trying to fork processes for computationally heavy jobs.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/8d4d650e-79b2-43ca-843b-02320c64dd4b%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Jungho Ahn

[Vitaly Bogomolov]

AFAIK, python threads cannot utilize all CPU cores due to GIL. So, I'm trying to fork processes for computationally heavy jobs.

GAE taskqueue makes this job more productively. (IMHO)

G0LOD

[Adam (Cloud Platform Support)]

Yes, multiprocessing is not supported in the Google App Engine standard environment. The information I'm giving you pertains to the App Engine flexible environment. I'm sorry if there was any confusion, however in your second response you specifically stated "i can't use multiprocess in Python flexible appengine."

On Tuesday, September 6, 2016 at 2:14:25 PM UTC-4, Jungho Ahn wrote:

Yes, there are no core restrictions, but multiprocessing.Process() is a fake thread.

I received an answer from Google like

'For security reasons, multiprocessing is not supported in App Engine Standard Environment.  More specifically, the multiprocess module is replaced with a stub that provides limited compatibility (hence the "fake" PIDs you are seeing), but does not actually spawn processes.'

AFAIK, python threads cannot utilize all CPU cores due to GIL. So, I'm trying to fork processes for computationally heavy jobs.

[Jungho Ahn]

When I tested in python flexible appengine, it wasn't supported.

Do you mean it should be supported?

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/e6f21a22-b0fa-4b55-917a-bf40c881fd24%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Jungho Ahn

[Adam (Cloud Platform Support)]

Yes, there should be no explicit restrictions on multiprocessing on the flexible environment, unless you are using the 'python-compat' runtime. If you could provide some answers to the questions I asked in my previous posts, specifically on September 6th, it will be much easier to help you out. So far you've not given any real details about how you've been testing anything.

On Friday, September 9, 2016 at 2:37:52 PM UTC-4, Jungho Ahn wrote:

When I tested in python flexible appengine, it wasn't supported.

Do you mean it should be supported?