Multiprocessing in Production

[Kel Markert]

Hi all,

Has anyone tried to implement multiprocessing on a production server? I am trying to use the map function from the default Python multiprocessing package for quick I/O of data and processing on a web app. It works just fine on the dev but seems to hang up when kicking off a process in production.

Here is the github repo for the Tethys app that I am trying to implement this in: https://github.com/servir/altex/

Any guidance would be greatly appreciated.

All the best,

Kel

[Scott Christensen]

I'm not sure what is going on, but it may be related to the uwsgi sever that is used in production. See this issue https://stackoverflow.com/questions/44476678/uwsgi-lazy-apps-and-threadpool

--
You received this message because you are subscribed to the Google Groups "Tethys Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tethysplatfor...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/tethysplatform/725d32ed-8f95-4a20-97ce-73411dbf0eb6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Zhiyu (Drew) Li]

Hi Kel,

We are having issues with multiprocessing code in production as well. We have a controller function that spawns a new Process (process.start()), and then this control is supposed to return a response immediately WITHOUT waiting for the spawned Process to finish. After the controller function returns, the spawned Process is supposed to continue running till its end. So it is kind of a non-blocking/async call.

The code works in dev mode locally. But in production mode the controller doesn't return until the spawned process finishes -- becoming a blocking/sync call.

I have been searching online over the weekend and have tried different methods but it is still not working. I tried different combinations of the following options in uwsgi service yml file with no luck:

close-on-exec: true

lazy-apps: true

enable-threads: true

 buffer-size: 32768

 post-buffering: true

 gevent: 100

 async: true

I took a quick look at your code. If I understand it correctly you want to your controller to wait for the spawned Processes to finish, right? What error or unexpected behavior you saw in production?

I will keep you updated on my progress. Please also post back your progress.

Thanks

Drew

@BYU

Our code is like this:

import multiprocessing

import time

def newProcessFunc():

     print "newProcessFunc Start"

     # do something here

     time.sleep(30) # sleep 30 second

     print "newProcessFunc Stop"

def myControllerFunc (request):

     print "myControllerFunc Start"

     # spawn a new process

     p = multiprocessing.Process(target=newProcessFunc)

     p.start() 

           

     print "myControllerFunc End"

     return JsonResponse({''status": "success"}) 

On Sun, Feb 25, 2018 at 7:37 PM, Scott Christensen <scottych...@gmail.com> wrote:

I'm not sure what is going on, but it may be related to the uwsgi sever that is used in production. See this issue https://stackoverflow.com/questions/44476678/uwsgi-lazy-apps-and-threadpool

On Sun, Feb 25, 2018, 5:46 PM Kel Markert <kel.m...@nasa.gov> wrote:

Hi all,

Has anyone tried to implement multiprocessing on a production server? I am trying to use the map function from the default Python multiprocessing package for quick I/O of data and processing on a web app. It works just fine on the dev but seems to hang up when kicking off a process in production.

Here is the github repo for the Tethys app that I am trying to implement this in: https://github.com/servir/altex/

Any guidance would be greatly appreciated.

All the best,

Kel

--
You received this message because you are subscribed to the Google Groups "Tethys Platform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to tethysplatform+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/tethysplatform/725d32ed-8f95-4a20-97ce-73411dbf0eb6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "Tethys Platform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to tethysplatform+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/tethysplatform/CAJyNUrrzOWVb%3DKoNvvuyrDdST5sgobE9xrhq67F4t4PU11aaLQ%40mail.gmail.com.

[Zhiyu (Drew) Li]

We finally got it working by changing to the latest gunicorn 19.7.1 from conda-forge.

We created a tethys.gunicorn.service file at /etc/systemd/system/, and run "systemctl start tethys.gunicorn.service" instead of uwsgi service.

Also nginx should be changed to forward all incoming requests to this gunicorn server.

[Unit]

Description=Tethys gunicorn

After=syslog.target

[Service]

ExecStartPre=/bin/bash -c 'mkdir -p /var/log/gunicorn; chown www-data:www-data /var/log/gunicorn'

ExecStart=/home/tethys/tethys/miniconda/envs/tethys/bin/gunicorn tethys_portal.wsgi:application --name tethys-gunicorn --workers 8 --user www-data --group www-data --bind 127.0.0.1:49153 --timeout 600 --max-requests 2000 --env HOME=/home/www-data --log-level debug --access-logfile /var/log/gunicorn/access.log --log-file /var/log/gunicorn/error.log --capture-output --pythonpath=/home/tethys/tethys/miniconda/envs/tethys

ExecStartPost=/bin/bash -c 'chown -R www-data:www-data /var/log/gunicorn'

Restart=always

KillSignal=SIGQUIT

Type=simple

StandardError=syslog

NotifyAccess=all

[Install]

WantedBy=multi-user.target

Thanks

Drew

[swainn]

I had a similar issue with multi-threading and uwsgi. The problem was that it is not configured to allow multi-threading (and likely multi-processing) by default. As soon as the request is finished processing, the process dies and all child threads were killed with it. Getting around this was fairly simple though. There is a parameter you can add to the config or as a command line argument when you start uwsgi to enable multi-threading called "enable-threads". I don't know if it will work for multi-processing, but it is worth a shot.

Here is the option: http://uwsgi-docs.readthedocs.io/en/latest/Options.html?highlight=enable-threads#enable-threads

This option can be specified either in the config file or as a command line argument: http://uwsgi-docs.readthedocs.io/en/latest/Configuration.html?highlight=commandline#configuring-uwsgi

This is a good reference on uwsgi and mentions that multithreading is not supported by default: http://uwsgi-docs.readthedocs.io/en/latest/ThingsToKnow.html