Issue with loadmodule in mac

[Vishnu Prasad]

I download mod_wsgi and ran python setup.py install. As per your suggestion, am moving away from simple httpserver to Flask enabled one. 

I get following on running module-config

LoadFile "/Users/vph/anaconda/lib/libpython3.6m.dylib"

LoadModule wsgi_module "/Users/vph/anaconda/lib/python3.6/site-packages/mod_wsgi-4.5.16-py3.6-macosx-10.7-x86_64.egg/mod_wsgi/server/mod_wsgi-py36.cpython-36m-darwin.so"

WSGIPythonHome "/Users/vph/anaconda"

I got the following on running install-module

 sudo mod_wsgi-express install-module

LoadFile "/Users/vph/anaconda/lib/libpython3.6m.dylib"

LoadModule wsgi_module "/usr/libexec/apache2/mod_wsgi-py36.cpython-36m-darwin.so"

WSGIPythonHome "/Users/vph/anaconda"

My apache2 httpd.conf is :

LoadModule wsgi_module "/usr/libexec/apache2/mod_wsgi-py36.cpython-36m-darwin.so"

If I start mod_wsgi, get following.

Server URL         : http://localhost:8000/

Server Root        : /tmp/mod_wsgi-localhost:8000:501

Server Conf        : /tmp/mod_wsgi-localhost:8000:501/httpd.conf

Error Log File     : /tmp/mod_wsgi-localhost:8000:501/error_log (warn)

Request Capacity   : 5 (1 process * 5 threads)

Request Timeout    : 60 (seconds)

Startup Timeout    : 15 (seconds)

Queue Backlog      : 100 (connections)

Queue Timeout      : 45 (seconds)

Server Capacity    : 20 (event/worker), 20 (prefork)

Server Backlog     : 500 (connections)

Locale Setting     : en_US.UTF-8

However, when I run sudo apachectl configtest, get following which suggests something wrong. 

httpd: Syntax error on line 172 of /private/etc/apache2/httpd.conf: Cannot load /usr/libexec/apache2/mod_wsgi-py36.cpython-36m-darwin.so into server: dlopen(/usr/libexec/apache2/mod_wsgi-py36.cpython-36m-darwin.so, 10): Library not loaded: @rpath/libpython3.6m.dylib\n  Referenced from: /usr/libexec/apache2/mod_wsgi-py36.cpython-36m-darwin.so\n  Reason: image not found

How do I fix this ? I am using default apache2 coming with os, command for apachectl -V gives following:

Server version: Apache/2.4.16 (Unix)

Server built:   Jul 22 2015 21:03:09

Server's Module Magic Number: 20120211:47

Server loaded:  APR 1.4.8, APR-UTIL 1.5.2

Compiled using: APR 1.4.8, APR-UTIL 1.5.2

Architecture:   64-bit

Server MPM:     prefork

  threaded:     no

    forked:     yes (variable process count)

Server compiled with....

 -D APR_HAS_SENDFILE

 -D APR_HAS_MMAP

 -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)

 -D APR_USE_FLOCK_SERIALIZE

 -D APR_USE_PTHREAD_SERIALIZE

 -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT

 -D APR_HAS_OTHER_CHILD

 -D AP_HAVE_RELIABLE_PIPED_LOGS

 -D DYNAMIC_MODULE_LIMIT=256

 -D HTTPD_ROOT="/usr"

 -D SUEXEC_BIN="/usr/bin/suexec"

 -D DEFAULT_PIDLOG="/private/var/run/httpd.pid"

 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"

 -D DEFAULT_ERRORLOG="logs/error_log"

 -D AP_TYPES_CONFIG_FILE="/private/etc/apache2/mime.types"

 -D SERVER_CONFIG_FILE="/private/etc/apache2/httpd.conf"

 python -V

Python 3.6.1 :: Anaconda 4.4.0 (x86_64)

Please do help, am stuck.

[Graham Dumpleton]

On 4 Jun 2017, at 4:50 AM, Vishnu Prasad <vishnu...@gmail.com> wrote:

I download mod_wsgi and ran python setup.py install. As per your suggestion, am moving away from simple httpserver to Flask enabled one. 

I get following on running module-config

LoadFile "/Users/vph/anaconda/lib/libpython3.6m.dylib"

LoadModule wsgi_module "/Users/vph/anaconda/lib/python3.6/site-packages/mod_wsgi-4.5.16-py3.6-macosx-10.7-x86_64.egg/mod_wsgi/server/mod_wsgi-py36.cpython-36m-darwin.so"

WSGIPythonHome "/Users/vph/anaconda"

I got the following on running install-module

 sudo mod_wsgi-express install-module

LoadFile "/Users/vph/anaconda/lib/libpython3.6m.dylib"

LoadModule wsgi_module "/usr/libexec/apache2/mod_wsgi-py36.cpython-36m-darwin.so"

WSGIPythonHome "/Users/vph/anaconda"

You do not need to do both of these, you only do one.

In both cases you need to copy the output and put it in your Apache config file.

My apache2 httpd.conf is :

LoadModule wsgi_module "/usr/libexec/apache2/mod_wsgi-py36.cpython-36m-darwin.so"

In other words, this is incomplete, as you didn't copy the LoadFile and WSGIPythonHome directives in the Apache configuration.

Note that the order is important and must be done as shown.

If you are on MacOS X, I am not even sure how running mod_wsgi-express install-module even worked.

MacOS X System Integrity Protection (SIP), unless they changed it in Sierra, prohibits you from install extra Apache modules into the system Apache modules directory.

Thus on MacOS X, you must use the mod_wsgi-express module-config method as the install-module method would usually fail.

If I start mod_wsgi, get following.

Server URL         : http://localhost:8000/

Server Root        : /tmp/mod_wsgi-localhost:8000:501

Server Conf        : /tmp/mod_wsgi-localhost:8000:501/httpd.conf

Error Log File     : /tmp/mod_wsgi-localhost:8000:501/error_log (warn)

Request Capacity   : 5 (1 process * 5 threads)

Request Timeout    : 60 (seconds)

Startup Timeout    : 15 (seconds)

Queue Backlog      : 100 (connections)

Queue Timeout      : 45 (seconds)

Server Capacity    : 20 (event/worker), 20 (prefork)

Server Backlog     : 500 (connections)

Locale Setting     : en_US.UTF-8

Just remember that you don't need to worry about mod_wsgi-express start-server if you are going to manually configure the system Apache and use it alone.

You could still use mod_wsgi-express start-server for development, but not for permanent running of the site if setting up system Apache to do it.

-- 
You received this message because you are subscribed to the Google Groups "modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.
To post to this group, send email to mod...@googlegroups.com.
Visit this group at https://groups.google.com/group/modwsgi.
For more options, visit https://groups.google.com/d/optout.

[Vishnu Prasad]

Thanks Graham. Got my code ready @ https://github.com/vishnuprasadh/flaskdynamicimgprocessor. Will Perf test and update, was a very wise guidance from you.

[Graham Dumpleton]

On 5 Jun 2017, at 2:04 AM, Vishnu Prasad <vishnu...@gmail.com> wrote:

Thanks Graham. Got my code ready @ https://github.com/vishnuprasadh/flaskdynamicimgprocessor. Will Perf test and update, was a very wise guidance from you.

Depending on how the image library being used is implemented, using a high number of threads as you have in:

    WSGIDaemonProcess imgprocessor user=vph home=/var/www/imgprocessor threads=40

may be a bad idea.

The problem is that the global interpreter lock (GIL) in Python means that only one piece of Python code can run at a time.

If the image library being used isn't able to release the GIL while doing the image resize, and use C level operations only which don't need to lock the Python object, then you can't have multiple image resize operations running in parallel. Being a CPU bound activity also doesn't help your course.

For CPU bound activities, especially where the GIL cannot be released and the operation can take a while, you should favour use of multiple processes over threads as then you can get proper concurrency by virtue of the CPU intensive task running on separate processes.

Use of a high number of threads should only be done when you have a I/O bound application. Even then, running 40 threads rings alarm bells. Usually you still want to run a lot less than that if can.

Also, as far as testing performance, ensure you aren't testing by simply overloading your web server. That is completely the wrong way of going about it.

I would recommend you only supply as much traffic as would cause a single web application process to be within the 40-60% CPU range if using Python code and so limited by the GIL. If you are reaching more than that, you should be creating more processes.

Note that even if you put threads at 40 and do overload your application, it doesn't necessarily mean that all threads are being used. Most likely in a CPU intensive application, most threads sit there unused and so wasting a bit of memory. That is why need to monitor CPU usage of processes, but also the thread utilisation factor.

I did a talk about this at regional PyCon some time back. See:

* https://www.youtube.com/watch?v=SGleKfigMsk

Lastly, neither of these is needed in the configuration:

        WSGIScriptReloading On

        Options +ExecCGI

Script reloading is on by default. ExecCGI is only needed if using AddHandler, not when using WSGIScriptAlias.

And:

        Order allow,deny

        Allow from all

        Require all granted

is using both Apache 2.2 and 2.4 methods of granting access. I wouldn't show both. If need to, use:

<IfVersion < 2.4>

        Order allow,deny

        Allow from all

</IfVersion>

<IfVersion >= 2.4>

        Require all granted

</IfVersion>

Graham

[Vishnu Prasad]

Thanks Graham. I have put in place a plan to test with 1ProcessX3Threads,2ProcessX3Threads,4ProcessX3Threads and likewise for 6,10,20,40 threads. Again for me benchmark is the CPU usage with response time. Anything above 75% CPU, will try to mark it as a comfort zone....in addition planning to review how load average varies with the above.

Will share what i end up..

thanks

[Graham Dumpleton]

If want to get more serious with, have a read of:

http://blog.dscpl.com.au/2015/06/implementing-request-monitoring-within.html

There are APIs in mod_wsgi to extract metrics information.

A key one in that is the thread utilisation and number of activated threads. They can help to tune the settings properly.

The PyCon talk video was using that metrics API.

Graham