mod_wsgi: ImportError: No module named 'django'

[Shanti Suresh]

Greetings,

Thank you for your help, in advance!  It has been a frustrating day.

I have a virtualenv configured with Python3.4.3 on RedHat release 7.2.

I configured mod_wsgi-3.4 as:
./configure --with-apxs=/usr/bin/apxs --with-python=/usr/local/dev/env/bin/python3
make
sudo cp .libs/mod_wsgi.so /etc/httpd/modules/

--
My wsgi.py file reads:
import os
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "knack_djangoapp.settings_temp")

from django.core.wsgi import get_wsgi_application
application = get_wsgi_application()
--
--
In httpd.conf, I have:

<VirtualHost *:81>
  DocumentRoot /var/www/html/dev
  ServerName dev.domain.edu
  WSGIDaemonProcess dev python-path=/usr/local//dev/djangoapp:/usr/local/dev/env/lib/python3.4/site-packages/ lang='en_US.UTF-8' locale='en_US.UTF-8'
  WSGIProcessGroup dev
  WSGIScriptAlias / /usr/local/dev/djangoapp/wsgi.py process-group=dev
  <Directory /usr/local/dev/djangoapp/>
      <Files wsgi.py>
          Require all granted
      </Files>
  </Directory>
 </VirtualHost>
--

--
In my vrtualenv:
pip freeze gives:
Django==1.8.15
mod-wsgi==4.5.7
--

Upon starting httpd as /sbin/service httpd start, I get:
...Adding '/usr/local/dev/env/lib/python3.4/site-packages/' to path
...Target WSGI script '/usr/local/dev/djangoapp/wsgi.py' cannot be loaded as Python module.
...Exception occurred processing WSGI script '/usr/local/dev/djangoapp/wsgi.py'
...from django.core.wsgi import get_wsgi_application
...ImportError: No module named 'django'

I greatly appreciate any and all help.  I am at my wits' end.  mod_wsgi seems to be using the system python interpreter.  It is not using the virtualenv appropriately.

Thank you so much!

                 -Shanti

[Graham Dumpleton]

On 20 Oct 2016, at 10:56 AM, Shanti Suresh <shantis...@gmail.com> wrote:

Greetings,

Thank you for your help, in advance!  It has been a frustrating day.

I have a virtualenv configured with Python3.4.3 on RedHat release 7.2.

I configured mod_wsgi-3.4 as:
./configure --with-apxs=/usr/bin/apxs --with-python=/usr/local/dev/env/bin/python3

Are you using your own Python installation or that from the Software Collections Library? I am presuming it is your own rather than SCL as SCL latest is Python 3.4.2. Just want to make sure.

Anyway, even if you intend to use a virtual environment, it is often better to still compile mod_wsgi against the main Python installation. This will avoid complications if later want to have multiple daemon process groups using different Python virtual environments.

So would recommend doing a:

  make distclean

then rerun configure but this time use the python3 from where it was originally installed.

make
sudo cp .libs/mod_wsgi.so /etc/httpd/modules/

--
My wsgi.py file reads:
import os
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "knack_djangoapp.settings_temp")

from django.core.wsgi import get_wsgi_application
application = get_wsgi_application()
--
--
In httpd.conf, I have:

Because best practice is to always use daemon mode, I would recommend adding outside of VirtualHost, usually just after where the wsgi_module is loaded:

    WSGIRestrictEmbedded On

This will prevent Python being initialised in the Apache child worker processes and will only be initialised in daemon processes.

If your configuration isn’t quite right and for some reason it tries to run stuff in embedded mode, you will now get an error saying something is wrong. Thus helps to avoid mistakes.

<VirtualHost *:81>
  DocumentRoot /var/www/html/dev
  ServerName dev.domain.edu

  WSGIDaemonProcess dev python-path=/usr/local//dev/djangoapp:/usr/local/dev/env/lib/python3.4/site-packages/ lang='en_US.UTF-8' locale='en_US.UTF-8’

Recommend using:

    WSGIDaemonProcess dev python-home=/usr/local/dev/env python-path=/usr/local//dev/djangoapp lang='en_US.UTF-8' locale='en_US.UTF-8’

    WSGIApplicationGroup %{GLOBAL}

The important bit here is using python-home to set the location of the virtual environment. This will be what sys.prefix is set to for Python when run under the virtual environment. Don’t set up the virtual environment using site-packages in python-path.

The WSGIApplicationGroup is also good practice if only have one application per daemon process group, as is recommended, as avoids problems with some third party C extensions for Python that will not work in Python sub interpreters.

  WSGIProcessGroup dev

This is actually redundant as as process-group option on WSGIScriptAlias.

  WSGIScriptAlias / /usr/local/dev/djangoapp/wsgi.py process-group=dev
  <Directory /usr/local/dev/djangoapp/>
      <Files wsgi.py>
          Require all granted
      </Files>
  </Directory>
 </VirtualHost>
--

--
In my vrtualenv:
pip freeze gives:
Django==1.8.15
mod-wsgi==4.5.7
--

Upon starting httpd as /sbin/service httpd start, I get:
...Adding '/usr/local/dev/env/lib/python3.4/site-packages/' to path
...Target WSGI script '/usr/local/dev/djangoapp/wsgi.py' cannot be loaded as Python module.
...Exception occurred processing WSGI script '/usr/local/dev/djangoapp/wsgi.py'
...from django.core.wsgi import get_wsgi_application
...ImportError: No module named 'django'

I greatly appreciate any and all help.  I am at my wits' end.  mod_wsgi seems to be using the system python interpreter.  It is not using the virtualenv appropriately.

So those are things I would recommend, but how you had it should still have worked.

What I would suggest is temporarily replace the wsgi.py file with test script in:

    http://modwsgi.readthedocs.io/en/develop/user-guides/checking-your-installation.html#python-installation-in-use

This will log some details about the Python virtual environment being used. You can then verify is the virtual environment you expect it to be.

Other things to check are that the user that Apache run as has access to everything in the virtual environment. That would usually mean access to ‘others’ for files and directories.

The only other thing can think of is that SELinux is causing problems, but doubt that as it wouldn’t be able to access the wsgi.py even if that was the case.

So try that test WSGI script to see if correct virtual environment. Also in Python from command line do:

    import django

    print(django.__file__)

and see if where Django actually is, corresponds to where you expect it to and where mod_wsgi is looking. Modify the test script to print out the value of sys.path to see if the directory is in the module search path if necessary.

Graham

[Shanti Suresh]

Hi Graham,

Thanks so much for your response.

With those changes, I get:

Fatal Python error: Py_Initialize: Unable to get the locale encoding
ImportError: No module named 'encodings'

I also removed "lang='en_US.UTF-8' locale='en_US.UTF-8’" just for fun, but I still get the error about encodings.
I don't have an "encodings" module in my Django project.

Thanks again,

               -Shanti

[Graham Dumpleton]

Do you have multiple Python 3 versions installed on your system?

It may be picking up wrong shared library at run time:

    http://modwsgi.readthedocs.io/en/develop/user-guides/checking-your-installation.html#python-shared-library

The solution to that if it is the case, is to work out where the shared library is for the main Python installation of version you want to use is, and set the LD_RUN_PATH environment at the time of compilation to where that directory is. For example, if Python 3 you want to use is under /usr/local, use:

    make distclean

    ./configure --with-apxs=/usr/bin/apxs --with-python=/usr/local/bin/python3

    LD_RUN_PATH=/usr/local/lib make

    sudo make install

BTW, are you using Anaconda Python?

Did you use virtualenv or pyvenv to create the Python virtual environment.

The pyvenv approach is broken for when embedding Python. I had a workaround for it, but is possible some recent changes I made so that Anaconda Python would work has upset that. Or the workaround will not work for Anaconda Python anyway.

Graham

-- 
You received this message because you are subscribed to the Google Groups "modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.
To post to this group, send email to mod...@googlegroups.com.
Visit this group at https://groups.google.com/group/modwsgi.
For more options, visit https://groups.google.com/d/optout.

[Shanti Suresh]

Hi Graham,

Thank you.

I tried with both pyenv as well as virtualenv and both times, it is the "encodings" module.

Thank God, it is not 'django' anymore.  I will try recompiling.  The Python binaries include:
/usr/bin/python2.7
/usr/bin/python2.7-config
/usr/bin/python3.4
/usr/bin/python3.4m
/usr/bin/python3.4m-x86_64-config

                   -Shanti

[Graham Dumpleton]

If your Python 3 is the system Python, then LD_RUN_PATH shouldn’t be needed.

I presume you mean pyvenv and not pyenv as pyenv is a tool for installing whole Python installations, not just virtual environment.

The preference is to use virtualenv over pyvenv as know virtualenv always works.

Graham

[Graham Dumpleton]

The only other can think of is that the main Python installation was patched/upgraded between when virtual environment was created and when it was used. This can cause some strange problems to occur and necessitates recreation of the virtual environment.

Graham

[Shanti Suresh]

Hi Graham,

As much distracting as the debate is, the build of mod_wsgi actually isn't without event.  I have to go in and modify the Makefile.
I change -lpython3.4 to -lpython3

And then the make succeeds.  Otherwise, "make" fails as follows:

$ make | tee make5.log
...
/usr/lib64/apr-1/build/libtool --silent --mode=link gcc -std=gnu99 -Wl,-z,relro,-z,now   -o mod_wsgi.la  -rpath /usr/lib64/httpd/modules -module -avoid-version    mod_wsgi.lo -L/usr/lib64 -L/usr/lib64/python3.4/config -lpython3.4 -lpthread -ldl -lutil -lm
/usr/bin/ld: cannot find -lpython3.4
collect2: error: ld returned 1 exit status
apxs:Error: Command failed with rc=65536

Could that be the problem?

Thanks,

              -Shanrti

[Graham Dumpleton]

What do you get for:

    ls -las /usr/lib64/libpython*

There should be a libpython3.4. If there isn’t there is something a bit wonky.

Is the Python 3.4 definitely a system package? I actually didn’t think RHEL 7.2 had Python 3.

As far as doing software development on RHEL, one generally should avoid using the system Python and instead use Python installed from Software Collections Library. These are not installed under /usr, but down under /opt somewhere.

Graham

[Shanti Suresh]

Hi Graham,

I see.  I appreciate that feedback. 

I did:
sudo yum install python34 python34-devel

$ sudo yum install python34 python34-devel
Loaded plugins: langpacks, product-id, rhnplugin, search-disabled-repos, subscription-manager
This system is receiving updates from RHN Classic or Red Hat Satellite.
Package python34-3.4.3-7.el7.x86_64 already installed and latest version
Package python34-devel-3.4.3-7.el7.x86_64 already installed and latest version
Nothing to do

I don't mind downgrading to Python 2.7.

Also,
$  ls -las /usr/lib64/libpython*
   0 lrwxrwxrwx. 1 root root      19 Feb 20  2016 /usr/lib64/libpython2.7.so -> libpython2.7.so.1.0
1780 -rwxr-xr-x. 1 root root 1822520 Oct 11  2015 /usr/lib64/libpython2.7.so.1.0
   0 lrwxrwxrwx. 1 root root      20 Oct 19 12:45 /usr/lib64/libpython3.4m.so -> libpython3.4m.so.1.0
2556 -rwxr-xr-x. 1 root root 2615808 Aug  9 13:13 /usr/lib64/libpython3.4m.so.1.0
   8 -rwxr-xr-x. 1 root root    6688 Aug  9 13:13 /usr/lib64/libpython3.so

Thanks,

                  -Shanti

[Graham Dumpleton]

So the Makefile is meant to use -lpython3.4m.

There was a time when the ‘m’ was being left off, but that should have been fixed a long time ago.

I will need to dig into what RHEL is doing and why the configuration that Python generates for me doesn’t have the ‘m’.

So using -lpython3 or -lpython3.4m will work.

Anyway, with it building okay, where are we up to now.

Still have encoding issue?

Graham

[Shanti Suresh]

Hi Graham,

Yes, still having the Importerror on "encoding".  But the good news is that, Python2.7 and RHEL-stock mod_wsgi works well.

So thank you for that!  I am glad that we determined the RHEL still has some work left to be done with Python3.

In case you are able to find out what went wrong with Python3 on RHEL, please let me know.  It is nor urgent.  I am all set.

I can't thank you enough, Graham  Hope you have a wonderful evening.

                   -Shanti

[Graham Dumpleton]

Okay. Glad you got something working. I will have to keep in mind trying to check RHEL in some way. Is a bit of pain for me to get RHEL images and CentOS is probably different enough that wouldn’t see issue.

Graham

[Shanti Suresh]

Hi Graham,

I just wanted to add that the Django module, "mod_wsgi", works really well and is a breeze to get working.  I used Django 1.8 + Python3.4.

The mod_wsgi Django module uses the correct python executable from the virtual environment and recognizes STATIC_ROOT etc. to generate a fully functional httpd.conf.  I was pleasantly surprised by how simple it is to get a working application up and running by using "apachectl start".

Just one thing on my whimsical wishlist is if the module would also generate a file of environment variables that one may then use in setting up the HTTPD process as a system service.

Thanks,

               -Shanti

[Graham Dumpleton]

On 22 Oct 2016, at 1:46 AM, Shanti Suresh <shantis...@gmail.com> wrote:

Hi Graham,

I just wanted to add that the Django module, "mod_wsgi", works really well and is a breeze to get working.  I used Django 1.8 + Python3.4.

The mod_wsgi Django module uses the correct python executable from the virtual environment and recognizes STATIC_ROOT etc. to generate a fully functional httpd.conf.  I was pleasantly surprised by how simple it is to get a working application up and running by using "apachectl start".

Just one thing on my whimsical wishlist is if the module would also generate a file of environment variables that one may then use in setting up the HTTPD process as a system service.

Can you elaborate more on what you mean?

Generating stuff automatically in some way to help with integration into a system startup mechanism is problematic for a number of reasons. The main one is that there are so many different startup mechanisms in use across Linux systems and MacOS X so what do you target. Environment variables for configuration of WSGI application is also a bit of a problem with Apache because of Apache being something that can handle multiple deployed applications in the one instance. You can’t necessarily set environment variables for the whole of Apache as it could affect more than just your one application.

As far as hosting of WSGI applications and relying on environment variables for configuration in general as people interpret as what 12factor manifesto says, that is wrong to a degree. What 12factor says is that configuration should come from the environment. That is, it is not part of your application code or the bundle you are deploying. You should use the abilities of the hosting environment to provide the configuration. Using environment variables is just one way of doing that. Another maybe that the environment provides a way of you inserting configuration files into the running application environment for the application to find.

Anyway, there are lots of considerations when it comes to trying to configure deployments and get a WSGI application running, so will help for you to clarify what specifically you are after or wanting to do.

Graham