Adding python-path to WSGIDaemonProcess causes site.USER_SITE to fail with attribute error

[Steve Fielding]

The actual value of python-path does not seem to matter. I show it set to the tensorflow research directory because, it is my ultimate target, and that code base needs to access site.USER_SITE

config file:

<VirtualHost *:80>
        ServerName checkwsgi.com
        WSGIDaemonProcess checkwsgi.com home=/home/stevefielding_ca/github/REST-tutorial python-home=/home/stevefielding_ca/.virtualenvs/cv/bin/python python-path=/home/stevefielding_ca/github/models/research:/home/stevefielding_ca/github/models/research/slim
        WSGIProcessGroup checkwsgi.com
        WSGIScriptAlias / /home/stevefielding_ca/github/REST-tutorial/checkInstall.py
        <Directory /home/stevefielding_ca/github/REST-tutorial/>
                Require all granted
        </Directory>
</VirtualHost>

This is my script:

import sys
import site
def application(environ, start_response):
  status = '200 OK'  
  output = ''
  output += 'sys.version = {}\n'.format( repr(sys.version))
   output += 'sys.prefix = {}\n'.format(repr(sys.prefix))
   output += 'sys.path = {}\n'.format(repr(sys.path))
   output += 'site.USER_SITE = {}\n'.format(site.USER_SITE)
   response_headers = [('Content-type', 'text/plain'),                    ('Content-Length', str(len(output)))]  start_response(status, response_headers)
  print(output)
  output = bytes(output,'utf-8')
   return output

I get the following error when I try to access my script:

[Wed Aug 28 20:53:06.010105 2019] [wsgi:error] [pid 9148:tid 139859258083072] [remote 127.0.0.1:39307]     output += 'site.USER_SITE = {}\\n'.format(site.USER_SITE)
[Wed Aug 28 20:53:06.010121 2019] [wsgi:error] [pid 9148:tid 139859258083072] [remote 127.0.0.1:39307] AttributeError: module 'apache' has no attribute 'USER_SITE'

If I remove the python-path from my wsgi conf file, then everything works fine.

[Graham Dumpleton]

The per user site-packages directory under $HOME is not used if you are using a Python virtual environment. You should not be installing packages using 'pip install --user' if using a virtual environment, they should be installed into the virtual environment.

Further, under Apache the code doesn't even run as your user, but the Apache user, and wouldn't even be the same $HOME directory anyway.

If the tensorflow package is relying on the per user site packages directory always being used and the site.USER_SITE variable being present, it is arguably broken.

Why is tensorflow expecting site.USER_SITE to exist and what is it using the directory for?

--
You received this message because you are subscribed to the Google Groups "modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/c65ef285-5c13-4494-b594-2f3e82d60628%40googlegroups.com.

[Steve Fielding]

Thanks for the feedback Graham.

Confused as to why the 'USER_SITE' attribute is present if I do not try to set python-path. If I just remove the python-path setting from the config file, this is what site.USER_SITE is set to:

site.USER_SITE = /var/www/.local/lib/python3.5/site-packages

Tensorflow is checking the site directories in the __init__.py file, and the error occurs as soon as I import Tensorflow.

From the code shown below it seems that tensorflow is inspecting site.USER_SITE, and other site paths to determine if their locations are a prefix for the location of the __init__.py file. If any prefix matches, then this is used as an idication that tensorflow is running from a pip package, and then the appropriate modules are loaded.

# Get sitepackages directories for the python installation.
_site_packages_dirs = []
_site_packages_dirs += [_site.USER_SITE]
_site_packages_dirs += [_p for _p in _sys.path if 'site-packages' in _p]
if 'getsitepackages' in dir(_site):
  _site_packages_dirs += _site.getsitepackages()

if 'sysconfig' in dir(_distutils):
  _site_packages_dirs += [_distutils.sysconfig.get_python_lib()]

_site_packages_dirs = list(set(_site_packages_dirs))

# Find the location of this exact file.
_current_file_location = _inspect.getfile(_inspect.currentframe())

def _running_from_pip_package():
  return any(
      _current_file_location.startswith(dir_) for dir_ in _site_packages_dirs)

if _running_from_pip_package():
  for s in _site_packages_dirs:
    # TODO(gunan): Add sanity checks to loaded modules here.
    plugin_dir = _os.path.join(s, 'tensorflow-plugins')
    if _fi.file_exists(plugin_dir):
      _ll.load_library(plugin_dir)

On Wednesday, August 28, 2019 at 2:50:19 PM UTC-7, Graham Dumpleton wrote:

The per user site-packages directory under $HOME is not used if you are using a Python virtual environment. You should not be installing packages using 'pip install --user' if using a virtual environment, they should be installed into the virtual environment.

Further, under Apache the code doesn't even run as your user, but the Apache user, and wouldn't even be the same $HOME directory anyway.

If the tensorflow package is relying on the per user site packages directory always being used and the site.USER_SITE variable being present, it is arguably broken.

Why is tensorflow expecting site.USER_SITE to exist and what is it using the directory for?

To unsubscribe from this group and stop receiving emails from it, send an email to mod...@googlegroups.com.

[Graham Dumpleton]

In your script where you are printing out value of site.USER_SITE, what do you get if you add:

    import site

    print('NAME', site.__name__)

    print('FILE', site.__file__)

Graham

To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/0628ef26-c970-47b0-b726-0c6105e2d198%40googlegroups.com.

[Steve Fielding]

In the case where site.USER_SITE attribute is NOT present:

[Fri Aug 30 00:55:05.338223 2019] [wsgi:error] [pid 2447:tid 140424335841024] NAME mod_wsgi
[Fri Aug 30 00:55:05.338242 2019] [wsgi:error] [pid 2447:tid 140424335841024] FILE /home/stevefielding_ca/.virtualenvs/cv/lib/python3.5/site-packages/mod_wsgi/__init__.py

In the case where site.USER_SITE attribute is present:

[Fri Aug 30 00:59:12.435555 2019] [wsgi:error] [pid 2748:tid 140613069281024] NAME site
[Fri Aug 30 00:59:12.437237 2019] [wsgi:error] [pid 2748:tid 140613069281024] FILE /home/stevefielding_ca/.virtualenvs/cv/lib/python3.5/site.py

On Thursday, August 29, 2019 at 4:06:48 PM UTC-7, Graham Dumpleton wrote:

In your script where you are printing out value of site.USER_SITE, what do you get if you add:

    import site

    print('NAME', site.__name__)

    print('FILE', site.__file__)

Graham

To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/0628ef26-c970-47b0-b726-0c6105e2d198%40googlegroups.com.