sys.path on windows

70 views
Skip to first unread message

Banesiu.Sever

unread,
Jul 29, 2007, 3:03:59 AM7/29/07
to modwsgi
I can't import global modules in my entry file (the one where
application function is defined) because it looks like sys.path is set
to ['C:\\Program Files\\Bazaar\\python25.zip', '.\\DLLs', '.\\lib', '.\
\lib\\plat-win', '.\\lib\\lib-tk', 'C:\\Program Files\\Apache Software
Foundation\\Apache2.2\\bin']. Does anyone knows why?

Graham Dumpleton

unread,
Jul 29, 2007, 3:26:49 AM7/29/07
to mod...@googlegroups.com

Did you install Python as an Administrator or as yourself?

If Apache is run as a service it runs as a special user and if you
installed Python as yourself and not an administrator then the Windows
registry may be missing keys in the machine wide registry which Python
uses to where out where it is installed

This has been an issue for mod_python as well:

http://www.modpython.org/pipermail/mod_python/2006-September/021979.html

See if the Windows registry has those keys in it and perhaps reinstall
Python as an administrator if they don't exist or look incomplete.

Post what the registry keys are that you do find.

Graham

Banesiu.Sever

unread,
Jul 29, 2007, 3:47:40 AM7/29/07
to modwsgi
On Jul 29, 10:26 am, "Graham Dumpleton" <graham.dumple...@gmail.com>
wrote:

This is the Windows registry tree I have for Python:

[HKEY_CURRENT_USER\Software\Python]
[HKEY_CURRENT_USER\Software\Python\PythonCore]
[HKEY_CURRENT_USER\Software\Python\PythonCore\2.5]
[HKEY_CURRENT_USER\Software\Python\PythonCore\2.5\Help]
[HKEY_CURRENT_USER\Software\Python\PythonCore\2.5\Help\Main Python
Documentation]
@="C:\\users\\sever\\Python25\\Doc\\Python25.chm"
[HKEY_CURRENT_USER\Software\Python\PythonCore\2.5\InstallPath]
@="C:\\users\\sever\\Python25\\"
[HKEY_CURRENT_USER\Software\Python\PythonCore\2.5\InstallPath
\InstallGroup]
@="Python 2.5"
[HKEY_CURRENT_USER\Software\Python\PythonCore\2.5\Modules]
[HKEY_CURRENT_USER\Software\Python\PythonCore\2.5\PythonPath]
@="C:\\users\\sever\\Python25\\Lib;C:\\users\\sever\\Python25\\DLLs;C:\
\users\\sever\\Python25\\Lib\\lib-tk"

Note, Apache is not installed as a service.

Graham Dumpleton

unread,
Jul 29, 2007, 3:53:50 AM7/29/07
to mod...@googlegroups.com
On 29/07/07, Banesiu.Sever <Banesi...@gmail.com> wrote:
>

Which Python distribution are you using?

The HKEY_CURRENT_USER\Software\Python\PythonCore\2.5\PythonPath key
doesn't list site-packages directory which is another issue seen with
mod_python.

I suspect that standard Python installer does the correct thing, but
that some third party installers do not and miss out site-packages in
the default PythonPath defined in the registry.

See comments in quoted section at end of:

http://www.modpython.org/pipermail/mod_python/2007-March/023333.html

about site-packages being missing in path.

Instead of looking for where mod_python was installed, look for where
the global modules you can't find are installed and verify they are in
site-packages. Then add that directory.

Graham

Adal Chiriliuc

unread,
Jul 29, 2007, 4:20:00 AM7/29/07
to modwsgi
It looks like you have two Pythons installed - the official one and
another one used by Bazaar.

Maybe in your PATH Bazaar (C:\Program Files\Bazaar) is listed before
the official Python folder
(C:\users\sever\Python25). Try changing the order or add the last one
to PATH if it's missing
(Control Panel - System - Advanced - Environment Variables - PATH).

To test this, open a command line console, go to C:\ (cd \), start
python.exe and try to import your module.

Note: [HKEY_CURRENT_USER\Software\Python\PythonCore\2.5\PythonPath]
doesn't have to
contain site-packages, this folder is actually added to sys.path by
site.py.

Banesiu.Sever

unread,
Jul 29, 2007, 4:22:16 AM7/29/07
to modwsgi
On Jul 29, 10:53 am, "Graham Dumpleton" <graham.dumple...@gmail.com>
wrote:

> Which Python distribution are you using?

I'm using python 2.5.1, Apache 2.2.4, Windows Vista

> The HKEY_CURRENT_USER\Software\Python\PythonCore\2.5\PythonPath key
> doesn't list site-packages directory which is another issue seen with
> mod_python.

Well, even if site-packages directory is not in PythonPath this
directory can be found in sys.path when using python's interactive
interpreter.

> I suspect that standard Python installer does the correct thing, but
> that some third party installers do not and miss out site-packages in
> the default PythonPath defined in the registry.

I used "standard" Windows .msi installer downloaded form python.org to
install python.

> See comments in quoted section at end of:
>
> http://www.modpython.org/pipermail/mod_python/2007-March/023333.html
>
> about site-packages being missing in path.
>
> Instead of looking for where mod_python was installed, look for where
> the global modules you can't find are installed and verify they are in
> site-packages. Then add that directory.

I'm using decimal module. It is located in C:\users\sever\Python25\Lib
and this directory is already listed in PythonPath.

Sever

Graham Dumpleton

unread,
Jul 29, 2007, 4:24:57 AM7/29/07
to mod...@googlegroups.com
Good, Adal to the rescue.

I try and help with this Windows stuff even though not a Windows
person, but I can only echo suggestions made by others in the past
with mod_python. Thus have no idea if what I am saying is correct. :-(

Graham

Adal Chiriliuc

unread,
Jul 29, 2007, 4:40:09 AM7/29/07
to modwsgi
Well, you provide better Windows help that what Linux help I could
provide ;)

One question: any particular reason why WSGIPythonExecutable,
WSGIPythonHome and WSGIPythonPath are disabled on Windows?
Py_SetProgramName and Py_SetPythonHome don't have any special Windows
mention from what I see (and in fact I think I used them when I
embedded Python). They would help a lot. For example I could test all
Apache/Python combinations without having to temporary rename folders
or modify the registry. And you could use mod_wsgi without caring what
the registry contains (you would just need the a folder containing the
Python files).

On 29 Jul, 11:24, "Graham Dumpleton" <graham.dumple...@gmail.com>
wrote:


> Good, Adal to the rescue.
>
> I try and help with this Windows stuff even though not a Windows
> person, but I can only echo suggestions made by others in the past
> with mod_python. Thus have no idea if what I am saying is correct. :-(
>
> Graham
>

Banesiu.Sever

unread,
Jul 29, 2007, 4:49:34 AM7/29/07
to modwsgi
On Jul 29, 11:20 am, Adal Chiriliuc <adal.chiril...@gmail.com> wrote:
> It looks like you have two Pythons installed - the official one and
> another one used by Bazaar.
>
> Maybe in your PATH Bazaar (C:\Program Files\Bazaar) is listed before
> the official Python folder
> (C:\users\sever\Python25). Try changing the order or add the last one
> to PATH if it's missing
> (Control Panel - System - Advanced - Environment Variables - PATH).
>
> To test this, open a command line console, go to C:\ (cd \), start
> python.exe and try to import your module.
>
> Note: [HKEY_CURRENT_USER\Software\Python\PythonCore\2.5\PythonPath]
> doesn't have to
> contain site-packages, this folder is actually added to sys.path by
> site.py.

Thanks Adal! Thanks Graham!
Indeed, changing the order of directories in Path did the job (Bazzar
was before Python).

I think there is another issue. In sys.path the current directory (the
one where my entry file is located) is not listed. This means I can't
import relative to current directory / entry file. Am I missing
something here?

Sever

Adal Chiriliuc

unread,
Jul 29, 2007, 4:58:39 AM7/29/07
to modwsgi
You have two options:

import os
import sys

sys.path.append(os.path.dirname(__file__))

This should be put BEFORE any other import directives or operations.

OR you can do this:

create a myproject.pth file in the C:\users\sever\Python25 folder
which contains the folder with your files. This file should just be a
text file containing the path. No " or ' or \\ required.

Banesiu.Sever

unread,
Jul 29, 2007, 5:11:54 AM7/29/07
to modwsgi
On Jul 29, 11:58 am, Adal Chiriliuc <adal.chiril...@gmail.com> wrote:
> You have two options:
>
> import os
> import sys
>
> sys.path.append(os.path.dirname(__file__))
>
> This should be put BEFORE any other import directives or operations.
>
> OR you can do this:
>
> create a myproject.pth file in the C:\users\sever\Python25 folder
> which contains the folder with your files. This file should just be a
> text file containing the path. No " or ' or \\ required.

Why can't this be done automatically? It doesn't look pythonic, it's
not something you usually expect to and is not documented. :-)

Graham Dumpleton

unread,
Jul 29, 2007, 5:18:06 AM7/29/07
to mod...@googlegroups.com
I can't post the documentation I am working on which covers this as it
is on my other computer, but you shouldn't add the directory the
script file is in to sys.path, or at least it is not advisable. This
is because the script files aren't treated exactly like normal Python
modules and if you do an import of a script file which has a .py
extension, it will load as a second independent module to the magic
script file, resulting in two copies in memory.

Two options here. The first is to always use a .wsgi extension on
script file so no danger of it being imported as a normal Python
module. Then you can safely add the directory using:

import os
import sys

here = os.path.dirname(__file__)
if not here in sys.path:
sys.path.append(here)

The check to see if the path is in sys.path is to avoid it being added
multiple times if relying on script reloading, or if running multiple
script files under one interpreter (application group).

If you have common data or code which multiple scripts files need, put
it in a proper Python module with .py extension in the same directory.

Second option is to add to sys.path some totally different directory
outside of the document tree exposed to Apache and have all your
common Python modules there instead. This is somewhat safer as if you
stuff up your Apache configuration people will not be able to
inadvertently download source code for those modules.

Basically, the best ideas is to put as little as possible in the
script file, ie., just enough to import real Python modules from
elsewhere and setup the application entry point. Setting sys.path
and/or os.environ might be needed in some cases.

Graham

Adal Chiriliuc

unread,
Jul 29, 2007, 5:22:02 AM7/29/07
to modwsgi
The admin scripts of most frameworks (Django, Pylons, ...) do this
automatically behind your back. When using mod_wsgi to serve them this
is not done anymore, so somebody else needs to :-)

Graham Dumpleton

unread,
Jul 29, 2007, 5:26:46 AM7/29/07
to mod...@googlegroups.com
On 29/07/07, Banesiu.Sever <Banesi...@gmail.com> wrote:
>

Because the script files aren't treated exactly like real Python
modules. What happens is that the script files are imported to create
a Python module, but the name given to the module is a magic string
(md5 hash) of the absolute path of the script file. This is done as
opposed to the name of the module just being the basename of the file,
to allow multiple script files in different directories being loaded
in the same interpreter to use the same name. If the modules weren't
somehow named differently, every script file would have to have a
unique name across all directories.

This problem with Python module names needing to being unique has
caused huge problems with mod_python other the years and there the
module importer for mod_python had to be completely rewritten to
similarly name modules based on their absolute path rather than name.

Another thing which has caused a fair number of problems with
mod_python is how mod_python allowed sys.path to be modified in the
Apache configuration using PythonPath directive. This caused
randomness of path order and in conjunction with unique naming meant
the wrong module could end up being loaded. Thus, avoided using a
directive in mod_wsgi to allow sys.path to be modified.

Without knowing the full history of all the problems these sorts of
things caused in mod_python it might be hard to understand the reasons
for why mod_wsgi works how it does, but so you have a rough idea of
how big a problem module importing was for mod_python you might read:

http://www.dscpl.com.au/wiki/ModPython/Articles/ModuleImportingIsBroken

You will see some of these issues mentioned in there.

Graham

Graham Dumpleton

unread,
Jul 29, 2007, 7:04:43 PM7/29/07
to mod...@googlegroups.com
On 29/07/07, Adal Chiriliuc <adal.ch...@gmail.com> wrote:
>
> Well, you provide better Windows help that what Linux help I could
> provide ;)
>
> One question: any particular reason why WSGIPythonExecutable,
> WSGIPythonHome and WSGIPythonPath are disabled on Windows?
> Py_SetProgramName and Py_SetPythonHome don't have any special Windows
> mention from what I see (and in fact I think I used them when I
> embedded Python). They would help a lot. For example I could test all
> Apache/Python combinations without having to temporary rename folders
> or modify the registry. And you could use mod_wsgi without caring what
> the registry contains (you would just need the a folder containing the
> Python files).

I was actually going to ask you whether this stuff would work on
Windows. I didn't enable it as I wasn't sure which of them would work.
I thought perhaps Windows relied more on Windows registry settings
than environment variables and the like. Also wasn't sure whether
Python on Windows did the trick of searching PATH for Python
executable and whether it would even help to override executable
location if wrong DLL was picked up because of PATH order.

If you can play with those sections of code and tell me which ones
work and whether they might need to be changed somehow then that would
be great. Relevant bits of code are:

#ifndef WIN32
if (wsgi_server_config->python_path) {
putenv(apr_psprintf(p, "PYTHONPATH=%s",
wsgi_server_config->python_path));
}
#endif

#ifndef WIN32
AP_INIT_TAKE1("WSGIPythonExecutable", wsgi_set_python_executable, NULL,
RSRC_CONF, "Python executable absolute path name."),
AP_INIT_TAKE1("WSGIPythonHome", wsgi_set_python_home, NULL,
RSRC_CONF, "Python prefix/exec_prefix absolute path names."),
AP_INIT_TAKE1("WSGIPythonPath", wsgi_set_python_path, NULL,
RSRC_CONF, "Python module search path."),
#endif

Not sure WSGIPythonPath will work as not sure putenv() will set
variable where Python will find it on Windows. With WSGIPythonHome,
what separator is used if listing both directories, on UNIX it would
be ':' which would conflict with drive specifier. Or is that a non
issue as only need to list one on Windows. Finally, will
WSGIPythonExecutable matter if wrong DLL is picked up.

As you can see, just wasn't sure whether they would work or not.

Thanks in advance.

Graham

Adal Chiriliuc

unread,
Aug 2, 2007, 2:24:28 PM8/2/07
to modwsgi
I was able to play with these settings today. Here are my findings:

WSGIPythonExecutable can't work on Windows because the value set by
Py_SetProgramName is ignored on this platform. The python executable
is always derived from the module containing the running process.

WSGIPythonHome and WSGIPythonPath work.

The separator if multiple paths are present is ";" (the usual Windows
path separator for environment variables like PATH, LIB, ...).

Both / and \ can be used to separate path components. sys.path entries
will always use only \, regardless of the original separator.

Contents of sys.path when the following two directives was used, with
the Python DLL picked up from C:\Soft\Python:

WSGIPythonHome "C:\Soft\PythonVC80"
WSGIPythonPath "C:\Projects\Anim\Python;C:\Projects\Anim\Website
\AnimWeb;C:\Projects\Anim\Vendor\PythonLib"

C:\Projects\Anim\Python
C:\Projects\Anim\Website\AnimWeb
C:\Projects\Anim\Vendor\PythonLib
C:\Soft\Python\python25.zip <<< Python DLL lives in this dir
C:\Soft\PythonVC80\DLLs
C:\Soft\PythonVC80\lib
C:\Soft\PythonVC80\lib\plat-win
C:\Soft\PythonVC80\lib\lib-tk
C:\Projects\Anim\Vendor\Apache\bin <<< current dir
C:\Soft\PythonVC80
C:\Soft\PythonVC80\lib\site-packages
C:\Soft\PythonVC80\lib\site-packages\win32 <<< from .pth file
C:\Soft\PythonVC80\lib\site-packages\win32\lib <<< from .pth file
C:\Soft\PythonVC80\lib\site-packages\Pythonwin <<< from .pth file
C:\Soft\PythonVC80\lib\site-packages\wx-2.8-msw-unicode <<< from .pth
file

Note that python25.zip is inserted in sys.path before the paths
computed from WSGIPythonHome. This could cause problems if that file
actually exists. WSGIPythonPath paths are inserted before all others.

WSGIPythonHome indeed supports two paths. Both of them were used to
generate sys.path (I didn't study exactly how).

You can't user these two directives to "activate" a folder where you
have installed eggs (at least I don't know how). This is because the
egg paths are setup by easy-install.pth, but this file is not parsed
if it's not located in site-packages. To parse it you need to execute
a line like this:
import site; site.addsitedir(r"C:\Projects\Anim\Vendor\PythonLib").
But you can add that line in the .wsgi file. So using this trick and
WSGIPythonPath I was able to serve a Pylons application without having
any sort of reference to it (.pth file or easy_install stuff) in the
Python installation folder.

This means that on Windows it's possible to serve different WSGI
applications each one with different eggs installed without disturbing
the main Python installation and without requiring something like
working_env.py (I'm not saying that not possible on Linux). But this
will only work for mod_wsgi, serving the application using Paste or
updating/installing new eggs won't work directly like this.

I've also tested this situation: I've removed python25.dll from PATH
and copied it next to httpd.exe, then started Apache with
WSGIPythonHome pointing to a Python installation folder and it worked.

A small note: WSGIPythonOptimize doesn't actually work. In Python 2.5,
pyc and pyo files are identical (at least on Windows). In Python 2.4
they were different. Of course, this is not mod_wsgi fault ;)

On 30 Jul, 02:04, "Graham Dumpleton" <graham.dumple...@gmail.com>
wrote:

Graham Dumpleton

unread,
Aug 7, 2007, 6:48:11 AM8/7/07
to mod...@googlegroups.com
On 03/08/07, Adal Chiriliuc <adal.ch...@gmail.com> wrote:
>
> I was able to play with these settings today. Here are my findings:
>
> WSGIPythonExecutable can't work on Windows because the value set by
> Py_SetProgramName is ignored on this platform. The python executable
> is always derived from the module containing the running process.
>
> WSGIPythonHome and WSGIPythonPath work.

I enabled these latter two directives in subversion a little while
back, but been busy so had forgotten to mention it.

> A small note: WSGIPythonOptimize doesn't actually work. In Python 2.5,
> pyc and pyo files are identical (at least on Windows). In Python 2.4
> they were different. Of course, this is not mod_wsgi fault ;)

Do you know if setting:

WSGIPythonOptimize 2

does anything different on Windows?

This is meant to result in doc strings being removed when Python code
which needs to be compiled.

Graham

Adal Chiriliuc

unread,
Aug 7, 2007, 8:01:14 AM8/7/07
to modwsgi
I've tested WSGIPythonOptimize 2. The doc strings are still present in
the .pyo files.

I've done some testing using "python.exe -O" and "python.exe -OO"
using a simple test module which contains a couple of doc-strings and
an assert. The two options (which should correspond to python optimize
1 and 2) generate the same .pyo files (same size, same md5 hash). The
doc-strings are still present, the assert is removed (the .pyo files
are smaller than the .pyc files and the assert does not trigger).
Could you try a test like this on your system?

On Aug 7, 1:48 pm, "Graham Dumpleton" <graham.dumple...@gmail.com>
wrote:

Graham Dumpleton

unread,
Aug 10, 2007, 7:25:13 AM8/10/07
to mod...@googlegroups.com
On 07/08/07, Adal Chiriliuc <adal.ch...@gmail.com> wrote:
>
> I've tested WSGIPythonOptimize 2. The doc strings are still present in
> the .pyo files.
>
> I've done some testing using "python.exe -O" and "python.exe -OO"
> using a simple test module which contains a couple of doc-strings and
> an assert. The two options (which should correspond to python optimize
> 1 and 2) generate the same .pyo files (same size, same md5 hash). The
> doc-strings are still present, the assert is removed (the .pyo files
> are smaller than the .pyc files and the assert does not trigger).
> Could you try a test like this on your system?

On MacOS X with file:

"""Module doc string"""
print __file__

I get for 'python':

grumpy:~/tmp grahamd$ python
Python 2.3.5 (#1, Mar 20 2005, 20:38:20)
[GCC 3.3 20030304 (Apple Computer, Inc. build 1809)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import opt
opt.py
>>> help(opt)

Help on module opt:

NAME
opt - Module doc string

FILE
/Users/grahamd/tmp/opt.py

$ ls -las
total 16
0 drwxr-xr-x 4 grahamd grahamd 136 Aug 10 21:13 .
0 drwxr-xr-x 63 grahamd grahamd 2142 Aug 10 21:13 ..
8 -rw-r--r-- 1 grahamd grahamd 40 Aug 10 21:13 opt.py
8 -rw-r--r-- 1 grahamd grahamd 146 Aug 10 21:13 opt.pyc

For 'python -O':

grumpy:~/tmp grahamd$ python
Python 2.3.5 (#1, Mar 20 2005, 20:38:20)
[GCC 3.3 20030304 (Apple Computer, Inc. build 1809)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import opt
opt.py
>>> help(opt)

Help on module opt:

NAME
opt - Module doc string

FILE
/Users/grahamd/tmp/opt.py

$ ls -las
total 16
0 drwxr-xr-x 4 grahamd grahamd 136 Aug 10 21:13 .
0 drwxr-xr-x 63 grahamd grahamd 2142 Aug 10 21:13 ..
8 -rw-r--r-- 1 grahamd grahamd 40 Aug 10 21:13 opt.py
8 -rw-r--r-- 1 grahamd grahamd 146 Aug 10 21:13 opt.pyo

For 'python -OO':

Python 2.3.5 (#1, Mar 20 2005, 20:38:20)
[GCC 3.3 20030304 (Apple Computer, Inc. build 1809)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import opt
opt.py
>>> help(opt)

Help on module opt:

NAME
opt

FILE
/Users/grahamd/tmp/opt.py

$ ls -las
total 16
0 drwxr-xr-x 4 grahamd grahamd 136 Aug 10 21:14 .
0 drwxr-xr-x 63 grahamd grahamd 2142 Aug 10 21:13 ..
8 -rw-r--r-- 1 grahamd grahamd 40 Aug 10 21:13 opt.py
8 -rw-r--r-- 1 grahamd grahamd 106 Aug 10 21:14 opt.pyo

Thus with -OO the doc string does vanish.

Important thing here is that I removed compiled file between each
test. If you don't do that it will just use the existing one on disk.
So running -OO where .pyo file from -O already exists will show
docstring. Run -O where .pyo file from -OO and docstring is gone.
Thus, what was used when .pyo file is generated first time is what is
important.

Thus, using -OO will probably not gain you anything unless you ensure
that no .pyo files laying around at all, allowing them to be generated
as -OO in the first place.

Graham

Adal Chiriliuc

unread,
Aug 11, 2007, 12:35:50 PM8/11/07
to modwsgi
I was careful to remove the file ;)

I believe it worked for you because you used Python 2.3.5. I remember
testing this in Python 2.3 and 2.4.

But in 2.5 this changed. Another regression is that the file name is
stored for each function object, instead of once per .pyc file.

Here is a screen shot from a Xubuntu virtual machine with 2.5.1c1:
http://adal.chiriliuc.com/temp/opt.png

The doc-string did appear in the last command run (alone in the window
than the terminal text was restored).

Note: tst.py is as displayed in the editor window, despite the * in
the window title.

On 10 Aug, 14:25, "Graham Dumpleton" <graham.dumple...@gmail.com>
wrote:

Reply all
Reply to author
Forward
0 new messages