[Q] installing Subversion SWIG bindings in a virtualenv?
Olaf Meeuwissen
I've been looking into setting up a server with multiple Trac envs,
using a combination of virtualenv and WSGI to keep the environments
separate. As I'd also like to keep the server's Python environment
separate from that used by the Trac environments, I created another
`virtualenv --no-site-packages` and used that for WSGIPythonHome.
To this WSGIPythonHome virtualenv you can easy_install whatever you
need for all the Trac environments.
I got that to work fine as long as you don't have a Trac environment
that uses a Subversion repository. Trac requires the Subversion SWIG
binding (as opposed to PySVN which is just an easy_install away) and
I have no clue how to install that (short of `make swig-py; sudo make
install-swig-py`, which is a no-go on a production machine).
Is there an easy way to install the Subversion SWIG bindings in a
virtualenv, or two?
Thanks in advance,
--
Olaf Meeuwissen FLOSS Engineer -- AVASYS Corporation
FSF Associate Member #1962 sign up at http://member.fsf.org/
Noah Kantrowitz
> Hi all,
>
> I've been looking into setting up a server with multiple Trac envs,
> using a combination of virtualenv and WSGI to keep the environments
> separate. As I'd also like to keep the server's Python environment
> separate from that used by the Trac environments, I created another
> `virtualenv --no-site-packages` and used that for WSGIPythonHome.
> To this WSGIPythonHome virtualenv you can easy_install whatever you
> need for all the Trac environments.
>
> I got that to work fine as long as you don't have a Trac environment
> that uses a Subversion repository. Trac requires the Subversion SWIG
> binding (as opposed to PySVN which is just an easy_install away) and
> I have no clue how to install that (short of `make swig-py; sudo make
> install-swig-py`, which is a no-go on a production machine).
>
> Is there an easy way to install the Subversion SWIG bindings in a
> virtualenv, or two?
Subversion installs the bindings to $PREFIX/lib[64]/svn-python/ so if
you set $PREFIX to be some temp folder, and then move the svn/ and
libsvn/ folders into the virtualenv site-packages it should work fine.
--Noah
Olaf Meeuwissen
Eh, I think I also need the various *.py files, correct?
I had a look at the content of Debian's python-subversion package and
it looks like I can use that as is by just doing a `dpkg-deb -x` in
the right place.
... me goes off to give that a try ...
Tough luck. I couldn't get that to work.
Resyncing from within a virtualenv gives:
Command failed: Unsupported version control system "svn": "No module named _fs"
Apparently some of the search paths are missing bits and pieces.
At least the all the *.(so|py) files in python-subversion do not seem
to refer to absolute filenames. So am under the impression that Noah
is right and the $PREFIX approach should work fine but I do not have
the time to test that right now :-(
Any other suggestions?
osimons
On Aug 21, 5:02 am, Olaf Meeuwissen <olaf.meeuwis...@avasys.jp> wrote:
> Is there an easy way to install the Subversion SWIG bindings in a
> virtualenv, or two?
usual, and in each virtualenv sitepackages derived from this Python
add a .pth file that points to location of svn libraries.
Like,
$ echo "/opt/local/lib/svn-python2.4" > /path/to/my/virutalenv/lib/
python2.4/site-packages/svn-python.pth
(My /opt/local/lib/svn-python2.4 contains the directories 'svn' and
'libsvn')
:::simon
https://www.coderesort.com
Olaf Meeuwissen
osimons <odds...@gmail.com> writes:
> On Aug 21, 5:02 am, Olaf Meeuwissen <olaf.meeuwis...@avasys.jp> wrote:
>
>> Is there an easy way to install the Subversion SWIG bindings in a
>> virtualenv, or two?
>
> Install the Subversion bindings against your correct main Python as
> usual,
That would be `apt-get install python-subversion` in my case. Note
that this package installs the *.py files and the native .so libs in
different places.
*.py in /usr/share/python-support/python-subversion/{svn,libsvn}
*.so in /usr/lib/python-support/python-subversion/python2.[45]/libsvn
The libsvn_swig_py2.[45] .so's are in /usr/lib/.
> and in each virtualenv sitepackages derived from this Python
> add a .pth file that points to location of svn libraries.
>
> Like,
>
> $ echo "/opt/local/lib/svn-python2.4" > /path/to/my/virutalenv/lib/
> python2.4/site-packages/svn-python.pth
Did that.
$ cat /path/to/virtualenv/lib/python2.5/site-packages/svn-python.pth
/usr/lib/python-support/python-subversion/python2.5
/usr/lib/python-support/python-subversion
/usr/share/python-support/python-subversion
Restarted apache2 and fired up a new browser for good measure.
> (My /opt/local/lib/svn-python2.4 contains the directories 'svn' and
> 'libsvn')
Unfortunately, that didn't work.
Here's (what I think is) the relevant piece of the log:
2008-08-26 08:25:48,471 Trac[svn_fs] INFO: Failed to load Subversion bindings
Traceback (most recent call last):
File "/path/to/virtualenv/lib/python2.5/site-packages/Trac-0.11.1-py2.5.egg/trac/versioncontrol/svn_fs.py", line 253, in __init__
_import_svn()
File "/path/to/virtualenv/lib/python2.5/site-packages/Trac-0.11.1-py2.5.egg/trac/versioncontrol/svn_fs.py", line 69, in _import_svn
from svn import fs, repos, core, delta
File "/usr/share/python-support/python-subversion/svn/fs.py", line 19, in <module>
from libsvn.fs import *
File "/usr/share/python-support/python-subversion/libsvn/fs.py", line 7, in <module>
import _fs
ImportError: No module named _fs
FWIW, I'm using virtualenv-1.1 and using a virtualenv with site
packages (that is, created without the --no-site-packages option) as
my WSGIPythonHome works fine.
Any ideas?
Graham Dumpleton
import sys
print >> sys.stderr, str(sys.path)
This will cause value of sys.path to be logged to Apache error log. Go
through what is dumped out to just make sure that .pth files contents
added correctly and that all the other directories added look
reasonable and refer to your virtual environment.
Graham
On Aug 26, 9:38 am, Olaf Meeuwissen <olaf.meeuwis...@avasys.jp> wrote:
> Sorry, meant to get back to this earlier but work got in the way.
>
> _import_svn()
> File "/path/to/virtualenv/lib/python2.5/site-packages/Trac-0.11.1-py2.5.egg/trac /versioncontrol/svn_fs.py", line 69, in _import_svn
Olaf Meeuwissen
> In the WSGI script file, add:
>
> import sys
>
> print >> sys.stderr, str(sys.path)
>
> This will cause value of sys.path to be logged to Apache error log. Go
> through what is dumped out to just make sure that .pth files contents
> added correctly and that all the other directories added look
> reasonable and refer to your virtual environment.
Thanks for the tip. Did that and here is what ends up in my Apache
error log:
[Wed Aug 27 08:45:58 2008] [error] ['/path/to/WSGIHome/lib/python2.5/site-packages/setuptools-0.6c8-py2.5.egg', '/path/to/WSGIHome/lib/python2.5/site-packages/Pygments-0.10-py2.5.egg', '/path/to/WSGIHome/lib/python2.5/site-packages/pytz-2008c-py2.5.egg', '/path/to/SandBox/lib/python2.5/site-packages/setuptools-0.6c8-py2.5.egg', '/path/to/SandBox/lib/python2.5/site-packages/pysqlite-2.4.1-py2.5-linux-i686.egg', '/path/to/SandBox/lib/python2.5/site-packages/Trac-0.11.1-py2.5.egg', '/path/to/SandBox/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg', '/path/to/WSGIHome/lib/python2.5', '/path/to/WSGIHome/lib/python2.5/plat-linux2', '/path/to/WSGIHome/lib/python2.5/lib-tk', '/path/to/WSGIHome/lib/python2.5/lib-dynload', '/usr/lib/python2.5', '/usr/lib/python2.5/lib-tk', '/path/to/WSGIHome/lib/python2.5/site-packages', '/path/to/SandBox/lib/python2.5/site-packages', '/usr/share/python-support/python-subversion']
My WSGIPythonHome is set to /path/to/WSGIHome which is a symlink
pointing to a --no-site-packages virtualenv. I use the symlink so I
can switch easily between virtualenvs w/ and w/o site packages. The
/path/to/SandBox is another no-site-packages virtualenv and here's
where I added a python-subversion.pth with the contents shown by the
last element of the sys.path.
Still, I get an error message in the browser and the following in my
trac.log:
2008-08-27 08:45:57,961 Trac[svn_fs] INFO: Failed to load Subversion bindings
Traceback (most recent call last):
File "/path/to/SandBox/lib/python2.5/site-packages/Trac-0.11.1-py2.5.egg/trac/versioncontrol/svn_fs.py", line 253, in __init__
_import_svn()
File "/path/to/SandBox/lib/python2.5/site-packages/Trac-0.11.1-py2.5.egg/trac/versioncontrol/svn_fs.py", line 69, in _import_svn
from svn import fs, repos, core, delta
File "/usr/share/python-support/python-subversion/svn/fs.py", line 19, in <module>
from libsvn.fs import *
File "/usr/share/python-support/python-subversion/libsvn/fs.py", line 7, in <module>
import _fs
ImportError: No module named _fs
I'm clueless. Anyone have any ideas?
Graham Dumpleton
> > In the WSGI script file, add:
>
> > import sys
>
> > print >> sys.stderr, str(sys.path)
>
> > This will cause value of sys.path to be logged to Apache error log. Go
> > through what is dumped out to just make sure that .pth files contents
> > added correctly and that all the other directories added look
> > reasonable and refer to your virtual environment.
>
> Thanks for the tip. Did that and here is what ends up in my Apache
> error log:
>
> My WSGIPythonHome is set to /path/to/WSGIHome which is a symlink
> pointing to a --no-site-packages virtualenv. I use the symlink so I
> can switch easily between virtualenvs w/ and w/o site packages. The
> /path/to/SandBox is another no-site-packages virtualenv and here's
> where I added a python-subversion.pth with the contents shown by the
> last element of the sys.path.
/path/to/SandBox/lib/python2.5/site-packages
added to sys.path?
If this contains .pth files you can't just use:
sys.path.append(....)
as that doesn't result in .pth files being parsed with content added
to sys.path in the process.
So, post exact details of the following.
1. Version of mod_wsgi.
2. What WSGIPythonHome and WSGIPythonPath are set to in Apache
configuration.
3. Whether you are using WSGIDaemonProcess or WSGIProcessGroup and if
so what they are set to.
4. What 'ls -las' output is on your sandbox site packages directory.
5. The contents of the .pth files in that sandbox site packages
directory.
6. What your WSGI script file is doing in respect of manipulations of
sys.path or calls to site.addsitedir().
Graham
> Still, I get an error message in the browser and the following in my
> trac.log:
>
> 2008-08-27 08:45:57,961 Trac[svn_fs] INFO: Failed to load Subversion bindings
> Traceback (most recent call last):
> _import_svn()
> File "/path/to/SandBox/lib/python2.5/site-packages/Trac-0.11.1-py2.5.egg/trac/ve rsioncontrol/svn_fs.py", line 69, in _import_svn
Olaf Meeuwissen
> On Aug 27, 9:57 am, Olaf Meeuwissen <olaf.meeuwis...@avasys.jp> wrote:
>> Graham Dumpleton <Graham.Dumple...@gmail.com> writes:
>> > In the WSGI script file, add:
>>
>> > import sys
>>
>> > print >> sys.stderr, str(sys.path)
>>
>> > This will cause value of sys.path to be logged to Apache error log. Go
>> > through what is dumped out to just make sure that .pth files contents
>> > added correctly and that all the other directories added look
>> > reasonable and refer to your virtual environment.
>>
>> Thanks for the tip. Did that and here is what ends up in my Apache
>> error log:
>>
>> [Wed Aug 27 08:45:58 2008] [error] ['/path/to/WSGIHome/lib/python2.5/site-packages/setuptools-0.6c8-py2.5.egg' , '/path/to/WSGIHome/lib/python2.5/site-packages/Pygments-0.10-py2.5.egg', '/path/to/WSGIHome/lib/python2.5/site-packages/pytz-2008c-py2.5.egg', '/path/to/SandBox/lib/python2.5/site-packages/setuptools-0.6c8-py2.5.egg', '/path/to/SandBox/lib/python2.5/site-packages/pysqlite-2.4.1-py2.5-linux-i6 86.egg', '/path/to/SandBox/lib/python2.5/site-packages/Trac-0.11.1-py2.5.egg', '/path/to/SandBox/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686 .egg', '/path/to/WSGIHome/lib/python2.5', '/path/to/WSGIHome/lib/python2.5/plat-linux2', '/path/to/WSGIHome/lib/python2.5/lib-tk', '/path/to/WSGIHome/lib/python2.5/lib-dynload', '/usr/lib/python2.5', '/usr/lib/python2.5/lib-tk', '/path/to/WSGIHome/lib/python2.5/site-packages', '/path/to/SandBox/lib/python2.5/site-packages', '/usr/share/python-support/python-subversion']
>>
>> My WSGIPythonHome is set to /path/to/WSGIHome which is a symlink
>> pointing to a --no-site-packages virtualenv. I use the symlink so I
>> can switch easily between virtualenvs w/ and w/o site packages. The
>> /path/to/SandBox is another no-site-packages virtualenv and here's
>> where I added a python-subversion.pth with the contents shown by the
>> last element of the sys.path.
>
> How in your configuration are you getting:
>
> /path/to/SandBox/lib/python2.5/site-packages
>
> added to sys.path?
The same way as /path/to/WSGIHome/lib/python2.5/site-packages, I'd
presume. That is, I have no clue, really. I think virtualenv and/or
easy_install automagically take care of that.
> If this contains .pth files you can't just use:
>
> sys.path.append(....)
>
> as that doesn't result in .pth files being parsed with content added
> to sys.path in the process.
The /path/to/SandBox/lib/python2.5/site-packages directory contains a
.pth file with the following contents:
/usr/share/python-support/python-subversion
so it seems to get added allright.
> So, post exact details of the following.
>
> 1. Version of mod_wsgi.
$ dpkg-query -W libapache2-mod-wsgi
libapache2-mod-wsgi 2.1-2
$
> 2. What WSGIPythonHome and WSGIPythonPath are set to in Apache
> configuration.
$ grep -r WSGIPythonHome /etc/apache2/sites-enabled/
/etc/apache2/sites-enabled/WSGIHome:WSGIPythonHome /path/to/WSGIHome
$ grep -r WSGIPythonPath /etc/apache2/sites-enabled/
$
> 3. Whether you are using WSGIDaemonProcess or WSGIProcessGroup and if
> so what they are set to.
$ grep -A3 WSGIDaemonProcess /etc/apache2/sites-enabled/SandBox
WSGIDaemonProcess SandBox user=olaf group=olaf \
threads=15 maximum-requests=10000 \
python-path=/path/to/SandBox/lib/python2.5/site-packages
$ grep WSGIProcessGroup /etc/apache2/sites-enabled/SandBox
WSGIProcessGroup SandBox
$
> 4. What 'ls -las' output is on your sandbox site packages directory.
$ ls -las /path/to/SandBox/lib/python2.5/site-packages/
total 356
4 drwxr-xr-x 5 olaf olaf 4096 2008-08-27 08:25 .
4 drwxr-xr-x 4 olaf olaf 4096 2008-08-21 16:25 ..
4 drwxr-xr-x 4 olaf olaf 4096 2008-08-21 16:26 Genshi-0.5.1-py2.5-linux-i686.egg
4 drwxr-xr-x 4 olaf olaf 4096 2008-08-21 16:26 Trac-0.11.1-py2.5.egg
4 -rw-r--r-- 1 olaf olaf 312 2008-08-21 16:26 easy-install.pth
4 drwxr-xr-x 5 olaf olaf 4096 2008-08-21 16:26 pysqlite-2.4.1-py2.5-linux-i686.egg
4 -rw-r--r-- 1 olaf olaf 44 2008-08-27 08:45 python-subversion.pth
324 -rw-r--r-- 1 olaf olaf 324858 2008-05-21 06:20 setuptools-0.6c8-py2.5.egg
4 -rw-r--r-- 1 olaf olaf 29 2008-08-21 16:25 setuptools.pth
$
> 5. The contents of the .pth files in that sandbox site packages
> directory.
$ cat /path/to/SandBox/lib/python2.5/site-packages/*.pth
/usr/share/python-support/python-subversion
import sys; sys.__plen = len(sys.path)
./setuptools-0.6c8-py2.5.egg
./pysqlite-2.4.1-py2.5-linux-i686.egg
./Trac-0.11.1-py2.5.egg
./Genshi-0.5.1-py2.5-linux-i686.egg
import sys; new=sys.path[sys.__plen:]; del sys.path[sys.__plen:]; p=getattr(sys,'__egginsert',0); sys.path[p:p]=new; sys.__egginsert = p+len(new)
./setuptools-0.6c8-py2.5.egg
$
> 6. What your WSGI script file is doing in respect of manipulations of
> sys.path or calls to site.addsitedir().
I'm using the trac.wsgi script generated by trac-admin's deploy
command modulo the sys.stderr logging you suggested. FWIW, I've
attached the script.
Hope this helps and thanks for helping me trouble shoot this.
Graham Dumpleton
*.py in /usr/share/python-support/python-subversion/{svn,libsvn}
*.so in /usr/lib/python-support/python-subversion/python2.[45]/
libsvn
$ cat /path/to/virtualenv/lib/python2.5/site-packages/svn-
python.pth
/usr/lib/python-support/python-subversion/python2.5
/usr/lib/python-support/python-subversion
/usr/share/python-support/python-subversion
Shouldn't the directory containing the .so files, ie.,
/usr/lib/python-support/python-subversion/python2.5
still be in the .pth file? You now only have:
/usr/share/python-support/python-subversion
which would mean only .py files would be found.
Yes/No?
Graham
> 1KViewDownload
Graham Dumpleton
On Aug 27, 3:33 pm, Graham Dumpleton <Graham.Dumple...@gmail.com>
wrote:
$ python
Python 2.3.5 (#1, Nov 23 2005, 12:05:23)
[GCC 3.3.2] on sunos5
Type "help", "copyright", "credits" or "license" for more information.
>>> import libsvn
>>> libsvn.__file__
'/usr/local/sys/python/local/lib/python2.3/site-packages/libsvn/
__init__.py'
>>> import svn
>>> svn.__file__
'/usr/local/sys/python/local/lib/python2.3/site-packages/svn/
__init__.py'
and show what you get.
One other possibility is that .so can be found but library
dependencies cant be found. I would though expect different Python
exception, but if subversion bindings are masking the true error, this
might be problem.
Thus, run for your paths:
ldd /usr/local/sys/python/local/lib/python2.3/site-packages/libsvn/
*.so
as the user that the mod_wsgi daemon process would run as.
Graham
Olaf Meeuwissen
> More below.
>
> On Aug 27, 3:33 pm, Graham Dumpleton <Graham.Dumple...@gmail.com> wrote:
>> On Aug 27, 2:54 pm, Olaf Meeuwissen <olaf.meeuwis...@avasys.jp> wrote:
>> > Graham Dumpleton <Graham.Dumple...@gmail.com> writes:
>> [snip!]
Tried that, but it didn't make things work. The additional entries in
the .pth file do get listed in the Apache error logs, though.
BUT! If I also add the libsvn directory that lives below
/usr/lib/python-support/python-subversion/python2.5/ to the .pth file
things work! As in, that 'module _fs not found' error goes away and
I can browse the repository.
For the record, my python-subversion.pth now contains:
/usr/share/python-support/python-subversion
/usr/lib/python-support/python-subversion/python2.5/libsvn
Me now wonders why all this works out of the box using a virtualenv
*with* site-packages ...
A `virtualenv --no-site-packages` use the following sys.path:
['', '/path/to/WSGIHome/lib/python2.5/site-packages/setuptools-0.6c8-py2.5.egg', '/path/to/WSGIHome/lib/python2.5/site-packages/Pygments-0.10-py2.5.egg', '/path/to/WSGIHome/lib/python2.5/site-packages/pytz-2008c-py2.5.egg', '/path/to/WSGIHome/lib/python25.zip', '/path/to/WSGIHome/lib/python2.5', '/path/to/WSGIHome/lib/python2.5/plat-linux2', '/path/to/WSGIHome/lib/python2.5/lib-tk', '/path/to/WSGIHome/lib/python2.5/lib-dynload', '/usr/lib/python2.5', '/usr/lib/python2.5/lib-tk', '/path/to/WSGIHome/lib/python2.5/site-packages']
And a "regular" `virtualenv` uses:
['', '/path/to/WSGIHome/lib/python2.5/site-packages/setuptools-0.6c8-py2.5.egg', '/path/to/WSGIHome/lib/python2.5/site-packages/Pygments-0.10-py2.5.egg', '/path/to/WSGIHome/lib/python2.5/site-packages/pytz-2008c-py2.5.egg', '/path/to/WSGIHome/lib/python25.zip', '/path/to/WSGIHome/lib/python2.5', '/path/to/WSGIHome/lib/python2.5/plat-linux2', '/path/to/WSGIHome/lib/python2.5/lib-tk', '/path/to/WSGIHome/lib/python2.5/lib-dynload', '/usr/lib/python2.5', '/usr/lib/python2.5/lib-tk', '/path/to/WSGIHome/lib/python2.5/site-packages', '/usr/local/lib/python2.5/site-packages', '/usr/lib/python2.5/site-packages', '/usr/lib/python2.5/site-packages/PIL', '/var/lib/python-support/python2.5']
There's gotta be some kind of magic going on in those last four
directories. The first one is empty and I doubt PIL has anything to
do with. The leaves only two but I don't know enough about Python to
figure out where and what. A cluebat is appreciated.
> [snip]
Thanks!