sys.path ordering troubles with pip/virtualenv/ipython and readline on Mac OS X - solved?

Showing 1-1 of 1 messages
sys.path ordering troubles with pip/virtualenv/ipython and readline on Mac OS X - solved? JK Laiho 12/25/09 3:07 PM
Due to licensing issues, OS X uses libedit instead of GNU readline,
and the system's Python (on Leopard and Snow Leopard at least) have a file in /System/Library/Frameworks/Python.framework/
Versions/2.5/lib/python2.5/lib-dynload that is linked against libedit
(2.6 instead of 2.5 for SL). A separate readline package can be
installed from PyPi, but there's a gotcha.

I ran into difficulties while running IPython, which stubbornly
refused to use the readline I'd installed into site-packages with pip.
Reading some blogs, it turned out that easy_installing readline would
do the trick, and indeed it did. The reason is simple: easy_install
installs the pre-built egg from PyPi (or builds an egg if pointed to
the source tarball) and inserts that in easy_install.pth in front of
the usual search paths (including the lib-dynload one mentioned
above), meaning "import readline" will import the easy_installed, GNU
readline-using version instead of the system default, making IPython
work fine with it. But pip doesn't touch easy_install.pth when
installing non-editable packages, so the system will get
imported instead as it lives further up in sys.path.

In this instance I was installing readline globally into /Library/
Python/2.5/site-packages, but this behaviour is also problematic for
my other use case, where I'm running several Python projects from
isolated virtual environments, using pip requirements files for
constructing and populating them. After several hours of trial and
error, I was able to come up with a solution of sorts, although it's a
bit more complex than I'd like.

I'm posting this here since I don't have a blog yet, and I'm hoping
that people with the same issue will eventually be able to find this
post by googling the things I googled while researching this :-). I'm
also hoping that the smarter-than-myself readers of this group might
be able to improve on the used method and correct any misinformation I
may present.

Anyway, here goes.

Since the construction of a virtual environment symlinks the entire
lib-dynload directory from under /System/Library/(...) into the
virtual environment, just deleting its contained from
inside the environment to make the in the environment's
site-packages found first won't cut it. Messing with the system Python
distribution is plain wrong. There might be a clean way to do a
permanent reordering of sys.path inside the virtual environment after
doing the initial "pip install -E project_env -r pip-requirements.txt"
to create it. All that needs to be done on that front is to place the /
Library/Python/2.x/site-packages path in front of all the /System/
Library/(...) paths in some automated fashion, maybe with creative
modifications to easy_install.pth. Unfortunately, pip doesn't have
post-install hooks (that I know of), and I'm not aware of any other
ways to do so.

Another solution might be to host a copy of readline in a local VCS
repository that could be used with "pip install -e git+ssh://(etc.)",
since pip inserts editable packages into easy_install.pth so that they
appear before the /System paths. Not an elegant solution, either.

A very simple solution would be to just run "easy_install -a readline"
once in the activated virtual environment, but I thought we were
trying to replace easy_install here ;-). That's also one more manual
step to perform (and it suffers from a gotcha that is detailed below).

The most promising solution is the use of virtualenv's bootstrap
script facility. Pip apparently can't use a custom virtualenv script
created with that facility when creating an environment with the -E
parameter, so a 'python --no-site-packages
project_env' would have to be run before the pip command mentioned

I've gotten this far with the script creation, modelling it after the
official example at virtualenv's PyPi page:

import virtualenv, textwrap
output = virtualenv.create_bootstrap_script(textwrap.dedent("""
import os, subprocess
def after_install(options, home_dir):
    # TODO: only run this if OS X is detected, since some
    # developers run Linux[join(home_dir, 'bin', 'easy_install'), '-a',
f = open('', 'w').write(output)

When running the created script to create the
virtual environment, the -a parameter to easy_install will prevent it
from simply adding the path to the readline package living in the
global site-packages to the virtual environment's easy_install.pth,
and force it to copy it to the environment instead. It's a bit silly
that even the easy_install script living inside an activated virtual
environment is still able to "see" into the global site-packages
directory. Smells like a virtualenv isolation bug to me, although I
wouldn't know if there was some reason why this can't be avoided
(which is conceivable, since easy_install isn't built to respect
virtual environments).

I have yet to find an incantation that would force easy_install to
disregard the copy of readline living inside the global site-packages
directory and always download it from PyPi instead, so in that regard
this solution is incomplete; if a virtual environment needs to have a
different version than the one in site-packages, this solution isn't
good enough.

Improvements, alternate solutions and corrections are most welcome. If
there's a way to avoid using a custom bootstrap script to create the
virtual environment with GNU readline, and instead just use whatever
facilities pip has hidden under the hood while running it with the -E
and -r parameters against the requirements file, that would be great.

- JK Laiho