How to use paver.virtual.bootstrap?

[Matt Kangas]

I'm attempting to use paver.virtual.bootstrap in a multi-package build
process, and I'm feeling slightly confused & wondering if I'm going
down the right path...

I'm still working on the same project noted earlier:
- N sub-projects using distutils or setuptools (it's a mix)
- Want to automate the whole build using paver

By "automate" I mean to define top-level tasks for:
- "clean" - clean all subpackages
- "build" - build all subpackages
- "install" - install all subpackages into a virtualenv
- "lint" - run pychecker/pyflakes across all *installed* code
- "test" - run nosetests against all *installed* code

I have "clean" and "build" working, "install" working too -- except
for my one newer project which uses setuptools, and it's complaining:

-------
## install: ./telemetry
> cd ./telemetry; /usr/bin/paver install --prefix /home/kangas/devel/trunk/install
---> install
Checking .pth file support in /home/kangas/devel/trunk/install/lib/
python2.4/site-packages/
/usr/bin/python -E -c pass
TEST FAILED: /home/kangas/devel/trunk/install/lib/python2.4/site-
packages/ does NOT support .pth files
error: bad install directory or PYTHONPATH

You are attempting to install a package to a directory that is not
on PYTHONPATH and which Python does not read ".pth" files from.
-------

Ok, this is not really surprising: I ran that Paver "install" task
outside of the virtualenv.

What's the best way of invoking that task within the virtualenv? And
invoking my other tasks, "lint" and "test"?

The options I see thus far include:

1. Use options.virtualenv.paver_command_line
2. Invoke INSTALL_DIR/bin/python in my subprocess instead of
sys.executable when it exists
(created by the "bootstrap" task)

Am I missing another option?

options.virtualenv.paver_command_line seems hugely limiting, because
(AFAIK) it defines one fixed set of tasks which can be run within the
virtualenv. But I want to be able to run any combination of different
tasks against my code, after it's built+installed into the virtualenv.

Here's the relevant parts of my pavement.py so far:

---------

#
# Set Options
# -----------

options(
setup = dict(),
virtualenv = Bunch(
script_name="install/bootstrap.py",
# packages_to_install=('nose',),
# paver_command_line=""
)
)

#
# Begin Tasks
# -----------

@task
def work_dirs():
"""Create working dirs"""
for d in WORKDIRS:
d.mkdir()

@task
def clean():
"""Clean all build artifacts"""
for d in WORKDIRS:
d.rmtree()
recur_task("clean", args="-a", ignore_error=True)


@task
@needs(['work_dirs','paver.virtual.bootstrap'])
def bootstrap():
"""create virtualenv in ./install"""
sh('cd install; %s bootstrap.py' % sys.executable)

@task
@needs('work_dirs')
def build():
"""Build all subprojects"""
recur_task("build", args=SUBTASK_ARGS['build'])

@task
@needs(['work_dirs','bootstrap'])
def install():
recur_task("install", args=SUBTASK_ARGS['install'])

@task
def test():
"""Test all subprojects"""
recur_task("test")

## definitions for recur_task() and related functions deleted for
clarity

[Kevin Dangoor]

I'll definitely have to get some more docs put together around
paver.virtual when I finally get a chance to finish off what I'm
working on with it now.

The idea of the bootstrap script is that the bootstrap script is
responsible for creating a virtualenv with stuff installed and then
running paver to finish things off. So that bootstrap script +
pavement.py combine to set up an environment to do *something*
(whatever it is you need done). I've used it for setting up
development environments thus far.

So, the bootstrap is something you run just the first time. After
that, the easiest thing to do is

source bin/activate

and then just run paver and python as normal and have everything
happen within the virtualenv. Does that make sense?

Kevin

--
Kevin Dangoor
Product Manager
SitePen, Inc.
Web development experts:
development, support, training

ke...@sitepen.com

[Matt Kangas]

Kevin,

That makes sense. But it sounds like I'm attempting to do more with
paver.virtual than you've bitten off thus far. :-)

In a nutshell, I'm trying to implement a continuous-integration build.
The build would run in reaction to check-ins on my SVN repository,
with no human intervention at all. It will run many times/day.

The build must start with a "clean-all" step that expunges all traces
of previous builds. Then I want to build packages in dependency order,
install them in some new (paver-created) directory, and run my full
test suite.

I'm trying to do this all from one top-level command: (haven't gotten
to a "test" step yet)
"svn up; paver clean install"

Some obvious problems:

- The top-level "paver" process is necessarily running outside of the
virtualenv. It could be running in _a_ virtualenv, but I wouldn't want
it to be the same virtualenv that my build artifacts get installed
into.
- At some point in my build process, I create the virtualenv, then
begin running tasks "inside" the virtualenv. All of this must be
spawned automatically from the initial process.
- My test suites can only run on installed code, not the source tree:
I have some C modules that I'm building.

Does that make sense?

The way I'm trying to approach it thus far:

- kick off initial "paver" task using my system-installed Python
- clean: rm -rf ./install
- mkdir ./install
- "bootstrap" task: create ./install/bootstrap.py, then run it
- run my subtasks ("python setup.py install") using ./install/bin/
python

As you'll note, I don't use bin/activate -- I'm not running any manual
shell commands whatsoever. This must all be kicked off by a master
process. (Hence, why I'm looking at Paver in the first place.)

There's also little point in recycling a previously-build virtualenv.
To reuse it, I would have to guarantee absolutely all of my previous
build artifacts were removed. It's probably better to just nuke it
entirely and focus on recreating it quickly. (And thus have a build-
install-test process that is stateless and trivially bootstrapped on
new boxes.)

I'm currently hitting an issue where a system site-packages module
(cjson) isn't getting copied into the virtualenv. Any idea how to
solve this?

It's also unfortunate that easy_install always makes requests to
pypi.python.org every time, but I can probably install a Squid proxy
and run the requests through that to remove some of the overhead.

--Matt

[Matt Kangas]

> I'm currently hitting an issue where a system site-packages module
> (cjson) isn't getting copied into the virtualenv. Any idea how to
> solve this?

I've found my problem here, and the solution may be useful to anyone
else trying to use paver+virtualenv on x64 Linux. (I'm using CentOS
5.1 x86_64)

@task
@needs(['work_dirs', 'paver.virtual.bootstrap'])
def bootstrap():
"""create virtualenv in ./install"""

# run bootstrap
sh('cd %s; %s bootstrap.py' % (INSTALL_DIR, sys.executable))

# HACQUE FOR CENTOS x64:
# Create a ".pth" file in the virtualenv "site-packages" dir.
# When the virtualenv python starts, site.py will process this
file
# and "/usr/lib64/.../site-packages" will be added to sys.path
pyvname = "python"+sys.version[:3]
lib64_sitedir = path(sys.prefix) / "lib64" / pyvname / "site-
packages"
if lib64_sitedir.exists():
pth = INSTALL_DIR / 'lib'/ pyvname / 'site-packages' / 'global-
site.pth'
pth.write_text(lib64_sitedir)

[Kevin Dangoor]

On Sep 18, 2008, at 4:20 PM, Matt Kangas wrote:

>
>> I'm currently hitting an issue where a system site-packages module
>> (cjson) isn't getting copied into the virtualenv. Any idea how to
>> solve this?

I ran into something (this may be the same problem... not sure) just
getting a virtualenv installed in the first place. I was trying to run
virtualenv as a non-root user and it was failing because it was trying
to install setuptools globally. It turns out that the system python
dir had a distutils.cfg in lib/python/distutils. My solution was to
override that with a setup.cfg in the current directory (the top of
the virtualenv, where I was running the command).

Kevin

[Kevin Dangoor]

On Sep 17, 2008, at 5:05 PM, Matt Kangas wrote:

> The way I'm trying to approach it thus far:
>
> - kick off initial "paver" task using my system-installed Python
> - clean: rm -rf ./install
> - mkdir ./install
> - "bootstrap" task: create ./install/bootstrap.py, then run it
> - run my subtasks ("python setup.py install") using ./install/bin/
> python
>
> As you'll note, I don't use bin/activate -- I'm not running any manual
> shell commands whatsoever. This must all be kicked off by a master
> process. (Hence, why I'm looking at Paver in the first place.)

Usually, the "bootstrap" task is run once (or infrequently). You would
check the resulting script into your source control or distribute it
in some other fashion. That way, the target system does not need
virtualenv at all.

> There's also little point in recycling a previously-build virtualenv.
> To reuse it, I would have to guarantee absolutely all of my previous
> build artifacts were removed. It's probably better to just nuke it
> entirely and focus on recreating it quickly. (And thus have a build-
> install-test process that is stateless and trivially bootstrapped on
> new boxes.)

Yeah, nuking the build virtualenv makes sense.

> I'm currently hitting an issue where a system site-packages module
> (cjson) isn't getting copied into the virtualenv. Any idea how to
> solve this?

Does virtualenv actually copy the system site-packages modules in, or
just include them in the pythonpath? I didn't think it actually copied
them. I could be wrong, though, because I hadn't looked at that
machinery.

> It's also unfortunate that easy_install always makes requests to
> pypi.python.org every time, but I can probably install a Squid proxy
> and run the requests through that to remove some of the overhead.
>

Check out Ian Bicking's new pyinstall: http://blog.ianbicking.org/2008/09/23/pyinstall-a-new-hope/

That will likely help you already, and Ian is certainly very open to
patches.

Kevin