virtualenv, PIP, fabric and patches to external projects

[Alex Rades]

Hi,
I know this is probably not 100% django related (well, actually is 0%)
but I think most of you have worked with virtualenv and PIP and this
looks like a good place to talk about deployment scenarios.

Basically, I'd like to handle dependencies of my virtual environment
with PIP, but I have some patches to external projects that I have to
apply to apply before I can use them.
Looks like PIP does not offer a facility to apply patches to
downloaded dependencies. Maybe pip is not the right tool to do this.

How do you handle your patching needs in your environments? I've
started thinking about using fabric to do this, but I'm not really
sure.

Thank you very much.

[Kevin Teague]

With patching Python code, there is always the option of dynamically
modifying the code at run-time ... aka doing it monkey style. You can
do this in a bit more orderly fashion by using something like the
'monkey' package, which lets you use a signature on the function being
packaged, so that you become aware when you update the code being
patched that it's not longer the same code.

http://pypi.python.org/pypi/monkey

But beyond that, PIP is probably not the right tool for applying
patches. It's probably also not the right tool for setting up
databases, configuring LDAP, cleaning the cats liter, etc. I use
Buildout for this. Which a number of people use in conjunction with
virtualenv and pip (there is a Buildout recipe for driving pip for
example). Although there are also recipes which let you manage
packages in an isolated fashion without using virtualenv+pip - which
is usually what I do since they allow you to re-use the same packages
between deployments. I think if I installed all the deployments on my
machine using PIP where the code is repeated for every deployment I
would run out of i-nodes.

Anyways, patching in Buildout, that is patching C code, etc. ... aka
real patching or manly patching, is usually done with a 'configure;
make; make install;' recipe. With the zc.recipe.cmmi (http://
pypi.python.org/pypi/zc.recipe.cmmi) this looks like:

[buildout]
parts = foo

[foo]
recipe = zc.recipe.cmmi
url = http://url-path/to/some-file.tgz
patch = ${buildout:directory}/patches/config.patch
patch_options = -p0

I haven't actually used Fabric yet (keep meaning to switch some
deployments over some day, but I guess I'm too busy doing manuall
deployments to have time for this ;P), but it's my understanding this
is a tool for executing remote commands (e.g. commands to invoke pip
or buildout). I guess you can always accomplish quite a bit with BASH
commands, but I much prefer Buildout as it's a declarative
configuration system as opposed to an imperative series of commands.

[Wes Winham]

We solve this problem by keeping forks of the external projects on
github, bitbucket, etc. Since PIP works so nicely with those tools,
it's pretty easy to do. You'll also find that beyond the initial work
of creating your own fork (which is actually really really easy on
both github and bitbucket), maintaining your patches is easier in the
fork than keeping your patch files laying around. It's also nice in
that if your patch is useful to the project, it's easy for them to
merge it in, which is good for both you and the project. Of the 19
external django apps we use, we maintain our own branches on github/
bitbucket for 7 of them because of minor patching we had to do.

One tip though, it took me a while to get this through my head, but
explicit versions in your requirements file are almost always the way
to go. You might think it's a hassle to update your requirements file
every time you make a new commit to your fork of a project, but it
ensures that no change every sneaks up on you and it also lets you add
a useful commit message to your actual project like "Updated
requirements for django-foo to add support for feature X and fix bug
Y" which means you only have one commit log to grep through to find
what you need later. It also means that you can always reproduce old
builds if needed.

-Wes