carton evangelism

[Jeff Hammel]

so i wrote a thing....

carton
======

make a self-extracting virtualenv from directories or URLs of packages

See the docstring of carton.py for documentation:
http://k0s.org/mozilla/hg/carton/file/tip/carton.py#l3

Carton provides a one-file portable package of a virtualenv and all of
its software.

Carton lives at http://k0s.org/mozilla/hg/carton

Tests illustrating carton's use are at
http://k0s.org/mozilla/hg/carton/file/tip/tests/doctest.txt

A self-extracting carton of, um, carton is at
http://k0s.org/mozilla/hg/carton/file/tip/carton-env.py

Why did you write this thing?
-----------------------------

For whatever reasons, people have trouble using virtualenv so I
figured I'd make that easy. People also seem to believe that you can
zip up a virtualenv and treat it as portable. In general, virtualenvs
are architecture and library dependent, so this isn't a good idea.
carton works around this problem by packaging the instructions
necessary to make the virtualenv instead of dealing with
library/hardware incompatabilities.

Should I use carton for my deployment strategy?
-----------------------------------------------

Maybe. Carton is pretty minimalist. It does effectively one thing:
makes a file that unfolds to a virtualenv with software setup for
development. If this fits in with what you want to do, then sure! If
not, then probably not.

TODO
----

carton is essentially finished. Features can be added by request
(just mail me: jhammel __at__ mozilla __dot__ com), but the scope of
the project is pretty much fulfilled.

There are a few items I would like to add:

- currently, carton only works if connected to a network or if a local
virtualenv tarball is specified with `--virtualenv`. I would like to
make a script, `regen_virtualenv.py`, that would add a hashed value
of the virtualenv tarball to the global `VIRTUAL_ENV`
variable. Similarly, one should be able to retrieve the stripped
down version of the script without this hash if desired

- ... and being able to take virtualenv from its import path, if
available

- it would be nice to be able to specify a post-deployment script to
be run

[Kumar McMillan]

On Mon, Jul 11, 2011 at 3:06 PM, Jeff Hammel <k0s...@gmail.com> wrote:
> make a self-extracting virtualenv from directories or URLs of packages

I feel like I've been trying to solve this problem for most of my life!

In AMO and other Mozilla sites we just cram everything into a "vendor"
git submodule. This means that the developer just does:
git clone <url> --recursive
and he/she has everything minus compiled modules. There is the start
of a management script to add/upgrade with pip:
https://github.com/jbalogh/vending-machine It's obviously git only
but this approach has a ton of benefits.

I first tried to make virtualenv setup easy with this eco command line util:
https://bitbucket.org/kumar303/eco
It is dumb though (don't use it) because it requires a command line tool.

Next I scrapped that and created velcro:
https://bitbucket.org/kumar303/velcro/
which is better integrated into virtualenv/pip but it still requires a
command line to *start* your project. The dev then gets to use it as
a virtualenv bootstrap script which works pretty well. I see now that
I put virtualenv as a global requirement in the docs but I think the
new style bootstrap scripts fix that. I ran into problems with
isolating command line scripts in the virtualenv for globally shared
packages but who does that anyway.

The ultimate goal I've been going for is to hand someone a project
that they can install in the easiest way possible using a standard
Python interpreter.

btw, you may get more mileage with carton out of converting it to a
bootstrap script:
http://www.virtualenv.org/en/latest/index.html#creating-your-own-bootstrap-scripts
I do like how carton gives you a zip bundle of dependencies.

-Kumar

[Ian Bicking]

BTW, newer Python versions (2.6+?) can also execute zip files, which might be advantageous (smaller, no base64, and you don't have to load the whole thing into memory).  I believe it's just if you put a file __main__.py in the root of the zip file, it should be executed when you do "python bundle.zip".  Unfortunately the failure on earlier versions won't be pleasant.

And also tangential, you might find ScriptTest useful for testing: http://pythonpaste.org/scripttest/

On Mon, Jul 11, 2011 at 3:06 PM, Jeff Hammel <k0s...@gmail.com> wrote:

[Jeff Hammel]

Replies inline:

On Mon, Jul 11, 2011 at 1:35 PM, Kumar McMillan
<kumar.m...@gmail.com> wrote:
> On Mon, Jul 11, 2011 at 3:06 PM, Jeff Hammel <k0s...@gmail.com> wrote:
>> make a self-extracting virtualenv from directories or URLs of packages
>
> I feel like I've been trying to solve this problem for most of my life!
>
> In AMO and other Mozilla sites we just cram everything into a "vendor"
> git submodule.  This means that the developer just does:
> git clone <url> --recursive
> and he/she has everything minus compiled modules.  There is the start
> of a management script to add/upgrade with pip:
> https://github.com/jbalogh/vending-machine  It's obviously git only
> but this approach has a ton of benefits.
>
> I first tried to make virtualenv setup easy with this eco command line util:
> https://bitbucket.org/kumar303/eco
> It is dumb though (don't use it) because it requires a command line tool.
>
> Next I scrapped that and created velcro:
> https://bitbucket.org/kumar303/velcro/
> which is better integrated into virtualenv/pip but it still requires a
> command line to *start* your project.  The dev then gets to use it as
> a virtualenv bootstrap script which works pretty well.  I see now that
> I put virtualenv as a global requirement in the docs but I think the
> new style bootstrap scripts fix that.  I ran into problems with
> isolating command line scripts in the virtualenv for globally shared
> packages but who does that anyway.

I've addd these to http://brasstacks.mozilla.com/toolbox/ ; feel free to edit.

> The ultimate goal I've been going for is to hand someone a project
> that they can install in the easiest way possible using a standard
> Python interpreter.
>
> btw, you may get more mileage with carton out of converting it to a
> bootstrap script:
> http://www.virtualenv.org/en/latest/index.html#creating-your-own-bootstrap-scripts

I'm not sure how this would help, exactly? But patches welcome :)

[Jeff Hammel]

Thanks for the pointers!