buildout development vs. deployment

[Reikje]

Hi, I am looking into buildout to deploy a django webapp and all it's dependencies. I find it quite useful even during development because it forces you to keep track of your dependencies. A question regarding some best practices. Lets say I have buildout.cfg and setup.py in my project root and checked in into SCM. My webapp is listed under develop in buildout.cfg. While this is great, during deployment this is probably not what you want because you wanna lock the version of your source code. I want to do a git revision checkout to archive this. So i guess I need to maintain two different buildout.cfg files, one for development and one for deployment. How can this be organized to avoid DRY?

On a side note, what are the alternatives to buildout. Maybe there is something even better :)

[Tom Evans]

I have no experience of buildout, but I have spent a lot of time
working on SCM for my django sites, so I'll describe what I do.

Everything here is built around subversion, we don't use git. If you
haven't used subversion, 'svn:externals' check out another repository
in the specified location. It's a way of composing disparate bits of
versioned repositories.

The first thing to discuss is the project source code. This lives in
svn/projectname/trunk, with release branches in
svn/projectname/releases. Minor work happens on trunk, and is merged
to release branch(es). Major work happens on feature branches, which
is then merged to trunk and then to release branches.

Of course, project source code in itself is not enough to run a site.
The project must be deployed and configured for a specific host. We do
this with 'deployments'. Each deployment corresponds to an instance of
a project running on a specific server, and lives in
svn/projectname/deployments/servername, and consists of a skeleton
directory structure (derived from a base version), and lots of
'svn:externals' links.

So the deployment contains the structure of the project, and links to
check out the project code and other required files, like scripts for
starting/stopping the project, a bootstrap script for creating the
virtualenv environment, the settings files, etc, which are also all
stored in subversion.

The next stage is libraries. For this, we use virtualenv and pip. We
have a bootstrap script which creates the virtualenv if it is not
present, and a requirements.pip file, which lists the libraries and
specific versions to install.
All of our libraries are installed via pip, and we have a process
which checks out both our own libraries and patched vendor libraries
(eg, like django-debug-toolbar), builds packages from them, and pushes
them to our own pypi site¹. All our packages are then installed from
this pypi site.
The requirements.pip is another 'svn:externals' file, checked out from
svn/projectname/configurations/. All the production sites share one
requirements.pip.

The project has a default_settings.py file, and each deployment also
checks out a deployment specific settings.py, which imports from
default_settings.py and adds the host-specific settings for that site.

Every check-in on the project results in a buildbot updating a
buildbot specific deployment and running tests. There is a buildbot
for trunk and for each active release branch.

All schema migrations are taken care of by south.

Finally, all the production deployments are managed by a Fabric
fabfile. This contains programmatic instructions for doing various
tasks, eg update app servers to latest version, which:
takes one set of backend servers out of the rotation
stops DB replication from that deployment
updates that deployment
installs/uninstalls libraries as indicated in (updated) requirements.pip
runs south sschema migrations
marks that backend as the only active backend
restarts DB replication
updates all other servers.

If you're not using fabric, or some other automated rollout tool, to
manage your services, you should be!

For our 'bulletproof' sites, we have an exact replica of a production
stack in test. We never update the live site until we've updated this
test replica successfully.

Creating a new backend deployment is as simple as copying the stock
deployment, creating the site specific settings.py, checking it out on
a host and running the bootstrap script. In fact, it's slightly easier
than that, its 'fab new_prod_site hostname'.

Since everything is in subversion, we can easily re-constitute a site
as it was, or reproduce the exact same configuration in development as
we have in production. This aids debugging and increases confidence in
the system.

Cheers

Tom

Tools:

virtualenv: http://www.virtualenv.org/en/latest/index.html
pip: http://www.pip-installer.org/en/latest/index.html
south: http://south.aeracode.org/
fabric: http://fabfile.org/
buildbot: http://trac.buildbot.net/

¹ It's not as good as the real pypi site, its really just a big folder
served over http with all the packages in it. Works well enough.

[Reik Schatz]

Hi Tom,

great answer, thanks. A lot of other sites mentioned Fabric. I'll definitely check it out.

/Reik

--
You received this message because you are subscribed to the Google Groups "Django users" group.

To post to this group, send email to django...@googlegroups.com.
To unsubscribe from this group, send email to django-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-users?hl=en.

[Reinout van Rees]

On 25-04-12 12:21, Reikje wrote:
> Hi, I am looking into buildout to deploy a django webapp and all it's
> dependencies. I find it quite useful even during development because it
> forces you to keep track of your dependencies. A question regarding some
> best practices. Lets say I have buildout.cfg and setup.py in my project
> root and checked in into SCM. My webapp is listed under develop in
> buildout.cfg. While this is great, during deployment this is probably
> not what you want because you wanna lock the version of your source
> code. I want to do a git revision checkout to archive this. So i guess I
> need to maintain two different buildout.cfg files, one for development
> and one for deployment. How can this be organized to avoid DRY?

I have a production.cfg that extends buildout.cfg. I can then add extra
parts ("set up a supervisord to run gunicorn") of modify settings in
existing ones ("use port 10026 instead of 8000").

I get to keep it pretty DRY that way.

Note that I make proper packages out of my development apps and stick
them somewhere that buildout can find them. You could try to use the
mr.developer add-on to check out tags of your stuff.

For hints on getting your own packages up on a pypi of your own:
http://reinout.vanrees.org/weblog/2009/11/09/eggproxy-plus-private-packages.html
That ought to give you some hints.

> On a side note, what are the alternatives to buildout. Maybe there is
> something even better :)

Buildout provides two things:

- Installation + isolation of python packages, just like the
pip+virtualenv environment. With the note that buildout works just fine
on windows and that pip won't install binary windows eggs. Otherwise the
effect is mostly the same, though I like buildout's version handling
better as it is impossible to forget it (unlike forgetting to pass pip a
requirements.txt file).

- Add-on recipes for extra automation. Great for deployment. Generating
nginx/apache config files, setting up a cron job, creating
your_site_dir/var/log directories and so on.

=> for handling everything *within* your project directory, nothing
beats the functionality of buildout. And it combines fine with tools
like fabric that can handle everything outside of the project directory.


Reinout


--
Reinout van Rees http://reinout.vanrees.org/
rei...@vanrees.org http://www.nelen-schuurmans.nl/
"If you're not sure what to do, make something. -- Paul Graham"

[Reikje]

Excellent, thanks for sharing that blog post.

Are you using a setup.py file with buildout? Currently I am not using one. My buildout.cfg lists all the dependencies and I checkout my source code from a specific git revision. Afterwards I go into the source code folder and add local files (i.e. production ready Django settings file).

Am I right that using setup.py and running bin/buildout.py setup would create an egg of my source folder and the egg is created in the develop-eggs folder? Using this approach I wouldn't be able to do local modifications correct, as everything is bundled within a egg file. (Let me add that I am new to Django/Python deployment :)

/Reik

[Reinout van Rees]

On 30-04-12 13:08, Reikje wrote:
> Are you using a setup.py file with buildout? Currently I am not using
> one. My buildout.cfg lists all the dependencies

Yes, I use a setup.py. For me, a setup.py is for listing hard
dependencies of the code, especially for packages you include in your
buildout. That way you can also reuse them and be sure that everything
you need is included.

The core point is that setup.py's are "recursive" where buildouts are not.