What does an ideal django workflow setup look like?

[Ken]

*I've Googled, but have not found any really organized and layman-
friendly overview, which is why I came here*

I know how to get python/django working on my computer (PC or Mac) and
have developed simple test apps using django's built-in dev server and
a mysql back-end. So I can get things to work and don't need help on
installing stuff.

But as a coding hobbyist, I don't know how to set up a pro workflow. I
don't know best practices on how to setup and maintain a dev and
production environment, how to efficiently publish to production, how
to integrate all that into a version control system (i.e., git),
basically anything that a pro coder at a startup would take for
granted as "the way to do things".

I understand there are many different ways and products to use to
setup a great workflow for developing in django, but would like to
hear how you or your startup team (or corporate dev group) does it.
Specifics would be amazing, as I need a little hand holding, i.e.
please cover anything and everything that I should know in order to
develop efficiently, robustly, and eventually collaboratively.

Basically, please explain it in a way that a layman can follow the
steps and setup a workflow without pulling his hair out. =P

[Max Countryman]

There was a great article on HN regarding project structure although now I can't seem to find it. Perhaps someone has it bookmarked?

> --
> You received this message because you are subscribed to the Google Groups "Django users" group.
> To post to this group, send email to django...@googlegroups.com.
> To unsubscribe from this group, send email to django-users...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/django-users?hl=en.
>

[Phil Gyford]

After reading a few articles on this kind of thing, I tried to figure
out a good, repeatable way to do some of this, and wrote up the whole
process here: http://www.gyford.com/phil/writing/2010/09/29/django-environment.php

It doesn't (yet) cover the best ways of integrating dev/production
servers. I'm only really getting started myself, so would welcome any
further suggestions for improvement.

Phil

> --
> You received this message because you are subscribed to the Google Groups "Django users" group.
> To post to this group, send email to django...@googlegroups.com.
> To unsubscribe from this group, send email to django-users...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/django-users?hl=en.
>
>

--
http://www.gyford.com/

[Brian Bouterse]

There are whole books devoted to this topic, which is more of an industry best practices question than a django question.  Personally I like this book on continuous integration.  If not that one try googling for "continuous integration"

Brian

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To post to this group, send email to django...@googlegroups.com.
To unsubscribe from this group, send email to django-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-users?hl=en.

--
Brian Bouterse
ITng Services

[Kenneth Gonsalves]

On Fri, 2010-10-22 at 08:02 -0700, Ken wrote:
> Basically, please explain it in a way that a layman can follow the
> steps and setup a workflow without pulling his hair out. =P

my method (I am not a professional programmer):

1. set up my code and create a mercurial repository. Test each change on
my local machine - I use nginx+runserver.

2. once the code is working - push the changes to a repository on the
internet (I use bitbucket)

3. from the repository pull the code to the staging server for the
client to test with real data (usually this data would be taken from the
production server)

4. once the client is happy - pull the code to the production server -
usually apache+wsgi or nginx+fcgi or tornado.

5. for changes in the database tables: run manage.py sqlall and save it
as before.sql. make the changes, again run sqlall and save it as
after.sql. Create a diff between the two and massage that into a script.
Test the script on the development machine and then on the staging
server, finally do it on the production server.

5. Note: settings.py is not put under version control as this holds
sensitive information and paths differ according to the server it is on.
I create a default.settings.py which has the sensitive information
removed. That is kept under version control.
--
regards
Kenneth Gonsalves
Senior Associate
NRC-FOSS at AU-KBC

[shacker]

On Oct 22, 10:09 pm, Kenneth Gonsalves <law...@au-kbc.org> wrote:

> 5. Note: settings.py is not put under version control as this holds
> sensitive information and paths differ according to the server it is on.
> I create a default.settings.py which has the sensitive information
> removed. That is kept under version control.

Another approach to this problem:

settings.py IS in version control and includes settings that are
universal to all environments (local dev, staging, production), but
has dummy info or empty strings for passwords or paths or other
information that vary between environments.

local_settings.py is NOT in version control, and redefines any
passwords or settings specific to the various environments with actual
values.

Technically, it should not be necessary to define dummy values in
settings.py first, but we've seen issues where things break if you
don't set them first and then override them. No idea why.

The following bit at the end of settings.py looks for and reads in
info from local_settings.


##############################
# Default settings go above this line.
# This allows local settings to override.
##############################
try:
from local_settings import *
except ImportError, exp:
pass

[Phlip]

> I know how to get python/django working on my computer (PC or Mac) and
> have developed simple test apps using django's built-in dev server and
> a mysql back-end. So I can get things to work and don't need help on
> installing stuff.

I take all the other advice in these threads (beginning with
relentless automated tests), and I put them into fabfile.py.

Programmers should consider command lines like 'python manage.py
syncdb' and 'git push origin master' as just components of a kit. We
have no reason to type those command lines over and over again, with
minor variations each time. We should use the kit to assemble a
programmer-friendly environment.

Here's a Fabric fabfile.py example:

def sh(cmd): local(cmd, capture=False)

def _manage(command, flavor, extra=''):
sh('python manage.py %s --settings=settings.%s %s' % (command,
flavor, extra))

def test(): _manage('test', 'test' , '--verbosity=0')
def shell(): _manage('shell', 'local')

def run():
_manage('syncdb', 'local')
_manage('loaddata', 'local', 'sample_database') # CONSIDER These
data should differ from the tests.py data.
_manage('runserver', 'local')

I also move the clutter of test_settings.py, local_settings.py, etc to
a folder settings/, contain test.py, common_settings.py, local.py,
etc.

That fab file allows for short command lines:

fab run # build a sample database from your test fixtures & launch
a server

fab shell # use iPython for the best shell.

fab ci:'what I did' # run all the tests then integrate

Also, read the book "Release It!", if you think you know how to battle-
harden a website for production!

--
Phlip
http://zeekland.zeroplayer.com/

[Kenneth Gonsalves]

On Sat, 2010-10-23 at 08:41 -0700, shacker wrote:
> On Oct 22, 10:09 pm, Kenneth Gonsalves <law...@au-kbc.org> wrote:
>
> > 5. Note: settings.py is not put under version control as this holds
> > sensitive information and paths differ according to the server it is
> on.
> > I create a default.settings.py which has the sensitive information
> > removed. That is kept under version control.
>
> Another approach to this problem:
>
> settings.py IS in version control and includes settings that are
> universal to all environments (local dev, staging, production), but
> has dummy info or empty strings for passwords or paths or other
> information that vary between environments.

this is a better approach - much more professional (I *did* say I am an
amateur;-))

[Steve Holden]

On 10/25/2010 2:02 AM, Kenneth Gonsalves wrote:
[shacker]

>> Another approach to this problem:
>>
>> settings.py IS in version control and includes settings
>> that are universal to all environments (local dev, staging,
>> production), but has dummy info or empty strings for
>> passwords or paths or other information that vary between
>> environments.
> this is a better approach - much more professional (I *did* say I
> am an amateur;-))

A technique I use, having picked it up from Jacob Kaplan Moss, IIRC, is
to keep settings.py under source control but have it include as its last
statement

from local_settings import *

The local_settings.py file is then maintained outside source control on
each target platform. This works very nicely, and allows you to maintain
common settings without having to edit a file that's under code control
to vary settings between platforms.

regards
Steve
--
DjangoCon US 2010 September 7-9 http://djangocon.us/

[Tom Evans]

When this came up another time, a better approach was explained to me.
Rather than settings.py 'under version control'*, you keep
settings_default.py as your universal settings file. This is then
imported into settings.py.

The advantage of this is that you don't need to do clever tricks in
your settings.py to update things like INSTALLED_APPS,
MIDDLEWARE_CLASSES etc.

Having said that, I still import from setting_local in all my projects.

Cheers

Tom

* This is such a misnomer - *all* your source and configuration files
should be under version control, especially local settings (I keep
mine in a separate repository, which is then externall'ed into place
in the instance's deployment repo). How else can you effectively
control change management?

[Jumpfroggy]

This is such a great thread! I'd love to see a wiki page detailing
some good setup ideas. It's such a leap forward from the nuts & bolts
of "How do I install django" and "How do I run a complex query" to
this... "How do I use django more effectively, end to end?"

I agree - most of these tips will be generic to project dev &
deployment, but it seems there are a few django specific ones.

My thoughts...

- Use VCS. First step. I use SVN, but others are probably better.
Even if you're a solo dev, even for small projects, it's worth it.

- Definitely have local, test, and prod servers. Local is where the
development happens (on your machine). Testing is where you can test
the latest commits. Ideally you'd have continuous integration as
well, which would also run on the latest revision. Prod is the live
site. If you're a single dev on a small project, you can get away
with local & prod.

- +1 on using "settings.py" and "local_settings.py". The problem with
keeping local settings files in the VCS is when you have multiple
servers with different settings. You could store each file as
local_settings_test_server.py and so on. Then you could have
something like "settings_server_name.py" that's not in VCS, and it
only has:

SERVER_NAME = 'prod'

And in your settings.py:

import * from settings_server_name

if SERVER_NAME == 'prod':
import * from settings_prod_server
elif SERVER_NAME == 'test':
...
etc

That way almost everything is in VCS, and you only have one file to
write for each server.

- Try to develop & test on servers similar to your PROD server. It
stinks to develop and test, then realize you had python X and PROD
server is python X - 0.1, which means some features/libraries aren't
available (or syntax is wrong).

- Use "south" for database changes! It takes just a bit to get used
to, but will make database changes (both schema and data) so much
easier to keep track of. This is basically what Kenneth Gonsalves
described, but in reliable/convenient/automated form. It's amazing.
Go south! Use south! (don't think about dmigrations. It does not
exist.)

- Use some kind of deployment script. I wrote my own and it's simple
and awesome. You can also use something like fabric, which seems
complicated but is very useful. Add things like "svn update force" to
update a repsitory, "touch *.wsgi" to reload the app, "python
manage.py migrate" to update any DB changes (using south), etc.
Anything you'd do on the server, create scripts for. Even stuff like
"copy PROD DB to TEST server", so you have real data to play with.

That's what I can think of at the moment. It'd be nice to have a
"best practices" page on the django site, since a lot of these
suggestions will be repeated for most users.

[Brian Bouterse]

I really can't encourage the use of Hudson (or something like it).  It's a great tool to plug scripts into, and gives you added features if configured properly such as running test cases automatically, giving code coverage reports, pylint reports, and if your build "succeeds" can even roll out your successful build to the staging servers.

I like to tie our VCS using git-hooks to kick off automated builds of Hudson with each commit.

In turn, your staging servers could run some smoke tests and then do a script based automated roll-out using fabric as suggested.  +1 for fabric

+1 on the use of South

Chaining all of these tools together bring together one powerful development pipeline, but even using them all independently is still very powerful (and the first step).

my 2 cents,

Brian

--

You received this message because you are subscribed to the Google Groups "Django users" group.
To post to this group, send email to django...@googlegroups.com.
To unsubscribe from this group, send email to django-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-users?hl=en.

[Kenneth Gonsalves]

On Mon, 2010-10-25 at 16:06 -0400, Brian Bouterse wrote:
> I really can't encourage the use of Hudson <http://hudson-ci.org/> (or
> something like it).

'cant encourage'? or 'can encourage'?

[Tom Evans]

On Mon, Oct 25, 2010 at 7:04 PM, Jumpfroggy <rocket...@gmail.com> wrote:
> This is such a great thread!  I'd love to see a wiki page detailing
> some good setup ideas.  It's such a leap forward from the nuts & bolts
> of "How do I install django" and "How do I run a complex query" to
> this... "How do I use django more effectively, end to end?"
>
> I agree - most of these tips will be generic to project dev &
> deployment, but it seems there are a few django specific ones.
>
> My thoughts...
>
> - Use VCS.  First step.  I use SVN, but others are probably better.
> Even if you're a solo dev, even for small projects, it's worth it.
>

Others are 'cooler'. I'm not sufficiently versed in git or mercurial
to comment on whether they can handle the structures I'll describe
below..

>
> - +1 on using "settings.py" and "local_settings.py".  The problem with
> keeping local settings files in the VCS is when you have multiple
> servers with different settings.  You could store each file as
> local_settings_test_server.py and so on.  Then you could have
> something like "settings_server_name.py" that's not in VCS, and it
> only has:
>
>    SERVER_NAME = 'prod'
>
> And in your settings.py:
>
>    import * from settings_server_name
>
>    if SERVER_NAME == 'prod':
>        import * from settings_prod_server
>    elif SERVER_NAME == 'test':
>        ...
>        etc
>

Bit clunky isn't it? Every time you add/remove an instance of the
project, you have to update settings.py.

This is how I structure my projects in the svn repository (hope you
can all see unicode line segments!):

sample_project
├── configurations
│   ├── development-requirements.pip
│   ├── production-requirements.pip
│   ├── settings_local-app01.py
│   ├── settings_local-app02.py
│   ├── settings_local-dev-app01.py
│   └── settings_local-dev-app02.py
├── deployments
│   ├── app01
│   │   ├── logs
│   │   └── scripts
│   ├── app02
│   │   ├── logs
│   │   └── scripts
│   ├── dev-app01
│   │   ├── logs
│   │   └── scripts
│   └── dev-app02
│   ├── logs
│   └── scripts
├── scripts
│   ├── bootstrap
│   └── project
└── src
├── releases
│   ├── 1.0
│   │   └── htdocs
│   ├── 1.1
│   │   └── htdocs
│   └── 1.2
│   └── htdocs
└── trunk
└── htdocs

Each deployed instance of the project gets a folder in
sample_project/deployments. The deployment then uses svn externals to
include the parts of the project it requires:

src/<some branch> is externalled into dep/sample_project
configurations/settings_local-<instance>.py is file externalled into
dep/sample_project/settings_local.py
configurations/<one of the pip files> is file externalled into
dep/scripts/requirements.pip
scripts/{bootstrap,project} are file externalled into dep/scripts/

You end up with a structure like this:

instance
├── environ
│   ├── bin
│   ├── include
│   └── lib
├── htdocs
├── logs
├── run
├── sample_project
│   ├── __init__.py
│   ├── settings.py
│   ├── settings_local.py
│   └── urls.py
└── scripts
├── bootstrap
├── project
└── requirements.pip

The 'bootstrap' script is a simple script that sets up the virtualenv
and installs/upgrades all packages to the specification specified in
requirements.pip, using pip.
The 'project' script is a rc script for plugging into the OS's rc
infrastructure.

Every single file is tracked by VCS, every change in the project
structure is versioned and controlled for change management processes.
Changes to a production environment can be reliably tested, checked
and repeated.
New instances simply require a new settings_local-<instance>.py
created in the configuration repo, new packages can be tried in
development and it is simple and repeatable to deploy to production.

There are a number of other tools that I use to do change management
(Fabric and south mainly), but it would take too long to go into those
as well!

Cheers

Tom

[Brian Bouterse]

I can't encourage it *enough*.  This was writen hastily, thanks for the close read.

Brian

--

You received this message because you are subscribed to the Google Groups "Django users" group.
To post to this group, send email to django...@googlegroups.com.
To unsubscribe from this group, send email to django-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-users?hl=en.

[David De La Harpe Golden]

On 25/10/10 19:04, Jumpfroggy wrote:

> - +1 on using "settings.py" and "local_settings.py". The problem with
> keeping local settings files in the VCS is when you have multiple
> servers with different settings. You could store each file as
> local_settings_test_server.py and so on.

We use a proj.settings along the lines of:

from proj.common_settings import *
import imp
imp.load_source('proj_local_settings', '/etc/proj/settings.py')
from proj_local_settings import *

to keep our host-specific settings outside VCS on each host. A range of
more complex schemes are possible, of course.

[Celso González]

On Fri, Oct 22, 2010 at 08:02:56AM -0700, Ken wrote:

Hi

> I understand there are many different ways and products to use to
> setup a great workflow for developing in django, but would like to
> hear how you or your startup team (or corporate dev group) does it.
> Specifics would be amazing, as I need a little hand holding, i.e.
> please cover anything and everything that I should know in order to
> develop efficiently, robustly, and eventually collaboratively.

There are common ideas like +1 South, local_settings.py, vcs and
helper scripts like fabric

> Basically, please explain it in a way that a layman can follow the
> steps and setup a workflow without pulling his hair out. =P

ok, my system

Every project has its own virtual enviroment

-virtualenv
--bin
--include
--lib
--requirements.txt
--myproject
---apps
...
---static
---templates

*All the structure and contents goes into git or vcs of choice
*pip freeze > requirements.txt and pip install -r requirements.txt
to handle the external soft installed
*in .gitignore I override bin, include, lib, src directories

Just using git and virtualenv im able to create the same enviroment
quickly in several machines (dev, testing, production)

Still working on the database exports

--
Celso Gonz�lez (PerroVerd)
http://mitago.net

[Ken]

Ok -

I put together more research. It's a huge pile of links. Hope this
helps as additional places to learn/look. As a disclaimer, I'm a noob,
which is why I am providing Internet research instead of a personal
expert opinion.

------Simon Willison (co-Founder of Django) answered the question at
Quora.com------

Short answer: virtualenv, pip, south for migrations, fabric for
deployment.

Long answer: I haven't seen the perfect tutorial covering all of the
above yet, which is a shame, but here are some good links:

http://morethanseven.net/2009/07/27/fabric-django-git-apache-mod_wsgi-virtualenv-and-p.html

http://www.saltycrane.com/blog/2009/05/notes-using-pip-and-virtualenv-django/

http://www.clemesha.org/blog/modern-python-hacker-tools-virtualenv-fabric-pip

http://www.caktusgroup.com/blog/2010/04/22/basic-django-deployment-with-virtualenv-fabric-pip-and-rsync/

-----------Hacker News threads------------

http://news.ycombinator.com/item?id=1824171

http://news.ycombinator.com/item?id=1827160

http://news.ycombinator.com/item?id=135829

The next few are more git-focused.

http://news.ycombinator.com/item?id=1343753

http://news.ycombinator.com/item?id=441670

----------Old Reddit threads------------

http://www.reddit.com/r/django/comments/dpnfr/fabric_django_git_apache_mod_wsgi_virtualenv_and/

http://www.reddit.com/r/django/comments/95oeo/django_development_workflow/

http://www.reddit.com/r/programming/comments/a5j8b/mozilla_addons_team_switching_to_django_git_and/