Django deployment practices -- do people use setup.py?

[Tom Eastman]

Hey guys,

I'm looking for deployment best practices for Django projects. Google
searches seem to show countless numbers of them, many of them somewhat
contradictory :-)

So as a simple discussion point: I'm curious to know if lots of people
use setup.py to deploy a django project? That'd mean setup.py would
install your python packages to somewhere on sys.path, maybe in your
python site-packages or maybe in a virtualenv.

Is this considered a common/recommended practice? If so -- where do you
put your settings files?

Maybe you want settings files in /etc/myproject? and add that to
sys.path? Do people do this?

Or do people skip the setup.py thing, and just check out their source to
some directory and add that directory to sys.path?

What I'm doing at the moment:

- Fabric command sets up apache and directories
- Another one uses setup.py to create an archive, delivers the archive
and runs setup.py install etc.

Is this silly?

What are your thoughts?

--
Tom Eastman // Catalyst IT Ltd. // +64 4 803 2432

[Thomas Guettler]

Hey,

I read your post some days ago and waited what other people say. But
nobody answered.

I posted a related question some time ago: Staging (dev,test,prod) in django. I explain
my setup there.
http://markmail.org/thread/wqwvordnlhyizwyp

A related thread on django-dev:
http://groups.google.com/group/django-developers/browse_thread/thread/d919da361273dbc0/8de3c5cf9369eca3?#8de3c5cf9369eca3

I think it would be very good to have the fundamental parts in django:
* stages: DEV, TEST, PROD
* get next system: user, host, install-prefix

Several deployment implementation could use these methods and variables. Switching
between a deployment implementation would be easier.

Thomas

--
Thomas Guettler, http://www.thomas-guettler.de/
E-Mail: guettli (*) thomas-guettler + de

[bruno desthuilliers]

On Mar 12, 11:55 pm, Tom Eastman <t...@catalyst.net.nz> wrote:
> Hey guys,
>
> I'm looking for deployment best practices for Django projects. Google
> searches seem to show countless numbers of them, many of them somewhat
> contradictory :-)
>
> So as a simple discussion point: I'm curious to know if lots of people
> use setup.py to deploy a django project?

We dont - we use virtualenvs (no-site-packages), pip requirements, and
git.

[Benedict Verheyen]

Same setup here.
I only use setup.py for creating pure Python packages, not Django packages.

Cheers,
Benedict

[Andrew Cutler]

On 15 March 2012 02:40, bruno desthuilliers

<bruno.des...@gmail.com> wrote:
> We dont - we use virtualenvs (no-site-packages), pip requirements, and
> git.

We do the same here. Our deployment looks something like

# mkvirtualenv --no-site-packages foo
# workon foo
# cdvirtualenv
# svn checkout bar
# pip install -r deployment/pip-reqs.txt

It works. But it's ugly. And the pip-reqs are a pain to maintain.

Now I've got almost the same requirements as the OP for a new project.
We want to be able to distribute our Django 'app' as a standalone
module. This is pretty easy to do with a distutils and a well
constructed setup.py.

We also want to be able to distribute a self contained Django project
with sane defaults (and this is where it gets tricky).

In total we're looking at two packages:

dms_django (Django Project + base settings)
dms_core (Django app, and required packages)

So for a quick install we can do something like

# mkvirtualenv --no-site-packages dms
# workon dms
# cdvirtualenv
# pip install svn+https://mysvnrepo/dms_django/ # Install the
dms_django package and pull in dms_core as a requirement

... sit back and everything is taken care of. (well that's the idea)

However packaging a Django "project" using distutils is not straight forward:

Issues:

The default Django manage.py, assumes that settings.py is in the same
directory. So if we mark this as a script (in setup.py) so it gets
installed into ./bin and it wont work, unless we customise it first.

And where do we put local settings? Can we have setup.py create a
local project directory, with any local configs, user template
overrides. etc?

I'm hoping someone has already done this before and can advise on what
works and what doesn't...

Cheers, Andrew

[Matt Schinckel]

I don't use a setup.py for my _project_ deployment, but I do have a setup.py for each of the apps that I use within a project (or at least those that are 'reusable')

For project deployment, I use a fabfile that does the following:

* installs public keys onto the server (if necessary)

* creates the directory structure required (if necessary)

* copies the project onto the server

* installs requirements from REQUIREMENTS.txt

* runs collectstatic, migrate, etc

* restarts the web server (apache/nginx/whatever).

So, I can do:

$ fab deploy --host production.server

I can do each of these parts separately, if necessary, or the whole lot. I also have a wrapper around django-admin.py:

$ fab django:collectstatic --host server

And one around web server restart:

$ fab apache:graceful --host server

Matt.

[Tom Eastman]

On 14/03/12 21:47, Thomas Guettler wrote:
> Hey,
>
> I read your post some days ago and waited what other people say. But
> nobody answered.
>
> I posted a related question some time ago: Staging (dev,test,prod) in
> django. I explain
> my setup there.
> http://markmail.org/thread/wqwvordnlhyizwyp
>
> A related thread on django-dev:
> http://groups.google.com/group/django-developers/browse_thread/thread/d919da361273dbc0/8de3c5cf9369eca3?#8de3c5cf9369eca3
>
>
> I think it would be very good to have the fundamental parts in django:
> * stages: DEV, TEST, PROD
> * get next system: user, host, install-prefix
>
> Several deployment implementation could use these methods and variables.
> Switching
> between a deployment implementation would be easier.
>
> Thomas

Hi Thomas,

Yeah, I didn't get a lot of feedback -- but in the intervening time I've
been using zc.buildout with two of my projects and I've quite liked the
way it does things. Doesn't solve all my problems -- but it does seem to
do a nice job of combining what using virtualenv and setup.py gets you.

Cheers,

Tom

[John Griessen]

On 03/14/2012 09:13 PM, Matt Schinckel wrote:
> For project deployment, I use a fabfile that does the following:
>
> * installs public keys onto the server (if necessary)
> * creates the directory structure required (if necessary)
> * copies the project onto the server
> * installs requirements from REQUIREMENTS.txt
> * runs collectstatic, migrate, etc
> * restarts the web server (apache/nginx/whatever).

I hadn't thought of doing database migrates along with deploy
commands -- I'd thought they needed special handling

How much migrating of database info is reasonable to
script like this? Does anyone work with web storefronts
that do some of the inventory bookkeeping, and so they
have transaction info that cannot be lost, and how do you
force the django app to stop doing any new transactions
before you do a migrate? Do the usual Django web store modules
handle this idea already? Do any migrate scripts handle stopping
new transactions first?

John

[Andrew Cutler]

On 15 March 2012 12:33, Andrew Cutler <and...@adlibre.com.au> wrote:
>
> We also want to be able to distribute a self contained Django project
> with sane defaults (and this is where it gets tricky).
>

Answering my own question here... But I've finally achieved the "holy
grail" of Django project deployment using setup.py (setuptools) and
pip.

In a clean virtualenv... run one deployment command... sit back and done.

Here we have the deployment command for a Django timesheet system we've written:

# pip install git+git://github.com/adlibre/Adlibre-TMS.git

A (trivially) customised manage.py is put into ./adlibre_tms/ along
with a sample local_settings.py, leaving all the immutable source
files in ./lib/python/site-packages/adlibre_tms/. The way settings.py
is imported is a little magical, but so far it hangs together nicely.
Directories for media, static files and sqlite database are also
created.

Now... I probably wouldn't recommend this method unless you're planing
on re-distributing a Django project as a complete application. If
you're just deploying your own stuff then dealing with
distutils/setuptools is probably not worth the heartache and
hair-pulling.

But at least this shows that it is possible. It is also possible to
pull in your own package dependencies direct from github using
setuptools 'dependency_links' urls and github tarball links. (Which is
truly magical when it works)

Hopefully this will save someone else some time...

Cheers,
Andrew

[Daniel Sokolowski]

Hi Andrew,

Can you clarify if you approach can handle things that are not installable
through pip? For example I run into an issue with geodjango requirements - I
was not able to get everything installed through PIP and had to resort to
manually using the Debian package management.

Cheers,
Andrew

--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To post to this group, send email to django...@googlegroups.com.
To unsubscribe from this group, send email to
django-users...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

Daniel Sokolowski
Web Engineer
Danols Web Engineering
http://webdesign.danols.com/
Office: 613-817-6833
Fax: 613-817-4553
Toll Free: 1-855-5DANOLS
Kingston, ON K7L 1H3, Canada

Notice of Confidentiality:
The information transmitted is intended only for the person or entity to
which it is addressed and may contain confidential and/or privileged
material. Any review re-transmission dissemination or other use of or taking
of any action in reliance upon this information by persons or entities other
than the intended recipient is prohibited. If you received this in error
please contact the sender immediately by return electronic transmission and
then immediately delete this transmission including all attachments without
copying distributing or disclosing same.

[Andrew Cutler]

On 18 April 2012 23:21, Daniel Sokolowski <daniel.s...@klinsight.com> wrote:

Can you clarify if you approach can handle things that are not installable through pip? For example I run into an issue with geodjango requirements - I was not able to get everything installed through PIP and had to resort to manually using the Debian package management.

Do you mean install requirements for system libraries, databases. That sort of thing? Or do you mean non python files, like scripts, documentation ?

Cheers, Andrew

[Daniel Sokolowski]

Both, system libraries and not python files --- anything outside of Python land that PIP can’t handle. How do you handle that if you want a self contained easy to deploy project?

 

From: Andrew Cutler

Sent: Saturday, April 21, 2012 12:58 AM

To: django...@googlegroups.com

Subject: Re: Django deployment practices -- do people use setup.py?

 

[Andrew Cutler]

On 24 April 2012 02:21, Daniel Sokolowski

<daniel.s...@klinsight.com> wrote:
>
> Both, system libraries and not python files --- anything outside of Python
> land that PIP can’t handle. How do you handle that if you want a self
> contained easy to deploy project?

Well then it's safe to assume that if it's not pure Python, or
directly related to your project then it's out of scope for standard
Python packaging tools.

Python runs just about everywhere, and every platform handles
libraries and app dependencies differently. I don't think that an
elegant, and general cross-platform solution exists. Distutils does
however provide a general method for installing Python (C) extension
modules in a cross platform fashion and there is also a 'data_files'
option for installing non Python files within your virtualenv but this
is as far as it goes.

For Windows you'll need to create an EXE installer or MSI package with
every redistributable that you need (eg Apache/MySQL... oh and
Python). For *nix platforms use the relevant package format eg
RPM/DEB/DMG to install whatever your app depends on, whether it be
MySQL, or some Python module or system library.

The way we've solved it with our Django apps is quite simple. We have
a README file that explains what the general requirements are, and how
to install these requirements for the most popular Linux distros (eg
yum install foo). Our setup.py takes care of everything Python. It's
up to the end user/sysadmin to plumb everything together. Perhaps
later on I'll roll some generic Debs and RPMS that satisfy the
required dependencies, but for now we assume our end users are
technically proficient, so we can get away with a less integrated
approach.

Cheers,

--
Andrew Cutler

[Daniel Sokolowski]

Thank you for taking the time to explain; I've reached a similar 'readme'
approach here but was hoping for a better solution.

-----Original Message-----
From: Andrew Cutler

Sent: Tuesday, April 24, 2012 8:21 AM
To: django...@googlegroups.com
Subject: Re: Django deployment practices -- do people use setup.py?

On 24 April 2012 02:21, Daniel Sokolowski
<daniel.s...@klinsight.com> wrote:
>
> Both, system libraries and not python files --- anything outside of Python

> land that PIP can�t handle. How do you handle that if you want a self

[pritesh modi]

Hello Andrew

i am looking for same actually can u provide me information about that. i think what i understand is here finally the .py code should not go at deployment server and only compile code is going at server and run at deployment server.

so can you provide me the information ?



thanks

On Wed, Apr 25, 2012 at 7:26 PM, Daniel Sokolowski <daniel.s...@klinsight.com> wrote:

Thank you for taking the time to explain; I've reached a similar 'readme' approach here but was hoping for a better solution.

-----Original Message----- From: Andrew Cutler
Sent: Tuesday, April 24, 2012 8:21 AM

To: django...@googlegroups.com
Subject: Re: Django deployment practices -- do people use setup.py?

On 24 April 2012 02:21, Daniel Sokolowski

<daniel.sokolowski@klinsight.com> wrote:

Both, system libraries and not python files --- anything outside of Python

land that PIP can’t handle. How do you handle that if you want a self

To unsubscribe from this group, send email to django-users+unsubscribe@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/django-users?hl=en.

Daniel Sokolowski
Web Engineer
Danols Web Engineering
http://webdesign.danols.com/

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To post to this group, send email to django...@googlegroups.com.

To unsubscribe from this group, send email to django-users+unsubscribe@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/django-users?hl=en.

--
Pritesh  Modi

 

[Andrew Cutler]

On 27 April 2012 14:27, pritesh modi <pritesh...@gmail.com> wrote:
>
> i am looking for same actually can u provide me information about that. i think what i understand is here finally the .py code should not go at deployment server and only compile code is going at server and run at deployment server.
>

I think this can be achieved in two or three ways.

Distutils, and some hacking as suggested here:

http://bytes.com/topic/python/answers/42434-distutils-binary-distribution

Or using setup tools and a rolling a binary egg:

http://python.6.n6.nabble.com/Distributing-only-pyc-td2010644.html

Alternatively you can rolling your own package / deployment
toolchain... this really would depend on your target platform. This
would probably make sense if you're targeting windows and want to
bundle all your requirements.

I really can't comment on which is the preferred approach, as I don't
have any experience with binary only packaging (all our distributed
code is open source). But these at least should give you a starting
point for your own investigation.

Regards,

--
Andrew Cutler

[Simon Bächler]

Hi

We have been using git and git submodules and just started using virtualenv and pip. 

Submodules works well but you need a git repo of your packages. I wrote a blog post on using them: http://www.feinheit.ch/blog/2012/04/18/using-git-submodules/

Now we use pip for working packages like south or feincms. I still use submodules for packages still in development.

Regards

Simon

[Daniel Sokolowski]

Hi Simon,

 

I read your blog and am wondering if you considered the ‘pip install –e’ (edit) option of pip? I use it and it does what you are trying achieve with git submodules.

 

For example:

    $ pip install -e git+git://github.com/danielsokolowski/django-chunks.git#egg=danols-django-chunks

 

Would install django-chunks from my repo under `virtualenv/src/danols-django-chunks` and it would be a full svn repo (or git or hg) that you can push pull from.

You can also add @commit_num to install a specific commit; the benefit of this is that you can place this in your projects `requirements.txt` file as is.

 

If you I don’t include the `-e` option then it would install to `virtualenv/lib/python/site-packages/`.  If you did consider the `–e` option but choose not to use it

could you sum up why you find your approach better?

 

In the end I have a mix of official packages, git repos, and project specific code split up in a directory structure like so (I feel like sharing ) :

 

    ./

    ./raw_media – ras project assets

    ./src

    ./src/django-project         – the client website, the django projects

    ./src/django-guardian     – an example of an official app modified just to this specific site but not backward compatible or worthy to be pushed back to an official repo

    ./virtualenv

    ./virtualenv/[...]         – virtual env stuff including stuff install with just ‘pip install’

    ./virtualenv/src/        – repos installed with ‘pip install –e’ options that I can edit and push back

    .project                      – eclipse IDE file

    .pydevproject             – eclipse IDE file

    apache-conf.httpd     – apache conf that I symlink to

    django-project.wsgi     – mod_wsgi setting file

    readme.txt

    requirements.readme.txt

    requirements.txt

 

 

 

From: Simon Bächler

Sent: Friday, April 27, 2012 3:19 AM

To: django...@googlegroups.com

Subject: Re: Django deployment practices -- do people use setup.py?

 

--
You received this message because you are subscribed to the Google Groups "Django users" group.

To view this discussion on the web visit https://groups.google.com/d/msg/django-users/-/UsHUTCQQwEwJ.

To post to this group, send email to django...@googlegroups.com.

To unsubscribe from this group, send email to django-users...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/django-users?hl=en.