How to update a live / deployed Django project?

[Tobias Dacoir]

We've build a small web application using Django 1.8 over the past couple of months and we are going live with version 1.0 soon (one last closed beta test is coming up).
So far we have deployed it on an Ubuntu VM (in Azure) using Apache / Nginx and a local MySQL Server.

I've already found some good tutorials on how to properly set it up (http://rogueleaderr.com/post/65157477648/the-idiomatic-guide-to-deploying-django-in#disqus_thread ) however what I am mising now is how to handle updates to our code.

For our development server we have a git hook that automatically deploys new commits to the server. However that often requires manual interaction - running manage.py makemigrations etc. Sometimes Django also does not pick up our changed models and we have to manually fiddle with the Database in order to get it back running. We even had times where we lost all our data since it was easier to just start with a new database.

Obviously this can't happen once we are live and obviously we want downtime to be as small as possible. So is there any guide on how to update code of a live, deployed Django application?

[James Schneider]

On Feb 10, 2016 12:50 AM, "Tobias Dacoir" <fal...@gmail.com> wrote:
>
> We've build a small web application using Django 1.8 over the past couple of months and we are going live with version 1.0 soon (one last closed beta test is coming up).
> So far we have deployed it on an Ubuntu VM (in Azure) using Apache / Nginx and a local MySQL Server.
>
> I've already found some good tutorials on how to properly set it up (http://rogueleaderr.com/post/65157477648/the-idiomatic-guide-to-deploying-django-in#disqus_thread ) however what I am mising now is how to handle updates to our code.
>
> For our development server we have a git hook that automatically deploys new commits to the server. However that often requires manual interaction - running manage.py makemigrations etc. Sometimes Django also does not pick up our changed models and we have to manually fiddle with the Database in order to get it back running. We even had times where we lost all our data since it was easier to just start with a new database.
>

You need to fix your deployment process before going live. What are you doing where Django doesn't catch changed models? That's a red flag for me. Either you're doing something strange with your models, your code is not updating the way you think it is, or you've uncovered a bug in Django and should report it. You shouldn't need to run 'makemigrations' anywhere but in your development environment. Are your migrations carried through as part of your Git hook update? It sounds like they aren't. If your development servers are seeing database changes, your production servers should probably get those same changes in most cases. Squashing migrations should be performed in development before the final updated deployment package is out together and then sent to production after testing.

I'd recommend setting up test servers which are clones of your production servers (maybe even including a snapshot of your live data). Your changes on your development servers should be tested on the test servers, including the deployment process. Git alone may not be enough to complete a deployment (probably not a lot of control in a failure scenario), but something like Fabric may help. There are lots of ways to deploy though. Many use mass configuration managers like CFEngine, Puppet, Chef, and Ansible.

> Obviously this can't happen once we are live and obviously we want downtime to be as small as possible. So is there any guide on how to update code of a live, deployed Django application?
>

Depends on the available infrastructure. For a single server, you'll incur downtime for shutting down the web service, applying migrations, and bringing things back online. If you have a load balancer in use, you may be able to pull out a single real server and update it, and put it back with little or no interruption. Obviously you'll have to account for existing session and shared databases among servers in that case.

If your database changes are something simple like adding columns or tables that aren't referenced in existing code, you may be able to perform the migration live, then update the code, and reload the web server process, incurring minimal, if any, downtime. Really depends on what your DB changes are. I would hope that with a 1.0 non-beta release that your DB structure would remain stable, though.

Test, test, test!

Backup, backup, backup!

-James

[Tobias Dacoir]

Thanks for the reply.

Thanks for the advice. We should include the migrate command into our git hook as well. But like I said, sometimes it's not always possible as manual interaction is required. At least for 1.0 our model should not change much, except for future extensions and new models.

We also have a development / test server that is a direct clone of the future live server. However I still don't have the workflow laid out. Should we use KISS and just upload a zip of our new version and just replace the whole folder on the live server with the new code and issue an Apache restart afterwards (not even sure that this is needed in order to pick up the new code)?

[Andreas Kuhne]

Hi,

What I tend to do when deploying is:

1. Create a fabric script that contains a reference to the server that should be updated (defaults to dev server).

2. Use git archive to zip the contents of a git tag to a local file.

3. Upload the file to the server

4. Stop the uwsgi process

5. Update files

6. Run migrate, collectstatic

7. Start uwsgi process

8. Delete archive file from server.

9. Done!

I really don't see when a migration would need manual updates? If that is happening, you are probably doing migrations wrong - as someone pointed out earlier. We are running several production workloads with this configuration and have never needed to manually apply any migrations - except when migrating the database from postgres to mysql. 

I like the deployment process to be as automated as possible, and usually not need any manual "help" to get it working (even to the point of being deployed from a CI server automatically). 

Regards,

Andréas

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/a8ae9869-83cb-4023-a884-91ebf8190724%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Tobias Dacoir]

Thanks for sharing your workflow. Sounds simple enough and is more or less a scripted version of what I had in mind all along.

About the manual intervention: At some points during development we had to provide a one-time manual entry when Django was migrating the database, either by adding or removing properties to existing models. This shouldn't happen after we go live with 1.0 - hopefully. We always have ideas for new features to be build in :)

[Andreas Kuhne]

The thing is, you should be able to catch those issues when migrating locally on you development machines. In that case, just fix the migration and then it should work perfectly in production. You can always add certain values in the migration scripts themselves.

Regards,

Andréas

2016-02-10 14:51 GMT+01:00 Tobias Dacoir <fal...@gmail.com>:

Thanks for sharing your workflow. Sounds simple enough and is more or less a scripted version of what I had in mind all along.

About the manual intervention: At some points during development we had to provide a one-time manual entry when Django was migrating the database, either by adding or removing properties to existing models. This shouldn't happen after we go live with 1.0 - hopefully. We always have ideas for new features to be build in :)

--

You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/2566cd3d-478b-4276-aba7-3917141ae104%40googlegroups.com.

[Radek Svarz]

Tobias,

I know I will not help you directly, but maybe my setup + process can help someone else. I am deploying towards Redhat Openshift 2 PaaS.

I was in the similar situation choosing the best fit option for continuous deployment.

Final process:

- Merging the feature code to the master branch and pushing to github.

- Codeship autograbs the code from github, does CI tests.

- If tests pass it pushes to Openshift with hot_deploy setup (i.e. the old version is still running during the deploy phase). 

- Once the new version is on Openshift incl. all updated requirements it restarts the WSGI process (= very short downtime) and the new version is up.

- TODO some automated smoke tests towards Openshift production

The whole thing lasts a few minutes totally on the background, I am only notified of the success / failure.

And I can make some hot fixes through github's web interface :)

note: fabric is not compatible with Python 3, so its nogo for me. 

Radek

[Vernon D. Cole]

I have been through several variations on what you are doing.  The solution which has worked best for me is to use SaltStack to control the update.  It will pull your updates using git, apply as needed, and restart your servers. If you are building a new server (for testing or prototype usually) it will install everything. The last time I needed to test a new version of django it took about two hours to build a whole new prototype server on a virtual machine on my laptop.  Version updates on a running server take an eyeblink.

I put an example up on github/salt_demo. It needs some work since two years ago, but should give you the idea. It will run standalone, but I really recommend setting up a Salt master, even if you only have a few machines.  It can can really save you -- for example, I have had times when an ssh daemon quit working, and I was able to regain control using Salt.  The demo uses PostgreSQL -- I will send you my mariadb state if you wish.
--
Vernon Cole

[Andrew Farrell]

> I have been through several variations on what you are doing.  The solution which has worked best for me is to use SaltStack to control the update. 

If you are interested in learning SaltStack, I just recently wrote a step-by-step tutorial on it, targeted for django developers. It walks you through writing a configuration to deploy a django site (with postgres, gunicorn, and nginx) on a local virtual machine and then VPS as well as one way of writing automated tests for the configuration.

If you have feedback on the tutorial or suggestions on how to improve the learner experience, please let me know.

cheers,

Andrew

--

You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/3aeba092-2834-455e-a5e8-0c15be79d245%40googlegroups.com.

[Tobias Dacoir]

Thanks for all the great input. I'll definetely look into salt too (only worked with puppet so far) but for now I guess fabric will be fine. We are using Python 2.7 everywhere anyway.

Like I said, right now we plan to host it on a single physical machine that we get from the university (it's a research project). But I see that nobody is using the Apache / WSGI solution that is explained in Django's documentation. So I'll also have to look into nginx + gunicorn as well. Phew...seems like even more work than actually developing our webapp :)

[Rok Jaklič]

We are using two git branches, one for development and master. When dev branch is good enough, we merge it to master and then we have a script which does git pull and restarts uwsgi.

Rok

On Wednesday, February 10, 2016 at 9:50:22 AM UTC+1, Tobias Dacoir wrote: