Need advise for Django deployment on Amazon EC2 with Loadbalancers

[Martin]

Hi all,

has anyone on this list experience with deployments on the Amazon cloud?

Our stack is like this:

* Loadbalancer spawns AWI instances

* AWI instance has Django sites

* Each Django site hast the same PostgreSQL DB defined in their local_settings

* The PostgreSQL is it's own small instance

What is the best practice to do a deployment? The best I could come up with is the following:

1. Tell the loadbalancer to stop autoscaling
2. Kill all instances but one
3. git pull on that last instance, run syncdb, migrate, collectstatic, makemessages, rebuild_index, etc.
4. Deployment is done Kill fcgi processes on that last instance
5. Create a snapshot of that last instance
6. Tell the loadbalancer that new instances, from now on, can be created off of that new snapshot
7. Allow the loadbalancer to do autoscaling again

Am I missing anything? Does anyone have experience with this? Is there a better way to do it?

Many thanks in advance!

Kind regards,

Martin

[Calvin Cheng]

Hello Martin,

What you are asking for, is in essence, server automation (a.k.a "DevOps" processes) comprising 3 levels of scripting and configuration:-

(1) Server bootstrapping layer: where you might need to introduce some kernel specific features which virtual machines on EC2 or Linode do not have

(2) Application Service deployment layer:  Push automation, through the use of tools like Fabric (python) or Chef (ruby)

(3) Configuration Management tool layer: Indempotent-style tools like Puppet (ruby) and Bcfg2/kokki (python)

The complete implementation of the 3 layers is non-trivial but can be done (I am almost done with it and will be publishing a whole bunch of open source libraries relating to it).

In the 3rd layer specifically, you have a Master server instance, which Slave instances will clone from and get "launched" based on specific criteria you define in your script.

It's fun stuff. But takes up a lot of time to finish up.

[Martin]

Thanks Calvin! I'm aware of all this.

We will do as much as possible via Fabric. I have it all running (almost) but some parts are just so hackerish that it gives me the creeps, that's why I'm asking for advise :)

I can smell that there must be a simple, proven and elegant solution.

--
Chat: http://hackerspace.sg/chat

[Ciaran Lyons]

I just found out this week that there is a DevOps Singapore meetup group organised by Andy Marks of ThoughtWorks.  

Here it is:

http://www.meetup.com/devops-singapore/

Has anyone on the list been to any of their meetups?

Ciaran

CIARAN LYONS | PARTNER

NEWTON CIRCUS | 138 CECIL STREET, #01-01 CECIL COURT, SINGAPORE 069538

M +65 97956169 | CIARAN...@NEWTON-CIRCUS.COM | SKYPE CIARANLYONS-NEWTONCIRCUS

WWW.NEWTON-CIRCUS.COM

--
Chat: http://hackerspace.sg/chat

[Lc]

There isn't an 'elegant' solution.

Indempotent solutions like puppet and bcfg2 have their own warts, in their own particular ways.

Much like the early days of messy 'web frameworks', the state of 'server scaling frameworks' is currently in its infancy.

Web frameworks have grown and become mature over time - 8 to 10 years since the dot com boom/bust days.

Server scaling frameworks will mature in due course.

Sent from my iPhone

--
Chat: http://hackerspace.sg/chat

[Lin Chuan]

Hello Ciaran,

Didn't know about that.  

But yes, DevOps is indeed a topic of great interest to me, with the advent of commercial solutions like Google's AppEngine ("autoscaling" without a need for a Sysadmin), Heroku (ditto), Redhat's OpenShift, the open source OpenStack solution and the more traditional scaling "VPS" type solutions like puppet and bcfg2 or kokki.

I would like to see more open source code for "server scaling frameworks" grow and mature into the likes of django, rails etc. Which is why I released a bunch of code releasing to that and contributed a couple of fixes to bcfg2.

On a separate note...

I tried using meetup.com at one point in time but somehow did not find it useful and the "connections" not that relevant; so I stopped using it and unsubscribed.  

Facebook serves my needs (and my peers' needs very well). I don't see a reason to use meetup.com

Regards,

www.calvinx.com
Stalk me! @ www.twitter.com/calvinchengx

[Senthil Kumaran]

On Fri, May 18, 2012 at 04:09:02PM +0800, Martin wrote:
> What is the best practice to do a deployment? The best I could come
> up with is the following:

> 1. Tell the loadbalancer to stop autoscaling
> 2. Kill all instances but one
> 3. git pull on that last instance, run syncdb, migrate, collectstatic,
> makemessages, rebuild_index, etc.
> 4. Deployment is done Kill fcgi processes on that last instance
> 5. Create a snapshot of that last instance
> 6. Tell the loadbalancer that new instances, from now on, can be created off
> of that new snapshot
> 7. Allow the loadbalancer to do autoscaling again

>
> Am I missing anything? Does anyone have experience with this? Is there a better
> way to do it?

I would say, leave the specifics aside and concentrate on strategy
first because specifics vary greatly. I see that you are talking about
a load balancer different from what I have seen (I have seen - A10
doing DNS only load balancing, and custom written request based proxy
load balancing) and I am not sure what a Amazons Customer owned
load-balancing would be.

Speaking of general strategies, the first aim is to nullify downtime
when doing deploy. Which means that you will have one instance
running, while you are upgrading another and then roles will reverse
in order to complete the process.

In large scales, an automation strategy is adopted where you do a push
based deploy , one zone at a time, and then when all zones are
deployed the let the load balancer manage the instances. So, it looks
me to that you are doing it right for your solution and you are
perhaps looking to that little extra automation.

My solution has been to let a service on the host machine be read to
start other services on events and then do push based deploy so that
application is deployed and then the service listening for events
would recognize the new app and restart the service. The DNS load
balancer would be zone based so based and take effect after the
process.

Hope this helps,
Senthil

[Benjamin Scherrey]

Martin we're doing much of what you describe, using fabric as I believe you are. Haven't found any "off the shelf" solutions. All our stuff is custom scripts which we try to implement is as functional (no side effect) a manner as we can which is challenging when dealing with real-world systems. I guess that's why they call it DevOps.

 -- Ben

--
Chat: http://hackerspace.sg/chat

[Lin Chuan]

The problem with "push" style solutions is that they are not indempotent, which means that they would execute commands and scripts imperatively without first checking whether or not they ought to do so.  This might cause unintended side effects.  

It is of course possible to write "if" checks virtually everywhere and micromanage every script to check before execute like what python cuisine does with its _ensure idiom but this approach can be quite hackish and bulks up your fabric script significantly.

"Push only" approaches are fine for small scale stuff (2-10 servers), but once you have a lot of servers, a push system will begin to show its limits, unless of course you implement multiprocessing in your scripts.  

"Pull" type solutions on the other hand are declarative (i.e. indempotent) in nature, so no silly "if" checks everywhere; but the downside is that they are much more complex and time consuming to implement .

Which is why people usually use a mix of push-and-pull in a real life deployment.  

*  Push-style scripts for provisioning application code and services.

*  Pull-type solutions for initial server configuration and configuration file management (managing conf files for postgresql, memcached, task server, web server, other specialised services with a conf file etc).

--
Chat: http://hackerspace.sg/chat

[Senthil Kumaran]

On Fri, May 18, 2012 at 11:03:21PM +0800, Lin Chuan wrote:
> The problem with "push" style solutions is that they are not indempotent, which
> means that they would execute commands and scripts imperatively without first
> checking whether or not they ought to do so. This might cause unintended side
> effects.

Sorry. I do not get this. But I assume you are meaning that each
time a push is done, the result may be different. In fact, no. Push
is done for the entire feature and once it is complete the service
should start this feature and application from start.

Push type deploy is used by all major distributed systems. They spin
the OS with the configuration settings for services and then do push
deploy.

> Which is why people usually use a mix of push-and-pull in a real life
> deployment.
> * Push-style scripts for provisioning application code and services.
> * Pull-type solutions for initial server configuration and configuration file
> management (managing conf files for postgresql, memcached, task server, web
> server, other specialised services with a conf file etc).

Yup, thats reasonable.

Thanks,
Senthil

[Andy Marks]

Hi Lin Chuan,

Would be great to have you come along to DevOps Singapore one night and talk about your experiences with this stuff in Django world.  We've been Ruby and Java focussed so far, so it would be a great balance.

Let me know if you'd like to run a session around Django deployments and/or your work with bcfg2 or other such things.

Cheers,
Andy

Andy MARKS

Technical Principal

ThoughtWorks Singapore

email: ama...@thoughtworks.com

mobile: +65 9788 0259

web: www.thoughtworks.com

--
Chat: http://hackerspace.sg/chat

[Lin Chuan]

Hello Andy,

Sure. When and where are your meet-ups btw?  My office's at Samsung Hub Building at the moment.

I would love to attend your meet-ups probably after June?  At the moment, I am super engrossed with Diablo 3 (and, to be fair, my legitimate excuse is that I will be trying to figure out what to do with Diablo 3's API. Ideas welcome... :-p) 

And, also busy with an ongoing project - major iteration #2 due to release by the end of June.

Anything beyond that - no problem... Unless another major project comes our way again.  :-)

Cheers!

[Andy Marks]

Hi Lin Chuan,

All the meetup details appear firstly at http://www.meetup.com/devops-singapore/, but you can join the meetup group if you'd like email as well.  We haven't announced any meetups beyond June at the moment, but that will change after the June one has completed.

Cheers,
Andy

Andy MARKS

Technical Principal

ThoughtWorks Singapore

email: ama...@thoughtworks.com

mobile: +65 9788 0259

web: www.thoughtworks.com

--
Chat: http://hackerspace.sg/chat

[Martin Bähr]

On Tue, May 22, 2012 at 07:52:19PM +0800, Lin Chuan wrote:
> I am super engrossed with Diablo 3 (and, to be fair, my legitimate excuse
> is that I will be trying to figure out what to do with Diablo 3's API.

having a legitimate excuse to play is never fair, at least not to those
who don't have such an excuse ;-)

my excuse will be "researching games for my children". diablo doesn't
need to apply here though...

greetings, martin.
--
cooperative communication with sTeam - caudium, pike, roxen and unix
services: debugging, programming, training, linux sysadmin, web development
--
pike programmer working in china societyserver.(org|net)
foresight developer community.gotpike.org foresightlinux.org
unix sysadmin (open-steam|www.caudium).org realss.com
Martin B�hr http://societyserver.org/mbaehr/