Better web app deployment

[Michael Snoyman]

Hi all,

(Note: I originally was going to just send this to the Yesod mailing list, but frankly, most of the code I'm setting up would work for any web framework. If people using other frameworks want to use this deployment code, and are interested in working with me to make sure it's universally applicable, please be in touch.)

I'm planning on improving the deployment code starting next week. Instead of comparing to what we currently have (and which I think only I use), let me just give a high-level approach. Please give input on how you'd like to change things. Once I have the basic structure put together, I'll put it up on Github, and contributions will be greatly appreciated:

* We'll use Nginx as a reverse proxy. This will allow us to have virtual hosting, more efficient static file serving (possibly from a separate domain), and zero-downtime deployments.
* The scaffolding will accept a number of environment variables for configuring it, such as PostgreSQL connection information, approot, static root, and port number. (Most of this is already implemented.)
* Each site will have a deployment configuration file, for things like domain name, whether a PostgreSQL database is required, and any system packages are required to be installed (we'll assume a Debian-based system for now). The deployer will automatically create databases and provide that information via environment variables.
* Apps will have a specific file format, which is essentially a deployment config file and a bunch of things (static folder, executable, etc) it needs. An app will then be a tarball of those folders.
* The deployer will watch a folder for changes. Deployments will be performed by copying files into this folder.
* When a new app is deployed, it will be unpacked, started up, and monitored. If it stops, it will be restarted. The Nginx config file will be updated to reverse proxy to the app's port (randomly assigned) and then reloaded.
* When a new version of the app is deployed, we'll unpack and start it up to a new folder with a different port. Once the app responds to an HTTP HEAD request (doesn't matter *how* it responds), we'll update the Nginx config file, reload Nginx, send the TERM signal to the old app, and then delete the old app's folder.
* When an app is deleted, remove it from the Nginx config file, reload Nginx, send the app the TERM signal, and delete the folder.
* We'll automatically store log files for apps (recommendations on good approaches here are welcome).
* The deployer itself will provide a web interface to monitor status of apps and view logs.
* I have no specific plans for running shell scripts or the like. I believe most of that can be part of the app loading. With the zero-downtime deployment, it's acceptable for the load time to be significant. If anyone has better ideas here, let me know.

If we get a good deployment tool going, I think I would next want to set up an AMI (Amazon Machine Image) to make it dead simple to spin up an instance capable of hosting Yesod apps. If we can get it to configure DNS as well, and get a command line tool that automatically performs the EC2 instance creation, I think we'd be golden.

Michael

[Lyndon Maydwell]

This is absolutely on par with what I've been thinking about over the
last week. I take it you are drawing some inspiration from the Twelve
Factor App Manifesto that you linked me to? It certainly seems that
way :-)

I started building a management application to act as an interface for
this kind of architecture, however, I'm stuck grappling with the Yesod
form model as I haven't had much time to spend on it yet.

A couple of ideas:

Are you planning to have strong isolation between the applications
using some kind of sand-boxing? Given an Nginx reverse proxy this
should be made quite trivial by having the apps communicating over a
port or socket. Just a question of how it would be implemented.

I feel that a framework like this would benefit enormously from having
strong partnership with a continuous integration framework for
actually building the applications. This could run as an application
inside the application deployment framework in order to be nicely
circular! A benefit of this would be that you could guarantee that the
build and deploy architecture is the same - freeing developers from
the worry that their local machine is a different CPU architecture or
operating system.

Love the idea. You're always a step ahead.

[Greg Weber]

>
> * We'll use Nginx as a reverse proxy. This will allow us to have virtual
> hosting, more efficient static file serving (possibly from a separate
> domain), and zero-downtime deployments.

Its important to note that this level of infrastrucure can provide SSL also.
Pound can also be used in place of Nginx as a lighter-weight
alternative if all of Nginx's features aren't needed.
AWS provides their own load balancer: that can be used instead of Pound.

It all looks good!
With respect to Lyndon's questions of building, the ideal is to spin
up a build server identical to production, build, copy the new
application over, start the new app, and turn off the build server.
Jeremy is mentoring a Haskell build server GSoC project.

[Vagif Verdi]

I have an intranet app that has a couple of long running operations. Say batch print 50 pdf files to printer. Or submit 10,000 invoices to ADP Tax server (end of month).

Maybe killing the app server should be optional.

[Lyndon Maydwell]

I don't think that it's generally considered a good idea to have
long-running processes occurring in-app. I know that on Heroku there
are 'workers' that you can provision to take care of such tasks. These
could even run as their own apps that connect to the same resources
for queueing, etc, in order to reuse the deployment interfaces and
infrastructure.

Deciding to hold off on a restart of an application due to what would
usually be asynchronous operations seems a bit fishy.

[Vagif Verdi]

Why not ? It's an intranet app, few users, it does what we need. Its simple to deploy (one app instead of several separate apps). Much simpler infrastructure. Now i understand those awesome startups who deploy to Heroku of AWS or what have you in hopes to build next Facebook. Sure, go wild, nosql, thousands workers complex infrastructure.

But why would i overcomplicate my small intranet app, just because your deployment tool is not flexible enough ? It would be simpler for me just not to use your deployment tool :)

On Friday, May 4, 2012 10:10:32 AM UTC-7, Lyndon wrote:

I don't think that it's generally considered a good idea to have
long-running processes occurring in-app. I know that on Heroku there
are 'workers' that you can provision to take care of such tasks. These
could even run as their own apps that connect to the same resources
for queueing, etc, in order to reuse the deployment interfaces and
infrastructure.

Deciding to hold off on a restart of an application due to what would
usually be asynchronous operations seems a bit fishy.

[Lyndon Maydwell]

True - There are different optimal solutions for different situations
and flexibility is good.

[Michael Snoyman]

I was actually very specific about what we should do: send an
interrupt signal. In your case, you could catch that signal and allow
your long-running processes to finish. Likewise, an app which is has
any kind of long-running responses would want to take that approach.

[Michael Snoyman]

On Fri, May 4, 2012 at 6:17 PM, Lyndon Maydwell <mayd...@gmail.com> wrote:
> This is absolutely on par with what I've been thinking about over the
> last week. I take it you are drawing some inspiration from the Twelve
> Factor App Manifesto that you linked me to? It certainly seems that
> way :-)
>
> I started building a management application to act as an interface for
> this kind of architecture, however, I'm stuck grappling with the Yesod
> form model as I haven't had much time to spend on it yet.
>
> A couple of ideas:
>
> Are you planning to have strong isolation between the applications
> using some kind of sand-boxing? Given an Nginx reverse proxy this
> should be made quite trivial by having the apps communicating over a
> port or socket. Just a question of how it would be implemented.

I wasn't planning on it, but you're right that it should be feasible.
I suppose I've been taking the approach so far of letting the apps
have much more freedom than say a Heroku deployment. The idea would be
that only trusted apps would be running on the system.

> I feel that a framework like this would benefit enormously from having
> strong partnership with a continuous integration framework for
> actually building the applications. This could run as an application
> inside the application deployment framework in order to be nicely
> circular! A benefit of this would be that you could guarantee that the
> build and deploy architecture is the same - freeing developers from
> the worry that their local machine is a different CPU architecture or
> operating system.

Like Greg said, the ideal would be building this on a separate virtual
machine to avoid bogging down your app server with compilation. One
possible idea would be to provide a nice, powerful EC2 instance as a
community, and get a simple setup with Github commit hooks to be able
to deploy from it automatically. I definitely want to explore this,
but after the deployment system itself is in place.

[Max Cantor]

Have you considered using AWS ELBs (Elastic Load Balancers) instead of nginx? They have a few advantages:
1) configuration is stupidly easy
2) ssl config is also stupidly easy
3) lighten the load on your machine by putting the load balancer on a separate, but much cheaper instance.

I know the downside is that it ties us to AWS, but that might be worth it.

Just my $0.02.

max

[Michael Snoyman]

I'd be very surprised if configuration is simpler than Nginx (you can
see the current code here[1]). I haven't set up SSL yet here, but I've
done it in the past, and it's also very simple. The main advantage
would be lightening the load. On the flip side, we're tied into AWS
(*major* downside, I work with clients for example that require their
Yesod apps run on their server, and I don't think that's so uncommon)
and it can greatly increase hosting prices (I'm paying ~$8/month for
my reserved instance, and ELB would be ~$19/month).

Plus, if we want to have static file serving, we'd still need to run
Nginx locally.

[1] https://github.com/snoyberg/keter/blob/master/Keter/Nginx.hs#L117