How do you deploy code?

[Ori Lahav]

Time to share:

How do you deploy code?
At what scale of machines numbers?
Do you use tools for that? proprietary scripts? Manually (god forbid)?
Do you test it was successfully deployed? how?

Le's hear you.

[David Peleg]

I "deploy" to my little web hosting by copying PHP files over FTP :)

(yes, I know you are not interested in such "deployment methods"...)

My old way to deploy to tomcat is, I assume, the most trivial and widely-used: shut down the server, delete old WAR directory, copy new WAR file / directory and restart the server.

If you designed the configuration / state files to be outside of the WAR file - you're lucky. Otherwise you have to dig inside and set things before you start tomcat again.

I don't think it's that worse method in case you have another running instance of tomcat and load balancer. Actually it's like walking on 2 legs - while walking, one foot is on the air part of the time, but at last you're standing.

Running is not suggested on production :)

David.

[Ori Lahav]

nice :)

now imagine you have 1000 legs.

how long such deployment will take? what retention bonus would you give to the guy deploying this all day?

that's why I was asking about tools.

--
http://olahav.typepad.com

[David Peleg]

People say that the "Erlang way" is working. You hot-swap versions of processes (actually these are pieces of code) all the time and you never have to shut down the server.

Maybe someone has an experience with this...

David.

[Eishay Smith]

It might not be a good idea doing application hot-swap on a JVM. Java's class objects (class definitions) are kept in the JVM memory space called Perm Gen (String interns are there as well). Once the permgen is full it forces a full stop-the-world GC on the heap even if your GC configs are to CMS, its a huge performance killer especially in large heaps. If it can't clean the permgen you get nice OOM exceptions.

When reloading applications without killing the JVM you're adding class objects to the permgen. Even if its the same class but loaded from a different class loaders (as it happens in j2ee containers) you get a new class object. Some application containers (as tomcat) have known class memory leak where they keep references to old class definitions so even the stop the world GC won't help you.

Solution we use: have services running in clusters of instances. To deploy, stop traffic to subset of instances, kill them, restart them with a new version of the service, self check for errors : if(happy) {deploy the next max(n*2,cap) instances} else {roll back cluster to prev version}.

Building a basic tool should take 3-5 days assuming you have deployment, discoverability, service routing env already in place. Time to deploy N nodes is proportional to the cluster's unused capacity; more capacity that can go down for upgrade faster the cluster will get upgraded. Defining that capacity can be dynamic according to current traffic.

--es

[Ran Tavory]

There are a few ways to hot-deploy java apps, one is using the container's hot-deploy method (I know tomcat has this support through the manager app and I assume others too) and another is OSGi. I haven't used OSGi in practice, though. From what I read, it's a powerful but rather complicated beast. 

For taking servers down and restarting I can think of two main techniques - one is to multiply the service and deploy them behind a load balancer so you can afford to seamlessly take one of the instances or more down while updating the service and have the load balancer choose wisely. 

The other is using message queues, which conceptually are also nice as load balancers but they also have the up side of being able to buffer even if none of the instances are up (or just not keeping up with the pace) and the downside of added complexity and latency.

This is all still very high level, having that kind of system work reliably is the challenge.

It'd be interesting if folks could share some more specific techniques or tools.

On Mon, Nov 22, 2010 at 8:51 AM, Eishay Smith <eis...@gmail.com> wrote:

It might not be a good idea doing application hot-swap on a JVM. Java's class objects (class definitions) are kept in the JVM memory space called Perm Gen (String interns are there as well). Once the permgen is full it forces a full stop-the-world GC on the heap even if your GC configs are to CMS, its a huge performance killer especially in large heaps. If it can't clean the permgen you get nice OOM exceptions.

When reloading applications without killing the JVM you're adding class objects to the permgen. Even if its the same class but loaded from a different class loaders (as it happens in j2ee containers) you get a new class object. Some application containers (as tomcat) have known class memory leak where they keep references to old class definitions so even the stop the world GC won't help you.

fwiw tomcat 7 (or 6) should have this fixed 

--

/Ran

[Eran Sandler]

Speaking of doubling the load (and sorry for bringing cloud back into this thread), but this is the most common way to deploy in cloud environments.

You basically setup an (almost) complete environment with the new code on new instances, test it and when you are fine with it, update the load balancers to go to the new version. You can still run side-by-side with the old version and switch back if all hell breaks lose.

Once you are comfortable with the new code just shutdown the old instances and that's it.

The major benefit here is that you are never in an under capacity situation while deploying new code.

The best way of doing that is either having a image prepared with the "latest" version, or have a blank image and simply let the script "build" the whole machine from the ground up (which also means it never actually gets dirty :-) ).

Eran