Best Practices for Continuous Deployment on AppEngine

[Aleem Mawani]

I haven an appengine app and we currently do about 5-10 deploys per day. These deploys all have a NEW app version and are then set to default.

We notice that while switching over the default version, we get about a minute or 2 of instability while instances are switching over.

We plan on increasing the number of deploys to 30/day. Does it make sense to still deploy to a new version everytime?

What are the best practices here? How do you handle a lot of deploys?

[Vinny P]

On Thu, Aug 8, 2013 at 1:25 PM, aloo <aleem....@gmail.com> wrote:

I haven an appengine app and we currently do about 5-10 deploys per day. These deploys all have a NEW app version and are then set to default.

We notice that while switching over the default version, we get about a minute or 2 of instability while instances are switching over.

 

 

Use traffic splitting ( https://developers.google.com/appengine/docs/adminconsole/trafficsplitting ) to incrementally move more and more users to the new version of an application. That buys time for instances to spool up for the new version and spool down for the previous version.

On Thu, Aug 8, 2013 at 1:25 PM, aloo <aleem....@gmail.com> wrote:

We plan on increasing the number of deploys to 30/day. Does it make sense to still deploy to a new version everytime?

What are the best practices here? How do you handle a lot of deploys?

 

 

Is there a particular reason that you need to deploy new versions so rapidly? 30 deploys a day is fine for a test/low-production application, but if you're a high-traffic app it's a good idea to deploy on a much less frequent basis.

 

 

-----------------

-Vinny P

Technology & Media Advisor

Chicago, IL

App Engine Code Samples: http://www.learntogoogleit.com

 

 

 

[Jay]

I am curious. What kind of instability do you see? I haven't seen or at least noticed that. I'm not suggesting it doesn't exist. If there is something to watch out for there I would like to know.

[Aleem Mawani]

The instability is mostly around 500's serving. I guess its because new instances need to be booted up? I thought the new code would just run on the existing instances that are already running but I guess thats not the case.

[Aleem Mawani]

On Thursday, August 8, 2013 1:54:55 PM UTC-7, Vinny P wrote:

On Thu, Aug 8, 2013 at 1:25 PM, aloo <aleem....@gmail.com> wrote:

I haven an appengine app and we currently do about 5-10 deploys per day. These deploys all have a NEW app version and are then set to default.

We notice that while switching over the default version, we get about a minute or 2 of instability while instances are switching over.

 

 

Use traffic splitting ( https://developers.google.com/appengine/docs/adminconsole/trafficsplitting ) to incrementally move more and more users to the new version of an application. That buys time for instances to spool up for the new version and spool down for the previous version.

Interesting. So new versions of the app spin up new instances? Existing instances don't get the newly deployed code?

 

On Thu, Aug 8, 2013 at 1:25 PM, aloo <aleem....@gmail.com> wrote:

We plan on increasing the number of deploys to 30/day. Does it make sense to still deploy to a new version everytime?

What are the best practices here? How do you handle a lot of deploys?

 

 

Is there a particular reason that you need to deploy new versions so rapidly? 30 deploys a day is fine for a test/low-production application, but if you're a high-traffic app it's a good idea to deploy on a much less frequent basis.

We practice continuos deployment. As soon as a feature is ready or a bug is fixed we deploy it, individually, immediately. We find that much better than having larger releases with more code changes in each release.

[Vinny P]

On Friday, August 9, 2013 12:27:39 PM UTC-5, aloo wrote:

So new versions of the app spin up new instances? Existing instances don't get the newly deployed code?

I wouldn't say "new versions spin up new instances," that leads to confusion about what's happening. The more correct way to say it is "different app versions get their own instances."  

Remember that App Engine doesn't know which version is new or old (the version doesn't have to be a number, it can be a word too!). All it knows is that you have different versions of the app uploaded, and that you want to run them. So it allocates instances to run each version. By using Traffic Splitting, you can slowly move traffic from one version to another, and that sends a hint to the scheduler: "hey, this version is getting less traffic, maybe I should spool down the instances running that version. But wait, this other version over here is getting more traffic, so I'll allocate more instances to that."

If you want to directly replace a version of your app, then upload your application with the same version as the currently deployed default app.

[Aleem Mawani]

I guess my real question is can a single instance run more than one version?

[Vinny P]

On Fri, Aug 9, 2013 at 5:26 PM, aloo <aleem....@gmail.com> wrote:

I guess my real question is can a single instance run more than one version?

 

 

 

No.

 

One instance can run only one version of a given application.

[Nick]

We do continuous deployment as well, although not 30 a day.
I can't quite remember when there started being issues with 500s being served when deploying an app, but it wasn't always the case. Appengine used to just handle deploying newer code to the same version and start up new instances, shutting down old ones naturally.

We basically have two versions, one set to default, and another used for smoke test and to host while we're deploying. We upload new code to the smoke testing environment, smoke test, then switch it to default. Then we deploy to the other version and flip it back to default.

Occasionally the behaviour doing this changes (for a while the old instances on the old version didn't shutdown, and had to be done manually), but that's the nature of appengine.

Issue to star if you're interested (this affects everyone, any deployment will currently cause downtime to consumers for a small window - 2 seconds to a couple of minutes, there will be no indication of this in your logs at all):
http://code.google.com/p/googleappengine/issues/detail?id=7874&q=Deploy&colspec=ID%20Type%20Component%20Status%20Stars%20Summary%20Language%20Priority%20Owner%20Log

[Jeff Schnitzer]

We do zero-downtime rolling deployments. Our solution is to deploy to a new version each time (identified by a timestamp), warm up the version, and then switch. If you have a high-traffic app you may need to use something like 'ab' to flood the new version and spin up enough instances. 

Here's our ant script: https://gist.github.com/stickfigure/6201192

Unfortunately there's no API to delete old versions, so every couple deploys we have to go into the GAE admin UI and delete several versions by hand.

We're in the process of mavenizing this app so this is all going to change again of course...

Jeff

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

[Gopal Patel]

You can set default from command line.

[Nick]

It should be easy to replicate, when you deploy to your default version, hit the website in your web browser. Hey presto, 500. The window of failure varies, possibly depending on artifact size.

[Renzo Nuccitelli]

 Could you show de documentation about setting de default version from cmd line? I've searched for it but could not find.

 Thanks,
 Renzo

[Jeff Schnitzer]

The ant script in the gist includes an option to switch the default version automatically.

Jeff

[Jacob Taylor]

Partial copy of appcfg.py's output....last line is what you want

appcfg.py set_default_version .

JacobAirOne:tapnexus jacob$ appcfg.py 

Usage: appcfg.py [options] <action>

Action must be one of:

  backends: Perform a backend action.

  backends configure: Reconfigure a backend without stopping it.

  backends delete: Delete a backend.

  backends list: List all backends configured for the app.

  backends rollback: Roll back an update of a backend.

  backends start: Start a backend.

  backends stop: Stop a backend.

  backends update: Update one or more backends.

  create_bulkloader_config: Create a bulkloader.yaml from a running application.

  cron_info: Display information about cron jobs.

  download_app: Download a previously-uploaded app.

  download_data: Download entities from datastore.

  help: Print help for a specific action.

  list_versions: List all uploaded versions for an app.

  request_logs: Write request logs in Apache common log format.

  resource_limits_info: Get the resource limits.

  rollback: Rollback an in-progress update.

  set_default_version: Set the default (serving) version.

On Mon, Aug 12, 2013 at 1:11 PM, Renzo Nuccitelli <ren...@gmail.com> wrote:

--
Jacob Taylor

CTO, Kahuna

(650) 302-1864

www.useKahuna.com

http://www.linkedin.com/in/thejacobtaylor

[Aleem Mawani]

On Saturday, August 10, 2013 10:08:01 AM UTC-7, Jeff Schnitzer wrote:

We do zero-downtime rolling deployments. Our solution is to deploy to a new version each time (identified by a timestamp), warm up the version, and then switch. If you have a high-traffic app you may need to use something like 'ab' to flood the new version and spin up enough instances. 

Can you describe the proicess you use to "flood" the new version with requests?

[Jeff Schnitzer]

I don't flood it - I just make one fetch to make sure a single instance is up. Our app is very busty and usually has low enough traffic that this prevents any major UX issues. I just wanted to mention it just in case someone does try this with a high-traffic app.

Shelling out to 'ab' is pretty simple. This should get a handful of instances off the ground:

ab -n 100 -c 10 http://www.example.com/

See http://en.wikipedia.org/wiki/ApacheBench

Jeff

[Aleem Mawani]

Thanks thats very helpful!
ᐧ

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/Q9v_0bqm9ZI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengi...@googlegroups.com.

[Rafael]

For the continuous deployment, we use maven EAR to run all together in scripting. We run everything in our local servers, so we don't need to be warming up instances. 

We had to code our own dispatch.xml handler in order to be able to develop locally in a multi module setup. Since appengine doesn't support that in local. 

I have to do the AB trick every single time as well. With the same parameters :) 

It would be so easy for the appengine team to put a "warmup XX" instances whenever releasing a new version. 

Although, they're probably busy enabling a new language support rather than making existing customers happy :)

thanks

rafa

On Wed, Sep 25, 2013 at 8:19 AM, Jeff Schnitzer <je...@infohazard.org> wrote:

[Aleem Mawani]

Rafael - what parameters do you use with ab? I.e. how many requests and how many concurrent? I'm doing this from my local box and my connection keeps getting "reset by peer". I suspect this is some DoS protection....
ᐧ

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/Q9v_0bqm9ZI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengi...@googlegroups.com.

[alexh]

Hi Rafa,

We recently made the switch to using modules and sadly discovered that set_default_version is not supported with an EAR. Have you developed a workaround for this?

There's currently an open issue.

https://code.google.com/p/googleappengine/issues/detail?id=9920