Some instances running unreleased version of the App Engine SDK?

[Julian Bunn]

We noticed on Sep 15 and 16 (a couple of days ago) that some of our GAE instances were running version 1.9.27 of the SDK, rather than the released version 1.9.26.

(We coincidentally had several serious issues with our application.)

Could someone please advise on how this can happen, and what we can do to prevent it happening in the future (i.e. force instances to only run the latest production release of the SDK)? Is there an application setting somewhere?

I apologize if this is the wrong Group for such questions.

Many thanks!

Julian

[PK]

Hi Julian,

Google routinely does this when they release a new runtime that they believe it is backwards incompatible. It mostly works OK but I did experience downtime for several hours on July 24th because a new version was rolled out that had an issue. It was a Friday, I was in an intercontinental flight without internet when the roll out happened and it was a small disaster. Google did take the issue seriously when I finally brought it to their attention but it took many hours to get back to stable ground. 

I think Google can do some things to improve this. First, it needs to pass some control of when we upgrade the runtime to developers hands so the world does not change suddenly and unexpectedly under our feet. Furthermore, as they roll out new versions they should automatically revert applications back to the previous version if the error rate was almost zero before and spikes as soon as instances in the new version come onboard.

I have filed this issue in the public tracker. I highly encourage others to star it and add your ideas before you find yourselves in my or Julian’s shoes.

Best 

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/4ac6bcf5-9a32-40fe-ac5a-0886e4eceeb9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

PK

p...@gae123.com

[Julian Bunn]

Thanks, PK - I will take a look at the issue you filed and comment accordingly.

We suffered pretty major disruption to our application, and lost many hours of data, unfortunately. The symptoms were extra null fields in some Datastore objects, which caused storage of new uploaded data from clients to fail, and a greatly increased Error rate from our app. 

Julian

[Jesse Scherer (Google Cloud Support)]

Julian there should be somebody from the engineering team reaching out to you soon -- if not already -- to understand what happened in your case.

As I mentioned in PK's issue we're working on ways to handle canaries better. BTW the most obvious approach, a flag for "never run my app on a canary app server" is not that great because it would simply delay discovery of problems until the appserver version which works for you is no longer current.

[PK]

Jesse, agreed. But at a minimum would like to control when it happens so it is a mutually convenient time.

BTW, you guys used to do SDK pre-releases. This might also help to catch some issues. I sed to always download the pre-release and run my tests on it.

On Sep 18, 2015, at 1:15 PM, Jesse Scherer (Google Cloud Support) <jsch...@google.com> wrote:

BTW the most obvious approach, a flag for "never run my app on a canary app server" is not that great because it would simply delay discovery of problems until the appserver version which works for you is no longer current.

PK

p...@gae123.com

[Nick]

I would imagine anyone doing anything half serious has a series of environments, all my projects have at least a -Dev and -UAT environment. I'd much prefer if those got the newer sdk first.

Or just actual environmental support built in - that would solve everyone's problem (and was underway at one point, right?)