Updates to managed VMs stopped working

179 views
Skip to first unread message

Andrei Volgin

unread,
Jan 15, 2016, 4:41:15 PM1/15/16
to Google App Engine
Starting from last night updates to modules using managed VMs started to fail for my project (id="filemambo"). appcfg update now results in


java.lang.RuntimeException: Fatal problem encountered during deployment. Please refer to the logs for more information.


       at com.google.appengine.tools.admin.AppVersionUpload.isServing(AppVersionUpload.java:1096)

       at com.google.appengine.tools.admin.AppVersionUpload.access$200(AppVersionUpload.java:46)

       at com.google.appengine.tools.admin.AppVersionUpload$2.call(AppVersionUpload.java:929)

       at com.google.appengine.tools.admin.AppVersionUpload$2.call(AppVersionUpload.java:926)

       at com.google.appengine.tools.admin.AppVersionUpload$3.call(AppVersionUpload.java:1233)

       at com.google.appengine.tools.admin.AppVersionUpload$3.call(AppVersionUpload.java:1230)

       at com.google.appengine.tools.admin.AppVersionUpload.retryWithBackoffOptional(AppVersionUpload.java:1270)

       at com.google.appengine.tools.admin.AppVersionUpload.retryWithBackoff(AppVersionUpload.java:1228)

       at com.google.appengine.tools.admin.AppVersionUpload.commit(AppVersionUpload.java:926)

       at com.google.appengine.tools.admin.AppVersionUpload.uploadFilesTransaction(AppVersionUpload.java:228)

       at com.google.appengine.tools.admin.AppVersionUpload.doUpload(AppVersionUpload.java:200)

       at com.google.appengine.tools.admin.AppAdminImpl.doUpdate(AppAdminImpl.java:571)

       at com.google.appengine.tools.admin.AppAdminImpl.update(AppAdminImpl.java:58)

       at com.google.appengine.tools.admin.AppCfg$UpdateAction.execute(AppCfg.java:1537)

       at com.google.appengine.tools.admin.AppCfg.executeAction(AppCfg.java:360)

       at com.google.appengine.tools.admin.AppCfg.<init>(AppCfg.java:222)

       at com.google.appengine.tools.admin.AppCfg.<init>(AppCfg.java:127)

       at com.google.appengine.tools.admin.AppCfg.main(AppCfg.java:123)


com.google.appengine.tools.admin.AdminException: Unable to update app: Fatal problem encountered during deployment. Please refer to the logs for more information.

       at com.google.appengine.tools.admin.AppAdminImpl.doUpdate(AppAdminImpl.java:577)

       at com.google.appengine.tools.admin.AppAdminImpl.update(AppAdminImpl.java:58)

       at com.google.appengine.tools.admin.AppCfg$UpdateAction.execute(AppCfg.java:1537)

       at com.google.appengine.tools.admin.AppCfg.executeAction(AppCfg.java:360)

       at com.google.appengine.tools.admin.AppCfg.<init>(AppCfg.java:222)

       at com.google.appengine.tools.admin.AppCfg.<init>(AppCfg.java:127)

       at com.google.appengine.tools.admin.AppCfg.main(AppCfg.java:123)


Caused by: java.lang.RuntimeException: Fatal problem encountered during deployment. Please refer to the logs for more information.

       at com.google.appengine.tools.admin.AppVersionUpload.isServing(AppVersionUpload.java:1096)

       at com.google.appengine.tools.admin.AppVersionUpload.access$200(AppVersionUpload.java:46)

       at com.google.appengine.tools.admin.AppVersionUpload$2.call(AppVersionUpload.java:929)

       at com.google.appengine.tools.admin.AppVersionUpload$2.call(AppVersionUpload.java:926)

       at com.google.appengine.tools.admin.AppVersionUpload$3.call(AppVersionUpload.java:1233)

       at com.google.appengine.tools.admin.AppVersionUpload$3.call(AppVersionUpload.java:1230)

       at com.google.appengine.tools.admin.AppVersionUpload.retryWithBackoffOptional(AppVersionUpload.java:1270)

       at com.google.appengine.tools.admin.AppVersionUpload.retryWithBackoff(AppVersionUpload.java:1228)

       at com.google.appengine.tools.admin.AppVersionUpload.commit(AppVersionUpload.java:926)

       at com.google.appengine.tools.admin.AppVersionUpload.uploadFilesTransaction(AppVersionUpload.java:228)

       at com.google.appengine.tools.admin.AppVersionUpload.doUpload(AppVersionUpload.java:200)

       at com.google.appengine.tools.admin.AppAdminImpl.doUpdate(AppAdminImpl.java:571)

       ... 6 more




I see that a managed VM instance is killed, a new instance is created, but this new instance has 0 bytes in memory and is not working (503 error: "The service you requested is not available yet).

There have been no changes in project structure, setup, etc. I noticed that this problem started exactly when the error message from deploys (about outdated SDK) disappeared. I upgraded to 1.9.31, but it did not help.

gcloud preview app deploy does not work either - it results in "Timed out when starting VMs. Rolling back the update."

I tried deploying module as a new version, deleting it completely and redeploying, etc. 

Any ideas how to resolve this?

Adam (Cloud Platform Support)

unread,
Jan 15, 2016, 8:07:26 PM1/15/16
to Google App Engine
Does this happen for non-default modules or the default module? If non-default, have you tried re-deploying the default module to a new version as well? Under 'Compute Engine' -> VM instances, do you see many 'gae-*' instances in an unhealthy state (including any gae-builder instances)?

Andrei Volgin

unread,
Jan 15, 2016, 8:15:05 PM1/15/16
to google-a...@googlegroups.com
1. I tried to redeploy all modules. The default is App Engine instances, and it works fine. The backend module is managed VMs, and it refuses to deploy. It worked yesterday.

2. When I try to deploy (gcloud preview app deploy), I get an error message, including , but I do see the age-builder instance for a while ("gae-builder-vm-1d", where 1d is module version). Then I see a vm instance, like "gae-backend-1d-xxoc". It spins for a long time, and finally turns green. It still shows 503 "The service you requested is not available yet." when I try to access it from the client.

Andrei

> --
> You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/qATq7olKdZ8/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to google-appengi...@googlegroups.com.
> To post to this group, send email to google-a...@googlegroups.com.
> Visit this group at https://groups.google.com/group/google-appengine.
> To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/b725d5ea-5309-48c6-bb9d-df05d756471e%40googlegroups.com.
>
> For more options, visit https://groups.google.com/d/optout.

Adam (Cloud Platform Support)

unread,
Jan 17, 2016, 4:47:59 PM1/17/16
to Google App Engine
The only recommendation I have is to try deleting all versions of Managed VM modules from App Engine -> Versions, and also any remaining Managed VM instances from Compute Engine -> Instances and trying a redeploy. I don't have any reports of known issues with Managed VM deployments at this time but I will be sure to keep this thread updated if I do. For troubleshooting purposes you might want to try testing deployment to a new project to test if the issue is project-local.
> To unsubscribe from this group and all its topics, send an email to google-appengine+unsubscribe@googlegroups.com.
> To post to this group, send email to google-appengine@googlegroups.com.

Andrei Volgin

unread,
Jan 18, 2016, 10:57:47 PM1/18/16
to google-a...@googlegroups.com
I already tried deleting all versions/instances, but it did not help.

Deploying the same code to a completely new project worked. What can be the cause? Both projects are on the same billing account. The only difference I can think of as relevant is the SSL resource that I created for compute engine instances in the project that fails to deploy managed VMs. Maybe the fact that I tried to deploy CE instances made some changes in the settings, but I don't see any differences in the console.

Any help is greatly appreciated. We have Gold support for the other project, and I am #5 on StackOverflow based on the number of answered questions (477) about App Engine :)

Andrei

> To unsubscribe from this group and all its topics, send an email to google-appengi...@googlegroups.com.
> To post to this group, send email to google-a...@googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/qATq7olKdZ8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/6dfe7497-13a9-4b8b-b28f-311b25757d8d%40googlegroups.com.

Adam (Cloud Platform Support)

unread,
Jan 19, 2016, 6:51:01 PM1/19/16
to Google App Engine
If it's local to the project, I'd start by checking the metadata for each project for discrepancies and also note the region you are using to create instances. Check the GCE quota details for limits as this also will affect your ability to create Managed VM instances.
> To unsubscribe from this group and all its topics, send an email to google-appengine+unsubscribe@googlegroups.com.
> To post to this group, send email to google-appengine@googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/qATq7olKdZ8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.

Adam (Cloud Platform Support)

unread,
Jan 19, 2016, 7:03:38 PM1/19/16
to Google App Engine
I'll also add that if you have Gold support on one project it's valid for both if they are under the same billing account, so your best route is likely opening a support case.
Reply all
Reply to author
Forward
0 new messages