Google App Engine is slow to deploy, hangs on "Updating service [someproject]..."

15,803 views
Skip to first unread message
Message has been deleted

Kevin Lau

unread,
Sep 29, 2016, 3:48:26 PM9/29/16
to Google App Engine

Is there a faster way to deploy an app? Google App Engine is slow to deploy, hangs on "Updating service [someproject]..." I am using a flexible environment with PHP.

Nick (Cloud Platform Support)

unread,
Oct 3, 2016, 4:39:50 PM10/3/16
to Google App Engine
Hey Kevin,

The speed of deployment can be changed by limiting the size of the uploaded app, limiting the complexity of the build necessary in the Dockerfile, if present, and by ensuring a fast and reliable internet connection. Other than these variables, it's hard to comment on what specifically might be making deployment exhibit the timing that it does on your system. 

I hope this is helpful - let me know if you have any further questions! I'll be happy to help!

Cheers,

Nick
Cloud Platform Community Suppor

Stanislas Marion

unread,
Mar 18, 2017, 12:03:46 PM3/18/17
to Google App Engine
Hi Nick,
I have tweaked my build so that all the image building is done outside of GAE.
After I have pushed my docker image to Google (using gcloud docker -- push gcr.io/...) which takes very little time, it seems like the "Updating service" step takes forever (7-8min) whereas all that is needed is a simple variant of 'docker run new_image' > `migrate traffic to new_image'. Why does that last 7-8 minutes ??? Is there any way to speed it up? It is very painful when developing.
Best,

"This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender immediately"

Nick (Cloud Platform Support)

unread,
Mar 20, 2017, 4:25:19 PM3/20/17
to Google App Engine
Hey Stanislas,

Could you let me know which region your app is based in? I'll attempt to reproduce this behaviour and hope to either have a reply explaining a bit of what's going on or else I can file a Public Issue Tracker issue for you to follow as a Feature Request on improving deployment times.

Cheers,

Nick
Cloud Platform Community Support

Nick (Cloud Platform Support)

unread,
Mar 20, 2017, 4:27:12 PM3/20/17
to Google App Engine
Ah, apologies for sending that last message too quick - I'll also need to know roughly the details of your application's contents. How long does it take your code to start up (you can check this by running the container on your own machine, assuming it has stats comparable to the deployment machine specs). Along with this information, could you share your Dockerfile contents with me in response to the email I've just sent you from esup...@google.com?

Cheers,

Nick

Cloud Platform Community Support

On Saturday, March 18, 2017 at 12:03:46 PM UTC-4, Stanislas Marion wrote:

Stanislas Marion

unread,
Mar 21, 2017, 6:00:11 AM3/21/17
to google-a...@googlegroups.com
Hi Nick,
My app is based in us-central.
My app is a very simple nodejs app, that runs an expressjs server. It starts in a few seconds (at most) with the command "node app.js --config config.json"
I'm attaching my Dockerfile and default.yaml.
Keep in mind that I deploy using the "gcloud app deploy default.yaml --image-url gcr.io/my-project/default"
Cheers,

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/hZMEkmmObDU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/5aa7088c-a33e-4135-974b-232adc1c4c08%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Dockerfile
default.yaml

Nick (Cloud Platform Support)

unread,
Mar 21, 2017, 12:18:43 PM3/21/17
to Google App Engine
Hey Stanislas,

Thanks for the information. I'll see what I can find out about this and will report back here.


Cheers,

Nick
Cloud Platform Community Support

On Tuesday, March 21, 2017 at 6:00:11 AM UTC-4, Stanislas Marion wrote:
Hi Nick,
My app is based in us-central.
My app is a very simple nodejs app, that runs an expressjs server. It starts in a few seconds (at most) with the command "node app.js --config config.json"
I'm attaching my Dockerfile and default.yaml.
Keep in mind that I deploy using the "gcloud app deploy default.yaml --image-url gcr.io/my-project/default"
Cheers,

On Mon, Mar 20, 2017 at 9:27 PM 'Nick (Cloud Platform Support)' via Google App Engine <google-appengine@googlegroups.com> wrote:
Ah, apologies for sending that last message too quick - I'll also need to know roughly the details of your application's contents. How long does it take your code to start up (you can check this by running the container on your own machine, assuming it has stats comparable to the deployment machine specs). Along with this information, could you share your Dockerfile contents with me in response to the email I've just sent you from esup...@google.com?

Cheers,

Nick

Cloud Platform Community Support

On Saturday, March 18, 2017 at 12:03:46 PM UTC-4, Stanislas Marion wrote:
Hi Nick,
I have tweaked my build so that all the image building is done outside of GAE.
After I have pushed my docker image to Google (using gcloud docker -- push gcr.io/...) which takes very little time, it seems like the "Updating service" step takes forever (7-8min) whereas all that is needed is a simple variant of 'docker run new_image' > `migrate traffic to new_image'. Why does that last 7-8 minutes ??? Is there any way to speed it up? It is very painful when developing.
Best,

On Monday, October 3, 2016 at 10:39:50 PM UTC+2, Nick (Cloud Platform Support) wrote:
Hey Kevin,

The speed of deployment can be changed by limiting the size of the uploaded app, limiting the complexity of the build necessary in the Dockerfile, if present, and by ensuring a fast and reliable internet connection. Other than these variables, it's hard to comment on what specifically might be making deployment exhibit the timing that it does on your system. 

I hope this is helpful - let me know if you have any further questions! I'll be happy to help!

Cheers,

Nick
Cloud Platform Community Suppor

On Thursday, September 29, 2016 at 3:48:26 PM UTC-4, Kevin Lau wrote:

Is there a faster way to deploy an app? Google App Engine is slow to deploy, hangs on "Updating service [someproject]..." I am using a flexible environment with PHP.

"This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender immediately"

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/hZMEkmmObDU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.

Nick (Cloud Platform Support)

unread,
Mar 21, 2017, 3:42:41 PM3/21/17
to Google App Engine
Hey Stanislas,

My initial hunch was that the issue was the deployment of other resources necessary to support the containers running. My analysis of deployment-related logs appears to confirm this:

I created a simple NodeJS app using your dockerfile and default.yaml. I then pushed the docker image to gcr.io and ran "gcloud app deploy --image-url ..."

After about 1 minute of waiting, all resources associated with the deployment had apparently completed, but the command had not returned yet:

```
$ gcloud deployment-manager resources list --deployment aef-default-20170321t185300

NAME                               TYPE                                     STATE      ERRORS  INTENT
aef-default-20170321t185300-00     compute.beta.regionInstanceGroupManager  COMPLETED  []
aef-default-20170321t185300-00ahs  compute.v1.httpsHealthCheck              COMPLETED  []
aef-default-20170321t185300-00it   compute.v1.instanceTemplate              COMPLETED  []
aef-default-20170321t185300-bs     compute.v1.backendService                COMPLETED  []
aef-default-20170321t185300-hcfw   compute.v1.firewall                      COMPLETED  []
aef-default-20170321t185300-hcs    compute.v1.httpsHealthCheck              COMPLETED  []
```

At around the same time that the last of the above completed, I see the following in the Console logs when selecting the "Deployment" logs source:

```
{
 protoPayload: {…}
 insertId: "54B422B11D921.AE9070D.A80D9821"
 resource: {
  type: "deployment"
  labels: {
   project_id: "<PROJECTID>"
   name: "<PROJECTID>-gclb"
  }
 }
 timestamp: "2017-03-21T18:54:05.999Z"
 severity: "ERROR"
 logName: "projects/<PROJECTID>/logs/cloudaudit.googleapis.com%2Factivity"
}
```

I haven't expanded the protoPayload, but it shows that this was an attempt to delete the deployment "<PROJECTID>-gclb" if it exists. It's near-immediately followed by a log representing an update on the deployment "<PROJECTID>-gclb" with createPolicy: "CREATE_OR_ACQUIRE". I believe it's reasonable to assume gclb stands for "Google Cloud Load Balancer", but I could be wrong here.

About six minutes later, the deployment command finally completed. At about the same time, the following shows up in the logs:

```
18:59:35.632
{"@type":"type.googleapis.com/google.cloud.audit.AuditLog","status":{"code":404,"message":"The object 'projects/<PROJECTID>/global/deployments/<PROJECTID>-gclb' is not found."},"authenticationInfo":{"principalEmail":"NUM...@cloudservices.gserviceaccount.com"},"requestMetadata":{"calle...
 {
 protoPayload: {…}
 insertId: "54B423EB72139.A6A3558.C2071C50"
 resource: {
  type: "deployment"
  labels: {
   name: "<PROJECTID>-gclb"
   project_id: "<PROJECTID>"
  }
 }
 timestamp: "2017-03-21T18:59:35.632Z"
 severity: "ERROR"
 logName: "projects/<PROJECTID>/logs/cloudaudit.googleapis.com%2Factivity"
}
```

This represents the deployment infrastructure checking whether the deployment "<PROJECTID>-gclb" has been created yet, whether the create operation scheduled above has finished yet. However, after another 25 seconds, approximately, we see the following:

```
19:00:00.122
{"@type":"type.googleapis.com/google.cloud.audit.AuditLog","authenticationInfo":{"principalEmail":"NUM...@cloudservices.gserviceaccount.com"},"requestMetadata":{"callerIp":"10.131.45.3","callerSuppliedUserAgent":"App Engine Flex Google-API-Java-Client Google-HTTP-Java-Client/1.22.0-SNAPSHOT (...
 {
 protoPayload: {
  authenticationInfo: {
   principalEmail: "NUM...@cloudservices.gserviceaccount.com"     
  }
  requestMetadata: {
   callerIp: "10.131.45.3"     
   callerSuppliedUserAgent: "App Engine Flex Google-API-Java-Client Google-HTTP-Java-Client/1.22.0-SNAPSHOT (gzip)"     
  }
  serviceName: "deploymentmanager.googleapis.com"    
  methodName: "v2.deploymentmanager.deployments.update"    
  resourceName: "projects/<PROJECTID>/global/deployments/aef-gclb"    
 }
 insertId: "54B42402DA050.AE94741.97003020"   
 resource: {
  type: "deployment"    
  labels: {
   name: "aef-gclb"     
   project_id: "<PROJECTID>"     
  }
 }
 timestamp: "2017-03-21T19:00:00.122Z"   
 severity: "NOTICE"   
 logName: "projects/<PROJECTID>/logs/cloudaudit.googleapis.com%2Factivity"   
 operation: {
  id: "operation-1490122775787-54b423eb8f5f9-5492ae56-bcfc4ec6"    
  last: true    
 }
}
```

Notice "type: deployment" and "last: true". This appears to signal the completion of a deployment "aef-gclb" at around the same time that the command line returned news that the deployment was ready to serve.

This all seems to point to the fact that there are some resources which need to be deployed before the app can serve, quite apart from fetching the container image from gcr.io. It appears that creating a load balancer / updating its rules is an important part of this which naturally will take a long time since a whole array of distributed machines in the routing infrastructure will need to sign off that they contain the new rules before the user can be confident the deployment has completed.

I hope this helps clarify things for you. As we continue to improve the infrastructure, doubtless times involved in these steps will decrease. We definitely analyze the data related to deployments and all kinds of other operations to determine how to streamline our systems. If you ever notice something in future that needs explanation, feel free to post in this group to inquire.


Cheers,

Nick
Cloud Platform Community Support

On Tuesday, March 21, 2017 at 6:00:11 AM UTC-4, Stanislas Marion wrote:
Hi Nick,
My app is based in us-central.
My app is a very simple nodejs app, that runs an expressjs server. It starts in a few seconds (at most) with the command "node app.js --config config.json"
I'm attaching my Dockerfile and default.yaml.
Keep in mind that I deploy using the "gcloud app deploy default.yaml --image-url gcr.io/my-project/default"
Cheers,

On Mon, Mar 20, 2017 at 9:27 PM 'Nick (Cloud Platform Support)' via Google App Engine <google-appengine@googlegroups.com> wrote:
Ah, apologies for sending that last message too quick - I'll also need to know roughly the details of your application's contents. How long does it take your code to start up (you can check this by running the container on your own machine, assuming it has stats comparable to the deployment machine specs). Along with this information, could you share your Dockerfile contents with me in response to the email I've just sent you from esup...@google.com?

Cheers,

Nick

Cloud Platform Community Support

On Saturday, March 18, 2017 at 12:03:46 PM UTC-4, Stanislas Marion wrote:
Hi Nick,
I have tweaked my build so that all the image building is done outside of GAE.
After I have pushed my docker image to Google (using gcloud docker -- push gcr.io/...) which takes very little time, it seems like the "Updating service" step takes forever (7-8min) whereas all that is needed is a simple variant of 'docker run new_image' > `migrate traffic to new_image'. Why does that last 7-8 minutes ??? Is there any way to speed it up? It is very painful when developing.
Best,

On Monday, October 3, 2016 at 10:39:50 PM UTC+2, Nick (Cloud Platform Support) wrote:
Hey Kevin,

The speed of deployment can be changed by limiting the size of the uploaded app, limiting the complexity of the build necessary in the Dockerfile, if present, and by ensuring a fast and reliable internet connection. Other than these variables, it's hard to comment on what specifically might be making deployment exhibit the timing that it does on your system. 

I hope this is helpful - let me know if you have any further questions! I'll be happy to help!

Cheers,

Nick
Cloud Platform Community Suppor

On Thursday, September 29, 2016 at 3:48:26 PM UTC-4, Kevin Lau wrote:

Is there a faster way to deploy an app? Google App Engine is slow to deploy, hangs on "Updating service [someproject]..." I am using a flexible environment with PHP.

"This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender immediately"

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/hZMEkmmObDU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.

Stanislas Marion

unread,
Mar 21, 2017, 3:50:34 PM3/21/17
to Google App Engine
Hi Nick,
Thanks a lot for the lengthy explanation. 
In this light, is there anything I can do to speed things up? Like for instance take care of the load-balancer myself? Indeed I don't see a reason why it should need to be changed. Could I do this with GAE or would I have to move to G Container/Compute E?
Cheers,

On Mon, Mar 20, 2017 at 9:27 PM 'Nick (Cloud Platform Support)' via Google App Engine <google-a...@googlegroups.com> wrote:
Ah, apologies for sending that last message too quick - I'll also need to know roughly the details of your application's contents. How long does it take your code to start up (you can check this by running the container on your own machine, assuming it has stats comparable to the deployment machine specs). Along with this information, could you share your Dockerfile contents with me in response to the email I've just sent you from esup...@google.com?

Cheers,

Nick

Cloud Platform Community Support

On Saturday, March 18, 2017 at 12:03:46 PM UTC-4, Stanislas Marion wrote:
Hi Nick,
I have tweaked my build so that all the image building is done outside of GAE.
After I have pushed my docker image to Google (using gcloud docker -- push gcr.io/...) which takes very little time, it seems like the "Updating service" step takes forever (7-8min) whereas all that is needed is a simple variant of 'docker run new_image' > `migrate traffic to new_image'. Why does that last 7-8 minutes ??? Is there any way to speed it up? It is very painful when developing.
Best,

On Monday, October 3, 2016 at 10:39:50 PM UTC+2, Nick (Cloud Platform Support) wrote:
Hey Kevin,

The speed of deployment can be changed by limiting the size of the uploaded app, limiting the complexity of the build necessary in the Dockerfile, if present, and by ensuring a fast and reliable internet connection. Other than these variables, it's hard to comment on what specifically might be making deployment exhibit the timing that it does on your system. 

I hope this is helpful - let me know if you have any further questions! I'll be happy to help!

Cheers,

Nick
Cloud Platform Community Suppor

On Thursday, September 29, 2016 at 3:48:26 PM UTC-4, Kevin Lau wrote:

Is there a faster way to deploy an app? Google App Engine is slow to deploy, hangs on "Updating service [someproject]..." I am using a flexible environment with PHP.

"This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender immediately"

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/hZMEkmmObDU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.

"This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender immediately"

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/hZMEkmmObDU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.

Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/dcfd42f7-f1da-4c0f-8adb-9e3cfdd361ca%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Nick (Cloud Platform Support)

unread,
Mar 21, 2017, 4:21:56 PM3/21/17
to google-a...@googlegroups.com
Hey Stanislas,

The exact explanation speculated on in my last post shouldn't be taken as any description of what's necessarily going on, however it was an estimate of what might be happening based on the logs observed. I'm corresponding with experts in this area to get a more clear answer at the moment.

You could look into deploying on Container Engine, which would mean that the front-end management done by the App Engine Flexible Environment infrastructure wouldn't be happening, rather it would be the responsibility of the resources you deploy on Container Engine (a managed service based pretty transparently on Kubernetes). Surely deploying new container images to your pool of instances of your app (in Kubernetes, nodes running pods) in a cluster (or multiple clusters) would be quite fast, since the master sitting in front of your clusters and the nodes in the cluster are not as massively distributed as the App Engine Flexible Environment serving infrastructure, hence updating their routing rules would be relatively fast. This is something to look into and experiment with if you don't want to wait on the more detailed word from experts, but don't rush to that if you're not all that curious.

I'll get back to this thread with more details when they're forthcoming in our investigation.


Cheers,

Nick
Cloud Platform Community Support 

On Tuesday, March 21, 2017 at 3:50:34 PM UTC-4, Stanislas Marion wrote:
Hi Nick,
Thanks a lot for the lengthy explanation. 
In this light, is there anything I can do to speed things up? Like for instance take care of the load-balancer myself? Indeed I don't see a reason why it should need to be changed. Could I do this with GAE or would I have to move to G Container/Compute E?
Cheers,
{"@type":"type.googleapis.com/google.cloud.audit.AuditLog","status":{"code":404,"message":"The object 'projects/<PROJECTID>/global/deployments/<PROJECTID>-gclb' is not found."},"authenticationInfo":{"principalEmail":"NUMBER@cloudservices.gserviceaccount.com"},"requestMetadata":{"calle...
 {
 protoPayload: {…}
 insertId: "54B423EB72139.A6A3558.C2071C50"
 resource: {
  type: "deployment"
  labels: {
   name: "<PROJECTID>-gclb"
   project_id: "<PROJECTID>"
  }
 }
 timestamp: "2017-03-21T18:59:35.632Z"
 severity: "ERROR"
 logName: "projects/<PROJECTID>/logs/cloudaudit.googleapis.com%2Factivity"
}
```

This represents the deployment infrastructure checking whether the deployment "<PROJECTID>-gclb" has been created yet, whether the create operation scheduled above has finished yet. However, after another 25 seconds, approximately, we see the following:

```
19:00:00.122
{"@type":"type.googleapis.com/google.cloud.audit.AuditLog","authenticationInfo":{"principalEmail":"NUMBER@cloudservices.gserviceaccount.com"},"requestMetadata":{"callerIp":"10.131.45.3","callerSuppliedUserAgent":"App Engine Flex Google-API-Java-Client Google-HTTP-Java-Client/1.22.0-SNAPSHOT (...
 {
 protoPayload: {
  authenticationInfo: {
   principalEmail: "NUMBER@cloudservices.gserviceaccount.com"     
On Mon, Mar 20, 2017 at 9:27 PM 'Nick (Cloud Platform Support)' via Google App Engine <google-appengine@googlegroups.com> wrote:
Ah, apologies for sending that last message too quick - I'll also need to know roughly the details of your application's contents. How long does it take your code to start up (you can check this by running the container on your own machine, assuming it has stats comparable to the deployment machine specs). Along with this information, could you share your Dockerfile contents with me in response to the email I've just sent you from esup...@google.com?

Cheers,

Nick

Cloud Platform Community Support

On Saturday, March 18, 2017 at 12:03:46 PM UTC-4, Stanislas Marion wrote:
Hi Nick,
I have tweaked my build so that all the image building is done outside of GAE.
After I have pushed my docker image to Google (using gcloud docker -- push gcr.io/...) which takes very little time, it seems like the "Updating service" step takes forever (7-8min) whereas all that is needed is a simple variant of 'docker run new_image' > `migrate traffic to new_image'. Why does that last 7-8 minutes ??? Is there any way to speed it up? It is very painful when developing.
Best,

On Monday, October 3, 2016 at 10:39:50 PM UTC+2, Nick (Cloud Platform Support) wrote:
Hey Kevin,

The speed of deployment can be changed by limiting the size of the uploaded app, limiting the complexity of the build necessary in the Dockerfile, if present, and by ensuring a fast and reliable internet connection. Other than these variables, it's hard to comment on what specifically might be making deployment exhibit the timing that it does on your system. 

I hope this is helpful - let me know if you have any further questions! I'll be happy to help!

Cheers,

Nick
Cloud Platform Community Suppor

On Thursday, September 29, 2016 at 3:48:26 PM UTC-4, Kevin Lau wrote:

Is there a faster way to deploy an app? Google App Engine is slow to deploy, hangs on "Updating service [someproject]..." I am using a flexible environment with PHP.

"This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender immediately"

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/hZMEkmmObDU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.

"This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender immediately"

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/hZMEkmmObDU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.

Stanislas Marion

unread,
Mar 21, 2017, 6:41:52 PM3/21/17
to Google App Engine
Great, thank you so much for your help, I'll be very interested in the details you'll get from your investigation.

On Tue, Mar 21, 2017 at 9:22 PM 'Nick (Cloud Platform Support)' via Google App Engine <google-a...@googlegroups.com> wrote:
Hey Stanislas,

The exact explanation speculated on in my last post shouldn't be taken as any description of what's necessarily going on, however it was an estimate of what might be happening based on the logs observed. I'm corresponding with experts in this area to get a more clear answer at the moment.

You could look into deploying on Container Engine, which would mean that the front-end management done by the App Engine Flexible Environment infrastructure wouldn't be happening, rather it would be the responsibility of the resources you deploy on Container Engine (a managed service based pretty transparently on Kubernetes). Surely deploying new container images to your pool of instances in a cluster (or multiple clusters) would be quite fast, since the master sitting in front of your clusters and the nodes in the cluster are not as massively distributed as the App Engine Flexible Environment serving infrastructure, hence updating their routing rules would be relatively fast. This is something to look into and experiment with if you don't want to wait on the more detailed word from experts, but don't rush to that if you're not all that curious.


I'll get back to this thread with more details when they're forthcoming in our investigation.


Cheers,

Nick
Cloud Platform Community Support 

On Tuesday, March 21, 2017 at 3:50:34 PM UTC-4, Stanislas Marion wrote:
Hi Nick,
Thanks a lot for the lengthy explanation. 
In this light, is there anything I can do to speed things up? Like for instance take care of the load-balancer myself? Indeed I don't see a reason why it should need to be changed. Could I do this with GAE or would I have to move to G Container/Compute E?
Cheers,

{"@type":"type.googleapis.com/google.cloud.audit.AuditLog","status":{"code":404,"message":"The object 'projects/<PROJECTID>/global/deployments/<PROJECTID>-gclb' is not found."},"authenticationInfo":{"principalEmail":"NUM...@cloudservices.gserviceaccount.com"},"requestMetadata":{"calle...
 {
 protoPayload: {…}
 insertId: "54B423EB72139.A6A3558.C2071C50"
 resource: {
  type: "deployment"
  labels: {
   name: "<PROJECTID>-gclb"
   project_id: "<PROJECTID>"
  }
 }
 timestamp: "2017-03-21T18:59:35.632Z"
 severity: "ERROR"
 logName: "projects/<PROJECTID>/logs/cloudaudit.googleapis.com%2Factivity"
}
```

This represents the deployment infrastructure checking whether the deployment "<PROJECTID>-gclb" has been created yet, whether the create operation scheduled above has finished yet. However, after another 25 seconds, approximately, we see the following:

```
19:00:00.122
{"@type":"type.googleapis.com/google.cloud.audit.AuditLog","authenticationInfo":{"principalEmail":"NUM...@cloudservices.gserviceaccount.com"},"requestMetadata":{"callerIp":"10.131.45.3","callerSuppliedUserAgent":"App Engine Flex Google-API-Java-Client Google-HTTP-Java-Client/1.22.0-SNAPSHOT (...
 {
 protoPayload: {
  authenticationInfo: {
   principalEmail: "NUM...@cloudservices.gserviceaccount.com"     
To unsubscribe from this group and all its topics, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.

"This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender immediately"

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/hZMEkmmObDU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.

"This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender immediately"

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/hZMEkmmObDU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.

Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/01edf030-61cd-4bed-b4c5-3564749c05fe%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Nick (Cloud Platform Support)

unread,
Apr 5, 2017, 7:28:11 PM4/5/17
to Google App Engine
Hey Stanislas,

So, I can confirm that my prior investigation and intuitions were on the right track. I can confirm that a majority of deployment lag comes from programming the Google Cloud Load Balancers (GCLB). This is what we've seen in this case. Updates have to go out across the entire infrastructure while still respecting certain locks used to keep configurations consistent. These Load Balancers are not visible to users in the Console and not user-configurable as they're infrastructural and meant to be exactly as they are, else users may try to modify or accidentally delete them, which would cause a lot of issues.

We're devoting a lot of energy to decreasing GCLB configuration push times, so rest assured that our efforts in that direction should pay off going forward.

Cheers,

Nick
Cloud Platform Community Support

On Tuesday, March 21, 2017 at 6:41:52 PM UTC-4, Stanislas Marion wrote:
Great, thank you so much for your help, I'll be very interested in the details you'll get from your investigation.

On Tue, Mar 21, 2017 at 9:22 PM 'Nick (Cloud Platform Support)' via Google App Engine <google-appengine@googlegroups.com> wrote:
Hey Stanislas,

The exact explanation speculated on in my last post shouldn't be taken as any description of what's necessarily going on, however it was an estimate of what might be happening based on the logs observed. I'm corresponding with experts in this area to get a more clear answer at the moment.

You could look into deploying on Container Engine, which would mean that the front-end management done by the App Engine Flexible Environment infrastructure wouldn't be happening, rather it would be the responsibility of the resources you deploy on Container Engine (a managed service based pretty transparently on Kubernetes). Surely deploying new container images to your pool of instances in a cluster (or multiple clusters) would be quite fast, since the master sitting in front of your clusters and the nodes in the cluster are not as massively distributed as the App Engine Flexible Environment serving infrastructure, hence updating their routing rules would be relatively fast. This is something to look into and experiment with if you don't want to wait on the more detailed word from experts, but don't rush to that if you're not all that curious.

I'll get back to this thread with more details when they're forthcoming in our investigation.


Cheers,

Nick
Cloud Platform Community Support 

On Tuesday, March 21, 2017 at 3:50:34 PM UTC-4, Stanislas Marion wrote:
Hi Nick,
Thanks a lot for the lengthy explanation. 
In this light, is there anything I can do to speed things up? Like for instance take care of the load-balancer myself? Indeed I don't see a reason why it should need to be changed. Could I do this with GAE or would I have to move to G Container/Compute E?
Cheers,

{"@type":"type.googleapis.com/google.cloud.audit.AuditLog","status":{"code":404,"message":"The object 'projects/<PROJECTID>/global/deployments/<PROJECTID>-gclb' is not found."},"authenticationInfo":{"principalEmail":"NUMBER@cloudservices.gserviceaccount.com"},"requestMetadata":{"calle...
 {
 protoPayload: {…}
 insertId: "54B423EB72139.A6A3558.C2071C50"
 resource: {
  type: "deployment"
  labels: {
   name: "<PROJECTID>-gclb"
   project_id: "<PROJECTID>"
  }
 }
 timestamp: "2017-03-21T18:59:35.632Z"
 severity: "ERROR"
 logName: "projects/<PROJECTID>/logs/cloudaudit.googleapis.com%2Factivity"
}
```

This represents the deployment infrastructure checking whether the deployment "<PROJECTID>-gclb" has been created yet, whether the create operation scheduled above has finished yet. However, after another 25 seconds, approximately, we see the following:

```
19:00:00.122
{"@type":"type.googleapis.com/google.cloud.audit.AuditLog","authenticationInfo":{"principalEmail":"NUMBER@cloudservices.gserviceaccount.com"},"requestMetadata":{"callerIp":"10.131.45.3","callerSuppliedUserAgent":"App Engine Flex Google-API-Java-Client Google-HTTP-Java-Client/1.22.0-SNAPSHOT (...
 {
 protoPayload: {
  authenticationInfo: {
   principalEmail: "NUMBER@cloudservices.gserviceaccount.com"     
To unsubscribe from this group and all its topics, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.

"This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender immediately"

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/hZMEkmmObDU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.

"This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender immediately"

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/hZMEkmmObDU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.

Stanislas Marion

unread,
Apr 6, 2017, 4:27:30 AM4/6/17
to Google App Engine
Hi Nick,
Thank you so much for your time and detailed explanations. I've been investigating GKE and it seems like it strikes the right balance between speed and convenience, so I'll probably move progressively towards it.
Best,

On Thu, Apr 6, 2017 at 1:28 AM 'Nick (Cloud Platform Support)' via Google App Engine <google-a...@googlegroups.com> wrote:
Hey Stanislas,

So, I can confirm that my prior investigation and intuitions were on the right track. I can confirm that a majority of deployment lag comes from programming the Google Cloud Load Balancers (GCLB). This is what we've seen in this case. Updates have to go out across the entire infrastructure while still respecting certain locks used to keep configurations consistent. These Load Balancers are not visible to users in the Console and not user-configurable as they're infrastructural and meant to be exactly as they are, else users may try to modify or accidentally delete them, which would cause a lot of issues.

We're devoting a lot of energy to decreasing GCLB configuration push times, so rest assured that our efforts in that direction should pay off going forward.

Cheers,

Nick
Cloud Platform Community Support

On Tuesday, March 21, 2017 at 6:41:52 PM UTC-4, Stanislas Marion wrote:
Great, thank you so much for your help, I'll be very interested in the details you'll get from your investigation.

{"@type":"type.googleapis.com/google.cloud.audit.AuditLog","status":{"code":404,"message":"The object 'projects/<PROJECTID>/global/deployments/<PROJECTID>-gclb' is not found."},"authenticationInfo":{"principalEmail":"NUM...@cloudservices.gserviceaccount.com"},"requestMetadata":{"calle...
 {
 protoPayload: {…}
 insertId: "54B423EB72139.A6A3558.C2071C50"
 resource: {
  type: "deployment"
  labels: {
   name: "<PROJECTID>-gclb"
   project_id: "<PROJECTID>"
  }
 }
 timestamp: "2017-03-21T18:59:35.632Z"
 severity: "ERROR"
 logName: "projects/<PROJECTID>/logs/cloudaudit.googleapis.com%2Factivity"
}
```

This represents the deployment infrastructure checking whether the deployment "<PROJECTID>-gclb" has been created yet, whether the create operation scheduled above has finished yet. However, after another 25 seconds, approximately, we see the following:

```
19:00:00.122
{"@type":"type.googleapis.com/google.cloud.audit.AuditLog","authenticationInfo":{"principalEmail":"NUM...@cloudservices.gserviceaccount.com"},"requestMetadata":{"callerIp":"10.131.45.3","callerSuppliedUserAgent":"App Engine Flex Google-API-Java-Client Google-HTTP-Java-Client/1.22.0-SNAPSHOT (...
 {
 protoPayload: {
  authenticationInfo: {
   principalEmail: "NUM...@cloudservices.gserviceaccount.com"     
To unsubscribe from this group and all its topics, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.

"This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender immediately"

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/hZMEkmmObDU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.

"This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender immediately"

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/hZMEkmmObDU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.

"This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender immediately"

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/hZMEkmmObDU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.

Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/a3520170-b3d5-4ddc-afce-a9dbc838d4d5%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Nick (Cloud Platform Support)

unread,
Apr 11, 2017, 9:05:46 AM4/11/17
to Google App Engine
Hey Stanislas,

Great to hear. I think you'll be glad you did, from what I see the flexibility and power of kubernetes is something really amazing if you can leverage it. Feel free to post to the kubernetes-users group forum if you have any open-ended discussions or questions about this. Troubleshooting can be done on Stack Overflow and issues can be reported at the issuetracker.google.com.


Cheers,

Nick
Cloud Platform Community Support

On Thursday, April 6, 2017 at 4:27:30 AM UTC-4, Stanislas Marion wrote:
Hi Nick,
Thank you so much for your time and detailed explanations. I've been investigating GKE and it seems like it strikes the right balance between speed and convenience, so I'll probably move progressively towards it.
Best,
On Thu, Apr 6, 2017 at 1:28 AM 'Nick (Cloud Platform Support)' via Google App Engine <google-appengine@googlegroups.com> wrote:
Hey Stanislas,

So, I can confirm that my prior investigation and intuitions were on the right track. I can confirm that a majority of deployment lag comes from programming the Google Cloud Load Balancers (GCLB). This is what we've seen in this case. Updates have to go out across the entire infrastructure while still respecting certain locks used to keep configurations consistent. These Load Balancers are not visible to users in the Console and not user-configurable as they're infrastructural and meant to be exactly as they are, else users may try to modify or accidentally delete them, which would cause a lot of issues.

We're devoting a lot of energy to decreasing GCLB configuration push times, so rest assured that our efforts in that direction should pay off going forward.

Cheers,

Nick
Cloud Platform Community Support

On Tuesday, March 21, 2017 at 6:41:52 PM UTC-4, Stanislas Marion wrote:
Great, thank you so much for your help, I'll be very interested in the details you'll get from your investigation.

{"@type":"type.googleapis.com/google.cloud.audit.AuditLog","status":{"code":404,"message":"The object 'projects/<PROJECTID>/global/deployments/<PROJECTID>-gclb' is not found."},"authenticationInfo":{"principalEmail":"NUMBER@cloudservices.gserviceaccount.com"},"requestMetadata":{"calle...
 {
 protoPayload: {…}
 insertId: "54B423EB72139.A6A3558.C2071C50"
 resource: {
  type: "deployment"
  labels: {
   name: "<PROJECTID>-gclb"
   project_id: "<PROJECTID>"
  }
 }
 timestamp: "2017-03-21T18:59:35.632Z"
 severity: "ERROR"
 logName: "projects/<PROJECTID>/logs/cloudaudit.googleapis.com%2Factivity"
}
```

This represents the deployment infrastructure checking whether the deployment "<PROJECTID>-gclb" has been created yet, whether the create operation scheduled above has finished yet. However, after another 25 seconds, approximately, we see the following:

```
19:00:00.122
{"@type":"type.googleapis.com/google.cloud.audit.AuditLog","authenticationInfo":{"principalEmail":"NUMBER@cloudservices.gserviceaccount.com"},"requestMetadata":{"callerIp":"10.131.45.3","callerSuppliedUserAgent":"App Engine Flex Google-API-Java-Client Google-HTTP-Java-Client/1.22.0-SNAPSHOT (...
 {
 protoPayload: {
  authenticationInfo: {
   principalEmail: "NUMBER@cloudservices.gserviceaccount.com"     
To unsubscribe from this group and all its topics, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.

"This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender immediately"

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/hZMEkmmObDU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.

"This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender immediately"

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/hZMEkmmObDU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.

"This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender immediately"

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/hZMEkmmObDU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.

E EE

unread,
May 2, 2017, 3:11:12 PM5/2/17
to Google App Engine
I am having the same problem in region US central and a hello world Java app (app engine flexible). This is a tiny app with one servlet that prints hello, it does not get smaller than that.

It happens from both my local machine as well as the browser console, using both gradle as well as "gcloud app deploy" command.

Nick (Cloud Platform Support)

unread,
May 2, 2017, 3:52:35 PM5/2/17
to Google App Engine
Hey E,

You should analyze (or sanitize and then attach here) the logs produced by the deployment. You can find the logs location by running 'gcloud info'.

Cheers,

Nick
Cloud Platform Community Support

E EE

unread,
May 2, 2017, 4:59:05 PM5/2/17
to Google App Engine
The latest attempt to deploy from the browser console resulted in:

ERROR: (gcloud.app.deploy) Error Response: [13] Unexpected Error. (The API call stubby.Send() took too long to respond and was cancelled.)

Attached the log, nothing jumps out there.
gcloud_logs.txt

Nick (Cloud Platform Support)

unread,
May 3, 2017, 4:20:12 PM5/3/17
to Google App Engine
Hey E,

The log file location is given when you run the gcloud info command. In your case, the output showed:

Logs Directory: [/tmp/tmp.ijTCcTysEa/logs]
Last Log File: [None]

It's odd that there should be "None" for the last log file. Did you run the failing command before running gcloud info? And are there any log files in /tmp/tmp.ijTCcTysEa/logs? It's possible with a file path like that that the log directory is temporary, so the logs from a prior deployment may not be saved unless you actively retrieve them after the fact within a given timeframe.

Cheers,

Nick
Cloud Platform Community Support

E EE

unread,
May 3, 2017, 5:27:24 PM5/3/17
to Google App Engine
Here is the log file from cloud shell

You can see many:
Operation [apps/my-project/operations/d8c581c51b0-e9a02f] not complete. Waiting 5s.
log.txt

Nick (Cloud Platform Support)

unread,
May 3, 2017, 5:43:50 PM5/3/17
to Google App Engine
Hey E,

Alright, now given that I can't look into what operation that is, you should check what it is using gcloud, using the command below:

gcloud beta app operations describe d8c581c51b0-e9a02f

Possibly it's the same issue discussed above, the creation of the load balancer.


Cheers,

Nick
Cloud Platform Community Support

E EE

unread,
May 4, 2017, 4:03:08 PM5/4/17
to Google App Engine
From the logs it seems like the operation is the deployment of a new version. See attached dump of the command.
describe.txt

Nick (Cloud Platform Support)

unread,
May 4, 2017, 6:01:17 PM5/4/17
to Google App Engine
Hey E,

I've seen it before where the app is actually serving but it takes some time for the deployment command to complete. In the operation info, it says, "servingStatus: SERVING". Could you check, when this occurs, whether the app isn't serving before the command ends?
Cheers,

Nick
Cloud Platform Community Support

E EE

unread,
May 4, 2017, 7:07:31 PM5/4/17
to Google App Engine
From my experiments the new version is live about the same time the upload command returns.

Gunar Cassiano Gessner

unread,
Aug 16, 2017, 9:02:06 AM8/16/17
to Google App Engine
I experience the same slowness.
It makes sense that it the problematic part is GCloud's Load Balancer,
as pushing code and building the container seems relatively fast.

Are there any updates on this on Google's part?

Gunar Cassiano Gessner

unread,
Aug 16, 2017, 9:53:08 AM8/16/17
to Google App Engine
Hi Nick, thank you for the information.

Are there any updates on decreasing GCLB configuration push times?
Is there a way for me to bypass it if using "manual_scaling: instances: 1"?

Thank you,
Gunar.


On Wednesday, April 5, 2017 at 8:28:11 PM UTC-3, Nick (Cloud Platform Support) wrote:
Hey Stanislas,

So, I can confirm that my prior investigation and intuitions were on the right track. I can confirm that a majority of deployment lag comes from programming the Google Cloud Load Balancers (GCLB). This is what we've seen in this case. Updates have to go out across the entire infrastructure while still respecting certain locks used to keep configurations consistent. These Load Balancers are not visible to users in the Console and not user-configurable as they're infrastructural and meant to be exactly as they are, else users may try to modify or accidentally delete them, which would cause a lot of issues.

We're devoting a lot of energy to decreasing GCLB configuration push times, so rest assured that our efforts in that direction should pay off going forward.

Cheers,

Nick
Cloud Platform Community Support

On Tuesday, March 21, 2017 at 6:41:52 PM UTC-4, Stanislas Marion wrote:
Great, thank you so much for your help, I'll be very interested in the details you'll get from your investigation.

{"@type":"type.googleapis.com/google.cloud.audit.AuditLog","status":{"code":404,"message":"The object 'projects/<PROJECTID>/global/deployments/<PROJECTID>-gclb' is not found."},"authenticationInfo":{"principalEmail":"NUM...@cloudservices.gserviceaccount.com"},"requestMetadata":{"calle...
 {
 protoPayload: {…}
 insertId: "54B423EB72139.A6A3558.C2071C50"
 resource: {
  type: "deployment"
  labels: {
   name: "<PROJECTID>-gclb"
   project_id: "<PROJECTID>"
  }
 }
 timestamp: "2017-03-21T18:59:35.632Z"
 severity: "ERROR"
 logName: "projects/<PROJECTID>/logs/cloudaudit.googleapis.com%2Factivity"
}
```

This represents the deployment infrastructure checking whether the deployment "<PROJECTID>-gclb" has been created yet, whether the create operation scheduled above has finished yet. However, after another 25 seconds, approximately, we see the following:

```
19:00:00.122
{"@type":"type.googleapis.com/google.cloud.audit.AuditLog","authenticationInfo":{"principalEmail":"NUM...@cloudservices.gserviceaccount.com"},"requestMetadata":{"callerIp":"10.131.45.3","callerSuppliedUserAgent":"App Engine Flex Google-API-Java-Client Google-HTTP-Java-Client/1.22.0-SNAPSHOT (...
 {
 protoPayload: {
  authenticationInfo: {
   principalEmail: "NUM...@cloudservices.gserviceaccount.com"     
To unsubscribe from this group and all its topics, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.

"This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender immediately"

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/hZMEkmmObDU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.

"This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender immediately"

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/hZMEkmmObDU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Message has been deleted
Message has been deleted

Adrien Di Pasquale

unread,
Aug 21, 2017, 3:06:19 PM8/21/17
to Google App Engine
Hi ! I'm experiencing the same slowness : the "Update service ...|" step is taking anywhere from 5 to 30 minutes. 

Here is what I've tried so far to fix it, without success: 
- use a different region (I've switched from europe-west to us-central)
- deploy the hello_world app from the sample github repo
- use the `no-promote` option
- use the `manual_scaling: instances: 1` option in app.yml

I've isolated it down to a single operation that hangs, it's always the same :
```
$ gcloud app operations describe [...]
metadata:
  ephemeralMessage: Waiting for instances to become available for serving traffic.
  insertTime: '2017-08-21T13:18:17.363Z'
  method: google.appengine.v1.Versions.CreateVersion
  target: [...]
  user: [...]
name: [...]
```

After it finally completes (5-30 minutes), when I check back the operation with the same command, it consistently shows an endTime ~= insertTime + 10s. So I'm guessing there's a buggy delay in the callback that marks the operation as completed (this echoes a previous comment in this thread, that the operation is actually done, but not marked as such).

Hope this can help ! This is ruining my first steps with GAE :/

FYI I'm using Python Flexible Environment with python 3.6, so I'm guessing my next steps will be to try with 2.7 or the Standard environment.

Samuel Cousin

unread,
Aug 24, 2017, 3:19:11 PM8/24/17
to Google App Engine
Thanks for the debug info! I'm experiencing the exact same issue but with a nodejs 8.x environment. This inertia to deploy a mere 100 lines of JS is really testing my limit...

Jean-Marc Arsan

unread,
Oct 17, 2017, 8:59:35 AM10/17/17
to Google App Engine
Experiencing the same issue with google-appengine :(. I was trying to migrate from AWS; however, after banging my head for the past few hours. I'll probably quit if it can't complete by tomorrow.

voice...@gmail.com

unread,
Nov 15, 2017, 6:35:38 PM11/15/17
to Google App Engine
From limited testing with Java Flexible, it happens with 

server: jetty9

Significantly less issues with standard Java, though we want to user the jetty built in container.

What we see is that initial deployment is OK, and every consecutive deployment slows down exponentially, until above half an hour it does not happen at all (or at least we consider it dead).

I assume there is some problem of GAE tracking the instances.


Daniel Sirota

unread,
Nov 15, 2017, 6:59:05 PM11/15/17
to google-a...@googlegroups.com
Happenned to me on the AppEngine Standard. Cleaning the staging.PROJECTNAME bucket on GCS fixed it.
It appears the folder gets bigger and bigger and it really slows the deploy.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.

Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/e20829dd-2d13-4453-bea9-6dbb3eed771b%40googlegroups.com.

Yngvar Kristiansen

unread,
Nov 19, 2017, 10:48:17 AM11/19/17
to Google App Engine
I am experiencing deploy times of 9 minutes, for every deploy, without code changes. All I have done is following a hello world tutorial for a Spring Boot app with Maven.

I have searched without finding a solution, unfortunately.

I think people get a bad first impression of the App Engine when deploy takes this long.

See attached logs.

On Thursday, November 16, 2017 at 12:59:05 AM UTC+1, Daniel Sirota wrote:
Happenned to me on the AppEngine Standard. Cleaning the staging.PROJECTNAME bucket on GCS fixed it.
It appears the folder gets bigger and bigger and it really slows the deploy.
On Nov 16, 2017 01:35, <voice...@gmail.com> wrote:
From limited testing with Java Flexible, it happens with 

server: jetty9

Significantly less issues with standard Java, though we want to user the jetty built in container.

What we see is that initial deployment is OK, and every consecutive deployment slows down exponentially, until above half an hour it does not happen at all (or at least we consider it dead).

I assume there is some problem of GAE tracking the instances.


--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
deploylog.txt

Vijay M

unread,
Nov 20, 2017, 10:41:40 PM11/20/17
to Google App Engine
I have been struggling with the same issue. I tried deploying a very simple app and waited for several minutes before the process timed out. I even ran into the same timeout issue with Google's own samples from 

https://github.com/GoogleCloudPlatform/python-docs-samples/appengine/flexible/hello_world


Coming from AWS, it has been very frustrating working with GCP. One more day of this and I am going back to AWS. 

Gunar Cassiano Gessner

unread,
Nov 21, 2017, 3:46:32 PM11/21/17
to google-a...@googlegroups.com
Speaking from experience here.
Just move on.
Google App Engine (esp. Flexible Engine) is not worthy of your time.

Take care,
@gunar

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/hZMEkmmObDU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.

Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/0c5f7fed-3551-4426-88e4-1383b4151084%40googlegroups.com.

DeWitt Clinton

unread,
Dec 12, 2017, 12:37:43 PM12/12/17
to Google App Engine
Hi all,

I’m on the App Engine team and I agree, flexible deployments are currently too slow. The observations in this thread generally match our own: Deploying a new version of an app takes around six minutes median (50th percentile), with tail spikes several times that (99th percentile). Language choice or app size isn’t a factor, as these setup costs are independent of the runtime or code being deployed. 

For many apps, where production versions are pushed via long-running, asynchronous CI/CD pipelines, the exact timing may not matter so much. But for everyone who depends on app deployments for development purposes, this is too much.

For those cases, we are testing an improved deployment flow that will let you update an existing version in-place in around 150 seconds median. We hope to release the fast in-place updates for everyone next month, but if you’d like to help us test it out a little sooner please tell me your project id here and I’ll whitelist it.

If you miss this signup, don’t worry, we plan to turn faster in-place updates on for all projects by default in the new year.

Thanks!

-DeWitt


On Tuesday, November 21, 2017 at 12:46:32 PM UTC-8, Gunar Cassiano Gessner wrote:
Speaking from experience here.
Just move on.
Google App Engine (esp. Flexible Engine) is not worthy of your time.

Take care,
@gunar
On Tue, Nov 21, 2017 at 1:24 AM, Vijay M <vijay...@gmail.com> wrote:
I have been struggling with the same issue. I tried deploying a very simple app and waited for several minutes before the process timed out. I even ran into the same timeout issue with Google's own samples from 

https://github.com/GoogleCloudPlatform/python-docs-samples/appengine/flexible/hello_world


Coming from AWS, it has been very frustrating working with GCP. One more day of this and I am going back to AWS. 

On Thursday, September 29, 2016 at 12:48:26 PM UTC-7, Kevin Lau wrote:

Is there a faster way to deploy an app? Google App Engine is slow to deploy, hangs on "Updating service [someproject]..." I am using a flexible environment with PHP.

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/hZMEkmmObDU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.

Filipe Caldas

unread,
Dec 19, 2017, 4:59:08 AM12/19/17
to Google App Engine
Hi DeWitt,

  Will this also impact the deploy time on managed Appengine applications?

Best,

Tom Jowett

unread,
Jan 18, 2018, 9:03:39 AM1/18/18
to Google App Engine
Thanks DeWitt!

I just signed up using that form for my app - is this still slated for release this month?

Cheers!
Tom

Ivan Aguirre

unread,
Jan 18, 2018, 9:03:39 AM1/18/18
to Google App Engine
Hi DeWitt,

Do you have updates on this? Is there an issue tracker I can follow?

best regards!!

AntoineA

unread,
Jan 23, 2018, 4:04:14 PM1/23/18
to Google App Engine
+1 - Also interested

DeWitt Clinton

unread,
Jan 25, 2018, 8:17:32 PM1/25/18
to google-a...@googlegroups.com
Hi all,

I've whitelisted everyone who filled out the form, and the experiment is going well so far. I hope to see it rolled out to all projects and publicly documented in the next several weeks.

Thanks for your help and feedback!

-DeWitt


Dinçer Kavraal

unread,
Feb 1, 2018, 8:58:40 AM2/1/18
to Google App Engine
I don't know about the experimentation but deployments are failing for the last a few days and taking hours.
Has anyone noticed that?

Also I am not on the whitelist.

Yuri Bürger

unread,
Feb 1, 2018, 12:14:52 PM2/1/18
to Google App Engine
I've noticed huge decrease in deployment time today. It was not fast before, but now it is very slow. Also, it seems randomly fail with "time out"

четвер, 1 лютого 2018 р. 15:58:40 UTC+2 користувач Dinçer Kavraal написав:
Message has been deleted

Dinçer Kavraal

unread,
Feb 1, 2018, 3:52:07 PM2/1/18
to Google App Engine
I have noticed, deleting old image files (google storage) or deploying services into a clean project made things a bit faster.
I don't know if it was a coincidence. They are probably trying to fix this.

I ran into random timeout errors too. At one case, it said "server at .....appspot.com could not be found" or sth like this.


On Thursday, 1 February 2018 20:52:12 UTC+3, Mike Hardy wrote:
Same with us. Our deployment hangs at `updating service` and eventually fails.

Franck Diédrich

unread,
Feb 8, 2018, 3:23:23 PM2/8/18
to Google App Engine
Hello,

I'm experiencing serious latency too on a flexible env.

Deployments used to last around 9 minutes (which is quite long) and now they time out without explicit reason.

When will deployement improvement flow be available?

Mike Hardy

unread,
Feb 12, 2018, 2:54:38 PM2/12/18
to Google App Engine
Same boat as Franck and others - this is unusable

DeWitt Clinton

unread,
Feb 12, 2018, 7:07:10 PM2/12/18
to google-a...@googlegroups.com
Hi all,

Thanks for being patient, everyone. I waited to reply here until the experiment was fully ramped-up and verified in prod, but all projects should now be eligible to start using the faster in-place version updates.

To perform an in-place update, please pass the --version flag to the deploy command and specify the version you would like to deploy over. Additionally, you’ll likely want to prevent that version from becoming the default, so be sure to include the --no-promote flag. For example:

  $ gcloud app deploy --version <your-existing-version-here> --no-promote 

We recommend you use this flow only for development purposes, and continue to deploy full versions for any production traffic.

Thanks to everyone who tested this out so far, and please let me know if you run into any issues or concerns along the way.

-DeWitt

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/hZMEkmmObDU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.

Justin Yip

unread,
Feb 12, 2018, 9:02:54 PM2/12/18
to Google App Engine
Hello,

My team is not able to deploy anything to appengine since 4pm (PST). The following message spins forever:

Updating service [****] (this may take several minutes)


Has anyone experienced similar problem?

Justni


On Monday, February 12, 2018 at 4:07:10 PM UTC-8, DeWitt Clinton wrote:
Hi all,

Thanks for being patient, everyone. I waited to reply here until the experiment was fully ramped-up and verified in prod, but all projects should now be eligible to start using the faster in-place version updates.

To perform an in-place update, please pass the --version flag to the deploy command and specify the version you would like to deploy over. Additionally, you’ll likely want to prevent that version from becoming the default, so be sure to include the --no-promote flag. For example:

  $ gcloud app deploy --version <your-existing-version-here> --no-promote 

We recommend you use this flow only for development purposes, and continue to deploy full versions for any production traffic.

Thanks to everyone who tested this out so far, and please let me know if you run into any issues or concerns along the way.

-DeWitt

On Mon, Feb 12, 2018 at 11:55 AM Mike Hardy <mha...@localinsights.io> wrote:
Same boat as Franck and others - this is unusable

On Thursday, February 8, 2018 at 12:23:23 PM UTC-8, Franck Diédrich wrote:
Hello,

I'm experiencing serious latency too on a flexible env.

Deployments used to last around 9 minutes (which is quite long) and now they time out without explicit reason.

When will deployement improvement flow be available?

Le jeudi 29 septembre 2016 21:48:26 UTC+2, Kevin Lau a écrit :

Is there a faster way to deploy an app? Google App Engine is slow to deploy, hangs on "Updating service [someproject]..." I am using a flexible environment with PHP.

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/hZMEkmmObDU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.

Samuel Woo

unread,
Feb 12, 2018, 11:09:54 PM2/12/18
to Google App Engine
I can't make a new deployment and I can't event start a previously deployed (and working) version. The server appears to start, but it does not respond to web requests.

Mike Hardy

unread,
Feb 13, 2018, 12:24:22 PM2/13/18
to Google App Engine
DeWitt, thanks for informing us but I see a few potential problems.

1. we've been passing `--version` in our gcloud app deploy command for the last 6 months to name the new release. Isn't this going to conflict with users like us who want to name the newest version?

2. I tested the solution and it just deploys to the previous version. It doesn't actually shut down the previous version and create a new one. In my opinion, one of the biggest features of app engine is rolling back to a previous version. It looks like this deployment method eliminates that feature.

voice...@gmail.com

unread,
Feb 13, 2018, 2:18:09 PM2/13/18
to Google App Engine
As far as I remember this is caused by a slow startup. The app takes too long to start responding to load balancer requests, so the load balancer gives up on it.
The app is still running fine, just not accessible from the outside world. You will be able to see the app logs in stack driver and even SSH to it, but not access it using HTTP.
This is at least my understanding of it.

The solution is to find a way to make your app start faster. Sometimes more machine resources help (Memory or even another core). This mean that you will need to run your app with much higher resources than it really needs, making it more expensive.

I think there is another ticket on adding a human readable failure message that tells you what went wrong, but apparently user friendliness is not a priority.




On Monday, February 12, 2018 at 8:09:54 PM UTC-8, Samuel Woo wrote:

DeWitt Clinton

unread,
Feb 13, 2018, 3:48:43 PM2/13/18
to google-a...@googlegroups.com
Hi Mike, response inline.

On Tue, Feb 13, 2018 at 9:24 AM, Mike Hardy <mha...@localinsights.io> wrote:
DeWitt, thanks for informing us but I see a few potential problems.

1. we've been passing `--version` in our gcloud app deploy command for the last 6 months to name the new release. Isn't this going to conflict with users like us who want to name the newest version?

This will still work as expected. If you use --version to name the version and we've never seen that version before, it will create a new one.

2. I tested the solution and it just deploys to the previous version. It doesn't actually shut down the previous version and create a new one. In my opinion, one of the biggest features of app engine is rolling back to a previous version. It looks like this deployment method eliminates that feature.

Redeploying over existing versions has always been a mutation rather than a full deployment, so if you really want a complete tear-down for the rollback scenario, you'll want to create a new version.  But if you just need fast rollbacks to a known-good version, I think the --version <known-good> flag may be exactly what you want. 

Maybe even create new versions with the --no-stop-previous-version flag and use traffic migration to shift traffic back and forth? Be sure to turn down old versions eventually, though.

-DeWitt

 
To unsubscribe from this group and all its topics, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.

Kyle Smaagard

unread,
Feb 18, 2018, 12:13:25 PM2/18/18
to Google App Engine


Just to add my own $0.02
Switched from the standard env to flexible and went from 30 second deploys to 5+ min.  Superrrrr bad experience especially when this occurs on Google's own samples!!

Sylvain de vathaire

unread,
Mar 9, 2018, 9:00:32 AM3/9/18
to Google App Engine
Sames issues on flexible..
Not able to build and deploy anymore !

this a a simple Wordpress web site but Build time out after 10 minutes (deadlineexceedederrors ..)

Let's go to a simple CAAS and forget about google bloated GAE

Baptiste Lafontaine

unread,
Apr 19, 2018, 11:54:18 AM4/19/18
to Google App Engine
Still the same issue in april 2018. 

Deployment of a simple PHP Symfony application in the flex environment takes at least 5 minutes even when specifying the version number and --nopromote. 

gcloud app deploy --version 20180419t141737 --no-promote  
 4:56.09 total

I've seen in the bugtracker (https://issuetracker.google.com/u/1/issues/35900606) that PHP 7.2 supports is coming to non-flex env with the promise of <1 minute deployment. 
Let's hope they move fast on this :) 

Prashanth Acharya

unread,
Oct 8, 2018, 8:53:35 AM10/8/18
to Google App Engine
Is there any updated. I have Rails flex instances timeout after 45mins( 2 services).

Katayoon (Cloud Platform Support)

unread,
Oct 8, 2018, 2:17:30 PM10/8/18
to Google App Engine

Hi Prashanth,


As DeWitt mentioned, having a delay around 6 minutes median (50th percentile) for a new version of App Engine Flex. deployment is the expected behavior since when you deploy your first application to the flexible environment, your virtual machine (VM) and other infrastructure are set up.


In your case, 45 minutes is too long and it seems something is misconfigured. So, I recommend to double check if you deploy each service with a separate app.yaml file and if the issue persists, you may report the issue to the Issue Tracker and provide us with the redacted output of the deploy command you are using (with the --verbosity tag set to debug) along with the output of the “gcloud info --anonymize” command so that we would be able investigate the root cause of the issue.



Prashanth Acharya

unread,
Oct 9, 2018, 1:59:15 AM10/9/18
to Google App Engine
Hi Katayoon,
   The app tries to connect to redis on startup.
    I realised that redis was in a different region hence unable to connect. 
    Would that be the cause of long delays during deployment?

sami...@google.com

unread,
Oct 9, 2018, 4:28:50 PM10/9/18
to Google App Engine
Yes, you are right. In order to connect to the Redis instance, App Engine applications must be in the same region as the Redis Instance and on the same network to be authorized. Please check the following article for reference.

Shiva Wu

unread,
Dec 29, 2018, 12:34:29 PM12/29/18
to Google App Engine
So I guess no update on the load balancer update speed, huh? It has been more than a year...

On Wednesday, 5 April 2017 16:28:11 UTC-7, Nick (Cloud Platform Support) wrote:
Hey Stanislas,

So, I can confirm that my prior investigation and intuitions were on the right track. I can confirm that a majority of deployment lag comes from programming the Google Cloud Load Balancers (GCLB). This is what we've seen in this case. Updates have to go out across the entire infrastructure while still respecting certain locks used to keep configurations consistent. These Load Balancers are not visible to users in the Console and not user-configurable as they're infrastructural and meant to be exactly as they are, else users may try to modify or accidentally delete them, which would cause a lot of issues.

We're devoting a lot of energy to decreasing GCLB configuration push times, so rest assured that our efforts in that direction should pay off going forward.

Cheers,

Nick
Cloud Platform Community Support

On Tuesday, March 21, 2017 at 6:41:52 PM UTC-4, Stanislas Marion wrote:
Great, thank you so much for your help, I'll be very interested in the details you'll get from your investigation.

On Tue, Mar 21, 2017 at 9:22 PM 'Nick (Cloud Platform Support)' via Google App Engine <google-a...@googlegroups.com> wrote:
Hey Stanislas,

The exact explanation speculated on in my last post shouldn't be taken as any description of what's necessarily going on, however it was an estimate of what might be happening based on the logs observed. I'm corresponding with experts in this area to get a more clear answer at the moment.

You could look into deploying on Container Engine, which would mean that the front-end management done by the App Engine Flexible Environment infrastructure wouldn't be happening, rather it would be the responsibility of the resources you deploy on Container Engine (a managed service based pretty transparently on Kubernetes). Surely deploying new container images to your pool of instances in a cluster (or multiple clusters) would be quite fast, since the master sitting in front of your clusters and the nodes in the cluster are not as massively distributed as the App Engine Flexible Environment serving infrastructure, hence updating their routing rules would be relatively fast. This is something to look into and experiment with if you don't want to wait on the more detailed word from experts, but don't rush to that if you're not all that curious.

I'll get back to this thread with more details when they're forthcoming in our investigation.


Cheers,

Nick
Cloud Platform Community Support 

On Tuesday, March 21, 2017 at 3:50:34 PM UTC-4, Stanislas Marion wrote:
Hi Nick,
Thanks a lot for the lengthy explanation. 
In this light, is there anything I can do to speed things up? Like for instance take care of the load-balancer myself? Indeed I don't see a reason why it should need to be changed. Could I do this with GAE or would I have to move to G Container/Compute E?
Cheers,

On Tue, Mar 21, 2017 at 8:42 PM 'Nick (Cloud Platform Support)' via Google App Engine <google-a...@googlegroups.com> wrote:
Hey Stanislas,

My initial hunch was that the issue was the deployment of other resources necessary to support the containers running. My analysis of deployment-related logs appears to confirm this:

I created a simple NodeJS app using your dockerfile and default.yaml. I then pushed the docker image to gcr.io and ran "gcloud app deploy --image-url ..."

After about 1 minute of waiting, all resources associated with the deployment had apparently completed, but the command had not returned yet:

```
$ gcloud deployment-manager resources list --deployment aef-default-20170321t185300

NAME                               TYPE                                     STATE      ERRORS  INTENT
aef-default-20170321t185300-00     compute.beta.regionInstanceGroupManager  COMPLETED  []
aef-default-20170321t185300-00ahs  compute.v1.httpsHealthCheck              COMPLETED  []
aef-default-20170321t185300-00it   compute.v1.instanceTemplate              COMPLETED  []
aef-default-20170321t185300-bs     compute.v1.backendService                COMPLETED  []
aef-default-20170321t185300-hcfw   compute.v1.firewall                      COMPLETED  []
aef-default-20170321t185300-hcs    compute.v1.httpsHealthCheck              COMPLETED  []
```

At around the same time that the last of the above completed, I see the following in the Console logs when selecting the "Deployment" logs source:

```
{
 protoPayload: {…}
 insertId: "54B422B11D921.AE9070D.A80D9821"
 resource: {
  type: "deployment"
  labels: {
   project_id: "<PROJECTID>"
   name: "<PROJECTID>-gclb"
  }
 }
 timestamp: "2017-03-21T18:54:05.999Z"
 severity: "ERROR"
 logName: "projects/<PROJECTID>/logs/cloudaudit.googleapis.com%2Factivity"
}
```

I haven't expanded the protoPayload, but it shows that this was an attempt to delete the deployment "<PROJECTID>-gclb" if it exists. It's near-immediately followed by a log representing an update on the deployment "<PROJECTID>-gclb" with createPolicy: "CREATE_OR_ACQUIRE". I believe it's reasonable to assume gclb stands for "Google Cloud Load Balancer", but I could be wrong here.

About six minutes later, the deployment command finally completed. At about the same time, the following shows up in the logs:

```
18:59:35.632
{"@type":"type.googleapis.com/google.cloud.audit.AuditLog","status":{"code":404,"message":"The object 'projects/<PROJECTID>/global/deployments/<PROJECTID>-gclb' is not found."},"authenticationInfo":{"principalEmail":"NUM...@cloudservices.gserviceaccount.com"},"requestMetadata":{"calle...
 {
 protoPayload: {…}
 insertId: "54B423EB72139.A6A3558.C2071C50"
 resource: {
  type: "deployment"
  labels: {
   name: "<PROJECTID>-gclb"
   project_id: "<PROJECTID>"
  }
 }
 timestamp: "2017-03-21T18:59:35.632Z"
 severity: "ERROR"
 logName: "projects/<PROJECTID>/logs/cloudaudit.googleapis.com%2Factivity"
}
```

This represents the deployment infrastructure checking whether the deployment "<PROJECTID>-gclb" has been created yet, whether the create operation scheduled above has finished yet. However, after another 25 seconds, approximately, we see the following:

```
19:00:00.122
{"@type":"type.googleapis.com/google.cloud.audit.AuditLog","authenticationInfo":{"principalEmail":"NUM...@cloudservices.gserviceaccount.com"},"requestMetadata":{"callerIp":"10.131.45.3","callerSuppliedUserAgent":"App Engine Flex Google-API-Java-Client Google-HTTP-Java-Client/1.22.0-SNAPSHOT (...
 {
 protoPayload: {
  authenticationInfo: {
   principalEmail: "NUM...@cloudservices.gserviceaccount.com"     
  }
  requestMetadata: {
   callerIp: "10.131.45.3"     
   callerSuppliedUserAgent: "App Engine Flex Google-API-Java-Client Google-HTTP-Java-Client/1.22.0-SNAPSHOT (gzip)"     
  }
  serviceName: "deploymentmanager.googleapis.com"    
  methodName: "v2.deploymentmanager.deployments.update"    
  resourceName: "projects/<PROJECTID>/global/deployments/aef-gclb"    
 }
 insertId: "54B42402DA050.AE94741.97003020"   
 resource: {
  type: "deployment"    
  labels: {
   name: "aef-gclb"     
   project_id: "<PROJECTID>"     
  }
 }
 timestamp: "2017-03-21T19:00:00.122Z"   
 severity: "NOTICE"   
 logName: "projects/<PROJECTID>/logs/cloudaudit.googleapis.com%2Factivity"   
 operation: {
  id: "operation-1490122775787-54b423eb8f5f9-5492ae56-bcfc4ec6"    
  last: true    
 }
}
```

Notice "type: deployment" and "last: true". This appears to signal the completion of a deployment "aef-gclb" at around the same time that the command line returned news that the deployment was ready to serve.

This all seems to point to the fact that there are some resources which need to be deployed before the app can serve, quite apart from fetching the container image from gcr.io. It appears that creating a load balancer / updating its rules is an important part of this which naturally will take a long time since a whole array of distributed machines in the routing infrastructure will need to sign off that they contain the new rules before the user can be confident the deployment has completed.

I hope this helps clarify things for you. As we continue to improve the infrastructure, doubtless times involved in these steps will decrease. We definitely analyze the data related to deployments and all kinds of other operations to determine how to streamline our systems. If you ever notice something in future that needs explanation, feel free to post in this group to inquire.


Cheers,

Nick
Cloud Platform Community Support

On Tuesday, March 21, 2017 at 6:00:11 AM UTC-4, Stanislas Marion wrote:
Hi Nick,
My app is based in us-central.
My app is a very simple nodejs app, that runs an expressjs server. It starts in a few seconds (at most) with the command "node app.js --config config.json"
I'm attaching my Dockerfile and default.yaml.
Keep in mind that I deploy using the "gcloud app deploy default.yaml --image-url gcr.io/my-project/default"
Cheers,

On Mon, Mar 20, 2017 at 9:27 PM 'Nick (Cloud Platform Support)' via Google App Engine <google-a...@googlegroups.com> wrote:
Ah, apologies for sending that last message too quick - I'll also need to know roughly the details of your application's contents. How long does it take your code to start up (you can check this by running the container on your own machine, assuming it has stats comparable to the deployment machine specs). Along with this information, could you share your Dockerfile contents with me in response to the email I've just sent you from esup...@google.com?

Cheers,

Nick

Cloud Platform Community Support

On Saturday, March 18, 2017 at 12:03:46 PM UTC-4, Stanislas Marion wrote:
Hi Nick,
I have tweaked my build so that all the image building is done outside of GAE.
After I have pushed my docker image to Google (using gcloud docker -- push gcr.io/...) which takes very little time, it seems like the "Updating service" step takes forever (7-8min) whereas all that is needed is a simple variant of 'docker run new_image' > `migrate traffic to new_image'. Why does that last 7-8 minutes ??? Is there any way to speed it up? It is very painful when developing.
Best,

On Monday, October 3, 2016 at 10:39:50 PM UTC+2, Nick (Cloud Platform Support) wrote:
Hey Kevin,

The speed of deployment can be changed by limiting the size of the uploaded app, limiting the complexity of the build necessary in the Dockerfile, if present, and by ensuring a fast and reliable internet connection. Other than these variables, it's hard to comment on what specifically might be making deployment exhibit the timing that it does on your system. 

I hope this is helpful - let me know if you have any further questions! I'll be happy to help!

Cheers,

Nick
Cloud Platform Community Suppor


On Thursday, September 29, 2016 at 3:48:26 PM UTC-4, Kevin Lau wrote:

Is there a faster way to deploy an app? Google App Engine is slow to deploy, hangs on "Updating service [someproject]..." I am using a flexible environment with PHP.

"This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender immediately"

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/hZMEkmmObDU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.

For more options, visit https://groups.google.com/d/optout.

"This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender immediately"

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/hZMEkmmObDU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.

For more options, visit https://groups.google.com/d/optout.

"This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender immediately"

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/hZMEkmmObDU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.

For more options, visit https://groups.google.com/d/optout.

"This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender immediately"

Shiva Wu

unread,
Dec 29, 2018, 12:34:30 PM12/29/18
to Google App Engine
So I guess no update on the GCLB deployment speed, huh? It has been more than a year...

Edgar (Google Cloud Platform Support)

unread,
Jan 3, 2019, 10:55:34 PM1/3/19
to Google App Engine
As DeWitt mentioned, currently our median deployment time for production goes around 6 minutes as it is working as intended on the flexible environment. There has been some additional work and testing done for faster deployment for development purposes.

The updated workflow("in-place version updates") to provide faster deployment when developing an application for the Flexible environment is described in the comment of 2/12/18 from DeWitt. For ease of reading here is the transcript of the workflow:


"To perform an in-place update, please pass the --version flag to the deploy command and specify the version you would like to deploy over. Additionally, you’ll likely want to prevent that version from becoming the default, so be sure to include the --no-promote flag. For example:

$ gcloud app deploy --version <your-existing-version-here> --no-promote

We recommend you use this flow only for development purposes, and continue to deploy full versions for any production traffic."

- Should you like to go further into the GCLB deployment speed, I invite you to open another Groups discussion with your details regarding the GCLB and reference this thread, as it will be easier to follow through the new one as the previous comment here mostly pertains to the slow deployment on the flexible environment for developing purposes and has been addressed here as explained above.

You would also be able to file a Feature Request as well directly should you want to see the current deployment speed improved. Providing the use case and the details for the slow deployment in the feature request will be needed for the engineering team to review for implementation.

Karolis Krulis

unread,
Jan 18, 2019, 3:15:29 PM1/18/19
to Google App Engine
That's not really a solution. If I deploy the same exact file to AWS it takes ~15s on a slow network. I spent 10 minutes deploying on google. 

Matt Haff

unread,
Jan 18, 2019, 3:25:29 PM1/18/19
to Google App Engine
I'm having the same issue, I posted a separate question here per Nick's request - https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!topic/google-appengine/NJMzqz5JqZQ

Mohammad I (Cloud Platform Support)

unread,
Jan 24, 2019, 12:01:28 AM1/24/19
to Google App Engine

Hello Matt,


As the issue is related to user’s deployment failure sometimes which is an indicator that this issue shouldn't be reported on Google Groups as this forum is meant for general discussion of the platform, not specific technical issues.


What's needed for anybody who would want to report a technical issue, is to use the issue tracker rather than google groups so that we can properly involve the engineering team and further investigate this deployment issue in the public issue tracker. If it is required to share any private or project specific information please choose the “Private” component so that the information are not visible to the public.


I wish you the best of luck in proceeding to either open a proper issue report in the public issue tracker.


I would also encourage you to share the public issue tracker link in this thread if you keep that public so that other users can go to the link for resolution or they can see the updates on the issue.


Matt Haff

unread,
Jan 24, 2019, 10:47:41 AM1/24/19
to Google App Engine
Thanks Mohammad, I have created an issue there as well.

Andreas Virkus

unread,
Jun 27, 2019, 2:11:12 PM6/27/19
to Google App Engine
I feel it necessary to echo this. Zeit, Heroku, AWS, [you-name-it] all deliver these micro-updates under a minute.
In todays world of rapid development/delivery/iteration, it's odd to see a company of Google's magnitude be contempt with being 10+ times slower than their competition.

I guess GAE at its current state is either still immature or simply meant to serve people with different needs. Hope GKE proves to be a better experience.

Magnus Markling

unread,
Apr 22, 2020, 8:12:18 AM4/22/20
to Google App Engine
Hey Nick!

How are these devoted efforts coming along? I just started a brand new GAE project with a simple Node application. Deployment takes 8-10 mins, the major part of which is still spent waiting for "Updating service". I'm not particularly impressed.

Cheers,
Magnus

Den torsdag 6 april 2017 kl. 01:28:11 UTC+2 skrev Nick (Cloud Platform Support):
We're devoting a lot of energy to decreasing GCLB configuration push times, so rest assured that our efforts in that direction should pay off going forward.

Cheers,

Nick
Cloud Platform Community Support

On Tuesday, March 21, 2017 at 6:41:52 PM UTC-4, Stanislas Marion wrote:
Great, thank you so much for your help, I'll be very interested in the details you'll get from your investigation.

On Tue, Mar 21, 2017 at 9:22 PM 'Nick (Cloud Platform Support)' via Google App Engine <google-a...@googlegroups.com> wrote:
Hey Stanislas,

The exact explanation speculated on in my last post shouldn't be taken as any description of what's necessarily going on, however it was an estimate of what might be happening based on the logs observed. I'm corresponding with experts in this area to get a more clear answer at the moment.

You could look into deploying on Container Engine, which would mean that the front-end management done by the App Engine Flexible Environment infrastructure wouldn't be happening, rather it would be the responsibility of the resources you deploy on Container Engine (a managed service based pretty transparently on Kubernetes). Surely deploying new container images to your pool of instances in a cluster (or multiple clusters) would be quite fast, since the master sitting in front of your clusters and the nodes in the cluster are not as massively distributed as the App Engine Flexible Environment serving infrastructure, hence updating their routing rules would be relatively fast. This is something to look into and experiment with if you don't want to wait on the more detailed word from experts, but don't rush to that if you're not all that curious.

I'll get back to this thread with more details when they're forthcoming in our investigation.


Cheers,

Nick
Cloud Platform Community Support 



Cheers,

Nick
Cloud Platform Community Support

On Tuesday, March 21, 2017 at 6:00:11 AM UTC-4, Stanislas Marion wrote:
Hi Nick,
My app is based in us-central.
My app is a very simple nodejs app, that runs an expressjs server. It starts in a few seconds (at most) with the command "node app.js --config config.json"
I'm attaching my Dockerfile and default.yaml.
Keep in mind that I deploy using the "gcloud app deploy default.yaml --image-url gcr.io/my-project/default"
Cheers,

On Mon, Mar 20, 2017 at 9:27 PM 'Nick (Cloud Platform Support)' via Google App Engine <google-a...@googlegroups.com> wrote:
Ah, apologies for sending that last message too quick - I'll also need to know roughly the details of your application's contents. How long does it take your code to start up (you can check this by running the container on your own machine, assuming it has stats comparable to the deployment machine specs). Along with this information, could you share your Dockerfile contents with me in response to the email I've just sent you from esup...@google.com?

Cheers,

Nick

Cloud Platform Community Support

On Saturday, March 18, 2017 at 12:03:46 PM UTC-4, Stanislas Marion wrote:
Hi Nick,
I have tweaked my build so that all the image building is done outside of GAE.
After I have pushed my docker image to Google (using gcloud docker -- push gcr.io/...) which takes very little time, it seems like the "Updating service" step takes forever (7-8min) whereas all that is needed is a simple variant of 'docker run new_image' > `migrate traffic to new_image'. Why does that last 7-8 minutes ??? Is there any way to speed it up? It is very painful when developing.
Best,


On Monday, October 3, 2016 at 10:39:50 PM UTC+2, Nick (Cloud Platform Support) wrote:
Hey Kevin,

The speed of deployment can be changed by limiting the size of the uploaded app, limiting the complexity of the build necessary in the Dockerfile, if present, and by ensuring a fast and reliable internet connection. Other than these variables, it's hard to comment on what specifically might be making deployment exhibit the timing that it does on your system. 

I hope this is helpful - let me know if you have any further questions! I'll be happy to help!

Cheers,

Nick
Cloud Platform Community Suppor

On Thursday, September 29, 2016 at 3:48:26 PM UTC-4, Kevin Lau wrote:

Is there a faster way to deploy an app? Google App Engine is slow to deploy, hangs on "Updating service [someproject]..." I am using a flexible environment with PHP.

"This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender immediately"

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/hZMEkmmObDU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-a...@googlegroups.com.

"This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender immediately"

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/hZMEkmmObDU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-a...@googlegroups.com.

"This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender immediately"

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/hZMEkmmObDU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-a...@googlegroups.com.

Olu

unread,
Apr 22, 2020, 10:23:06 AM4/22/20
to Google App Engine
I understand your concern about the Deployment time. These are on-going efforts and changes that sadly have no ETA at this time as the team is constantly looking into ways to get better deployment time for all App Engine environments. There are similar reports on these issue links[1][2] that the Internal GCLB engineers are aware of. I recommend you read the details shared and track the issue links for regular updates, particularly when the fixes are completed. At this time, however, recommendations offered on this thread are still what are viable. 

Reply all
Reply to author
Forward
0 new messages