deploy is very slow
4,529 views
Skip to first unread message
Adam Mathias Bittlingmayer
Feb 2, 2017, 9:02:29 AM2/2/17
to Google App Engine
Deployment with gcloud deploy app is slow (in the low dozens of minutes).
The build is fast, it is the push steps that are slow.
Both steps like Mounted from google_appengine/python and Updating service [test]...\ ie steps that are in the datacentre, not the upload itself.
Thus the size of the code seems to have no effect, it is small enough in this case. It happens with both python 2 and 3.
Is there a way to increase an allocation to speed this up? Are there other factors?
Nick (Cloud Platform Support)
Feb 3, 2017, 3:40:25 PM2/3/17
to Google App Engine
Hey Adam,
Are you deploying a Flexible Environment app? If so, what do the container builder logs from the Console show as far as activity during this time?
Also, is this still occurring, or was it only for a certain period of time? If it's transient, it might just be a momentary service degradation, which for a Beta product is to be expected at this stage. When a product goes into General Availability, we'll often have certain Service Level targets in terms of latency, uptime, that you can rely on (for example, read here for Compute Engine SLA's)
Cheers,
Nick
Cloud Platform Community Support
Are you deploying a Flexible Environment app? If so, what do the container builder logs from the Console show as far as activity during this time?
Also, is this still occurring, or was it only for a certain period of time? If it's transient, it might just be a momentary service degradation, which for a Beta product is to be expected at this stage. When a product goes into General Availability, we'll often have certain Service Level targets in terms of latency, uptime, that you can rely on (for example, read here for Compute Engine SLA's)
Cheers,
Nick
Cloud Platform Community Support
Adam Mathias Bittlingmayer
Feb 4, 2017, 2:30:00 AM2/4/17
to Google App Engine
Yes, it is the Flex Env.
Reviewing the container build logs, it does not seem deterministic. The total time varies, and the place in the process where multiple minutes pass between two log lines varies.
Reviewing the container build logs, it does not seem deterministic. The total time varies, and the place in the process where multiple minutes pass between two log lines varies.
Nick (Cloud Platform Support)
Feb 6, 2017, 12:58:05 PM2/6/17
to Google App Engine
Hey Adam,
Could you capture the deployment logs using the flags "--verbosity debug" and "--log-http", piping this output into the "ts" utility to prepend a timestamp to each line, and then email me the logs? I'll be able to analyze what might be going on that way. On linux / Mac that will look like:
(tee is used so that you can still read the output as it hits the console, and note that you'll have to put whatever version you want in the command where ${VERSION} is written)
On windows, you can use powershell to run the same command line after creating the "ts" filter to prepend timestamps by running the following line:
Cheers,
Nick
Could you capture the deployment logs using the flags "--verbosity debug" and "--log-http", piping this output into the "ts" utility to prepend a timestamp to each line, and then email me the logs? I'll be able to analyze what might be going on that way. On linux / Mac that will look like:
gcloud --quiet --verbosity debug --log-http app deploy app.yaml --version ${VERSION} 2>&1 | ts | tee -a deploy_log.txt
(tee is used so that you can still read the output as it hits the console, and note that you'll have to put whatever version you want in the command where ${VERSION} is written)
On windows, you can use powershell to run the same command line after creating the "ts" filter to prepend timestamps by running the following line:
filter timestamp {"$(Get-Date -Format s): $_"}
Cheers,
Nick
Nick (Cloud Platform Support)
Feb 6, 2017, 1:40:48 PM2/6/17
to Google App Engine
Just a quick update:
As well, to avoid the timestamp filtering commands above, you could simply run the deployment with the --verbosity debug and --log-http flags and then run gcloud info to check for where the "latest log file" is stored, and upload that file. The output line telling its location will look like:
As well, to avoid the timestamp filtering commands above, you could simply run the deployment with the --verbosity debug and --log-http flags and then run gcloud info to check for where the "latest log file" is stored, and upload that file. The output line telling its location will look like:
Last Log File: [/home/anon/.config/gcloud/logs/2017.02.06/18.30.49.455744.log]
Luis F De Pombo
Feb 12, 2017, 11:57:14 AM2/12/17
to Google App Engine
Hi Nick,
I am having the same issue as Adam but I am using a nodejs Flex env. I have the file you point to, what is the best email to get this to you? Thank you.
Nick (Cloud Platform Support)
Feb 13, 2017, 3:44:20 PM2/13/17
to Google App Engine
Hey Luis,
The best way in fact would be to create a Public Issue Tracker thread, and we'll be able to follow up there. Post the link here once you've created a thread, and I'll be happy to assist, sending you an email which you can reply to, attaching the log.
The best way in fact would be to create a Public Issue Tracker thread, and we'll be able to follow up there. Post the link here once you've created a thread, and I'll be happy to assist, sending you an email which you can reply to, attaching the log.
Cheers,
Nick
Cloud Platform Community Support
Alex Komoroske
Feb 20, 2017, 11:19:58 AM2/20/17
to Google App Engine
I'm running into this too with the Go flexible environment. Deploys are taking 30+ minutes.
Nick (Cloud Platform Support)
Feb 22, 2017, 6:11:06 PM2/22/17
to Google App Engine
Hey Alex,
Slow deploys can be caused by many factors. If you want to submit a Public Issue Tracker issue for us to look at, you should consider including the following information:
1. The size on disk of your project
2. Whether you're using the flexible environment or not, and whether you have a Dockerfile (if so, the contents of the Dockerfile)
3. The number of files in the project
4. The deploy log stored in a location given by running gcloud info and seeing the "Last Log File" field.
With this information we'll be able to better understand whether the deploy time is expected or exceptional.
Slow deploys can be caused by many factors. If you want to submit a Public Issue Tracker issue for us to look at, you should consider including the following information:
1. The size on disk of your project
2. Whether you're using the flexible environment or not, and whether you have a Dockerfile (if so, the contents of the Dockerfile)
3. The number of files in the project
4. The deploy log stored in a location given by running gcloud info and seeing the "Last Log File" field.
With this information we'll be able to better understand whether the deploy time is expected or exceptional.
Cheers,
Nick
Cloud Platform Community Support
Cole Peterson
Mar 31, 2017, 7:24:03 AM3/31/17
to Google App Engine
Hey Nick. Just go and deploy a service using glcoud deploy and you will see the issue.
It is unbearably slow.
Reply all
Reply to author
Forward
0 new messages