DEADLINE_EXCEEDED error while deploying
3,368 views
Skip to first unread message
pankaj kumar
Apr 6, 2016, 12:01:46 AM4/6/16
to Google App Engine
We have a custom runtime docker container that I deploy on google app engine. While running the command "gcloud preview app deploy app.yaml", I often get the following error but sometimes it goes through. I am not sure what deadline gets exceeded.
Where should I look to find the deadline that I am exceeding?
Thanks
ERROR: (gcloud.preview.app.deploy) Error Response: [4] DEADLINE_EXCEEDED
npm ERR! Linux 3.19.0-51-generic
npm ERR! argv "/nodejs/bin/node" "/nodejs/bin/npm" "run" "deploy"
npm ERR! node v4.2.3
npm ERR! npm v3.8.5
npm ERR! code ELIFECYCLE
npm ERR! DataC...@1.0.0 deploy: ` gcloud config set project datacabinet-backend && gcloud preview app deploy app.yaml`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the DataC...@1.0.0 deploy script ' gcloud config set project datacabinet-backend && gcloud preview app deploy app.yaml'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the DataCabinet package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR! gcloud config set project datacabinet-backend && gcloud preview app deploy app.yaml
npm ERR! You can get information on how to open an issue for this project with:
npm ERR! npm bugs DataCabinet
npm ERR! Or if that isn't available, you can get their info via:
npm ERR! npm owner ls DataCabinet
npm ERR! There is likely additional logging output above.
npm ERR! Please include the following file with any support request:
npm ERR! /datacabinet-backend/npm-debug.log
Nicholas (Google Cloud Support)
Apr 6, 2016, 11:54:12 AM4/6/16
to Google App Engine
DEADLINE_EXCEEDED normally takes place when some part of the deploy process times out. It's not entirely clear from the console output why this is timing out. From the output above, the following seems to be happening:
- Executing a npm deploy script
- Script executes gcloud config set project datacabinet-backend && gcloud preview app deploy app.yaml and fails
- gcloud preview app deploy times out returning DEADLINE_EXCEEDED error
- npm catches the exception and returns
If this is not the case, please clarify what your actual build/deploy steps are. If this is the case, I would suggest the --verbosity debug flag with the gcloud preview app deploy command for more informative gcloud output and separating the two gcloud commands:
> gcloud config set project datacabinet-backend
> gcloud preview app deploy app.yaml --verbosity debug
This may provide a better idea of where in the deploy process this timeout occurs. Please note that very often, timeout errors are best addressed with retries.
pankaj kumar
Apr 6, 2016, 4:54:24 PM4/6/16
to Google App Engine
Thanks for your followup. Here is the error I receive when trying with debug verbosity.
DEBUG: Operation [operations/build/datacabinet-backend/NjgyN2VkNGUtMThjOC00NGVhLWE3NDgtY2U2YWFhYmFhM2ViOlVT] not complete. Waiting 1s.
DEBUG: Reading GCS logfile: 206 (read 461 bytes)
gcc -std=gnu99 -DSTANDALONE_RSERVE -DRSERV_DEBUG -DNODAEMON -I. -Iinclude -I/Rbins/lib/R/include -DNDEBUG -DRSERVE_PKG -I. -Iinclude -pthread -I/usr/local/include -fpic -g -O2 -DRSERVE_PKG -I. -Iinclude -pthread -o Rserve.dbg standalone.c md5.c session.c qap_decode.c qap_encode.c sha1.c base64.c websockets.c RSserver.c tls.c http.c oc.c rsio.c ulog.c ioc.c utils.c -lssl -lcrypto -ldl -lcrypt -L/Rbins/lib/R/lib -lR -lssl -lcrypto -ldl -lcrypt
DEBUG: Operation [operations/build/datacabinet-backend/NjgyN2VkNGUtMThjOC00NGVhLWE3NDgtY2U2YWFhYmFhM2ViOlVT] complete. Result: {
"metadata": {
"build": {
"finishTime": "2016-04-06T20:50:37.382268Z",
"status": "TIMEOUT",
"timeout": "600.000s",
"projectId": "datacabinet-backend",
"id": "6827ed4e-18c8-44ea-a748-ce6aaabaa3eb",
"source": {
"storageSource": {
"object": "datacabinet-backend.default.20160406t204017",
"bucket": "staging.datacabinet-backend.appspot.com"
}
},
"steps": [
{
"args": [
],
"name": "gcr.io/cloud-builders/dockerizer"
}
],
"startTime": "2016-04-06T20:40:37.120120Z",
"logsBucket": "staging.datacabinet-backend.appspot.com",
"images": [
],
"createTime": "2016-04-06T20:40:36.449834Z"
}
},
"done": true,
"name": "operations/build/datacabinet-backend/NjgyN2VkNGUtMThjOC00NGVhLWE3NDgtY2U2YWFhYmFhM2ViOlVT",
"error": {
"message": "DEADLINE_EXCEEDED",
"code": 4
}
}
DEBUG: (gcloud.preview.app.deploy) Error Response: [4] DEADLINE_EXCEEDED
Traceback (most recent call last):
File "/root/google-cloud-sdk/lib/googlecloudsdk/calliope/cli.py", line 654, in Execute
result = args.cmd_func(cli=self, args=args)
File "/root/google-cloud-sdk/lib/googlecloudsdk/calliope/backend.py", line 1390, in Run
resources = command_instance.Run(args)
File "/root/google-cloud-sdk/lib/surface/preview/app/deploy.py", line 510, in Run
config_cleanup)
File "/root/google-cloud-sdk/lib/googlecloudsdk/api_lib/app/deploy_command_util.py", line 197, in BuildAndPushDockerImages
storage_client, http)
File "/root/google-cloud-sdk/lib/googlecloudsdk/api_lib/app/deploy_command_util.py", line 247, in _BuildImagesWithCloudBuild
image.repo_tag, cloudbuild_client, http)
File "/root/google-cloud-sdk/lib/googlecloudsdk/api_lib/app/cloud_build.py", line 175, in ExecuteCloudBuild
retry_callback=log_tailer.Poll)
File "/root/google-cloud-sdk/lib/googlecloudsdk/api_lib/app/api/operations.py", line 69, in WaitForOperation
encoding.MessageToPyValue(completed_operation.error)))
OperationError: Error Response: [4] DEADLINE_EXCEEDED
ERROR: (gcloud.preview.app.deploy) Error Response: [4] DEADLINE_EXCEEDED
Josh Moore
Apr 6, 2016, 5:00:01 PM4/6/16
to google-a...@googlegroups.com
If you can set up docker locally, try using the flag `--docker-build local` as documented here[1]. That will at least provide a workaround.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/1baf8f99-c445-4348-ab62-faf1be339f75%40googlegroups.com.--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
pankaj kumar
Apr 6, 2016, 6:29:56 PM4/6/16
to Google App Engine
I can create one locally just fine. It is when I create with gcloud that I get this error.
Josh Moore
Apr 6, 2016, 6:38:36 PM4/6/16
to google-a...@googlegroups.com
The timeout is occurring in the build phase. The timeout is shown as:
"timeout": "600.000s",
Thus, the image has 10 minutes to build before timing out, and the timestamps in the response bear that out. You'll have to either optimize your dockerfile to build more quickly or build the file locally, for now. How long does the image take to build locally?
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/507a0293-0e15-486b-82c7-a4c857689d50%40googlegroups.com.
pankaj kumar
Apr 7, 2016, 1:31:56 AM4/7/16
to Google App Engine
I am just trying to build one package from sources which is taking all the time. Is there a way to bump up the timeout? We intend to modify the package later and would like to compile from sources. We would only add more features and the build time will go up in future.
I think there is another step of provisioning the build machine when running with gcloud. It takes some time and the overall time probably goes beyond 10 minutes. I will confirm shortly.
Thanks
Nicholas (Google Cloud Support)
Apr 12, 2016, 11:24:49 AM4/12/16
to Google App Engine
At this time, the deadline cannot be changed. You can however find several approaches to avoiding such errors in this helpful article along with other important information about such exceptions. Hope this helps!
On Wednesday, April 6, 2016 at 12:01:46 AM UTC-4, pankaj kumar wrote:
Jason Kriesel
Jun 22, 2016, 8:08:20 PM6/22/16
to Google App Engine
With the new local build approach (docker-build local doesn't appear to be an option for me), how do you deploy the docker file once it's built?
Zachary Kloepping
Aug 20, 2017, 3:40:58 PM8/20/17
to Google App Engine
I wasn't using docker containers but I was having this same issue deploying a Rails app and it ended up being because I had some large log files and database dumps in my logs/ and tmp/ directories. I fixed by adding this to the app.yml
skip_files:
- tmp/
- log/
Todd Morrison
Feb 16, 2018, 8:59:00 AM2/16/18
to Google App Engine
You can now increase the build time, allowing for the docker image to build for a longer amount of time, with: `gcloud config set app/cloud_build_timeout 2000`
Joshua Crowley
Jul 13, 2019, 3:04:58 PM7/13/19
to Google App Engine
I'm finding that gcloud config set app/cloud_build_timeout is ignored? All my builds timeout after 10 mins. Does it still apply?
David Charles Martinez
Jul 15, 2019, 4:30:11 PM7/15/19
to google-a...@googlegroups.com
Hello Joshua,
About “gcloud config set app/cloud_build_timeout“ command, please follow this Google Issue Tracker where this issue will be investigated as I have confirmed the gcloud command seems not to be working properly. As a workaround, you can try using the [--timeout=DURATION] flag to change the timeout threshold which should work properly at the moment of running the command.
Joshua Crowley
Jul 15, 2019, 9:12:59 PM7/15/19
to Google App Engine
Thanks for looking into it.
ERROR: (gcloud.app.deploy) unrecognized arguments: --timeout=1000 (did you mean '--http-timeout'?)
There's no --timeout argument for gcloud.app.deploy
Workaround I've found for app engine is to use the build command (which follows timeout config), and then deploy the image.
"gcloud builds submit --tag asia.gcr.io/... && gcloud app deploy app.yaml --image-url=asia.gcr.io/...",
David Charles Martinez
Jul 16, 2019, 12:00:34 PM7/16/19
to Google App Engine
Hello Joshua,
Yes, you would need to use "gcloud builds submit" to use the --timeout flag.
Reply all
Reply to author
Forward
0 new messages