Mayank Bhagya
Mar 28, 2016, 5:55:08 AM3/28/16
to Google App Engine
We face this issue every day. And no, I am not exaggerating.
EVERY SINGLE DAY!
Every day, from about 12 in the afternoon to 5 in the evening (GMT+0530), the push-to-deploy bitnami-jenkins setup complains that it couldn't copy files to cloud storage.
The error is intermittent. Certain deployments succeed in first attempt, some others in second and some others don't complete for hours!
Today we have to deploy a critical fix and this thing is not ready to deploy!
Here are some logs:
Updating module [default]...- Updating module [default]...\ Updating module [default]...| Updating module [default].../ Updating module [default]...- Updating module [default]...failed. ERROR: (gcloud.preview.app.deploy) Error Response: [13] Errors were encountered while copying files to App Engine. Details: [ [ { "@type": "type.googleapis.com/google.rpc.ResourceInfo", "description": "Error reading from Google Cloud Storage.", "resourceName": "https://storage.googleapis.com/staging.myappname.appspot.com/9daf160fc37b8d609092a9f47367188d596ab34e.jpeg", "resourceType": "file" } ] ]
Do you think adding more features and not addressing bugs like these will make your customers' life easy?
Whatever happened to quality.
Can Google folks be serious enough and understand the impact of such issues?
Having downtime in an application is not a joke.
Not everybody ships with 'beta' quality.
Please look into this issue asap and let me know if you know anything from my side.
Nick (Cloud Platform Support)
Mar 28, 2016, 12:59:07 PM3/28/16
to Google App Engine
Hey Mayank,
I understand that you're seeing some issues in deployment to via your bitnami-jenkins GCE instance. While connection issues in a cloud environment are to be expected, the tools we use to make connections should be fault-tolerant although this isn't always automatically the case. It appears that some aspect of your deployment system is failing when a connection to Cloud Storage fails. Your post didn't make it quite clear exactly how this manifests, as full system and logs context were not given. However, there is enough information to make clear that you need to contact either Cloud Platform or Bitnami support. You could either try to report it in the Cloud Platform Public Issue Tracker, or use the recommended path from the documentation, which explains:
Either way you will shortly be in touch with someone who can assist you further. Be sure to provide full context of how you run the deployment, how you gather the error, all messages which are output, etc.
If there's anything else you feel I could assist with here, feel free to let me know and I'll do my best to answer any questions.
Sincerely,
Nick
Cloud Platform Community Support
I understand that you're seeing some issues in deployment to via your bitnami-jenkins GCE instance. While connection issues in a cloud environment are to be expected, the tools we use to make connections should be fault-tolerant although this isn't always automatically the case. It appears that some aspect of your deployment system is failing when a connection to Cloud Storage fails. Your post didn't make it quite clear exactly how this manifests, as full system and logs context were not given. However, there is enough information to make clear that you need to contact either Cloud Platform or Bitnami support. You could either try to report it in the Cloud Platform Public Issue Tracker, or use the recommended path from the documentation, which explains:
Free support is provided via our community site by the Bitnami team. For free support, we do not provide a guaranteed response time, however we do our best to respond to questions within 24 hours Monday through Friday.
Go to Bitnami support (community.bitnami.com)
Additional Google Cloud Platform technical support is available as a paid package from Google Cloud Support.
Either way you will shortly be in touch with someone who can assist you further. Be sure to provide full context of how you run the deployment, how you gather the error, all messages which are output, etc.
If there's anything else you feel I could assist with here, feel free to let me know and I'll do my best to answer any questions.
Sincerely,
Nick
Cloud Platform Community Support
Mayank Bhagya
Mar 29, 2016, 2:49:12 AM3/29/16
to Google App Engine
Thanks for the reply, Nick.
While connection issues in a cloud environment are to be expected, the tools we use to make connections should be fault-tolerant although this isn't always automatically the case.
Do you mean we should expect these tools to fail?
Your post didn't make it quite clear exactly how this manifests, as full system and logs context were not given.
Well, the appengine push-to-deploy setup, states very clearly, how to use 'gcloud' on a compute-engine instance to deploy to appengine.
The attached logs were the output of 'gcloud preview app deploy', as visible on line#7 of the logs.
However, there is enough information to make clear that you need to contact either Cloud Platform or Bitnami support. You could either try to report it in the Cloud Platform Public Issue Tracker, or use the recommended path from the documentation, which explains:
That doesn't help a lot Nick.
Either way you will shortly be in touch with someone who can assist you further. Be sure to provide full context of how you run the deployment, how you gather the error, all messages which are output, etc.
Thanks. This should help.
Do let me know if you need more info or if you need me to run gcloud in a debug / verbose mode.
Nick (Cloud Platform Support)
Mar 29, 2016, 9:27:15 AM3/29/16
to Google App Engine
Hey Mayank,
I'm sorry that you found my comments unhelpful. I'll try to clarify:
It is to be expected that errors will occur when dealing with any software product which is beta (such as gcloud). As well, in a distributed (cloud) computing environment, connection issues are a fact of life, even for the best-intentioned developers and companies, and it's not necessarily a matter of attitude toward quality, which I'm sure we all agree is the most important factor. The question becomes, how to deal with inevitable connection error situations? Software should attempt to retry failed connections, generally, using an incremental backoff scheme to prevent flooding the target server. However, when software doesn't behave in this nice way, what's a developer to do? In this case, it appears that a failure to connect to Cloud Storage led to the entire bitnami-jenkins setup failing.
The reason I had directed you to the appropriate channels of support is because in those locations you will find people expressly-oriented to addressing your specific issue here, while this forum is meant for more general discussion of the Google Cloud Platform and services, not specific technical issues. When you post to Bitnami support, or Cloud Platform support (this could also mean the Public Issue Tracker), you should be sure to include all relevant details, such as which command you are running or which documentation you are following, what version of all relevant softwares and OS's are involved, etc. I'm unsure at this point whether you're following documentation on our side or from Bitnami, if you could clarify that would be great.
All that said, I'll be happy to offer whatever assistance I can here, in whatever form I can. Feel free to supply more info and we can try to get to the bottom of this.
Regards,
I'm sorry that you found my comments unhelpful. I'll try to clarify:
It is to be expected that errors will occur when dealing with any software product which is beta (such as gcloud). As well, in a distributed (cloud) computing environment, connection issues are a fact of life, even for the best-intentioned developers and companies, and it's not necessarily a matter of attitude toward quality, which I'm sure we all agree is the most important factor. The question becomes, how to deal with inevitable connection error situations? Software should attempt to retry failed connections, generally, using an incremental backoff scheme to prevent flooding the target server. However, when software doesn't behave in this nice way, what's a developer to do? In this case, it appears that a failure to connect to Cloud Storage led to the entire bitnami-jenkins setup failing.
The reason I had directed you to the appropriate channels of support is because in those locations you will find people expressly-oriented to addressing your specific issue here, while this forum is meant for more general discussion of the Google Cloud Platform and services, not specific technical issues. When you post to Bitnami support, or Cloud Platform support (this could also mean the Public Issue Tracker), you should be sure to include all relevant details, such as which command you are running or which documentation you are following, what version of all relevant softwares and OS's are involved, etc. I'm unsure at this point whether you're following documentation on our side or from Bitnami, if you could clarify that would be great.
All that said, I'll be happy to offer whatever assistance I can here, in whatever form I can. Feel free to supply more info and we can try to get to the bottom of this.
Regards,
Nick
Cloud Platform Community Support
Mayank Bhagya
Mar 30, 2016, 1:59:15 AM3/30/16
to Google App Engine
Thanks Nick.
It is to be expected that errors will occur when dealing with any software product which is beta (such as gcloud). As well, in a distributed (cloud) computing environment, connection issues are a fact of life, even for the best-intentioned developers and companies, and it's not necessarily a matter of attitude toward quality, which I'm sure we all agree is the most important factor.
I understand that connection errors are a common thing. But if an error occurs for five hours EVERY DAY, then it should probably be looked at with some importance than just waiving it off, attributing it to beta quality or cloud environments.
The question becomes, how to deal with inevitable connection error situations? Software should attempt to retry failed connections, generally, using an incremental backoff scheme to prevent flooding the target server. However, when software doesn't behave in this nice way, what's a developer to do?
I understand what fault tolerance is Nick. Thanks for explaining.
In this case, it appears that a failure to connect to Cloud Storage led to the entire bitnami-jenkins setup failing.
The reason I had directed you to the appropriate channels of support is because in those locations you will find people expressly-oriented to addressing your specific issue here, while this forum is meant for more general discussion of the Google Cloud Platform and services, not specific technical issues.
Alright, I'll write to the google-cloud-sdk group.
When you post to Bitnami support, or Cloud Platform support (this could also mean the Public Issue Tracker), you should be sure to include all relevant details, such as which command you are running or which documentation you are following, what version of all relevant softwares and OS's are involved, etc.
'bitnami-jenkins' is a pre-built image offered on google cloud. So the OS is fixed. It comes with google-cloud-sdk installed. So that is also fixed. I'm unsure at this point whether you're following documentation on our side or from Bitnami, if you could clarify that would be great.
As far as the command is concerned, it is 'gcloud preview app deploy', as visible in the logs.
The entire thing was setup and functional as per Google's article on 'push-to-deploy' setup.
This was setup a few months ago and has been performing on and off with this error.
The documentation regarding the same has disappeared from the docs now. See this.
Anyway, thanks for all the help.
I'll write this on the google-cloud-sdk group.
Nick (Cloud Platform Support)
Mar 30, 2016, 1:11:28 PM3/30/16
to Google App Engine
Hey Mayank,
I understand that you've been seeing an error for five hours every day, on and off for a few months. This is obviously a problem which is worth investigating, and I'm doing everything I can to make sure the issue is reported and dealt with in the most efficient, clear, and appropriate way possible. This is why I recommended that the full issue detail be posted, and in a location where the highest probability of detailed assistance is available - that would be bitnami support or a Cloud Platform Public Issue Tracker, or Cloud Platform Support. Posting to the Cloud SDK Groups forum is probably not the best location, since it's unlikely the error is originating in the Cloud SDK - the Cloud SDK is merely reporting that an error occurred (and again it's worth posting the full debug output to potentially see more information).
As said before, I'll be happy to offer whatever assistance I can here, in whatever form I can. Feel free to supply more info and we can try to get to the bottom of this.
I understand that you've been seeing an error for five hours every day, on and off for a few months. This is obviously a problem which is worth investigating, and I'm doing everything I can to make sure the issue is reported and dealt with in the most efficient, clear, and appropriate way possible. This is why I recommended that the full issue detail be posted, and in a location where the highest probability of detailed assistance is available - that would be bitnami support or a Cloud Platform Public Issue Tracker, or Cloud Platform Support. Posting to the Cloud SDK Groups forum is probably not the best location, since it's unlikely the error is originating in the Cloud SDK - the Cloud SDK is merely reporting that an error occurred (and again it's worth posting the full debug output to potentially see more information).
As said before, I'll be happy to offer whatever assistance I can here, in whatever form I can. Feel free to supply more info and we can try to get to the bottom of this.
Sincerely,
Nick
Cloud Platform Community Support
Reply all
Reply to author
Forward
0 new messages