Stackdriver Uptime Check won't save..

[SD User]

..the test works fine, but will not save. I get this error. Google search knows nothing about it. Any suggestions? The VM has been running for days, but occasionally stops, and I want to know when it does. I also tried resetting the VM and installing the SD Agent for monitoring.

"There was an error saving the uptime check config: Internal error storing config in project..."

[Mary Koes]

+Daniel Klein 

On Tue, Sep 17, 2019 at 3:20 PM SD User <thisyisyf...@gmail.com> wrote:

..the test works fine, but will not save. I get this error. Google search knows nothing about it. Any suggestions? The VM has been running for days, but occasionally stops, and I want to know when it does. I also tried resetting the VM and installing the SD Agent for monitoring.

"There was an error saving the uptime check config: Internal error storing config in project..."

--
© 2016 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 
Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdr...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.
---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-d...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-stackdriver-discussion/9d966b72-9b2b-45fc-8f14-71c5011c660a%40googlegroups.com.

[SD User]

I wanted to add that I tested the Save function from a generic website 'http://whatsmyrouterip.com/' which also gave the error. And that the Uptime Checks actually seem to be saved because they are listed on the Overview, but clicking into them they say no checks have been run and no data, and I'm not able to edit them, though I have double-checked the settings.  Thank you

[Daniel Klein]

We're aware of it, it is an issue in Prober Service, it should be fixed within 30m according to PS-oncall.

____________

Daniel V. Klein | Cloud Uptime PM (on rotation from SWE), ex-SRE

d...@google.com | 412-370-0862 (feel free to call if it is urgent)

Concurrency is ha12rd.  ramout 10 progmers w get itrong.  ¯\_(ツ)_/¯

[Daniel Klein]

The underlying problem has been resolved, and we've verified that all checks are now saving correctly from all locations.

____________

Daniel V. Klein | Cloud Uptime PM (on rotation from SWE), ex-SRE

d...@google.com | 412-370-0862 (feel free to call if it is urgent)

Concurrency is ha12rd.  ramout 10 progmers w get itrong.  ¯\_(ツ)_/¯

[Daniel Klein]

I'd like to take this opportunity to apologize both for the brevity of my responses and for the inconvenience that you experienced. I am the new PM for Uptime, and initially we thought that this was an internal issue only. I'm sorry that it bled over and affected your user-experience, too! Your timely feedback was very helpful for us in identifying the root cause, and I'd like to thank you for taking the time to report it (and for using Stackdriver)! If you have any further issues or questions, please let me know.

____________

Daniel V. Klein | Cloud Uptime PM (on rotation from SWE), ex-SRE

d...@google.com | 412-370-0862 (feel free to call if it is urgent)

Concurrency is ha12rd.  ramout 10 progmers w get itrong.  ¯\_(ツ)_/¯

[SD User]

Thanks Daniel, the save now works.. Unfortunately, I now have "All Locations Failed" on my check even though after resetting the VM the webpage is functioning fine for several minutes (tried refreshing pages, etc.)

I'm also getting "Responded with Connection Error - Host Unreachable in 10000 ms." on the new check I created, again while the webpage is working. I created a new check because there was no way to manually run (test) the original nor edit the any parameters (those would be very helpful features to add). Will trudge on to see what the problem is.. (a debug or error log would be helpful here). ..I don't think there is any way for me to debug. I will wait another hour, try again, reset the VM, and try again.

+Daniel Klein 

Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdriver-discu...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.

---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-discussion+unsub...@googlegroups.com.

[SD User]

Thanks Daniel, the save now works.. Unfortunately, I now have "All Locations Failed" on my check even though after resetting the VM the webpage is functioning fine for several minutes (tried refreshing pages, etc.)

I'm also getting "Responded with Connection Error - Host Unreachable in 10000 ms." on the new check I created, again while the webpage is working. I created a new check because there was no way to manually run (test) the original nor edit the any parameters (those would be very helpful features to add). Will trudge on to see what the problem is.. (a debug or error log would be helpful here). ..I don't think there is any way for me to debug. I will wait another hour, try again, reset the VM, and try again.

On Tuesday, September 17, 2019 at 12:20:57 PM UTC-7, SD User wrote:

[SD User]

Thanks Daniel, the save now works.. Unfortunately, I now have "All Locations Failed" on my check even though after resetting the VM the webpage is functioning fine for several minutes (tried refreshing pages, etc.)

I'm also getting "Responded with Connection Error - Host Unreachable in 10000 ms." on the new check I created, again while the webpage is working. I created a new check because there was no way to manually run (test) the original nor edit the any parameters (those would be very helpful features to add). Will trudge on to see what the problem is.. (a debug or error log would be helpful here). ..I don't think there is any way for me to debug. I will wait another hour, try again, reset the VM, and try again.

On Tuesday, September 17, 2019 at 2:36:35 PM UTC-7, Daniel Klein wrote:

+Daniel Klein 

Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdriver-discu...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.

---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-discussion+unsub...@googlegroups.com.

[SD User]

Sorry, for the double post. I was getting 

"An error occurred while communicating with the server.
RELOAD
from the forum site after submitting the Captcha.. the computing gods are not with me today apparently. ..actually still getting this error :(

[SD User]

Uptime Checks are still not connecting. Even after creating a new check to 'test' the same page. Page is working on browser and with output via SSH.

I just 'tested' google.com on port 80, which is working. My webservice is setup by default on a different port. This may be part of why it's not connecting, although it did work properly within the last several days.

[SD User]

Uptime Checks are still not connecting. Even after creating a new check to 'test' the same page. Page is working on browser and with output via SSH.

I just 'tested' google.com on port 80, which is working. My webservice is setup by default on a different port. This may be part of why it's not connecting, although it did work properly within the last several days.

[SD User]

I just tried setting up a check with the webservice FollowThatPage and it returned: "905 Server 'xxx.xxx.xxx.xxx:xxx' not found in the DNS system."

So it may be the DNS is parsing the port out correctly. But again it worked on 4 or 5 checks I created using the 'test' function when the save function wasn't working, so I'm thinking something changed within SD.

[Nikki Oyekunbi]

Hello,

  Can you please file feedback via the Stackdriver UI? That will allow us to properly investigate the issue.

Thanks,

        Nikki

--

© 2016 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 

Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdr...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.

---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-d...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-stackdriver-discussion/389eacbe-f562-4580-972a-f2b7baf14066%40googlegroups.com.

[Daniel Klein]

So, I see a few issues that I can comment on:

1. No worries about the double posts :-)
2. There may have been a database synchronization issue involving checks that were unsuccessfully changed during the problem you reported and then successfully changed after the problem. This is possibly why your specific checks didn't run. Resetting your VM is probably not the issue, but we just recently implemented a synchronization fix, and hopefully this will resolve your problems (please let us know).
3. Regarding the "Responded with Connection Error - Host Unreachable in 10000 ms", there could be firewall issues, DNS issues (which you have seen), or port issues with your test target, and as Nikki suggested, feedback via the UI will help us find and fix the problem.

Thank you again for your patience and understanding (and for being willing to work with us on this), and we'll try to help you get everything resolved as soon as possible!

[SD User]

Thanks again Daniel. So, this is interesting.. I realized it might be my firewall rules, and sure enough it was. I now have exceptions for 

IP ranges

35.239.0.0/16

35.199.0.0/16

Are these the main StackDriver Uptime Check IPs?

The interesting part is that the test checks were connecting to the webservice before. That remains a mystery..

[SD User]

And 35.185.0.0 and 35.186.0.0. I only have US origins selected. 

It would be *really* great if GCP had a 'country block' IP filter for the firewall. As it is I am the only one accessing this site. But when I had all IPs allowed I was getting massive scans from Russia and China (and California) every minute. I'm assuming the GCP ranges are a high value target. In the future, I would love to have a more public site, but disallow origins from 'bad actor' states. It doesn't solve everything, but at least limiting to the US would limit to servers protected by the rule of law. (As an aside, I think this is what needs to be done with Facebook). As an aside, my ASUS router has a function that blocks port scans. This would also be helpful with security.

It would be great to have a default firewal filter for StackDriver like there is for SSH and RDP (isn't this a security risk? I turn both off when not using.

[Daniel Klein]

Oh dear! You don't need to open up whole netblocks, you can specify the exact list of IP addresses we use. They are available through an API call, uptimeCheckIps.list (and that page will also give you the API explorer which will allow you to run the API call in your browser and get a list top copy/paste. I hope this helps!

[SD User]

Thanks. Is there a tool to translate this json into the firewall rule ip field? Or is that an SSH thing with Iptables?

[SD User]

This will find/replace the json fluff, but have to replace the location. Then you get the exalted blessing of entering them all in individually, because the field wont parse separators. 

"\n    },\n    {\n      "region": "USA",\n      "location": "Oregon",\n      "ipAddress": "

There are a couple pure regex solutions on the web, but very time consuming.

[Daniel Klein]

My apologies for the delay in replying to this... I found a shortcut for you. It is not a perfect solution to your problem, but it helps:

If you go to https://app.google.stackdriver.com/uptime and click on the download icon on the top right, you'll get a file that looks like this:

{"ipAddress":"35.203.129.73","region":"USA","location":"Oregon"},{"ipAddress":"35.185.252.44","region":"USA","location":"Oregon"},{"ipAddress":"35.233.165.146","region":"USA","location":"Oregon"},{"ipAddress":"35.186.164.184","region":"USA","location":"Virginia"},{"ipAddress":"35.188.230.101","region":"USA","location":"Virginia"},

...

Since every address is on one long line, tweaking the contents should be pretty easy (with or without a regex), and you can get your list of IP addresses much quicker. The regex solution is to

1,$s/},/\n/g

1,$s/[^0-9.]*//g

And to make it easier for you, here is the current list of IPs that I extracted while testing my solution:

23.251.144.62

35.185.178.105

35.185.252.44

35.186.144.97

35.186.159.51

35.186.164.184

35.186.167.85

35.186.176.31

35.187.114.193

35.187.242.246

35.188.230.101

35.192.92.84

35.195.128.75

35.197.117.125

35.197.32.224

35.198.18.224

35.198.194.122

35.198.221.49

35.198.224.104

35.198.248.66

35.198.36.209

35.198.39.162

35.199.12.162

35.199.123.150

35.199.126.168

35.199.157.7

35.199.27.30

35.199.66.47

35.199.67.79

35.199.77.186

35.199.90.14

35.203.129.73

35.203.157.42

35.205.205.242

35.205.234.10

35.205.72.231

35.221.55.249

35.224.249.156

35.233.165.146

35.233.167.246

35.233.206.171

35.236.207.68

35.236.221.2

35.238.118.210

35.238.3.7

35.239.194.85

35.240.124.58

35.240.151.105

104.155.110.139

104.155.77.122

104.197.30.241

146.148.119.250

146.148.41.163

146.148.59.114