App returning 404s since Feb 18

[jhf555]

I run a website on App Engine and since what looks like Feb 18 it has just been returning 404s.  I haven't changed anything that I know if in quite a while so I'm not sure why this would suddenly have started.  And if I visit the "appspot" version of the site directly, it works fine.  The sites are:

404ing:

http://www.bigriddles.com/

And here's the working version:

https://bigriddles-hrd.appspot.com/

I'm digging around the app engine console but don't see anything obvious yet.  My billing info looks up to date.  DNS for www.bigriddles appears to be pointing to the right place.  The Quota page isn't showing any obvious issues for being over quotas.  Any ideas what could be wrong?

Thanks,

Jonathan

[jhf555]

I scrolled down to posts from the day this started happening to me and it looks like it hit others as well.  I'll try out their fix: https://groups.google.com/forum/#!msg/google-appengine/LhQ2NEydn0w/tt6v0-RzBAAJ

[jhf555]

I've added a TXT record for my domain as per the other post, though it doesn't work (at least not yet).  Would appreciate some help from someone at Google.  Thanks!

[jhf555]

Latest update: GoDaddy won't let me add the TXT record...it says it's being added but when I refresh (even a day later) per the instructions on https://support.google.com/quickfixes/answer/6252336?hl=en&rd=1, the record isn't actually there.  I'm not sure if this is because Google itself is managing the DNS account and I can't make certain changes...whatever it is, it's confusing.

I also tried upgrading my app to use Python 2.7 (from 2.5) and redeploying but this still causes 404s.

Next step is going to be transferring my domain away from Google/GoDaddy to see if I can actually control the DNS.  

[Adam (Cloud Platform Support)]

You can see that the domain is still hosted on GoDaddy's servers if you do a 'dig' on the domain or use a site like network-tools.com to do a lookup on the domain:

;ANSWER

bigriddles.com. 3599 IN NS ns50.domaincontrol.com.

bigriddles.com. 3599 IN NS ns49.domaincontrol.com.

No TXT record though. Again you can use network-tools.com or 'dig' to see if it's there. If the record isn't being published, the Google verification tool isn't going to work either.

It's possible that the record may not be entered correctly or there is a problem with GoDaddy's DNS servers or your DNS configuration. Posting a screen capture of how the TXT record looks in your DNS control panel may help.

[Adam (Cloud Platform Support)]

As a follow up, it's also a good idea to double check your Developers Console settings to make sure it's set up properly. 

[jhf555]

Thanks for the response Adam.  I'm actually gonna transfer my domain away from GoDaddy altogether...I just initiated the domain transfer this morning so it will be a few days before it's completed.  I still have no idea why their UI said I had added the TXT record successfully, but it didn't stick for whatever reason.  Even so, I'm not sure why I'd actually need that TXT record.  I bought the domain through Google (which used GoDaddy as the registrar) and I don't remember needing to do anything further to verify the domain when I set up my app (and I'm not using SSL).  That being said, I set this up over 5 years ago I think so I might be forgetting something.

Ignoring DNS for the moment, the 404s are puzzling, and I couldn't find any log messages in the Google Cloud Log Viewer that tipped me off as to why they suddenly started.  Everything worked fine locally when running with dev_appserver.py.  My best guess was that something recently got deprecated that I was using, so I decided to try to update my app to more modern libraries (my app was still using Python 2.5).  I'm now down the rabbit hole of actually trying to upgrade everything to see if that fixes things.  I'm also adding extra logging to my app to see if I can figure out where the 404 gets triggered.

Not sure if you have the ability to look, but is there likely a good way to figure out what's causing the 404s other than just adding extensive logging?  The app-engine console shows me nothing useful that I can find other than the graph of 404s, and like I said everything works locally.  I tried curling as well to see if you send back any debug headers but didn't see anything.

Thanks.

[jhf555]

I've updated to Python 2.7 and gotten things working locally, but still getting 404s on the server.  Given that the appstpot url is working directly, maybe this really is a domain verification problem.  I'm running into lots of roadblocks again, which is frustrating.  Like I said, I can't add a TXT record for some reason.  And now I'm trying to follow the instructions on

https://www.google.com/webmasters/verification/verification?authuser=0&domain=bigriddles.com

to verify the domain by signing in to GoDaddy in the pop-up window.  And for whatever reason the GoDaddy credentials supplied by Google in the admin console do not work in that popup.

[Adam (Cloud Platform Support)]

The 404s would not show up in your Developers Console logs since the request is not even making it to the app server. The CNAME resolves to ghs.google.com which relies on an internal mapping to route traffic to App Engine.

Unfortunately the mapping will not work if the domain is not verified. Since no TXT record existed on the domain this is a good sign that the verification was never completed previously. The only way to fix this is to complete the verification process.

Likely the reason why your credentials don't work in the Webmaster Central popup is because they are for a different login which is only accessible through the Domains section of the Google Apps admin console (since this is a Google Apps domain).

[jhf555]

Thanks again for your replies Adam.  I finally got my app working again today and figured I'd post an update.

So the problem was in fact the domain no longer being "verified" in Google's eyes.  Like I mentioned, when I originally read about some other folks running into this problem, the first thing I tried was to add the verification TXT record through GoDaddy, but despite their UI claiming the record was being added correctly, it wasn't actually saved when I reloaded their DNS UI, and so I couldn't verify that way.  I ended up just moving my domain registration over to Gandi, and this morning I was able to add the TXT record there, verify the domain, and "add" the domain back in via the Google Cloud UI for my appengine app, and things started working again.  

Overall this was a bummer, and I'm not sure how much SEO karma I lost from my site being down for so long.  I guess I'll find out in the coming weeks.

A lot of things went wrong here, ranging from minor to major: 

- Why did my domain suddenly become unverified?  Given that this happened to others on the same day (see the thread I linked to from my original post), I have to assume this was either a bug or a poorly-communicated change of some sort.  FWIW, in the Google Webmasters UI, it said that the domain had previously been verified by an HTML meta tag some years ago.

- It would have been helpful if the 404 Google returned gave some hint as to what was wrong.  I ended up having to search these forums and find people complaining on the same date where my outage started to get a hint at the problem.  If Google returned a message or an HTTP header with the 404 with the problem ("The domain name you used on this request isn't verified for this app." or something) it would have been really helpful.  Even though I was pointed to this problem early on, I wasn't sure it was the issue and ended up down many other rabbit holes.

- Why wouldn't GoDaddy let me add my TXT record?  (not Google's fault I'm pretty sure)

- Minor issue, but for reasons I don't understand the new UI for Google App Engine (Google Cloud Console) sometimes only showed me "Dashboard" and "Activity" in the left nav (instead of all the options, like Versions, Instances, Settings, etc..).  Not sure why, but it made it harder to find certain things like the settings page.

Jonathan