docjohn

Long-time site in Google and no previous problems. Only started paying
attention to Google's webmaster tools as they've improved them this
year (thank you!).

Just noticed that Google hasn't had a successful crawl on my site
since Sept. 5. The error message for the main page is:

"We can't currently access your home page because of a timeout."

The site is online and functional 24/7 with significant traffic (even
from Google's Adsensebot crawl servers, no problemo). But Googlebot
has nearly stopped completely, down from hundreds of pages crawled per
day to just a dozen or two since 9/5.

So without any further information, I decided I'd better upload a
sitemap and see if that would give me further info. Well, it did:

"URL timeout: DNS lookup timeout"

The problem is, DNS lookups are fine for my domain. We use a third-
party enterprise DNS hosting service that lists 5 geographically-
diverse DNS nameservers for our domain. Naturally, these are all up
and running fine. DNStools reports no problems with either our domain
or its DNS.

Looking in the log files for Googlebot instances shows that the few
times it is crawling these days, it's downloading the files fine
(e.g., byte count and result code is what is expected and what they've
always been).

So a few suggestions come to mind from this experience:

1. Google should add more detail to the default error message that is
reported on the dashboard's main page for a URL about the specific
timeout issue. I only discovered additional detailed information about
the error from uploading a sitemap.

2. Error messages are fairly useless to a webmaster unless they
suggest a troubleshooting procedure. Luckily, I know the
troubleshooting procedure for DNS timeouts and now I've done
everything I can do on my side of things... What's next? How do I
contact Google about this problem? (I assume this is it.)

3. A button that says, "Okay, I've fixed the error, please clear your
cache and crawl again next time you're scheduled to do so" would be
great. I assume it's not there because Google is *always* crawling and
automatically rechecks these things on its own. That's cool, but it
appears my domain is stuck in some sort of error cache thing or
something on Google's search crawl side (while Google Adsensebot
continues to churn along just fine). I hope/assume this will fix
itself, since there doesn't appear to be anything else I can do on my
end.

4. Google's internal systems and bots should be able to talk to one
another. Google Adsensebot continues to function fine, while Googlebot
thinks my domain's DNS is broken. If Googlebot could talk to Google
Adsensebot, they'd say, "Hey, I'm getting through just fine. Try
hitting it from a different IP address or something and clear your
error cache."

John