Long-time site in Google and no previous problems. Only started paying attention to Google's webmaster tools as they've improved them this year (thank you!).
Just noticed that Google hasn't had a successful crawl on my site since Sept. 5. The error message for the main page is:
"We can't currently access your home page because of a timeout."
The site is online and functional 24/7 with significant traffic (even from Google's Adsensebot crawl servers, no problemo). But Googlebot has nearly stopped completely, down from hundreds of pages crawled per day to just a dozen or two since 9/5.
So without any further information, I decided I'd better upload a sitemap and see if that would give me further info. Well, it did:
"URL timeout: DNS lookup timeout"
The problem is, DNS lookups are fine for my domain. We use a third- party enterprise DNS hosting service that lists 5 geographically- diverse DNS nameservers for our domain. Naturally, these are all up and running fine. DNStools reports no problems with either our domain or its DNS.
Looking in the log files for Googlebot instances shows that the few times it is crawling these days, it's downloading the files fine (e.g., byte count and result code is what is expected and what they've always been).
So a few suggestions come to mind from this experience:
1. Google should add more detail to the default error message that is reported on the dashboard's main page for a URL about the specific timeout issue. I only discovered additional detailed information about the error from uploading a sitemap.
2. Error messages are fairly useless to a webmaster unless they suggest a troubleshooting procedure. Luckily, I know the troubleshooting procedure for DNS timeouts and now I've done everything I can do on my side of things... What's next? How do I contact Google about this problem? (I assume this is it.)
3. A button that says, "Okay, I've fixed the error, please clear your cache and crawl again next time you're scheduled to do so" would be great. I assume it's not there because Google is *always* crawling and automatically rechecks these things on its own. That's cool, but it appears my domain is stuck in some sort of error cache thing or something on Google's search crawl side (while Google Adsensebot continues to churn along just fine). I hope/assume this will fix itself, since there doesn't appear to be anything else I can do on my end.
4. Google's internal systems and bots should be able to talk to one another. Google Adsensebot continues to function fine, while Googlebot thinks my domain's DNS is broken. If Googlebot could talk to Google Adsensebot, they'd say, "Hey, I'm getting through just fine. Try hitting it from a different IP address or something and clear your error cache."
> Long-time site in Google and no previous problems. Only started paying > attention to Google's webmaster tools as they've improved them this > year (thank you!).
> Just noticed that Google hasn't had a successful crawl on my site > since Sept. 5. The error message for the main page is:
> "We can't currently access your home page because of a timeout."
> The site is online and functional 24/7 with significant traffic (even > from Google's Adsensebot crawl servers, no problemo). But Googlebot > has nearly stopped completely, down from hundreds of pages crawled per > day to just a dozen or two since 9/5.
> So without any further information, I decided I'd better upload a > sitemap and see if that would give me further info. Well, it did:
> "URL timeout: DNS lookup timeout"
> The problem is, DNS lookups are fine for my domain. We use a third- > party enterprise DNS hosting service that lists 5 geographically- > diverse DNS nameservers for our domain. Naturally, these are all up > and running fine. DNStools reports no problems with either our domain > or its DNS.
> Looking in the log files for Googlebot instances shows that the few > times it is crawling these days, it's downloading the files fine > (e.g., byte count and result code is what is expected and what they've > always been).
> So a few suggestions come to mind from this experience:
> 1. Google should add more detail to the default error message that is > reported on the dashboard's main page for a URL about the specific > timeout issue. I only discovered additional detailed information about > the error from uploading a sitemap.
> 2. Error messages are fairly useless to a webmaster unless they > suggest a troubleshooting procedure. Luckily, I know the > troubleshooting procedure for DNS timeouts and now I've done > everything I can do on my side of things... What's next? How do I > contact Google about this problem? (I assume this is it.)
> 3. A button that says, "Okay, I've fixed the error, please clear your > cache and crawl again next time you're scheduled to do so" would be > great. I assume it's not there because Google is *always* crawling and > automatically rechecks these things on its own. That's cool, but it > appears my domain is stuck in some sort of error cache thing or > something on Google's search crawl side (while Google Adsensebot > continues to churn along just fine). I hope/assume this will fix > itself, since there doesn't appear to be anything else I can do on my > end.
> 4. Google's internal systems and bots should be able to talk to one > another. Google Adsensebot continues to function fine, while Googlebot > thinks my domain's DNS is broken. If Googlebot could talk to Google > Adsensebot, they'd say, "Hey, I'm getting through just fine. Try > hitting it from a different IP address or something and clear your > error cache."
Long time indexed, no problems, then 9/5 suddenly we start getting the 'hompage timeout' errors. Same with the site map-- DNS timeout. But Yahoo! gets the sitemap just fine, and we didn't change anything on our end-- neither in DNS zone files, nor in server setup, nor the sitemap files themselves.
Googlebot is still pulling down other files from our server, just not the sitemap and index (and robots). Indexing is down *significantly*-- so this is not just a funny coincidence.
The last download of robots.txt was: September 5, 2007 3:49:14 PM PDT. Does this match yours?
Any further thoughts? I can't find a way to get in touch with google.
> Long-time site in Google and no previous problems. Only started paying > attention to Google's webmaster tools as they've improved them this > year (thank you!).
> Just noticed that Google hasn't had a successful crawl on my site > since Sept. 5. The error message for the main page is:
> "We can't currently access your home page because of a timeout."
> The site is online and functional 24/7 with significant traffic (even > from Google's Adsensebot crawl servers, no problemo). But Googlebot > has nearly stopped completely, down from hundreds of pages crawled per > day to just a dozen or two since 9/5.
> So without any further information, I decided I'd better upload a > sitemap and see if that would give me further info. Well, it did:
> "URL timeout: DNS lookup timeout"
> The problem is, DNS lookups are fine for my domain. We use a third- > party enterprise DNS hosting service that lists 5 geographically- > diverse DNS nameservers for our domain. Naturally, these are all up > and running fine. DNStools reports no problems with either our domain > or its DNS.
> Looking in the log files for Googlebot instances shows that the few > times it is crawling these days, it's downloading the files fine > (e.g., byte count and result code is what is expected and what they've > always been).
> So a few suggestions come to mind from this experience:
> 1. Google should add more detail to the default error message that is > reported on the dashboard's main page for a URL about the specific > timeout issue. I only discovered additional detailed information about > the error from uploading a sitemap.
> 2. Error messages are fairly useless to a webmaster unless they > suggest a troubleshooting procedure. Luckily, I know the > troubleshooting procedure for DNS timeouts and now I've done > everything I can do on my side of things... What's next? How do I > contact Google about this problem? (I assume this is it.)
> 3. A button that says, "Okay, I've fixed the error, please clear your > cache and crawl again next time you're scheduled to do so" would be > great. I assume it's not there because Google is *always* crawling and > automatically rechecks these things on its own. That's cool, but it > appears my domain is stuck in some sort of error cache thing or > something on Google's search crawl side (while Google Adsensebot > continues to churn along just fine). I hope/assume this will fix > itself, since there doesn't appear to be anything else I can do on my > end.
> 4. Google's internal systems and bots should be able to talk to one > another. Google Adsensebot continues to function fine, while Googlebot > thinks my domain's DNS is broken. If Googlebot could talk to Google > Adsensebot, they'd say, "Hey, I'm getting through just fine. Try > hitting it from a different IP address or something and clear your > error cache."
Suddenly Google is giving us an ERROR message when we try to update our sitemap.xml files on Google Webmaster Tools.
Following is my online chat with Adam, our hosting company's support rep.
Question: Google seems to be having a problem accessing our files. Each of our sites is getting a "URL timeout." We know for certain that all our files are right. Bluehost support rep the other day told me that it was probably an issue that our non-Bluehost domain registrars need to fix, but in fact that is very unlikely. Most likely is that we need Bluehost to adjust folder permissions or firewall settings to allow Google access.
Adam: Hello! Welcome to our live chat. Please allow me a moment to read over your question. Keep in mind that we take multiple chats at one time so my response to your questions may take a few moments.
Lawrence: Google seems to be having a problem accessing our files. Each of our sites is getting a "URL timeout." We know for certain that all our files are right. Bluehost support rep the other day told me that it was probably an issue that our non-Bluehost domain registrars need to fix, but in fact that is very unlikely. Most likely is that we need Bluehost to adjust folder permissions or firewall settings to allow Google access.
Adam: As long as you don't have a robots.txt file blocking google you should be fine. We wouldn't block google's bot for any reason.
Lawrence: Hmmm... However, it's not fine. :-( Here's the error message: "URL timeout: DNS lookup timeout We encountered an error while trying to access your Sitemap. Please ensure your Sitemap follows our guidelines and can be accessed at the location you provided and then resubmit" Are you familiar with Google sitemaps.xml files?
Adam: Yes.
Lawrence: We have four different websites hosted on Bluehost. I setup sitemap.xml files on each last year to help Google. They were able to accept them without problem until recently. Nothing changed on our side. As you know, it's nearly impossible to open a support ticket with Google. They refer users to Google Groups instead. This issue was discussed by the Google Webmaster Help group
Adam: Yeah, I've noticed that.
Lawrence: each solution named something to do with hosting Have no other Bluehost clients encountered the issue?
Adam: Not that I am aware of, no.
Lawrence: You know, it's always a mystery how Google works, and we're all doing guesswork all the time. Some have suggested that since DNS is mentioned it may have to do with the domains' registrars But we have multiple registrars for different sites and I find it unlikely that all would become problematic at the same time. Know what I mean? It seems more likely that there would be a permissions or firewall snafu on Bluehost (where all these sites are hosted) Or, the problem is with Google, in which case I'll have to throw up my hands and just give up having a business. (and die penniless in the gutter somewhere)
Adam: I'm afraid we really wouldn't gain anything from blocking googlebot :(
Lawrence: Now, googlebot indexes web pages, right?
Adam: Yes, that is the part of google that does that.
Lawrence: but when we request Google to come and download our latest updated sitemap.xml do they index it the same way? same IP as their normal googlebot? what i mean is,
Adam: Yes, although I've never seen it come from the same IP address. That's why the robots.txt file is important if you want to block it.
Lawrence: i don't actually know whether they're indexing our web pages okay this week. maybe or maybe not, i don't know. But what i do know is that they are unable to access our sitemap.xml files we don't use robots.txt
Adam: Are they in the same location you stated when you submitted the first time?
Lawrence: yes in fact, 3 out of 4 of them haven't been changed at all since the last successful grabbing by Google so, i can only conclude that it's something changed at Google or something changed at Bluehost, (or some new and unfortunate interaction effect between the two) one day they're all fine, then suddenly all ERROR here's a blogger who encountered the same problem http://blog.mcnicholl.com/2007/02/15/google-sitemaps-url-timeout-dns-...
Adam: Although I can't completely rule us out, I would suggest trying to get an answer out of google.
Lawrence: hmmm... okay will i get a transcript of this chat after i close the window?
Adam: I'll continue asking around, but I'm not aware of any change that we have made that would cause this. Yes, after you close the chat it will ask you if you want it to be emailed to you.
Lawrence: Adam, thanks for your help so far... I'll see what response I get from Google bye for now
Naively, I tried sending my question to [email address]
What I got back was:
"Thank you for writing to Google. We'd like to assist you, but we only respond to messages submitted through our online contact form. Please visit http://www.google.com/support/ to submit your message, and we'll get back to you soon. We apologize for any inconvenience, and we look forward to hearing from you.
Regards, The Google Team"
So, when you go to google.com/support looking for the "online contact form" to "submit your message," guess what you find? A never-ending network of FAQs and no contact form anywhere where you can submit your message.
> Suddenly Google is giving us an ERROR message when we try to update > our sitemap.xml files on Google Webmaster Tools.
> Following is my online chat with Adam, our hosting company's support > rep.
> Question: Google seems to be having a problem accessing our files. > Each of our sites is getting a "URL timeout." We know for certain that > all our files are right. Bluehost support rep the other day told me > that it was probably an issue that our non-Bluehost domain registrars > need to fix, but in fact that is very unlikely. Most likely is that we > need Bluehost to adjust folder permissions or firewall settings to > allow Google access.
> Adam: Hello! Welcome to our live chat. Please allow me a moment to > read over your question. Keep in mind that we take multiple chats at > one time so my response to your questions may take a few moments.
> Lawrence: Google seems to be having a problem accessing our files. > Each of our sites is getting a "URL timeout." We know for certain that > all our files are right. Bluehost support rep the other day told me > that it was probably an issue that our non-Bluehost domain registrars > need to fix, but in fact that is very unlikely. Most likely is that we > need Bluehost to adjust folder permissions or firewall settings to > allow Google access.
> Adam: As long as you don't have a robots.txt file blocking google you > should be fine. We wouldn't block google's bot for any reason.
> Lawrence: Hmmm... However, it's not fine. :-( > Here's the error message: > "URL timeout: DNS lookup timeout We encountered an error while trying > to access your Sitemap. Please ensure your Sitemap follows our > guidelines and can be accessed at the location you provided and then > resubmit" > Are you familiar with Google sitemaps.xml files?
> Adam: Yes.
> Lawrence: We have four different websites hosted on Bluehost. > I setup sitemap.xml files on each last year to help Google. > They were able to accept them without problem until recently. > Nothing changed on our side. > As you know, it's nearly impossible to open a support ticket with > Google. They refer users to Google Groups instead. > This issue was discussed by the Google Webmaster Help group
> Adam: Yeah, I've noticed that.
> Lawrence: each solution named something to do with hosting > Have no other Bluehost clients encountered the issue?
> Adam: Not that I am aware of, no.
> Lawrence: You know, it's always a mystery how Google works, and we're > all doing guesswork all the time. > Some have suggested that since DNS is mentioned it may have to do with > the domains' registrars > But we have multiple registrars for different sites and I find it > unlikely that all would become problematic at the same time. Know what > I mean? > It seems more likely that there would be a permissions or firewall > snafu on Bluehost (where all these sites are hosted) > Or, the problem is with Google, in which case I'll have to throw up my > hands and just give up having a business. (and die penniless in the > gutter somewhere)
> Adam: I'm afraid we really wouldn't gain anything from blocking > googlebot :(
> Lawrence: Now, googlebot indexes web pages, right?
> Adam: Yes, that is the part of google that does that.
> Lawrence: but when we request Google to come and download our latest > updated sitemap.xml do they index it the same way? same IP as their > normal googlebot? > what i mean is,
> Adam: Yes, although I've never seen it come from the same IP address. > That's why the robots.txt file is important if you want to block it.
> Lawrence: i don't actually know whether they're indexing our web pages > okay this week. maybe or maybe not, i don't know. But what i do know > is that they are unable to access our sitemap.xml files > we don't use robots.txt
> Adam: Are they in the same location you stated when you submitted the > first time?
> Lawrence: yes > in fact, 3 out of 4 of them haven't been changed at all since the last > successful grabbing by Google > so, i can only conclude that it's something changed at Google or > something changed at Bluehost, (or some new and unfortunate > interaction effect between the two) > one day they're all fine, then suddenly all ERROR > here's a blogger who encountered the same problemhttp://blog.mcnicholl.com/2007/02/15/google-sitemaps-url-timeout-dns-...
> Adam: Although I can't completely rule us out, I would suggest trying > to get an answer out of google.
> Lawrence: hmmm... okay > will i get a transcript of this chat after i close the window?
> Adam: I'll continue asking around, but I'm not aware of any change > that we have made that would cause this. > Yes, after you close the chat it will ask you if you want it to be > emailed to you.
> Lawrence: Adam, thanks for your help so far... I'll see what response > I get from Google > bye for now
My site is hosted with ThePlanet... so this is not confined to one host. The issue seems to either be on Google's end, OR they started somehow strictly enforcing some rules that in the past they were lax with (E.g., making sure your DNS record is perfect).
Let's keep bumping this up... I spent about an hour trying to find the right contact form and came up with *nothing*... I hope Google staff monitors this forum-- this is important and we really need your help to get this straightened out!
> Naively, I tried sending my question to [email address]
> What I got back was:
> "Thank you for writing to Google. We'd like to assist you, but we only > respond to messages submitted through our online contact form. Please > visithttp://www.google.com/support/to submit your message, and we'll > get > back to you soon. We apologize for any inconvenience, and we look > forward > to hearing from you.
> Regards, > The Google Team"
> So, when you go to google.com/support looking for the "online contact > form" to "submit your message," guess what you find? A never-ending > network of FAQs and no contact form anywhere where you can submit your > message.
> Hmmmmmm.... We're doomed....
> On Sep 10, 3:12 pm, LawrenceJT wrote:
> > Same here!!
> > Suddenly Google is giving us an ERROR message when we try to update > > our sitemap.xml files on Google Webmaster Tools.
> > Following is my online chat with Adam, our hosting company's support > > rep.
> > Question: Google seems to be having a problem accessing our files. > > Each of our sites is getting a "URL timeout." We know for certain that > > all our files are right. Bluehost support rep the other day told me > > that it was probably an issue that our non-Bluehost domain registrars > > need to fix, but in fact that is very unlikely. Most likely is that we > > need Bluehost to adjust folder permissions or firewall settings to > > allow Google access.
> > Adam: Hello! Welcome to our live chat. Please allow me a moment to > > read over your question. Keep in mind that we take multiple chats at > > one time so my response to your questions may take a few moments.
> > Lawrence: Google seems to be having a problem accessing our files. > > Each of our sites is getting a "URL timeout." We know for certain that > > all our files are right. Bluehost support rep the other day told me > > that it was probably an issue that our non-Bluehost domain registrars > > need to fix, but in fact that is very unlikely. Most likely is that we > > need Bluehost to adjust folder permissions or firewall settings to > > allow Google access.
> > Adam: As long as you don't have a robots.txt file blocking google you > > should be fine. We wouldn't block google's bot for any reason.
> > Lawrence: Hmmm... However, it's not fine. :-( > > Here's the error message: > > "URL timeout: DNS lookup timeout We encountered an error while trying > > to access your Sitemap. Please ensure your Sitemap follows our > > guidelines and can be accessed at the location you provided and then > > resubmit" > > Are you familiar with Google sitemaps.xml files?
> > Adam: Yes.
> > Lawrence: We have four different websites hosted on Bluehost. > > I setup sitemap.xml files on each last year to help Google. > > They were able to accept them without problem until recently. > > Nothing changed on our side. > > As you know, it's nearly impossible to open a support ticket with > > Google. They refer users to Google Groups instead. > > This issue was discussed by the Google Webmaster Help group
> > Adam: Yeah, I've noticed that.
> > Lawrence: each solution named something to do with hosting > > Have no other Bluehost clients encountered the issue?
> > Adam: Not that I am aware of, no.
> > Lawrence: You know, it's always a mystery how Google works, and we're > > all doing guesswork all the time. > > Some have suggested that since DNS is mentioned it may have to do with > > the domains' registrars > > But we have multiple registrars for different sites and I find it > > unlikely that all would become problematic at the same time. Know what > > I mean? > > It seems more likely that there would be a permissions or firewall > > snafu on Bluehost (where all these sites are hosted) > > Or, the problem is with Google, in which case I'll have to throw up my > > hands and just give up having a business. (and die penniless in the > > gutter somewhere)
> > Adam: I'm afraid we really wouldn't gain anything from blocking > > googlebot :(
> > Lawrence: Now, googlebot indexes web pages, right?
> > Adam: Yes, that is the part of google that does that.
> > Lawrence: but when we request Google to come and download our latest > > updated sitemap.xml do they index it the same way? same IP as their > > normal googlebot? > > what i mean is,
> > Adam: Yes, although I've never seen it come from the same IP address. > > That's why the robots.txt file is important if you want to block it.
> > Lawrence: i don't actually know whether they're indexing our web pages > > okay this week. maybe or maybe not, i don't know. But what i do know > > is that they are unable to access our sitemap.xml files > > we don't use robots.txt
> > Adam: Are they in the same location you stated when you submitted the > > first time?
> > Lawrence: yes > > in fact, 3 out of 4 of them haven't been changed at all since the last > > successful grabbing by Google > > so, i can only conclude that it's something changed at Google or > > something changed at Bluehost, (or some new and unfortunate > > interaction effect between the two) > > one day they're all fine, then suddenly all ERROR > > here's a blogger who encountered the same problemhttp://blog.mcnicholl.com/2007/02/15/google-sitemaps-url-timeout-dns-...
> > Adam: Although I can't completely rule us out, I would suggest trying > > to get an answer out of google.
> > Lawrence: hmmm... okay > > will i get a transcript of this chat after i close the window?
> > Adam: I'll continue asking around, but I'm not aware of any change > > that we have made that would cause this. > > Yes, after you close the chat it will ask you if you want it to be > > emailed to you.
> > Lawrence: Adam, thanks for your help so far... I'll see what response > > I get from Google > > bye for now
Our case is the same. We've made no changes to our DNS, no changes to our server configuration, nothing. We're hosted in a large, mainstream hosting facility, and our DNS is hosted by a very large, enterprise- grade DNS hosting service (that we've had zero problems with).
So my original instinct was, "Oh, this is just a server networking problem on Google's end. They'll figure it out."
But as it comes up on a week and I notice Googlebot is still staying away, that's becoming a concern.
As I said, we've run the DNSreports tool at dnsstuff.com and it comes back perfectly clean and okay. Not a thing wrong or misconfigured.
> My site is hosted with ThePlanet... so this is not confined to one > host. The issue seems to either be on Google's end, OR they started > somehow strictly enforcing some rules that in the past they were lax > with (E.g., making sure your DNS record is perfect).
> Let's keep bumping this up... I spent about an hour trying to find the > right contact form and came up with *nothing*... I hope Google staff > monitors this forum-- this is important and we really need your help > to get this straightened out!
> On Sep 9, 11:31 pm, LawrenceJT wrote:
> > Naively, I tried sending my question to [email address]
> > What I got back was:
> > "Thank you for writing to Google. We'd like to assist you, but we only > > respond to messages submitted through our online contact form. Please > > visithttp://www.google.com/support/tosubmit your message, and we'll > > get > > back to you soon. We apologize for any inconvenience, and we look > > forward > > to hearing from you.
> > Regards, > > The Google Team"
> > So, when you go to google.com/support looking for the "online contact > > form" to "submit your message," guess what you find? A never-ending > > network of FAQs and no contact form anywhere where you can submit your > > message.
> > Hmmmmmm.... We're doomed....
> > On Sep 10, 3:12 pm, LawrenceJT wrote:
> > > Same here!!
> > > Suddenly Google is giving us an ERROR message when we try to update > > > our sitemap.xml files on Google Webmaster Tools.
> > > Following is my online chat with Adam, our hosting company's support > > > rep.
> > > Question: Google seems to be having a problem accessing our files. > > > Each of our sites is getting a "URL timeout." We know for certain that > > > all our files are right. Bluehost support rep the other day told me > > > that it was probably an issue that our non-Bluehost domain registrars > > > need to fix, but in fact that is very unlikely. Most likely is that we > > > need Bluehost to adjust folder permissions or firewall settings to > > > allow Google access.
> > > Adam: Hello! Welcome to our live chat. Please allow me a moment to > > > read over your question. Keep in mind that we take multiple chats at > > > one time so my response to your questions may take a few moments.
> > > Lawrence: Google seems to be having a problem accessing our files. > > > Each of our sites is getting a "URL timeout." We know for certain that > > > all our files are right. Bluehost support rep the other day told me > > > that it was probably an issue that our non-Bluehost domain registrars > > > need to fix, but in fact that is very unlikely. Most likely is that we > > > need Bluehost to adjust folder permissions or firewall settings to > > > allow Google access.
> > > Adam: As long as you don't have a robots.txt file blocking google you > > > should be fine. We wouldn't block google's bot for any reason.
> > > Lawrence: Hmmm... However, it's not fine. :-( > > > Here's the error message: > > > "URL timeout: DNS lookup timeout We encountered an error while trying > > > to access your Sitemap. Please ensure your Sitemap follows our > > > guidelines and can be accessed at the location you provided and then > > > resubmit" > > > Are you familiar with Google sitemaps.xml files?
> > > Adam: Yes.
> > > Lawrence: We have four different websites hosted on Bluehost. > > > I setup sitemap.xml files on each last year to help Google. > > > They were able to accept them without problem until recently. > > > Nothing changed on our side. > > > As you know, it's nearly impossible to open a support ticket with > > > Google. They refer users to Google Groups instead. > > > This issue was discussed by the Google Webmaster Help group
> > > Adam: Yeah, I've noticed that.
> > > Lawrence: each solution named something to do with hosting > > > Have no other Bluehost clients encountered the issue?
> > > Adam: Not that I am aware of, no.
> > > Lawrence: You know, it's always a mystery how Google works, and we're > > > all doing guesswork all the time. > > > Some have suggested that since DNS is mentioned it may have to do with > > > the domains' registrars > > > But we have multiple registrars for different sites and I find it > > > unlikely that all would become problematic at the same time. Know what > > > I mean? > > > It seems more likely that there would be a permissions or firewall > > > snafu on Bluehost (where all these sites are hosted) > > > Or, the problem is with Google, in which case I'll have to throw up my > > > hands and just give up having a business. (and die penniless in the > > > gutter somewhere)
> > > Adam: I'm afraid we really wouldn't gain anything from blocking > > > googlebot :(
> > > Lawrence: Now, googlebot indexes web pages, right?
> > > Adam: Yes, that is the part of google that does that.
> > > Lawrence: but when we request Google to come and download our latest > > > updated sitemap.xml do they index it the same way? same IP as their > > > normal googlebot? > > > what i mean is,
> > > Adam: Yes, although I've never seen it come from the same IP address. > > > That's why the robots.txt file is important if you want to block it.
> > > Lawrence: i don't actually know whether they're indexing our web pages > > > okay this week. maybe or maybe not, i don't know. But what i do know > > > is that they are unable to access our sitemap.xml files > > > we don't use robots.txt
> > > Adam: Are they in the same location you stated when you submitted the > > > first time?
> > > Lawrence: yes > > > in fact, 3 out of 4 of them haven't been changed at all since the last > > > successful grabbing by Google > > > so, i can only conclude that it's something changed at Google or > > > something changed at Bluehost, (or some new and unfortunate > > > interaction effect between the two) > > > one day they're all fine, then suddenly all ERROR > > > here's a blogger who encountered the same problemhttp://blog.mcnicholl.com/2007/02/15/google-sitemaps-url-timeout-dns-...
> > > Adam: Although I can't completely rule us out, I would suggest trying > > > to get an answer out of google.
> > > Lawrence: hmmm... okay > > > will i get a transcript of this chat after i close the window?
> > > Adam: I'll continue asking around, but I'm not aware of any change > > > that we have made that would cause this. > > > Yes, after you close the chat it will ask you if you want it to be > > > emailed to you.
> > > Lawrence: Adam, thanks for your help so far... I'll see what response > > > I get from Google > > > bye for now
> Our case is the same. We've made no changes to our DNS, no changes to > our server configuration, nothing. We're hosted in a large, mainstream > hosting facility, and our DNS is hosted by a very large, enterprise- > grade DNS hosting service (that we've had zero problems with).
> So my original instinct was, "Oh, this is just a server networking > problem on Google's end. They'll figure it out."
> But as it comes up on a week and I notice Googlebot is still staying > away, that's becoming a concern.
> As I said, we've run the DNSreports tool at dnsstuff.com and it comes > back perfectly clean and okay. Not a thing wrong or misconfigured.
> Hoping for a response from Google! :)
> On Sep 10, 4:08 am, armenb wrote:
> > This is pretty horrible.
> > My site is hosted with ThePlanet... so this is not confined to one > > host. The issue seems to either be on Google's end, OR they started > > somehow strictly enforcing some rules that in the past they were lax > > with (E.g., making sure your DNS record is perfect).
> > Let's keep bumping this up... I spent about an hour trying to find the > > right contact form and came up with *nothing*... I hope Google staff > > monitors this forum-- this is important and we really need your help > > to get this straightened out!
> > On Sep 9, 11:31 pm, LawrenceJT wrote:
> > > Naively, I tried sending my question to [email address]
> > > What I got back was:
> > > "Thank you for writing to Google. We'd like to assist you, but we only > > > respond to messages submitted through our online contact form. Please > > > visithttp://www.google.com/support/tosubmityour message, and we'll > > > get > > > back to you soon. We apologize for any inconvenience, and we look > > > forward > > > to hearing from you.
> > > Regards, > > > The Google Team"
> > > So, when you go to google.com/support looking for the "online contact > > > form" to "submit your message," guess what you find? A never-ending > > > network of FAQs and no contact form anywhere where you can submit your > > > message.
> > > Hmmmmmm.... We're doomed....
> > > On Sep 10, 3:12 pm, LawrenceJT wrote:
> > > > Same here!!
> > > > Suddenly Google is giving us an ERROR message when we try to update > > > > our sitemap.xml files on Google Webmaster Tools.
> > > > Following is my online chat with Adam, our hosting company's support > > > > rep.
> > > > Question: Google seems to be having a problem accessing our files. > > > > Each of our sites is getting a "URL timeout." We know for certain that > > > > all our files are right. Bluehost support rep the other day told me > > > > that it was probably an issue that our non-Bluehost domain registrars > > > > need to fix, but in fact that is very unlikely. Most likely is that we > > > > need Bluehost to adjust folder permissions or firewall settings to > > > > allow Google access.
> > > > Adam: Hello! Welcome to our live chat. Please allow me a moment to > > > > read over your question. Keep in mind that we take multiple chats at > > > > one time so my response to your questions may take a few moments.
> > > > Lawrence: Google seems to be having a problem accessing our files. > > > > Each of our sites is getting a "URL timeout." We know for certain that > > > > all our files are right. Bluehost support rep the other day told me > > > > that it was probably an issue that our non-Bluehost domain registrars > > > > need to fix, but in fact that is very unlikely. Most likely is that we > > > > need Bluehost to adjust folder permissions or firewall settings to > > > > allow Google access.
> > > > Adam: As long as you don't have a robots.txt file blocking google you > > > > should be fine. We wouldn't block google's bot for any reason.
> > > > Lawrence: Hmmm... However, it's not fine. :-( > > > > Here's the error message: > > > > "URL timeout: DNS lookup timeout We encountered an error while trying > > > > to access your Sitemap. Please ensure your Sitemap follows our > > > > guidelines and can be accessed at the location you provided and then > > > > resubmit" > > > > Are you familiar with Google sitemaps.xml files?
> > > > Adam: Yes.
> > > > Lawrence: We have four different websites hosted on Bluehost. > > > > I setup sitemap.xml files on each last year to help Google. > > > > They were able to accept them without problem until recently. > > > > Nothing changed on our side. > > > > As you know, it's nearly impossible to open a support ticket with > > > > Google. They refer users to Google Groups instead. > > > > This issue was discussed by the Google Webmaster Help group
> > > > Adam: Yeah, I've noticed that.
> > > > Lawrence: each solution named something to do with hosting > > > > Have no other Bluehost clients encountered the issue?
> > > > Adam: Not that I am aware of, no.
> > > > Lawrence: You know, it's always a mystery how Google works, and we're > > > > all doing guesswork all the time. > > > > Some have suggested that since DNS is mentioned it may have to do with > > > > the domains' registrars > > > > But we have multiple registrars for different sites and I find it > > > > unlikely that all would become problematic at the same time. Know what > > > > I mean? > > > > It seems more likely that there would be a permissions or firewall > > > > snafu on Bluehost (where all these sites are hosted) > > > > Or, the problem is with Google, in which case I'll have to throw up my > > > > hands and just give up having a business. (and die penniless in the > > > > gutter somewhere)
> > > > Adam: I'm afraid we really wouldn't gain anything from blocking > > > > googlebot :(
> > > > Adam: Yes, that is the part of google that does that.
> > > > Lawrence: but when we request Google to come and download our latest > > > > updated sitemap.xml do they index it the same way? same IP as their > > > > normal googlebot? > > > > what i mean is,
> > > > Adam: Yes, although I've never seen it come from the same IP address. > > > > That's why the robots.txt file is important if you want to block it.
> > > > Lawrence: i don't actually know whether they're indexing our web pages > > > > okay this week. maybe or maybe not, i don't know. But what i do know > > > > is that they are unable to access our sitemap.xml files > > > > we don't use robots.txt
> > > > Adam: Are they in the same location you stated when you submitted the > > > > first time?
> > > > Lawrence: yes > > > > in fact, 3 out of 4 of them haven't been changed at all since the last > > > > successful grabbing by Google > > > > so, i can only conclude that it's something changed at Google or > > > > something changed at Bluehost, (or some new and unfortunate > > > > interaction effect between the two) > > > > one day they're all fine, then suddenly all ERROR > > > > here's a blogger who encountered the same problemhttp://blog.mcnicholl.com/2007/02/15/google-sitemaps-url-timeout-dns-...
> > > > Adam: Although I can't completely rule us out, I would suggest trying > > > > to get an answer out of google.
> > > > Lawrence: hmmm... okay > > > > will i get a transcript of this chat after i close the window?
> > > > Adam: I'll continue asking around, but I'm not aware of any change > > > > that we have made that would cause this. > > > > Yes, after you close the chat it will ask you if you want it to be > > > > emailed to you.
> > > > Lawrence: Adam, thanks for your help so far... I'll see what response > > > > I get from Google > > > > bye for now
> Same problem here. DNS looks ok every way I can think of testing it. > But for the last 5 days, googlebot and Sitemap submissions are timing > out on DNS.
> On Sep 10, 2:13 pm, docjohn wrote:
> > Our case is the same. We've made no changes to our DNS, no changes to > > our server configuration, nothing. We're hosted in a large, mainstream > > hosting facility, and our DNS is hosted by a very large, enterprise- > > grade DNS hosting service (that we've had zero problems with).
> > So my original instinct was, "Oh, this is just a server networking > > problem on Google's end. They'll figure it out."
> > But as it comes up on a week and I notice Googlebot is still staying > > away, that's becoming a concern.
> > As I said, we've run the DNSreports tool at dnsstuff.com and it comes > > back perfectly clean and okay. Not a thing wrong or misconfigured.
> > Hoping for a response from Google! :)
> > On Sep 10, 4:08 am, armenb wrote:
> > > This is pretty horrible.
> > > My site is hosted with ThePlanet... so this is not confined to one > > > host. The issue seems to either be on Google's end, OR they started > > > somehow strictly enforcing some rules that in the past they were lax > > > with (E.g., making sure your DNS record is perfect).
> > > Let's keep bumping this up... I spent about an hour trying to find the > > > right contact form and came up with *nothing*... I hope Google staff > > > monitors this forum-- this is important and we really need your help > > > to get this straightened out!
> > > On Sep 9, 11:31 pm, LawrenceJT wrote:
> > > > Naively, I tried sending my question to [email address]
> > > > What I got back was:
> > > > "Thank you for writing to Google. We'd like to assist you, but we only > > > > respond to messages submitted through our online contact form. Please > > > > visithttp://www.google.com/support/tosubmityourmessage, and we'll > > > > get > > > > back to you soon. We apologize for any inconvenience, and we look > > > > forward > > > > to hearing from you.
> > > > Regards, > > > > The Google Team"
> > > > So, when you go to google.com/support looking for the "online contact > > > > form" to "submit your message," guess what you find? A never-ending > > > > network of FAQs and no contact form anywhere where you can submit your > > > > message.
> > > > Hmmmmmm.... We're doomed....
> > > > On Sep 10, 3:12 pm, LawrenceJT wrote:
> > > > > Same here!!
> > > > > Suddenly Google is giving us an ERROR message when we try to update > > > > > our sitemap.xml files on Google Webmaster Tools.
> > > > > Following is my online chat with Adam, our hosting company's support > > > > > rep.
> > > > > Question: Google seems to be having a problem accessing our files. > > > > > Each of our sites is getting a "URL timeout." We know for certain that > > > > > all our files are right. Bluehost support rep the other day told me > > > > > that it was probably an issue that our non-Bluehost domain registrars > > > > > need to fix, but in fact that is very unlikely. Most likely is that we > > > > > need Bluehost to adjust folder permissions or firewall settings to > > > > > allow Google access.
> > > > > Adam: Hello! Welcome to our live chat. Please allow me a moment to > > > > > read over your question. Keep in mind that we take multiple chats at > > > > > one time so my response to your questions may take a few moments.
> > > > > Lawrence: Google seems to be having a problem accessing our files. > > > > > Each of our sites is getting a "URL timeout." We know for certain that > > > > > all our files are right. Bluehost support rep the other day told me > > > > > that it was probably an issue that our non-Bluehost domain registrars > > > > > need to fix, but in fact that is very unlikely. Most likely is that we > > > > > need Bluehost to adjust folder permissions or firewall settings to > > > > > allow Google access.
> > > > > Adam: As long as you don't have a robots.txt file blocking google you > > > > > should be fine. We wouldn't block google's bot for any reason.
> > > > > Lawrence: Hmmm... However, it's not fine. :-( > > > > > Here's the error message: > > > > > "URL timeout: DNS lookup timeout We encountered an error while trying > > > > > to access your Sitemap. Please ensure your Sitemap follows our > > > > > guidelines and can be accessed at the location you provided and then > > > > > resubmit" > > > > > Are you familiar with Google sitemaps.xml files?
> > > > > Adam: Yes.
> > > > > Lawrence: We have four different websites hosted on Bluehost. > > > > > I setup sitemap.xml files on each last year to help Google. > > > > > They were able to accept them without problem until recently. > > > > > Nothing changed on our side. > > > > > As you know, it's nearly impossible to open a support ticket with > > > > > Google. They refer users to Google Groups instead. > > > > > This issue was discussed by the Google Webmaster Help group
> > > > > Adam: Yeah, I've noticed that.
> > > > > Lawrence: each solution named something to do with hosting > > > > > Have no other Bluehost clients encountered the issue?
> > > > > Adam: Not that I am aware of, no.
> > > > > Lawrence: You know, it's always a mystery how Google works, and we're > > > > > all doing guesswork all the time. > > > > > Some have suggested that since DNS is mentioned it may have to do with > > > > > the domains' registrars > > > > > But we have multiple registrars for different sites and I find it > > > > > unlikely that all would become problematic at the same time. Know what > > > > > I mean? > > > > > It seems more likely that there would be a permissions or firewall > > > > > snafu on Bluehost (where all these sites are hosted) > > > > > Or, the problem is with Google, in which case I'll have to throw up my > > > > > hands and just give up having a business. (and die penniless in the > > > > > gutter somewhere)
> > > > > Adam: I'm afraid we really wouldn't gain anything from blocking > > > > > googlebot :(
> > > > > Adam: Yes, that is the part of google that does that.
> > > > > Lawrence: but when we request Google to come and download our latest > > > > > updated sitemap.xml do they index it the same way? same IP as their > > > > > normal googlebot? > > > > > what i mean is,
> > > > > Adam: Yes, although I've never seen it come from the same IP address. > > > > > That's why the robots.txt file is important if you want to block it.
> > > > > Lawrence: i don't actually know whether they're indexing our web pages > > > > > okay this week. maybe or maybe not, i don't know. But what i do know > > > > > is that they are unable to access our sitemap.xml files > > > > > we don't use robots.txt
> > > > > Adam: Are they in the same location you stated when you submitted the > > > > > first time?
> > > > > Lawrence: yes > > > > > in fact, 3 out of 4 of them haven't been changed at all since the last > > > > > successful grabbing by Google > > > > > so, i can only conclude that it's something changed at Google or > > > > > something changed at Bluehost, (or some new and unfortunate > > > > > interaction effect between the two) > > > > > one day they're all fine, then suddenly all ERROR > > > > > here's a blogger who encountered the same problemhttp://blog.mcnicholl.com/2007/02/15/google-sitemaps-url-timeout-dns-...
> > > > > Adam: Although I can't completely rule us out, I would suggest trying > > > > > to get an answer out of google.
> > > > > Lawrence: hmmm... okay > > > > > will i get a transcript of this chat after i close the window?
> > > > > Adam: I'll continue asking around, but I'm not aware of any change > > > > > that we have made that would cause this. > > > > > Yes, after you close the chat it will ask you if you want it to be > > > > > emailed to you.
> > > > > Lawrence: Adam, thanks for your help so far... I'll see what response > > > > > I get from Google > > > > > bye for now
> Same problem here. DNS looks ok every way I can think of testing it. > But for the last 5 days, googlebot and Sitemap submissions are timing > out on DNS.
> On Sep 10, 2:13 pm, docjohn wrote:
> > Our case is the same. We've made no changes to our DNS, no changes to > > our server configuration, nothing. We're hosted in a large, mainstream > > hosting facility, and our DNS is hosted by a very large, enterprise- > > grade DNS hosting service (that we've had zero problems with).
> > So my original instinct was, "Oh, this is just a server networking > > problem on Google's end. They'll figure it out."
> > But as it comes up on a week and I notice Googlebot is still staying > > away, that's becoming a concern.
> > As I said, we've run the DNSreports tool at dnsstuff.com and it comes > > back perfectly clean and okay. Not a thing wrong or misconfigured.
> > Hoping for a response from Google! :)
> > On Sep 10, 4:08 am, armenb wrote:
> > > This is pretty horrible.
> > > My site is hosted with ThePlanet... so this is not confined to one > > > host. The issue seems to either be on Google's end, OR they started > > > somehow strictly enforcing some rules that in the past they were lax > > > with (E.g., making sure your DNS record is perfect).
> > > Let's keep bumping this up... I spent about an hour trying to find the > > > right contact form and came up with *nothing*... I hope Google staff > > > monitors this forum-- this is important and we really need your help > > > to get this straightened out!
> > > On Sep 9, 11:31 pm, LawrenceJT wrote:
> > > > Naively, I tried sending my question to [email address]
> > > > What I got back was:
> > > > "Thank you for writing to Google. We'd like to assist you, but we only > > > > respond to messages submitted through our online contact form. Please > > > > visithttp://www.google.com/support/tosubmityourmessage, and we'll > > > > get > > > > back to you soon. We apologize for any inconvenience, and we look > > > > forward > > > > to hearing from you.
> > > > Regards, > > > > The Google Team"
> > > > So, when you go to google.com/support looking for the "online contact > > > > form" to "submit your message," guess what you find? A never-ending > > > > network of FAQs and no contact form anywhere where you can submit your > > > > message.
> > > > Hmmmmmm.... We're doomed....
> > > > On Sep 10, 3:12 pm, LawrenceJT wrote:
> > > > > Same here!!
> > > > > Suddenly Google is giving us an ERROR message when we try to update > > > > > our sitemap.xml files on Google Webmaster Tools.
> > > > > Following is my online chat with Adam, our hosting company's support > > > > > rep.
> > > > > Question: Google seems to be having a problem accessing our files. > > > > > Each of our sites is getting a "URL timeout." We know for certain that > > > > > all our files are right. Bluehost support rep the other day told me > > > > > that it was probably an issue that our non-Bluehost domain registrars > > > > > need to fix, but in fact that is very unlikely. Most likely is that we > > > > > need Bluehost to adjust folder permissions or firewall settings to > > > > > allow Google access.
> > > > > Adam: Hello! Welcome to our live chat. Please allow me a moment to > > > > > read over your question. Keep in mind that we take multiple chats at > > > > > one time so my response to your questions may take a few moments.
> > > > > Lawrence: Google seems to be having a problem accessing our files. > > > > > Each of our sites is getting a "URL timeout." We know for certain that > > > > > all our files are right. Bluehost support rep the other day told me > > > > > that it was probably an issue that our non-Bluehost domain registrars > > > > > need to fix, but in fact that is very unlikely. Most likely is that we > > > > > need Bluehost to adjust folder permissions or firewall settings to > > > > > allow Google access.
> > > > > Adam: As long as you don't have a robots.txt file blocking google you > > > > > should be fine. We wouldn't block google's bot for any reason.
> > > > > Lawrence: Hmmm... However, it's not fine. :-( > > > > > Here's the error message: > > > > > "URL timeout: DNS lookup timeout We encountered an error while trying > > > > > to access your Sitemap. Please ensure your Sitemap follows our > > > > > guidelines and can be accessed at the location you provided and then > > > > > resubmit" > > > > > Are you familiar with Google sitemaps.xml files?
> > > > > Adam: Yes.
> > > > > Lawrence: We have four different websites hosted on Bluehost. > > > > > I setup sitemap.xml files on each last year to help Google. > > > > > They were able to accept them without problem until recently. > > > > > Nothing changed on our side. > > > > > As you know, it's nearly impossible to open a support ticket with > > > > > Google. They refer users to Google Groups instead. > > > > > This issue was discussed by the Google Webmaster Help group
> > > > > Adam: Yeah, I've noticed that.
> > > > > Lawrence: each solution named something to do with hosting > > > > > Have no other Bluehost clients encountered the issue?
> > > > > Adam: Not that I am aware of, no.
> > > > > Lawrence: You know, it's always a mystery how Google works, and we're > > > > > all doing guesswork all the time. > > > > > Some have suggested that since DNS is mentioned it may have to do with > > > > > the domains' registrars > > > > > But we have multiple registrars for different sites and I find it > > > > > unlikely that all would become problematic at the same time. Know what > > > > > I mean? > > > > > It seems more likely that there would be a permissions or firewall > > > > > snafu on Bluehost (where all these sites are hosted) > > > > > Or, the problem is with Google, in which case I'll have to throw up my > > > > > hands and just give up having a business. (and die penniless in the > > > > > gutter somewhere)
> > > > > Adam: I'm afraid we really wouldn't gain anything from blocking > > > > > googlebot :(
> > > > > Adam: Yes, that is the part of google that does that.
> > > > > Lawrence: but when we request Google to come and download our latest > > > > > updated sitemap.xml do they index it the same way? same IP as their > > > > > normal googlebot? > > > > > what i mean is,
> > > > > Adam: Yes, although I've never seen it come from the same IP address. > > > > > That's why the robots.txt file is important if you want to block it.
> > > > > Lawrence: i don't actually know whether they're indexing our web pages > > > > > okay this week. maybe or maybe not, i don't know. But what i do know > > > > > is that they are unable to access our sitemap.xml files > > > > > we don't use robots.txt
> > > > > Adam: Are they in the same location you stated when you submitted the > > > > > first time?
> > > > > Lawrence: yes > > > > > in fact, 3 out of 4 of them haven't been changed at all since the last > > > > > successful grabbing by Google > > > > > so, i can only conclude that it's something changed at Google or > > > > > something changed at Bluehost, (or some new and unfortunate > > > > > interaction effect between the two) > > > > > one day they're all fine, then suddenly all ERROR > > > > > here's a blogger who encountered the same problemhttp://blog.mcnicholl.com/2007/02/15/google-sitemaps-url-timeout-dns-...
> > > > > Adam: Although I can't completely rule us out, I would suggest trying > > > > > to get an answer out of google.
> > > > > Lawrence: hmmm... okay > > > > > will i get a transcript of this chat after i close the window?
> > > > > Adam: I'll continue asking around, but I'm not aware of any change > > > > > that we have made that would cause this. > > > > > Yes, after you close the chat it will ask you if you want it to be > > > > > emailed to you.
> > > > > Lawrence: Adam, thanks for your help so far... I'll see what response > > > > > I get from Google > > > > > bye for now
Continues to get weirder! It just picked up my robots.txt file this morning (last updated date on the robots.txt link of Webmaster tools). Previously, this was stuck on September 5th. The resubmission of the sitemap just failed minutes ago, with the DNS timeout error again, however.
Either things are going to get better once their crawler DNS cache expires (not likely given I've seen this problem a bunch of times in the archives and that's not ever the resolution), or we still have a major malfunction where every page resolves minus the sitemaps and the home page!
> Fitz, DocJohn, and others: who is your registrar? Maybe that's a > commonality?
> We're registered with GoDaddy. DNS hosted with ThePlanet.
> On Sep 10, 5:25 am, Fitz wrote:
> > Same problem here. DNS looks ok every way I can think of testing it. > > But for the last 5 days, googlebot and Sitemap submissions are timing > > out on DNS.
> > On Sep 10, 2:13 pm, docjohn wrote:
> > > Our case is the same. We've made no changes to our DNS, no changes to > > > our server configuration, nothing. We're hosted in a large, mainstream > > > hosting facility, and our DNS is hosted by a very large, enterprise- > > > grade DNS hosting service (that we've had zero problems with).
> > > So my original instinct was, "Oh, this is just a server networking > > > problem on Google's end. They'll figure it out."
> > > But as it comes up on a week and I notice Googlebot is still staying > > > away, that's becoming a concern.
> > > As I said, we've run the DNSreports tool at dnsstuff.com and it comes > > > back perfectly clean and okay. Not a thing wrong or misconfigured.
> > > Hoping for a response from Google! :)
> > > On Sep 10, 4:08 am, armenb wrote:
> > > > This is pretty horrible.
> > > > My site is hosted with ThePlanet... so this is not confined to one > > > > host. The issue seems to either be on Google's end, OR they started > > > > somehow strictly enforcing some rules that in the past they were lax > > > > with (E.g., making sure your DNS record is perfect).
> > > > Let's keep bumping this up... I spent about an hour trying to find the > > > > right contact form and came up with *nothing*... I hope Google staff > > > > monitors this forum-- this is important and we really need your help > > > > to get this straightened out!
> > > > On Sep 9, 11:31 pm, LawrenceJT wrote:
> > > > > Naively, I tried sending my question to [email address]
> > > > > What I got back was:
> > > > > "Thank you for writing to Google. We'd like to assist you, but we only > > > > > respond to messages submitted through our online contact form. Please > > > > > visithttp://www.google.com/support/tosubmityourmessage, and we'll > > > > > get > > > > > back to you soon. We apologize for any inconvenience, and we look > > > > > forward > > > > > to hearing from you.
> > > > > Regards, > > > > > The Google Team"
> > > > > So, when you go to google.com/support looking for the "online contact > > > > > form" to "submit your message," guess what you find? A never-ending > > > > > network of FAQs and no contact form anywhere where you can submit your > > > > > message.
> > > > > Hmmmmmm.... We're doomed....
> > > > > On Sep 10, 3:12 pm, LawrenceJT wrote:
> > > > > > Same here!!
> > > > > > Suddenly Google is giving us an ERROR message when we try to update > > > > > > our sitemap.xml files on Google Webmaster Tools.
> > > > > > Following is my online chat with Adam, our hosting company's support > > > > > > rep.
> > > > > > Question: Google seems to be having a problem accessing our files. > > > > > > Each of our sites is getting a "URL timeout." We know for certain that > > > > > > all our files are right. Bluehost support rep the other day told me > > > > > > that it was probably an issue that our non-Bluehost domain registrars > > > > > > need to fix, but in fact that is very unlikely. Most likely is that we > > > > > > need Bluehost to adjust folder permissions or firewall settings to > > > > > > allow Google access.
> > > > > > Adam: Hello! Welcome to our live chat. Please allow me a moment to > > > > > > read over your question. Keep in mind that we take multiple chats at > > > > > > one time so my response to your questions may take a few moments.
> > > > > > Lawrence: Google seems to be having a problem accessing our files. > > > > > > Each of our sites is getting a "URL timeout." We know for certain that > > > > > > all our files are right. Bluehost support rep the other day told me > > > > > > that it was probably an issue that our non-Bluehost domain registrars > > > > > > need to fix, but in fact that is very unlikely. Most likely is that we > > > > > > need Bluehost to adjust folder permissions or firewall settings to > > > > > > allow Google access.
> > > > > > Adam: As long as you don't have a robots.txt file blocking google you > > > > > > should be fine. We wouldn't block google's bot for any reason.
> > > > > > Lawrence: Hmmm... However, it's not fine. :-( > > > > > > Here's the error message: > > > > > > "URL timeout: DNS lookup timeout We encountered an error while trying > > > > > > to access your Sitemap. Please ensure your Sitemap follows our > > > > > > guidelines and can be accessed at the location you provided and then > > > > > > resubmit" > > > > > > Are you familiar with Google sitemaps.xml files?
> > > > > > Adam: Yes.
> > > > > > Lawrence: We have four different websites hosted on Bluehost. > > > > > > I setup sitemap.xml files on each last year to help Google. > > > > > > They were able to accept them without problem until recently. > > > > > > Nothing changed on our side. > > > > > > As you know, it's nearly impossible to open a support ticket with > > > > > > Google. They refer users to Google Groups instead. > > > > > > This issue was discussed by the Google Webmaster Help group
> > > > > > Adam: Yeah, I've noticed that.
> > > > > > Lawrence: each solution named something to do with hosting > > > > > > Have no other Bluehost clients encountered the issue?
> > > > > > Adam: Not that I am aware of, no.
> > > > > > Lawrence: You know, it's always a mystery how Google works, and we're > > > > > > all doing guesswork all the time. > > > > > > Some have suggested that since DNS is mentioned it may have to do with > > > > > > the domains' registrars > > > > > > But we have multiple registrars for different sites and I find it > > > > > > unlikely that all would become problematic at the same time. Know what > > > > > > I mean? > > > > > > It seems more likely that there would be a permissions or firewall > > > > > > snafu on Bluehost (where all these sites are hosted) > > > > > > Or, the problem is with Google, in which case I'll have to throw up my > > > > > > hands and just give up having a business. (and die penniless in the > > > > > > gutter somewhere)
> > > > > > Adam: I'm afraid we really wouldn't gain anything from blocking > > > > > > googlebot :(
> > > > > > Adam: Yes, that is the part of google that does that.
> > > > > > Lawrence: but when we request Google to come and download our latest > > > > > > updated sitemap.xml do they index it the same way? same IP as their > > > > > > normal googlebot? > > > > > > what i mean is,
> > > > > > Adam: Yes, although I've never seen it come from the same IP address. > > > > > > That's why the robots.txt file is important if you want to block it.
> > > > > > Lawrence: i don't actually know whether they're indexing our web pages > > > > > > okay this week. maybe or maybe not, i don't know. But what i do know > > > > > > is that they are unable to access our sitemap.xml files > > > > > > we don't use robots.txt
> > > > > > Adam: Are they in the same location you stated when you submitted the > > > > > > first time?
> > > > > > Lawrence: yes > > > > > > in fact, 3 out of 4 of them haven't been changed at all since the last > > > > > > successful grabbing by Google > > > > > > so, i can only conclude that it's something changed at Google or > > > > > > something changed at Bluehost, (or some new and unfortunate > > > > > > interaction effect between the two) > > > > > > one day they're all fine, then suddenly all ERROR > > > > > > here's a blogger who encountered the same problemhttp://blog.mcnicholl.com/2007/02/15/google-sitemaps-url-timeout-dns-...
> > > > > > Adam: Although I can't completely rule us out, I would suggest trying > > > > > > to get an answer out of google.
> > > > > > Lawrence: hmmm... okay > > > > > > will i get a transcript of this chat after i close the window?
> > > > > > Adam: I'll continue asking around, but I'm not aware of any change > > > > > > that we have made that would cause this. > > > > > > Yes, after you close the chat it will ask you if you want it to be > > > > > > emailed to you.
> > > > > > Lawrence: Adam, thanks for your help so far... I'll see what response > > > > > > I get from Google > > > > > > bye for now
> > > > > > Adam: Anytime, let us know how it goes.
The registrar is fine, they can't mess up geenrally.
The DNS is where your hosting is . Unless you use somehtign like dyndns where you set up a seprate dns zone from what the hoster would.
The DNS at your hosting can be messed up and often is. Nameservers sometimes are lame, there may be leakage.
That's what you need to concentrate on.
Unfortunately this old staple tool from dnsreport.com has become a paid service and I cannot use it to troubleshoot dns problem without being a paid up member (free lookups long used up). I'm not paying for it, if you want to, go ahead.
Your hoster ought to have some tools too .. unless they are small fry like me ;) Bluehost doesn't seem to fit the description.
> Fitz, DocJohn, and others: who is your registrar? Maybe that's a > commonality?
> We're registered with GoDaddy. DNS hosted with ThePlanet.
> On Sep 10, 5:25 am, Fitz wrote:
> > Same problem here. DNS looks ok every way I can think of testing it. > > But for the last 5 days, googlebot and Sitemap submissions are timing > > out on DNS.
> > On Sep 10, 2:13 pm, docjohn wrote:
> > > Our case is the same. We've made no changes to our DNS, no changes to > > > our server configuration, nothing. We're hosted in a large, mainstream > > > hosting facility, and our DNS is hosted by a very large, enterprise- > > > grade DNS hosting service (that we've had zero problems with).
> > > So my original instinct was, "Oh, this is just a server networking > > > problem on Google's end. They'll figure it out."
> > > But as it comes up on a week and I notice Googlebot is still staying > > > away, that's becoming a concern.
> > > As I said, we've run the DNSreports tool at dnsstuff.com and it comes > > > back perfectly clean and okay. Not a thing wrong or misconfigured.
> > > Hoping for a response from Google! :)
> > > On Sep 10, 4:08 am, armenb wrote:
> > > > This is pretty horrible.
> > > > My site is hosted with ThePlanet... so this is not confined to one > > > > host. The issue seems to either be on Google's end, OR they started > > > > somehow strictly enforcing some rules that in the past they were lax > > > > with (E.g., making sure your DNS record is perfect).
> > > > Let's keep bumping this up... I spent about an hour trying to find the > > > > right contact form and came up with *nothing*... I hope Google staff > > > > monitors this forum-- this is important and we really need your help > > > > to get this straightened out!
> > > > On Sep 9, 11:31 pm, LawrenceJT wrote:
> > > > > Naively, I tried sending my question to [email address]
> > > > > What I got back was:
> > > > > "Thank you for writing to Google. We'd like to assist you, but we only > > > > > respond to messages submitted through our online contact form. Please > > > > > visithttp://www.google.com/support/tosubmityourmessage, and we'll > > > > > get > > > > > back to you soon. We apologize for any inconvenience, and we look > > > > > forward > > > > > to hearing from you.
> > > > > Regards, > > > > > The Google Team"
> > > > > So, when you go to google.com/support looking for the "online contact > > > > > form" to "submit your message," guess what you find? A never-ending > > > > > network of FAQs and no contact form anywhere where you can submit your > > > > > message.
> > > > > Hmmmmmm.... We're doomed....
> > > > > On Sep 10, 3:12 pm, LawrenceJT wrote:
> > > > > > Same here!!
> > > > > > Suddenly Google is giving us an ERROR message when we try to update > > > > > > our sitemap.xml files on Google Webmaster Tools.
> > > > > > Following is my online chat with Adam, our hosting company's support > > > > > > rep.
> > > > > > Question: Google seems to be having a problem accessing our files. > > > > > > Each of our sites is getting a "URL timeout." We know for certain that > > > > > > all our files are right. Bluehost support rep the other day told me > > > > > > that it was probably an issue that our non-Bluehost domain registrars > > > > > > need to fix, but in fact that is very unlikely. Most likely is that we > > > > > > need Bluehost to adjust folder permissions or firewall settings to > > > > > > allow Google access.
> > > > > > Adam: Hello! Welcome to our live chat. Please allow me a moment to > > > > > > read over your question. Keep in mind that we take multiple chats at > > > > > > one time so my response to your questions may take a few moments.
> > > > > > Lawrence: Google seems to be having a problem accessing our files. > > > > > > Each of our sites is getting a "URL timeout." We know for certain that > > > > > > all our files are right. Bluehost support rep the other day told me > > > > > > that it was probably an issue that our non-Bluehost domain registrars > > > > > > need to fix, but in fact that is very unlikely. Most likely is that we > > > > > > need Bluehost to adjust folder permissions or firewall settings to > > > > > > allow Google access.
> > > > > > Adam: As long as you don't have a robots.txt file blocking google you > > > > > > should be fine. We wouldn't block google's bot for any reason.
> > > > > > Lawrence: Hmmm... However, it's not fine. :-( > > > > > > Here's the error message: > > > > > > "URL timeout: DNS lookup timeout We encountered an error while trying > > > > > > to access your Sitemap. Please ensure your Sitemap follows our > > > > > > guidelines and can be accessed at the location you provided and then > > > > > > resubmit" > > > > > > Are you familiar with Google sitemaps.xml files?
> > > > > > Adam: Yes.
> > > > > > Lawrence: We have four different websites hosted on Bluehost. > > > > > > I setup sitemap.xml files on each last year to help Google. > > > > > > They were able to accept them without problem until recently. > > > > > > Nothing changed on our side. > > > > > > As you know, it's nearly impossible to open a support ticket with > > > > > > Google. They refer users to Google Groups instead. > > > > > > This issue was discussed by the Google Webmaster Help group
> > > > > > Adam: Yeah, I've noticed that.
> > > > > > Lawrence: each solution named something to do with hosting > > > > > > Have no other Bluehost clients encountered the issue?
> > > > > > Adam: Not that I am aware of, no.
> > > > > > Lawrence: You know, it's always a mystery how Google works, and we're > > > > > > all doing guesswork all the time. > > > > > > Some have suggested that since DNS is mentioned it may have to do with > > > > > > the domains' registrars > > > > > > But we have multiple registrars for different sites and I find it > > > > > > unlikely that all would become problematic at the same time. Know what > > > > > > I mean? > > > > > > It seems more likely that there would be a permissions or firewall > > > > > > snafu on Bluehost (where all these sites are hosted) > > > > > > Or, the problem is with Google, in which case I'll have to throw up my > > > > > > hands and just give up having a business. (and die penniless in the > > > > > > gutter somewhere)
> > > > > > Adam: I'm afraid we really wouldn't gain anything from blocking > > > > > > googlebot :(
> > > > > > Adam: Yes, that is the part of google that does that.
> > > > > > Lawrence: but when we request Google to come and download our latest > > > > > > updated sitemap.xml do they index it the same way? same IP as their > > > > > > normal googlebot? > > > > > > what i mean is,
> > > > > > Adam: Yes, although I've never seen it come from the same IP address. > > > > > > That's why the robots.txt file is important if you want to block it.
> > > > > > Lawrence: i don't actually know whether they're indexing our web pages > > > > > > okay this week. maybe or maybe not, i don't know. But what i do know > > > > > > is that they are unable to access our sitemap.xml files > > > > > > we don't use robots.txt
> > > > > > Adam: Are they in the same location you stated when you submitted the > > > > > > first time?
> > > > > > Lawrence: yes > > > > > > in fact, 3 out of 4 of them haven't been changed at all since the last > > > > > > successful grabbing by Google > > > > > > so, i can only conclude that it's something changed at Google or > > > > > > something changed at Bluehost, (or some new and unfortunate > > > > > > interaction effect between the two) > > > > > > one day they're all fine, then suddenly all ERROR > > > > > > here's a blogger who encountered the same problemhttp://blog.mcnicholl.com/2007/02/15/google-sitemaps-url-timeout-dns-...
> > > > > > Adam: Although I can't completely rule us out, I would suggest trying > > > > > > to get an answer out of google.
> > > > > > Lawrence: hmmm... okay > > > > > > will i get a transcript of this chat after i close the window?
> > > > > > Adam: I'll continue asking around, but I'm not aware of any change > > > > > > that we have made that would cause this. > > > > > > Yes, after you close the chat it will ask you if you want it to be > > > > > > emailed to you.
> > > > > > Lawrence: Adam, thanks for your help so far... I'll see what response > > > > > > I get from Google > > > > > > bye for now
> > > > > > Adam: Anytime, let us know how it goes.- Hide quoted text -
Just wanted to let you know that we're looking around to see if this is being caused by something on our end. Also, as webado noted, it helps if you post the URL in question.
> Just wanted to let you know that we're looking around to see if this > is being caused by something on our end. Also, as webado noted, it > helps if you post the URL in question.
Exactly the same situation here....I stumbled upon this later...but I posted exactly the same question on a separate thread...I am reporducing that thread here.... I feel better that it is not only me...but lot of others are also facing the same problem.. Google has been cralwing hindreds sometimes thousand of pages without any problem...but since 6th I have been facing url timeout problem like many of the guys above....
> Long-time site in Google and no previous problems. Only started paying > attention to Google's webmaster tools as they've improved them this > year (thank you!).
> Just noticed that Google hasn't had a successful crawl on my site > since Sept. 5. The error message for the main page is:
> "We can't currently access your home page because of a timeout."
> The site is online and functional 24/7 with significant traffic (even > from Google's Adsensebot crawl servers, no problemo). But Googlebot > has nearly stopped completely, down from hundreds of pages crawled per > day to just a dozen or two since 9/5.
> So without any further information, I decided I'd better upload a > sitemap and see if that would give me further info. Well, it did:
> "URL timeout: DNS lookup timeout"
> The problem is, DNS lookups are fine for my domain. We use a third- > party enterprise DNS hosting service that lists 5 geographically- > diverse DNS nameservers for our domain. Naturally, these are all up > and running fine. DNStools reports no problems with either our domain > or its DNS.
> Looking in the log files for Googlebot instances shows that the few > times it is crawling these days, it's downloading the files fine > (e.g., byte count and result code is what is expected and what they've > always been).
> So a few suggestions come to mind from this experience:
> 1. Google should add more detail to the default error message that is > reported on the dashboard's main page for a URL about the specific > timeout issue. I only discovered additional detailed information about > the error from uploading a sitemap.
> 2. Error messages are fairly useless to a webmaster unless they > suggest a troubleshooting procedure. Luckily, I know the > troubleshooting procedure for DNS timeouts and now I've done > everything I can do on my side of things... What's next? How do I > contact Google about this problem? (I assume this is it.)
> 3. A button that says, "Okay, I've fixed the error, please clear your > cache and crawl again next time you're scheduled to do so" would be > great. I assume it's not there because Google is *always* crawling and > automatically rechecks these things on its own. That's cool, but it > appears my domain is stuck in some sort of error cache thing or > something on Google's search crawl side (while Google Adsensebot > continues to churn along just fine). I hope/assume this will fix > itself, since there doesn't appear to be anything else I can do on my > end.
> 4. Google's internal systems and bots should be able to talk to one > another. Google Adsensebot continues to function fine, while Googlebot > thinks my domain's DNS is broken. If Googlebot could talk to Google > Adsensebot, they'd say, "Hey, I'm getting through just fine. Try > hitting it from a different IP address or something and clear your > error cache."
Just to report, Sitemap submission is working again for me, no more DNS timeout. If it was something you guys fixed on your end, then thank you very much. If not, then thank you anyway :-)
> Just wanted to let you know that we're looking around to see if this > is being caused by something on our end. Also, as webado noted, it > helps if you post the URL in question.
Sitemap is now being read OK (thanks!), but home page still shows as inaccessible (front page of Webmaster Tools)-- "We can't currently access your home page because of a timeout."
I assume this will clear up now too?
Bravo to the G-team for fixing this so quickly after we brought it to your attention.
> Just to report, Sitemap submission is working again for me, no more > DNS timeout. If it was something you guys fixed on your end, then > thank you very much. If not, then thank you anyway :-)
> On Sep 10, 7:59 pm, Susan Moskwa wrote:
> > Hi folks--
> > Just wanted to let you know that we're looking around to see if this > > is being caused by something on our end. Also, as webado noted, it > > helps if you post the URL in question.
I think I've got essentially the same problem. No crawl for weeks on http://www.zoe-young.co.uk/ and Webmaster Tools is reporting "network unreachable - robots.txt unreachable" whenever I submit a new sitemap.
In my case, everything used to be fine - and all the pages have previously found their way into the cache - but though Webmaster Tools says it has tried to reach my sitemap.txt half-a-dozen time today alone (because I keep resubmitting the file) my raw weblogs are showing not a single attempt.
Siteground Support say they've checked firewalls and claim it's not down to them; and all the other spiders are accessing robots.txt without any problem.
Since Google doesn't appear to be reaching me at all, it can't be a problem with any of my files, surely? Is there any way I can get any more diagnostics out of Google?
> Just wanted to let you know that we're looking around to see if this > is being caused by something on our end. Also, as webado noted, it > helps if you post the URL in question.
It was an issue on our end, and should be cleared up now. Thanks for all your reports!
If anyone is still seeing DNS-related errors in their Webmaster Tools account, check the "last found" date. If it's dated before today, wait for us to recrawl the URL and the error should go away; otherwise, feel free to post the details here and we can look into it further.
Barry, would you do me a favor and ask if there is any update on peoples homepage going missing while the rest of the site is indexed? Apparently you have the magic touch to get answers.
This hasn't resolved the issue I reported in [19] above.. I'd hoped that was a DNS issue, but maybe it's something a little different. I'm not alone in seeing it - plenty of reports elswhere, but no advice on cause or resolution. Is it posible to look at this?
It's discussed in more detail in the thread "Network unreachable: robots.txt unreachable" a lttle further down this group:
> It was an issue on our end, and should be cleared up now. Thanks for > all your reports!
> If anyone is still seeing DNS-related errors in their Webmaster Tools > account, check the "last found" date. If it's dated before today, wait > for us to recrawl the URL and the error should go away; otherwise, > feel free to post the details here and we can look into it further.
> Barry, would you do me a favor and ask if there is any update on > peoples homepage going missing while the rest of the site is > indexed? Apparently you have the magic touch to get answers.
|
