Almost a month ago I posted here asking for ideas as to why a new site wasn't getting indexed by the search engines.
Well earlier this week I caught Google in the log, and I saw that the bot had come to the site looking for robots.txt, didn't find it (500), and pissed off. Apparently now Google REQUIRES robots.txt. (?) Who knew.
I placed a robots.txt in the site and now it's getting spidered normally.
-- "Because all you of Earth are idiots!" ¯`ˇ.¸¸.ˇ´¯`ˇ-> freemontŠ <-ˇ´¯`ˇ.¸¸.ˇ´¯
"freemont" <freem...@spammenotfreemontsoffice.com> wrote ...
> Apparently now Google REQUIRES robots.txt
Not so. Be wary of generalising from one example; other things may be going on. --
Andrew seo2seo.com sick-site-syndrome.com
UK Residents: STOP THE "10p Tax Ripoff" Sign the petition to stop the government stealing from the very poorest tell your friends about this petition: http://petitions.pm.gov.uk/10penceband/
freemont wrote: > Almost a month ago I posted here asking for ideas as to why a new site > wasn't getting indexed by the search engines.
> Well earlier this week I caught Google in the log, and I saw that the bot > had come to the site looking for robots.txt, didn't find it (500), and > pissed off. Apparently now Google REQUIRES robots.txt. (?) Who knew.
> I placed a robots.txt in the site and now it's getting spidered normally.
No, it was the 500 (Internal Error) which caused the googlebot to stop.
Why did your server send a 500 instead of the more correct 404 (Not Found)? The latter is the correct return code.
-- ================== Remove the "x" from my email address Jerry Stuckle JDS Computer Training Corp. jstuck...@attglobal.net ==================
On Thu, 15 May 2008 16:39:33 +0100, GeorgeLondon writ:
>>> Apparently now Google REQUIRES robots.txt
>> Not so. >> Be wary of generalising from one example; other things may be going on.
> He's right you know. It is indeed not so.
You're right. It was early when I posted that and it was worded poorly. Imagine me saying "Apparently now Google REQUIRES robots.txt. Who knew." sarcastically and with an exasperated look. :-)
It was just a strange problem.
-- "Because all you of Earth are idiots!" ¯`ˇ.¸¸.ˇ´¯`ˇ-> freemontŠ <-ˇ´¯`ˇ.¸¸.ˇ´¯
On Thu, 15 May 2008 17:07:59 +0200, John Hosking writ:
> freemont wrote: >> Well earlier this week I caught Google in the log, and I saw that the bot >> had come to the site looking for robots.txt, didn't find it (500), and
> 500?!? Not 404?
Yep, 500
>> pissed off. Apparently now Google REQUIRES robots.txt. (?) Who knew.
>> I placed a robots.txt in the site and now it's getting spidered normally.
> Are we sure googlebot just doesn't like 500-series HTTP responses? I > believe there are *a lot* of sites out there without robots.txts.
It appears they won't go further when a request for the file returns 500. I, too, have made sites without robots.txt that were indexed straight away. -- "Because all you of Earth are idiots!" ¯`ˇ.¸¸.ˇ´¯`ˇ-> freemontŠ <-ˇ´¯`ˇ.¸¸.ˇ´¯
"freemont" <freem...@spammenotfreemontsoffice.com> wrote: > Imagine me saying "Apparently now Google REQUIRES robots.txt. Who knew." > sarcastically and with an exasperated look. :-) > It was just a strange problem.
