The group you are posting to is a Usenet group. Messages posted to this group will make your email address visible to anyone on the Internet.
Your reply message has not been sent.
Your post was successful
 |
Newsgroups: alt.www.webmaster
From: freemont <freem...@spammenotfreemontsoffice.com>
Date: Thu, 15 May 2008 19:09:24 +0000
Local: Thurs, May 15 2008 3:09 pm
Subject: Re: site wouldn't show in searches
On Thu, 15 May 2008 17:07:59 +0200, John Hosking writ:
> freemont wrote: Yep, 500 >> Well earlier this week I caught Google in the log, and I saw that the bot >> had come to the site looking for robots.txt, didn't find it (500), and > 500?!? Not 404? >> pissed off. Apparently now Google REQUIRES robots.txt. (?) Who knew. It appears they won't go further when a request for the file returns 500. >> I placed a robots.txt in the site and now it's getting spidered normally. > Are we sure googlebot just doesn't like 500-series HTTP responses? I I, too, have made sites without robots.txt that were indexed straight away. -- "Because all you of Earth are idiots!" ¯`ˇ.¸¸.ˇ´¯`ˇ-> freemontŠ <-ˇ´¯`ˇ.¸¸.ˇ´¯ You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
| |||||||||||||