On 10/15/07, Graham Dumpleton <Graham.Dumple...@gmail.com> wrote:
> Just realised that the robots.txt file on code.google.com blocks
> access to other search engines:
> User-Agent: *
> Disallow: /
> Noindex: /
> Given that nothing was said in the information available before
> signing up that "BTW, you can host your project here but don't expect
> any other search engine to be able to find it", all I can say is I am
> not happy at this point. My dissatisfaction isn't helped by the fact
> that Google Search has somehow screwed up and has dropped my Google
> Code home page from the Google Search results where it used to be the
> top ranking site for the name used for the project. As a result,
> search traffic and thus downloads of the software have started
> dropping off because no one can find the site anymore. Am still
> getting traffic through site referrals but overall traffic is down
> from what it was before.
> Anyway, although most people probably use Google Search, thought it
> might be worthwhile to point out to others that Google is explicitly
> blocking other search engines through its robots.txt file, so don't
> expect other search engine users to find your code and just hope that
> Google Search doesn't loose your Google Code site for some reason.
Wow. I really appreciate your bug report, and especially giving us the
benefit of the doubt on this. But seriously, taking off our tinfoil hats
for a second, there's no conspiracy going on here. I'm really glad that you
pointed this out--we made some infrastructure changes in the last few weeks
that caused this erroneous robots.txt to be installed on Google Code. We're
working ASAP to have it removed. It was most definitely *not* an
intentional thing to do--we want everyone to be able to find your projects
(and our APIs!) on Google Code, no matter what search engine they use.
I'll follow up when this has been fixed.
-Fitz
