Gmail Calendar Documents Reader Web more »
Recently Visited Groups | Help | Sign in
Google Groups Home
Message from discussion When does link exchange between sites owned by the same company cause a google penalty
The group you are posting to is a Usenet group. Messages posted to this group will make your email address visible to anyone on the Internet.
Your reply message has not been sent.
Your post was successful
 
From:
To:
Cc:
Followup To:
Add Cc | Add Followup-to | Edit Subject
Subject:
Validation:
For verification purposes please type the characters you see in the picture below or the numbers you hear by clicking the accessibility icon. Listen and type the numbers you hear
 
JohnMu Google employee  
View profile  
 More options Dec 1 2008, 7:21 am
From: JohnMu
Date: Mon, 1 Dec 2008 04:21:38 -0800 (PST)
Local: Mon, Dec 1 2008 7:21 am
Subject: Re: When does link exchange between sites owned by the same company cause a google penalty
Hi BBdeath

If we can't access the robots.txt (say the server is unreachable or
returns a 5xx error result code) then we won't crawl the website at
all. So in other words, if anything, the robots.txt file should be
reachable if you want the website to be crawled and indexed :).
Usually if the robots.txt is unreachable or returns a server side
error, then the rest of the site is similarly inaccessible though.

Looking a bit further, if your server goes down or you are doing
maintenance and you need to use the robots.txt file to control
crawling of the website, then you MUST make sure not to return the
"The server is down" page with a 200 result code. Doing that could
result in us seeing this page instead of the robots.txt page, which
could result in us crawling and indexing content that we shouldn't
once the site is back up. Optimally, all such "the server is down"
URLs should return result code 503 (service unavailable). Doing that
also prevents search engines from crawling and indexing the error
pages :-). Sometimes I'm surprised at how many large sites forget to
do this...

Cheers
John


    Forward  
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.

Create a group - Google Groups - Google Home - Terms of Service - Privacy Policy
©2009 Google