I have a network of sites that relate to the same category but they
are not the same. Just related. It makes sense that such sites are
inter linked in many places.
All of the sites are unique, have good content and also have natural
links, articles in newspapers etc.
Can relatively heavy interlinking cause a filter - sudden drop in
rankings of all participating sites?
Here is what I gathered from the net:
" Matt Cutts April 2008: Regarding interlinking between sites, he said
its fine to interlink if the sites are related, but he said not to
overdo it. When pressed, he said over 10 sites interlinking might be
asking for trouble. He said it would also be ok to break out your
network of sites and interlink sites within a certain category. The
specific example was a network of local sites, and Matt said you could
either have a single portal with links to all the geo-portals, or
maybe interlink between all the various plumbing sites.
Matt mentioned that sites dont automatically get pagerank just for
existing. They need backlinks to get pagerank. Also, he said if you
have a network of sites and add a bunch more sites, its like
spreading the same amount of peanut butter across a bigger piece of
bread. In that case, each site in the network gets a smaller share of
the pagerank distribution. http://www.seo.com/blog/google/matt-cutts-does-domain-roundtable/ taken from live interview by John Andrews in April 2008 @
http://www.johnon.com/543/mattcutts-domainroundtable.html
PS I personally don't see anything wrong with linking to other,
related sites that the same person owns, as long as it is done within
reason. Cross-linking a few related and relevant sites in the footer
is generally not a problem, but if you have too many sites, I would
recommend linking to one main "about us" page where you list them. Be
reasonable "
But then there is also:
From: http://www.ksl-consulting.co.uk/google_penalty.html "If you run more than one website and the Google penalty hits all
sites at the same time, check the interlinking (cross linking) between
those sites. Extensive interlinking of websites, particularly if they
are on the same C Class IP address (same ISP) can be viewed as "link
schemes" by Google, breaking their terms of service. The risks are
even higher where site A site wide links to site B and site B site
wide links back to site A. If you must use site wide links, make sure
they are not reciprocal links. Link schemes built around links in the
footer of each webpage are particularly risky. The reality is that
site wide links do little to increase site visibility in the Google
SERPS, nor do they improve Page Rank more than a single link, as
Google only counts one link from a site to another. KSL Consulting
also believe that Yahoo! now applies a similar policy. There is some
evidence that the extensive use of site wide links can lower website
Google trust value, which can subsequently reduce ranking."
Looking at your posting history and handle you seem to have a thing
about links.
When you are trying to work out what rules are you have to work out to
whom they may apply. They will apply differently to different types of
site, Matt was trying to explain that.
I have exactly what he describes a network of local sites that
currently have cross links, the bulk of the lower level cross-links
are re="nofollow" to remove 'excessiveness'. They also link to other
sites that we control but only at junctions where the content has
common relevance and the link add value to the user. None of these
links appear to cause issues for me. I probably break the rules on
some of the 2,200 pages especially the older ones but in overall terms
to a negligible amount. You can put as many links you like that are
"nofollow" because Google will ignore them. So the question is if you
feel your users need the link but you are worried that Google might
not like it, then "nofollow" them. But if you will not "nofollow" them
there must be another reason - that's the problem - then you're
pushing for PR.
Anyway the rules I try to follow with these are totally different to
that of my main site, all 2.6 million pages of it which includes two
content specific sub-domains. (1.3 million in the index, the bulk of
the rest are "nondexed" by me as unworthy of being in the index like
forms, applications and site specific instructions).
With these there is extensive cross-linking where content requires it
for example; the main site has the location, one sub-domain all the
images, the other sub-domain the wildlife that can be found there. By
themselves they are three distinct sites that can be used as such but
they also join together. The only reason to break them apart was the
sheer size.
In the UK, the BBC websites are the same, every major newspaper and
those other major media creators, eg.g the Mirror Group main site
links to over 160 websites of it's regional newspapers. They are all
in the index and all have extensive cross-linking, few of which are
"nofollow", but nothing to do with chasing PR just very relevant.
And that's the difference, rules depend upon what you are doing and
WHY.
> I have a network of sites that relate to the same category but they
> are not the same. Just related. It makes sense that such sites are
> inter linked in many places.
> All of the sites are unique, have good content and also have natural
> links, articles in newspapers etc.
> Can relatively heavy interlinking cause a filter - sudden drop in
> rankings of all participating sites?
> Here is what I gathered from the net:
> " Matt Cutts April 2008: Regarding interlinking between sites, he said
> its fine to interlink if the sites are related, but he said not to
> overdo it. When pressed, he said over 10 sites interlinking might be
> asking for trouble. He said it would also be ok to break out your
> network of sites and interlink sites within a certain category. The
> specific example was a network of local sites, and Matt said you could
> either have a single portal with links to all the geo-portals, or
> maybe interlink between all the various plumbing sites.
> Matt mentioned that sites dont automatically get pagerank just for
> existing. They need backlinks to get pagerank. Also, he said if you
> have a network of sites and add a bunch more sites, its like
> spreading the same amount of peanut butter across a bigger piece of
> bread. In that case, each site in the network gets a smaller share of
> the pagerank distribution.http://www.seo.com/blog/google/matt-cutts-does-domain-roundtable/ > taken from live interview by John Andrews in April 2008 @http://www.johnon.com/543/mattcutts-domainroundtable.html
> PS I personally don't see anything wrong with linking to other,
> related sites that the same person owns, as long as it is done within
> reason. Cross-linking a few related and relevant sites in the footer
> is generally not a problem, but if you have too many sites, I would
> recommend linking to one main "about us" page where you list them. Be
> reasonable "
> But then there is also:
> From:http://www.ksl-consulting.co.uk/google_penalty.html > "If you run more than one website and the Google penalty hits all
> sites at the same time, check the interlinking (cross linking) between
> those sites. Extensive interlinking of websites, particularly if they
> are on the same C Class IP address (same ISP) can be viewed as "link
> schemes" by Google, breaking their terms of service. The risks are
> even higher where site A site wide links to site B and site B site
> wide links back to site A. If you must use site wide links, make sure
> they are not reciprocal links. Link schemes built around links in the
> footer of each webpage are particularly risky. The reality is that
> site wide links do little to increase site visibility in the Google
> SERPS, nor do they improve Page Rank more than a single link, as
> Google only counts one link from a site to another. KSL Consulting
> also believe that Yahoo! now applies a similar policy. There is some
> evidence that the extensive use of site wide links can lower website
> Google trust value, which can subsequently reduce ranking."
One wonders with the ever repeating subject of links that if as much
effort was put into content creation whether you, a few others, John
Mu and I would have this second job!
> None of which seems immediately explicable - I've seen sites hit by
> penalties that I've assumed were caused by only one of those, let
> alone both.
jevstar wrote:
> One wonders with the ever repeating subject of links that if as much
> effort was put into content creation whether you, a few others, John
> Mu and I would have this second job!
It would be a different population - but the same size.
It would be all those displaced by those who'd starting doing some
work.
> True, but he now comes #1 for the vanity search that he first
> complained about - and even has sitelinks.
I cannot see.
But it's right this way- what is so easily detectable can be simply
neglected in most cases, and penalized in just a few.
Just I cannot understand why is it not obvious that these tricks
doesn't work- the link shemes (around "link echange dude") looks to be
even too excessive for Yahoo- only the homepage (of the original site)
indexed.
> None of which seems immediately explicable - I've seen sites hit by
> penalties that I've assumed were caused by only one of those, let
> alone both.
So I understand that the consensus over here is that running a link
exchange program where you create a directory and manually edit and
approve each link exachange is sthg that google frowns upon? Even
though the link exchange is one of the best ways to gain popularity
among sites? This is how the internet developed.....
Note that
http://www.hoteladvisor.com/links.html the file is called links.html - we do not hide that this is just link
exchange and that's all between related sites out there.
I would be grateful if someone could look and advise what to change in
order not to violate any guidelines.
> > True, but he now comes #1 for the vanity search that he first
> > complained about - and even has sitelinks.
> I cannot see.
> But it's right this way- what is so easily detectable can be simply
> neglected in most cases, and penalized in just a few.
> Just I cannot understand why is it not obvious that these tricks
> doesn't work- the link shemes (around "link echange dude") looks to be
> even too excessive for Yahoo- only the homepage (of the original site)
> indexed.
> On Nov 28, 3:33 pm, Phil Payne wrote:
> > jevstar wrote:
> > > Looking at your posting history and handle you seem to have a thing
> > > about links.
> > True, but he now comes #1 for the vanity search that he first
> > complained about - and even has sitelinks.
> > None of which seems immediately explicable - I've seen sites hit by
> > penalties that I've assumed were caused by only one of those, let
> > alone both.- Hide quoted text -
> So I understand that the consensus over here is that running a link
> exchange program where you create a directory and manually edit and
> approve each link exachange is sthg that google frowns upon? Even
> though the link exchange is one of the best ways to gain popularity
> among sites? This is how the internet developed.....
> Note thathttp://www.hoteladvisor.com/links.html > the file is called links.html - we do not hide that this is just link
> exchange and that's all between related sites out there.
> I would be grateful if someone could look and advise what to change in
> order not to violate any guidelines.
> Thanks!
> On Nov 28, 6:41 pm, BBdeath wrote:
> > > True, but he now comes #1 for the vanity search that he first
> > > complained about - and even has sitelinks.
> > I cannot see.
> > But it's right this way- what is so easily detectable can be simply
> > neglected in most cases, and penalized in just a few.
> > Just I cannot understand why is it not obvious that these tricks
> > doesn't work- the link shemes (around "link echange dude") looks to be
> > even too excessive for Yahoo- only the homepage (of the original site)
> > indexed.
> > On Nov 28, 3:33 pm, Phil Payne wrote:
> > > jevstar wrote:
> > > > Looking at your posting history and handle you seem to have a thing
> > > > about links.
> > > True, but he now comes #1 for the vanity search that he first
> > > complained about - and even has sitelinks.
> > > None of which seems immediately explicable - I've seen sites hit by
> > > penalties that I've assumed were caused by only one of those, let
> > > alone both.- Hide quoted text -
If you disallow a page in robots.txt the robot will never see it was
meant to have nofollow on it. Links found in previous crawl of that
page will likely remain cached.
Best to not disallow it . You can use the Noindex: directive instead
of the Disallow: directive and leave the meta tag to contain
"nofollow" (or "noindex,nofollow") .
Otherwise add rel="nofollow" to any external links as needed ( in case
some external links are actually legitimately related to your site and
are therefore OK to have).
The web was not built based on link exchanges. It was and is based on
links. Period. They have always been meant to be natural links, not
paid or traded. It used to be hard to tell programmatically so link
exchanges used to work for a long time. No longer the case. Google can
sniff them easily now even when disguised in roundabout schemes. Maybe
not immediately, on a first crawl, but eventually it comes around full
circle and figures it all out and then Wham! The sites that are
involved won't know what hit them.
> Set the <meta> tags on those pages "noindex,nofollow" and add
> re="nofollow" to the external URLs.
> If all the link stuff is in the same area on your site you could also
> "Disallow" those areas in your robots.txt file
> For safety's sake you could do all of it.
> Regards
> JV
> On Nov 30, 3:36 pm, Link Exchange Dude wrote:
> > Hello. Thanks for helping here.
> > So I understand that the consensus over here is that running a link
> > exchange program where you create a directory and manually edit and
> > approve each link exachange is sthg that google frowns upon? Even
> > though the link exchange is one of the best ways to gain popularity
> > among sites? This is how the internet developed.....
> > Note thathttp://www.hoteladvisor.com/links.html > > the file is called links.html - we do not hide that this is just link
> > exchange and that's all between related sites out there.
> > I would be grateful if someone could look and advise what to change in
> > order not to violate any guidelines.
> > Thanks!
> > On Nov 28, 6:41 pm, BBdeath wrote:
> > > > True, but he now comes #1 for the vanity search that he first
> > > > complained about - and even has sitelinks.
> > > I cannot see.
> > > But it's right this way- what is so easily detectable can be simply
> > > neglected in most cases, and penalized in just a few.
> > > Just I cannot understand why is it not obvious that these tricks
> > > doesn't work- the link shemes (around "link echange dude") looks to be
> > > even too excessive for Yahoo- only the homepage (of the original site)
> > > indexed.
> > > On Nov 28, 3:33 pm, Phil Payne wrote:
> > > > jevstar wrote:
> > > > > Looking at your posting history and handle you seem to have a thing
> > > > > about links.
> > > > True, but he now comes #1 for the vanity search that he first
> > > > complained about - and even has sitelinks.
> > > > None of which seems immediately explicable - I've seen sites hit by
> > > > penalties that I've assumed were caused by only one of those, let
> > > > alone both.- Hide quoted text -
Judging by the number of question in this group regarding robots.txt
that don't work, I think that the safest route is still to "noindex"
the displaying page if all that it has on it is external links and to
include the rel="nofollow" in the anchor tag of those external you
wish excluded.
The chances that un-skilled persons will make a mistake with their
robots text leaving the external links still active is a risk that I
think is now to high to take.
> If you disallow a page in robots.txt the robot will never see it was
> meant to have nofollow on it. Links found in previous crawl of that
> page will likely remain cached.
> Best to not disallow it . You can use the Noindex: directive instead
> of the Disallow: directive and leave the meta tag to contain
> "nofollow" (or "noindex,nofollow") .
> Otherwise add rel="nofollow" to any external links as needed ( in case
> some external links are actually legitimately related to your site and
> are therefore OK to have).
> The web was not built based on link exchanges. It was and is based on
> links. Period. They have always been meant to be natural links, not
> paid or traded. It used to be hard to tell programmatically so link
> exchanges used to work for a long time. No longer the case. Google can
> sniff them easily now even when disguised in roundabout schemes. Maybe
> not immediately, on a first crawl, but eventually it comes around full
> circle and figures it all out and then Wham! The sites that are
> involved won't know what hit them.
> On Nov 30, 10:48 am, jevstar wrote:
> > Hello again
> > Set the <meta> tags on those pages "noindex,nofollow" and add
> > re="nofollow" to the external URLs.
> > If all the link stuff is in the same area on your site you could also
> > "Disallow" those areas in your robots.txt file
> > For safety's sake you could do all of it.
> > Regards
> > JV
> > On Nov 30, 3:36 pm, Link Exchange Dude wrote:
> > > Hello. Thanks for helping here.
> > > So I understand that the consensus over here is that running a link
> > > exchange program where you create a directory and manually edit and
> > > approve each link exachange is sthg that google frowns upon? Even
> > > though the link exchange is one of the best ways to gain popularity
> > > among sites? This is how the internet developed.....
> > > Note thathttp://www.hoteladvisor.com/links.html > > > the file is called links.html - we do not hide that this is just link
> > > exchange and that's all between related sites out there.
> > > I would be grateful if someone could look and advise what to change in
> > > order not to violate any guidelines.
> > > Thanks!
> > > On Nov 28, 6:41 pm, BBdeath wrote:
> > > > > True, but he now comes #1 for the vanity search that he first
> > > > > complained about - and even has sitelinks.
> > > > I cannot see.
> > > > But it's right this way- what is so easily detectable can be simply
> > > > neglected in most cases, and penalized in just a few.
> > > > Just I cannot understand why is it not obvious that these tricks
> > > > doesn't work- the link shemes (around "link echange dude") looks to be
> > > > even too excessive for Yahoo- only the homepage (of the original site)
> > > > indexed.
> > > > On Nov 28, 3:33 pm, Phil Payne wrote:
> > > > > jevstar wrote:
> > > > > > Looking at your posting history and handle you seem to have a thing
> > > > > > about links.
> > > > > True, but he now comes #1 for the vanity search that he first
> > > > > complained about - and even has sitelinks.
> > > > > None of which seems immediately explicable - I've seen sites hit by
> > > > > penalties that I've assumed were caused by only one of those, let
> > > > > alone both.- Hide quoted text -
I'm just saying that if you disallow the page AFTER it's already been
cached and indexed, it may be too late to indicate that outgoing links
from it should be disregarded if you disallow it in robots.txt.
I would use the "noindex,nofollow" meta tag first until the page drops
out of the index.
After that I would disallow it in robots.txt so it won't come back
again.
Hopefully. Because some of mine handled exactly that way keep
reappearing (no title, no snippet), so that I will have to use the
Noindex: directive instead or in addition to the Disallow: directive.
I keep forgetting to do it.
> Judging by the number of question in this group regarding robots.txt
> that don't work, I think that the safest route is still to "noindex"
> the displaying page if all that it has on it is external links and to
> include the rel="nofollow" in the anchor tag of those external you
> wish excluded.
> The chances that un-skilled persons will make a mistake with their
> robots text leaving the external links still active is a risk that I
> think is now to high to take.
> Regards
> JV
> On Nov 30, 4:10 pm, webado wrote:
> > If you disallow a page in robots.txt the robot will never see it was
> > meant to have nofollow on it. Links found in previous crawl of that
> > page will likely remain cached.
> > Best to not disallow it . You can use the Noindex: directive instead
> > of the Disallow: directive and leave the meta tag to contain
> > "nofollow" (or "noindex,nofollow") .
> > Otherwise add rel="nofollow" to any external links as needed ( in case
> > some external links are actually legitimately related to your site and
> > are therefore OK to have).
> > The web was not built based on link exchanges. It was and is based on
> > links. Period. They have always been meant to be natural links, not
> > paid or traded. It used to be hard to tell programmatically so link
> > exchanges used to work for a long time. No longer the case. Google can
> > sniff them easily now even when disguised in roundabout schemes. Maybe
> > not immediately, on a first crawl, but eventually it comes around full
> > circle and figures it all out and then Wham! The sites that are
> > involved won't know what hit them.
> > On Nov 30, 10:48 am, jevstar wrote:
> > > Hello again
> > > Set the <meta> tags on those pages "noindex,nofollow" and add
> > > re="nofollow" to the external URLs.
> > > If all the link stuff is in the same area on your site you could also
> > > "Disallow" those areas in your robots.txt file
> > > For safety's sake you could do all of it.
> > > Regards
> > > JV
> > > On Nov 30, 3:36 pm, Link Exchange Dude wrote:
> > > > Hello. Thanks for helping here.
> > > > So I understand that the consensus over here is that running a link
> > > > exchange program where you create a directory and manually edit and
> > > > approve each link exachange is sthg that google frowns upon? Even
> > > > though the link exchange is one of the best ways to gain popularity
> > > > among sites? This is how the internet developed.....
> > > > Note thathttp://www.hoteladvisor.com/links.html > > > > the file is called links.html - we do not hide that this is just link
> > > > exchange and that's all between related sites out there.
> > > > I would be grateful if someone could look and advise what to change in
> > > > order not to violate any guidelines.
> > > > Thanks!
> > > > On Nov 28, 6:41 pm, BBdeath wrote:
> > > > > > True, but he now comes #1 for the vanity search that he first
> > > > > > complained about - and even has sitelinks.
> > > > > I cannot see.
> > > > > But it's right this way- what is so easily detectable can be simply
> > > > > neglected in most cases, and penalized in just a few.
> > > > > Just I cannot understand why is it not obvious that these tricks
> > > > > doesn't work- the link shemes (around "link echange dude") looks to be
> > > > > even too excessive for Yahoo- only the homepage (of the original site)
> > > > > indexed.
> > > > > On Nov 28, 3:33 pm, Phil Payne wrote:
> > > > > > jevstar wrote:
> > > > > > > Looking at your posting history and handle you seem to have a thing
> > > > > > > about links.
> > > > > > True, but he now comes #1 for the vanity search that he first
> > > > > > complained about - and even has sitelinks.
> > > > > > None of which seems immediately explicable - I've seen sites hit by
> > > > > > penalties that I've assumed were caused by only one of those, let
> > > > > > alone both.- Hide quoted text -
> > > > > - Show quoted text -- Hide quoted text -
For the person who can administer the order of what they are doing
carefully.
1) Firstly add rel="nofollow" to the external URL's
2) Leave everything else to allow Google to spider the page and
discover the new attributes.
3) Wait until you see the new cache that Google has taken of that
page. Open the cache in text only mode and then view the cache source
code. You can then see that Google has picked up the rel="nofollow" in
each link
4) When it has done so for all the required external links, if the
page is wholly and only containing external links as is often the case
- then mark the page <meta> "noindex,nofollow"
5) If you have a large number of external link pages in a directory
proceed
in this manner treating each page individually.
6) Once all the pages in the directory have been attended to then the
directory can be blocked with a robots.txt statement
By following this process you can be sure that all the link
attributes, that is rel="nofollow" have been absorbed by Google
without exception.
First off, thank you very much for your amazing involvement here in
the groups! It's great to see new faces who are helping out with so
much good information!
I just wanted to add a comment regarding part of what you wrote:
> Judging by the number of question in this group regarding robots.txt
> that don't work, I think that the safest route is still to "noindex"
> the displaying page if all that it has on it is external links and to
> include the rel="nofollow" in the anchor tag of those external you
> wish excluded.
Should you -- or anyone else -- ever feel that we are not honoring the
robots.txt directives, please take the time to make a big fuss about
it, change the subject, include lots of exclamation marks, etc :-).
Really, it's important to us. If anything, we should always be
following those directives.
That said, there are a few caveats which I thought I'd mention while
I'm here:
- Keep in mind that the "disallow" directive is (by design) for
crawling, not for indexing. So if a page is already known, we may keep
it in the index even if we can't crawl it anymore. It might be
relevant to the users, so we might want to send them there. If a page
is not yet known, we may still index it with the URL only. It's rare
that we would provide just the URL to a user searching for something
in specific, but it's possible.
- If you want to have a page removed from the index, using the robots
noindex meta tag is probably the easiest way. However, we'd only see
that if we were allowed to crawl it :).
- If the site is running on IIS or any other non-case-sensitive
webserver, you will need to be very careful with regards to the
robots.txt file. The robot.txt file is case-sensitive (as URLs are by
definition), so with a directive "disallow: /folder/" you could still
have http://example.com/Folder/ crawled and indexed. Technically, you
would have to specify all permutations of the upper/lower case folder
name (which in practice is difficult), so it would make sense to use
robots meta tags in addition for non-case-sensitive servers.
I hope that makes sense, let me know if you all have any more
questions :-) and keep up the good work, everyone!!
> First off, thank you very much for your amazing involvement here in
> the groups! It's great to see new faces who are helping out with so
> much good information!
> I just wanted to add a comment regarding part of what you wrote:
> > Judging by the number of question in this group regarding robots.txt
> > that don't work, I think that the safest route is still to "noindex"
> > the displaying page if all that it has on it is external links and to
> > include the rel="nofollow" in the anchor tag of those external you
> > wish excluded.
> Should you -- or anyone else -- ever feel that we are not honoring the
> robots.txt directives, please take the time to make a big fuss about
> it, change the subject, include lots of exclamation marks, etc :-).
> Really, it's important to us. If anything, we should always be
> following those directives.
> That said, there are a few caveats which I thought I'd mention while
> I'm here:
> - Keep in mind that the "disallow" directive is (by design) for
> crawling, not for indexing. So if a page is already known, we may keep
> it in the index even if we can't crawl it anymore. It might be
> relevant to the users, so we might want to send them there. If a page
> is not yet known, we may still index it with the URL only. It's rare
> that we would provide just the URL to a user searching for something
> in specific, but it's possible.
> - If you want to have a page removed from the index, using the robots
> noindex meta tag is probably the easiest way. However, we'd only see
> that if we were allowed to crawl it :).
> - If the site is running on IIS or any other non-case-sensitive
> webserver, you will need to be very careful with regards to the
> robots.txt file. The robot.txt file is case-sensitive (as URLs are by
> definition), so with a directive "disallow: /folder/" you could still
> havehttp://example.com/Folder/crawled and indexed. Technically, you
> would have to specify all permutations of the upper/lower case folder
> name (which in practice is difficult), so it would make sense to use
> robots meta tags in addition for non-case-sensitive servers.
> I hope that makes sense, let me know if you all have any more
> questions :-) and keep up the good work, everyone!!
If we can't access the robots.txt (say the server is unreachable or
returns a 5xx error result code) then we won't crawl the website at
all. So in other words, if anything, the robots.txt file should be
reachable if you want the website to be crawled and indexed :).
Usually if the robots.txt is unreachable or returns a server side
error, then the rest of the site is similarly inaccessible though.
Looking a bit further, if your server goes down or you are doing
maintenance and you need to use the robots.txt file to control
crawling of the website, then you MUST make sure not to return the
"The server is down" page with a 200 result code. Doing that could
result in us seeing this page instead of the robots.txt page, which
could result in us crawling and indexing content that we shouldn't
once the site is back up. Optimally, all such "the server is down"
URLs should return result code 503 (service unavailable). Doing that
also prevents search engines from crawling and indexing the error
pages :-). Sometimes I'm surprised at how many large sites forget to
do this...
I personally have not experience the issue of Googlebots ignoring the
robots.txt. It was something mentioned earlier in the thread.
I am aware however of the number of people who have difficulty with
compiling a good robots.txt which is something I think you (Google)
are aware of too, hence the robots checker in WMT.
As a consequence of this I think the individual rel="nofollow" on each
anchor tag is the safest route for many folks.
> If we can't access the robots.txt (say the server is unreachable or
> returns a 5xx error result code) then we won't crawl the website at
> all. So in other words, if anything, the robots.txt file should be
> reachable if you want the website to be crawled and indexed :).
> Usually if the robots.txt is unreachable or returns a server side
> error, then the rest of the site is similarly inaccessible though.
> Looking a bit further, if your server goes down or you are doing
> maintenance and you need to use the robots.txt file to control
> crawling of the website, then you MUST make sure not to return the
> "The server is down" page with a 200 result code. Doing that could
> result in us seeing this page instead of the robots.txt page, which
> could result in us crawling and indexing content that we shouldn't
> once the site is back up. Optimally, all such "the server is down"
> URLs should return result code 503 (service unavailable). Doing that
> also prevents search engines from crawling and indexing the error
> pages :-). Sometimes I'm surprised at how many large sites forget to
> do this...
JohnMu wrote:
> Looking a bit further, if your server goes down or you are doing
> maintenance and you need to use the robots.txt file to control
> crawling of the website, then you MUST make sure not to return the
> "The server is down" page with a 200 result code. Doing that could
> result in us seeing this page instead of the robots.txt page, which
> could result in us crawling and indexing content that we shouldn't
> once the site is back up.
Although that's very good advice, I currently suspect that Google
requires not only a 200 on robots.txt but also a successful parsing of
the file before it proceeds. There seems to have been a major change
in robots.txt processing in the early summer introducing "failsafe"
handling of the file. Which makes great sense from a legal exposure
reduction point of view.
I've certainly seen at least three cases of robots.txt files being
delivered with 200 server responses but a BOM at the start - and
Google has stopped crawling that site. Removing the BOM has restarted
crawling.
If there's one thing HIGH on my wish list, it's a message from Google
whenever it fails to parse a file. I'm convinced that there are many
instances where such a message would be win/win for Google and
webmasters alike. For "control" type input like sitemaps, robots.txt,
etc., then the form would be: "Failed to parse xxxx successully"
leaving the user to find tools, validators, etc., to solve the
problem. For web pages, perhaps just: "Recovery occured during
parsing - some content may have been flushed."
Only a heads-up - nothing more than that. And if it addressed only
provable syntax errors and the like, no downside for Google.
> I'm just saying that if you disallow the page AFTER it's already been
> cached and indexed, it may be too late to indicate that outgoing links
> from it should be disregarded if you disallow it in robots.txt.
> I would use the "noindex,nofollow" meta tag first until the page drops
> out of the index.
> After that I would disallow it in robots.txt so it won't come back
> again.
> Hopefully. Because some of mine handled exactly that way keep
> reappearing (no title, no snippet), so that I will have to use the
> Noindex: directive instead or in addition to the Disallow: directive.
> I keep forgetting to do it.
> On Nov 30, 11:34 am, jevstar wrote:
> > Judging by the number of question in this group regarding robots.txt
> > that don't work, I think that the safest route is still to "noindex"
> > the displaying page if all that it has on it is external links and to
> > include the rel="nofollow" in the anchor tag of those external you
> > wish excluded.
> > The chances that un-skilled persons will make a mistake with their
> > robots text leaving the external links still active is a risk that I
> > think is now to high to take.
> > Regards
> > JV
> > On Nov 30, 4:10 pm, webado wrote:
> > > If you disallow a page in robots.txt the robot will never see it was
> > > meant to have nofollow on it. Links found in previous crawl of that
> > > page will likely remain cached.
> > > Best to not disallow it . You can use the Noindex: directive instead
> > > of the Disallow: directive and leave the meta tag to contain
> > > "nofollow" (or "noindex,nofollow") .
> > > Otherwise add rel="nofollow" to any external links as needed ( in case
> > > some external links are actually legitimately related to your site and
> > > are therefore OK to have).
> > > The web was not built based on link exchanges. It was and is based on
> > > links. Period. They have always been meant to be natural links, not
> > > paid or traded. It used to be hard to tell programmatically so link
> > > exchanges used to work for a long time. No longer the case. Google can
> > > sniff them easily now even when disguised in roundabout schemes. Maybe
> > > not immediately, on a first crawl, but eventually it comes around full
> > > circle and figures it all out and then Wham! The sites that are
> > > involved won't know what hit them.
> > > On Nov 30, 10:48 am, jevstar wrote:
> > > > Hello again
> > > > Set the <meta> tags on those pages "noindex,nofollow" and add
> > > > re="nofollow" to the external URLs.
> > > > If all the link stuff is in the same area on your site you could also
> > > > "Disallow" those areas in your robots.txt file
> > > > For safety's sake you could do all of it.
> > > > Regards
> > > > JV
> > > > On Nov 30, 3:36 pm, Link Exchange Dude wrote:
> > > > > Hello. Thanks for helping here.
> > > > > So I understand that the consensus over here is that running a link
> > > > > exchange program where you create a directory and manually edit and
> > > > > approve each link exachange is sthg that google frowns upon? Even
> > > > > though the link exchange is one of the best ways to gain popularity
> > > > > among sites? This is how the internet developed.....
> > > > > Note thathttp://www.hoteladvisor.com/links.html > > > > > the file is called links.html - we do not hide that this is just link
> > > > > exchange and that's all between related sites out there.
> > > > > I would be grateful if someone could look and advise what to change in
> > > > > order not to violate any guidelines.
> > > > > Thanks!
> > > > > On Nov 28, 6:41 pm, BBdeath wrote:
> > > > > > > True, but he now comes #1 for the vanity search that he first
> > > > > > > complained about - and even has sitelinks.
> > > > > > I cannot see.
> > > > > > But it's right this way- what is so easily detectable can be simply
> > > > > > neglected in most cases, and penalized in just a few.
> > > > > > Just I cannot understand why is it not obvious that these tricks
> > > > > > doesn't work- the link shemes (around "link echange dude") looks to be
> > > > > > even too excessive for Yahoo- only the homepage (of the original site)
> > > > > > indexed.
> > > > > > On Nov 28, 3:33 pm, Phil Payne wrote:
> > > > > > > jevstar wrote:
> > > > > > > > Looking at your posting history and handle you seem to have a thing
> > > > > > > > about links.
> > > > > > > True, but he now comes #1 for the vanity search that he first
> > > > > > > complained about - and even has sitelinks.
> > > > > > > None of which seems immediately explicable - I've seen sites hit by
> > > > > > > penalties that I've assumed were caused by only one of those, let
> > > > > > > alone both.- Hide quoted text -
> > > > > > - Show quoted text -- Hide quoted text -
> Although that's very good advice, I currently suspect that Google
> requires not only a 200 on robots.txt but also a successful parsing of
> the file before it proceeds. There seems to have been a major change
> in robots.txt processing in the early summer introducing "failsafe"
> handling of the file. Which makes great sense from a legal exposure
> reduction point of view.
