We may want to update that FAQ with some additional info later, but in the meantime, I thought I'd clear the air a bit:
1) Penalty? When your site has pages in our supplemental index, it does *not* indicate that your site has been penalized. In particular, we do not move a site's pages from our main to our supplemental index in response to any violations of our Webmaster Guidelines.
2) Freshness? You can expect to see a fresher supplemental index in the coming quarters. By the definition of "supplemental," however, I don't forsee it becoming as comprehensive or frequently updated as our main index.
3) Cure? Get more quality backlinks. This is a key way that our algorithms will view your pages as more valuable to retain in our main index.
If you'd simply like to get some old pages OUT of our supplemental index (and out of Google altogether), then you can use our automated removal tool: http://www.google.com/support/webmasters/bin/topic.py?topic=8459 Beware: Once removed from our index, the pages will STAY removed for 6 months and won't be reincluded during that time, so remove carefully (and only use this tool if absolutely necessary).
Thanks for your help Adam although i am confused by 'we do not move a site's pages from our main to our supplemental index in response to any violations of our Webmaster Guidelines'. Sorry if i have missed the obvious but why do you move it to the supplemental index? Your guidelines suggest it may be because the URL has too many parameters to crawl however if its impossible to crawl how did it end up in the supplemental list (it must have been crawled.) Do you mean quality backlinks to the site generally or to the specific page that was supplemented? If its the later then how do i get a quality backlink to a page on say 'privacy policy' or any other unique -to-the-site topic. 'see a fresher supplemental index in the coming quarters.' does this mean even more pages will be supplemented? Is there any chance you could introduce a user friendly search facilty for the supplemental indexes as most of our customers would never think of using a 'site' command .
> 1) Penalty? > When your site has pages in our supplemental index, it does *not* > indicate that your site has been penalized. In particular, we do not > move a site's pages from our main to our supplemental index in response > to any violations of our Webmaster Guidelines.
> 3) Cure? > Get more quality backlinks. This is a key way that our algorithms will > view your pages as more valuable to retain in our main index.
Wonderful, Adam. This confirms most of my gut feelings on the subject.
I'd only quesiton this a bit though:
"Once removed from our index, the pages will STAY removed for 6 months and won't be reincluded during that time ..."
I've been battling some old urls that have not existed in a very long time (over a year I think), left over from a total remake of my site where I replaced a crappy mambo installation with my own code. After redirecting some of the old zanny mambo url's for a while, I really wanted ALL that dead wood out of the index: active and supplemental, so I stopped redirecting and let them generate 404's. I've had them removed numerous times. They seemed to stay removed for a while after the last removal during this past summer (but not as long as 6 months, I'm sure of that) and then suddenly reappeared with a vengeance, back into suppplemental and more numerous than ever I remember them being (Mambo does generate an awful lot of crap url's). I gave up, hoping Googlebot will also get fed up with them. They all return 404's if accessed. They should die. They don't want to die LOL
I don't know if there are rogue links to those crummy pages on the net - I expect some old cached something or other soemwhere may exist. This would explain why they keep coming back. If ever there has been a legitimate link to any anywhere, if would have been to one page alone - anythign else woudl be from god knows what site rippers or caches or such. I could live with that one url, I could redirect it as well. I can't stand to see the old ones.
But can there ever be a satisfactory solution to this?
I understand the philosophy fully. I do have a problem understanding why sites with newer URLs, fewer backlinks, with pseudo copies of URLs from my site STILL have most or all URLs active, while most of my URLs including authority pages on some products have become supplemental. Over 10,000 URLs have gone supp including URLs with backlinks and PR leaving 54 current while competitors still have many thousands.
I would like to see most ecommerce URLs sent to supplemental and force ecommerce webmasters to use Google Base/Froogle to sell products. Then we would all be on a level playing field.
> 3) Cure? > Get more quality backlinks. This is a key way that our algorithms will > view your pages as more valuable to retain in our main index.
Well, that's just dandy! The way other webmasters find your content to make a determination on whether they should link to your site is to find you in Google's index for a particular search. But because supplemental pages with no backlinks are rarely if ever in any SERPs, no one will ever find your unique and relevant content as long as it exists only in supplementals, so it never gets any backlinks. Therefore, the only way to get backlinks to improve page rank is to buy them, which is counter to Google's stated position that purchased backlinks are bad, and can be penalized. Google has created one gigantic catch-22 feedback loop with this approach, and by doing so, is actually encouraging link buying schemes. Sorry, but I just don't see how this can be good for anyone.
Never had to buy or sollicit a backlink. Never. I'd never do it. I never add a link to a site either unless I personally find the site interesting and valuable in some aspect that's of interest and relevancae to my site - which means to me. If others do the same to my site, great. Not going to force them.
I's justy like saying Please and Thank you. It's good manners, not obligatory.
I guess we'll have to respectfully agree to disagree. Any site that makes a linking determination purely on the basis of where a site is in ANY index or what that site's PageRank is or whatever... clearly is not aiming to serve its users well. It's like saying "I'm only going to recommend this restaurant to my friends if it's already popular, listed in the NY Times" and so on.
Surely there is some feedback loop, that's undeniable, and that's human nature. But to suggest that the only way to get noticed and get links is to be already popular... well, that's discounting the vast number of Mom 'n' Pop sites (info, commerce, etc.) who have garnered attention and cash by doing amazingly un-radical (and fully appropriate) things in the promotional world.
Sell widgets? Post thoughtful, helpful comments on blogs that talk about the care of widgets, traveling to see widgets, widget troubleshooting, etc. Be a part of the online community. Get out into the "real world," too, speaking at tradeshows, volunteering some time with pro-bono assistance, whatever... but get out there. IMHO, any site that *depends* solely upon search engines for publicity and traffic is just asking for trouble. That's like a brick-and-mortar company saying, "Well, we put up a sign by our shop. Can't we just sit and wait for the phone calls to come in?!"
The days of getting tons of search traffic just by putting up a site are (thankfully) long-gone. If you don't make an effort to engage others or -- worse -- simply put up a flashy (or contrastingly boring-as-heck) Web site that's an island unto itself, why SHOULD others care, much less visit?
I think there is certainly content that Google can do a better job indexing and listing. And indeed, a core part of my job AND passion is to make things easier for smaller (good) sites to get seen. But in the meantime, I can only shake my head when people suggest that there are no productive, within-Webmaster-Guideline ways to garner awareness and traffic. It's just not so.
Forgive my toughlove today, but whether you take it as a kick in the pants or as a sign that -- on this gray Sunday afternoon -- I really feel passionately about the potential for smaller sites in Google, well, that's up to you.
> Surely there is some feedback loop, that's undeniable, and that's human
nature. But to suggest that the only way to get noticed and get links is to be already popular... well, that's discounting the vast number of
Mom 'n' Pop sites (info, commerce, etc.) who have garnered attention and cash by doing amazingly un-radical (and fully appropriate) things in the promotional world.
How can you reconcile that with the simple fact that http://freespace.virgin.net/keith.higgins2/ - and they don't come more "Mom and Pop" - hasn't even been spidered since 30 April? If you don't look, you don't see.
A small craft business, unique in its field, ignored by Google. In fact, banned, Can't see why. #3 on Yahoo for "pattern cutter yorkshire". #4 on Lycos.
Similar sitaution to Phil. As an example we are the only people in the whole world to make and sell a 'mother of pearl banana bar' which is why we are #1 in all the search engines for those search terms, except Google where it is not to be found. We have nutured publicity through all the normal chanels, trade shows etc and we are known for this unusual and unique item. The problem is internet users see that Googles results do not show any site related to this specific item, so they either assume no such item was ever made or they turn to the other search engines for help. As i see it, the reason the page (forgive the link) http://www.silverstall.com/bodyjew/banana-bar_mother-of-pearl_8.html is not in the googles main index is that it has been supplemented based on the cached page 'as retrieved on 2 May 2006' That page was updated a long time ago, so why does it take so long for Google to re-examine a supplemented page. I know our initial page was poorly titled and lacked contenet however is the any way a review of supplemental pages could be speeded up to say once a month?
