We may want to update that FAQ with some additional info later, but in the meantime, I thought I'd clear the air a bit:
1) Penalty? When your site has pages in our supplemental index, it does *not* indicate that your site has been penalized. In particular, we do not move a site's pages from our main to our supplemental index in response to any violations of our Webmaster Guidelines.
2) Freshness? You can expect to see a fresher supplemental index in the coming quarters. By the definition of "supplemental," however, I don't forsee it becoming as comprehensive or frequently updated as our main index.
3) Cure? Get more quality backlinks. This is a key way that our algorithms will view your pages as more valuable to retain in our main index.
If you'd simply like to get some old pages OUT of our supplemental index (and out of Google altogether), then you can use our automated removal tool: http://www.google.com/support/webmasters/bin/topic.py?topic=8459 Beware: Once removed from our index, the pages will STAY removed for 6 months and won't be reincluded during that time, so remove carefully (and only use this tool if absolutely necessary).
Thanks for your help Adam although i am confused by 'we do not move a site's pages from our main to our supplemental index in response to any violations of our Webmaster Guidelines'. Sorry if i have missed the obvious but why do you move it to the supplemental index? Your guidelines suggest it may be because the URL has too many parameters to crawl however if its impossible to crawl how did it end up in the supplemental list (it must have been crawled.) Do you mean quality backlinks to the site generally or to the specific page that was supplemented? If its the later then how do i get a quality backlink to a page on say 'privacy policy' or any other unique -to-the-site topic. 'see a fresher supplemental index in the coming quarters.' does this mean even more pages will be supplemented? Is there any chance you could introduce a user friendly search facilty for the supplemental indexes as most of our customers would never think of using a 'site' command .
> 1) Penalty? > When your site has pages in our supplemental index, it does *not* > indicate that your site has been penalized. In particular, we do not > move a site's pages from our main to our supplemental index in response > to any violations of our Webmaster Guidelines.
> 3) Cure? > Get more quality backlinks. This is a key way that our algorithms will > view your pages as more valuable to retain in our main index.
Wonderful, Adam. This confirms most of my gut feelings on the subject.
I'd only quesiton this a bit though:
"Once removed from our index, the pages will STAY removed for 6 months and won't be reincluded during that time ..."
I've been battling some old urls that have not existed in a very long time (over a year I think), left over from a total remake of my site where I replaced a crappy mambo installation with my own code. After redirecting some of the old zanny mambo url's for a while, I really wanted ALL that dead wood out of the index: active and supplemental, so I stopped redirecting and let them generate 404's. I've had them removed numerous times. They seemed to stay removed for a while after the last removal during this past summer (but not as long as 6 months, I'm sure of that) and then suddenly reappeared with a vengeance, back into suppplemental and more numerous than ever I remember them being (Mambo does generate an awful lot of crap url's). I gave up, hoping Googlebot will also get fed up with them. They all return 404's if accessed. They should die. They don't want to die LOL
I don't know if there are rogue links to those crummy pages on the net - I expect some old cached something or other soemwhere may exist. This would explain why they keep coming back. If ever there has been a legitimate link to any anywhere, if would have been to one page alone - anythign else woudl be from god knows what site rippers or caches or such. I could live with that one url, I could redirect it as well. I can't stand to see the old ones.
But can there ever be a satisfactory solution to this?
I understand the philosophy fully. I do have a problem understanding why sites with newer URLs, fewer backlinks, with pseudo copies of URLs from my site STILL have most or all URLs active, while most of my URLs including authority pages on some products have become supplemental. Over 10,000 URLs have gone supp including URLs with backlinks and PR leaving 54 current while competitors still have many thousands.
I would like to see most ecommerce URLs sent to supplemental and force ecommerce webmasters to use Google Base/Froogle to sell products. Then we would all be on a level playing field.
> 3) Cure? > Get more quality backlinks. This is a key way that our algorithms will > view your pages as more valuable to retain in our main index.
Well, that's just dandy! The way other webmasters find your content to make a determination on whether they should link to your site is to find you in Google's index for a particular search. But because supplemental pages with no backlinks are rarely if ever in any SERPs, no one will ever find your unique and relevant content as long as it exists only in supplementals, so it never gets any backlinks. Therefore, the only way to get backlinks to improve page rank is to buy them, which is counter to Google's stated position that purchased backlinks are bad, and can be penalized. Google has created one gigantic catch-22 feedback loop with this approach, and by doing so, is actually encouraging link buying schemes. Sorry, but I just don't see how this can be good for anyone.
Never had to buy or sollicit a backlink. Never. I'd never do it. I never add a link to a site either unless I personally find the site interesting and valuable in some aspect that's of interest and relevancae to my site - which means to me. If others do the same to my site, great. Not going to force them.
I's justy like saying Please and Thank you. It's good manners, not obligatory.
I guess we'll have to respectfully agree to disagree. Any site that makes a linking determination purely on the basis of where a site is in ANY index or what that site's PageRank is or whatever... clearly is not aiming to serve its users well. It's like saying "I'm only going to recommend this restaurant to my friends if it's already popular, listed in the NY Times" and so on.
Surely there is some feedback loop, that's undeniable, and that's human nature. But to suggest that the only way to get noticed and get links is to be already popular... well, that's discounting the vast number of Mom 'n' Pop sites (info, commerce, etc.) who have garnered attention and cash by doing amazingly un-radical (and fully appropriate) things in the promotional world.
Sell widgets? Post thoughtful, helpful comments on blogs that talk about the care of widgets, traveling to see widgets, widget troubleshooting, etc. Be a part of the online community. Get out into the "real world," too, speaking at tradeshows, volunteering some time with pro-bono assistance, whatever... but get out there. IMHO, any site that *depends* solely upon search engines for publicity and traffic is just asking for trouble. That's like a brick-and-mortar company saying, "Well, we put up a sign by our shop. Can't we just sit and wait for the phone calls to come in?!"
The days of getting tons of search traffic just by putting up a site are (thankfully) long-gone. If you don't make an effort to engage others or -- worse -- simply put up a flashy (or contrastingly boring-as-heck) Web site that's an island unto itself, why SHOULD others care, much less visit?
I think there is certainly content that Google can do a better job indexing and listing. And indeed, a core part of my job AND passion is to make things easier for smaller (good) sites to get seen. But in the meantime, I can only shake my head when people suggest that there are no productive, within-Webmaster-Guideline ways to garner awareness and traffic. It's just not so.
Forgive my toughlove today, but whether you take it as a kick in the pants or as a sign that -- on this gray Sunday afternoon -- I really feel passionately about the potential for smaller sites in Google, well, that's up to you.
> Surely there is some feedback loop, that's undeniable, and that's human
nature. But to suggest that the only way to get noticed and get links is to be already popular... well, that's discounting the vast number of
Mom 'n' Pop sites (info, commerce, etc.) who have garnered attention and cash by doing amazingly un-radical (and fully appropriate) things in the promotional world.
How can you reconcile that with the simple fact that http://freespace.virgin.net/keith.higgins2/ - and they don't come more "Mom and Pop" - hasn't even been spidered since 30 April? If you don't look, you don't see.
A small craft business, unique in its field, ignored by Google. In fact, banned, Can't see why. #3 on Yahoo for "pattern cutter yorkshire". #4 on Lycos.
Similar sitaution to Phil. As an example we are the only people in the whole world to make and sell a 'mother of pearl banana bar' which is why we are #1 in all the search engines for those search terms, except Google where it is not to be found. We have nutured publicity through all the normal chanels, trade shows etc and we are known for this unusual and unique item. The problem is internet users see that Googles results do not show any site related to this specific item, so they either assume no such item was ever made or they turn to the other search engines for help. As i see it, the reason the page (forgive the link) http://www.silverstall.com/bodyjew/banana-bar_mother-of-pearl_8.html is not in the googles main index is that it has been supplemented based on the cached page 'as retrieved on 2 May 2006' That page was updated a long time ago, so why does it take so long for Google to re-examine a supplemented page. I know our initial page was poorly titled and lacked contenet however is the any way a review of supplemental pages could be speeded up to say once a month?
A rhetorical question at this point - I have seen what it is. Still scratching my head over it .... LOL
But see this means the people already know exactly what they are looking for.
Maybe I am just interested in body piercing jewelery (god forbid, but whatever LOL). Then I will maybe find the site and browse it and maybe I'll find something I like. I dont' think I'd search for a banana bar anything - would I? Well, maybe some kids today would, don't know ...
I had a site that, 12 months ago, was at zero. No page rank, alexa traffic rank at 4+ million and basically no visitors.
I put up about 25 links on a 'links page'. No reciprocal programs, no unrelated sites, just sites that I thought would be good for my audience. Then I sent a quick email to those webmasters basically saying "nice site, I linked to you, here I am, feel free to link back".
Well, a funny thing happened on the way to the forum. While I wasn't even paying attention about 9 months ago traffic started to pick up steadily. A friend sends an email to me and says "how did you get a Google page rank or 5?". I didn't even know. Sure enough, Google traffic was up, PR was 5, alexa was under 1 million and showing 17 backlinks.
It really wasn't very hard. It just took about 3-6 months.
> > 3) Cure? > > Get more quality backlinks. This is a key way that our algorithms will > > view your pages as more valuable to retain in our main index.
> Well, that's just dandy! The way other webmasters find your content to > make a determination on whether they should link to your site is to > find you in Google's index for a particular search. But because > supplemental pages with no backlinks are rarely if ever in any SERPs, > no one will ever find your unique and relevant content as long as it > exists only in supplementals, so it never gets any backlinks. > Therefore, the only way to get backlinks to improve page rank is to buy > them, which is counter to Google's stated position that purchased > backlinks are bad, and can be penalized. Google has created one > gigantic catch-22 feedback loop with this approach, and by doing so, is > actually encouraging link buying schemes. Sorry, but I just don't see > how this can be good for anyone.
Hi Guys and thanks for all your input. As an experienced web-master pointed out to me putting 'mother' 'pearl' banana' and 'bar' all in the same phrase has connotations which even the most banned pornographic sites would have trouble creating. Damn i knew we should have called it 'belly accessory made from shell.'
It does however highlight the fact that Google seems far stricter than the other search engines - maybe as a result of having to deal with so many more web-pages. I can understand the strict criteria for competitive keywords such as money, mortgage etc however for less compettive (or in our case zero competitive) can't google loosen up a bit?
Phil, i'm sure the 'freespace.net' bit of the URL is a killer with google - maybe i'm wrong
Thanks for posting this Adam. I'm hoping you'll be good enough to address a few things in detail. First, WHY goes G even have to have something as useless and hurtful as this? Being in supplemental in ITSELF **IS a penalty**, you're as good as deleted because no one will ever find you!
Please explain what exactly is meant by: "*the number of parameters in a URL* might exclude a site from being crawled for inclusion in our main index"? Is the "number of parameters" meaning what's physically IN the URL, like hyphens, dashes, number of words, etc.? If so, what is too many? Why would G be THE ONLY SE that penalizes people for trying to use USER FRIENDLY and EASILY product-IDENTIFIED URL's?
And:
> 2) Freshness? > You can expect to see a fresher supplemental index in the coming > quarters. By the definition of "supplemental," however, I don't foresee > it becoming as comprehensive or frequently updated as our main index.
That is only seriously hurting people, in particular, your "Mom & Pop" places (of which you indicated you are so fond).
> 3) Cure? > Get more quality backlinks. This is a key way that our algorithms will > view your pages as more valuable to retain in our main index.
Ok, so HOW, is anyone supposed to get hundreds of people to link to *ALL* of your pages? Does G think everyone is Amazon.com?? That's not only impractical, it's impossible. That also is ruining Mom & Pop and small business places! Now hopefully, you're talking about IBL's to one's HOMEPAGE....right? If not, that criteria is just plain insane. Like another said, you'd have to go out and bribe/pay hundreds of websites to link to all of your pages. No, the "quality" or "need" of the pages has nothing to do with it. You just can't get people to link to every product's sales page unless it's a PERMANENT product on your site (which no one has unless they've INVENTED the products), have the cheapest price on the planet, and a massive amount of the population needs it! That's just screwing people trying to make ends meet, catering to the multi-million $ monopolies, and ruining the very backbone of the (American) economy-small privately owned business.
You, nor anyone, will address WHY, WHY pages are placed in the sup index. Please tell me, why webpages that are: unique, one-of-a-kind, white-hat, well-linked-from the main or a main webpage; AND were FIRST on the FIRST page of results for searches, get deleted from the main index?? That is just totally senseless. If this is a "bug", then please say so. If it is not a bug, then please say so, and *why* G is insistent upon doing it.
You *should* know, that all is this is just making G only a shadow of its former self. One used to be able to find ANYTHING on G, now, that is no longer the case because so many pages....ARE IN THE SUPPLEMENTAL INDEX!
PLEASE address these points, I'm only one of millions that need to know. Thank you for your time.
> I put up about 25 links on a 'links page'. No reciprocal programs, no > unrelated sites, just sites that I thought would be good for my > audience. Then I sent a quick email to those webmasters basically > saying "nice site, I linked to you, here I am, feel free to link back".
> The days of getting tons of search traffic just by putting up a site > are (thankfully) long-gone. If you don't make an effort to engage > others or -- worse -- simply put up a flashy (or contrastingly > boring-as-heck) Web site that's an island unto itself, why SHOULD > others care, much less visit?
> Regards, > Adam
This statement just makes me want to cry! I guess Affiliate Marketing is now DEAD.
I wish I knew this 6 months before I started building a site. Right now, every month I just keep paying my website hosting bills with no revenue from any Ads that show on the site.
Now, I'll just shut down my sites and move on back to doing something else.
We'll I;ll still keep my sites up, just to practice my programming skills and web building skills but. I won't expect anyone to visit them :...-(
As a single person trying to build the website, I don't have the energy or the time to reach out to the public and advertise my site in other ways.
I do have a few offline-advertising ideas like the pro-bono ones. But hmm, I'll have to consider them.
Adam Lasnik wrote: > Any site that makes a linking determination purely on the basis of > where a site is in ANY index or what that site's PageRank is or > whatever... clearly is not aiming to serve its users well.
Perhaps not. HOWEVER, G has FORCED people to do this!! No one can have a site any longer that is just based on "aiming to serve its users well" because GOOGLE doesn't like it!! You have to "aim to SERVE GOOGLE", and that is NOT right nor the way it should be, yet, it is! It is GOOGLE that has created this!
> Surely there is some feedback loop, that's undeniable, and that's human > nature. But to suggest that the only way to get noticed and get links > is to be already popular... well, that's discounting the vast number of > Mom 'n' Pop sites (info, commerce, etc.) who have garnered attention > and cash by doing amazingly un-radical (and fully appropriate) things > in the promotional world.
That cannot be applied to EVERY niche. Furthermore, that IS indeed accurate: "the only way to get noticed and get links is to be already popular", because no one is going to WANT to link to pages (that DID exist and SHOULD exist) but now don't exist in the main index!
> IMHO, any > site that *depends* solely upon search engines for publicity and > traffic is just asking for trouble. That's like a brick-and-mortar > company saying, "Well, we put up a sign by our shop. Can't we just sit > and wait for the phone calls to come in?!"
Well, as far as G goes, YES.
The B&M analogy isn't exactly accurate. ;-) A far greater number of people use SE's than drive by a store. Those that drive by your store, aren't looking for what you sell. They are going from point A to point B.
> The days of getting tons of search traffic just by putting up a site > are (thankfully) long-gone.(1) If you don't make an effort to engage > others or -- worse -- simply put up a flashy (or contrastingly > boring-as-heck) Web site that's an island unto itself, why SHOULD > others care, much less visit?(2)
(1) Spoken like a true person that couldn't care less about SERP's because you DON'T HAVE A BUSINESS that needs to be IN the SERP's! Walk in my, and millions of other's shoes, then say that...you wouldn't. You have no compassion nor caring for those innocents that are being victimized by YOUR company. (2) Any "effort to engage others" is totally useless and waste of time if your pages **ARE NOT IN G**. Additionally, people should not be penalized for not having the money to hire top webmasters and designers, or for being "boring". That's another way G is discriminatory.
> I think there is certainly content that Google can do a better job > indexing and listing. And indeed, a core part of my job AND passion is > to make things easier for smaller (good) sites to get seen.
Easier? Then why doesn't G DELETE or put into supplemental, all the SPAMMY ILLEGAL BLACKHAT G-TOS-violating websites that are HIJACKING and STEALING MY data causing MY pages to get deleted from the index??
Please address these Adam. Thank you, and thanks again for this thread.
Adam i think in the real worl of marketing you need to establish a demand and need through as you say 'un-radical (and fully appropriate) things in the promotional world.' The problem is that despite doing that Google is ignoring that marketing and basing its results firstly on an extremely pedantic and strict criteria and secondly on links. As a user i am frequently finding what i need from your rival search engines because you have not indexed the relevant page due to Google throwing it in the supplemental pages. (the banana bar thingy mentioned in this thread is a good example although god forbid its not the type of thing i personally would look for).The problem from a marketing perspective is that if a new online product, artcile or book etc is launched at great expense, all that effort and energy is diminished if months later Google effectively tells 90% of all internet users that no such product exists. Unless Google relaxs their quality (or whatever it is) filters i fear the quality of Googles search results will continue to decline. It needs to speed up the process of rechecking the supplemental indexes as there are many pages in them that haven't been re-examined for months - sometimes years. I think Rick has a point when he says G has FORCED people to determine their link strategy on serp position and PR - i know i would not like a reciprocal link to a site with a PR0 no matter how relevant to my business that site sould be. As for 'Post thoughtful, helpful comments on blogs' is that not encouraging commercial sites to spam blogs. I thought the whole point of Blogs was to rapidy publish thoughts and ideas not as a marketing tool for promoting products. Nevertheless Adam i am grateful that you and Google have taken the time and consideratrion to talk to us humble mortals but please try and take on board some of the comments that have been made here as it will help Google become a better search engine.
> This statement just makes me want to cry! I guess Affiliate Marketing > is now DEAD.
No, it's not dead, but it's more difficult than (perhaps) it once was.
Do you want to go to a page full of adverts? No? Then why do you think anybody else does?
Google aims to index CONTENT. Maybe it doesn't always succeed in that aim, but that's what they're trying to do. If you can build a site with useful content, from which users can also buy stuff that will make you money, then affiliate marketing can still work for you.
If you just want to slap up a virtual billboard of affiliate links and wait for the millions to roll in, you're gonna be disappointed.
apologies to interupt the debate on Affiliate marketing but just thought i'd mention that our banana bar page is on page 2 of google's main index. This reaffrims the point that even if the page is in the supplemental index it can still appear in the main Google index and i must apologise to Adam for my previous bleetings. (except the one about speeding up the re-examination of supplemental pages)
To return to affiliate marketing, it has gained a bad reputation (rightly or wrongly) firstly as it was considered to be a major source of e-mail spam and seconly many affiliates create large amounts of autogenerated webpages each devoted to different competitive keywords, which is the reason, as i understand Google apply a stricter criteria.
>From my perspective the only way to make it work is to set up your own
network and carefully police everyone who joins that network, making sure that content is not duplicated throughout that network. I may be talking totally complete rubbish as i have no personal experience of affiliate marketing except avoiding any web-page festooned with adverts.
silverstall wrote: > apologies to interupt the debate on Affiliate marketing but just > thought i'd mention that our banana bar page is on page 2 of google's > main index. > This reaffrims the point that even if the page is in the supplemental > index it can still appear in the main Google index and i must apologise > to Adam for my previous bleetings. (except the one about speeding up > the re-examination of supplemental pages)
Dude that IS NOT THE CASE with most people and sure's hell is not the case in my......well, case. You are SO LUCKY if you're seeing that with your URL's! I can search for all my search phrases and (at **BEST**) see my MINOR page(s) showing up INSTEAD, pages that NO ONE would ever click on in the SERP's because they don't even have the freakin' PRODUCT in the SERP's snip, and they are almost NON RELEVANT to the search! Usually my page(s) won't even show up AT ALL!! Then I have to go to page 10, then 20, then 30, etc., all the way to the VERY END of the go*damn results then click that "line from hell": "In order to show you the most relevant results, we have omitted some entries very similar to the X already displayed. If you like, you can repeat the search with the omitted results included."
Supplemental is supposed to be separate, so I don't see how you're seeing any of your pages LABELED "supplemental result" mixed in the main index's results. I'm not saying you're not, I believe you, I just don't comprehend HOW. Are you SURE what you're seeing is labeled as "supplemental result"?
> To return to affiliate marketing, it has gained a bad reputation > (rightly or wrongly) firstly as it was considered to be a major source > of e-mail spam and seconly many affiliates create large amounts of > autogenerated webpages each devoted to different competitive keywords, > which is the reason, as i understand Google apply a stricter criteria. > >From my perspective the only way to make it work is to set up your own > network and carefully police everyone who joins that network, making > sure that content is not duplicated throughout that network. I may be > talking totally complete rubbish as i have no personal experience of > affiliate marketing except avoiding any web-page festooned with adverts.
That's a lot of BS one would have to go through isn't it, just to be POSSIBLY, MAYBE, indexed by "the Google God"! Site owners are just going to give up, and that's going to leave G with NO users eventually because word will, and IS getting out, that they are NOT indexing CORRECTLY what they SHOULD be indexing and users are going to start dropping them like soiled diaper. It used to be the OTHER SE's could learn a lot from G. Now, it's GOOGLE that could learn a lot from the OTHER SE's!
