I had a quick question I wanted to get some clarification on from Google please. We want to allow Google to crawl and index our paid members' pages of our website which has a lively forum in it. It is has controlled access via cookies so only people that are paid and members can access it.
We see www.webmasterworld.com run a similar setup whereby Google is allowed into their site and index's it even thought it is a paid, cookie controlled forum. Could you please clarify if we are allowed to show Google these pages and get them indexed even though when the generally public click on them in the SERP's they will be taken to the homepage if they don't have the correct cookie.
Is this classed as cloaking? It would be pretty easy with my script to set it via useragent to let Google crawl my members forum, and then set the nocache option - but isn't that against your TOS or is this ok to do?
If it is can you please comment on how Webmasterworld.com get away with it and what method they use to make this legal thanks (a simple search of webmaster world + linking) will show indexed pages but when you click on them (assuming you don't have their cookie set) you will be taken to the "pay up" page.
While I can't comment on any other sites in this situation, I can say that it is generally not possible for sites to show Googlebot one page and disallow users from accessing those pages without payment and still adhere to our Webmaster guidelines.
On a happier note, my colleagues and I are working on an arrangement which I think you'll be pleased with... balancing many Webmasters' interest in requiring community membership or signin to content-rich pages while still showing content in Google's search results. Stay tuned :) (we'll make an announcement in the Webmaster Central blog)
> While I can't comment on any other sites in this situation, I can say > that it is generally not possible for sites to show Googlebot one page > and disallow users from accessing those pages without payment and still > adhere to our Webmaster guidelines.
> On a happier note, my colleagues and I are working on an arrangement > which I think you'll be pleased with... balancing many Webmasters' > interest in requiring community membership or signin to content-rich > pages while still showing content in Google's search results. Stay > tuned :) (we'll make an announcement in the Webmaster Central blog)
> Regards, > Adam
Quoting your "happier note" aren't you just saying that what I said above will be legal then? Becuase that is exactly what it is doing.
Even if you can't comment on a partical site there MUST be a legal way to do what I am asking as the site I mention does it (and has been around for years). Can you please comment on a legal way to do what I am asking thanks.
> While I can't comment on any other sites in this situation, I can say > that it is generally not possible for sites to show Googlebot one page > and disallow users from accessing those pages without payment and still > adhere to our Webmaster guidelines.
> On a happier note, my colleagues and I are working on an arrangement > which I think you'll be pleased with... balancing many Webmasters' > interest in requiring community membership or signin to content-rich > pages while still showing content in Google's search results. Stay > tuned :) (we'll make an announcement in the Webmaster Central blog)
> Regards, > Adam
"Stay tuned :)" - how long away before we can expect this please?
Adam Lasnik wrote: > While I can't comment on any other sites in this situation, I can say > that it is generally not possible for sites to show Googlebot one page > and disallow users from accessing those pages without payment and still > adhere to our Webmaster guidelines.
Hi Adam (if you still read this thread)
I have to disagree with you about your guidelines. It doesn't break any of your guidelines when a website, such as WebmasterWorld, allows googlebot to crawl parts (or all) of a site that are disallowed to people who are not registered. As long as a site's USERS receive the same content as googlebot, there is no breach of the guidelines. When people are outside the site (e.g. in Google's site, looking at the search results), they are not the site's users and there is no need to deliver the same pages to them that googlebot received.
However, there is every reason for a search engine NOT to list pages that the engine's users cannot go directly to by clicking a link in the serps - at least not without marking them as such, which may be what your announcement will be about.
I still can't understand why I can reach any Webmasterworld public thread listed in Google's index when I am sitting on a cybercafe in Illinois but when I am in my personal computer I am required to log in to read any topic, even foo. That is, coming from Google's listings. Heck, I can even read the public threads sitting at the cybercafe around the corner without having to log in and be tracked!
What have they done to my computer that when searching Google and clicking on any result for webmasterworld I always have to log in? Have heard about ISPs and spam but the cyber around the corner uses the same ISP as myself. I find it unfair that Google offers me -possibly- relevant information from that forum which I then cannot access freely.
As a user this is a poor experience and makes Google -and all other search engines- look bad. It could just be something that such forum is doing but unfortunately it makes Google appear guilty by association.
Maybe your accessing that forum from the cyber cafe rides on some repviosu user having actually logged into that forum before and a cookie being placed on that pc.
But I have also found that some forums are only pretending to require a login. Scroll down past that notice and past the ads and you're in.
I'm sure that webado is correct. Somebody must have logged in from the cafes and the computers have WMW's cookie, which automatically logs you in. You ought to be able to see a username.
I didn't know that scrolling below the login form allowed you to get in. I seem to remember that the logged-in-only started just with the Google forum (can't guarantee my memory though), because that's where the bulk of the traffic goes, so maybe they have developed it a bit since then.
How about this for a guess as to what the announcement will be:-
They will come up with a new meta tag that can be put on pages that are behind closed doors, so that Google can mark them in the serps. The meta tag will have two possible values to indicate subscription required, and registration required, and the listings in the serps will state "subscription required" or "registration required". That's my best guess, and I'd put a little bit of money down to back it :)
PhilC, I don't know how it works, but how do they do the subscription required when using the news search? Do they use a tag, or is it a domain wide sign-up? Either way it's a good work around for everybody. The forums that use conditional IP serving don't do it to fool google, or probably even MSN or Yahoo, they do it to find the Bad bots, the scrapers and content stealers out there.
The only downside I see, and perhaps it isn't a big deal is that you've created another industry of harvesting email addresses and names that can be highly specialized. Jenny's Dog Grooming Service may not be able to make any money on the web, but building a list of people interested in dog grooming may. Some are more valuable than others in that case, the New York times list is pretty big cross section of the world but in MWM the majority of the users are computer savy, internet wise etc. Of course the answer to this is caveat emptor, if Google marks the site as requiring a subcription, and you choose to subscribe with a real email address, then you take the risk. It amazes me in my ecommerce site that people will trust the site enough to send thousands of dollars via credit card, but usually still use a throw away yahoo or hotmail email address.
Here's my parsing of the response, reading between the PR (not pagerank) lines.
Adam Lasnik wrote:
> While I can't comment on any other sites in this situation,
-Not sure if he meant CAN'T or actually WON'T. Generally in this forum google employees don't respond to specific requests, i.e) that site is banned because they did this. But they try to answer in generalizations to help as many as possible. -On the other hand, if subcription redirection is allowed on a few hand selected sites, which raises all kinds of concerns regarding equal access, then there may be a policy to not comment.
>I can say > that it is generally not possible for sites to show Googlebot one page > and disallow users from accessing those pages without payment and still > adhere to our Webmaster guidelines.
-What wasn't said in this line was "show Googlebot one page and people a different page" which is the traditional definition of cloaking, which I am going to have to assume is really really against the webmast guidelines. -"Payment" seems to be the critical word here. It doesn't say logging in, without cookies, using a MAC, or using firefox, he just said requiring Payment. It would make sense that they don't want to offer conversions to a site owner with the Google snippets being the free preview tool. Does this also mean the converse is true? If payment is not required can you disallow users from accessing those page with a sign up? -In this case I am considering the pronoun "users" to be users of google, not the site returned in the Serp. Google probably still considers them their user on the first page, as that is the page they sent them to and the "user's" experience is still based on the recommendation from google. If the person clicks to the next page, then they are now considered the site's user. -Doing all of this will result in you not adhering to the Webmaster Guidelines. What happens then is up to google, could be a site wide ban, could be those pages dropped from the index, could be nothing.
> On a happier note, my colleagues and I are working on an arrangement > which I think you'll be pleased with... balancing many Webmasters' > interest in requiring community membership or signin to content-rich > pages while still showing content in Google's search results. Stay > tuned :) (we'll make an announcement in the Webmaster Central blog)
- We've now switched from "payment" to "sign-in" two very different concepts. There will have to be some specific guidelines and review on sign-in page. It would be too easy for the shad web designer to have three screens deep of sales pitch for the paid membership and a link buried down in the bottom, slighty hidden, for the free access to just the page returned in the Serp.
JLH wrote: > PhilC, I don't know how it works, but how do they do the subscription > required when using the news search? Do they use a tag, or is it a > domain wide sign-up?
I never use the news search, but I just had a quick look, and it looks like the news providers provide feeds to Google, and, if a page is subscription/registration only, the info will be in the feed.
> Either way it's a good work around for everybody. > The forums that use conditional IP serving don't do it to fool google, > or probably even MSN or Yahoo, they do it to find the Bad bots, the > scrapers and content stealers out there.
They do it to handle bad bots, but they allow googlebot through where people need to log in (or have the cookie). The real issue, imo, is not the websites - it's whether or not search engines should knowingly allow pages to be listed in the serps that people cannot go directly to by clicking on the listing. That's what gets up people's noses. They claim it to be spam when it isn't, because they want it to be wrong (spam).
It has been discussed at length in various places for quite a long time, and I do think that Google has listened.
Adam wrote: > I can say > that it is generally not possible for sites to show Googlebot one page > and disallow users from accessing those pages without payment and still > adhere to our Webmaster guidelines.
and you commented:
> What wasn't said in this line was "show Googlebot one page and people > a different page" which is the traditional definition of cloaking, > which I am going to have to assume is really really against the webmast > guidelines.
Cloaking is against Google's guidelines, but what the sites in question do isn't cloaking. The actual guideline is:
"Don't ... present different content to search engines than you display to users, which is commonly referred to as cloaking."
But users of those sites DO receive the same content as googlebot, so it isn't cloaking. For a fuller explanation read http://www.webworkshop.net/blog/?p=17
I actually disagreed with Adam about the part you quoted, because there isn't anything in the guidelines to back up his statement.
Anywhere I go, even foreign countries I am able to READ all public threads just by clicking on serps from google as a GUEST. Verified that nobdoy else had been logged before and made sure there was that GUEST notice on top. And after loging in, that computer would turn bad and always require login thereafter. Even if all cookies had been erased. Is WMW changing the registry of logged computers? I once changed my home computer and I was able to access public threads from clicking on serps normally. It was only after logged in for the first time this started to happen. At that time I did see for just a nanosecond a DOS shell opening in a flash and my first thought was... they just ran a script on me.
Webado, I am sorry but I can't pass the login/subscribe notice when clicking on Google's results for their public/free listings. All I see is the log in box or subscribe notice and then the link of the URL I was supposed to reach but when clicking on it, it yields a 404. And I tried many other serps from Google always with the same issue. Briefly, I can search Google all day but unless I log in I can't read what Google offers as free information and wmw tags as public.
I don't know, I have only found one site where they claim you need to be logged in to see the answers, at the top of the page, yet I can go right down lower and see all of it. I can't remember the name of the site. Experts something.
WMW doesn't seem to need any login even though they say so - I haven't run into any spot where I had to log in. Can somebody give me a concrete example?
Maybe they go by visitor country or browser or something and decide who gets it and who doesn't for free.
If you delete your cookies, you'll find that WMW will stop you at a login page when you try to open threads. You must be registered there, and have a cookie.
Experts exchange I get to see all answers by scrolling down the page. I've never registered there. I wouldn't do it if they paid me let alone me pay LOL
WMW - I am not a member there either. Don't think I've ever felt the need to sign up there.
Cookies are all gone from all computers. At work: they get cleaned up when I close my session. At home I have recently deleted all cookies and temp files.
I simply don't need to login. Unless there is one particular area.
Have you tried going into an actual thread? Most of the sub-forums at WMW allow you to go as far as the thread listings, but not to see an actual thread. Try the Google News forum.
Featured Home Page Discussion This 43 message thread spans 2 pages: < < 43 ( 1 [2] )
Adam Lasnik on Duplicate Content
mattg3
#:3193314 3:24 pm on Dec. 19, 2006 (utc 0)
I disagree.
I was talking about a Wiki not a review site. I think most of it is taken care of at least in Mediawiki that it will lead to a script. Though some users might type the url directly and a sitemap cronjob would pick these up. I have never seen ie Wikipedia history pages and index.php?bla files in Google so I hope the topic is hopefully mute.
But we see again life is more complex and opinions differ.
LOL, no but maybe checking the ip and from there getting a geo-location and deciding that they'll request memebership for certain location and not others. The fact remains that when I visit from home or from work (different IP's and ISP's) I am not faced with the requirement of logging in - I am in. Obviously I cannot post but I can read.
I see the login page all the time, webado. It's a real pain. I regularly dump my cookies (that sounds bad, lol) and it constantly means logging back into wmw whenever I accidentally click one of their entries in the search results ("accidentally" because I try to avoid it exactly for that reason).
Besides the pain of having to log in, I see their page mainly as a way to "con" newbies into "subscribing" for something that's available for free and that is advertised in Googles search results. Why can't they just buy Adwords like everyone else? Why do they get free advertising? Why is (here comes Phil again :-)) cloaking bad but WMW and some other sites like the NYT tolerated? Don't tell me it is the same content: the page that it goes to *does not* have it on there. And don't say it's for free -- if even Matt Cutts had to do a post on how to register for WMW for free (albeit giving them your email address and a password), many people will have paid for access on accident.
The main problem I see with this whole story is that newbie webmasters are already thinking that "large companies" have bought their search rankings or are getting a special treatment on Googles side. It's very hard to convince them that it isn't so (at least not directly :-)). This story shows that yes - you can go against the webmaster guidelines IF "Google likes your site" (for whatever reasons).
It sends webmasters the wrong signals: cheat your way to the top and if you're lucky, Google will cover you.
"Following these guidelines will help Google find, index, and rank your site. Even if you choose not to implement any of these suggestions, we strongly encourage you to pay very close attention to the "Quality Guidelines," which outline some of the illicit practices that may lead to a site being removed entirely from the Google index (*)."
The footnote is missing on the page: "(*) unless your site is WMW, NYT, or some other hand-picked sites we like"
|
