Dear Google

I had a quick question I wanted to get some clarification on from
Google please.
We want to allow Google to crawl and index our paid members' pages of
our website which has a lively forum in it. It is has controlled access
via cookies so only people that are paid and members can access it.

We see www.webmasterworld.com run a similar setup whereby Google is
allowed into their site and index's it even thought it is a paid,
cookie controlled forum.
Could you please clarify if we are allowed to show Google these pages
and get them indexed even though when the generally public click on
them in the SERP's they will be taken to the homepage if they don't
have the correct cookie.

Is this classed as cloaking?  It would be pretty easy with my script to
set it via useragent to let Google crawl my members forum, and then set
the nocache option - but isn't that against your TOS or is this ok to
do?

If it is can you please comment on how Webmasterworld.com get away with
it and what method they use to make this legal thanks (a simple search
of webmaster world + linking) will show indexed pages but when you
click on them (assuming you don't have their cookie set) you will be
taken to the "pay up" page.

Look forward to your response.

aff_developer

Hey aff_developer,

While I can't comment on any other sites in this situation, I can say
that it is generally not possible for sites to show Googlebot one page
and disallow users from accessing those pages without payment and still
adhere to our Webmaster guidelines.

On a happier note, my colleagues and I are working on an arrangement
which I think you'll be pleased with... balancing many Webmasters'
interest in requiring community membership or signin to content-rich
pages while still showing content in Google's search results.  Stay
tuned :) (we'll make an announcement in the Webmaster Central blog)

Regards,
Adam

Quoting your "happier note" aren't you just saying that what I said
above will be legal then? Becuase that is exactly what it is doing.

Even if you can't comment on a partical site there MUST be a legal way
to do what I am asking as the site I mention does it (and has been
around for years). Can you please comment on a legal way to do what I
am asking thanks.

"Stay tuned :)" - how long away before we can expect this please?

any update on when this will be happening?

still no news on this :(

Hi Adam (if you still read this thread)

I have to disagree with you about your guidelines. It doesn't break any
of your guidelines when a website, such as WebmasterWorld, allows
googlebot to crawl parts (or all) of a site that are disallowed to
people who are not registered. As long as a site's USERS receive the
same content as googlebot, there is no breach of the guidelines. When
people are outside the site (e.g. in Google's site, looking at the
search results), they are not the site's users and there is no need to
deliver the same pages to them that googlebot received.

However, there is every reason for a search engine NOT to list pages
that the engine's users cannot go directly to by clicking a link in the
serps - at least not without marking them as such, which may be what
your announcement will be about.

I still can't understand why I can reach any Webmasterworld public
thread listed in Google's index when I am sitting on a cybercafe in
Illinois but when I am in my personal computer I am required to log in
to read any topic, even foo. That is, coming from Google's listings.
Heck, I can even read the public threads sitting at the cybercafe
around the corner without having to log in and be tracked!

What have they done to my computer that when searching Google and
clicking on any result for webmasterworld I always have to log in? Have
heard about ISPs and spam but the cyber around the corner uses the same
ISP as myself. I find it unfair that Google offers me -possibly-
relevant information from that forum which I then cannot access freely.

As a user this is a poor experience and makes Google -and all other
search engines- look bad. It could just be something that such forum is
doing but unfortunately it makes Google appear guilty by association.

Maybe your accessing that forum from the cyber cafe rides on some
repviosu user having actually logged into that forum before and a
cookie being placed on that pc.

But I have also found that some forums are only pretending to require a
login. Scroll down past that notice and past the ads and you're in.

I'm sure that webado is correct. Somebody must have logged in from the
cafes and the computers have WMW's cookie, which automatically logs you
in. You ought to be able to see a username.

I didn't know that scrolling below the login form allowed you to get
in. I seem to remember that the logged-in-only started just with the
Google forum (can't guarantee my memory though), because that's where
the bulk of the traffic goes, so maybe they have developed it a bit
since then.

How about this for a guess as to what the announcement will be:-

They will come up with a new meta tag that can be put on pages that are
behind closed doors, so that Google can mark them in the serps. The
meta tag will have two possible values to indicate subscription
required, and registration required, and the listings in the serps will
state "subscription required" or "registration required". That's my
best guess, and I'd put a little bit of money down to back it :)

PhilC, I don't know how it works, but how do they do the subscription
required when using the news search?  Do they use a tag, or is it a
domain wide sign-up?  Either way it's a good work around for everybody.
 The forums that use conditional IP serving don't do it to fool google,
or probably even MSN or Yahoo, they do it to find the Bad bots, the
scrapers and content stealers out there.

The only downside I see, and perhaps it isn't a big deal is that you've
created another industry of harvesting email addresses and names that
can be highly specialized.  Jenny's Dog Grooming Service may not be
able to make any money on the web, but building a list of people
interested in dog grooming may.  Some are more valuable than others in
that case, the New York times list is pretty big cross section of the
world but in MWM the majority of the users are computer savy, internet
wise etc.  Of course the answer to this is caveat emptor, if Google
marks the site as requiring a subcription, and you choose to subscribe
with a real email address, then you take the risk.  It amazes me in my
ecommerce site that people will trust the site enough to send thousands
of dollars via credit card, but usually still use a throw away yahoo or
hotmail email address.

Here's my parsing of the response, reading between the PR (not
pagerank) lines.

-Not sure if he meant CAN'T or actually WON'T.  Generally in this forum
google employees don't respond to specific requests, i.e) that site is
banned because they did this.  But they try to answer in
generalizations to help as many as possible.
-On the other hand, if subcription redirection is allowed on a few hand
selected sites, which raises all kinds of concerns regarding equal
access, then there may be a policy to not comment.

-What wasn't said in this line was "show Googlebot one page and people
a different page" which is the traditional definition of cloaking,
which I am going to have to assume is really really against the webmast
guidelines.
-"Payment" seems to be the critical word here.  It doesn't say logging
in, without cookies, using a MAC, or using firefox, he just said
requiring Payment.  It would make sense that they don't want to offer
conversions to a site owner with the Google snippets being the free
preview tool.  Does this also mean the converse is true?  If payment is
not required can you disallow users from accessing those page with a
sign up?
-In this case I am considering the pronoun "users" to be users of
google, not the site returned in the Serp.  Google probably still
considers them their user on the first page, as that is the page they
sent them to and the "user's" experience is still based on the
recommendation from google.  If the person clicks to the next page,
then they are now considered the site's user.
-Doing all of this will result in you not adhering to the Webmaster
Guidelines.  What happens then is up to google, could be a site wide
ban, could be those pages dropped from the index, could be nothing.

- We've now switched from "payment" to "sign-in" two very different
concepts.  There will have to be some specific guidelines and review on
sign-in page.  It would be too easy for the shad web designer to have
three screens deep of sales pitch for the paid membership and a link
buried down in the bottom, slighty hidden, for the free access to just
the page returned in the Serp.