I own a website and recently have been getting tons of (bots?
spammers?)
visitors from the country of Africa. I do not want anyone outside of
America
visiting my site.
I want to ban the entire continent of Africa, but am wary of going to
such sites as:
Donwload an ip-to-country database, install it (maintain it regularly)
and capture every ip that comes, look it up in the database, and
decide what you want to do.
All this needs to be done with server-side scripting added to all your
pages.
Then live with the consequences of blocking visitors based on possibly
outdated, erroneous information. At any moment maybe 5% of IP's may
be reported wrong.
> Donwload an ip-to-country database, install it (maintain it regularly)
> and capture every ip that comes, look it up in the database, and
> decide what you want to do.
> All this needs to be done with server-side scripting added to all your
> pages.
> Then live with the consequences of blocking visitors based on possibly
> outdated, erroneous information. At any moment maybe 5% of IP's may
> be reported wrong.
> On Jun 28, 7:28 pm, Autocrat wrote:
> > What type of Server?
> > Are you using a 'dynamic' site?
> > Have you asked your host about firewalling your site or installing
> > protection scripts?
> > Have you looked in to anti-spam-bot and badbot scripts?
> I am most worried about accidentally banning GoogleBot.
> Is there a comprehensive list of GoogleBot IP addresses so I can
> whitelist them?
> Or how do I set up my .htaccess file to allow GoogleBot so I can go on
> a banning spree.
> On Jun 28, 5:22 pm, webado wrote:
> > Donwload an ip-to-country database, install it (maintain it regularly)
> > and capture every ip that comes, look it up in the database, and
> > decide what you want to do.
> > All this needs to be done with server-side scripting added to all your
> > pages.
> > Then live with the consequences of blocking visitors based on possibly
> > outdated, erroneous information. At any moment maybe 5% of IP's may
> > be reported wrong.
> > On Jun 28, 7:28 pm, Autocrat wrote:
> > > What type of Server?
> > > Are you using a 'dynamic' site?
> > > Have you asked your host about firewalling your site or installing
> > > protection scripts?
> > > Have you looked in to anti-spam-bot and badbot scripts?- Hide quoted text -
Blocking all users outside of the US from being able to see your site
would likely be considered cloaking and would be against our Webmaster
Guidelines. Instead of blocking these users automatically, I would
recommend that you add blocks based on the user's activity, not based
on his location.
> I want to ban the entire continent of Africa, but am wary of going to
> Anyone (or any Google employee) suggest a great way to ban foreign
> countries (china, india, korea, russia, africa etc) from coming to a
> site?
Ultimately, you can't - they can always use US-located proxies.
> Blocking all users outside of the US from being able to see your site
> would likely be considered cloaking and would be against our Webmaster
> Guidelines. Instead of blocking these users automatically, I would
> recommend that you add blocks based on the user's activity, not based
> on his location.
> Hope it helps!
> John
Sorry JohnMu, a 403 error hardly constitutes cloaked "content". Very
poor advice. Hopefully, a senior Google employee will set the record
staight.
Msnrub, ban the hell out of Africa if you want. Even if Google cared
they'd never know anyhow.
> a 403 error hardly constitutes cloaked "content".
I'm not a Google employee but, if you block users outside the US but
not Googlebots visiting from data centers world-wide it's cloaking.
For example if a Googlebot located in Africa visits your site, it
should see the same content users in Africa see. I think the key here
is to treat Googlebot the same as any other user.
> > a 403 error hardly constitutes cloaked "content".
> I'm not a Google employee but, if you block users outside the US but
> not Googlebots visiting from data centers world-wide it's cloaking.
> For example if a Googlebot located in Africa visits your site, it
> should see the same content users in Africa see. I think the key here
> is to treat Googlebot the same as any other user.
There are no Googlebot's located in Africa. Like it would matter
anyhow. A 403 Forbidden error is not content. Webmaster Guidelines are
clear- do not cloak content!
> There are no Googlebot's located in Africa. Like it would matter
> anyhow. A 403 Forbidden error is not content. Webmaster Guidelines are
> clear- do not cloak content!
Sure, there may not be Googlebot's located in Africa currently that
was just an example.
To recap, Mrsnrub wants to block users in Africa (potentially also
"china, india, korea, russia") but not Googlebot in any of those
areas.
> I want to ban the entire continent of Africa, but am wary of going to
> such sites as:
> Because I am fearful that someone may include GoogleBot's IP address
> in one of the lists
> and I would unwillingly ban GoogleBot from coming to my site.
> Anyone (or any Google employee) suggest a great way to ban foreign
> countries (china, india, korea, russia, africa etc) from coming to a
> site?e
This presents a potential for being detected both as cloaking and post-
cloaking. The cloaking happens when Googlebot sees content but users
in the same ip range don't. Idea being no content isn't the same
content. Post-cloaking happens when users click on search results but
are served different content than Googlebot received from the same
URL.
Either way, it seems returning content to Googlebot but no content to
users is the same as cloaking by user agent.
"Serving up different results based on user agent may cause your site
to be perceived as deceptive and removed from the Google index."
This reply from Google does not make any sense.
Cloacking is when you show different content to Google than the
content you show your users.
It is done to make the search engine think that you have content that
is different from the content you actually show your users.
the simple example is to show the search engine "free ipod" and show
the people adult entertainment.
if all Europe requests are redirected to a page that says "This site
is not for your region, please don't come back",
there is nothing wrong with doing that and if Google thinks it is
wrong that's too bad.
> if all Europe requests are redirected to a page that says "This site
> is not for your region, please don't come back",
> there is nothing wrong with doing that and if Google thinks it is
> wrong that's too bad.
Which leaves _you_ precisely where?
You think you stand _any_ chance of changing Google's mind?
Hi Key_Master,
I don't think that would be fair to users around the world who use our
services and rely on them to get results that are useful to them.
If you have pages which you want to limit to a small population of
users, then I do not think it would make sense to include these in the
world-wide results in the search engines. You're free to put those
pages online of course. However, if we realize that pages in our index
are useless to a part of our users, we'll likely take action to
provide other, more useful results for our users.
> Hi Key_Master,
> I don't think that would be fair to users around the world who use our
> services and rely on them to get results that are useful to them.
> If you have pages which you want to limit to a small population of
> users, then I do not think it would make sense to include these in the
> world-wide results in the search engines. You're free to put those
> pages online of course. However, if we realize that pages in our index
> are useless to a part of our users, we'll likely take action to
> provide other, more useful results for our users.
> John
The "useless to a part of our users" part of your reply is a thorn in
my side. It could apply to any visitor, or any ISP, that has been
banned from a site. So basically what you are saying, webmasters are
required to provide visitors full access to their site regardless of
which country they come from or what the visitors intent is? That
wasn't Google's position when they banned a large swath of Comcast
users a couple of years back.
By the way, Google alters (cloaks) search engine results for China.
Doesn't that means Google is in violation of their own guidelines?
Googlers in glass houses shouldn't throw stones.
You're obviously free to limit access to your site in whatever way you
see fit. However, if you're trying to show URLs from your site in our
search results, we ask that users are able to see what is being
promised in the title and snippet after clicking on the result.
To me, that makes sense not only for us but also for the webmaster:
which user is going to want to come back to a site that sends him away
after being enticed to visit through a search result? How can you be
so sure that this user is not someone who might be interested in
buying something, perhaps a user who happens to have gotten an IP
address mapped to a particular country in your IP/Location database or
perhaps someone just traveling?
The situation is obviously different when certain users are blocked
for legal reasons, that's something we both can't influence. However,
just blocking all users outside of the USA (as mentioned in the
original post) generally does not fall into that category.
As someone living in Europe where I often see websites that either
block my access, do not allow me to view product pages or redirect me
to some other trimmed down & useless "international" version, it
personally bothers me to see that this is something that is still
seriously being considered.
> You're obviously free to limit access to your site in whatever way you
> see fit. However, if you're trying to show URLs from your site in our
> search results, we ask that users are able to see what is being
> promised in the title and snippet after clicking on the result.
John, I'm afraid you're dealing with a big issue here, which if your
answers were true, puts Google on the evil side
of the coin.
There are myriad of reason why a site should be blocking IP addresses
based on Geo location.
The first that comes to my mind is an US advertiser paying commission
to affiliates for their visitors filling out some forms.
If the business is targeting particular location it is natural that
the form itself would be accessible only from that location.
It is just one of tens, if not hundreds of other legitimate reasons to
use IP blocking/redirecting and it has nothing to do with cloaking.
If Google is unable to send bots from that particular location then
Google is having an algo problem. It's not right to downgrade the
rankings
globally for a site who blocks particular range of IP addresses, thus
including eventual Google bot.
It is perfectly OK not to show the site in results for that location,
but to 'punish' the site for doing it would be very wrong.
I think everyone would appreciate the 'second opinion' from your team.
Thanks.
To quote Matt Cutts (http://www.mattcutts.com/blog/a-quick-word-about- cloaking/) : "Cloaking is serving different content to users than to
search engines."
Maile Ohye states on The Official Google Webmaster blog (http://
googlewebmastercentral.blogspot.com/2008/06/how-google-defines-ip-
delivery.html) :
"Geolocation: Serving targeted/different content to users based on
their location. As a webmaster, you may be able to determine a user's
location from preferences you've stored in their cookie, information
pertaining to their login, or their IP address. For example, if your
site is about baseball, you may use geolocation techniques to
highlight the Yankees to your users in New York.
The key is to treat Googlebot as you would a typical user from a
similar location, IP range, etc. (i.e. don't treat Googlebot as if it
came from its own separate country—that's cloaking)."
I have a client for whom I wrote a simple script to make their contact
page only throw an error to a specific country because of spam. This
client does not have any interest in serving clients from this
particular country and receives an extremely large amount of "we want
to offer your business our services" type emails everyday. I believe
that this fits in the Geolocation guideline, not the cloaking
guideline since we do not make any distinction between a spider using
those IP ranges or the Googlebot, individual users etc. Please advise
if I am reading the webmaster guidelines and Official Blog
incorrectly.
> You're obviously free to limit access to your site in whatever way you
> see fit. However, if you're trying to show URLs from your site in our
> search results, we ask that users are able to see what is being
> promised in the title and snippet after clicking on the result.
> To me, that makes sense not only for us but also for the webmaster:
> which user is going to want to come back to a site that sends him away
> after being enticed to visit through a search result? How can you be
> so sure that this user is not someone who might be interested in
> buying something, perhaps a user who happens to have gotten an IP
> address mapped to a particular country in your IP/Location database or
> perhaps someone just traveling?
> The situation is obviously different when certain users are blocked
> for legal reasons, that's something we both can't influence. However,
> just blocking all users outside of the USA (as mentioned in the
> original post) generally does not fall into that category.
> As someone living in Europe where I often see websites that either
> block my access, do not allow me to view product pages or redirect me
> to some other trimmed down & useless "international" version, it
> personally bothers me to see that this is something that is still
> seriously being considered.
After a bit of double-checking, I have a clarification where I was
mistaken. Sorry about the confusion!
The important part is that you do not treat the Googlebots any
different than other users from that region. So if your site blocks
users in the region where the Googlebot comes from (based on the IP
address and your IP/Location lookups), you should be blocking it as
well. Blocking users outside of the Googlebot's region would generally
be ok.
For more questions, feel free to post here, thanks!
> After a bit of double-checking, I have a clarification where I was
> mistaken. Sorry about the confusion!
> The important part is that you do not treat the Googlebots any
> different than other users from that region. So if your site blocks
> users in the region where the Googlebot comes from (based on the IP
> address and your IP/Location lookups), you should be blocking it as
> well. Blocking users outside of the Googlebot's region would generally
> be ok.
> For more questions, feel free to post here, thanks!
> well. Blocking users outside of the Googlebot's region would generally
> be ok.
Thanks for making that clear and maybe there is confusion, I never
took blocking outside a Googlbot's region as being the question. I
took the original question as asking if it was ok to treat Googlebot
differently than another user visiting from the same IP / same
region? So, sorry if I've added to anyone's confusion but I'm pretty
sure that was the question being asked?
------------------
Either way, saw the above on my way to post the following:
>There are myriad of reason why a site should be blocking IP addresses
>based on Geo location.
bihonline, I don't think anyone is saying a site shouldn't block IP
addresses
based on Geo location.
>I have a client for whom I wrote a simple script to make their contact
>page only throw an error to a specific country because of spam.
Assuming Googlebot could visit your site from the country being
blocked based on the IP and/or range being blocked, would Googbot be
blocked just like any other user?
If the answer is yes, I'm pretty certain you're fine! As you point
out nullvariable, Google says "The key is to treat Googlebot as you
would a typical user from a similar location, IP range, etc."
> > well. Blocking users outside of the Googlebot's region would generally
> > be ok.
> Thanks for making that clear and maybe there is confusion, I never
> took blocking outside a Googlbot's region as being the question. I
> took the original question as asking if it was ok to treat Googlebot
> differently than another user visiting from the same IP / same
> region? So, sorry if I've added to anyone's confusion but I'm pretty
> sure that was the question being asked?
> ------------------
> Either way, saw the above on my way to post the following:
> >There are myriad of reason why a site should be blocking IP addresses
> >based on Geo location.
> bihonline, I don't think anyone is saying a site shouldn't block IP
> addresses
> based on Geo location.
> >I have a client for whom I wrote a simple script to make their contact
> >page only throw an error to a specific country because of spam.
> Assuming Googlebot could visit your site from the country being
> blocked based on the IP and/or range being blocked, would Googbot be
> blocked just like any other user?
> If the answer is yes, I'm pretty certain you're fine! As you point
> out nullvariable, Google says "The key is to treat Googlebot as you
> would a typical user from a similar location, IP range, etc."
|
