Hi Scott. Google does use algorithms and different techniques to block
excessive automated queries and scraping, especially when a someone is
hitting Google quite hard. The reason is that scraping consumes server
resources. We don't want real users to be slowed down or affected just
because a bot is sending bunches of automated queries to Google.
We do turn off a number of tools/bots/IP addresses that scrape us too
heavily. It's a common enough phenomenon that we did a blog post on
Google's Online Security Blog about the subject:
http://googleonlinesecurity.blogspot.com/2007/07/reason-behind-were-s...
. In fact, I know that just a week or so ago our algorithms turned off
an IP belonging to one of the entities that you mentioned in your
post.
In general, I would approach the bizdev folks at Google about how to
send automated queries to Google with permission. Failing that, be
aware that if a tool sends too many queries to Google, we do reserve
the right to disable the IP address(es) of that tool. One thing I
would *not* recommend is that if a tool is blocked for bad behavior,
trying to make the tool more "sneaky" (e.g. trying to make the tool
look closer to a web browser). Attempts to fake out Google and pretend
to be more like a web browser (after you've been blocked once already)
is an example of the sort of thing that is really bad in our opinion.
Hope that helps,
Matt Cutts
On Feb 6, 9:32 am, Scott Goodyear wrote:
> I'm not really expecting a reply but if there is one I'd appreciate
> it.
> So, I've been reading about how automated queries related to Google,
> Yahoo, etc.http://blog.wired.com/business/2007/11/google-blocks-y.html
> Ideas to open data to the public.http://code.google.com/apis/opensocial/
> Problem when search engine robots which perform programmatic queries
> on other engines, sites...http://www.davidnaylor.co.uk/yahoo-and-google-oops.html
> and how Google wants to encourage Net Neutrality:http://www.google.com/help/netneutrality.html
> How does this jive with the Google terms of service:
> "Don't use unauthorized computer programs to submit pages, check
> rankings, etc. Such programs consume computing resources and violate
> our Terms of Service. Google does not recommend the use of products
> such as WebPosition Gold(tm) that send automatic or programmatic queries
> to Google."http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=3...
> There are many well known SEO/SEM sites like SEOMoz, JimBoykin's site,
> etc. that offer automated tools that search against Google. Are they
> authorized or not? How does one contact Google to become an authorized
> program or create more Google friendly apps? I've been unsuccessful in
> trying to contact some one at Google on this.
> Thanks for any replies!
> Scott
|
