How does this jive with the Google terms of service:
"Don't use unauthorized computer programs to submit pages, check
rankings, etc. Such programs consume computing resources and violate
our Terms of Service. Google does not recommend the use of products
such as WebPosition Gold(tm) that send automatic or programmatic queries
to Google."
http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=3...
There are many well known SEO/SEM sites like SEOMoz, JimBoykin's site,
etc. that offer automated tools that search against Google. Are they
authorized or not? How does one contact Google to become an authorized
program or create more Google friendly apps? I've been unsuccessful in
trying to contact some one at Google on this.
This is an interesting topic. I'm not quite sure how all those links
apply to our terms of service, which do not allow these kinds of
automated tools. I wrote a bit about tools like that a short while
ago:
A tool accessing other websites should try to obey the rules set forth
by that website. In general, these rules are described in several
ways:
1. The robots.txt covers which URLs may be accessed and which ones are
disallowed. You'll notice that in our robots.txt we explicitly
disallow "/search", which is what most of the ranking tools generally
try to access.
2. The server result codes give more information when a URL is
accessed. When our network recognizes automated queries, it may return
a result code of 500 or similar.
3. The HTTP headers returned by the server can provide information
through the "x-robots-tag".
4. A HTML page may provide information through a "robots" (or in our
case, "googlebot") meta tag.
As far as I am aware, there are no "SEO-tools" that have permission to
access our web-search results in an automated way. I am also not aware
of any plans to change that in the near future.
Hi Scott. Google does use algorithms and different techniques to block
excessive automated queries and scraping, especially when a someone is
hitting Google quite hard. The reason is that scraping consumes server
resources. We don't want real users to be slowed down or affected just
because a bot is sending bunches of automated queries to Google.
We do turn off a number of tools/bots/IP addresses that scrape us too
heavily. It's a common enough phenomenon that we did a blog post on
Google's Online Security Blog about the subject:
http://googleonlinesecurity.blogspot.com/2007/07/reason-behind-were-s... . In fact, I know that just a week or so ago our algorithms turned off
an IP belonging to one of the entities that you mentioned in your
post.
In general, I would approach the bizdev folks at Google about how to
send automated queries to Google with permission. Failing that, be
aware that if a tool sends too many queries to Google, we do reserve
the right to disable the IP address(es) of that tool. One thing I
would *not* recommend is that if a tool is blocked for bad behavior,
trying to make the tool more "sneaky" (e.g. trying to make the tool
look closer to a web browser). Attempts to fake out Google and pretend
to be more like a web browser (after you've been blocked once already)
is an example of the sort of thing that is really bad in our opinion.
> How does this jive with the Google terms of service:
> "Don't use unauthorized computer programs to submit pages, check
> rankings, etc. Such programs consume computing resources and violate
> our Terms of Service. Google does not recommend the use of products
> such as WebPosition Gold(tm) that send automatic or programmatic queries
> to Google."http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=3...
> There are many well known SEO/SEM sites like SEOMoz, JimBoykin's site,
> etc. that offer automated tools that search against Google. Are they
> authorized or not? How does one contact Google to become an authorized
> program or create more Google friendly apps? I've been unsuccessful in
> trying to contact some one at Google on this.
John, Matt,
Thank you very much for the reply. I've been trying to get some one at
Google to talk to me for a while. OK so bizdev... not so obvious on
Google's site. I'll give that a shot. I could write several essays on
why http://www.google.com/help/netneutrality.html seems to be
paradoxical to "The reason is that scraping consumes server
resources" but I'll leave that to others for now.
> Hi Scott. Google does use algorithms and different techniques to block
> excessive automated queries and scraping, especially when a someone is
> hitting Google quite hard. The reason is that scraping consumes server
> resources. We don't want real users to be slowed down or affected just
> because a bot is sending bunches of automated queries to Google.
> We do turn off a number of tools/bots/IP addresses that scrape us too
> heavily. It's a common enough phenomenon that we did a blog post on
> Google's Online Security Blog about the subject:http://googleonlinesecurity.blogspot.com/2007/07/reason-behind-were-s...
> . In fact, I know that just a week or so ago our algorithms turned off
> an IP belonging to one of the entities that you mentioned in your
> post.
> In general, I would approach the bizdev folks at Google about how to
> send automated queries to Google with permission. Failing that, be
> aware that if a tool sends too many queries to Google, we do reserve
> the right to disable the IP address(es) of that tool. One thing I
> would *not* recommend is that if a tool is blocked for bad behavior,
> trying to make the tool more "sneaky" (e.g. trying to make the tool
> look closer to a web browser). Attempts to fake out Google and pretend
> to be more like a web browser (after you've been blocked once already)
> is an example of the sort of thing that is really bad in our opinion.
> Hope that helps,
> Matt Cutts
> On Feb 6, 9:32 am, Scott Goodyear wrote:
> > I'm not really expecting a reply but if there is one I'd appreciate
> > it.
> > How does this jive with the Google terms of service:
> > "Don't use unauthorized computer programs to submit pages, check
> > rankings, etc. Such programs consume computing resources and violate
> > our Terms of Service. Google does not recommend the use of products
> > such as WebPosition Gold(tm) that send automatic or programmatic queries
> > to Google."http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=3...
> > There are many well known SEO/SEM sites like SEOMoz, JimBoykin's site,
> > etc. that offer automated tools that search against Google. Are they
> > authorized or not? How does one contact Google to become an authorized
> > program or create more Google friendly apps? I've been unsuccessful in
> > trying to contact some one at Google on this.
> John, Matt,
> Thank you very much for the reply. I've been trying to get some one at
> Google to talk to me for a while. OK so bizdev... not so obvious on
> Google's site. I'll give that a shot. I could write several essays on
> whyhttp://www.google.com/help/netneutrality.htmlseems to be
> paradoxical to "The reason is that scraping consumes server
> resources" but I'll leave that to others for now.
> Scott
> On Feb 6, 10:49 pm, Matt Cutts wrote:
> > Hi Scott. Google does use algorithms and different techniques to block
> > excessive automated queries and scraping, especially when a someone is
> > hitting Google quite hard. The reason is that scraping consumes server
> > resources. We don't want real users to be slowed down or affected just
> > because a bot is sending bunches of automated queries to Google.
> > We do turn off a number of tools/bots/IP addresses that scrape us too
> > heavily. It's a common enough phenomenon that we did a blog post on
> > Google's Online Security Blog about the subject:http://googleonlinesecurity.blogspot.com/2007/07/reason-behind-were-s...
> > . In fact, I know that just a week or so ago our algorithms turned off
> > an IP belonging to one of the entities that you mentioned in your
> > post.
> > In general, I would approach the bizdev folks at Google about how to
> > send automated queries to Google with permission. Failing that, be
> > aware that if a tool sends too many queries to Google, we do reserve
> > the right to disable the IP address(es) of that tool. One thing I
> > would *not* recommend is that if a tool is blocked for bad behavior,
> > trying to make the tool more "sneaky" (e.g. trying to make the tool
> > look closer to a web browser). Attempts to fake out Google and pretend
> > to be more like a web browser (after you've been blocked once already)
> > is an example of the sort of thing that is really bad in our opinion.
> > Hope that helps,
> > Matt Cutts
> > On Feb 6, 9:32 am, Scott Goodyear wrote:
> > > I'm not really expecting a reply but if there is one I'd appreciate
> > > it.
> > > How does this jive with the Google terms of service:
> > > "Don't use unauthorized computer programs to submit pages, check
> > > rankings, etc. Such programs consume computing resources and violate
> > > our Terms of Service. Google does not recommend the use of products
> > > such as WebPosition Gold(tm) that send automatic or programmatic queries
> > > to Google."http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=3...
> > > There are many well known SEO/SEM sites like SEOMoz, JimBoykin's site,
> > > etc. that offer automated tools that search against Google. Are they
> > > authorized or not? How does one contact Google to become an authorized
> > > program or create more Google friendly apps? I've been unsuccessful in
> > > trying to contact some one at Google on this.
> Hi Guys,
> For anyone who is lurking, I've put in a few emails and nothing so
> far.
> Scott
> On Feb 7, 8:53 am, Scott Goodyear wrote:
> > John, Matt,
> > Thank you very much for the reply. I've been trying to get some one at
> > Google to talk to me for a while. OK so bizdev... not so obvious on
> > Google's site. I'll give that a shot. I could write several essays on
> > whyhttp://www.google.com/help/netneutrality.htmlseemsto be
> > paradoxical to "The reason is that scraping consumes server
> > resources" but I'll leave that to others for now.
> > Scott
> > On Feb 6, 10:49 pm, Matt Cutts wrote:
> > > Hi Scott. Google does use algorithms and different techniques to block
> > > excessive automated queries and scraping, especially when a someone is
> > > hitting Google quite hard. The reason is that scraping consumes server
> > > resources. We don't want real users to be slowed down or affected just
> > > because a bot is sending bunches of automated queries to Google.
> > > We do turn off a number of tools/bots/IP addresses that scrape us too
> > > heavily. It's a common enough phenomenon that we did a blog post on
> > > Google's Online Security Blog about the subject:http://googleonlinesecurity.blogspot.com/2007/07/reason-behind-were-s...
> > > . In fact, I know that just a week or so ago our algorithms turned off
> > > an IP belonging to one of the entities that you mentioned in your
> > > post.
> > > In general, I would approach the bizdev folks at Google about how to
> > > send automated queries to Google with permission. Failing that, be
> > > aware that if a tool sends too many queries to Google, we do reserve
> > > the right to disable the IP address(es) of that tool. One thing I
> > > would *not* recommend is that if a tool is blocked for bad behavior,
> > > trying to make the tool more "sneaky" (e.g. trying to make the tool
> > > look closer to a web browser). Attempts to fake out Google and pretend
> > > to be more like a web browser (after you've been blocked once already)
> > > is an example of the sort of thing that is really bad in our opinion.
> > > Hope that helps,
> > > Matt Cutts
> > > On Feb 6, 9:32 am, Scott Goodyear wrote:
> > > > I'm not really expecting a reply but if there is one I'd appreciate
> > > > it.
> > > > How does this jive with the Google terms of service:
> > > > "Don't use unauthorized computer programs to submit pages, check
> > > > rankings, etc. Such programs consume computing resources and violate
> > > > our Terms of Service. Google does not recommend the use of products
> > > > such as WebPosition Gold(tm) that send automatic or programmatic queries
> > > > to Google."http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=3...
> > > > There are many well known SEO/SEM sites like SEOMoz, JimBoykin's site,
> > > > etc. that offer automated tools that search against Google. Are they
> > > > authorized or not? How does one contact Google to become an authorized
> > > > program or create more Google friendly apps? I've been unsuccessful in
> > > > trying to contact some one at Google on this.
