Gmail Calendar Documents Reader Web more »
Recently Visited Groups | Help | Sign in
Google Groups Home
Discussions > Crawling, indexing, and ranking > Googlebot eating up my bandwidth
There are currently too many topics in this group that display first. To make this topic appear first, remove this option from another topic.
There was an error processing your request. Please try again.
flag
  3 messages - Collapse all  -  Translate all to Translated (View all originals)
The group you are posting to is a Usenet group. Messages posted to this group will make your email address visible to anyone on the Internet.
Your reply message has not been sent.
Your post was successful
 
From:
To:
Cc:
Followup To:
Add Cc | Add Followup-to | Edit Subject
Subject:
Validation:
For verification purposes please type the characters you see in the picture below or the numbers you hear by clicking the accessibility icon. Listen and type the numbers you hear
 
IBUK  
View profile  
 More options May 22 2008, 6:06 pm
From: IBUK
Date: Thu, 22 May 2008 15:06:33 -0700 (PDT)
Local: Thurs, May 22 2008 6:06 pm
Subject: Googlebot eating up my bandwidth
Hi,

In the last month (April) Googlebot has used up over 130Gb of
bandwidth from my server!  This is really excessive and totally out of
the ordinary.

What could have caused this?


    Forward  
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Autocrat  
View profile  
 More options May 22 2008, 6:13 pm
From: Autocrat
Date: Thu, 22 May 2008 15:13:37 -0700 (PDT)
Local: Thurs, May 22 2008 6:13 pm
Subject: Re: Googlebot eating up my bandwidth
That seems to be an awful lot........

How big is the site?
Are you showing alot of big images etc.?
(Or maybe even huge PDFs?)

Are you sure it's the real 'googleBot' and not a spoof?

Can you see what pages are getting hte most activity... and can you
see anything wrong with those pages that could be cause the bots to
get 'stuck' ?


    Forward  
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Wysz Google employee  
View profile  
 More options May 22 2008, 7:10 pm
From: Wysz
Date: Thu, 22 May 2008 16:10:41 -0700 (PDT)
Local: Thurs, May 22 2008 7:10 pm
Subject: Re: Googlebot eating up my bandwidth
Hi IBUK,

One step you can take is to request a change of Googlebot's crawl rate
of your site using Google Webmaster Tools:
http://www.google.com/support/webmasters/bin/answer.py?answer=48620

To investigate the cause of this increased activity, take a look at
your logs and see if Googlebot is crawling any duplicate URLs. As
JohnMu suggested, these may include URLs containing session IDs or
other parameters not necessary for indexing. Examples of how this
issue can be caused and resolved are documented in the Help Center:
http://www.google.com/support/webmasters/bin/answer.py?answer=76329

As Autocrat suggests, you may also want to confirm that the busy bot
is indeed Googlebot. You can find instructions for doing that here:
http://googlewebmastercentral.blogspot.com/2006/09/how-to-verify-goog...

For more help with your particular site, feel free to post back here
with your URL so members of the group can take a look.

- Wysz

On May 22, 3:46 pm, JohnMu wrote:


    Forward  
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
End of messages
« Back to Discussions « Newer topic     Older topic »

Create a group - Google Groups - Google Home - Terms of Service - Privacy Policy
©2010 Google