Hi, this is the 2nd time now in a month that Google's bot is crashing
my server - I read an earler post an SEO101 suggested that we need to
contact Google to report this. How and where do we contact Google to
get them to fix this ASAP as its costing me bandwitdth and stopping
people access my site.
Well, posting here is always a good move... there's a chance one of
the G team will spot it.
Failing that... not idea.
.
Do you know how GBot is crashing your server?
It shouldn't be pulling that much information at one go to cause a
whole server to fall over.
Can you provide details/information from teh logs about crawl times,
bandwidth usage etc.?
.
Additionally, have you tried altering the Crawl Rate through your
Google WebMaster Tools account?
> Hi, this is the 2nd time now in a month that Google's bot is crashing
> my server - I read an earler post an SEO101 suggested that we need to
> contact Google to report this. How and where do we contact Google to
> get them to fix this ASAP as its costing me bandwitdth and stopping
> people access my site.
After the first time I altered the crawl rate to a slower crawl rate
and hoped that would work but this past weekend Google started to
crawl my site and cause my database to crash as it was trying to run
almost 100 simultaneous queries.
Erm....
you are running a 'real dynamic' site?
No caching... no pre-store.... do you even have Query Cache setup?
.
If the above is the case, then you are going to encounter serious
issues.
You do realise how many little bots there are?
What about visitor fluctuations?
Hell, what happens if your site becomes popular?
.
Yes, it could be GBot tipping your server... but I wouldn't say it's
GBots fault... jsut a side-effect of a possibly much larger issue.
Jeeze... it wouldn't even take much of a hacking attempt to keep your
site down... permenantly.
I can only suggest (most seriously) that you think of an alternative
setup/system/application.... and implement it immediately.
Looking at your site and what is happening on our side, I have a few
suggestions that might help make things better on your side.
- Caching, as mentioned by Autocrat, could likely help your site a
bit. However, from what I can tell, your URLs are generated completely
dynamically, so it is hard to tell how long they would be valid (and
cached).
- From a rough count, it looks like you have +/- infinite URLs. In a
situation like that, if we were to crawl an URL every couple of
minutes, it would take a pretty long time to cover your site. That
said, I don't see us crawling anywhere close to 100 (or even 5) URLs
simultaneously. Would it be possible that other crawlers are using the
Googlebot user agent and causing trouble on your site? You can verify
the Googlebot as mentioned in http://www.google.com/support/webmasters/bin/answer.py?answer=80553
- Seeing that you have so many URLs, I would suggest working on ways
to reduce the number of URLs that we should crawl. A simple way to do
this is to use the robots.txt file. For instance, I see lots of URLs
like the following which seem to 302 redirect:
http://www.mykindaprices.com/cgi-bin/mykindaprices/refer.pl?ShopCity:... I'm sure if you were to go through your server logs & website
analytics you could find a large number of URLs that might not be so
important to have listed in search engines. While doing that, I would
suggest using our Webmaster Guidelines as an aid as well. You can find
them at http://www.google.com/support/webmasters/bin/answer.py?answer=35769
With these measures I'm sure you can reduce the number of crawlable
URLs on your site to a manageable amount - which would help us to make
sure that we're crawling everything and help you in that we don't try
to crawl unnecessary URLs.
> Hi, this is the 2nd time now in a month that Google's bot is crashing
> my server - I read an earler post an SEO101 suggested that we need to
> contact Google to report this. How and where do we contact Google to
> get them to fix this ASAP as its costing me bandwitdth and stopping
> people access my site.
> Looking at your site and what is happening on our side, I have a few
> suggestions that might help make things better on your side.
> - Caching, as mentioned by Autocrat, could likely help your site a
> bit. However, from what I can tell, your URLs are generated completely
> dynamically, so it is hard to tell how long they would be valid (and
> cached).
> - From a rough count, it looks like you have +/- infinite URLs. In a
> situation like that, if we were to crawl an URL every couple of
> minutes, it would take a pretty long time to cover your site. That
> said, I don't see us crawling anywhere close to 100 (or even 5) URLs
> simultaneously. Would it be possible that other crawlers are using the
> Googlebot user agent and causing trouble on your site? You can verify
> the Googlebot as mentioned inhttp://www.google.com/support/webmasters/bin/answer.py?answer=80553
> - Seeing that you have so many URLs, I would suggest working on ways
> to reduce the number of URLs that we should crawl. A simple way to do
> this is to use the robots.txt file. For instance, I see lots of URLs
> like the following which seem to 302 redirect:http://www.mykindaprices.com/cgi-bin/mykindaprices/refer.pl?ShopCity:...
> I'm sure if you were to go through your server logs & website
> analytics you could find a large number of URLs that might not be so
> important to have listed in search engines. While doing that, I would
> suggest using our Webmaster Guidelines as an aid as well. You can find
> them athttp://www.google.com/support/webmasters/bin/answer.py?answer=35769
> With these measures I'm sure you can reduce the number of crawlable
> URLs on your site to a manageable amount - which would help us to make
> sure that we're crawling everything and help you in that we don't try
> to crawl unnecessary URLs.
> If you have your website registered with Webmaster Central you can
> reset your crawl rate to "slower" to keep Google from crashing your
> server.
> On Jun 29, 3:56 pm, danbaileyuk wrote:
> > Hi, this is the 2nd time now in a month that Google's bot is crashing
> > my server - I read an earler post an SEO101 suggested that we need to
> > contact Google to report this. How and where do we contact Google to
> > get them to fix this ASAP as its costing me bandwitdth and stopping
> > people access my site.
